Multimedia

Authors and titles for June 2023

[26] arXiv:2306.02623 (cross-list from cs.CV) [pdf, other]

Title: Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models

Jiabang He, Yi Hu, Lei Wang, Xing Xu, Ning Liu, Hui Liu, Heng Tao Shen

Comments: SIGIR 2023. The code and datasets for our Do-GOOD benchmark can be found at this https URL

Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Multimedia (cs.MM)

[27] arXiv:2306.02898 (cross-list from cs.CV) [pdf, other]

Title: Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark

Shuyu Yang, Yinan Zhou, Yaxiong Wang, Yujiao Wu, Li Zhu, Zhedong Zheng

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[28] arXiv:2306.03403 (cross-list from cs.CV) [pdf, html, other]

Title: SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation

Xuewei Li, Tao Wu, Zhongang Qi, Gaoang Wang, Ying Shan, Xi Li

Comments: Accepted by IJCAI 2023

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)

[29] arXiv:2306.03718 (cross-list from cs.SD) [pdf, other]

Title: Emotion-Conditioned Melody Harmonization with Hierarchical Variational Autoencoder

Shulei Ji, Xinyu Yang

Comments: Accepted by IEEE SMC 2023

Subjects: Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

[30] arXiv:2306.04216 (cross-list from cs.CV) [pdf, other]

Title: MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos

Jielin Qiu, Jiacheng Zhu, William Han, Aditesh Kumar, Karthik Mittal, Claire Jin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Ding Zhao, Bo Li, Lijuan Wang

Comments: Project website: this https URL

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[31] arXiv:2306.04321 (cross-list from cs.AI) [pdf, other]

Title: Generative Semantic Communication: Diffusion Models Beyond Bit Recovery

Eleonora Grassucci, Sergio Barbarossa, Danilo Comminiello

Subjects: Artificial Intelligence (cs.AI); Multimedia (cs.MM)

[32] arXiv:2306.04345 (cross-list from cs.CV) [pdf, other]

Title: An Overview of Challenges in Egocentric Text-Video Retrieval

Burak Satar, Hongyuan Zhu, Hanwang Zhang, Joo Hwee Lim

Comments: 4 pages, CVPR 2023 Joint Ego4D&EPIC Workshop, Extended Abstract

Subjects: Computer Vision and Pattern Recognition (cs.CV); Information Retrieval (cs.IR); Multimedia (cs.MM)

[33] arXiv:2306.04628 (cross-list from cs.SD) [pdf, other]

Title: Systematic Analysis of Music Representations from BERT

Sangjun Han, Hyeongrae Ihm, Woohyung Lim

Subjects: Sound (cs.SD); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

[34] arXiv:2306.05268 (cross-list from cs.LG) [pdf, other]

Title: Factorized Contrastive Learning: Going Beyond Multi-view Redundancy

Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, Ruslan Salakhutdinov

Comments: NeurIPS 2023. Code available at: this https URL

Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[35] arXiv:2306.05523 (cross-list from cs.CL) [pdf, other]

Title: FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering

Megha Chakraborty, Khushbu Pahwa, Anku Rani, Shreyas Chatterjee, Dwip Dalal, Harshit Dave, Ritvik G, Preethi Gurumurthy, Adarsh Mahor, Samahriti Mukherjee, Aditya Pakala, Ishan Paul, Janvita Reddy, Arghya Sarkar, Kinjal Sensharma, Aman Chadha, Amit P. Sheth, Amitava Das

Comments: arXiv admin note: text overlap with arXiv:2305.04329

Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[36] arXiv:2306.05584 (cross-list from cs.CV) [pdf, other]

Title: Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation

Jia-Xing Zhong, Ta-Ying Cheng, Yuhang He, Kai Lu, Kaichen Zhou, Andrew Markham, Niki Trigoni

Comments: To appear at NeurIPS 2023

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)

[37] arXiv:2306.05704 (cross-list from cs.CV) [pdf, other]

Title: Exploring Effective Mask Sampling Modeling for Neural Image Compression

Lin Liu, Mingming Zhao, Shanxin Yuan, Wenlong Lyu, Wengang Zhou, Houqiang Li, Yanfeng Wang, Qi Tian

Comments: 10 pages

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Image and Video Processing (eess.IV)

[38] arXiv:2306.06284 (cross-list from cs.SD) [pdf, other]

Title: Everybody Compose: Deep Beats To Music

Conghao Shen, Violet Z. Yao, Yixin Liu

Comments: Accepted MMSys '23

Journal-ref: Proceedings of the 14th Conference on ACM Multimedia Systems (2023)

Subjects: Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

[39] arXiv:2306.06979 (cross-list from cs.HC) [pdf, other]

Title: A Weakly Supervised Approach to Emotion-change Prediction and Improved Mood Inference

Soujanya Narayana, Ibrahim Radwan, Ravikiran Parameshwara, Iman Abbasnejad, Akshay Asthana, Ramanathan Subramanian, Roland Goecke

Comments: 9 pages, 3 figures, 6 tables, published in IEEE International Conference on Affective Computing and Intelligent Interaction

Subjects: Human-Computer Interaction (cs.HC); Machine Learning (cs.LG); Multimedia (cs.MM)

[40] arXiv:2306.07346 (cross-list from cs.CV) [pdf, html, other]

Title: Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training

Lorenzo Baraldi, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, Andrea Pilzer, Rita Cucchiara

Comments: Computer Vision and Image Understanding (2025)

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Multimedia (cs.MM)

[41] arXiv:2306.07646 (cross-list from cs.CV) [pdf, other]

Title: Enhanced Multimodal Representation Learning with Cross-modal KD

Mengxi Chen, Linyu Xing, Yu Wang, Ya Zhang

Comments: Accepted by CVPR2023

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[42] arXiv:2306.07678 (cross-list from cs.CV) [pdf, other]

Title: Localization of Just Noticeable Difference for Image Compression

Guangan Chen, Hanhe Lin, Oliver Wiedemann, Dietmar Saupe

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Image and Video Processing (eess.IV)

[43] arXiv:2306.07848 (cross-list from cs.CL) [pdf, html, other]

Title: GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognition

Yu Pan, Yanni Hu, Yuguang Yang, Wen Fei, Jixun Yao, Heng Lu, Lei Ma, Jianjun Zhao

Comments: 5 pages

Subjects: Computation and Language (cs.CL); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)

[44] arXiv:2306.07969 (cross-list from cs.CV) [pdf, other]

Title: GeneCIS: A Benchmark for General Conditional Image Similarity

Sagar Vaze, Nicolas Carion, Ishan Misra

Comments: CVPR 2023 (Highlighted Paper). Project page at this https URL

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)

[45] arXiv:2306.08657 (cross-list from cs.CV) [pdf, other]

Title: EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge

Mijanur Palash, Bharat Bhargava

Comments: Emotion Recognition, Deep Learning, Multi-modal, Convolutional neural network (CNN), LSTM, Situational-Knowledge, Novelty

Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM)

[46] arXiv:2306.08730 (cross-list from eess.SP) [pdf, html, other]

Title: Over-the-Air Learning-based Geometry Point Cloud Transmission

Chenghong Bian, Yulin Shao, Deniz Gunduz

Comments: 17 pages, accepted to IEEE JSAC SI on Intelligent Communications for Real-Time Computer Vision (Comm4CV)

Subjects: Signal Processing (eess.SP); Multimedia (cs.MM)

[47] arXiv:2306.08733 (cross-list from cs.CV) [pdf, other]

Title: Continuous Learning Based Novelty Aware Emotion Recognition System

Mijanur Palash, Bharat Bhargava

Comments: Automatic Emotion Detection, Novelty, Deep Learning

Journal-ref: AAAI Spring Symposium 2022

Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM)

[48] arXiv:2306.09085 (cross-list from cs.CV) [pdf, other]

Title: COSA: Concatenated Sample Pretrained Vision-Language Foundation Model

Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, Jing Liu

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Multimedia (cs.MM)

[49] arXiv:2306.09126 (cross-list from cs.SD) [pdf, other]

Title: STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events

Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, Tuomas Virtanen, Yuki Mitsufuji

Comments: 27 pages, 9 figures, accepted for publication in NeurIPS 2023 Track on Datasets and Benchmarks

Subjects: Sound (cs.SD); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Audio and Speech Processing (eess.AS); Image and Video Processing (eess.IV)

[50] arXiv:2306.09382 (cross-list from cs.SD) [pdf, other]

Title: Sound Demixing Challenge 2023 Music Demixing Track Technical Report: TFC-TDF-UNet v3

Minseok Kim, Jun Hyung Lee, Soonyoung Jung

Comments: 5 pages, 4 tables

Subjects: Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

[51] arXiv:2306.09489 (cross-list from cs.CV) [pdf, other]

Title: The 2023 Video Similarity Dataset and Challenge

Ed Pizzi, Giorgos Kordopatis-Zilos, Hiral Patel, Gheorghe Postelnicu, Sugosh Nagavara Ravindra, Akshay Gupta, Symeon Papadopoulos, Giorgos Tolias, Matthijs Douze

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Multimedia (cs.MM)

[52] arXiv:2306.09635 (cross-list from cs.SD) [pdf, other]

Title: CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models

Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg-Kirkpatrick, Julian McAuley

Comments: Accepted by WASPAA 2023. Demo: this https URL

Subjects: Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)

[53] arXiv:2306.09675 (cross-list from cs.LG) [pdf, other]

Title: Multi-View Class Incremental Learning

Depeng Li, Tianqi Wang, Junwei Chen, Kenji Kawaguchi, Cheng Lian, Zhigang Zeng

Comments: Accepted to Information Fusion

Journal-ref: Information Fusion, 2023, 102, 102021

Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[54] arXiv:2306.09817 (cross-list from cs.HC) [pdf, other]

Title: INDCOR white paper 4: Evaluation of Interactive Narrative Design For Complexity Representations

Christian Roth, Breanne Pitt, Lāsma Šķestere, Jonathan Barbara, Agnes Karolina Bakk, Kirsty Dunlop, Maria del Mar Grandio, Miguel Barreda, Despoina Sampatakou, Michael Schlauch

Comments: arXiv admin note: text overlap with arXiv:2010.10135

Subjects: Human-Computer Interaction (cs.HC); Multimedia (cs.MM)

[55] arXiv:2306.10054 (cross-list from cs.CY) [pdf, other]

Title: A Shift In Artistic Practices through Artificial Intelligence

Kıvanç Tatar, Petter Ericson, Kelsey Cotton, Paola Torres Núñez del Prado, Roser Batlle-Roca, Beatriz Cabrero-Daniel, Sara Ljungblad, Georgios Diapoulis, Jabbar Hussain

Journal-ref: Leonardo, 2024, 293-97

Subjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Multimedia (cs.MM)

[56] arXiv:2306.10065 (cross-list from eess.AS) [pdf, other]

Title: Taming Diffusion Models for Music-driven Conducting Motion Generation

Zhuoran Zhao, Jinbin Bai, Delong Chen, Debang Wang, Yubo Pan

Comments: Accepted by AAAI 2023 Summer Symposium with Best Paper Award

Subjects: Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Multimedia (cs.MM); Sound (cs.SD)

[57] arXiv:2306.10563 (cross-list from eess.AS) [pdf, other]

Title: Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Yuchen Hu, Ruizhe Li, Chen Chen, Chengwei Qin, Qiushi Zhu, Eng Siong Chng

Comments: 19 pages, 9 figures, Accepted by ACL 2023

Subjects: Audio and Speech Processing (eess.AS); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Sound (cs.SD)

[58] arXiv:2306.10567 (cross-list from eess.AS) [pdf, other]

Title: MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

Comments: 14 pages, 5 figures, Accepted by ACL 2023

Subjects: Audio and Speech Processing (eess.AS); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Sound (cs.SD)

[59] arXiv:2306.10684 (cross-list from cs.SD) [pdf, other]

Title: Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding

Zengjie Song, Zhaoxiang Zhang

Comments: Accepted to IEEE Transactions on Neural Networks and Learning Systems (T-NNLS)

Subjects: Sound (cs.SD); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

[60] arXiv:2306.10750 (cross-list from cs.CV) [pdf, other]

Title: WiCo: Win-win Cooperation of Bottom-up and Top-down Referring Image Segmentation

Zesen Cheng, Peng Jin, Hao Li, Kehan Li, Siheng Li, Xiangyang Ji, Chang Liu, Jie Chen

Comments: Accepted to IJCAI2023

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[61] arXiv:2306.11300 (cross-list from cs.CV) [pdf, html, other]

Title: RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Zilun Zhang, Tiancheng Zhao, Yulong Guo, Jianwei Yin

Comments: RS5M dataset v5

Journal-ref: TGRS-2023-06174

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multimedia (cs.MM)

[62] arXiv:2306.11724 (cross-list from eess.IV) [pdf, other]

Title: Low-complexity Multidimensional DCT Approximations

V. A. Coutinho, R. J. Cintra, F. M. Bayer

Comments: 28 pages, 5 figures, 5 tables

Journal-ref: IEEE Transactions on Image Processing, Volume 26, Issue 5, May 2017

Subjects: Image and Video Processing (eess.IV); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Systems and Control (eess.SY); Methodology (stat.ME)

[63] arXiv:2306.11746 (cross-list from cs.SI) [pdf, other]

Title: Focusing on Relevant Responses for Multi-modal Rumor Detection

Jun Li, Yi Bin, Liang Peng, Yang Yang, Yangyang Li, Hao Jin, Zi Huang

Comments: Submitted to TKDE

Subjects: Social and Information Networks (cs.SI); Multimedia (cs.MM)

[64] arXiv:2306.12249 (cross-list from cs.SD) [pdf, other]

Title: Knowledge-based Multimodal Music Similarity

Andrea Poltronieri

Comments: 11 pages, 1 figure

Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

[65] arXiv:2306.12795 (cross-list from cs.CV) [pdf, other]

Title: Learning Unseen Modality Interaction

Yunhua Zhang, Hazel Doughty, Cees G.M. Snoek

Comments: Published at NeurIPS 2023

Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM)

[66] arXiv:2306.14412 (cross-list from cs.CV) [pdf, other]

Title: A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inference

Chao Zhang, Shiwei Wu, Sirui Zhao, Tong Xu, Enhong Chen

Comments: 5 pages, 1 figure, technical report for track3 of CVPR 2023 LOVEU challenge

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[67] arXiv:2306.14432 (cross-list from eess.IV) [pdf, other]

Title: Subjective assessment of the impact of a content adaptive optimiser for compressing 4K HDR content with AV1

Vibhoothi, Angeliki Katsenou, François Pitié, Katarina Domijan, Anil Kokaram

Comments: Accepted Camera-ready version for the ICIP 2023 Paper

Subjects: Image and Video Processing (eess.IV); Human-Computer Interaction (cs.HC); Multimedia (cs.MM)

[68] arXiv:2306.14565 (cross-list from cs.CV) [pdf, html, other]

Title: Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, Lijuan Wang

Comments: 40 pages, 32 figures, ICLR 2024

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Computation and Language (cs.CL); Multimedia (cs.MM)

[69] arXiv:2306.14899 (cross-list from cs.CV) [pdf, html, other]

Title: FunQA: Towards Surprising Video Comprehension

Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, Ziwei Liu

Comments: Project Page: this https URL Codebase: this https URL

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multimedia (cs.MM)

[70] arXiv:2306.15246 (cross-list from cs.CR) [pdf, other]

Title: Errorless Robust JPEG Steganography Using Steganographic Polar Codes

Jimin Zhang, Xianfeng Zhao, Xiaolei He

Comments: 5 pages, 6 figures, submitted to IEEE Signal Processing Letters

Subjects: Cryptography and Security (cs.CR); Multimedia (cs.MM)

[71] arXiv:2306.15561 (cross-list from cs.CV) [pdf, other]

Title: You Can Mask More For Extremely Low-Bitrate Image Compression

Anqi Li, Feng Li, Jiaxin Han, Huihui Bai, Runmin Cong, Chunjie Zhang, Meng Wang, Weisi Lin, Yao Zhao

Comments: Under review

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Image and Video Processing (eess.IV)

[72] arXiv:2306.15910 (cross-list from cs.CV) [pdf, other]

Title: Incremental Learning on Food Instance Segmentation

Huu-Thanh Nguyen, Yu Cao, Chong-Wah Ngo, Wing-Kwong Chan

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[73] arXiv:2306.16175 (cross-list from cs.CV) [pdf, html, other]

Title: $\mathbf{C}^2$Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection

Maoxun Yuan, Xingxing Wei

Journal-ref: IEEE Transactions on Geoscience and Remote Sensing,vol. 62, pp. 1-12, 2024

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

[74] arXiv:2306.16181 (cross-list from cs.CV) [pdf, other]

Title: Learning to Pan-sharpening with Memories of Spatial Details

Maoxun Yuan, Tianyi Zhao, Bo Li, Xingxing Wei

Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Image and Video Processing (eess.IV)

[75] arXiv:2306.16366 (cross-list from cs.IT) [pdf, other]

Title: Coding Estimation based on Rate Distortion Control of H.264 Encoded Videos for Low Latency Applications

Amitesh Kumar Singam

Subjects: Information Theory (cs.IT); Multimedia (cs.MM); Optimization and Control (math.OC)