https://arxiv.org/api/1DDg0VLWPRHo9RlbALTFI2dsDV8 2026-11-12T03:27:35Z 10341 0 15 http://arxiv.org/abs/2611.14004v1 2026-11-10T12:45:09Z

As multimodal tourism-related information collected from heterogeneous data sources has become increasingly available, learning effective representations has attracted attention for improving recommendation performance. Existing approaches often tightly couple multimodal fusion with the architecture and optimization objective of a specific downstream model, which limits the generality and reusability of the learned representations. To address these limitations, we propose LUMIR, a decoupled multimodal representation learning framework based on contrastive learning. Multimodal information collected from heterogeneous data sources is first semantically unified into a shared textual representation space using large language models (LLMs). An interaction-aware fusion mechanism is then designed to model cross-modal interactions, while sequential signals are incorporated as a structural constraint to guide representation learning. To evaluate the learned representations, experiments are conducted on a multimodal point-of-interest (POI) dataset under the next-POI recommendation setting across multiple sequential recommendation backbones. The results show that LUMIR consistently outperforms existing multimodal fusion baselines in both accuracy-oriented and beyond-accuracy metrics, demonstrating that decoupling representation learning from downstream optimization provides an effective paradigm for learning reusable multimodal representations. The code is publicly available at https://github.com/wzehui/LUMIR.

2026-11-10T12:45:09Z 16 pages. Accepted at the DaQuaMRec Workshop, co-located with ACM RecSys 2026 Zehui Wang Wolfram Höpken Dietmar Jannach http://arxiv.org/abs/2611.13972v1 2026-11-10T12:08:40Z

Modern recommender systems increasingly consider diverse data modalities to enhance model performance. However, the quality of the underlying multimodal data (e.g. data from social media platforms) has received limited attention so far. Focusing on tourism recommendation, we examine real-world challenges encountered when using datasets from Yelp and Instagram. Issues include noise from fake reviews, hallucinated content from LLMs, modality contradictions, and demographic bias. We outline future directions such as quality-aware fusion, structured and controlled LLM-based content generation to reduce hallucinations, and user group differentiation to improve the robustness and reliability of multimodal recommendation systems.

2026-11-10T12:08:40Z 4 pages. Presented at the DaQuaMRec Workshop, co-located with ACM RecSys 2025 CEUR Workshop Proceedings, Vol. 4188, pp. 20-23, 2026 Zehui Wang Wolfram Höpken Dietmar Jannach http://arxiv.org/abs/2611.13707v1 2026-11-10T08:00:41Z

Learned image compression (LIC) models achieve competitive rate-distortion performance with increasingly large networks, making network quantization important for efficient deployment. Directly quantizing LIC models remains difficult in post-training quantization (PTQ) because compression performance is highly sensitive to quantization noise. Existing methods typically focus on optimizing either the weights or the activations, resulting in substantial rate-distortion degradation. To address this limitation, we introduce a two-stage pipeline that uses SmoothQuant to balance activation and weight ranges, followed by GPTQ to adapt the transformed weights. The proposed joint search framework optimizes the balancing transformation and adapted weights under the rate-distortion objective, accounting for the coupling between the two stages. Experiments across multiple LIC architectures and activation quantization configurations show that the joint search strategy consistently reduces rate-distortion degradation after quantization.

2026-11-10T08:00:41Z Accepted at IEEE VCIP 2026. 5 pages, 3 figures Yile Tu Heming Sun http://arxiv.org/abs/2605.28063v2 2026-11-10T07:32:24Z

Generating unified audio that naturally integrates speech and sounds from free-form text prompts remains challenging. For example, the prompt ``A fast drum roll ends with a cymbal crash, after which a man shouts: `Come on!'.'' describes several events that should appear in order and form a coherent recording rather than independent tracks that lack coherent temporal and semantic interactions. Current methods broadly follow two paradigms: pipeline methods couple separate text-to-sound and text-to-speech models, while unified models rely on a single generator that still requires structured, multi-field prompt inputs. Both paradigms require upstream prompt decomposition or rewriting, limiting free-form flexibility and risking cascading errors. To enable direct generation from free-form prompts within a single model, we introduce PlanAudio, an end-to-end framework that maps unconstrained free-form text to unified audio without intermediate prompt preprocessing. To address the challenges of direct free-form generation, PlanAudio integrates latent planning into a single autoregressive framework, predicting continuous audio layout before acoustic token synthesis. We construct a large-scale training corpus and introduce Free-Form Unified Audio Generation Benchmark (FUAG) for evaluation, both spanning speech, sound, and joint speech-and-sound generation. Experiments show that a single PlanAudio outperforms existing pipeline and unified baselines on speech-and-sound generation, while remaining competitive with specialist models on speech and sound generation. Further analysis validates the effectiveness of audio layout planning over textual planning, acoustic planning, and direct generation. Sample Available: https://planaudio.github.io/PlanAudio2026/.

2026-05-27T07:15:35Z Accepted by NeurIPS 2026. This version renames concepts, reorganizes sections, and fixes Method notation; adds ablation experiments and core-mechanism analysis; updates Table 1 with larger training data (improved speech performance in speech-and-sound mixed scenarios). The main conclusions remain unchanged Yuyue Wang Xihua Wang Xin Cheng Yijing Chen Ruihua Song http://arxiv.org/abs/2611.13544v1 2026-11-10T04:07:07Z

Multimodal screening for depression, anxiety, and stress in children and adolescents uses verbal and nonverbal behaviour. Most existing multimodal methods learn from a single recording or interview. Yet a participant's behaviour changes with the speaking condition, and these changes hold evidence that is invisible in a single recording. We propose DASWeave to bring this evidence into a neural network that encodes each response and then combines all responses of a participant. We train supervised trees to find which behavioural statistics matter for screening, both within each response and in how they change across conditions. We feed the tree outputs into the network both before and after this combination. We fine-tune a large language model to relate what a participant says to how they say it across conditions, and feed its representation into the network after the combination. The language model reads behaviour as text descriptions. We therefore derive additional training targets for the network from a multimodal large language model that takes in the audio-visual features directly. On the official hidden test set of the ACM Multimedia 2026 AdoDAS Grand Challenge (Track 1), DASWeave ranks second among 95 eligible teams. Its mean F1 is 27.1% higher than that of the official audio-visual baseline. It also has the highest mean AUROC among the top five teams. Ablation studies on the validation set show that the supervised trees and the fine-tuned language model contribute most to mean F1.

2026-11-10T04:07:07Z Binyan Chen University of Science and Technology of China, Hefei, China Tong Pan University of Science and Technology of China, Hefei, China Yadong Liu University of Science and Technology of China, Hefei, China Xin Yun University of Science and Technology of China, Hefei, China Shangfei Wang University of Science and Technology of China, Hefei, China http://arxiv.org/abs/2604.19019v3 2026-11-10T02:04:16Z

We study when, where, and why 978 Holocaust survivors smile in video testimonies. We create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling. We produce narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. Smiles often occur during periods of intense negative affect and we find negative-affect smiles are associated with more positive subsequent valence trajectories across all three modalities. Smiling reduces eye dynamics and blink rates, with both of these effects modulated by narrative valence. Taken together, we conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection.

2026-04-21T03:12:14Z Received Best Paper Runner-Up Award at ICMI 2026 In Proceedings of the 28th ACM International Conference on Multimodal Interaction (ICMI '26), October 5-9, 2026, Napoli, Italy Marcus Ma Emily Zhou Leonard Ludwig Julia Hörath Christina Winkler Kleanthis Avramidis Tiantian Feng Gabor Mihaly Toth Alina Bothe Shrikanth Narayanan 10.1145/3776574.3831180 http://arxiv.org/abs/2611.13310v1 2026-11-09T22:23:50Z

Temporal Sentence Grounding (TSG) aims to localize temporal segments in untrimmed videos that correspond to natural language queries, a critical task for video understanding with applications in video retrieval, summarization, and human-computer interaction. Despite significant progress driven by transformer-based models, existing TSG methods suffer from two key limitations: quantization errors due to temporal downsampling, which misalign segment boundaries, and suboptimal ranking of predictions based on semantic matching rather than localization precision. To address these challenges, we propose the Adaptive Boundary Refinement and Ranking (ABRR) framework, a novel, model-agnostic post-processing approach that enhances the performance of any TSG model without retraining. ABRR integrates three innovative components: (1) Temporal Distribution Smoothing, which mitigates noise in predicted score distributions using Gaussian kernel convolution; (2) Boundary Refinement via Optimal Transport, which leverages the 2-Wasserstein distance and Sinkhorn iterations to align coarse segment boundaries with true temporal moments, grounded in Monge-Kantorovich optimal transport theory; and (3) Quality-Driven Re-Ranking, which reorders predictions using a composite score balancing localization precision and semantic relevance. By correcting quantization errors and prioritizing accurately localized segments, ABRR achieves substantial improvements over state-of-the-art TSG models. Extensive experiments on benchmark datasets, including Charades-STA, ActivityNet Captions, and QVHighlights, demonstrate consistent gains at stringent IoU thresholds (e.g., IoU=0.7), alongside enhanced ranking performance. The framework's computational efficiency and compatibility with existing models make it a versatile and impactful solution for real-world TSG applications.

2026-11-09T22:23:50Z Accepted by IEEE Transactions on Multimedia Xiang Fang Wanlong Fang Junhao Dong Daizong Liu http://arxiv.org/abs/2611.13178v1 2026-11-09T20:00:23Z

Generative pipelines can now turn every product page of an e-commerce catalog into a short video ad, but deciding whether such an ad is usable is not one good-or-bad call. We describe how we operationalized the review and editing of these ads so that both can keep pace with generation: reviewers go through each ad scene by scene and answer six questions, on the product, scene quality, pronunciation, script, sound and captions; judges built from vision-language, speech and language models file each finding under the matching question, and reviewers accept or reject each finding and add what the judges missed. Each confirmed issue then opens an edit tool matched to it. The questions and the routing from a finding to its fix do not depend on the models that generate the ad. Across 120 ads from our pipeline, reviewed with judge findings on screen, reviewers marked the product wrong on 60% of scenes, and script, caption or sound problems on under 10% each. On 60 of these ads, they marked the product wrong on 58% of scenes, far more than the 26% marked by two reviewers who later saw the ads with the judges hidden; one reviewer who saw the ads both ways marked 57% with judge findings and 19% without. On 120 blind-labeled scenes, a product judge that watches the clip with the catalog photos reached AUC 0.84, while video-quality and photo-similarity scores were close to chance. We tune the judges for recall: the product judge catches nearly all defects but also flags many clean scenes, so reviewers decide; together, the judges widen what reviewers mark and reviewers add what the judges miss. Without judge findings, one reviewer's marking faded over a session. We release a review-portal template and our review data (120 ads and 562 scenes with judge findings and human marks, plus 120 blind-labeled scenes with baseline scores) at https://github.com/anoubhav/ad-review-and-edit.

2026-11-09T20:00:23Z Accepted at the WVAQ workshop at WACV 2027 Anoubhav Agarwaal Sukesh Perla Karthikeyan Mohanraj Susmita Ghose http://arxiv.org/abs/2610.40309v2 2026-11-09T17:35:03Z

Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features constrains all directions equally, providing broad coverage without semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. We therefore propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.

2026-10-29T15:48:43Z Yubo Cui Xianchao Guan Zheng Zhang http://arxiv.org/abs/2603.09541v2 2026-11-09T15:47:48Z

Embodied Question Answering (EQA) couples active visual perception, vision-language reasoning, and sequential decision making, and has traditionally been evaluated in temporally stable environments where visual evidence can be accumulated reliably. In realistic multimedia settings such as dynamic human-populated scenes, however, moving people and transient occlusions introduce human-induced perceptual variations: task-relevant multimodal cues become transient, viewpoint-dependent, and incompletely captured in individual views. Conventional store-then-retrieve strategies generate redundant visual evidence and incur high inference cost, making them ill-suited to such variations. This exposes two core problems for embodied agents: resolving viewpoint-dependent ambiguity, and maintaining compact evidence for efficient inference. To address these challenges, we present DIVRR (Dynamic-Informed View Refinement with Relevance-guided Rotation and adaptive memory selection), a training-free active evidence-acquisition framework for EQA. DIVRR combines question-conditioned room guidance with relevance-triggered view augmentation to guide exploration and additional observation. Selective memory admission uses the same relevance signal as view augmentation to retain high-scoring evidence, reducing stored entries while keeping the retrieved visual context compact. To evaluate EQA under human-induced perceptual variations, we introduce DynHuman-EQA, a dynamic human-populated EQA dataset constructed in Habitat, comprising dynamic and static subsets under shared scene layouts. Experiments on DynHuman-EQA and HM-EQA show that DIVRR outperforms existing baselines in both accuracy and memory efficiency. The gains are most pronounced in dynamic human-populated scenes.

2026-03-10T11:51:54Z Xin Lu Rui Li Xun Huang Weixin Li Jiayuan Li Chuanqing Zhuang Zhengda Lu Jun Xiao http://arxiv.org/abs/2611.13029v1 2026-11-09T09:10:16Z

In this paper, we consider the problem of recompression of images from JPEG to JPEG AI format. We show that direct recompression from JPEG to JPEG~AI trained on uncompressed images is accompanied by the preservation of the JPEG image artifacts in the resulting JPEG~AI image. In order to avoid it, we evaluate an approach where an artifacts removal neural network is first applied to the input JPEG image as an external pre-filter. However, in this case, the processing time is increased by 29 times. Therefore, we propose an artifact-compensated training for JPEG AI and show that it is able to combine pre-filtering against target distortions with efficient compression without increase in the processing time.

2026-11-09T09:10:16Z Preprint. Submitted to IEEE Multimedia on 21 February 2026. This is the author's draft version; it has not undergone peer review. 15 pages, 8 figures Igor Podtsepko Yuriy Kim Karam Mahfod Evgeny Belyaev Kristian Kirs Jianping Lin Ilya Papiev http://arxiv.org/abs/2611.12061v1 2026-11-09T08:36:47Z

We present BARS (Beat-Adaptive Rap Synthesis), a non-autoregressive rapping voice generation model based on Flow Matching, conditioned on a beat track (instrumental backing), lyrics, and a short reference voice for zero-shot timbre control. Unlike prior autoregressive rap synthesis, where rap pace and output duration are strictly determined by the beat, BARS enables independent control over the rap tempo, allowing flexible-length generation without explicit duration prediction or phoneme-level alignment. We encode lyrics using a pretrained ByT5 encoder with a lightweight Transformer adapter trained jointly with the model, without relying on speech-pretrained text encoders, enabling natural handling of non-standard orthography (e.g., slang, abbreviations) and numerals common in rap lyrics. To address alignment instabilities, such as token skipping or repetition, under strong beat conditioning, we propose Asymmetric Scaled RoPE for cross-attention, which introduces a positional structure to facilitate sequential lyric-audio alignment learning. Furthermore, the model supports partial re-generation via span masking, enabling lyric-level editing while preserving the surrounding vocal style. Experiments demonstrate that the proposed model achieves high lyric fidelity, natural rap flow that is rhythmically coherent with the beat, and robust tempo control across a wide range of rapping styles and speeds.

2026-11-09T08:36:47Z Published in ISMIR 2026 (Best Paper Award nominee). Demo: https://bars-ismir2026.github.io/bars/ Proceedings of the 27th International Society for Music Information Retrieval Conference (ISMIR), 2026 Taemin Cho Hector Martel Enyan Koh http://arxiv.org/abs/2611.13002v1 2026-11-08T16:41:22Z

Short-form video has become one of the dominant formats of online content, yet large-scale datasets enabling the analysis of what makes these videos successful remain limited. Existing resources often focus on post-publication popularity signals or rely on non-interpretable representations such as high-dimensional embeddings. Here, we introduce YTShorts-92K, a dataset of 92,654 YouTube Shorts collected from 345 creators and spanning 12 manually defined content categories. Each video is enriched with an interpretable multimodal feature set covering metadata, textual information, visual characteristics from frames and thumbnails, and cross-modal consistency measures. We describe the collection methodology, data cleaning process, and feature extraction pipeline, followed by descriptive analyses of the diversity of creators, topics, and extracted features. This dataset offers a reusable resource for studying short-form video content, online attention, and multimodal content analysis.

2026-11-08T16:41:22Z 10 pages, 3 figures, 3 tables Celian Charrin Sidonie Abuzakuk Artúr Vlaszov Nándor Birher Roland Molontay http://arxiv.org/abs/2611.11438v1 2026-11-08T15:02:47Z

Advances in spatial understanding have enabled frontier models to power coding agents that control a real robot arm zero-shot and end-to-end, without task-specific demonstrations or retraining. This brings two new safety challenges. First, the extensive knowledge of frontier models lets these agents perform tasks that earlier VLA and WAM paradigms could not, opening the door to new forms of capability misuse. Second, coding agents can access digital resources such as web pages and emails. Threats in these resources, such as indirect prompt injection, can carry over into the physical world when the agent controls a robot. We therefore introduce ArmedAgent, the first systematic benchmark to evaluate both risks for coding agents performing zero-shot robot control. It covers robustness, malicious use, and indirect injection across 36 tasks in 5 scenarios, evaluated in simulation and validated on a physical robot arm. A distinctive portion of the tasks target harmful behaviors that earlier VLA paradigms could not perform. Testing several frontier models, including GPT-6-Astra, under widely used agent frameworks, we find substantial safety risks in current systems.

2026-11-08T15:02:47Z Technical report, work in progress Yiming Sun Mi Zhang http://arxiv.org/abs/2611.12997v1 2026-11-08T14:06:38Z

Multimodal (e.g. vision-language, in this work) attribution aims to interpret the models without access to ground-truth explanatory supervision. Existing multimodal attribution methods typically rely on paired modalities as proxy supervision, implicitly assuming that image-text pairs are semantically aligned. Information bottleneck (IB)-based attribution methods follow this general paradigm and instantiate it through a global sufficiency-compression objective. In realistic multimodal datasets, however, this assumption is frequently violated due to partial semantic misalignment and unreliable cross-modal correspondences. Through theoretical analysis, we show that, under such misalignment, information-bottleneck-based attribution objectives exhibit a structural limitation: by assuming equal reliability across all proxy supervision signals, they induce posterior over-contraction and forced explanations. To address this issue, we propose TIB, a sample-wise Tempered Information Bottleneck framework that adaptively modulates attribution strength according to cross-modal reliability, without assuming semantic alignment. Extensive experiments on controlled benchmarks and large-scale datasets show the effectiveness of the proposed approach, while also validating the identified failure mode.

2026-11-08T14:06:38Z Accpeted to NeurIPS2026 Qiming Huang Peixi Liu Jianbo Jiao