https://arxiv.org/api/ha6BEra0dFKW+QeFTyJSoWCSum0 2026-09-11T19:09:55Z 10051 15 15 http://arxiv.org/abs/2609.04867v2 2026-09-09T03:05:45Z

Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it difficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench factorizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as-a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio.

2026-09-04T08:27:10Z 19 pages, 10 figures, 4 tables. Accepted at ACM Multimedia 2026 (MM '26). This arXiv version includes supplementary appendices not included in the conference proceedings version Yuchen Sun Qian Yang Jun Wang Detai Xin Guoqiao Yu Guanglu Wan Qi Jia 10.1145/3767308.3836100 http://arxiv.org/abs/2609.09579v1 2026-09-09T00:59:53Z

Applying QoE analysis to optimize usage of cellular spectrum is of high interest to mobile network operators. A key challenge is to be able to perform QoE measurement across very different types of apps, from DASH VoD to interactive applications such as Video Conferencing and Cloud Gaming. This paper presents AMVOTS, a QoE measurement system developed by AT&T, which is flexible enough to support a large range of application types and network conditions. We also discuss using AMVOTS as part of a closed loop to prototype QoE-aware radio resource allocation.

2026-09-09T00:59:53Z 4 pages, 3 figures. Accepted author manuscript of a paper published in the 2025 17th International Conference on Quality of Multimedia Experience (QoMEX), Madrid, Spain. The version of record is available at the DOI 2025 17th International Conference on Quality of Multimedia Experience (QoMEX), Madrid, Spain, 2025, pp. 1-4 Eric Petajan Jonathan Lynam Morey Antebi Hessam Moeini David Lindero Lars Ernstrom Gyanesh Patra Szilveszter Nadas 10.1109/QoMEX65720.2025.11219938 http://arxiv.org/abs/2609.09490v1 2026-09-08T22:12:33Z

Prior work has shown that QoE-aware resource sharing for real-time interactive video can support up to three times more simultaneous sessions at acceptable quality compared to rate-fair allocation. However, the required capabilities (QoE-targeted encoding, runtime spatial complexity estimation, and rich application-network APIs) are not yet available in commercial deployments. In this paper, we take an evolutionary approach: we design a system that delivers QoE-aware resource allocation using only capabilities that can be assembled in a lab today. We extend the utility-based allocation framework to the radio resource domain by introducing composite spatial complexity, which combines a session's video spatial complexity with its time-variant spectral efficiency into a single resource demand function. To operate with commercial real-time video streaming applications that use rate-based congestion control and lack capability to measure QoE, we use external tooling for QoE measurements. We develop an incremental reallocation algorithm with per-interval limits that encode both the congestion control algorithm's speed constraint and that spatial complexity estimates are reliable only near the current rate. The resulting prototype combines external QoE measurements with congestion-signal-based rate steering and does not require modification to commercial applications. We chart an evolution path from this prototype toward full QoE-aware resource sharing, mapping emerging standards (IETF SCONE, CAMARA, Media over QUIC) to the progressive capabilities they enable.

2026-09-08T22:12:33Z Accepted author manuscript. Published in Proc. IEEE QoMEX 2026, Cardiff, UK. (c) 2026 IEEE. 7 pages, 2 figures, 3 algorithms, 2 tables Proc. 2026 18th International Conference on Quality of Multimedia Experience (QoMEX), IEEE, 2026, pp. 1-7 Szilveszter Nádas Lars Ernström Dan Druta Igor Pruzhansky David Lindero Jonathan Lynam Eric Petajan 10.1109/QoMEX69967.2026.11618317 http://arxiv.org/abs/2609.08936v1 2026-09-08T15:59:16Z

We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.

2026-09-08T15:59:16Z Open-source at https://github.com/Tencent-Hunyuan/AuK Ziyang Ma Zhikang Niu Wenming Tu Tianrui Wang Ruiqi Yan Junxi Liu Yanru Huo Nickk Huang Yang Liu Qicong Xie Zeyu Xie Hui Wang Haitao Li Zixuan Jiang Yalin Li Jie Fang Yifan Duan Zeyue Tian Guangzheng Li Haina Zhu Shuyi Wang Jinwen Wang Mingyu Cui Tian Tan Auden Sen Liang Steve Yves Shan Yang Liefeng Bo Zilong Zheng Kai Yu Eng-Siong Chng Xie Chen http://arxiv.org/abs/2609.08517v1 2026-09-08T10:04:01Z

Diffusion models have become a core paradigm for multimedia generation, offering powerful concept-driven controllability for personalization, semantic editing, and selective unlearning. However, as semantic control extends beyond natural-language prompts to learned embeddings and intervention pipelines, the safety and governance of these systems become increasingly difficult to evaluate in a unified manner, especially for safety-sensitive, identity-linked, and other privacy-relevant concepts. Existing studies mainly rely on heuristic audits, adversarial probing, or task-specific erasure benchmarks, and therefore provide limited support for systematic comparison across models, conditioning channels, and deployment conditions. We present a concept-level probabilistic audit and reporting framework for diffusion models. We formalize governance-relevant concept behaviors as Bernoulli semantic events induced by stochastic generation, and define a Concept Risk Operator that maps model-channel configurations to structured risk profiles, enabling comparison across prompting interfaces, learned embedding channels, models, and recorded conditions. We apply sample-level post-hoc calibration and configuration-level risk aggregation, and show that probability error can change thresholded actions near policy boundaries. Experiments on SD1.5, SD2.1, and SDXL reveal consistent yet non-uniform operational risk patterns across concept families, channels, recorded conditions, and shifted protocols. In particular, embedding-based access and obfuscated prompts expose risks often understated by standard-prompt evaluation. A pooled multi-protocol calibrator improves held-out probability reliability, but we do not claim transfer from a standard-only calibrator. CLRC provides a common audit schema for probabilistic and decision-aware governance of multimedia generation systems.

2026-09-08T10:04:01Z Kun Xu Yushu Zhang Tao Wang Shuren Qi Barbara Carminati Elena Ferrari Yuming Fang 10.1145/3767308.3835863 http://arxiv.org/abs/2604.07991v2 2026-09-08T08:27:54Z

Unmanned aerial vehicles (UAVs) are increasingly crucial for low-altitude autonomy and complex environment understanding. World models enable UAVs to anticipate how future states may evolve under potential actions, providing predictive support for autonomous decision-making. For video world models, future video generation serves as a means of simulating future visual states, providing a direct basis for evaluating their predictive capability. However, existing UAV video resources typically focus on specific control signals or simulated environments. Standardized benchmarks for multi-condition UAV-view future video generation are still limited. To bridge this gap, we introduce MotionScape, a real-world UAV-view benchmark comprising 228 high-resolution video clips totaling 62,700 frames, with semantic annotations of weather and illumination conditions, scene environment, and camera-viewpoint motion. The clips are stratified into low-, medium-, and high-motion strata using optical-flow-based motion intensity. MotionScape supports Text2World, Image2World, and Video2World evaluation under a unified future-generation protocol. Technical validation with representative baseline models assesses performance across conditioning settings and motion strata, providing a unified platform for evaluating the future visual simulation capabilities of UAV world models.

2026-04-09T08:59:40Z Zile Guo Zhan Chen Xiaoxuan Liu Enze Zhu Kan Wei Yongkang Zou Sijia Ma Lei Wang http://arxiv.org/abs/2609.08221v1 2026-09-08T04:10:27Z

Micro-actions are subtle, low-intensity non-verbal behaviors that provide cues to fine-grained human states, including emotions and intentions. Recognizing them remains difficult because they are brief, contain weak visual changes, and often exhibit similar motion patterns across categories. This paper addresses these challenges with a fine-grained micro-action recognition method that combines full fine-tuning of InternVideo2.5, hierarchical soft fusion, and a lightweight candidate-label reranker. For the long-tailed label distribution in MA-52, we use class-balanced sampling and inverse-frequency reweighting to reduce the effect of frequent classes during training. We fine-tune InternVideo2.5 end to end and attach coarse and group-conditional fine-grained classification heads to the shared video representation, improving the consistency between coarse and fine predictions. For ambiguous samples, the candidate-label reranker uses hard samples and video-label matching to focus on easily confused fine-grained actions. Experiments validate the proposed method, which achieves a 79.99% F1-mean on MA-52 and ranks first in the 3rd Micro-Action Analysis Grand Challenge at ACM Multimedia 2026.

2026-09-08T04:10:27Z 7 pages, 2 figures, 3 tables. Accepted to the 34th ACM International Conference on Multimedia (MM '26). Ranked 1st in the 3rd Micro-Action Analysis Grand Challenge at ACM MM 2026 Yichi Zhang Zhichao Xia Yanjun Chi Lingsi Zhu Yuefeng Zou Jun Yu Qingsong Liu Jianqing Sun Shengping Liu http://arxiv.org/abs/2609.03584v2 2026-09-08T01:43:14Z

Synthetic chord substitutions offer controlled tests of music generation, but their effects can differ from those of a complete recognized chord sequence. We propose structure-matched calibration, which constructs synthetic chord sequences that preserve the locations and harmonic relations of recognizer-induced changes. Paired generation measures how closely these sequences reproduce the response to complete recognizer replay. On 29 of 30 MUSDB18-HQ songs, central four-second tritone corruption produces a larger target response than complete recognizer replay. On 24 held-out MoisesDB songs, structure matching reduces response distance to replay by 81% for MIDI-SAG and 77% for MusicGen-Chord. Distance to replay decreases on every song in both models. Joint matching also brings output chord sequences closer to replay than either temporal or relational matching alone. Calibration extends to AccoMontage's native beat-based interface, improving 23 of 24 songs. These results establish a method for making synthetic chord tests representative of recognized harmony, while distinguishing response magnitude from the harmonic structure of generated music.

2026-09-03T09:32:15Z 5 pages, 2 figures, 2 tables. Code: https://github.com/Viwennnnnn/local-chord-corruption-replay Weiwen Huang http://arxiv.org/abs/2609.07936v1 2026-09-07T19:52:20Z

As geopolitical tensions, energy crises, and energy-intensive AI infrastructure intensify concerns about demand, affordability, and resilience, communities increasingly encounter these challenges through everyday energy practices, particularly winter heating. Against this background, the doctoral exposé, "Designing Human-HVAC Interaction for Healthy, Affordable, and Sustainable Heating in Smart Homes", is structured around four chapters. First, a multidisciplinary literature review defines and positions Human-HVAC Interaction, focusing on heating in smart homes. Second, longitudinal living lab studies with design probes examine everyday heating practices, thermal comfort, and indoor environmental quality, with attention to thermally vulnerable groups such as older adults, pregnant or menopausal women, parents with infants, and people affected by allergies or airborne pollutants. Third, a VR-based smart home demonstrator explores how heating and IEQ scenarios can be prototyped and evaluated as a virtual living lab, while critically examining the limits of representing bodily indoor climate conditions through VR. Fourth, follow-up design studies examine how VR-based insights can be translated into physical-digital prototypes that combine digital fabrication, distributed environmental sensing, and diverse interface forms for critical heating and IEQ contexts. The thesis aims to contribute a design-oriented understanding of Human-HVAC Interaction by building from a multidisciplinary literature review to empirical living lab and co-design studies, VR-based prototyping, and physical system development, examining how smart home users make sense of, negotiate, and respond to smart HVAC system.

2026-09-07T19:52:20Z Mensch und Computer 2026 -- Tagungsband, Gesellschaft für Informatik e.V., 30. August - 02. September 2026, Duisburg, Germany Delong Korus-Du 10.18420/muc2026-mci-dc-228 http://arxiv.org/abs/2609.07414v1 2026-09-07T12:25:58Z

Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.

2026-09-07T12:25:58Z SIGGRAPH Asia 2026. Hejun and Jinxi are co-first authors. Code and data are available at: https://github.com/vLAR-group/RelightFormer Hejun Wang Jinxi Li Junwei Jiang Shiwei Mao Hu Cheng Shouwang Huang Bo Yang http://arxiv.org/abs/2609.07409v1 2026-09-07T12:23:42Z

Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight multimodal SER framework featuring an asymmetric Residual Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway. Combined with a BYOL-inspired cross-modal alignment objective and attention-guided pooling, the proposed framework improves multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric multimodal fusion is an effective alternative to interaction-heavy cross-modal transformers for real-time surveillance applications.

2026-09-07T12:23:42Z 6 pages, 4 figures, 3 tables. Accepted at the 2026 IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS 2026). (c) 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses Ngo Truong Dinh Tung-Lam Bui Chi-Trung Duong Vien Nguyen Thi Viet-Anh Nguyen Phuc-Lu Le http://arxiv.org/abs/2609.07311v1 2026-09-07T10:28:26Z

Searching large video collections is typically an interactive process in which users play two roles. First, they hold the search intent: the underlying goal that determines what content they seek and why. Second, users must operationalize this intent through an iterative search loop. Users translate their intent into queries, browse the retrieved candidates, and refine their queries based on the results. In this paper, we investigate the capabilities of modern Vision Language Models (VLM) and agentic approaches to reach search goals interactively and fully autonomously. Specifically, we study whether a provided initial specification of a search goal might be sufficient to solve traditionally interactive search tasks with an agentic system. Provided that the involved VLMs are not aware of the whole large video dataset in advance, the key challenge lies in the effective combination of an existing interactive video search system and a smart VLM agent controlling the system. While the search system provides indexing and efficient querying, the VLM-based agents analyze top-ranked items and make decisions about next actions. Our results show that modern agents can autonomously operate interactive video retrieval systems to solve many search tasks from an initial intent description, achieving performance competitive with strong historical expert-operated systems in several settings.

2026-09-07T10:28:26Z Submitted to the International Conference on Multimedia Modeling Bastian Jäckl Zuzana Vopálková Daniel A. Keim Jakub Lokoč http://arxiv.org/abs/2609.07038v1 2026-09-07T04:49:27Z

Adolescent depression, anxiety, and stress (D/A/S) call for scalable tools that complement, rather than replace, professional evaluation. Under a privacy-preserving policy, the AdoDAS Grand Challenge withholds minors' raw recordings and distributes anonymized audio-visual representations and ASR-derived text. Its 6,000 participants provide 24,000 segments across one scripted-reading and three open-response sessions. Two tracks assess multi-task binary D/A/S screening and ordinal prediction of 21 DASS-21 item responses. From 191 registrations, the final leaderboards included 95 eligible screening teams and 64 item-prediction teams. Audio-visual baselines achieved 0.4604 mean F1 and 0.2675 mean Quadratic Weighted Kappa; leading submissions reached 0.5921 and 0.2776. Representative systems emphasize cross-session modelling, temporal multimodal fusion, psychometric structure, and task-aware calibration.

2026-09-07T04:49:27Z 5 pages, 1 figure, 3 tables. To appear in the Proceedings of the 34th ACM International Conference on Multimedia (MM '26), November 10-14, 2026, Rio de Janeiro, Brazil. Zhaojie Luo and Junkun Wang contributed equally Zhaojie Luo Southeast University Shenzhen Loop Area Institute Junkun Wang Southeast University Tianhua Qi Southeast University Yuxuan Wu Southeast University Xin Zhao Southeast University Tetsuya Takiguchi Kobe University Tomoko Matsui Shenzhen Loop Area Institute Kun Qian Beijing Institute of Technology Fei Wang Nanjing Medical University Shuqiong Wu The University of Osaka Zhengjun Yue Shenzhen Loop Area Institute Hiroshi Ishiguro The University of Osaka Xinyuan Qian University of Science and Technology Beijing Haizhou Li The Chinese University of Hong Kong Shenzhen Loop Area Institute http://arxiv.org/abs/2609.06497v1 2026-09-06T09:25:32Z

Multimodal sentiment analysis requires effective modeling of both verbal semantics and non-verbal affective cues. A central challenge is to calibrate text-centered sentiment understanding with visual facial evidence in a controlled, adaptive, and parameter-efficient manner. Text usually serves as the semantic anchor, whereas visual cues provide complementary evidence for ambiguous or implicit expressions; however, indiscriminate fusion may introduce visual noise and distort textual semantics. Moreover, fully fine-tuning large visual and textual encoders is costly and prone to overfitting on limited and scenario-dependent MSA benchmarks. To address these issues, we propose Vision-Guided Text Prompt Tuning (VG-TPT), which formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations. VG-TPT injects visual affective cues into a frozen BERT encoder through layer-wise adaptive prompts, rather than relying on late-stage feature fusion or full backbone tuning. A co-guided router composes prompts from a trainable prompt bank according to both the evolving text state and the visual guidance feature, enabling sample-specific and layer-specific modulation. Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines and achieves competitive or superior performance compared with several full-modality methods, while updating only 2.4M trainable parameters. The code is available at https://github.com/ma-tubu/VG-TPT.

2026-09-06T09:25:32Z This paper has been accepted to IEEE MMSP 2026 Xiaoran Kou Jingyi Wu Peng Sun Yang Liu Hong Chen http://arxiv.org/abs/2609.06478v1 2026-09-06T08:50:45Z

Music videos are an important form of audiovisual expression in contemporary culture. They translate and extend the expressive content of songs through deliberate visual design. Existing automatic music video (MV) generation systems can generate visually plausible shots, yet often struggle with long-form coherence and song-grounded visual development. We present MVWeaver, a music video generation agent that integrates hierarchical planning with a learned song-to-visual bridge that translates song understanding into executable shot plans. The MVWeaver architecture comprises a comprehensive song analysis module, a visual planner that constructs hierarchical plans, and downstream image and video generation models that render the planned content. To equip a general-purpose LLM with MV-specific song-to-visual knowledge, we learn a bridge between song analysis and visual planning from real-MV-derived supervision and curate 1,861 real-world song--MV pairs with structured song-side, MV-side, and teacher-inferred song-to-visual rationale annotations. Using these annotations, we perform LoRA-based supervised fine-tuning (SFT) of a large language model to predict song-to-visual bridges that guide hierarchical visual planning. Our experiments demonstrate stronger song-grounded visual translation, richer visual development, and greater conceptual and shot-to-shot coherence, while ablations support the benefits of learned bridge conditioning.

2026-09-06T08:50:45Z 5 pages, 2 figures Sifei Li Minyan Luo Xu Li Guodong Qi Xincan Wang Hanwen Wang Chen Zhang Pengfei Wan Oliver Deussen Weiming Dong