https://arxiv.org/api/0e+spXwfCR/FPXZfkAE/r0rYTfM 2026-07-20T22:44:27Z 5392 45 15 http://arxiv.org/abs/2607.04668v1 2026-07-06T04:50:32Z

On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be dropped into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86-64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACK-latched epoch protocol that never waits on a named core: a seqlock-style generation-tagged latch composed with RCU/epoch-style membership consent, so each token's participant set is the intersection of the cores the gang requested and the cores that acked the current epoch. An un-acked core is outside this token and joins at most one token later. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8C/16T), inference output is bit-exact under verified per-token membership change on both a 135M and a 7B model, the property that makes elasticity safe in a kernel whose safety gate reads logits. Against fair static core partitions, elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75x (25%), 1.52x (50%), and 1.28x (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation. Returning a lent core costs 0.22 us (p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (~16 ms): a running tenant is never preempted mid-slice. Decode throughput saturates at gang width 8, so ceding cores past the knee is nearly free: elasticity auto-sizes the gang online.

2026-07-06T04:50:32Z 14 pages, 1 figure, 6 tables Daeyeon Son http://arxiv.org/abs/2411.07447v5 2026-07-05T13:47:06Z

LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. While LLM inference systems are capable of serving millions of requests from multiple users, they often lack theoretical models to determine whether they achieve the performance upper bounds of underlying hardware resources. Beyond online workload serving, merely analyzing existing systems-or developing yet another one-is both GPU-intensive and labor-intensive. This paper provides a comprehensive survey of LLM inference systems, focusing on their cache management policies and availability. We then show that simulations can be an effective tool to save GPU hours in the development and analysis phase of inference systems, revealing useful insights for developing better inference techniques, unlike how existing studies used simulations to find the best parameters inside a given system. Finally, we provide theoretical tools to estimate the optimal performance and formulate new ideas. Based on the theoretical analysis, especially on the cache management in LLM inference, we propose a simple yet effective cache replacement policy that can be easily plugged into existing preemptive schedulers and systems. We show that such a simple policy inspired from database systems can substantially save GPU hours in actual inference systems on online workloads. We share our experience submitting a journal paper to a database venue in November 2025 for anyone considering a similar path.

2024-11-12T00:10:34Z Kyoungmin Kim Jiacheng Li Kijae Hong Qunyou Liu Darong Huang Anastasia Ailamaki http://arxiv.org/abs/2607.04302v1 2026-07-05T13:39:52Z

We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16. To our knowledge, HiFA4 is the first Ascend-HIF4-targeted design of this kind evaluated on standard NLP benchmarks. HiFA4 combines two mechanisms. Smooth-QK applies a calibration-static per-channel equivalent rescaling to Q and K after RoPE, transferring quantization difficulty from K to Q without per-tile online reduction at inference. P-Reordering accumulates the softmax normalizer from the same quantized attention weights P_hat used in the PV GEMM, rather than from a higher-precision reconstruction. We show that this inconsistent formulation introduces a coherent output-scaling error, and validate the effect on a Qwen3-8B Layer-0 MMLU trace, where all 3.6M measured attention tiles exhibit net probability-mass loss with median epsilon_bar = -0.064. P-Reordering also allows the normalizer to be fused into the PV Cube GEMM. Across five LLMs, HiFA4 reduces quantization-induced decision drift. On Qwen3-8B, it recovers 37.5% of the accuracy gap introduced by direct HIF4 quantization, narrows the sample-weighted accuracy loss from 1.12 pp to 0.70 pp, reduces BF16-inconsistent MMLU predictions from 16.3% to 8.2%, and cuts MMLU accuracy regressions by 57% (1071 to 465). On Gemma2-9B, mild smoothing keeps HiFA4 within 0.7 pp of BF16 while reducing MMLU regressions by 27%. On LLaMA3.1-8B, Mistral-7B, and Phi-4B, where Smooth-QK is disabled, P-Reordering with the adopted Q-Mean auxiliary still reduces full-set MMLU regressions by 41-52%. A preliminary instruction-scheduling analysis projects a 35.4% critical-path latency reduction relative to BF16 by fusing the softmax normalizer into the PV Cube GEMM; on-hardware validation is left to future work.

2026-07-05T13:39:52Z 22 pages Hui Dong Yanzhao Li Jie Gao Chunlu Li Zhiyuan Zhang Yupeng Sun Zhenyuan Chen Zhiqiang Zou http://arxiv.org/abs/2607.04030v1 2026-07-04T21:15:59Z

Conditional functional dependencies (CFDs) are functional dependencies with a restricted scope: they specify the context in which a dependency holds and are useful for data-quality tasks, specifying complex integrity constraints, and extracting valuable insights from data. We study the CFD discovery problem, which is computationally demanding. We build on the state-of-the-art CFDFinder algorithm and introduce a set of algorithmic and engineering improvements, including a parallelization strategy, to produce ParCFDFinder. Our implementation is integrated into Desbordante - a high-performance open-source data profiler written in C++ that exposes a Python interface, enabling CFD discovery to be invoked from any Python program. Experimental results show that our enhancements speed up the algorithm by up to $318\times$ ($118\times$ on average) and reduce memory usage by up to $23\times$ ($14\times$ on average) compared with the existing Java-based implementation of Metanome. Integrating ParCFDFinder into Desbordante makes it possible, for the first time, to conveniently discover CFDs on datasets with hundreds of thousands of rows on a commodity machine within a reasonable time.

2026-07-04T21:15:59Z 2026 39th Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2026, pp. 130-141 Ivan Kozhukov Dmitry Fedoseev Maksim Emelyanov Artem Smola Pyotr Senichenkov Pavel Anosov George Chernishev 10.23919/FRUCT70069.2026.11506538 http://arxiv.org/abs/2607.03993v1 2026-07-04T19:27:48Z

One of the key promises of Mobile Edge Computing (MEC) is its low latency. Current large-scale IoT deployments rely on cloud for their reliability, low cost, and ease of use. For outdoor IoT deployments, 5G cellular networks offer significantly enhanced bandwidth and dramatically reduced latency compared to previous generations, enabling real-time data processing and control. Therefore, leveraging 5G connectivity is crucial for outdoor IoT applications requiring responsiveness and complex data handling. Combining MEC with 5G has the potential to provide the ease of cloud computing alongside low latency. We investigate the latency performance on a 5G cellular network with an experimental MEC setup. In our proof-of-concept, we demonstrate the benefits of using an edge-based compute server for real-time power transmission line analytics. We compare our solution with state-of-the-art multi-region cloud deployments and discuss the advantages of mobile edge computing (MEC). Our real-world evaluation demonstrates a low latency of 44.62 ms for MEC compared to cloud regions; however, the gap is narrowing. While such low latencies can benefit real-world deployments, they remain insufficient to meet the stringent requirements of smart power grid operations (~8 ms).

2026-07-04T19:27:48Z 12 pages, 6 figures Advanced Information Networking and Applications (AINA 2026), Springer Nature Switzerland, Cham, pp. 188-199 (2026) Aakash Sharma Sigmund Akselsen Anders Andersen Lars Ailo Bongo Arne Munch-Ellingsen 10.1007/978-3-032-23347-9_16 http://arxiv.org/abs/2607.03988v1 2026-07-04T19:14:08Z

The transition to quantum-resistant security is a critical priority for the next generation of mobile networks, particularly within the disaggregated architecture of 5G. This paper presents an energy-aware system-level evaluation of Post- Quantum Cryptography (PQC) integrated into the Transport Layer Security (TLS) handshake on embedded User Equipment (UE). Using Raspberry Pi 5s as representative embedded processing platforms, we evaluate the performance of NIST-standardized combinations of classical and post-quantum signature and key exchange mechanisms (KEM), incorporating direct on-device power measurements to estimate per-handshake energy consumption. Results experimentally validate a strong coupling between latency and energy consumption, indicating that execution time is the dominant contributor to energy cost. Hash-based signature schemes incur up to 4x higher latency and 2x energy compared to lattice-based alternatives, while the impact of KEMs is comparatively smaller. The analysis further reveals that overall system performance is primarily constrained by cryptographic computation and concurrency-induced contention rather than network transport effects. These findings provide practical guidance for PQC deployment in mobile environments and demonstrate that lattice-based signatures offer a more favorable balance between security, efficiency, and scalability for 5G systems.

2026-07-04T19:14:08Z Sanzida Hoque Abdullah Aydeger http://arxiv.org/abs/2601.10041v2 2026-07-03T19:43:04Z

Emergency department (ED) overcrowding and patient boarding represent critical systemic challenges that compromise care quality. We propose a threshold-based admission policy that redirects non-urgent patients to alternative care pathways, such as telemedicine, during peak congestion. The ED is modeled as a two-class $M/M/c$ preemptive-priority queuing system, where high-acuity patients are prioritized and low-acuity patients are subject to state-dependent redirection. Analyzed via a level-dependent Quasi-Birth-Death (QBD) process, the model determines the optimal threshold by maximizing a long-run time-averaged objective function comprising redirection-affected revenue and costs associated with patient balking and system occupancy. Structural analysis establishes monotone comparative statics relating the optimal threshold to all model parameters analytically. Numerical analysis using national healthcare data reveals that optimal policies are highly context-dependent. While rural EDs generally optimize at lower redirection thresholds, urban EDs exhibit performance peaks at moderate thresholds. Results indicate that our optimal policy yields significant performance gains of up to $4.84\%$ in rural settings and $5.90\%$ in urban environments. This research provides a mathematically rigorous framework for balancing clinical priority with operational efficiency across diverse ED settings.

2026-01-15T03:38:24Z 44 pages, 14 figures Sahba Baniasadi Paul M. Griffin Prakash Chakraborty http://arxiv.org/abs/2606.06510v3 2026-07-03T15:48:02Z

Conventional HPC holds that native hardware FP64 is the irreducible foundation of scientific computing. On AI-optimized GPUs of the NVIDIA B300 generation and beyond, native FP64 throughput has collapsed to ~1.3 TFLOPS even as FP8 tensor throughput has grown to multiple PFLOPS. We argue something stronger than that this is survivable: the FP8 tensor-core matrix-multiply is the sole computational primitive on which double-precision scientific computing needs to be built. Every canonical kernel -- dense and sparse linear algebra, spectral transforms, stencils -- and every application composing them reduces, via the Chinese Remainder Theorem-based Ozaki Scheme II, to sequences of FP8 matrix operations; the only non-FP8 arithmetic is a bounded, fixed-width integer accumulation at reconstruction. Native FP64 is thereby demoted from a hardware requirement to a derived accuracy guarantee obtained by composition over the FP8 primitive. We organize the claim as a five-layer hierarchy -- the FP8 op, Ozaki II, the basic kernels or Berkeley "dwarfs", composite solvers, and full applications -- and, because the dwarf taxonomy already spans scientific computing, establish it by exhibiting the reduction for every dwarf rather than a sample. The claim is falsifiable, and we build the instrument that tests it: a Tensor-Memory Equilibrium (TME) model extending the Roofline with emulation parameters (alpha, beta, gamma). We identify register-level fusion as the mechanism that keeps emulation memory-bound, project recovered FP64 performance across B300 and Rubin against an H100 baseline, and close the kernel coverage with a companion FFT analysis and compensated reductions. The model could have returned a negative verdict; instead it passes across the dwarfs and their compositions. This is the analytical half of a two-part program, with a follow-on implementation to validate the thesis on real silicon.

2026-05-28T03:40:05Z This is the third revised version (July 3rd) of the previous submission (May 28th) version. There is a companion Part (2) paper focusing on Ozaki-style FFT. We have added comprehensive analysis of the register fusion for Blackwell architecture, allowing beta to be within the sensitivity bounds Satoshi Matsuoka http://arxiv.org/abs/2607.03188v1 2026-07-03T10:47:11Z

Episode mining aims to extract subsequences of events that possess certain distinctive properties and constitute facts valuable to the user. Maximal frequent episode mining concentrates on discovery of frequently-appearing subsequences, which are not included into any other larger frequent subsequence. The state-of-the-art for this problem is the MaxFEM algorithm which enumerates possible subsequences, while applying various pruning techniques to accelerate the search. However, this is a computationally-intensive problem: reducing the minimum number of required subsequence occurrences or increasing the length of the subsequence both substantially raise running time, which limits practical use of MaxFEM. In this paper we describe our efforts in designing a high-performing algorithm for this problem. For this we: 1) develop an efficient C++ implementation of MaxFEM, and 2) devise an efficient technique to parallelizing it. As the result, we propose an improved parallel MaxFEM variant, which we call ParMaxFEM. Additionally, we integrate the improved algorithm into Desbordante - a high-performance, open-source data profiler with deep Python integration that treats patterns as first-class entities and allows users to develop their custom programs that can include discovery and validation of patterns. To evaluate our approach we compare both C++ implementations with the original SPMF implementation. Experiments demonstrated that our reimplemented version provides up to $8\times$ speedup over the SPMF baseline, while our parallelization technique provides up to $35\times$ improvement overall (on 8 cores).

2026-07-03T10:47:11Z 2026 39th Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2026, pp. 102-113 Maxim Ivanov Matvei Smirnov Alisa Strazdina George Chernishev 10.23919/FRUCT70069.2026.11506559 http://arxiv.org/abs/2607.02945v1 2026-07-03T04:40:04Z

In large-scale industrial applications, deep learning models that power recommendation and ranking have complex and diverse model architectures. These models are continuously developed and refined by large teams of machine learning engineers, rendering manual optimization infeasible. Consequently, graph-based optimization techniques have become an industry standard for boosting performance, with PyTorch FX transformations leading the charge. These transformations typically rely on a set of human-engineered module-level rewrite rules which are not scalable to diverse model architectures. To address this limitation, we introduce Optimus, a general-purpose model transformation framework built in the PyTorch 2.x (PT2) machine learning compiler. With a concise set of predefined patterns, Optimus applies an efficient greedy search algorithm for pattern matching and replacement, while preserving model semantic. It is designed and implemented as a highly customizable and extensible framework integrated into the PT2 stack. Our evaluation shows that the framework can achieve up to 63% speedup, 6% peak memory reduction, and over 400 second compile time decrease for our industry-scale recommendation models compared to baselines. Optimus is open-sourced together with PyTorch 2.x as a customizable model transformation layer.

2026-07-03T04:40:04Z In Proceedings of the 2026 ACM SIGKDD Conference on Knowledge Discovery and Data Mining Menglu Yu Jiaqi Xu Yuzhen Huang Yanbo Liang Jia Liu Shuai Yang Jason Ansel Elias Ellison Edward Yang Brian Hirsh Jia Chen Ren Will Feng Oguz Ulgen Xu Zhao Daohang Shi Huaqing Xiong Quanyu Zhu Mingming Ding Junqing Zhou Ruilin Chen Yuhang Yang Chi-Keung Luk http://arxiv.org/abs/2606.13501v2 2026-07-02T13:06:12Z

Diffusion Transformers (DiTs) have become the dominant architecture for image and video generation, creating growing demand for efficient DiT serving. Existing systems assign each request a fixed parallel configuration throughout its lifetime. However, DiT workloads exhibit substantial heterogeneity across requests, execution stages, and system conditions, making static parallelism inefficient and often leading to poor GPU utilization and degraded service quality. This paper argues that DiT serving should treat GPU parallelism as a first-class schedulable resource. We present GF-DiT, a policy-programmable runtime for elastic DiT serving that dynamically adapts the parallelism of running requests according to workload demands and service objectives. GF-DiT introduces an asynchronous execution abstraction that decomposes requests into independently schedulable trajectory tasks and enables online GPU reallocation. To make elastic parallelism practical, GF-DiT further proposes group-free collectives, a lightweight communication abstraction that supports low-overhead online formation and reconfiguration of arbitrary execution groups. We implement GF-DiT in vLLM-Omni and evaluate it on representative image and video diffusion workloads. Compared with fixed-pipeline execution with static parallelism, GF-DiT improves throughput by up to 6.01$\times$, reduces mean latency by up to 95%, lowers SLO violation rates by up to 90%, and reduces communication-group setup overhead from 778 ms to approximately 60 $μ$s. Our code is available at https://github.com/SJTU-Liquid/GF-DiT.

2026-06-11T15:51:10Z Xinwei Qiang Yifan Hu Shixuan Sun Jing Yang Han Zhao Chen Chen Yu Feng Jingwen Leng Minyi Guo http://arxiv.org/abs/2606.28565v2 2026-07-02T09:24:03Z

As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets. However, the end-to-end behavior of LLMs couples serving-layer policies with low-level GPU kernel execution and rapidly evolving architectures, forcing slow, deployment-specific benchmarking that is hard to generalize. We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns. It decomposes each serving step into a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, composed through a discrete-event scheduler that also captures mechanisms like prefix caching and continuous batching. KernelSight-LM offers two prediction tiers that trade target-GPU data for accuracy. The cross-generation tier uses no target-GPU measurements, only hardware specifications and kernel microbenchmarks from previously profiled GPUs, and predicts per-kernel latency on an unseen GPU generation to 12.1% error, a 1.8x improvement over the roofline baseline (22.0%). A second target-measured tier adds one model-agnostic kernel-microbenchmark sweep on the target GPU, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline (27.7%). Both tiers require far less target-GPU data than the prior systems they extend. In our simulator, these predictions yield end-to-end median (p50) errors across six model families of 15.4%, 12.8%, and 3.0% (TTFT, TPOT, throughput) in the cross-generation tier and 14.3%, 6.2%, and 2.7% in the target-measured tier, matching dedicated profiling tools while collecting far less on-device data. Beyond prediction, its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.

2026-06-26T19:43:38Z Xiteng Yao Taeho Kim Hengzhi Pei Xinle Liu Kyle Ulrich Leonard Lausen Ashish Khetan Xiang Song George Karypis Martin Herbordt http://arxiv.org/abs/2607.01914v1 2026-07-02T09:12:21Z

Parameter estimation for queueing systems is commonly performed using inter-arrival times, waiting times, or queue-length observations. However, such detailed observations are often unavailable in practical computer systems, where utilization data, such as CPU utilization, is much easier to collect. Utilization data provides only the fraction of time during which the system is busy within each monitoring interval, while the exact arrivals, services, phase transitions, and system states in unobservable periods remain hidden. This paper proposes an expectation-maximization (EM) algorithm for estimating the parameters of Markovian arrival process (MAP)-driven quasi-birth-death (QBD) queueing systems from utilization data. The proposed method formulates the underlying queueing dynamics as a QBD process and derives the expected sufficient statistics for sojourn times, phase transitions, arrivals, and services over both observable and unobservable intervals. These expectations are then used to iteratively update the MAP and service parameters under the maximum likelihood framework. In addition, Akaike's information criterion is introduced to select the appropriate number of MAP phases and mitigate overfitting. The proposed framework enables MAP-based queueing parameter estimation from incomplete utilization observations and provides a practical modeling approach for systems where detailed event-level measurements are difficult to obtain.

2026-07-02T09:12:21Z Chen Li Junjun Zheng Hiroyuki Okamura Tadashi Dohi http://arxiv.org/abs/2606.11357v2 2026-07-01T18:57:44Z

With the growing demand for on-device LLM inference, edge SoCs increasingly integrate NPUs to improve performance and energy efficiency under tight power and thermal budgets. However, practical LLM deployment on current client NPUs remains difficult: widely used quantization formats such as AWQ do not map cleanly onto many existing NPU software stacks, which are often proprietary and expose limited low-level control. In this work, we present TileFuse, a close-to-metal mixed-precision kernel library for AMD XDNA2 NPUs that targets GEMM/GEMV-based operators in quantized LLM inference. TileFuse brings practical low-bit formats such as AWQ-style W4A16 and W8A16 directly onto XDNA2, rather than forcing the model to be reshaped around an NPU-specific quantization scheme. TileFuse co-designs weight layout, metadata placement, mixed-precision microkernels, and array-level dataflow. Specifically, it fuses unpacking, dequantization, and GEMM/GEMV execution into a single kernel flow, introduces an interleaved pre-tiling layout that supports GEMM dimensions up to 32K, and redesigns GEMV dataflow to utilize the full 4x8 AIE array. Across kernel-level evaluations, TileFuse improves performance by up to 121.6% for GEMM and 281% for GEMV over full-precision baselines, while delivering more than 2x performance and energy-efficiency gains over strong iGPU baselines on GEMM. In end-to-end LLM experiments on Ryzen AI laptops, TileFuse achieves up to 2.0x lower prefilling latency with more than 64.6% lower energy consumption. Together, these results show that XDNA2 is a practical target for AWQ-style edge LLM inference and that native NPU support for off-the-shelf quantization can make NPUs substantially more usable in real client deployments.

2026-06-09T18:33:14Z 13 pages excluding reference, 11 figures Wesley Pang Gregory Hyegang Jun Feiyang Liu Deming Chen http://arxiv.org/abs/2607.00740v1 2026-07-01T10:23:00Z

Microservice availability is commonly assessed by fault injection and chaos experiments, but such experiments are costly, operationally risky, and difficult to repeat for every architectural change. Distributed tracing and deployment metadata provide cheaper evidence, yet they usually remain descriptive: they show which services interacted, not what endpoint-level availability property follows. This paper proposes a formal runtime availability model based on stochastic connectivity for resilience-oriented analysis of microservice endpoints. It treats endpoint availability under explicit fault scenarios as a measurable facet of microservice resilience, combining a typed service-dependency graph, a replication map, a probability measure over node and edge states, and request-specific success predicates. Its semantics separates computational failures of service replicas from communication failures of logical dependencies, showing that replication cannot compensate for bottleneck dependencies. The model can be reconstructed from traces and deployment artifacts, parameterized for architectural what-if analysis, and analyzed by Monte Carlo simulation before or alongside fault injection. We define the model, its trace-to-model construction, elementary semantic properties, and a synthetic adequacy study. The study matches closed-form oracle cases within sampling error and exposes boundaries caused by edge bottlenecks, correlated failures, missing traces, and time-dependent failures.

2026-07-01T10:23:00Z Anatoly A. Krasnovsky Anna Maslovskaya