uringscope: Portable, Low-Overhead Observability for io_uring

2026-07-09T00:47:13Z

io_uring moves I/O submission and completion into shared-memory rings. This makes it fast, and it also makes it invisible. strace sees only the ring setup, and the kernel tracepoints that expose the request flow are not stable ABI, so the few tools built on them work only on narrow kernel ranges. We present uringscope, a single-binary, language-agnostic observability tool for io_uring built on CO-RE (Compile Once, Run Everywhere) eBPF. uringscope makes four contributions. The first is a precise model of the request lifecycle and a method to reconstruct per-request flows from kernel events. The second is a technique for attaching portably to an unstable tracepoint surface, using BTF-probed program variants, CO-RE field flavors, and position-independent reads. The third is an evaluation of the tradeoff between overhead and fidelity: on device-bound NVMe workloads uringscope's aggregate mode costs 0.7 to 9.9% of throughput, which is cheaper than every full-fidelity alternative we measured. The fourth is a lightweight correctness mode that reuses the same reconstruction to detect submission-boundary hazards, together with a built-in doctor that turns the measurements into named pathologies with evidence, for operators who are debugging a tail-latency incident rather than browsing histograms.

SPEC CPU2026: Characterization, Representativeness, and Cross-Suite Comparison

2026-07-09T00:37:24Z

Specialized accelerators dominate AI workloads, but CPUs remain critical for orchestrating accelerators and running daily services. CPU performance therefore shapes end-to-end system efficiency, making benchmarks reflect modern workloads and bottlenecks. Yet it remains unclear whether the newest general-purpose CPU benchmark suite changes the architectural conclusions drawn from prior SPEC CPU generations. We present the first comprehensive characterization of SPEC CPU2026 across nine recent Intel, AMD, Ampere, and Nvidia platforms. Compared with SPEC CPU2017, SPEC CPU2026 increases instruction volume and memory footprint and shifts pressure toward emerging bottlenecks, especially instruction-cache stress. These shifts raise two practical questions: how much of the new suite is needed to preserve behavioral coverage, and how should that coverage be interpreted relative to modern domain-specific suites? Using clustering-based representativeness analysis, we identify compact subsets of 4-5 workloads per group that preserve 96.4-99.9% of full-suite behavior. We then compare SPEC CPU2026 against SPEC CPU2017, DCPerf, and MLPerf using cross-suite microarchitectural metrics. SPEC CPU2026 remains a complementary general-purpose suite: it moves closer to datacenter-like frontend pressure than prior SPEC CPU generations, while remaining less vector-intensive than MLPerf and less frontend-extreme than DCPerf. Finally, case studies on page sizes and allocators, prefetching, compiler optimizations, ISA sensitivity, many-core scaling, and rolling round-robin proxy workloads show that SPEC CPU2026 supports architectural studies beyond aggregate scores. Overall, SPEC CPU2026 updates the standardized general-purpose CPU baseline for the next decade of architecture evaluation.

Virtual-Memory Powersort

2026-07-08T16:18:01Z

We give a more space-efficient implementation of adaptive mergesort: Virtual-Memory Powersort. Using internal buffering techniques, we significantly reduce the memory consumption of the algorithm; specifically, for sorting $n$ objects the required buffer area is reduced from space for $n/2$ objects to $O(\sqrt{n \log n})$ objects. While this space-efficiency can be achieved (indeed reduced to $O(1)$) conceptually very easily with known inplace merging algorithms, using these as a drop-in replacement for the standard merge algorithm incurs a substantial slow-down. Virtual-Memory Powersort, by contrast, uses the same number of moves and comparisons as previous Powersort implementations up to an additive $O(n)$ term. We report on an empirical running-time study comparing our implementation against other Powersort variants and state-of-the-art stable sorting methods, demonstrating that almost in-place stable sorting can be achieved with negligible overhead in many scenarios.

Memory Scarcity, Open Models, and the Restructuring of the AI Industry, 2026-2030 -- A quantitative scenario analysis of inference economics, training-cost divergence, and infrastructure solvency

2026-07-08T09:43:14Z

We analyze how four forces restructure the AI industry over 2026-2030: the DRAM/HBM price surge, frontier-capable open-weight models (GLM-5.2), rapid inference-efficiency gains (near-Shannon-limit KV-cache compression, lightweight local runtimes), and the entry of Meta and xAI into compute resale on fleets bought before the memory repricing. Formulating inference economics in dollars per petabyte of bandwidth delivered (\$/PB) -- model-agnostic for bandwidth-bound decode -- we show the entrant-incumbent cost gap never closes: a depreciation conveyor delivers newly amortized fleets to incumbents faster than hardware prices normalize (3.2x in 2026, 1.9x in 2027, re-widening to 3-4x by 2029-30). Training bifurcates into a luxury tier (\$18-38B per frontier run by 2030) and a mass tier (previous-frontier parity via RL/distillation falling toward \$5M). Solvency of the announced buildout is confined to a corridor requiring roughly 2x annual token-demand growth for four years with sticky premium pricing; a measurement critique shows public token trackers overstate monetizable demand, and all pre-Q2-2026 projections predate the industry's shift from token maximization to token minimization. A vintage-breakeven analysis finds 2026 and 2028-29 capacity each fatally exposed to one pricing regime, with only the 2027 vintage robust. A greenfield custom-silicon entrant removes the merchant margin but not the memory premium (central outcome: 25% success/34% mediocre/41% loss, improvable via staged go/no-go gates). China's LineShine LX2 -- domestic HBM on a standard ISA -- decouples its cost curve from the memory crisis. Scenario probabilities: Rotating Landlord Oligopoly 25%, Commoditization Crash 25%, Jevons Absorption 20%, System-Layer Re-differentiation 18%, Geopolitical Bifurcation 12%. Solvency now depends on monetized bandwidth demand, premium stickiness, and vintage ownership.

Think Before You Grid-Search: Floor-First Triage for LLM Serving

2026-07-08T02:47:41Z

LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed. We argue for the reverse discipline: estimation is the analytical layer of profiling -- without it, optimization degenerates to grid search. Floor First is a residual-driven triage workflow. Each decode step is modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity); summing within a resource and maximizing across resources gives an optimistic floor, the plain sum a pessimistic one. Where a measurement lands inside this [max, sum] interval reads out overlap quality before any profiler is opened, and profilers escalate only on residuals above a stated threshold. Deployment alternatives are compared by wall ordering -- which resource wall binds first as load grows -- rather than by point benchmarks. The account is compositional: new attention or state-space variants enter by declaring one module, and the workflow ships as a zero-dependency calculator plus an agent skill that enforces the discipline in agentic optimization loops. As a case study we analyze a DeepSeek-V3.2-style 671B MoE/MLA model on 16 NVIDIA H20 GPUs, whose ridge point of ~74 FLOP/byte (vs ~590 for H100) makes it an extreme decode-oriented part. The floors show TP16 decoding is KV-capacity-limited to ~70 concurrent 8K requests; sparse attention removes the KV-bandwidth term but not the capacity wall; an EP16+DP-attention layout accepts slightly worse same-batch weight traffic for an order-of-magnitude higher capacity wall (~644) -- while single-stream latency favors TP by 2.4x. The layout judgment is thus a computable function of the operating point, explaining why production deployments on identical hardware have shipped opposite attention layouts.

Edge-Inference Governors Need Memory-Clock State

2026-07-08T00:31:46Z

On integrated edge SoCs whose memory fabric is governed independently of the compute clocks, frequency-aware latency estimators let deadline-aware DVFS governors schedule ML inference by modeling latency over CPU and GPU clocks -- but they do not condition on the memory clock (EMC), a deployment state that decides whether a governor meets its deadlines and at what energy. We show this with a deployed, measured governor on Jetson Orin: an EMC-blind GPU-only fit misses 25-28% of cycles at tight deadlines, whereas an EMC-aware two-cell refit holds misses to <=0.9% under a 2% QoS budget -- selecting a budget-feasible operating point proactively, where latency-only reactive calibration cannot repair memory-clock-dependent slope error. Across six models on two Orin SKUs, the core MobileNetV2 and ViT-Small results replicate on both boards; detection and LLM deployments reproduce the failure on the NX. Sustained deployment requires two further state layers: decode horizon, where KV-cache growth erodes tight-deadline feasibility over long responses, and GPU co-tenancy, whose occupancy opens queueing tails. A contract-admission policy composes the three layers from bounded probes and deploys live, including a joint cell decoding 2,000 tokens against an active GPU co-tenant where tenancy-blind admission misses 32% of tokens. Under the reference-probe maximum-guard accounting, every accepted contract is decisively measured-feasible. Fresh-probe stress tests then expose a probe-variance failure mode at knife-edge admissions; a dispersion-banded guard, validated on held-out trials, eliminates the observed probe-induced low-clock selections (held-out aggregate 1.19%); and one bin of headroom yielded observed per-launch compliance across eight fresh launches. A governor's guarantee is thus a measured conservatism ladder -- state repair, probe-variance banding, and headroom for launch realization.

A Sub-linear Low-Rank Solver for Poisson's Equation using Machine Learning Frameworks for GPU Acceleration

2026-07-07T09:02:06Z

In this paper we explore a fast Poisson solver for problems with a solution that is known to be low-rank. We use an adaptive and warm started cross approximation called Cross-DEIM that iterates between index selection and and cross approximation to generate a low-rank solution. This paper focuses on leveraging a modern machine learning framework, PyTorch, as a general purpose array language to implement low-rank solvers based on Cross-DEIM. PyTorch enables native access to GPUs and accelerators but with a user-friendly high-level interface. We investigate statistical leverage scores for the index selection for the cross approximation due to the cost associated with the pivoted algorithms used with the discrete empirical interpolation methods (DEIM and QDEIM) which are historically preferred. The cross approximation is naturally paired with a Discrete Sine Transform (DST) Poisson solver. This allows the Fast Fourier Transform (FFT) to be evaluated in batches along dimensions independently without any global transpose even in higher dimensions. We present performance results running on a A100 GPU and AMD EPYC CPU demonstrating the usefulness of the approach that enables problems sizes that previously were not feasible.

Energy-Efficient GPU DVFS for Fine-Tuning of SLMs on Resource-constrained Embedded Devices

2026-07-07T07:34:56Z

Dynamic Voltage Frequency Scaling (DVFS) on resource-constrained embedded GPU platforms is essential for energy-efficient small language model (SLM) fine-tuning, as privacy- and personalization-driven adaptation increasingly requires local execution and involves repeated forward-backward optimization over many mini-batches, making it substantially more time- and energy-intensive than single-pass inference. To this end, 1) we first characterize the fine-tuning behavior of representative encoder-only SLMs of BERT variants, and autoregressive decoder-only SLMs of Pythia variants on GLUE benchmarks. In addition to the characterizations, 2) we propose a simple yet effective ML-based model selection that selects energy-optimal GPU DVFS settings on resource-constrained embedded platforms. Our results on NVIDIA Jetson AGX Orin demonstrate average 13.11% energy savings (up to 26.73%) over MAXN Mode 0, which has no explicit power cap.

BitLogic: Training Framework for Gradient-Based FPGA-Native Neural Networks

2026-07-07T01:16:56Z

Gradient-based LUT- and logic-gate-based neural networks (LUTNet, LogicNets, DiffLogic, PolyLUT, NeuraLUT, WARP-LUT, DWN, LILogicNet, LightLUT) replace multiply-accumulate arithmetic with Boolean lookups. The same trained checkpoint deploys to GPU as bitwise ops on bit-packed activations, to FPGA as LUT primitives, and to ASIC as standard-cell gates, all from one code path. Yet each method ships its own training pipeline, encoder, connectivity rule, fan-in, and hardware-reporting convention. The natural practitioner question, which of these choices actually matter for accuracy and which for hardware cost, therefore has no answer in the current literature. We release \textbf{BitLogic}, a unified framework that factors the field into a five-axis design space (encoder, connectivity, fan-in, node parameterization, head) and instantiates every prior method under one shared training and evaluation protocol. The framework deliberately omits method-specific procedures such as calibration, pruning, and thresholding, and all evaluations are limited to two-layer feed-forward networks. Combining the per-axis winners identifies a new best-of-space configuration that outperforms every retrained prior on every (dataset, width) cell in which every compared prior fits the shared budget, across MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. We evaluate the best-of-space model on all three backends. On MNIST, the resulting two-layer network reaches ${\sim}126$\,MSamples/s on FPGA, ${\sim}15\times$ the throughput of a bit-packed GPU forward path that itself processes $64$ samples per $64$-bit operation, at four-to-five orders of magnitude less energy.

KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta

2026-07-06T22:18:07Z

Making deep learning recommendation model (DLRM) training and inference fast and efficient is important. However, this presents three key system challenges - model architecture diversity, kernel primitive diversity, and hardware generation and architecture heterogeneity. This paper presents KernelEvolve-an agentic kernel coding framework-to tackle heterogeneity at-scale for DLRM. KernelEvolve is designed to take kernel specifications as input and automate the process of kernel generation and optimization for recommendation model across heterogeneous hardware architectures. KernelEvolve does so by operating at multiple programming abstractions, from Triton and CuTe DSL to low-level hardware agnostic languages, spanning the full hardware-software optimization stack. The kernel optimization process is described as graph-based search with selection policy, universal operator, fitness function, and termination rule, dynamically adapts to runtime execution context through retrieval-augmented prompt synthesis. We designed, implemented, and deployed KernelEvolve to optimize a wide variety of production recommendation models across generations of NVIDIA and AMD GPUs, as well as Meta's AI accelerators. We validate KernelEvolve on the publicly-available KernelBench suite, achieving 100% pass rate on all 250 problems across three difficulty levels, and 160 PyTorch ATen operators across three heterogeneous hardware platforms, demonstrating 100% correctness. KernelEvolve reduces development time from weeks to hours and achieves substantial performance improvements over PyTorch baselines across diverse production use cases and for heterogeneous AI systems at-scale. Beyond performance efficiency improvements, KernelEvolve significantly mitigates the programmability barrier for new AI hardware by enabling automated kernel generation for in-house developed AI hardware.

Bounded-Memory Parallel Image Pulling for Large Container Images

2026-07-06T19:48:04Z

AI/ML workloads increasingly run as containers, where a container image must be downloaded to the host before the workload can start. This cold image pull lands on the critical path whenever a training or inference job scales up or a host is updated, and for GPU workloads it has become the dominant component of startup time as AI/ML images reach 31--48~GiB compressed. We present Disk-Backed Parallel Pull (DBPP), an alternative to the in-memory ordered reassembly used by containerd~2.2, the upstream container runtime. containerd splits layers into chunks fetched concurrently over HTTP range requests, but chunks that arrive out of order accumulate in the runtime heap until a sequential consumer drains them in order. This backlog grows with image size, and on GPU nodes where host memory is shared with frameworks and model weights, it leads to out of memory (OOM) termination of the runtime itself. DBPP writes each chunk directly to its target byte offset on disk, eliminating the ordering dependency and bounding memory regardless of image size. Because each layer lands on disk as a complete, seekable file, DBPP runs SHA-256 digest verification and decompression simultaneously, two passes containerd must run one after the other. In controlled experiments across five production-scale images (up to 48.5~GiB), DBPP reduces peak daemon memory by 8.7--25.3$\times$ while maintaining comparable pull throughput. On a memory-constrained node, containerd~2.2 is OOM-killed pulling a 31.4~GiB image while DBPP completes the same pull. The underlying idea reaches past container images: any pipeline that buffers data in memory only to enforce ordering can move that buffer to disk once the backing store is fast enough, trading a scarce, contended resource for an abundant one.

Adaptive Inference Batching using Policy Gradients

2026-07-06T16:20:19Z

Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic. We investigate whether reinforcement learning (RL) can learn adaptive batching and routing policies that outperform these heuristics, training REINFORCE and PPO agents on a discrete-event simulator validated against queuing theory and production traces (Azure Functions, BurstGPT). We formulate the problem as an MDP over queue state, request type and GPU availability, evaluating across standard Poisson traffic, extreme bursts, real-world traces and heterogeneous multi-GPU routing. Our central finding is a clear boundary condition for RL's value in systems problems. In single-GPU settings, a well-tuned static batching policy is already near-optimal under Poisson-like arrivals and RL offers only marginal gains (+0.1% to +1.0%). In multi-GPU heterogeneous routing, however, where fast and slow requests compete for shared resources, the agent discovers a workload-segregation policy that eliminates Head-of-Line blocking, yielding a 3.5x (348%) improvement over Round-Robin and a 48% improvement over the strongest heuristic baseline (Shortest-Queue), with 60% higher throughput and 25% lower latency while respecting SLA constraints. The policy generalizes to unseen bursty and real-world traffic despite training only on synthetic Poisson arrivals and an attention-augmented policy network converges roughly 20% faster than an MLP baseline. These results suggest RL's advantage over engineered heuristics concentrates in combinatorial, multi-resource decisions rather than single-resource temporal scheduling, a practical distinction for deciding where learned policies justify their engineering cost in production inference infrastructure.

TARE: Tail Aware Evaluation of HPC Job Runtime Prediction

2026-07-06T11:06:26Z

Runtime estimates affect reservation quality, backfilling opportunities, and queue delay in HPC schedulers. Under heavy tailed workloads, however, averaging over jobs can misrepresent scheduling impact because a small fraction of jobs dominates resource usage. This paper presents an empirical evaluation methodology for HPC job runtime prediction that focuses on the tail, combining GeoAccuracy weighted by resource usage with decile and split analyses. Using production traces from NREL Eagle and ALCF Mira/Intrepid, we compare XGBoost and Last2 against the user provided walltime estimate at submission (UserReq). Across all three datasets, evaluation focused on the tail changes the offline conclusion: MeanAccuracy keeps the methods relatively close, whereas GeoAccuracy reveals clearer separation and makes UserReq's strength in the upper tail visible. In the top decile, UserReq achieves the highest GeoAccuracy and lowest underestimation rate on all three datasets, and this pattern remains stable across rolling splits. We then translate this signal into a simple hybrid scheduling policy that keeps XGBoost for most jobs and routes the top decile by proxy_cost at submission to UserReq. Online replay on four production queues reduces mean wait time by up to 8% and increases backfilled jobs by 50% to 115%. These results show that offline evaluation focused on the tail better characterizes prediction quality relevant to scheduling and informs scheduling policy design.

ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution

2026-07-06T09:08:30Z

The transition from sequential to parallel computing is essential for modern high-performance applications but is hindered by the steep learning curve of concurrent programming. This challenge is magnified for irregular data structures (such as sparse graphs, unbalanced trees, and non-uniform meshes) where static scheduling fails and data dependencies are unpredictable. Current Large Language Models (LLMs) often fail catastrophically on these tasks, generating code plagued by subtle race conditions, deadlocks, and sub-optimal scaling. We bridge this gap with ParEVO, a framework designed to synthesize high-performance parallel algorithms for irregular data. Our contributions include: (1) The Parlay-Instruct Corpus, a curated dataset of 13,820 tasks synthesized via a "Critic-Refine" pipeline that explicitly filters for empirically performant algorithms that effectively utilize Work-Span parallel primitives; (2) specialized DeepSeek, Qwen, and Gemini models fine-tuned to align probabilistic generation with the rigorous semantics of the ParlayLib library; and (3) an Evolutionary Coding Agent (ECA) that improves the "last mile" of correctness by iteratively repairing code using feedback from compilers, dynamic race detectors, and performance profilers. On the ParEval benchmark, ParEVO achieves an average 106x speedup (with a maximum of 1103x) across the suite, and a robust 13.6x speedup specifically on complex irregular graph problems, outperforming state-of-the-art commercial models. Furthermore, our evolutionary approach matches state-of-the-art expert human baselines, achieving up to a 4.1x speedup on specific highly-irregular kernels. Source code and datasets are available at https://github.com/WildAlg/ParEVO.

Performance evaluation of scheduling tasks in many-core systems utilizing processes and threads

2026-07-06T08:52:20Z

This study assesses the scalability of process-based and thread-based schedulers for many-core shared-memory systems using a memory-intensive row-wise quick-sort workload on large three-dimensional tensors. The process-based evaluation considers bounded prolific, bounded collective, and three pipe-based producer-consumer schedulers: one-to-one, one-to-many, and many-to-many. These pipe schedulers dynamically stream task identifiers to worker processes, exchanging increased inter-process communication overhead for enhanced runtime load balancing and flexible chunk-based task dispatching. The thread-based evaluation examines static, dynamic, guided, chunk-based, chunk-stealing, adaptive chunk, and AIMD adaptive scheduling strategies. The AIMD scheduler employs an additive-increase multiplicative-decrease policy inspired by TCP congestion control, utilizing an exponentially weighted moving average (EWMA) of CPU utilization to regulate a contention window that limits the number of concurrently active chunks. The adaptive chunk scheduler further modifies chunk size based on observed per-thread execution speed. Experimental results on a 24-core x86-64 platform indicate that thread schedulers deliver the highest overall performance, with dynamic and guided scheduling yielding the most favorable practical outcomes. Among process schedulers, pipe-based designs demonstrate the strongest scalability, with one-to-one pipes excelling for smaller workloads and many-to-many pipes preferred for larger workloads. In summary, lightweight thread scheduling is optimal for shared-memory row sorting, while AIMD/adaptive scheduling and pipe-based process scheduling remain valuable for contention-aware execution, explicit inter-process coordination, and distributed-style heterogeneous workload management.