---

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose Stratum, a system-hardware co-design for Mixture-of-Experts (MoE) model inference. The system is predicated on a future memory technology, Monolithic 3D-Stackable DRAM (Mono3D DRAM), integrated with a Near-Memory Processing (NMP) logic die via hybrid bonding. The core contribution is a set of co-optimizations across the stack. At the hardware level, the authors propose an "in-memory tiering" mechanism to exploit simulated latency variations across the vertical layers of the Mono3D DRAM. At the system level, they introduce a topic-aware scheduler that uses a lightweight classifier to predict query topics, mapping frequently used "hot" experts to the faster memory tiers. The authors claim significant improvements in throughput (up to 8.29×) and energy efficiency (up to 7.66×) over conventional GPU-HBM baselines.

Strengths

1. Comprehensive Scope: The paper attempts a full cross-stack analysis, from device-level simulation of Mono3D DRAM (Sec 6.1.1) to system-level serving policies (Sec 5). This holistic approach is commendable, as optimizations at one level often have unexamined consequences at others.
2. Clear Problem Formulation: The work correctly identifies that MoE models, despite their computational sparsity, are fundamentally bottlenecked by the memory capacity and bandwidth required to store and access massive expert parameters. The motivation for exploring beyond-HBM memory architectures is well-founded.
3. Detailed NMP Microarchitecture: The proposed NMP architecture in Figure 7 (page 6) is described with a reasonable level of detail, including PE, PU, and chip-level organization. This provides a concrete basis for the performance and area modeling, moving beyond high-level conceptual claims.

Weaknesses

The paper’s ambitious claims rest on a chain of optimistic assumptions and insufficiently validated components. The entire structure is fragile, and a weakness in any single link calls the overall conclusion into question.

1. Foundational Reliance on Simulated, Forward-Looking Technology: The entire premise of the paper hinges on the specific performance characteristics of 1024-layer Mono3D DRAM, which does not exist commercially. The crucial latency variation between tiers (a 1.6x difference from fastest to slowest, Sec 6.2.1, pg 11), which is the sole motivation for the tiering mechanism, is derived from Coventor and NeuroSim simulations (Table 1, Figure 14). This is not measured data. If the real-world manufacturing process yields a device with less latency variation, or if thermal crosstalk between layers negates these differences under load, the primary benefit of Stratum's data placement strategy is severely diminished or eliminated. The paper presents these simulated results as fact without a sensitivity analysis.

2. The Fragility of the Topic-Aware Scheduling: The system's performance is critically dependent on a "lightweight topic classifier" (Sec 5.1). The authors report 85.0% accuracy on the Chatbot Arena dataset (pg 9). This implies a 15% misclassification rate. A single misclassification would presumably place a "hot" expert on a slow tier, resulting in worst-case memory latency for that token's computation. The evaluation does not quantify the performance degradation under misclassification. This is a critical omission. A 15% chance of hitting a major performance penalty is unacceptable in a production serving system. The system's performance in the face of this realistic failure mode is not analyzed.

3. Unsubstantiated and Potentially Misleading Performance Claims: The headline claim of "up to 8.29×" improvement is a classic red flag. As seen in Figure 16 (pg 13), this peak number occurs for the smallest model (OLMOE) at a specific sequence length. The gains for the larger and more complex Llama-4-Scout model are a much more modest ~4.5×. More importantly, the fairness of the GPU baseline comparison is questionable. The paper states the baseline is vLLM on H100 GPUs, but provides no details on the configuration. Was the baseline fully optimized? Was it truly memory-bound, or was it bottlenecked elsewhere? Without a detailed roofline analysis or performance counter data from the GPU, it is impossible to verify that the baseline is not a strawman. The system is designed to excel at memory-bound tasks; the authors must first rigorously prove that the baseline workloads are indeed memory-bound.

4. Practical Constraints are Glossed Over:

   • Expert Swapping Cost: Table 4 (pg 12) claims a sub-1% time overhead for expert swapping. This analysis appears to assume ideal conditions. The cost of moving gigabytes of parameter data between DRAM tiers, even with the proposed row-swap buffer, is non-trivial. The paper does not analyze the impact of memory bank conflicts or contention on the ring network during these swaps, especially under heavy load.
   • Thermal and Power Assumptions: The thermal analysis (Sec 6.2.2, pg 11) concludes a 45W power budget for the logic die is feasible with "high-end liquid cooling solutions." This is a best-case scenario. The paper does not model the performance impact of thermal throttling if this ideal cooling is not achieved. The power breakdown in Figure 15 (pg 11) is based on synthesis and simulation, which can often underestimate real-world dynamic power consumption.
   • Generality: The entire optimization relies on requests having clear, classifiable topics with predictable expert affinities. The system's performance on workloads without this property (e.g., general conversation, creative writing, multi-topic queries) is unaddressed. This severely limits the claimed applicability of the approach.

Questions to Address In Rebuttal

1. Please provide a sensitivity analysis showing how Stratum's throughput advantage changes as the simulated latency variation across Mono3D DRAM tiers is reduced. For example, what is the performance gain if the fastest-to-slowest tier latency ratio is only 1.2x, rather than the assumed 1.6x?
2. The topic classifier has a non-zero error rate. Please provide an ablation study that quantifies the impact of topic misclassification (e.g., at 5%, 15%, and 25% error rates) on the overall system throughput and latency distribution.
3. Please provide evidence that the GPU baselines were not configured as a strawman. Specifically, provide profiler data (e.g., from NSight) for the H100 running vLLM to demonstrate that the workload is fundamentally memory-bandwidth-bound and that the GPU's compute resources are not being underutilized for other reasons.
4. The expert swapping cost analysis in Table 4 seems to assume an idle system. How does this overhead change when swapping occurs concurrently with active inference requests that are contending for the same memory banks and on-chip network resources?
5. How does the system perform on a mixed workload of queries where a significant fraction (e.g., 50%) has no strong topic affinity and thus activates experts in a pseudo-random or uniform pattern? This would test the system's performance when its primary optimization heuristic fails.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Stratum, a comprehensive system-hardware co-design for accelerating the serving of Mixture-of-Experts (MoE) Large Language Models. The work's central and most compelling idea is the synergistic mapping between an emerging hardware technology and an emergent property of large AI models. Specifically, it leverages Monolithic 3D-Stackable DRAM (Mono3D DRAM), a technology characterized by non-uniform access latencies across its vertically stacked layers. The authors astutely observe that this physical heterogeneity can be exploited by mapping frequently accessed "hot" experts—identified via a topic-based prediction system—to the faster, upper memory tiers, while relegating "cold" experts to the slower, deeper tiers.

The proposed system integrates this tiered Mono3D DRAM with a near-memory processor (NMP) on a logic die, connected via high-density hybrid bonding. The co-design extends across the stack, encompassing a hardware architecture for the NMP (Section 3.2, page 5), operator mapping strategies for MoE and attention computations (Section 4, page 6), and a system-level scheduler that classifies incoming queries by topic to inform expert placement (Section 5, page 8). The cross-layer evaluation demonstrates significant improvements in throughput and energy efficiency over conventional GPU-HBM baselines.

Strengths

1. Novel and Elegant Co-Design Synergy: The core contribution is not merely the application of a new memory technology, but the profound insight that connects the physical properties of that technology to the behavioral properties of the target application. The paper brilliantly turns a potential hardware drawback—the variable access latency of deep Mono3D DRAM stacks (visualized in Figure 2, page 3)—into a key architectural feature. This is elegantly matched with the observation of topic-specific expert affinity in MoE models (profiled in Figure 4, page 4), creating a powerful, cross-stack optimization principle. This represents a mature form of co-design that goes beyond simple acceleration.

2. Forward-Looking and Relevant Problem Domain: The paper tackles two critical, forward-looking problems simultaneously: (1) the memory wall in serving extremely large models like MoEs, and (2) the architectural implications of next-generation 3D memory integration. By moving beyond the well-trodden ground of HBM-based PIM/NMP, the authors provide a valuable architectural blueprint for a technology (Mono3D DRAM) that is a strong candidate for future high-performance systems. This positions the work not as an incremental improvement, but as a pioneering exploration of a future design space.

3. Comprehensive, Multi-Layered Approach: The strength of the work lies in its completeness. The authors have considered the problem from the device level (DRAM timing parameters in Table 1, page 10), through circuit and architecture (NMP design in Figure 7, page 6), and up to the system software level (topic-aware scheduling in Figure 6, page 5). This end-to-end perspective lends significant credibility to the claimed performance benefits, as it accounts for constraints and overheads at each layer of the system.

4. Contextualization within the Field: The paper does a good job of situating itself relative to prior work in PIM/NMP for transformers (e.g., AttAcc, Duplex) and highlighting its key differentiators, primarily the shift to Mono3D DRAM and the exploitation of its unique properties (Section 7, page 13). It builds upon the established trend of moving compute closer to memory while introducing a novel axis of optimization (latency tiering).

Weaknesses

1. Contingency on an Emerging Technology: The work's greatest strength is also its primary weakness. The entire premise and the impressive results are predicated on the maturation and adoption of Monolithic 3D DRAM as described. While this is a hallmark of forward-looking architectural research, the practical impact is contingent on manufacturing trends and the resolution of potential yield and thermal challenges associated with such dense 3D integration.

2. Sensitivity to Model Behavior: The co-design is exquisitely tuned to the phenomenon of topic-expert specialization. This raises questions about its robustness. If future MoE training methodologies were to change—for instance, by explicitly encouraging more uniform expert usage to improve generalization—the core premise of Stratum's data placement strategy would be undermined. The system's performance is tightly coupled to a specific, albeit currently observed, emergent behavior of MoE models.

3. Potential Overheads in Dynamic Scenarios: The paper demonstrates that the overhead of swapping experts between tiers is negligible for a given batch transition (Table 4, page 12). However, in a real-world serving scenario with a highly diverse and rapidly changing mix of query topics, the frequency of these swaps could increase. There is a potential risk of "memory thrashing" at the tier level if the topic distribution of incoming requests is chaotic, which could degrade performance in ways not fully captured by the current evaluation.

Questions to Address In Rebuttal

1. Robustness to Model Drift: The core optimization relies on strong topic-expert affinity. How does the performance advantage of Stratum degrade as this affinity weakens? For example, what happens if the hot/cold expert distinction becomes less pronounced, with usage probabilities being more evenly distributed?

2. Impact of Prediction Inaccuracy: The system's effectiveness is front-loaded by a lightweight topic classifier. The evaluation in Section 5.1 (page 9) shows high accuracy, but not 100%. What is the performance penalty of a misclassification? For instance, if a "math" query is misclassified as "legal," the system would presumably preload the wrong experts into the fast tiers, leading to slower execution. Can the authors quantify this impact?

3. Generalizability of the Architecture: The Stratum NMP and tiered memory system is highly optimized for the sparse, dynamic nature of MoE models. Does this specialized architecture offer significant benefits for other classes of models? For example, could the tiered memory system be repurposed to accelerate traditional dense transformers by placing attention KV caches in faster tiers, or is its utility fundamentally tied to the expert-based structure of MoEs?

4. Scaling and Physical Constraints: The paper assumes up to 1024 vertically stacked layers. As the stack depth increases, the latency disparity between the top and bottom tiers also grows (as shown in Figure 14, page 11). Is there a point of diminishing returns where the slowest tier becomes so latent that it's impractical, or where thermal density becomes an insurmountable challenge for the NMP logic die?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes Stratum, a system-hardware co-design for serving Mixture-of-Experts (MoE) Large Language Models. The system is built upon an emerging memory technology, Monolithic 3D-Stackable DRAM (Mono3D DRAM), integrated with a Near-Memory Processor (NMP) on a logic die via hybrid bonding.

The primary novel claim, as I deconstruct it, is twofold: 1. At the hardware level: The architectural exploitation of an inherent physical property of Mono3D DRAM—namely, the layer-dependent access latency variation (Figure 2, page 3)—to create fine-grained, physical, intra-chip memory tiers. 2. At the system level: A co-design that maps a known software behavior of MoE models—topic-based expert affinity ("hot" vs. "cold" experts)—directly onto these novel physical memory tiers to optimize data access.

The work claims to be the first to propose such a co-design leveraging this specific memory technology for MoE serving.

Strengths

The core strength of this paper lies in its identification and architectural exploitation of a device-level characteristic.

1. Novel Architectural Insight: The central idea of turning a physical-layer non-ideality (latency variation across vertically stacked wordlines) into a system-level feature (memory tiering) is genuinely novel and insightful. Instead of designing for the worst-case latency as is conventional, the authors embrace the heterogeneity. This demonstrates a deep, cross-layer thinking that is rare and commendable. This is detailed in Section 2.1 (page 3) and visualized in Figure 14 (page 11).

2. Synergistic Co-Design: The novelty is further strengthened by the tight coupling between the hardware insight and the application domain. The concept of expert affinity in MoE models is known (e.g., [87], [33]), but prior work has not had a hardware substrate that so elegantly maps to this logical concept. The mapping of hot/cold experts to fast/slow physical tiers (Section 5.2, page 9) is a powerful and novel synthesis of existing ideas from different domains.

3. Significant Delta from HBM-based PIM/NMP: The paper correctly differentiates itself from prior art in NMP for LLMs (e.g., Duplex [89], AttAcc [67]), which are based on HBM. The architectural shift to Mono3D DRAM with its dense hybrid bonding fundamentally changes the design constraints (higher internal bandwidth, no TSV bottleneck), justifying a new NMP design. The novelty here is in the adaptation to and exploitation of this new memory paradigm.

Weaknesses

My critique is centered on the precise boundaries of the novelty and the potential transience of the underlying physical premise.

1. Constituent Ideas Are Not Novel: While the synthesis is novel, the paper could be more precise about the prior art for its constituent components. The observation of expert affinity is not new ([87]), memory tiering as a concept is decades old, and near-memory processing for transformers is an established research direction. The paper's novelty rests entirely on the combination and the physical mapping. The authors should frame their contribution more sharply as a novel architectural mapping rather than implying the invention of these base concepts.

2. NMP Architecture is Evolutionary, Not Revolutionary: The proposed Stratum NMP architecture (Figure 7, page 6) is a well-engineered solution but does not appear to introduce fundamentally new processing concepts. It combines tensor cores, a ring interconnect, and SIMD special function units, which are all well-understood building blocks in accelerator design. The "delta" compared to prior NMP designs like Duplex [89] seems to be primarily in the interconnect topology (ring vs. global buffer/crossbar) and the direct integration with Mono3D banks. This is a significant engineering adaptation, but its conceptual novelty as a processor architecture is limited.

3. Contingency on a Technological "Flaw": The entire premise of in-memory tiering hinges on the significant latency variation across Mono3D DRAM layers. This variation stems from the staircase structure for wordline contacts (Figure 2, page 3). It is conceivable that device and circuit designers will view this as a defect to be engineered away in future generations of the technology, striving for uniform access times. If they succeed, the core hardware motivation for this work vanishes. The novelty is thus tied to a potentially transient property of an emerging technology.

Questions to Address In Rebuttal

The authors should use the rebuttal to clarify the following points regarding the novelty and significance of their contribution.

1. The concept of topic-based expert affinity and classifying experts as "hot" or "cold" has been explored for optimizing MoE serving on existing hardware (e.g., [87]). Please confirm that your primary novel contribution is not the identification of this affinity, but rather the creation of a new hardware architecture (tiered Mono3D) that provides a physical substrate for this logical classification, and the co-design that maps between them.

2. The NMP architecture in Figure 7 integrates tensor cores, a ring network, and SIMD function units. Beyond the adaptation to leverage the high internal bandwidth of Mono3D DRAM, what are the fundamentally new architectural concepts in the processing unit or interconnect design itself when compared to the principles used in prior NMP systems like Duplex [89]?

3. The proposed tiering mechanism is predicated on the access latency heterogeneity in Mono3D DRAM. How fundamental is this property? Is it not a target for elimination by device-level engineers in future iterations of the technology? How would the value proposition of Stratum change if next-generation Mono3D DRAM achieved, for instance, less than a 20% latency variation between the fastest and slowest layers?

4. The system introduces significant complexity, including a topic classifier, a dynamic SLO-aware scheduler, and a memory mapper for expert swapping. Could a simpler baseline, such as caching the weights of the most globally popular experts (independent of topic) in a large SRAM on the logic die, achieve a substantial fraction of the performance benefit without the overhead of dynamic topic classification and physical data migration between tiers? A comparison against such a baseline would help quantify the benefit of the novel tiering mechanism itself.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose "Kelle," a hardware-software co-design for LLM inference on edge devices that uses eDRAM as the primary storage for the KV cache. The core contributions are twofold: 1) an algorithm named AERP that combines attention-based token eviction with a recomputation policy to manage the KV cache size, and 2) a two-dimensional adaptive refresh policy (2DRP) for the eDRAM that modulates refresh rates based on token importance and bit significance, thereby intentionally introducing errors to save power. These are implemented in a custom accelerator. The authors claim significant speedup (3.9×) and energy savings (4.5×) over an SRAM-based baseline. While the problem is relevant, the work rests on a foundation of questionable experimental design and insufficiently substantiated claims, particularly concerning the main performance comparison.

Strengths

1. Problem Formulation: The paper correctly identifies the KV cache as a primary memory bottleneck for LLM inference on resource-constrained edge devices, and the exploration of eDRAM as an alternative to SRAM is a logical direction.
2. Ablation Study Structure: The authors have made an effort to isolate the performance contributions of their various techniques. The comparisons between Original+SRAM, Original+eDRAM, AEP+SRAM, and AERP+SRAM (Section 8.1.2, Figure 13) provide some insight into the relative benefits of each component of their design, assuming the baselines are fair.
3. Comprehensive Algorithm Design: The proposed solution is multifaceted, addressing the eDRAM challenge from multiple angles: cache content management (AERP), refresh power (2DRP), and data lifetime (Kelle Scheduler).

Weaknesses

My primary role is to ensure the rigor of published work. This paper, in its current form, exhibits several critical weaknesses that undermine the validity of its conclusions.

1. Fundamentally Confounded Main Comparison: The central claim of 3.9× speedup and 4.5× energy savings is derived from a comparison between Kelle+eDRAM and an Original+SRAM baseline. As stated in Section 8.1.1, the SRAM baseline is area-matched, resulting in a smaller 24×24 systolic array compared to Kelle's 32×32 array. This is an unacceptable confounder. The reported gains are not solely from the memory subsystem innovation but are significantly influenced by the fact that the Kelle system has nearly 80% more compute resources (32²/24² ≈ 1.78). This renders the headline performance numbers highly misleading. A valid comparison would require matching the compute capabilities.

2. Insufficient Justification for Heuristics: The proposed AERP algorithm relies on a seemingly arbitrary threshold. In Section 4.1.2, the decision to recompute KV vectors for a token if it is retained in "at least 50% of the heads" lacks any theoretical or empirical justification. The paper presents no sensitivity analysis for this crucial parameter. How does performance change at 40% or 60%? Without this, the chosen value appears to be a "magic number" that may have been tuned for the specific experiments shown.

3. Superficial Evaluation of Error Injection: The 2DRP policy is predicated on the idea that LLMs are tolerant to a certain level of data corruption in the KV cache. The authors' evaluation of this tolerance is dangerously shallow. They rely primarily on perplexity (PPL) (Figure 8), which is a coarse, statistical measure of fluency. The claim in Section 7.1 that an average retention failure rate of 2e-3 has a negligible impact is not sufficiently proven. While Table 5 shows "comparable" scores on TruthfulQA, a minor drop in accuracy on a multiple-choice benchmark does not adequately characterize the risk of catastrophic failures, such as factual hallucination or safety violations, which could be triggered by specific bit-flips in critical tokens. A system that knowingly corrupts data requires a far more rigorous and targeted evaluation of its failure modes.

4. "Straw Man" Baseline Scheduler: The baseline computation pattern shown in Figure 12a appears intentionally inefficient, with long, serialized data dependencies that inflate the data lifetime in eDRAM. Any reasonably optimized system would attempt to co-schedule dependent operations to improve data locality and reduce lifetime. The gains attributed to the "Kelle Scheduler" may be significantly overstated by comparing against this naive baseline.

5. Understated System Complexity: The paper proposes a complex memory subsystem (Section 5.1, Figure 10) with data split into four groups (MSB/LSB for HST/LST tokens) across 32 banks, managed by custom eviction and refresh controllers. While the area of the systolic evictor is mentioned (Section 8.1.4), the overhead of the intricate control logic, potential timing challenges, and addressing complexity for this fine-grained management is not adequately discussed or quantified.

Questions to Address In Rebuttal

The authors must address the following points directly to salvage the credibility of this work.

1. Provide a new end-to-end performance comparison (speedup and energy) against an SRAM-based baseline that is compute-matched (i.e., also uses a 32×32 systolic array). This is the only way to isolate the true contribution of the Kelle memory system. The area and power of this new, larger SRAM baseline must be reported.

2. Present a sensitivity analysis for the 50% popularity threshold in the recomputation policy (Section 4.1.2). How do system performance, accuracy, and storage savings vary as this threshold is changed? Justify your final choice of 50%.

3. The evaluation of the 2DRP's error injection is insufficient. Please provide a more rigorous analysis of its impact on model factuality and safety. This should go beyond PPL and standard zero-shot tasks and utilize benchmarks specifically designed to detect factual inconsistency (e.g., FactScore) or adversarial safety risks.

4. Justify the selection of the "Baseline" scheduler in Figure 12. Is this representative of typical, optimized LLM inference schedulers, or is it a worst-case scenario designed to amplify the benefits of the Kelle scheduler? Please compare against a more aggressive, locality-aware baseline scheduler.

5. In Section 8.4.1, you claim Kelle can support long contexts (up to 60K tokens) by offloading to DRAM, stating that the paging process is simplified. However, prefetching this volume of KV data from DRAM for every token generation step would incur substantial latency and energy overhead. Please provide a quantitative analysis of this "paging" overhead and demonstrate that it does not negate the on-chip benefits for such long sequences.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper, "Kelle," presents a hardware-software co-design for efficient Large Language Model (LLM) inference on edge devices, targeting the significant bottleneck of the Key-Value (KV) cache. The core contribution is the principled replacement of traditional on-chip SRAM with embedded DRAM (eDRAM) as the primary storage for the KV cache. Recognizing that eDRAM's high density comes at the cost of power-intensive periodic refreshes, the authors propose a suite of tightly-coupled optimizations to mitigate this overhead. These include: 1) an Attention-based Eviction and Recomputation Policy (AERP) to manage the limited cache size and reduce the lifetime of stored data, and 2) a Two-Dimensional Adaptive Refresh Policy (2DRP) that cleverly exploits the inherent error tolerance of LLMs by reducing refresh rates for less significant bits and less important tokens. These policies are instantiated in a complete accelerator design featuring a custom memory subsystem and a novel "systolic evictor" to implement the AERP with minimal latency. The work demonstrates significant improvements in speed and energy efficiency, positioning eDRAM as a viable and powerful component for future edge AI systems.

Strengths

1. Excellent Problem-Solution Fit and Timeliness: The paper identifies one of the most pressing problems in deploying modern AI: the memory capacity wall for LLM inference. The proposal to use eDRAM is a fantastic synthesis of an existing, mature technology with a new and critical problem domain. While eDRAM has been explored for CPU caches and CNN accelerators, its application to the LLM KV cache is both novel and exceptionally well-motivated. The authors correctly identify that the KV cache is fundamentally a capacity and bandwidth problem, which aligns perfectly with eDRAM's primary strengths over SRAM.

2. Insightful and Elegant Co-design Principle: The true strength of this work lies not just in using eDRAM, but in the deep, synergistic co-design. The central insight, empirically validated in Figure 8 (page 5), is that LLMs exhibit graceful degradation to certain types of data corruption. The 2DRP policy (Section 4.2, page 5) is a direct and clever exploitation of this property, mapping a characteristic of the software model (variable importance of data) directly onto a physical knob in the hardware (refresh rate). This moves beyond a simple component swap into a truly holistic system design, which is commendable.

3. Comprehensive System-Level Approach: The authors present a complete system, from algorithm to architecture. They do not stop at the conceptual level but detail the hardware necessary to make their vision a reality, including the Kelle accelerator architecture (Figure 9, page 6), the memory subsystem layout (Figure 10, page 7), and the novel systolic evictor (Section 5.3, page 7). This end-to-end thinking significantly increases the credibility and impact of the work, demonstrating a clear path from idea to implementation.

4. Thorough and Illuminating Evaluation: The experimental evaluation is extensive, covering multiple models, datasets, and a strong set of ablation studies (Section 8.3, page 11). The comparison against four distinct baselines in Section 8.1.1 (page 10) is particularly effective, as it allows the reader to disentangle the benefits derived from simply using eDRAM versus those from the AERP and 2DRP algorithms. This methodical breakdown provides clear evidence for the efficacy of each proposed contribution.

Weaknesses

1. Limited Contextualization Against Other Memory Technologies: The paper does an excellent job of framing the SRAM vs. eDRAM trade-off. However, the broader field of computer architecture is actively exploring a rich landscape of emerging memory technologies (e.g., MRAM, RRAM, FeFETs) for on-chip memory. These technologies could potentially offer the density benefits of eDRAM without the refresh overhead, albeit with their own challenges (e.g., write latency, endurance). A discussion on where Kelle's principles fit within this wider context would strengthen the paper's long-term relevance. For instance, could the AERP policy be beneficial for NVMs to manage write endurance? This is a missed opportunity to position the work more broadly.

2. Understated Implementation Complexity: The proposed co-design, particularly the 2DRP policy, introduces non-trivial control logic. The memory controller must now track importance scores and manage multiple, fine-grained refresh domains dynamically. While the paper quantifies the overhead of the systolic evictor (Section 8.1.4, page 11), a more detailed analysis of the control overhead for the memory system itself would be beneficial. The elegance of the idea may hide a significant complexity cost that should be acknowledged and justified more explicitly.

3. The Fundamental Assumption of Error Tolerance: The 2DRP policy hinges on the assumption that LLMs are resilient to bit-flips in certain data. While the authors show this holds for their 16-bit evaluation, this property may become less robust as the field pushes towards aggressive, low-bit quantization (e.g., 4-bit, 3-bit, or even binary formats). In such schemes, every single bit carries substantially more information, and a single bit-flip could be catastrophic. The paper would be strengthened by a discussion of how its core principles might adapt or break under these more aggressive quantization scenarios.

Questions to Address In Rebuttal

1. Could the authors elaborate on how their co-design principles, particularly AERP and the concept of mapping data importance to hardware properties, might compare to or be adapted for other emerging on-chip memory technologies like MRAM or RRAM, which offer density without refresh but introduce challenges like write endurance and latency?

2. The 2DRP policy requires a sophisticated eDRAM controller capable of managing dynamic, fine-grained refresh domains. Could the authors provide a more detailed estimate of the area and power overhead of this control logic relative to a standard eDRAM controller? Is there a risk that this control complexity negates some of the energy savings from reduced refresh operations?

3. The viability of 2DRP rests on the error tolerance of the LLM. How do the authors expect this approach to perform with models that are aggressively quantized to 4-bit or lower representations? Does the reduced information redundancy in low-bit formats present a fundamental challenge to a strategy that relies on tolerating data corruption?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes "Kelle," a hardware-software co-design for efficient LLM inference on edge devices. The central idea is to use embedded DRAM (eDRAM) as the primary on-chip storage for the KV cache to leverage its high density and low leakage power compared to SRAM. To mitigate the primary drawback of eDRAM—the high cost of periodic refresh operations—the authors introduce a suite of techniques: (1) an Attention-based Eviction and Recomputation Policy (AERP) to manage the cache size and data lifetime, (2) a Two-Dimensional Adaptive Refresh Policy (2DRP) to reduce refresh frequency by exploiting data criticality, and (3) a custom hardware accelerator featuring a novel "Systolic Evictor" to implement these policies efficiently. The authors claim this co-design results in significant speedup and energy savings.

My review will assess the novelty of these constituent ideas by situating them within the landscape of prior art.

Strengths

The primary strength of this work lies in the synthesis of multiple techniques into a cohesive system targeted at a very specific and timely problem. While many of the individual concepts have precedent in other domains, their application and integration for managing an LLM KV cache in eDRAM is a novel endeavor.

The most genuinely novel contribution at the microarchitectural level appears to be the Systolic Evictor (Section 5.3, page 7). The concept of a computational unit that operates in a systolic, on-the-fly manner, tightly coupled with the main systolic array (RSA), to identify the eviction candidate without stalling the pipeline is a clever and, to my knowledge, new design. It solves a specific performance problem created by the proposed eviction policy in an elegant way.

Weaknesses

My main concern is that the paper presents several core algorithmic and policy-level ideas as fundamentally new, when they are in fact adaptations or direct parallels of well-established concepts from prior work. The "delta," or the degree of novelty, is smaller than implied.

1. Use of eDRAM for On-Chip Acceleration: The foundational premise of using eDRAM as a dense, low-power alternative to SRAM for on-chip neural network accelerators is not new. This path has been well-trodden, particularly in the domain of CNNs. For instance, DaDianNao [15] explored eDRAM for machine learning supercomputers, and RANA [76] specifically proposed refresh-optimized eDRAM for efficient neural acceleration. The novelty here is the application to LLMs, not the core architectural choice.

2. Attention-based Eviction Policy: The AERP eviction policy, which prunes tokens based on their summed attention scores (Equation 3, page 4), is conceptually almost identical to the "heavy-hitter" identification method proposed in H2O [98]. H2O also identifies important tokens by their high accumulated attention scores. The paper needs to clearly articulate the fundamental algorithmic innovation that differentiates its eviction policy from H2O, beyond the novelty of its hardware implementation (the Systolic Evictor). As it stands, the policy itself appears derivative.

3. Two-Dimensional Adaptive Refresh Policy (2DRP): The 2DRP is presented as a key innovation, but it is a synthesis of two known principles.

   • Importance-Aware Refresh: The idea of reducing refresh rates for less critical data is not new. RANA [76] did precisely this by linking refresh frequency to the impact of bit errors on CNN accuracy. Kelle's first dimension—refreshing High Score Tokens (HST) more frequently than Low Score Tokens (LST)—is a direct analogue of this principle applied to the LLM domain.
   • Bit-level Differential Refresh: The second dimension—refreshing Most Significant Bits (MSBs) more frequently than Least Significant Bits (LSBs)—is a classic technique in approximate and fault-tolerant memory design. The principle that errors in LSBs are less impactful than errors in MSBs is fundamental.
   • The novelty of 2DRP is therefore in the combination of these two axes for the specific use case of an LLM KV cache. It is a good piece of engineering, but not a fundamentally new concept in memory management.

4. KV Vector Recomputation: The trade-off of computation for memory is a cornerstone of computer science. While applying it to the KV cache is logical, the policy of recomputing based on token "popularity" (Section 4.1.2, page 5) is a heuristic. The novelty is limited to the specific formulation of this heuristic.

In summary, the paper constructs a powerful system, but it does so primarily by adapting and combining existing ideas. The work would be stronger if it more accurately positioned its contributions as a novel synthesis and application of these principles, rather than implying they are new from first principles.

Questions to Address In Rebuttal

1. Please explicitly detail the fundamental algorithmic difference between your attention-based eviction policy in AERP and the heavy-hitter identification method in H2O [98]. Why should your policy be considered a novel contribution distinct from this prior work?

2. Could the authors reframe the contribution of 2DRP? Given prior art like RANA [76] on importance-aware refresh for NNs and established work on bit-level differential reliability, please clarify if the novelty lies in the discovery of these principles or in their specific synthesis and application to the LLM KV cache.

3. The recomputation policy relies on a heuristic threshold where input vectors (x_n) are stored if the token is popular in "> 50% of the heads". How was this threshold determined? Please provide an analysis of the system's sensitivity to this specific value, as the robustness of a heuristic is key to evaluating its contribution.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper presents LongSight, an algorithm-hardware co-design that repurposes a CXL-based dense retrieval accelerator, DReX, for sparse attention in large-context LLMs. The core mechanism is a hybrid attention model: a dense sliding window of recent tokens is handled by the host GPU, while attention over the long-tail historical context is offloaded to DReX. This offload leverages a multi-stage filtering process, initiated by a sign-based filter (SCF) accelerated in-memory, to retrieve a top-k set of relevant Key-Value pairs. The authors claim this system can support context lengths up to one million tokens on a single GPU and achieves significant throughput improvements over dense baselines.

However, the work rests on a series of optimistic assumptions regarding the generalization of its sparse approximation, the practicality of its complex hyperparameter tuning, and the performance of an emulated hardware interface. The evaluation relies heavily on projections for its most impressive claims, leaving the robustness and real-world viability of the system in question.

Strengths

1. Pragmatic System Partitioning: The hybrid dense-sparse approach, which keeps recent tokens in GPU HBM for dense attention while offloading the historical context, is a logical and practical design choice. It correctly identifies that recent tokens are often most important, providing an accuracy backstop for the sparse approximation.
2. Detailed Hardware-Aware Data Layout: The paper provides a well-considered mapping of the logical data hierarchy (Key Blocks, Context Slices, User Partitions) onto the physical hardware of DReX (Section 7.3.3, page 9). This demonstrates a clear understanding of the need to co-design data structures with the memory system to maximize parallelism and bandwidth.
3. Repurposing Existing Hardware: The central idea of extending a dense retrieval accelerator for sparse attention is resourceful. It proposes a path to broader utility for specialized hardware, which is a commendable system-level goal.

Weaknesses

1. Core Claims are Based on Extrapolation, Not Measurement: The headline claim of supporting and accelerating 1M token contexts is not empirically demonstrated. As noted in the caption of Figure 7 (page 12), performance for context lengths above 128K is "projected based on performance at 128K context." A projection is not a result. For a systems paper making bold performance claims, the lack of measured data at the most challenging scales undermines the entire contribution. The scaling of system bottlenecks (e.g., CXL overhead, data structure management) may not be linear as assumed.
2. The Algorithmic Foundation is Potentially Fragile: The entire performance gain hinges on the effectiveness of the ITQ-enhanced SCF. The methodology for this is questionable:
   • The ITQ rotation matrix is trained on a mere 1K-token sequence (Section 5.4, page 6) but is expected to generalize and remain effective across a 1,000,000-token context. This is a heroic assumption. There is no evidence that the statistical properties of Key/Query vectors remain stable enough for this to hold.
   • The evaluation relies solely on perplexity. While a useful intrinsic metric, it is not a substitute for performance on downstream, long-context reasoning tasks. A small hit in perplexity can sometimes translate to a catastrophic failure in tasks requiring retrieval of specific, non-obvious facts from deep within the context.
3. Impractical Hyperparameter Sensitivity: The authors themselves concede in Section 9.3 (page 13) that "optimal parameters (i.e., window size, k, and SCF thresholds) are heavily context-dependent and impact end-to-end performance." This is a critical flaw for a general-purpose accelerator. It implies that for any new model, dataset, or even target context length, a user must engage in a complex, multi-dimensional tuning sweep to achieve the reported performance. This severely limits the system's practical usability and suggests the proposed method is more of a brittle proof-of-concept than a robust solution.
4. Insufficient and Misleading Baseline Comparisons: The primary baselines are 1-GPU and 2-GPU dense attention. It is a foregone conclusion that a sparse method will outperform a dense one at extreme context lengths. The more scientifically rigorous comparisons would be against state-of-the-art software-based sparse attention techniques (e.g., highly optimized block-sparse kernels) running on the same GPU hardware. The paper discusses such methods in the background (Section 3.1) but fails to compete against them in the evaluation (Section 9). The comparison against a simple sliding window attention (Figure 10, page 13) is a weak benchmark.
5. Hardware Performance is Based on Emulation: The evaluation "emulate[s] the CXL interface using a dual-socket Intel Xeon...platform" (Section 8.2, page 11). CXL is a complex interconnect, and its real-world performance involves subtle effects of protocol overhead, contention, and NUMA effects that are difficult to capture with such an emulation. A paper proposing a CXL-based hardware solution must be held to a higher standard of fidelity in its performance model. Relying on an emulation for the critical data path injects significant uncertainty into the final performance numbers.

Questions to Address In Rebuttal

1. Please provide a rigorous justification for projecting performance from 128K to 1M tokens. What evidence supports the assumption that no new system bottlenecks emerge at these larger scales? Can you provide data from a scaled-down model or hardware configuration that validates this linear scaling assumption?
2. The ITQ matrix is trained on a 1K token sequence. Please provide an ablation study showing how the model's accuracy (in perplexity and, ideally, a downstream task like needle-in-a-haystack) degrades as the context length increases from 1K to 128K. How can the authors be confident this approach does not fail catastrophically at 1M tokens?
3. Given that the system requires extensive, context-dependent hyperparameter tuning, what is the proposed methodology for a practitioner to deploy LongSight on a novel LLM or for a new application? Please quantify the tuning overhead.
4. Why were state-of-the-art software-only sparse attention methods, which require no specialized hardware, omitted as primary performance baselines in Figure 7? A fair comparison would show where LongSight provides a benefit over what is achievable on the commodity GPU alone.
5. How does your CXL emulation model account for protocol overhead and contention from multiple concurrent requests targeting the DReX device? Please quantify the sensitivity of your results to a 2x or 5x increase in CXL tail latency, which can be common in real-world systems under load.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces LongSight, an algorithm-hardware co-design framework to accelerate large language model (LLM) inference for extremely long contexts. The central and most compelling idea is the repurposing of a compute-enabled CXL memory expander called DReX, originally designed for dense retrieval, to serve as an active offloading device for the LLM's Key-Value (KV) cache. LongSight implements a hybrid attention mechanism: a conventional dense attention is performed on a sliding window of recent tokens stored in the GPU's local HBM, while sparse attention for the vast historical context is handled by DReX. The GPU offloads query vectors to DReX, which leverages its in- and near-memory processing capabilities to efficiently perform a top-k similarity search, returning only the most relevant Keys and Values. This approach dramatically reduces the memory and compute burden on the GPU, enabling a single-GPU system to efficiently handle context lengths of up to one million tokens, a scale that is currently only feasible with large multi-GPU setups.

Strengths

The primary strength of this work lies in its elegant synthesis of ideas from several distinct but converging fields of research.

1. Novel Connection of Problem Domains: The authors make a brilliant conceptual leap by identifying the architectural similarity between top-k dense retrieval (the problem DReX was built for) and the core operation of a common form of sparse attention (finding the keys with the highest dot-product similarity to a query). By repurposing a specialized accelerator, they provide a powerful, hardware-grounded solution to the sparse attention problem, rather than just proposing another software-based heuristic. This connection is the paper's core intellectual contribution and is genuinely insightful.

2. Addressing a Critical System Bottleneck: The paper tackles one of the most significant challenges in modern AI: the memory and computational explosion associated with large context windows. Their claim of enabling 1M token contexts on a single GPU (as shown in Figure 7, Page 12) is not merely an incremental improvement; it represents an order-of-magnitude increase in accessibility for this class of models. This has the potential to democratize research and application development for use cases that are currently out of reach for most, such as processing entire books, legal archives, or extensive codebases in a single inference pass.

3. A Compelling Use Case for Modern Hardware Trends: LongSight serves as a powerful "killer app" for emerging architectural trends. It demonstrates the tangible benefits of:

   • Compute Express Link (CXL): The low-latency, coherent memory sharing provided by CXL is essential for the fine-grained interaction between the GPU and DReX.
   • Processing-in-Memory (PIM): The use of PIM for the initial sign-bit filtering stage (Section 7.1, Page 8) is a perfect application of the technology—a simple, highly parallelizable operation performed directly where the data resides, avoiding massive data movement.
   • Disaggregated/Tiered Memory: The paper provides a concrete vision for a smarter memory hierarchy, where the CXL-attached tier is not just for passive capacity expansion but is an active computational partner to the main processor.

4. Holistic Algorithm-Hardware Co-Design: The solution is thoughtfully designed across the entire stack. The algorithm (hybrid dense-sparse with ITQ-enhancement) is tailored to the hardware's capabilities (sign-based PIM filtering, near-memory dot-products). The data layout within DReX is carefully orchestrated to maximize parallelism (Figure 6, Page 10). This comprehensive, system-level thinking is commendable.

Weaknesses

While the core idea is powerful, its current presentation is tightly coupled to a specific research artifact, which raises questions about its generalizability.

1. Dependence on the DReX Architecture: The work is presented as an extension of DReX [34]. While this makes for a concrete and well-evaluated system, it leaves the reader wondering about the fundamental principles. The paper would be significantly strengthened by abstracting away from DReX and defining the core requirements for a "LongSight-capable" memory device. What are the necessary PIM capabilities, the required CXL bandwidth, and the near-memory compute primitives that make this approach viable? Without this, the work risks being seen as a bespoke solution rather than a generalizable architectural paradigm.

2. Complexity of Hyperparameter Management: The authors acknowledge that the system's performance and accuracy depend on a set of interdependent hyperparameters: the dense window size (W), the number of sparse tokens (k), and the per-head SCF thresholds. The pareto frontier in Figure 10 (Page 13) shows that tuning is critical for optimal performance. This introduces a layer of complexity for practitioners. A more robust system would ideally feature a method for automatically and dynamically adapting these parameters.

3. Limited Comparison to Alternative Sparsity Patterns: The paper focuses exclusively on top-k similarity as the criterion for sparsity. This is a reasonable and popular choice, but the field has explored other patterns (e.g., block-sparse, strided, global tokens as in Longformer [2]). A discussion of whether the DReX hardware could be programmed or adapted to accelerate other sparsity patterns would help contextualize the flexibility and limitations of their proposed hardware.

Questions to Address In Rebuttal

1. The work is tightly coupled with the DReX architecture [34]. Can the authors elaborate on the core architectural requirements for a compute-enabled memory device to effectively implement the LongSight approach? For instance, what are the minimum PIM capabilities (e.g., is sign-bit XOR sufficient?) and CXL bandwidth needed? This would help readers understand how the concept could transcend this specific hardware implementation.

2. The performance of LongSight appears sensitive to several hyperparameters (window size, k, thresholds), which may depend on the context length and task (Section 9.3, Page 13). Could the authors discuss the potential for automating this tuning process? Is there a risk that the overhead of finding optimal parameters could negate the performance benefits in some practical, dynamic deployment scenarios?

3. The paper elegantly repurposes a dense retrieval accelerator for sparse attention. This concept of offloading key primitives to specialized CXL memory seems very powerful. Are there other computationally expensive primitives in modern transformer models (e.g., Mixture-of-Experts routing, speculative decoding verification) that could be similarly offloaded and accelerated by a DReX-like device? A brief discussion on the broader applicability of this "accelerated memory" paradigm would significantly enhance the paper's impact.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents LongSight, an algorithm-hardware co-design for accelerating large-context LLM inference. The core idea is to repurpose DReX [34], a compute-enabled CXL memory expander originally designed for dense retrieval (e.g., for RAG), to accelerate the sparse attention component of LLM inference. The authors propose a hybrid attention algorithm where a GPU handles a dense sliding window of recent tokens, while the vast history of the KV cache is offloaded to DReX. Within DReX, attention is treated as a top-k vector similarity search, leveraging DReX's in-memory sign-bit filtering and near-memory dot-product accelerators to efficiently find the most relevant Key vectors. The claimed novelty lies in this specific repurposing and the co-design of the algorithm, system integration, and hardware scheduling to enable dynamic, low-overhead sparse attention at a massive scale.

Strengths

The primary strength of this work is its novel synthesis of existing components to solve a well-defined and challenging problem. While individual elements are not entirely new, their combination is.

1. Novel Application of a Specialized Accelerator: The central novel claim—repurposing a dense retrieval accelerator (DReX) for dynamic LLM attention—is a compelling one. Prior art in hardware-accelerated attention (e.g., NeuPIMs [9], AttAcc [29]) has largely focused on accelerating dense attention computations. LongSight's approach of treating attention as a hardware-accelerated retrieval problem is a significant conceptual departure.

2. Addressing the Dynamic Update Challenge: The most significant point of novelty compared to other retrieval-based attention methods (e.g., Squeezed Attention [12], which uses standard ANNS) is how it handles the dynamic nature of the KV cache. Conventional ANNS methods require expensive index rebuilding upon data insertion, making them unsuitable for the per-token updates in autoregressive generation. LongSight’s use of DReX's index-free, sign-concordance filtering (SCF) mechanism provides a novel and practical solution to this critical limitation. This is the key "delta" that makes the contribution significant.

3. Novel System-Level Co-design: The contributions detailed in Section 7, particularly the extensions to the DReX CXL Controller (DCC) and the hierarchical data layout scheme (Figure 6), represent tangible, novel system design. This is not a simple "plug-and-play" use of DReX; it requires new hardware logic and a sophisticated software mapping to handle the granularity of multi-head, multi-layer, and multi-user attention requests.

Weaknesses

The work's novelty is somewhat constrained by its heavy reliance on a foundation of pre-existing concepts and a specific, previously proposed architecture.

1. Incremental Algorithmic Novelty: The hybrid dense-sparse attention algorithm itself is conceptually similar to established patterns. The idea of combining a dense sliding window for recency with a retrieval mechanism for long-term memory is present in various forms in prior work (e.g., Longformer [2], StreamingLLM [41]). The novelty is therefore not in the algorithm's high-level structure, but in its specific hardware-aware implementation.

2. Reliance on Closely-Related Prior Art: The proposed system is fundamentally an application built on top of DReX [34], a system proposed in a recent paper by many of the same authors. While repurposing an architecture is a valid contribution, it frames the work more as an extension or a new use case for DReX rather than a fundamentally new hardware architecture. The paper is transparent about this, but it bounds the scope of the novelty.

3. Synthesis vs. Fundamental Breakthrough: The work is a masterful piece of engineering and synthesis, combining ideas from sparse attention, vector databases, and near-data processing. However, it does not introduce a new, fundamental primitive. Its contribution is the clever and effective integration of existing primitives (CXL, PIM for filtering, near-memory acceleration) into a new system configuration.

Questions to Address In Rebuttal

1. The core idea of treating attention as a top-k retrieval problem is gaining traction, with Squeezed Attention [12] being a notable contemporary work. Your paper states that prior work supports a "fixed long context" (Section 4, page 4). Can you elaborate further on why standard ANNS methods are insufficient for the dynamic KV cache and how DReX's architectural features (specifically, the index-free filtering) are uniquely essential to overcoming this limitation? A more direct comparison would strengthen the novelty claim.

2. Your co-design is deeply tied to the specific architecture of DReX [34], including its two-stage filtering/scoring pipeline and PFU design. To what extent is the LongSight framework generalizable? Could the core principles be applied to other near-data or processing-in-memory architectures that may not feature the exact sign-concordance filtering mechanism? Or is the success of LongSight entirely contingent on the unique properties of DReX?

3. The use of Iterative Quantization (ITQ) [7] is presented as a key enabler for filter efficiency (Section 5.4, page 6). While effective, ITQ is a known technique for improving quantization performance. Is there any novelty in how ITQ is applied or integrated within the LongSight framework, or is its application here standard practice?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose ComPASS, a system aimed at improving the integration of Processing-In-Memory (PIM) devices into general-purpose systems. The solution consists of three main components: 1) a new memory command, PIM-ACT, intended to provide a compatible interface for initiating PIM operations across different architectures; 2) a PIM request generator within the memory controller to offload command generation from the host CPU; and 3) two scheduling policies, Static (ST-BLC) and Adaptive (AT-BLC) Throughput Balancers, to manage concurrent PIM and conventional memory requests. The evaluation, conducted via simulation, claims that the proposed system achieves high PIM performance (up to 10.75× GEMV speedup over non-PIM) while successfully co-existing with memory-intensive CPU workloads.

Strengths

1. The paper addresses a significant and timely problem in computer architecture: the practical integration of PIM hardware. The challenges of protocol incompatibility and resource scheduling are indeed major barriers to adoption.
2. The proposed solution is comprehensive, considering the protocol layer (PIM-ACT), hardware support (request generator), and system-level scheduling (ST-BLC/AT-BLC).
3. The evaluation of the scheduling policies, particularly the demonstration in Figure 8 that AT-BLC can meet PIM Quality-of-Service (QoS) targets where other policies fail, presents a compelling case for the adaptive approach.

Weaknesses

My analysis reveals several significant weaknesses that undermine the paper's core claims of compatibility, practicality, and rigor.

1. The Claim of a "Compatible" Protocol is Overstated and Misleading. The central premise of a unified PIM-ACT command is fundamentally weakened by the necessity of the "Architecture-Aware Optimization" (AAO) mechanism described in Section 4.5 (Page 5). The paper claims PIM-ACT allows "different PIM devices to communicate with the host using the same PIM-ACT interface, ensuring compatibility." However, AAO requires the memory controller to load device-specific timing parameters, bank activation granularities, and command semantics from the SPD. This means the controller is not interacting with a unified interface; it is interacting with a configurable one that requires explicit, a priori knowledge of the specific PIM device it is controlling. This is a crucial distinction. The complexity is not removed; it is merely shifted to SPD tables and the MC's interpretation logic. This approach seems far from the "drop-in" compatibility implied.

2. The Foundation of the Adaptive Scheduler (AT-BLC) Relies on an Unjustified Assumption. The paper's strongest result, the AT-BLC scheduler, is critically dependent on a "target completion time (T)" which "must be initialized before the PIM operation begins" (Section 5.4, Page 8). The paper provides no details on how this target T is determined. Is it provided by a compiler? A runtime profiler? An oracle? The feasibility and robustness of AT-BLC are entirely contingent on the accuracy of this prediction. The authors fail to analyze the sensitivity of their mechanism to mispredictions in T. A scheduler that only works with perfect future knowledge is of limited practical value. This omission represents a major logical flaw in the evaluation of the paper's primary scheduling contribution.

3. Key Overheads are Ignored or Dismissed Without Evidence.

   • Host CPU Offload: The paper claims the PIM request generator alleviates the burden on host processor cores (Section 4.2, Page 4), but this crucial benefit is never quantified. There is no measurement of CPU cycles saved or utilization reduced. Without this data, the contribution of the request generator is unsubstantiated.
   • Memory Management: The authors propose using HugeTLB for contiguous memory allocation (Section 4.6, Page 6). They acknowledge that memory compaction may be required if huge pages are fragmented but dismiss the overhead by claiming it is "amortized" for LLM inference. This is an unsupported assertion. For systems under high memory pressure or with different workload characteristics, compaction can induce significant, non-trivial latency spikes. The lack of any quantitative analysis on this front is a serious methodological weakness.

4. Performance Claims Lack Critical Nuance. The PIM-only evaluation in Section 7.1 (Page 9, Figure 7) shows that for GDDR6-AiM, PIM-ACT results in a performance regression for GEMV, even with AAO enabled (0.59% slower). For LPDDR-AiM, the regression is even larger (2.84% slower). To conclude from this data that the protocol "maintains performance comparable to... device-specific protocols" is an oversimplification. Furthermore, the system performance gain attributed to AAO is shown to be only 1.2-1.5% (Section 7.3, Page 12, Figure 10b). This marginal improvement calls into question whether the added complexity of the AAO mechanism is justified.

Questions to Address In Rebuttal

The authors must address the following points directly:

1. On Compatibility: Please reconcile the claim of a "compatible" and "unified" protocol with the explicit requirement for the memory controller to parse and implement device-specific behaviors via the AAO mechanism. Is "configurable" not a more accurate description? If so, how does this meaningfully reduce integration complexity compared to existing device-specific controller modifications?

2. On the AT-BLC Scheduler: The functionality of AT-BLC hinges entirely on the pre-supplied target time T.

   • a) What specific mechanism is proposed to calculate T in a real system?
   • b) Provide a sensitivity analysis showing how AT-BLC's performance (both PIM QoS and CPU throughput) degrades when T is mispredicted by ±10%, ±25%, and ±50%.

3. On Overheads:

   • a) Provide quantitative data demonstrating the reduction in host CPU utilization as a direct result of the PIM request generator. Compare a system with the generator to one without, where the host CPU issues all micro-requests.
   • b) What is the measured performance impact (e.g., tail latency for CPU requests, delay in PIM task initiation) of a memory compaction event triggered to create a huge page for a PIM workload?

4. On Performance Regressions: Please justify the claim that PIM-ACT offers "comparable" performance to native protocols when your own data (Figure 7) shows clear performance regressions for GEMV on AiM-based architectures. At what threshold does a performance loss become significant enough to disqualify a "compatible" claim?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

The paper presents ComPASS, a comprehensive solution aimed at solving the critical system-level integration challenges of Processing-In-Memory (PIM). The core contribution is a two-pronged approach that addresses both the hardware interface and the performance management aspects of PIM in general-purpose systems. The first prong is a compatible PIM protocol (PIM-ACT) and a memory controller extension (PIM request generator) designed to create a unified, flexible hardware interface that can support diverse PIM architectures (like HBM-PIM and GDDR6-AiM) while adhering to existing DRAM standards. The second prong is a set of scheduling policies, culminating in an adaptive throughput balancer (AT-BLC), to intelligently manage memory bus contention between PIM operations and conventional host CPU memory requests, ensuring that PIM meets its performance targets (QoS) without starving CPU applications.

Essentially, this work is not about inventing a new PIM device but rather about designing the standardized "plumbing" and "traffic control" necessary to make the burgeoning ecosystem of PIM devices a practical and integrated part of modern computer systems.

Strengths

1. Addresses a Critical and Timely Problem: The field has demonstrated the potential of PIM with several real-world devices. The primary bottleneck to widespread adoption is no longer the feasibility of in-memory computation itself, but the challenge of system integration. This paper correctly identifies the core obstacles—protocol incompatibility and resource contention—and proposes a holistic solution. It shifts the conversation from "Can we build PIM?" to "How do we use PIM effectively in a heterogeneous system?" This is exactly the direction the field needs to move in.

2. Pragmatic and Elegant Protocol Design: The PIM-ACT command is a clever and practical solution to the limited command space problem. By leveraging existing RFU commands or unused bits in commands like NOP (Section 4.1, Figure 3, page 4), it avoids the need for a radical departure from JEDEC standards. The inclusion of an optype field is the key to its power, creating a flexible abstraction layer. This allows device manufacturers to innovate on their specific PIM architectures while communicating with the host through a single, standardized command. The concept of Architecture-Aware Optimization (AAO) (Section 4.5, page 5), where device-specific timing or bank-grouping information is loaded from SPD, is an excellent mechanism for supporting this diversity without sacrificing compatibility.

3. Holistic System-Level Perspective: The authors recognize that a protocol alone is insufficient. The tight coupling of the PIM-ACT protocol with the PIM request generator in the memory controller and the AT-BLC scheduler demonstrates a strong system-level understanding. The request generator offloads the host CPU from the tedious task of issuing micro-operations, connecting it to a long line of work on "macro instructions" (e.g., TRiM [48], AESPA [27]). The AT-BLC scheduler directly addresses the inevitable performance interference, transforming PIM from a disruptive accelerator into a well-behaved citizen in the memory subsystem.

4. Strong Connection to Real-World Architectures: The work is well-grounded by demonstrating how ComPASS can be applied to existing commercial PIM architectures like Samsung's HBM-PIM and SK Hynix's GDDR6-AiM (Section 4.7, Table 1, page 7). This case study is crucial as it proves the proposal is not merely a theoretical exercise but a viable path toward unifying disparate industry efforts under a common framework.

Weaknesses

1. Hardware Complexity and Overhead are Underexplored: The paper proposes adding a non-trivial "PIM request generator" and additional scheduling logic to the memory controller. While conceptually sound, the potential cost in terms of die area, power consumption, and design complexity is not quantified. For a solution targeting practical adoption, understanding this overhead is crucial. A simple analysis of the storage requirements (request/data buffers) and decoding logic would strengthen the proposal significantly.

2. The Software-Hardware Interface is Abstract: The AT-BLC scheduler relies on the host providing the target completion time (T) and total number of micro-requests (W) for a PIM operation. The paper briefly mentions this is handled by the OS and PIM libraries (Section 4.4, page 5), but this interface is a critical and complex part of the system. How are these values accurately estimated by the runtime? What is the mechanism for passing them to the memory controller? What happens when these predictions are inaccurate? The robustness of the adaptive scheduler depends heavily on the quality of these inputs, and this link feels underdeveloped.

3. Limited Applicability of the Scheduling Model: The evaluation focuses on large, monolithic GEMV operations typical of LLM inference. In this context, a pre-calculated W and T is plausible. However, the future of PIM may include more diverse workloads, such as graph analytics or database queries, which feature more irregular, data-dependent memory access patterns and potentially many smaller, concurrent PIM tasks. It is unclear how well the AT-BLC model would adapt to scenarios where PIM execution is less predictable and cannot be easily characterized by a single W and T pair.

Questions to Address In Rebuttal

1. Could the authors provide an estimate, even if high-level, of the hardware overhead (e.g., buffer sizes in KB, gate count estimate for logic) introduced by the PIM request generator and the additional scheduling queues in the memory controller?

2. Could you please elaborate on the software stack's responsibility for the AT-BLC? Specifically, how does the runtime or driver determine the W and T parameters for a given PIM task, and what is the proposed hardware mechanism for communicating this information to the memory controller?

3. The proposed adaptive scheduler (AT-BLC) appears well-suited for predictable, throughput-oriented PIM workloads like GEMV. Could you discuss how the ComPASS framework, and particularly the AT-BLC, might be extended or adapted to support more dynamic and latency-sensitive PIM workloads with less predictable execution times?

4. The Architecture-Aware Optimization (AAO) concept is very compelling as a way to future-proof the protocol. Do you envision this information being communicated solely through a static mechanism like SPD, or could there be a more dynamic, runtime registration process for new PIM device capabilities?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors propose ComPASS, a solution aimed at improving the compatibility and efficiency of integrating Processing-In-Memory (PIM) devices into general-purpose systems. The work identifies two primary challenges: the lack of a standardized PIM protocol and the difficulty of scheduling PIM and conventional memory requests concurrently. The proposed solution consists of three core components:

1. PIM-ACT: A new, unified memory command that leverages unused command space (RFU or NOP bits) in existing DRAM standards to trigger multi-bank PIM operations. An "optype" field within the command allows it to be adapted to different PIM architectures.
2. PIM Request Generator: A hardware unit within the memory controller that offloads the host CPU by receiving high-level "macro requests" and decomposing them into a sequence of low-level "micro requests" for PIM execution.
3. Static and Adaptive Schedulers (ST-BLC and AT-BLC): Scheduling policies that manage the interleaving of PIM and non-PIM requests. While the static balancer uses a fixed threshold, the adaptive balancer dynamically adjusts scheduling priorities based on whether the PIM workload is meeting its Quality-of-Service (QoS) targets.

The central claim is that this combination of a compatible protocol and a QoS-aware scheduler provides a novel and effective solution for processor-PIM collaboration.

Strengths

The primary strength of this work lies in the synthesis of its components to address a practical and timely problem. While the individual concepts have precedents, their integration into a cohesive system architecture designed for compatibility is commendable.

The most novel element is the Adaptive Throughput Balancer (AT-BLC). The concept of a feedback loop where PIM execution progress (Wcur/Tcur vs. W/T') directly influences the memory scheduler's behavior (N) appears to be a new contribution in the context of PIM scheduling. This elevates the work beyond a simple static priority scheme.

The architectural decision to place the PIM request generator inside the memory controller (Section 4.2, Page 4), rather than within the PIM device itself, is a well-reasoned choice. It correctly identifies the limitation of prior work (e.g., TRiM [48], Darwin [32]) where an external generator obscures bank state from the MC, thereby preventing efficient request interleaving. This specific architectural delta is a key enabler for the proposed scheduling policies.

Weaknesses

While the paper presents a well-engineered system, its core ideas, when deconstructed, are largely incremental advancements or applications of known principles from other domains. My primary concern is the degree of fundamental novelty.

1. The PIM-ACT Command: The idea of using reserved/unused command bits to introduce new functionality is a standard industry practice, not a novel academic concept. The true contribution here is the proposed protocol that uses the command, not the mechanism of the command itself. Furthermore, both HBM-PIM and GDDR6-AiM already have mechanisms to trigger multi-bank operations. ComPASS's proposal is a unifying abstraction layer, which is a valuable engineering contribution but a limited conceptual leap. The authors themselves demonstrate in Table 1 (Page 7) that PIM-ACT primarily serves as a wrapper for existing PIM device functionality.

2. The PIM Request Generator: The concept of a hardware unit that translates macro-instructions into micro-operations is not new. This is functionally analogous to DMA controllers, command queue (CQ) mechanisms in NVMe, or instruction decoders in other types of accelerators. As noted in the paper's own related work section (Section 8, Page 12), works like TRiM [48] and Darwin [32] have already proposed instruction generators for PIM. The novelty of ComPASS is limited to the placement of this generator within the MC to enable better scheduling, which, while important, is an architectural refinement rather than a new paradigm.

3. The Static Throughput Balancer (ST-BLC): This is a classic threshold-based or round-robin arbitration policy. The paper acknowledges a recent work [15] that is "conceptually similar to our ST-BLC" (Section 8, Page 12). This suggests that the static scheduling approach is not novel. The paper's main contribution in scheduling is therefore entirely dependent on the adaptive nature of AT-BLC.

In summary, the novelty of this paper is not in the invention of new foundational concepts, but in the clever integration and adaptation of existing ones to the specific problem of PIM-CPU co-execution. The contribution is more of an elegant system design than a fundamental breakthrough.

Questions to Address In Rebuttal

The authors should use the rebuttal to more sharply define the novelty of their contributions against the closest prior art.

1. Regarding the Static Throughput Balancer (ST-BLC), the paper concedes it is "conceptually similar" to the virtual channel-based scheduler in [15]. Can the authors elaborate on the novel delta between ST-BLC and this prior work? If the novelty is minimal, the paper's contribution should be more narrowly framed around the adaptive policy (AT-BLC).

2. The novelty of the PIM Request Generator rests on its placement in the MC. While this enables scheduling, prior works like TRiM [48] also used macro instructions. Beyond location, is there any novelty in the macro-request interface itself (Figure 4c, Page 5) or the decoding logic that allows it to be more general-purpose than prior PIM-specific generators?

3. The PIM-ACT protocol's claim to compatibility is evaluated on two similar GEMV-focused accelerators (HBM-PIM and GDDR6-AiM). How would this protocol generalize to a fundamentally different PIM architecture, such as the general-purpose RISC-based cores in UPMEM-PIM [10]? Would the proposed 6-bit optype and the macro-request format be sufficient to express the diverse instruction set of such a device without becoming a bottleneck or requiring an impractically large optype mapping? Please justify the claimed generality of the protocol beyond the domain of neural network accelerators.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose PIM-CCA, an architecture that integrates a Configurable Compute Accelerator (CCA) into a Processing-in-Memory (PIM) processor, modeled after the UPMEM DPU. The stated goal is to alleviate computational bottlenecks that emerge in PIM systems when memory access latency is reduced. They present a compiler-driven approach to identify and offload "hot code" regions—primarily multiplication-based sequences—to a small, specialized CCA. The evaluation, conducted on a cycle-accurate simulator, claims significant performance gains (up to 1.55x) with what is described as minimal hardware overhead (0.036%). The work also analyzes the relationship between tasklet count and performance in this new architecture.

Strengths

1. Problem Motivation: The paper correctly identifies a critical issue in PIM systems: the performance bottleneck can shift from memory access to computation once workloads are moved closer to memory (Section 2.3). This observation is valid and provides a solid foundation for the work.

2. Grounded Simulation: The use of uPIMulator, a simulator based on a commercially available PIM system (UPMEM), provides a realistic baseline for evaluation (Section 2.1). This is preferable to purely abstract models.

3. Compiler-Architecture Co-design: The authors recognize that a hardware accelerator is ineffective without corresponding compiler support. The inclusion of a compiler flow to detect and replace code sections is a necessary component of the proposed solution (Section 3.3.3).

Weaknesses

My primary concerns with this paper relate to the generalizability of its claims, the transparency of its overhead analysis, and the rigor of its evaluation methodology.

1. Over-specialization to a Weak Baseline: The entire premise appears to be built upon the specific and significant weakness of the baseline UPMEM DPU: its extremely inefficient multi-cycle integer multiplication implemented via mul_step instructions (Section 3.2, page 6). The impressive 1.55x speedup seems less a testament to the general utility of a CCA and more a result of patching this one specific flaw. The "hot codes" identified are dominated by operations that are slow on the baseline (Figure 4a). The claims of broad applicability to PIM are unsubstantiated, as other PIM architectures (e.g., HBM-PIM) may not share this particular bottleneck.

2. Misleading Hardware Overhead Metric: The reported 0.036% area overhead is highly suspect (Section 4.6, page 12). This figure is almost certainly calculated against the total area of the entire PIM chip, which is overwhelmingly dominated by the DRAM arrays themselves. This metric obscures the true cost of the modification. The overhead should be reported relative to the area of the DPU's processor logic, which would provide a far more honest assessment of the design's complexity and cost. Without this, the claim of "minimal overhead" is not credible.

3. Convenient Benchmark Exclusion: The exclusion of key benchmarks like BFS and SpMV is a major red flag (Section 4.1, page 10). The justification given is "limitations in the instruction set supported by our baseline simulator." This is insufficient. These benchmarks are characterized by irregular memory access patterns and different computational kernels than the dense linear algebra workloads that dominate the successful results. Their exclusion raises serious doubts about the robustness of the compiler and the applicability of the CCA beyond simple, regular arithmetic patterns. The work is therefore evaluated on a cherry-picked set of benchmarks that are predisposed to benefit from the proposed accelerator.

4. Limited "Configurability" and Compiler Fragility: The "Configurable" Compute Accelerator appears to be little more than a set of three hard-wired custom function units for multiplication, accumulation, and max (Table 1, page 10). This is not what is typically understood as a reconfigurable fabric (like a CGRA). Furthermore, the compiler's hot code detection mechanism appears to be a simple pattern-matching scheme based on predefined templates (Figure 10, Algorithm 1). It is unclear how this would scale to more complex code structures or identify opportunities that do not exactly match the pre-canned patterns. The robustness of this compiler is unproven.

5. Contradictory Statements: In Section 4.1, the paper states that the Needle-Wushiman (NW) benchmark is unavailable in the simulator environment. However, the max operation, explicitly identified for NW, is included as a core CCA function (CCA code 0x2 in Table 1) and is designed into the hardware (Figure 7). Why design and include hardware for a benchmark you cannot run or evaluate? This internal inconsistency undermines confidence in the methodology.

Questions to Address In Rebuttal

The authors must address the following points to make this work acceptable:

1. Area Overhead: Please clarify the 0.036% area overhead claim. What is the total area used as the denominator for this calculation? Please provide the overhead as a percentage of the DPU's non-memory logic area to provide a more transparent comparison.

2. Baseline Dependency: The baseline UPMEM architecture's multi-cycle integer multiplication is a critical performance bottleneck. Could the reported speedups be primarily attributed to fixing this specific weakness, rather than a generalizable benefit of the CCA approach? How would PIM-CCA perform against a baseline PIM processor with a more reasonable, pipelined single-cycle integer multiplier?

3. Benchmark Exclusion: Provide a detailed technical justification for the exclusion of BFS and SpMV. Do these workloads contain computational patterns that the PIM-CCA compiler cannot identify or that the CCA hardware cannot accelerate? The absence of these key benchmarks casts doubt on the general applicability of the proposed solution.

4. Compiler Limitations: The compiler appears to rely on pattern matching for specific arithmetic sequences (Figure 10). What is the coverage of this approach? How does it handle hot code regions with complex control flow or patterns not pre-defined in the "logic palette"?

5. On the "Configurable" Claim: The CCA is configured for only three operation types. This seems more like a set of co-processors than a truly "configurable" accelerator. Please comment on the design's extensibility and the process for adding new, more complex functional units beyond the ones presented.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper, "PIM-CCA: An Efficient PIM Architecture with Optimized Integration of Configurable Functional Units," addresses an important, second-order problem emerging from the success of Processing-in-Memory (PIM) architectures: the creation of new computational bottlenecks within the memory device itself. The authors astutely observe that by alleviating the data movement bottleneck, many memory-intensive workloads become compute-bound on PIM's inherently resource-constrained processors (DPUs).

The core contribution is a holistic, co-designed solution that integrates a lightweight Configurable Compute Accelerator (CCA) into the PIM processor's pipeline. This is not merely a hardware proposal; it is a full system concept supported by a PIM-aware compiler that identifies and offloads hot computational subgraphs, and an analysis of the interplay between the accelerator and PIM's native task-based parallelism. The authors demonstrate through simulation that their PIM-CCA design can achieve up to a 1.55x performance improvement on compute-intensive kernels with a negligible hardware area overhead (0.036%), making it a practical proposal for real-world PIM systems.

Strengths

This is a well-conceived and timely piece of work that makes several strong contributions to the field.

1. Excellent Problem Formulation and Motivation: The paper's primary strength lies in its identification and clear articulation of a critical, next-generation problem for PIM. The analysis in Section 2.3 (page 4), particularly Figure 3, which shows the shift in workload characteristics from memory-bound to compute-bound once inside PIM, is insightful and provides a compelling motivation for the entire paper. This work is not solving a contrived problem; it is looking ahead at the natural evolution of PIM architectures and their limitations.

2. Elegant Synthesis of Architectural Concepts: This work sits at a valuable intersection of three major research areas: Processing-in-Memory, reconfigurable computing, and compiler-architecture co-design. Rather than inventing a new mechanism from scratch, the authors have skillfully applied the mature concept of a compiler-driven Configurable Compute Accelerator (CCA)—originally proposed for embedded systems—to the unique, resource-starved environment of a PIM processor. This cross-pollination of ideas is a powerful research methodology, and the result is an architecture that feels both novel in its application and grounded in established principles.

3. A Holistic, System-Level Perspective: The authors should be commended for not limiting their scope to just a hardware block. The proposal is a complete system. They have considered:

   • The Hardware: A carefully pipelined CCA logic that respects the tight constraints of the baseline DPU (Section 3.3.1, page 7).
   • The ISA: A minimal and compatible instruction set extension (cca and cca_move) that integrates cleanly into the existing pipeline (Section 3.3.2, page 7).
   • The Compiler: A mechanism for automatically identifying and replacing hot code regions, which is essential for programmability and usability (Section 3.3.3, page 8).
   • The Parallelism Model: An insightful analysis of how the CCA changes the optimal number of software tasklets, showing a deep understanding of the system's performance dynamics (Section 3.4, page 9).

4. Pragmatism and Realism: The proposed CCA is designed with the severe constraints of memory fabrication in mind. The extremely low reported area overhead (Section 4.6, page 12) makes this a highly practical and believable proposal. By focusing on accelerating a few key instruction patterns (like mul_step), the authors have found a "sweet spot" that delivers significant performance gains without requiring a radical or costly redesign of the PIM processor.

Weaknesses

While the core idea is strong, the paper could be improved by broadening its contextual analysis and exploring the boundaries of its proposed solution.

1. Limited Scope of Evaluated CCA Operations: The evaluation, while thorough for what it covers, is heavily focused on accelerating multiplication via the mul_step sequence found in the UPMEM architecture (Table 1, page 10). This is certainly the most obvious bottleneck and the correct place to start. However, the paper's broader claim is about a configurable accelerator. The current evaluation makes the CCA feel more like a specialized, fixed-function "multiplication co-processor" rather than a truly flexible unit. The work would be much stronger if it demonstrated how the framework could target other, more diverse computational patterns, even if just through a design study.

2. Insufficient Discussion of Design-Space Alternatives: The paper rightly argues that simply making the main DPU core more complex is infeasible. However, the CCA is not the only possible solution. A key alternative would be to add a small, fixed SIMD/vector datapath to the DPU. This is a very common architectural pattern for boosting compute throughput. A discussion comparing and contrasting the PIM-CCA approach (with its fine-grained, irregular-pattern matching) against a more traditional SIMD approach (with its regular, data-parallel focus) is missing. This would help to better delineate the specific advantages of the CCA paradigm in the PIM context.

3. Potential Over-Fitting of the Compiler and Logic: The PIM-CCA compiler and the CCA logic palette construction (Section 3.3.4, page 9) appear to be tightly coupled to the hot code patterns identified in the benchmark suite. While this is a valid methodology, it leaves open the question of generality. How would the framework adapt if presented with a new class of workloads with entirely different computational bottlenecks (e.g., bit-level manipulation, cryptography, or complex address calculations)? A more robust discussion of the compiler's ability to discover and map new, unseen patterns would strengthen the paper's claims of flexibility.

Questions to Address In Rebuttal

1. Beyond the multiplication and accumulation patterns evaluated, what other types of computational hot spots common in PIM workloads did you identify? Could you provide an example of a non-MAC-style code region and briefly explain how your compiler and CCA design methodology could be applied to accelerate it?

2. Could you elaborate on why a configurable accelerator (CCA) is a more suitable choice for PIM than a more conventional microarchitectural enhancement like a small, 2 or 4-lane SIMD unit? What specific workloads or computational patterns would be well-served by the CCA that a SIMD unit would handle poorly, and vice-versa?

3. Regarding the compiler framework, what happens when it encounters a hot loop that contains a mix of operations, some of which are mappable to the CCA and some of which are not? Is the compiler capable of partial offloading, or does the entire loop have to remain on the scalar DPU core if a perfect pattern match is not found?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes PIM-CCA, an architecture that integrates a compiler-guided Configurable Compute Accelerator (CCA) into a UPMEM-like Processing-in-Memory (PIM) system. The authors identify that as PIM systems alleviate memory-access bottlenecks, workloads can become compute-bound within the PIM units (DPUs) themselves. The proposed solution is to use a lightweight, reconfigurable CCA to offload these computational "hot code" regions, which are identified by a custom compiler that analyzes the application's dataflow graph. The work also includes an analysis of how this acceleration interacts with the PIM system's software-managed threading (tasklet) model. The primary claims are improved performance (up to 1.55x) with negligible hardware overhead (0.036%).

Strengths

1. Novel Synthesis of Concepts: The primary novel contribution of this paper is the application and adaptation of the Configurable Compute Accelerator concept, originally proposed by Clark et al. [6, 7], to the unique constraints and execution model of a modern, commercially-inspired PIM architecture. While PIM accelerators and CCAs exist independently, their synthesis to solve the emergent computational bottleneck within PIM is a novel research direction.

2. PIM-Specific Hardware Adaptation: The authors demonstrate a clear understanding of the target architecture's limitations. The design of the CCA is not a generic application but is tailored to the PIM environment. The insight to target low-level, multi-cycle instruction sequences (e.g., the mul_step operation on the UPMEM architecture, discussed in Section 3.2, page 6) and consolidate them into a single CCA operation is a well-reasoned, PIM-specific optimization that represents a clear delta over prior CCA work.

3. End-to-End System Proposal: The work presents a complete system view, including a compiler toolflow (Section 3.3.3, page 8) for pattern identification and code generation, a hardware architecture for the CCA itself (Section 3.3.1, page 6), and an analysis of its interaction with the system's concurrency model (Section 3.4, page 9). This holistic approach strengthens the contribution.

Weaknesses

1. Limited Delta from Foundational Prior Art: The core conceptual machinery is not new. The idea of a transparent, compiler-driven instruction set extension through a reconfigurable datapath is the central thesis of the original CCA papers [6, 7]. Similarly, the use of compiler analysis on dataflow graphs to identify and offload critical subgraphs is a foundational technique in compilation for custom hardware and reconfigurable computing. The paper's novelty rests almost entirely on the target of this existing methodology (PIM), not on a fundamental innovation in the methodology itself.

2. Questionable "Configurability" in Practice: The central premise of a CCA is its configurability to accelerate a diverse set of computational patterns. However, the evaluation relies on a CCA configured for a very limited set of operations: a 4-step multiply, an accumulation, and a maximum (Table 1, page 10). This configuration appears functionally closer to a collection of dedicated Custom Functional Units (CFUs) for multiply-accumulate and max operations rather than a truly "configurable" accelerator. The work does not sufficiently demonstrate that the overhead of the CCA's reconfigurable fabric is justified over simply implementing a few fixed-function units, which would be a far less novel contribution. The power of the general CCA framework seems underutilized and, consequently, its novelty is undermined.

3. Incremental Novelty of the Compiler Heuristic: The proposed "logic palette" construction algorithm (Algorithm 2, page 9) appears to be a straightforward greedy heuristic for mapping pattern graphs to hardware resources to maximize sharing. While necessary for their toolflow, the novelty of this specific algorithm, when compared to decades of prior art in High-Level Synthesis (HLS), logic synthesis, and technology mapping for FPGAs/CGRAs, is not clearly established. It solves a necessary problem for the authors but does not appear to be a standalone novel contribution in compiler or synthesis technology.

Questions to Address In Rebuttal

1. The key distinction of a CCA over a set of fixed-function units is its configurability. Given the evaluation uses only three core patterns (multiply, accumulate, max), please clarify the novelty and benefit of the generalizable CCA framework over simply proposing the addition of three dedicated CFUs to the DPU pipeline. What is the hardware overhead (area, wiring complexity) of the CCA's reconfigurability (e.g., the MUX network in Figure 7) compared to hard-wiring these three specific functions?

2. The paper positions itself in the PIM domain. However, other works have proposed co-locating more general-purpose reconfigurable logic (e.g., FPGAs) in the logic layer of 3D-stacked memory or on a DIMM. Please position PIM-CCA more clearly against these alternative forms of reconfigurable near-data processing. What is the fundamental novelty of the CCA model in this context compared to a small, near-memory FPGA or CGRA?

3. To substantiate the claim of a novel and generalizable framework, could the authors provide an example of a more complex or irregular computational pattern from a different application domain (e.g., bioinformatics, cryptography) that their PIM-CCA compiler (Figure 10, page 8) can successfully identify and for which the logic palette framework can generate an efficient CCA configuration? This would more convincingly demonstrate that the contribution is a new framework and not just a bespoke solution for accelerating GEMV-like kernels.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present 3D-PATH, a Processing-in-Memory (PIM) accelerator that utilizes 3D hybrid bonding to couple a DRAM die with a logic die. The core idea is a hierarchical Look-Up Table (LUT) system where a large DRAM-LUT stores precomputed results and a smaller, logic-based "fast-LUT" (implemented as a CAM) is used to accelerate searches and handle sparsity. The paper claims three main contributions: 1) A hierarchical, sparse-aware LUT architecture; 2) An efficient, multiplier-free method for floating-point (FP) computation; and 3) A "thermal-aware" hardware design to mitigate the thermal challenges of 3D stacking. While the paper addresses a relevant problem space, the methodology contains significant flaws, the evaluation relies on a weak baseline, and several core technical descriptions are either unclear or seemingly contradictory.

Strengths

1. Circuit-Level Rigor: The design and analysis of the custom circuit components, specifically the 9T CAM cell for the fast-LUT and the self-throttling sense amplifier, are thorough. The use of post-layout simulations with Monte Carlo analysis (Section 7.2.1, page 10, Fig. 12) demonstrates robustness at the cell level.
2. Detailed Thermal Modeling: The paper employs a professional tool (ANSYS Fluent) for thermal modeling (Section 6, page 7), providing a detailed analysis of heat dissipation under various cooling scenarios (Section 7.1, page 8). This is a commendable level of detail for an architecture paper.
3. Problem Identification: The authors correctly identify key challenges in existing LUT-PIM architectures, namely limited precision, storage redundancy, and inefficient off-table access (Section 1, page 1).

Weaknesses

1. Fundamentally Unclear Floating-Point Implementation: The description of the floating-point multiplication mechanism (Section 4.4.2, page 5) is critically flawed. The authors claim a "lossless transformation" where the multiplication of mantissas (MIN × Mw) is handled by a LUT. However, the paper states, "The LUT contains precomputed products of the input mantissa and the full-precision weight." This is a logical contradiction. The input mantissa (MIN) is a dynamic value determined at runtime; one cannot precompute a LUT for all possible combinations of dynamic inputs and stored weights without an astronomically large table. The paper fails to specify if mantissa slicing or another approximation is used, which would invalidate the "lossless" claim. Without a clear and viable explanation of this core mechanism, all reported FP performance results are unsubstantiated.

2. Use of an Unjustified "Analytical Baseline": The primary baseline for comparison, "3D-base," is described as an "analytical baseline model" (Section 'Baseline', page 8). Its performance characteristics are derived from "prior 3D design studies," not from a concrete implementation or even a cycle-accurate simulation. This is unacceptable for a rigorous comparison. The reported performance gains over this baseline (e.g., 1.68× on AI workloads, Fig. 16) are rendered meaningless, as the baseline can be arbitrarily defined to inflate the benefits of the proposed work. This invalidates a major portion of the claimed performance improvements.

3. Overstated "Thermal-Aware" Contribution: The "thermal-aware hardware" (Section 5, page 5) consists primarily of two circuit-level power reduction techniques: a sign-magnitude adder to reduce bit toggling and a self-throttling sense amplifier that power-gates on a mismatch. While these are valid optimizations for power, they do not constitute a "thermal management" system. A true thermal-aware system typically involves temperature sensors and a dynamic policy (e.g., DVFS) to manage heat globally. The authors' design is reactive and localized; framing it as a comprehensive thermal solution is a significant overstatement. The solution is power-saving, not thermal-aware in the conventional sense.

4. Crucial Architectural Details are Missing: The paper omits several details essential for evaluating the architecture's viability:

   • Fast-LUT Miss Policy: The entire performance model hinges on the fast-LUT. The authors provide no information on how a miss in the fast-LUT is handled. What is the performance penalty? Does it require a full scan or access to a different data structure in DRAM? This is a critical oversight.
   • Fast-LUT Hit Rate: Despite the fast-LUT being central to the sparse-aware computation, the authors provide no data on its hit rates for the evaluated benchmarks. Without this data, it is impossible to assess its effectiveness.
   • LUT Updates: The paper focuses exclusively on LUT reads. How are the DRAM and fast-LUTs populated and updated? DRAM writes are slow and power-intensive, which could be a major system bottleneck not accounted for in the evaluation.

5. Unfair GPU Baseline Comparison: For sparse workloads like ResNet-50 and LSS, the authors compare their sparse-aware accelerator against a GPU (Fig. 16). It is not stated whether the GPU baseline utilizes sparsity-aware optimizations (e.g., NVIDIA's cuSPARSE library, structured sparsity). If the comparison is against a dense GEMM implementation on the GPU, the reported speedups are misleading and simply reflect the benefits of sparsity itself, not necessarily the superiority of the proposed architecture.

Questions to Address In Rebuttal

1. Regarding the FP LUT (Section 4.4.2): Please provide a precise explanation of how the LUT for mantissa multiplication works.

   • How can it be precomputed if the input mantissa is dynamic?
   • If mantissa slicing is used, what is the bit-width of the slice, what is the resulting table size, and what is the impact on precision? Justify the "lossless" claim.

2. Regarding the "3D-base" Baseline (page 8): Please provide a detailed specification of the analytical model.

   • What specific assumptions were made regarding its compute units, on-chip network, memory access latency, and power consumption?
   • Justify why a simulated RTL or cycle-accurate model of a conventional 3D accelerator was not used as a more rigorous baseline.

3. Regarding the Fast-LUT (Section 4.2):

   • Describe the full pipeline for a fast-LUT miss and quantify the associated performance penalty in cycles.
   • Provide the measured hit rates for the fast-LUT across all evaluated AI benchmarks (ResNet-50, BERT, LSS, etc.).

4. Regarding Thermal Management (Section 5):

   • Justify the use of the term "thermal-aware hardware." Does the design include any form of temperature sensing or global thermal feedback loop? If not, please re-frame the contribution as a power-efficiency optimization.

5. Regarding GPU Comparisons (Fig. 16):

   • Clarify whether the GPU baseline for sparse models was configured to use hardware and software support for sparse matrix computation. If not, the comparison must be re-evaluated against a proper sparse-aware baseline.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents 3D-PATH, a novel processing-in-memory (PIM) architecture that synergistically combines three key technology trends: LUT-based computation, 3D hybrid bonding integration, and thermal-aware circuit co-design. The core contribution is the insight that the ultra-high bandwidth and fine-grained connectivity of hybrid bonding can be directly translated into high-throughput LUT operations. To manage the complexity and redundancy inherent in large LUTs, the authors propose a hierarchical structure: a small, fast, content-addressable LUT on a logic die acts as a filter and index for a large, high-capacity LUT implemented in a stacked DRAM die.

The work further extends this architectural foundation to address practical challenges. It introduces a multiplier-free method for floating-point operations and leverages the architecture's structure to handle sparse data efficiently. Recognizing that 3D stacking exacerbates thermal issues, the paper presents a holistic thermal co-design, including a self-throttling sense amplifier and a low-toggling adder, to mitigate hotspots without compromising performance. The result is a well-integrated, systems-level proposal that connects advances in semiconductor fabrication directly to architectural innovation for PIM.

Strengths

1. Timely and Visionary Synthesis: The paper's greatest strength is its successful synthesis of disparate but highly relevant research fields. It sits at the intersection of PIM architecture, advanced packaging, and low-power circuit design. Rather than treating hybrid bonding as a mere "faster wire," the authors use its unique properties to enable a new architectural paradigm (hierarchical LUT-PIM). This provides a compelling blueprint for what future heterogeneous systems could look like and is a significant contribution to the community's thinking about post-Moore's Law computing.

2. Holistic, Full-Stack Approach: The authors are to be commended for their end-to-end system perspective. They do not simply propose an abstract architecture; they ground it in a specific integration technology (hybrid bonding), identify the critical second-order problem that technology creates (thermal density, as discussed in Section 5.1, page 5), and propose concrete circuit-level solutions. This full-stack awareness, from device physics to architecture, is rare and makes the work far more credible and impactful.

3. Elegant Solutions to Known PIM Weaknesses: The paper directly tackles several well-known limitations of prior PIM and LUT-based accelerators:

   • Sparsity: The hierarchical fast-LUT/DRAM-LUT design is a very clever mechanism for handling sparsity. By using the CAM-based fast-LUT to identify non-zero elements, the system avoids redundant lookups in the much larger and slower DRAM-LUT, connecting directly to a major trend in AI/ML workloads.
   • Floating-Point Precision: The lack of efficient floating-point support has been a major barrier for PIM adoption in scientific computing and modern AI. The proposed transformation method (Section 4.4.2, page 5), which offloads the computationally intensive mantissa multiplication to the LUT while handling the exponent and sign in simple logic, is an elegant and practical solution.
   • The Memory Wall: The core premise attacks the memory wall by leveraging the massive parallelism (4096 parallel banks) afforded by hybrid bonding, turning a potential bandwidth firehose into productive computation.

Weaknesses

While the hardware proposal is compelling, the paper is viewed primarily through an architectural and circuits lens, leaving some crucial system-level questions open.

1. Programmability and the Software Stack: The paper is largely silent on how a developer or compiler would target the 3D-PATH architecture. Is the hierarchical LUT managed transparently by a hardware controller, or does it require explicit software management? Decomposing functions into LUTs, partitioning them between the fast and slow tiers, and managing updates are non-trivial software problems. Without a clear programmability model, 3D-PATH risks being a powerful but unusable piece of hardware.

2. Overhead of LUT Generation and Updates: The analysis focuses almost exclusively on the inference or lookup phase. However, a key advantage of LUTs is their reconfigurability. The paper does not quantify the latency or energy costs associated with populating or updating the LUTs in DRAM. For applications where the function changes dynamically (e.g., during training phases of machine learning, or with adaptive algorithms), the cost of writing these large tables could become a significant performance bottleneck, potentially negating the benefits of fast lookups. The discussion in Section 7.3.1 (page 10) mentions the update process but provides no performance data.

3. Scalability of the Fast-LUT: The fast-LUT is based on Content Addressable Memory (CAM), which is effective but known to be power-hungry and area-inefficient compared to standard SRAM. The paper evaluates a 32Kb configuration. It is unclear how the architecture's efficiency would scale to problems requiring a much larger set of "hot" indices that do not fit in the fast-LUT. This could create a performance cliff where frequent misses in the fast-LUT lead to iterative, high-latency searches in the DRAM die, undermining the core performance claim.

Questions to Address In Rebuttal

1. Could the authors elaborate on the intended programmability model for 3D-PATH? What is the division of responsibility between the hardware controller and the software/compiler for managing the two-level LUT hierarchy?

2. The proposed method for FP multiplication is clever. However, accumulation is also a critical part of many workloads (e.g., GEMM). The paper states that the outer-product approach avoids the need for pre-alignment across different banks (Section 4.4.2, page 5). How and where are the final partial products accumulated, and how does the system handle the necessary exponent alignment and normalization during this final reduction step? What is the associated hardware cost and latency?

3. Could the authors provide an analysis or estimation of the performance and energy overhead for updating a DRAM-LUT bank? How does this "write" or "reconfiguration" cost compare to the "read" or "lookup" cost, and how does it affect the suitability of 3D-PATH for workloads with dynamic or frequently changing functions?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents 3D-PATH, a processing-in-memory (PIM) architecture that leverages 3D hybrid bonding to implement a hierarchical look-up table (LUT) system. The core claims of novelty appear to be four-fold:

1. The primary architectural concept: a synthesis of 3D hybrid bonding with LUT-based PIM, creating a hierarchical system where a large, parallel DRAM-LUT is assisted by a small, fast SRAM-based LUT (fast-LUT) on the logic die.
2. The use of this hierarchical LUT structure to perform sparsity-aware computation, where the fast-LUT acts as an index and filter to avoid redundant accesses to the main DRAM-LUT.
3. A novel method for multiplier-free and, critically, alignment-free floating-point (FP) computation by combining representation transformation with a column-interleaved, outer-product data mapping.
4. The introduction of specific thermal-aware hardware, namely a self-throttling sense amplifier (SA) and a sign-magnitude dual-adder tree, to mitigate the thermal challenges inherent to the 3D-stacked architecture.

While the overall synthesis of these concepts into a cohesive system demonstrates novelty, several of the underlying techniques are adaptations of established principles. The most significant novel contributions lie in the architectural pattern enabled by 3D integration and the specific method for alignment-free FP computation.

Strengths

The primary strength of this work is the novel architectural synthesis. The central idea to "transform the high bandwidth of 3D integration into high LUT throughput" (Section 1, page 2) is a genuinely new and insightful architectural paradigm for PIM. While LUT-PIM (e.g., pLUTo [17]) and 3D integration for memory have been explored separately, their co-design into a hierarchical system where the logic die's fast-LUT serves as a directory/filter for the DRAM die's data-LUT is a compelling and novel approach. This is not merely an integration but a fundamentally new way to structure a PIM accelerator.

The second major novel contribution is the method for floating-point computation (Section 4.4.2, page 5). Prior PIM works that tackle FP arithmetic often struggle with the overhead of exponent alignment (e.g., FloatAP [71] uses bit-serial shifting). The proposed "alignment-free" method, achieved by mapping computation in an outer-product fashion where each bank handles an independent column, is a clever circumvention of this bottleneck. This represents a significant delta over the prior art in PIM systems and substantially broadens the applicability of LUT-based PIM.

Finally, the authors demonstrate a holistic design approach by identifying the thermal consequences of their architecture and proposing hardware solutions. This end-to-end consideration, from architecture to circuit, is commendable.

Weaknesses

My main concerns relate to the degree of novelty in some of the constituent components, particularly the thermal-aware hardware.

1. Self-Throttling Sense Amplifier: The concept of detecting a mismatch early in a CAM/TCAM search cycle and gating the discharge path to save power is a well-established technique in the circuit design literature. For example, selective-precharge schemes mentioned by the authors ([50], [60]) aim for a similar outcome. While the specific 7T circuit implementation in Figure 6 may be unique, the fundamental principle of "early gating to terminate current flow during mismatches" (Section 5.3.2, page 7) is not a fundamentally new invention. The novelty is more in its application to a thermal problem rather than a purely power-saving one, which is an incremental step.

2. Sign-Magnitude Low Toggling Adder: The use of sign-magnitude (SM) representation over two's complement (2C) to reduce switching activity in adders is a known low-power design strategy. The authors themselves cite prior work [3] that quantifies the energy efficiency benefits. The dual-adder tree is a standard approach to managing the complexity of SM addition/subtraction. Therefore, the novelty here is not in the arithmetic circuit itself, but in its integration into this specific architecture for the stated purpose of thermal mitigation.

The complexity of the overall 3D-PATH system is substantial, requiring advanced 3D integration. While the results for sparse workloads are impressive, the benefit of the hierarchical LUT for dense operations is less clear. For a dense matrix, the fast-LUT would presumably have a 100% hit rate, acting primarily as an address translator and introducing latency and power overhead without providing any filtering benefit. The novelty is therefore highly optimized for a specific data pattern (sparsity).

Questions to Address In Rebuttal

1. Regarding the thermal-aware hardware (Section 5.3, pages 6-7): Please clarify the novelty of the self-throttling SA and the SM dual-adder tree in the context of prior circuit-level art. Beyond applying known low-power techniques to a thermal problem, what is fundamentally new about the circuit topology or operation compared to existing mismatch-gated CAM SAs or low-power SM adders?

2. The hierarchical LUT is presented as a key innovation for handling sparsity. For a fully dense workload where no computation can be skipped, what is the performance and power overhead of the fast-LUT search step? Does the fast-LUT become a bottleneck or an inefficient power consumer in such a scenario, and how does the architecture's efficiency compare to a non-hierarchical "flat" LUT-PIM design in that specific case?

3. The alignment-free FP method is compelling. However, the transformation in Equation 2 (page 5) offloads only the mantissa multiplication ||MIN × MW||LUT to the table. The OLU must still handle exponent addition and final normalization. Could you elaborate on the overhead and potential precision-handling complexities (e.g., for subnormal numbers, rounding modes) that must be managed in the OLU post-lookup? How does this complexity trade-off against the benefit of avoiding pre-alignment?

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present a detailed microarchitectural analysis of the stack engine, a frontend optimization in modern x86 CPUs from Intel and AMD. They reverse engineer its behavior, focusing on the conditions that trigger a synchronization uop, namely unsupported operations and stack depth overflows. Based on these findings, they construct three leakage primitives and demonstrate their use in same-thread and cross-thread covert channels, as well as side-channel attacks against the cJSON and protobuf libraries. Finally, they identify and analyze undocumented MSRs ("chicken bits") on recent AMD CPUs that can disable the stack engine, measuring the performance impact of this mitigation. While the level of detail in the reverse engineering is commendable, the work suffers from questionable measurement reliability, a potentially contrived threat model, and claims that may not be fully substantiated by the provided evidence.

Strengths

1. Comprehensive Microarchitectural Analysis: The paper provides an impressively detailed investigation of the stack engine's behavior across a wide range of modern Intel and AMD microarchitectures, from Sandy Bridge to Zen 5. The systematic approach to characterizing properties like ARSP depth (Section 5.4) and support for add/sub (Section 5.5) is thorough.

2. Mitigation Analysis: The discovery and experimental validation of the undocumented MSRs on AMD Zen 4 and Zen 5 to control the stack engine (Section 9.1) is a significant finding. Measuring the real-world performance impact using SPEC CPU2017 provides a valuable data point on the trade-off between performance and security for this specific optimization.

Weaknesses

1. Unreliable Measurements Undermine Core Observations: The foundation of this paper—the reverse engineering in Section 5—appears to be built on a shaky measurement methodology. The authors frequently acknowledge measurement issues, such as "performance counter inaccuracies" and "high jitter" on Intel CPUs (caption of Figure 4, page 7), and "excessive noise" preventing a conclusive analysis of speculative execution (Section 5.7, page 8). Most concerningly, when key observations contradict their model (e.g., the lack of a sync uop on Golden Cove and Zen 1), they dismiss it as a "performance counter bug" (Section 5.2, page 6) without providing concrete evidence to support this claim. This is a critical weakness. An alternative and equally plausible explanation is that the hardware behaves differently, which would invalidate the generality of their findings and the primitives built upon them. The burden of proof is on the authors to demonstrate that these are measurement errors, not fundamental behavioral differences.

2. Questionable Novelty and Robustness of Attack Techniques:

   • The "new port contention technique" for cross-thread leakage (Section 7.2, page 10) is poorly motivated and described. The authors claim prior methods are insufficient for single-cycle uops but describe their solution as creating a dependency chain to delay execution. This is a standard approach to amplify contention signals, not a novel technique. The lack of a clear, formal description or a rigorous comparison against prior work makes the claim of novelty unsubstantiated.
   • The attack primitives themselves show a lack of generality. The Sync+Reload primitive is rendered ineffective on Zen 5, the latest AMD architecture, because sync uops are dispatched unconditionally (Section 5.2). This forces the authors to develop a more complex Prime+Sync+Probe primitive. This suggests a fragmented and architecture-specific set of leakage methods rather than a universal principle.

3. Contrived Threat Model and Fragile Attacks: The practicality of the demonstrated side-channel attacks is debatable.

   • The cJSON attack (Section 8.1) requires an attacker within the same address space to repeatedly invoke a parsing function on the same secret data but at different start offsets. This seems like a highly specific and unlikely scenario in a real-world setting like the FaaS environment described. A strong justification for the realism of this attack vector is missing.
   • The protobuf attack (Section 8.2) is admitted to only work when default compiler optimizations (which use vector instructions that reset the stack engine) are not used. It relies on a specific *__get_packed_size function. This makes the attack fragile and opportunistic, rather than a general threat to applications using protobuf. The paper fails to argue for the prevalence of such vulnerable code patterns in real-world software.

4. Overstated Claims: The abstract claims this work is the "first reverse engineering of the stack engine." This overstates the contribution. The existence and basic function of the stack engine have been known and described in public documents, including Agner Fog's optimization manuals [18], for years. While this paper provides a much deeper security-focused analysis, it is not the first reverse engineering. This should be rephrased for accuracy.

Questions to Address In Rebuttal

1. Regarding the claim of a "performance counter bug" on Golden Cove and Zen 1 (Section 5.2), what evidence can you provide to prove that a sync uop is indeed dispatched but simply not counted, as opposed to not being dispatched at all? Without this proof, how can you be confident in the universality of your Sync+Reload primitive?

2. Please formalize your "new dispatch alignment technique" (Section 7.2). How does it fundamentally differ from established techniques that use dependency chains to amplify contention for measurement? Please provide a more rigorous comparison to prior work like that of Aldaya et al. [3].

3. Could you elaborate on a more concrete and plausible end-to-end attack scenario for the cJSON side channel (Section 8.1)? Specifically, how would an attacker in a realistic in-process sandboxing environment gain the ability to repeatedly trigger the victim's parsing function on the same data with byte-level control over the starting offset?

4. Given that one of your main primitives (Sync+Reload) does not work on the latest AMD CPUs and your demonstrated attacks rely on non-default compiler flags (protobuf) or a highly specific invocation pattern (cJSON), do you believe your work demonstrates a widespread, practical threat, or rather a narrow, opportunistic one? Please justify your assessment.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a comprehensive, end-to-end investigation of the CPU stack engine—a long-standing but under-examined microarchitectural optimization—as a novel source of information leakage. The authors conduct a meticulous reverse engineering of the stack engine's behavior across a wide range of modern AMD and Intel processors, identifying the precise conditions that trigger synchronization events. Building on this foundational understanding, they construct a set of powerful leakage primitives. These primitives are then leveraged to build practical covert and side channels, culminating in a high-fidelity attack that exfiltrates structural information from a widely-used JSON parsing library, thereby leaking sensitive data. Crucially, the work does not stop at exploitation; the authors discover and verify undocumented "chicken bits" in recent AMD CPUs that can disable the stack engine, and they provide a sober analysis of the ~4% performance cost this mitigation incurs. This is a foundational piece of work that systematically transforms a seemingly benign performance optimization into a fully understood security liability.

Strengths

This paper has several significant strengths that place it firmly in the upper echelon of microarchitectural security research.

1. Novelty and Significance of the Target: While the community has spent years dissecting caches, branch predictors, and speculative execution, the stack engine has remained largely unexplored from a security perspective. This work is, to my knowledge, the first to subject it to a rigorous security analysis. By doing so, it opens up a new avenue of inquiry into how fundamental frontend optimizations can create security vulnerabilities, broadening the attack surface beyond the more commonly studied CPU components.

2. Methodological Rigor and Breadth: The authors’ approach is exceptionally thorough. The reverse engineering effort detailed in Section 5 (pages 5-8) is impressive, spanning multiple generations and architectures from both Intel and AMD. This provides a valuable comparative view and demonstrates that the vulnerability is not an isolated design flaw but an emergent property of a widely adopted optimization. The progression from reverse engineering to primitive-building to a full-fledged attack is logical and compelling.

3. A Complete "Vulnerability Lifecycle" Analysis: This is perhaps the paper’s greatest strength. It does not simply present an attack; it presents the entire story. It begins with curiosity about a microarchitectural feature, moves to deep understanding, demonstrates a practical exploit, and, most importantly, provides a concrete, hardware-verified mitigation. The discovery and functional analysis of the undocumented MSR bits in Section 9.1 (page 12) is a standout contribution, turning the paper from an offensive security work into a balanced and constructive piece of systems research. The performance evaluation of the mitigation provides the final, critical data point needed for CPU architects to make informed trade-off decisions.

4. Bridging Low-Level Primitives to High-Level Impact: The attack on the cJSON library (Section 8.1, page 11) is an excellent case study. It skillfully connects an esoteric microarchitectural effect (a sync uop being dispatched due to ARSP overflow) to a tangible security outcome (distinguishing between patient records based on the structure of parsed data). This demonstration is crucial for showing that the identified leakage channel is not merely theoretical but poses a genuine risk to real-world software.

Weaknesses

The weaknesses of this paper are minor and relate more to missed opportunities for contextualization than to flaws in the work itself.

1. Limited Contextualization within the Landscape of Frontend Attacks: The paper correctly identifies itself as a frontend attack in Section 10 (page 13). However, it could do more to position the stack engine channel relative to other known frontend channels (e.g., from the uop cache, Loop Stream Detector, or branch predictors). A brief discussion on the comparative properties—for instance, is the stack engine channel stealthier due to its transient nature? Is it lower or higher bandwidth? Is it more or less noisy than, say, a uop cache-based channel?—would help readers better situate this new vector within the broader taxonomy of microarchitectural threats.

2. Could Further Generalize the Underlying Principle: The paper correctly notes in Section 5.9 (page 8) that architectures like ARM and RISC-V do not require a stack engine due to their different ISA idioms. This is an important distinction. However, the underlying principle is that a performance optimization designed to handle a common software idiom (in this case, x86's push/pop stack management) creates state that can be leaked. The paper could strengthen its intellectual contribution by briefly speculating on whether analogous "idiom-specific" frontend optimizations in other ISAs might create similar, currently undiscovered, leakage vectors. This would elevate the core idea from an x86-specific finding to a more general principle of microarchitectural security.

Questions to Address In Rebuttal

1. The discovery of the MSR "chicken bits" is a fantastic contribution. Can the authors elaborate, even briefly, on the methodology used to find them? Was this the result of a brute-force MSR scan, or were there clues in documentation, patents, or kernel patches that pointed them in the right direction? Understanding this process could be valuable for other researchers.

2. How does the stack engine channel compare to other non-speculative, frontend channels in terms of its key properties? Specifically, considering a sophisticated defender, would this channel be considered more or less difficult to detect than a channel based on, for example, uop cache contention?

3. The paper makes a compelling case for the vulnerability of the stack engine in x86. Thinking more broadly, do the authors believe that the general principle—that stateful frontend optimizations for common software idioms are a ripe target for side channels—is applicable to other architectures? Could they provide a hypothetical example of what such an optimization and vulnerability might look like on an architecture like ARM or RISC-V?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents a systematic reverse engineering and security analysis of the "stack engine," a microarchitectural optimization in modern x86 CPUs that tracks the stack pointer (RSP) in the frontend to improve instruction-level parallelism. The authors characterize the behavior of this engine across a wide range of recent Intel and AMD microarchitectures, identifying its internal state (ARSP), its tracking depth, and the conditions that trigger a synchronization with the architectural RSP in the backend.

Based on these novel insights, the authors construct three new attack primitives (Direct Underflow, Sync+Reload, Prime+Sync+Probe) that exploit the stack engine's state to leak information. They demonstrate these primitives by building covert channels and a side-channel attack against the cJSON parsing library. Finally, they discover and analyze undocumented MSR "chicken bits" in recent AMD CPUs that can disable the stack engine, providing a mitigation and a way to quantify its performance impact.

The core novelty of this work lies in being the first to deeply investigate, characterize, and weaponize the stack engine as a source for information leakage. While the attack patterns are analogous to prior work in other domains (e.g., caches), the target, the channel, and the specific mechanisms are entirely new.

Strengths

1. Novelty of the Target Microarchitectural Unit: The primary strength of this paper is its focus on a previously unexamined microarchitectural component for security analysis. While the existence of stack pointer tracking is known in principle (citing Bekerman et al. [8] from 2000), this paper provides the first-ever detailed, empirical reverse engineering of its modern implementations (Section 5, pages 5-8). The characterization of properties like tracking depth (Figure 4, page 7), conditions for synchronization (Section 5.2, page 5), and support for add/sub (Section 5.5, page 7) across multiple generations of AMD and Intel CPUs is a significant and novel contribution to the community's understanding of the x86 frontend.

2. Novel Primitives Derived from First Principles: The attack primitives are not generic; they are meticulously derived from the specific behaviors uncovered during the reverse engineering phase. For example, Prime+Sync+Probe directly exploits the finite capacity of the internal ARSP register and the observable synchronization event on overflow. This tight coupling between reverse engineering and exploit development is a hallmark of high-quality microarchitectural security research. This represents the first time such stateful tracking in the stack engine has been exploited.

3. Novel Observation Technique for a Difficult Signal: The authors correctly identify that the signal from the stack engine—a single-cycle, independent ALU operation—is difficult to observe, especially cross-thread, and that prior port contention techniques [3, 23] are insufficient. Their development of a "tuned" port contention method with a dependency to align execution windows (Section 7.2, pages 10-11) is a subtle but important novel contribution in its own right, enabling the observation of this new class of faint signals.

4. Novel Discovery of Undocumented Mitigations: The discovery and characterization of undocumented "chicken bits" in AMD Zen 4 and Zen 5 CPUs to control the stack engine (Section 9.1, page 12) is a concrete and novel finding. This is not simply an application of existing knowledge but a genuine discovery that provides an immediate mitigation path and allows for a precise performance evaluation of the targeted feature.

Weaknesses

1. Conceptual Analogy to Existing Attack Patterns: While the target and mechanism are novel, the conceptual framework of the Prime+Sync+Probe primitive is a direct analogue to the classic Prime+Probe cache attack. The pattern is: (1) put the microarchitectural structure into a known state (prime), (2) let the victim execute, (3) check the state to see if the victim's activity changed it (probe). The paper should more explicitly position its contribution not as the invention of a new attack pattern, but as the novel discovery that the stack engine constitutes a previously unknown structure susceptible to this pattern.

2. The Demonstration is an Application, Not a Core Novelty: The successful attack on the cJSON library (Section 8.1, page 11) is an excellent demonstration of the primitives' effectiveness. However, from a novelty standpoint, this is an application of the core ideas rather than a new idea in itself. The core contribution remains the identification and exploitation of the stack engine, not the specific finding in a downstream software library.

Questions to Address In Rebuttal

1. The Prime+Sync+Probe primitive is functionally analogous to cache-based Prime+Probe. Can the authors elaborate on the non-trivial aspects of adapting this conceptual pattern to the stack engine's ARSP register? For instance, how does the non-cache-like, single-value nature of the ARSP state fundamentally differ from the set-based state of a cache during an attack?

2. In Section 5.7 ("Stack engine under speculation"), you confirm that sync operations are observable under transient execution but conclude that an attack "would only slightly expand the capabilities of a cross-thread attacker." This conclusion seems understated. Could a transient execution attack based on the stack engine enable leakage scenarios not possible with existing speculative attack primitives? Please clarify if there is a genuinely novel transient attack vector here that has been downplayed.

3. Your reverse engineering covers a wide range of x86 CPUs. You briefly state that ARM and RISC-V lack a stack engine due to their ISA design (Section 5.9, page 8). Are you aware of any analogous frontend optimizations in other non-x86 architectures that track register state (not just the stack pointer) in a similar stateful, finite-capacity manner that could be susceptible to the new class of primitives you have developed?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents an investigation into impedance-based side-channel leakage in emerging non-volatile memories (NVMs), specifically ReRAM, FRAM, and MRAM. The authors utilize a Vector Network Analyzer (VNA) to measure the S11 reflection parameter on the power distribution network (PDN) of commercial memory chips. They claim that variations in the measured impedance correlate with the stored data's Hamming weight (inter-HW) and specific data patterns (intra-HW). Using a feature selection process followed by Principal Component Analysis (PCA) and standard machine learning classifiers, the authors report high classification accuracy, concluding that impedance-based leakage is an "exploitable phenomenon" in these memory technologies.

Strengths

1. Empirical Breadth: The study evaluates three distinct and commercially relevant NVM technologies (ReRAM, FRAM, MRAM) from multiple vendors. This provides a broader base for its claims than a study focused on a single device.
2. Use of COTS Devices: The experiments are conducted on commercial off-the-shelf (COTS) components, which lends some external validity to the findings, as opposed to custom-designed test structures or simulations.
3. Detailed Experimental Setup: The paper provides specific details regarding the VNA configuration, frequency sweep, and data acquisition process (Section 6.3), which is a necessary, albeit basic, requirement for reproducibility.

Weaknesses

My primary concerns with this manuscript center on the validity of its core premise, the methodology used to isolate the phenomenon, and the interpretation of the results.

1. Conflation of System and Component Impedance: The fundamental flaw in this work is the lack of evidence isolating the impedance contribution of the memory cells themselves from the rest of the system. The measurements are taken on the 3.3V power rail (Section 6.3), which represents the aggregate impedance of the entire memory chip (including I/O buffers, sense amplifiers, row/column decoders, charge pumps) and the test PCB itself (decoupling capacitors, power planes, traces). The authors provide no control experiments—such as measuring the chip in a quiescent state or a baseline measurement of the PCB without the chip—to substantiate that the observed variations originate specifically from the N-bit memory state and not from peripheral circuitry that is trivially data-dependent. The simplified models in Section 4 (Fig. 6, 7) are cell-level, yet the measurements are system-level. This is a critical, unbridged gap.

2. Oversimplified Theoretical Foundation: The analytical models presented in Section 4 are rudimentary and fail to account for the dominant non-idealities in any real-world circuit. The equations (Eq. 3-8) ignore parasitic capacitance and inductance, process variations, temperature effects, and noise, all of which would heavily influence impedance measurements in the GHz range. To present these idealized equations as the basis for a complex, noisy, real-world phenomenon and then claim experimental validation is a significant logical leap. The work does not demonstrate that the measured effects are governed by these models versus other, more plausible circuit dynamics.

3. Questionable Signal Processing and Feature Selection Pipeline: The methodology in Section 7.1.1 is not adequately justified.

   • The choice of Pearson correlation for feature selection presupposes a linear relationship between impedance at a specific frequency and the stored data value. There is no theoretical or empirical justification provided for this assumption. Non-linear relationships would be missed entirely.
   • The two-step approach of selecting features via correlation and then applying PCA is unorthodox. PCA is designed to find the axes of maximal variance in a dataset. By pre-filtering the data, the authors may be biasing the analysis and discarding components that, while having lower individual correlation, might be highly informative in combination. A more rigorous approach would apply PCA to the full spectrum and analyze the resulting components or compare the chosen pipeline against standard alternatives.

4. Overstatement of Classification Results: While the reported classification accuracies in Table 1 seem high, the presentation obscures critical details.

   • The results are presented as ranges (e.g., F1-Score "84.3%-89.5%"). This is imprecise. Does this range represent variation across the 20 chips, across different NVM models within a technology type, or something else? This ambiguity hinders a critical assessment of the results' stability.
   • The confusion matrices in Figure 10 reveal non-trivial misclassifications. For MRAM (Fig. 10c), distinguishing HW0 has a False Discovery Rate (FDR) of 31.5%, and classifying HW8 only achieves a 77% True Positive Rate (TPR). This suggests the channel is significantly noisier and less reliable than the summary tables imply. To label a phenomenon with such error rates as definitively "exploitable" requires a more nuanced discussion of the attack context (e.g., error correction requirements) which is absent.

Questions to Address In Rebuttal

The authors must address the following points to substantiate their claims:

1. Isolation of Effect: How did the authors de-embed the impedance contribution of the NVM cells from the parasitic effects of the package, the PCB, and the chip's peripheral circuitry? Please provide control measurements or a detailed analysis that proves the observed impedance variations are not dominated by these other sources.
2. Model vs. Reality: Can the authors justify the leap from the simplified, ideal circuit models in Section 4 to the complex, system-level measurements? How do these models account for the frequency-dependent behavior of the PDN, including decoupling capacitor resonances, which are known to create significant impedance variations?
3. Methodological Justification: What is the justification for assuming a linear data-impedance relationship for the Pearson correlation feature selection? Please provide evidence comparing your two-step feature selection pipeline (correlation + PCA) with a baseline approach (e.g., PCA on the full spectrum) to demonstrate that your method is not biasing the results or discarding useful information.
4. Interpretation of Results: Please clarify precisely what the performance ranges in Table 1 represent. Furthermore, please address the significant misclassification rates in Figure 10 and discuss how they impact the practical exploitability of this side channel, which you claim is a key finding. An attack with a ~23% error rate on certain data patterns (MRAM HW8) is far from a foregone conclusion.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces and validates a novel side-channel attack vector against emerging non-volatile memories (NVMs) like ReRAM, FRAM, and MRAM. The core contribution is the demonstration that the fundamental data storage mechanism of these memories—their impedance state—can be directly and non-invasively measured through the Power Distribution Network (PDN) to leak the stored data. The authors challenge the implicit assumption that the shift from charge-based to impedance-based storage in NVMs inherently improves security against physical attacks.

Using S-parameter analysis with a Vector Network Analyzer (VNA), the authors show that data-dependent impedance variations are statistically significant across a range of commercial memory chips. Their methodology successfully distinguishes not only between data patterns with different Hamming weights (inter-HW) but also between patterns with the same Hamming weight (intra-HW), enabling powerful template-based attacks. The work serves as a foundational exploration of a new class of hardware vulnerabilities, highlighting that the physical properties of data storage can themselves become a source of leakage.

Strengths

1. Strong Conceptual Contribution: The primary strength of this paper is its conceptual clarity and novelty. It pivots the focus of side-channel analysis for NVMs from dynamic leakage (e.g., power consumption during write operations) to static leakage rooted in the physical state of the memory cells themselves. This is a crucial insight that opens a new and previously overlooked avenue for security research. By framing impedance not as a passive circuit parameter but as an active source of information leakage, the authors provide a valuable new perspective for the hardware security community.

2. Excellent Empirical Grounding: The work is not merely theoretical. The authors provide a robust and convincing experimental evaluation across three major NVM technologies (ReRAM, FRAM, MRAM) from multiple vendors (Section 6.1.1, page 7). This breadth significantly strengthens the generality of their findings. The systematic analysis of both inter- and intra-Hamming weight variations (Section 7.1, page 8) demonstrates the high resolution of the attack vector and its potential to defeat simple countermeasures.

3. Bridging Disparate Fields: The paper successfully connects two traditionally separate domains: the device physics of emerging memories and the RF/microwave measurement techniques used in signal integrity analysis. The repurposing of VNA and S-parameter analysis (Section 2.3, page 4), typically used for characterizing high-frequency circuits, as a tool for security vulnerability discovery is both clever and effective. This interdisciplinary approach is a model for future work in physical side-channel analysis.

Weaknesses

1. Understated Positioning Against Existing Impedance SCA: While the paper has a good "Related Works" section (Section 10, page 12), it could more forcefully articulate its contribution in the context of prior impedance side-channel work. Research on impedance leakage in FPGAs (e.g., [13, 65]) has already established the general principle. The authors should more explicitly highlight why their work is a significant leap forward—namely, the transition from analyzing dynamic switching in logic elements to extracting static data from high-density memory arrays. This is a non-trivial distinction that elevates the paper's impact, and it deserves more emphasis.

2. Practicality of the Threat Model: The threat model (Section 5, page 7), which assumes physical access and an identical reference chip for template building, is standard for this type of academic research. However, the work would have a greater impact if it briefly explored the boundaries of this model. For instance, a discussion on the potential for remote sensing (e.g., via backscattering) or the robustness of templates across different manufacturing batches would help contextualize the real-world threat level.

3. Countermeasures Section is Preliminary: The discussion on countermeasures in Section 8 (page 10) is comprehensive in its breadth but lacks depth. It serves as a good catalog of potential defenses but does not provide much intuition on which methods would be most effective or efficient against this specific static leakage channel. While a full evaluation is beyond the scope of this paper, a more focused discussion or a preliminary simulation of a promising countermeasure would transition the work from purely identifying a problem to pointing more concretely toward solutions.

Questions to Address In Rebuttal

The authors have presented a compelling and well-executed piece of research. I would appreciate their thoughts on the following points to further strengthen the work:

1. On the Novelty of Application: The related work in [13, 65] has explored impedance leakage in FPGAs. Could the authors further clarify the fundamental differences and challenges in extending this concept from configurable logic to dense, static NVM arrays? What makes this a non-trivial extension that constitutes a significant new contribution?

2. On the Robustness of Templates: The evaluation relies on template attacks using 20 chips of each type, which is a good practice. How sensitive are these impedance templates to environmental factors like process variations, temperature, and aging? Could a template trained on one batch of chips be successfully used against another, or is per-device (or at least per-batch) profiling necessary for a real-world attack?

3. On the Future of Defenses: Section 8 provides a good overview of potential countermeasures. Based on your findings, do you have an intuition about which category (e.g., architectural randomization, software-based masking, or device-level modifications) would be the most effective and cost-efficient defense against this specific type of static impedance leakage? For instance, would techniques that randomize memory layout be more effective than those that add noise to the PDN?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present "DExiM," an experimental study demonstrating that emerging Non-Volatile Memory (NVM) technologies—specifically ReRAM, FRAM, and MRAM—leak information about their statically stored data through their impedance characteristics. The core methodology involves using a Vector Network Analyzer (VNA) to measure the S11 reflection parameter of the memory chip's Power Distribution Network (PDN), from which the impedance profile is derived. The authors show that these impedance profiles contain statistically significant variations corresponding to both inter-Hamming weight (different numbers of '1's) and intra-Hamming weight (same number of '1's in different positions) data patterns. They use this leakage to train machine learning classifiers to distinguish the stored data with high accuracy. The central claim is that this represents a new class of hardware vulnerability for these memory types.

Strengths

1. Novel Application of a Known Technique to a New Domain: The primary strength of this paper is its application of impedance-based side-channel analysis to the domain of static data leakage from dedicated emerging NVM chips. While impedance analysis itself is not new, its use here to non-invasively read data-at-rest from technologies that encode information directly as impedance is a novel and logical extension. The work successfully identifies a vulnerability that is a direct consequence of the fundamental operating principle of these memories.

2. Systematic and Rigorous Validation: The authors provide a compelling existence proof through a thorough experimental evaluation. The use of three distinct NVM technologies (ReRAM, FRAM, MRAM) from multiple vendors, with tests conducted on 20 distinct chips for each model (as mentioned in Section 6.1.1, page 7), demonstrates that the observed phenomenon is not an artifact of a single device or architecture but a more fundamental property. This systematic approach adds significant weight to the core claim.

3. Clear Differentiation from Power Analysis: The paper effectively argues why this leakage vector is distinct from and potentially more potent than traditional power analysis for this class of devices. As noted in the introduction (Section 1, page 2), the low-power and limited switching activity of NVMs can make power SCA difficult, whereas impedance is a static property that remains measurable, presenting a fundamentally different attack surface.

Weaknesses

1. Incremental Advance Over Existing Impedance Leakage Work: The core idea of using impedance as a side channel is not new. The authors themselves cite the most relevant prior art, notably Monfared et al. [65] ("LeakyOhm") and a related preprint by the current authors [13]. "LeakyOhm" conclusively demonstrated data-dependent impedance leakage from FPGA logic elements (LUTs, flip-flops) and used it to mount a successful key-recovery attack on an AES implementation. The "delta" in this work is the change of target from programmable logic (FPGAs) to dedicated memory chips (NVMs). While this is a valid and important distinction, the conceptual framework—measuring data-dependent impedance variations via the PDN—is functionally identical. The paper's novelty rests entirely on this change of target, which makes the contribution more of an extension of an existing attack class to a new component type, rather than the discovery of a fundamentally new attack class.

2. Lack of a Demonstrated Novel Attack: The analysis stops at data classification. Prior work [65] took the critical next step of leveraging the identified leakage to perform a full cryptographic key extraction. By limiting the scope to classification of raw data patterns, the authors demonstrate a channel but not a novel attack with demonstrated real-world impact. This makes the contribution feel incomplete from a security perspective and lessens the significance of the novel finding. The paper hypothesizes about consequences like key extraction (Section 9, page 11) but does not provide evidence.

3. Generic and Non-Novel Countermeasures: The discussion of countermeasures in Section 8 (page 10) is a high-level overview of established side-channel defense principles (masking, randomization, ECC-aware design). There is no novel countermeasure proposed that is specifically tailored to the unique physical properties of impedance leakage in NVMs. This section lacks originality and reads like a summary of textbook defenses.

Questions to Address In Rebuttal

1. The closest prior art [65] demonstrated impedance leakage in FPGAs. Please elaborate on the fundamental physical differences in the source of impedance variations between a configured FPGA logic element (as in [65]) and a static NVM cell (as in your work). A more detailed comparison would help solidify the novelty of your contribution beyond simply a change in the device under test.

2. Given that prior work on impedance leakage [65] culminated in a full AES key extraction, why did your investigation stop at classifying 8-bit data patterns? Can you provide evidence or a compelling argument that the signal-to-noise ratio and observability of your discovered channel are sufficient for a similar, more complex attack on a real cryptographic primitive stored in one of these NVMs?

3. The countermeasures proposed in Section 8 are largely generic. Can you propose at least one concrete, novel countermeasure that is specifically designed to mitigate impedance leakage in NVMs at either the circuit or architectural level, which would not be a straightforward application of existing power/EM side-channel defenses?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents Sonar, a pre-silicon fuzzing framework that purports to systematically uncover contention-based side channels in processors. The core thesis is that multiplexers (MUXes) are the primary loci of resource contention, and by identifying these MUXes, one can define contention-critical states. The framework then uses these states, particularly the timing interval between requests (reqsIntvl), to guide testcase generation towards triggering contentions. The authors evaluate Sonar on two open-source RISC-V processors (BOOM and NutShell) and claim to have discovered 14 side channels, 11 of which are presented as new.

While the engineering of the framework appears functional for its defined scope, its fundamental premise is an oversimplification of microarchitectural contention. The methodology for guiding the fuzzer is naive, and the claims regarding the novelty and exploitability of the discovered vulnerabilities are not sufficiently substantiated.

Strengths

1. Automatable Heuristic for Contention Points: The core idea of leveraging MUX structures as a heuristic to identify potential contention points (Section 5.1, page 4) provides a scalable, automated starting point for analysis. The bottom-up tracing method is a logical way to group cascaded 2:1 MUXes into a single n:1 contention point.
2. Targeted Root-Cause Analysis: The dual-differential comparison method (Section 7, page 7), which correlates instruction commit cycle differences (CCD) with changes in contention-critical states, is a methodologically sound approach for narrowing down the source of a detected timing leak, reducing manual debugging effort.
3. Concrete Implementation: The framework is implemented and evaluated on two non-trivial out-of-order processor designs, demonstrating that the proposed instrumentation and analysis pipeline is functional.

Weaknesses

1. Oversimplified Contention Model: The central premise that MUXes are the definitive "hotspots" for all significant resource contention is fundamentally flawed. While MUXes are ubiquitous in signal selection, complex contention scenarios often arise from more intricate, stateful arbiters, queues with occupancy-dependent backpressure, and shared functional units whose contention logic is not fully captured by analyzing MUX input trees. This foundational assumption limits the scope of discoverable vulnerabilities to only those that manifest as simple signal selection conflicts, potentially missing more subtle and complex contention channels.

2. Naive Fuzzing Guidance Metric: The guidance metric, which exclusively seeks to minimize the request interval (reqsIntvl) (Section 6.2.1, page 6), is myopic. While forcing requests to be simultaneous (interval of zero) is a valid strategy for triggering simple volatile contentions, it fails to account for more complex scenarios. Many powerful side channels require precise, non-zero timing relationships (e.g., one request arriving exactly N cycles after another to exploit pipeline staging, buffer states, or arbiter fairness timers). The framework's singular focus on minimizing this interval is a greedy approach that likely prevents the discovery of such vulnerabilities and can easily get trapped in local optima where simple contentions mask the path to more complex ones.

3. Overstated Novelty of Findings: The claim of uncovering 11 "previously unknown" side channels (Abstract, page 1) is not supported by a rigorous analysis of the results. A close review of Table 3 (page 10) reveals that many of these "new" channels are simply instances of well-understood contention principles manifesting on the specific microarchitectures of BOOM and NutShell:

   • S1-S4 (TileLink Contention): This is a textbook case of bus/interconnect contention. Discovering that a long-running transaction can block a shorter one on a shared bus is not a novel vulnerability class.
   • S5 (MSHR Contention): This is a specific manifestation of MSHR pressure, a known side-channel vector. The paper itself notes its relation to Speculative Interference Attacks [11]. The "false sharing path blocking" appears to be a name for a specific trigger condition, not a new fundamental mechanism.
   • S11 & S12 (L1 DCache Contention): These are intra-thread variants of classic cache contention attacks (e.g., Prime+Probe, Flush+Reload). The novelty is limited to demonstrating they can be triggered without SMT, which is an interesting but incremental finding, not a new class of vulnerability.
   • The paper fails to demonstrate the discovery of a truly novel class of contention vulnerability.

4. Insufficient Evaluation of Exploitability: The exploitability analysis (Section 7.3, page 7 and Section 8.5, page 12) is superficial. Applying a generic Meltdown-style template demonstrates the existence of a transient execution window but provides no rigorous analysis of the signal-to-noise ratio, the difficulty of timing the attack under realistic system load, or the precise attacker capabilities required. The complete failure to construct a working PoC on NutShell (accuracy <2%) is a major red flag that is inadequately explained away by "earlier exception detection." This failure strongly suggests that the detected timing variations on NutShell are practically unexploitable and should not be classified as significant vulnerabilities without further evidence.

5. Limited Generalizability: The evaluation is confined to two academic, open-source RISC-V processors. The findings cannot be assumed to generalize to commercial-grade processors from vendors like Intel, AMD, or ARM, which feature far more complex and proprietary microarchitectures, interconnects, and prefetchers. Many of the uncovered issues appear to be artifacts of specific design choices in BOOM and NutShell rather than fundamental, universal principles.

Questions to Address In Rebuttal

1. Regarding the contention model: Can you provide a compelling argument or evidence that contention mechanisms not directly represented as MUX cascades (e.g., complex token-based arbiters, buffer occupancy management) are adequately covered by your approach? Please provide an example of a known side channel based on such a mechanism and explain how Sonar would detect it.

2. Regarding the guidance metric: How does the reqsIntvl minimization strategy avoid missing vulnerabilities that require a precise, non-zero timing interval between contending requests? Have you considered alternative guidance metrics that reward specific temporal patterns rather than just simultaneity?

3. Regarding novelty: Please justify the claim that channels S1-S4 and S11-S12 represent novel vulnerability classes, distinct from previously documented bus/interconnect and cache-based contention attacks, respectively. What is the fundamental new principle being demonstrated beyond its occurrence in a specific RISC-V core?

4. Regarding exploitability: Beyond citing "earlier exception detection," what specific microarchitectural investigation was performed to explain the <2% accuracy for the PoCs on NutShell? Does this result not suggest that the timing variations Sonar detects can be academic artifacts with no practical security impact? Why should these be considered exploitable vulnerabilities?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Sonar, a novel pre-silicon fuzzing framework designed to systematically uncover contention-based side channels in processor designs. The authors' core contribution is a microarchitecture-aware approach that departs from traditional random instruction fuzzing. Sonar is built on the key insight that resource contention often manifests at multiplexers (MUXes) within the circuit.

The framework operates in three stages: 1. It first uses a bottom-up tracing methodology starting from MUX outputs to automatically identify "contention points" and their associated states within the RTL design. 2. It then employs a state-guided fuzzing loop, using the timing interval between requests (reqsIntvl) at these contention points as a fitness metric to guide testcase mutation, progressively driving the design towards a state of contention. 3. Finally, it uses a "dual-differential comparison" method—comparing both instruction commit times and the microarchitectural contention states under different secret values—to accurately detect and pinpoint the root cause of the side channels.

Evaluated on the BOOM and NutShell open-source RISC-V cores, Sonar successfully identified 14 contention side channels, 11 of which are previously undocumented.

Strengths

The primary strength of this paper lies in its elegant and effective conceptual bridge between the architectural and microarchitectural domains for security verification.

1. A Powerful, Foundational Heuristic: The central idea to model resource contention as a problem of MUX arbitration (Section 5.1, page 4) is a significant contribution. It provides a concrete, automatable, and scalable way to find potential contention "hotspots" across a complex processor design without needing manual specifications. This moves the field beyond simply fuzzing instructions and hoping for the best, towards a more targeted and intelligent search. It's a foundational insight that could inform future work in hardware security verification.

2. Effective State-Guided Fuzzing: The paper successfully translates this microarchitectural insight into a practical feedback mechanism for fuzzing. Using the request interval (reqsIntvl) as a fitness function (Section 6.2.1, page 6) is a clever way to solve the notoriously difficult problem of triggering timing-sensitive events. It creates a gradient that the fuzzer can descend to force simultaneous requests, something that random mutation would struggle to achieve efficiently. This methodology represents a significant maturation of hardware fuzzing techniques for security.

3. Contextual Significance and Strong Results: This work is situated perfectly at the intersection of hardware verification and computer security. The need for pre-silicon detection of side channels is well-established, as post-silicon fixes are immensely costly. By finding 11 new, potentially exploitable channels in well-regarded open-source designs like BOOM (Table 3, page 10), the authors provide compelling evidence that their framework is not just a theoretical novelty but a practical and necessary tool. It effectively addresses a known gap left by both less-targeted fuzzers (like SpecDoctor) and less-scalable formal methods (like UPEC).

4. End-to-End Systematization: Sonar is presented not as a single trick, but as a complete, end-to-end framework. The combination of automated hotspot identification, guided triggering, and the dual-differential analysis for precise detection creates a systematic and repeatable process. This level of automation is crucial for integrating security analysis into the standard hardware design lifecycle.

Weaknesses

The weaknesses of the paper are primarily related to the boundaries of its core assumptions and its positioning relative to adjacent methodologies. These are opportunities for clarification and future exploration rather than fatal flaws.

1. Conceptual Limits of the MUX Model: The MUX-centric view is powerful for arbitrated resources (ports, buses, interconnects), but its applicability to other forms of contention is less clear. For example, the paper reports finding contention on a non-pipelined Multiply-Divide Unit (S13). While this is a valid contention channel, it stems from the unit's internal "busy" state rather than an input MUX selecting between competing requests. The paper would be strengthened by a discussion on the conceptual limits of the MUX model and how it generalizes (or doesn't) to other forms of resource occupancy.

2. Generalizability of the Implementation: The framework's implementation relies on analyzing FIRRTL, a high-level intermediate representation generated from Chisel (Section 8.2, page 8). While this is pragmatic for academic research using open-source cores, it raises questions about its applicability in industrial settings, which predominantly use Verilog/SystemVerilog and may not provide access to such a high-level IR. A discussion on the challenges and potential pathways for adapting the "bottom-up tracing" to standard Verilog RTL or even gate-level netlists would significantly broaden the perceived impact of this work.

3. Nuance in Comparison to Formal Methods: The paper positions itself against formal methods primarily on the basis of scalability. This is a fair and important point. However, the discussion could be more nuanced. Formal methods provide proofs of absence (within certain bounds), a powerful guarantee that fuzzing cannot offer. A brief exploration of the fundamental trade-offs would be valuable. Are there classes of contention channels Sonar might miss that a formal tool could theoretically find, and vice versa?

Questions to Address In Rebuttal

1. Could the authors elaborate on the conceptual boundaries of their MUX-based contention model? For instance, how does it capture contention for non-pipelined functional units (e.g., S9, S13 in Table 3) where the bottleneck isn't necessarily request arbitration at an input MUX, but rather the busy state of the unit itself? Is the framework implicitly monitoring this through other MUXes (e.g., on the writeback path), or does this suggest the need for a hybrid model?

2. The reliance on FIRRTL is pragmatic for the chosen evaluation targets. Can the authors comment on the feasibility of adapting the "bottom-up tracing" methodology to more standard, lower-level representations like Verilog RTL or even gate-level netlists, which would be crucial for industrial adoption?

3. Beyond scalability, could you provide more insight into the fundamental trade-offs between Sonar's dynamic approach and formal verification methods like UPEC? Are there specific types of contention channels (perhaps those requiring extremely complex state setup) that one is inherently better suited to find than the other?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

This paper presents Sonar, a pre-silicon fuzzing framework for detecting contention-based side channels in processor RTL designs. The authors propose a three-part methodology. First, they introduce a technique to systematically identify potential contention points by treating multiplexers (MUXes) as proxies for resource arbitration and using a bottom-up tracing algorithm. Second, they employ a guided fuzzing strategy where the feedback metric is the timing interval between competing requests (reqsIntvl) at these MUX-defined contention points, with the goal of minimizing this interval to trigger simultaneous contention. Third, they use a "dual-differential" analysis method to confirm the side channel, which compares both instruction commit-timing differences and the underlying microarchitectural contention states under different secret values.

The core novelty of this work rests on the combination of these three ideas: the systematic, structural identification of contention points via MUX analysis to guide a fuzzer, the use of a microarchitectural timing interval as a direct feedback loop for mutation, and the automated root-cause analysis that links observed timing effects to specific contention events. While individual components build on existing concepts (guided fuzzing, differential testing), their synthesis into a cohesive framework targeted specifically at contention side channels appears to be a novel contribution to the field of pre-silicon hardware security verification.

Strengths

From the perspective of novelty, the paper has several significant strengths:

• S1: A Novel Heuristic for Locating Contention. The central idea of using MUXes as a structural signature for contention points (Section 5.1, page 4) is a genuinely new and clever heuristic for the purpose of fuzzing. Prior fuzzing work (e.g., SpecDoctor, SIGFuzz) has relied on more abstract or effect-based monitoring (e.g., transient state coverage, commit timing). Sonar's approach is the first I am aware of to systematically parse the circuit structure itself (via FIRRTL) to derive targets for contention, which is a significant methodological advancement. The "bottom-up tracing" method to identify the full set of inputs to a cascaded MUX is a concrete, novel algorithm that enables this approach.

• S2: A Novel Feedback Mechanism for Triggering Contention. Guided fuzzing is not new, but the choice of feedback is critical and domain-specific. The authors' proposal to use the inter-request timing interval (reqsIntvl) as the fitness function for the fuzzer (Section 6.2.1, page 6) is a novel application tailored to the unique challenge of triggering timing-sensitive microarchitectural events. Driving mutations to explicitly minimize this value is a far more directed strategy than the random or coverage-guided approaches of prior work and represents a new technique in the hardware fuzzer's toolkit.

• S3: A Novel Method for Automated Root Cause Analysis. Standard differential testing can reveal the existence of a timing leak, but not necessarily its cause. Sonar’s "dual-differential comparison" method (Section 7, page 7) is a notable contribution. By simultaneously comparing the Commit Cycle Difference (CCD) to isolate the affected instruction and the contention state logs to identify the responsible MUX, the framework moves beyond mere detection to automated attribution. This linking of effect to cause is a qualitative improvement over prior art and represents a novel analysis workflow.

Weaknesses

The primary weaknesses relate to the framing of the novelty and the evaluation of its significance.

• W1: Overstated "First Mover" Claim. The abstract claims Sonar is the "first systematic and automated fuzzing framework designed to uncover contention side channels". This claim is too strong and potentially misleading. Frameworks like SpecDoctor [25] and SIGFuzz [26] are also fuzzing frameworks that can, and do, uncover contention side channels (e.g., port contention is a known Spectre-v4 variant). The true novelty of Sonar is not that it finds them, but how it finds them. The claim should be refined to state that it is the first framework to use structural analysis of arbitration logic (MUXes) to systematically guide a fuzzer towards triggering these contentions. The innovation is in the specific guidance mechanism, not in being the first tool in the category.

• W2: Insufficient Differentiation from Conceptually Adjacent Ideas. The concept of resource contention is fundamental to computer architecture. While using MUXes for fuzzing is new, the idea that arbiters are contention points is not. The paper would be stronger if it more clearly delineated why its automated MUX-based approach provides a significant advantage over a simpler, more manual approach where a designer might identify major arbiters (e.g., for execution ports, memory ports, bus interfaces) as monitoring targets. The novelty lies in the automation and scale, and this should be emphasized more.

• W3: Complexity vs. Benefit Analysis. The proposed methodology introduces significant complexity: a full-design RTL pass for MUX tracing and extensive instrumentation for dynamic reqsIntvl monitoring. The overheads are non-trivial, as shown in Table 2 (page 9). For the novelty to be truly significant, the benefit must clearly outweigh this cost. The comparison with SpecDoctor in Section 8.3.4 (page 9) shows Sonar triggers more contentions, which is a good result. However, this comparison doesn't fully assess the trade-off. Is it possible that a much simpler heuristic (e.g., monitoring a handful of key pipeline stall signals) could achieve 80% of the results with 10% of the complexity? A discussion on this trade-off would help solidify the significance of the proposed novel, but complex, approach.

Questions to Address In Rebuttal

1. Please refine your primary novelty claim. Given that prior fuzzers like SpecDoctor [25] can detect contention-based channels, can you clarify precisely what makes Sonar the "first" in its class? Is the novelty more accurately described as being the first structurally-guided fuzzer for this purpose?

2. The core of your guidance is the MUX-based heuristic. How robust is this heuristic? Are there common sources of resource contention in modern processors that are not implemented with straightforward MUX trees and would therefore be missed by Sonar? Conversely, does the automated tracing produce a large number of "contention points" at MUXes that are architecturally uninteresting or do not represent a meaningful shared resource, leading to wasted effort by the fuzzer?

3. Your reqsIntvl feedback mechanism is innovative but seems computationally intensive. For a complex contention point with N inputs, the fuzzer must track O(N^2) request pairs. How does the framework scale as the number of identified contention points and their input fan-in increases, especially in a large, complex SoC design beyond the evaluated BOOM/NutShell cores?

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents a symbiotic hardware prefetcher, the Task-Seeded Prefetcher (TSP), and a prefetch-aware task scheduler, the Memory Response Task Scheduler (MRS), designed for fine-grain, task-parallel manycore systems. The central thesis is that task descriptors, available to the scheduler before task execution, can seed a prefetcher to overcome the limitations of conventional prefetchers, which are ineffective for short-lived tasks. MRS then leverages the status of these prefetches to reorder task execution, prioritizing tasks whose data is already in cache. The authors evaluate this system across four scheduler types and two cache sizes, claiming significant speedups.

While the problem is well-defined and the high-level approach is conceptually plausible, the work suffers from a critical methodological flaw in its core premise of "predictability," and the evaluation reveals fundamental negative interactions that the proposed mechanisms exacerbate rather than solve. The paper identifies these serious pathologies but dismisses them as out of scope, undermining the claimed general applicability and robustness of the proposed solution.

Strengths

1. Problem Motivation: The paper correctly identifies a critical performance bottleneck in hardware-assisted task-parallel systems: the ineffectiveness of conventional memory prefetchers for very short tasks. The data in Figure 1 and Table 1 provides clear evidence of both the performance gap due to memory latency and the short duration of tasks, strongly motivating the need for a new approach.

2. Core Concept: The fundamental idea of using the task descriptor (function pointer and arguments), which is known well in advance of execution, to initiate prefetching is sound and represents a logical direction for these architectures.

3. Evaluation Breadth: The authors conduct an extensive evaluation, covering four distinct scheduling policies (combinations of speculative/non-speculative and random/spatial mapping) and two different cache configurations. This breadth attempts to demonstrate the general utility of their mechanisms.

Weaknesses

1. Circular Definition of Predictability: The paper's central claim rests on the "predictability" of memory accesses. However, this predictability is never independently validated. Table 1 (page 3) presents "Predictable loads (%)" as evidence, but the text on page 3 reveals how this is measured: by "counting the accesses for which TSP is able to issue prefetches." This is a tautology. The evidence for predictability is that the proposed predictor can predict them. This fails to provide any objective, foundational proof that these access patterns are inherently simple. A rigorous analysis would first classify access patterns (e.g., affine on arguments, multi-level pointer chasing) and then demonstrate the mechanism's coverage, rather than defining coverage as the metric itself.

2. Fundamental Negative Interaction 1: Priority Inversion: The symbiotic relationship between TSP and MRS is shown to be actively harmful for priority-ordered algorithms. The authors explicitly document a 1.3x slowdown for sssp-r on the NONSPECSPAT 1/16-cache system (Section 5.2, page 9). They correctly diagnose the cause: MRS's reordering of tasks based on data availability causes severe "priority inversions," leading to a 2.1x increase in the number of tasks executed. Their response to this fundamental design flaw is to state, "We leave applying MRS to such specialized schedulers as future work." This is unacceptable. A general-purpose scheduler that breaks a canonical, high-performance graph algorithm is not a general-purpose solution. The paper demonstrates a core tension between latency hiding and work efficiency and fails to resolve it.

3. Fundamental Negative Interaction 2: Increased Speculation Aborts: The evaluation shows that for msf-o, the proposed mechanisms lead to a performance degradation due to increased task aborts (Section 5.2, page 9). Aggressive prefetching, by its nature, brings data into the cache earlier, widening the window for data conflicts in a speculative system. This increases the likelihood of aborts, wasting significant work. Again, the authors diagnose a critical negative interaction that their "symbiotic" system creates but offer no solution within their framework. This suggests the two components are not truly symbiotic but can be mutually destructive under common conditions.

4. Inefficient Prefetching for Common Control Flow: The paper acknowledges that TSP prefetches aggressively for "moot tasks"—tasks that exit early based on a data-dependent condition (Section 5.3, page 12). This generates useless memory traffic and pollutes the cache. The authors' suggested remedy is to use a different architecture entirely (Hive), which is an admission that their mechanism is inefficient for a prevalent pattern in irregular applications. A robust prefetcher must be more intelligent about control flow.

5. Ad-Hoc Training Mechanism: The training FSM in Figure 5d relies on a "Randomize Dependence" transition when a learned pattern repeatedly fails. The paper provides no justification for this seemingly ad-hoc heuristic or analysis of its convergence properties. In a system with many potential dependencies (arguments and prior loads), random guessing could be highly inefficient and fail to converge on the correct pattern in a timely manner, if at all.

Questions to Address In Rebuttal

1. Please provide a mechanism-independent analysis of the memory access patterns in your benchmarks to substantiate the claim of high predictability. For instance, what percentage of dynamic loads are simple affine functions of task arguments, and what percentage are more complex indirect patterns?

2. The slowdown in sssp-r is attributed to priority inversion caused by MRS. This is not a minor outlier but a failure on a canonical algorithm for which priority scheduling is essential. How would you modify MRS to balance the goals of latency hiding and maintaining priority order to prevent such work-inefficiency pathologies? Simply stating it is "future work" is insufficient.

3. Your system increases abort rates in speculative execution for benchmarks like msf-o. How can the "symbiotic" relationship between TSP and MRS be modified to mitigate this effect? For example, should MRS consider the likelihood of data conflicts when reordering tasks?

4. Can you provide a more rigorous justification for the "Randomize Dependence" step in your training FSM? What is the search space for this randomization, and what is the expected convergence time for a task with N arguments and M prior loads? Have you considered more structured search heuristics?

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a symbiotic co-design of a data prefetcher (TSP) and a task scheduler (MRS) for fine-grained, hardware-assisted, task-parallel systems. The core thesis is that in such systems—where conventional prefetching fails due to the short lifetime and unpredictable dispatch order of tasks—the scheduler and prefetcher must work in tandem.

The authors' solution facilitates a bidirectional information flow: 1. Scheduler to Prefetcher: The scheduler "seeds" the Task-Seeded Prefetcher (TSP) with descriptors of tasks waiting in the queue, providing the crucial lookahead needed to initiate timely memory requests. TSP learns the memory access patterns of task functions, including direct, indirect, and striding patterns, based on their arguments. 2. Prefetcher to Scheduler: The Memory Response Task Scheduler (MRS) leverages the status of these prefetches. It adjusts the baseline scheduling policy to prioritize dispatching tasks whose data has already arrived in the cache, thereby improving core utilization and hiding memory latency.

This symbiotic approach is shown to yield significant speedups (gmeans up to 1.4x, peak 3.1x) on systems that are already highly optimized, effectively unlocking the performance potential that was previously gated by memory latency.

Strengths

The primary strength of this work lies in its elegant and insightful framing of a fundamental problem. It sits squarely at the intersection of two major research thrusts in computer architecture: overcoming the memory wall and exploiting fine-grained parallelism.

1. Identifies and Solves a Critical Bottleneck: The hardware task-parallelism model, exemplified by prior work like Swarm [39], was a major step forward for irregular applications. However, as this paper correctly and clearly articulates in Section 2 (page 2), this model created a new problem: it broke the assumptions that underpin decades of prefetching research. This paper doesn't just identify this gap; it provides a compelling and well-motivated solution. It is an essential, enabling technology for this entire class of architectures.

2. Novelty of the "Symbiotic" Co-design: While the components build on established concepts (e.g., TSP's learning mechanism is an intelligent adaptation of indirect memory prefetchers like IMP [94]), the true novelty is the tight, closed-loop integration. The idea that the task queue is not just a work container but a rich source of semantic lookahead for the memory system is powerful. Furthermore, using prefetch status to guide scheduling (MRS) moves beyond simple locality-aware scheduling into a new domain of "data-readiness-aware" scheduling. The illustrative example in Figure 4 (page 5) is exceptionally effective at communicating this core contribution.

3. Excellent Contextualization and Synthesis: The authors demonstrate a panoramic understanding of the field. They astutely draw parallels between their general-purpose approach and the hard-wired data-fetching mechanisms found in specialized accelerators. In essence, TSP and MRS can be seen as a way to achieve the data-fetching efficiency of an accelerator within a flexible, general-purpose, programmable tasking model. This is a significant conceptual contribution.

4. Strong and Convincing Evaluation: The experimental methodology is thorough, testing the proposed techniques across four distinct scheduler designs and two different cache configurations. The comparison of a 1/16th size cache with TSP/MRS against a full-size cache without a prefetcher is particularly powerful. The results, shown in Figure 6 (page 10), often demonstrate that a smaller, "smarter" cache with this symbiotic system is more effective than a large, "dumb" one, making a strong case for the area efficiency of this approach.

Weaknesses

The weaknesses of the paper are not in its core idea, which is sound, but in the completeness of the symbiotic relationship and its potential interactions with other system dynamics.

1. First-Order Symbiosis Ignores System-Level Pathologies: The paper is commendably transparent about scenarios where the system degrades performance, such as for sssp-r on the non-speculative spatial scheduler or msf-o on the speculative one (Section 5.2, page 9). The authors attribute this to negative interactions with priority inversion and increased speculation aborts. This suggests that the current symbiosis is "first-order"—it optimizes for the data readiness of individual tasks but is blind to second-order, system-wide effects like speculation pressure or fairness. A deeper integration might involve the prefetcher providing contention information to the scheduler, or the scheduler informing the prefetcher about a task's likelihood of being aborted.

2. Limited Scope of Predictability: The prefetching model is based on affine functions of task arguments and prior loads (Equation 1, page 4). This is a practical and effective choice for the workloads studied, but it represents a specific point in the design space. More complex, non-affine access patterns (e.g., hash-based or input-dependent control flow) would not be captured. While this is a reasonable limitation, the work would be stronger if it discussed where it fits on the spectrum between simple stride prefetchers and more heavyweight solutions like runahead execution.

3. Unexplored Behavior on "Friendly" Workloads: The evaluation focuses exclusively on irregular workloads, which is the target domain. However, to understand its place in a truly general-purpose system, it would be valuable to understand how TSP/MRS behaves on workloads where conventional stream/stride prefetchers excel. Is there a risk of negative interference, or does the system gracefully handle these cases as well? This would help contextualize the overall utility and robustness of the design.

Questions to Address In Rebuttal

1. The performance degradation due to increased aborts and priority inversion is a fascinating result. It suggests that aggressively reordering tasks via MRS can disrupt the delicate balance of speculative and priority-ordered systems. Could the symbiotic relationship be extended? For example, could the scheduler throttle MRS’s reordering when speculative state storage is nearly full, or when a large priority gap emerges between the highest-priority task and the highest-priority ready task?

2. Regarding the training mechanism for TSP: How quickly does it adapt to phase changes within an application, where a task function's memory access patterns might change? The FSM in Figure 5d (page 6) seems robust, but some discussion on training latency and adaptability would strengthen the paper.

3. Could you elaborate on the potential of this architecture beyond latency hiding? The prefetcher learns a task’s memory footprint. Could this information be used for other optimizations, such as guiding data placement in a NUCA system or informing fine-grained power management by predicting memory- or compute-intensive phases of a task? This speaks to the broader potential of the symbiotic model.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents two symbiotic hardware mechanisms, the Task-Seeded Prefetcher (TSP) and the Memory Response Task Scheduler (MRS), designed to address the memory latency bottleneck in manycore systems with hardware-assisted, fine-grain task parallelism. The authors identify a key problem: conventional prefetchers are ineffective for very short tasks (e.g., <200 cycles) because the task completes before the prefetcher can be trained and issue timely requests.

The core idea is to leverage the hardware task scheduler, which has access to task descriptors (function pointers and arguments) before a task is dispatched. TSP uses these descriptors to "seed" a prefetch engine, learning and predicting memory access patterns (argument-based, indirect, and striding) well before the task executes. MRS then augments the scheduler's dispatch logic, using feedback on prefetch completion from TSP to prioritize tasks whose data is already available in the cache, thereby reordering execution to maximize core utilization.

Strengths

The primary strength of this work lies in its novel integration of two distinct hardware components—the task scheduler and the data prefetcher—to solve a well-defined and challenging problem. While the constituent concepts have appeared in prior art, their specific combination and application context are new.

1. Novel Prefetch Trigger Mechanism: The central novel idea is using the task descriptor from a hardware task queue as the seed for a sophisticated data prefetcher. Conventional hardware prefetchers are triggered by the program counter (PC) and the stream of memory accesses from an executing thread. Prior work on programmer-assisted prefetching for tasks (e.g., Tesseract [5], Minnow [95]) required explicit hints or separate prefetch tasks. TSP proposes an automatic mechanism that learns complex access patterns initiated not by execution, but by the mere act of a task being queued for future execution. This is a genuinely new trigger for this class of prefetcher.

2. Novel Scheduler-Prefetcher Symbiosis: The feedback loop where the prefetcher's status directly and dynamically influences the scheduler's dispatch decision (MRS) is a novel form of hardware integration for general-purpose task-parallel systems. While the high-level concept of data-driven or locality-aware scheduling exists, MRS implements this as a reactive optimization on top of a conventional priority-ordered scheduler. It doesn't replace the scheduling paradigm but augments it, as clearly illustrated in Figure 4 (page 4). This symbiotic relationship is the paper's key conceptual contribution.

3. Clear Articulation of the "Delta": The paper does a commendable job of positioning its contributions against prior art. It correctly identifies that its learning mechanism for indirect and affine access patterns is an adaptation of the principles from the Indirect Memory Prefetcher (IMP) [94], as stated in Section 3.2 (page 5). The authors do not claim to have invented this learning logic, but rather to have found a new and effective way to trigger and apply it. This honest and precise positioning strengthens the paper's novelty claims.

Weaknesses

The novelty of this work is in the combination and application, not in the fundamental building blocks themselves. A critical assessment reveals the following:

1. Constituent Mechanisms are Adaptations of Prior Art: The address prediction logic within TSP—predicting addresses as an affine function of a reference value (address = scale * [ref] + offset) and chaining these predictions for indirect accesses—is functionally identical to the core mechanism of IMP [94] and its successors like DMP [30]. The novelty is confined to the source of the ref value (a task argument from the scheduler vs. a value from a striding access stream).

2. Conceptual Overlap with Dataflow Principles: The core idea of MRS—delaying task execution until its inputs are available—is the foundational principle of dataflow computing. While the authors' implementation as a reordering heuristic within a priority queue is a novel implementation for this domain, the underlying concept is not new. The paper would be stronger if it more explicitly differentiated its approach from classical dataflow execution models, particularly highlighting how MRS preserves a baseline priority ordering that pure dataflow lacks.

Questions to Address In Rebuttal

1. Generalizability of the Learning Mechanism: The TSP implementation appears tightly coupled to the affine function prediction model adapted from IMP [94]. Given the significant body of work on more advanced data prefetchers that recognize more complex patterns (e.g., graph traversals, irregular pointer chains), could TSP's front-end (the "task-seeding" trigger) be decoupled from its back-end (the prediction engine)? How much of the proposed architecture is specific to this one learning model versus being a general framework for pre-execution prefetching?

2. Distinction from Dataflow Models: Please clarify the novelty of the MRS policy compared to task scheduling in dataflow or stream processing architectures, where computation is naturally triggered by data availability. The key distinction appears to be that MRS modifies an existing priority order rather than deriving the schedule solely from data dependencies. Can the authors elaborate on why this distinction is significant and what benefits it provides over a pure data-driven model in the context of their target workloads?

3. Defense of "First" Claim: The conclusion (Section 7, page 13) claims TSP is the "first prefetcher to target hardware task-parallel systems without requiring programmer assistance." While this appears true within the specific domain of general-purpose manycore architectures, some specialized accelerators (e.g., for graph processing or deep learning) use work descriptors to pre-stage data automatically. Can the authors confirm the novelty of their integrated scheduler-prefetcher design against these more domain-specific architectures which may employ conceptually similar hardware mechanisms?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes Software Prefetch Multicast (SPM), a software-hardware co-design to improve bandwidth efficiency in manycore processors. The core idea is to use a new "sharer-exposed" prefetch instruction, generated by software analysis, to inform the hardware (specifically, the LLC) of which cores form a "sharer group." Within a group, a designated "leader" core issues prefetches, which trigger a multicast of the shared data from the LLC to all "follower" cores. The followers block their own redundant prefetches. To handle thread execution variance, the design incorporates a timeout mechanism for followers and a leader-switching mechanism. The authors claim significant NoC bandwidth savings and speedups over baseline systems.

However, the proposed mechanism introduces substantial complexity and a cascade of corrective measures (timeouts, leader switching) to patch fundamental vulnerabilities in its core leader-follower model. The evaluation, while showing positive results on regular benchmarks, raises serious questions about the general applicability, hidden overheads, and robustness of the system in real-world scenarios.

Strengths

1. Problem Motivation: The paper correctly identifies the escalating bandwidth pressure on the NoC and LLC in manycore systems as a critical bottleneck, providing a solid motivation for the work.
2. Leveraging Software Insight: The high-level approach of using static (or compiler) information to guide hardware coherence actions is a valid research direction, potentially avoiding the pitfalls of purely speculative hardware predictors.
3. Targeted Comparison: The analysis in Section 4.2 (page 8), which evaluates performance when the working set size exceeds the LLC capacity, is a relevant stress test that effectively highlights a key limitation of directory-history-based schemes like Push Multicast.

Weaknesses

1. Fragility and Compounding Complexity: The core leader-follower model is inherently fragile to thread asynchrony, a common case in parallel applications. The authors acknowledge this but address it with a series of complex, reactive patches. A follower waits (potentially stalling) for a slow leader. If it waits too long, it triggers a Timeout (Section 3.5, page 5), generating a unicast request and response, which partially defeats the purpose of multicast. If timeouts become frequent, a Leader-Followers Switching Mechanism (Section 3.6, page 6) is invoked, adding yet another layer of state management and network traffic. This design appears to replace the challenge of hardware prediction with an equally, if not more, complex challenge of hardware-based runtime synchronization management.

2. Introduction of New Performance Bottlenecks: The "Waiting Strategy" is a double-edged sword. While it reduces upstream traffic from followers, it can actively degrade performance by forcing a faster core to stall waiting for a multicast initiated by a lagging leader. The paper's own data confirms this failure mode. In Figure 13 (page 8), SPM performs worse than the standard L1Bingo-L2Stride prefetcher on backprop. The authors state this is because "a demand request may need to wait in the follower private cache." This is a critical admission that the mechanism can be actively harmful, yet this trade-off is not sufficiently analyzed.

3. Underestimation of System Overheads:

   • Hardware Cost: The claim of "light" hardware overhead (Section 4, page 7) is questionable. A 64-core system requires 304 Bytes/LLC slice, 280 Bytes/L2 cache, and a 64-entry (464 Bytes) Waiting Table per L2. This totals (64 cores * (280 B + 464 B)) + (64 slices * 304 B) ≈ 67 KB of configuration/state SRAM across the chip, which is not a trivial hardware cost.
   • Configuration Traffic: The Configuration Stage (Section 3.4, page 4) requires each sharer core to broadcast a config_req to all LLC slices. For a group of 16 sharers in a 64-slice system, this is 16 * 64 = 1024 configuration messages. While likely small packets, this initial broadcast storm is a non-trivial network overhead that is not quantified. The claim of "0.1 us" overhead is for a single round trip and does not account for this broadcast traffic or scenarios with many sharer groups.
   • Context Switch Penalty: The description of context switch handling in Section 3.7 (page 6) is superficial. A thread switch-out triggers a de-allocation message, broadcast to all LLCs. A new thread switch-in, if part of a sharer group, must then re-initiate the entire configuration stage. The performance penalty of this complete teardown and rebuild of sharer state upon a context switch seems prohibitive and is not evaluated.

4. Evaluation Scope and Parameter Tuning:

   • Benchmark Selection: The chosen benchmarks (e.g., cachebw, multilevel, conv3d) are characterized by highly regular, statically analyzable, bulk-synchronous sharing patterns. The proposed approach is tailor-made for these best-case scenarios. Its applicability to workloads with more dynamic, irregular, or pointer-based sharing is entirely unproven.
   • Parameter Sensitivity: Performance appears highly sensitive to the Timeout Threshold. As shown in Figure 19 (page 10), selecting a suboptimal value (e.g., 128 cycles instead of 512 for cachebw) can significantly degrade performance. The paper provides no methodology for how this critical parameter would be determined for arbitrary applications in a production environment, rendering the design impractical.

5. Inconsistent Bandwidth Savings Claims: Figure 17 (page 9) shows that for shared data, SPM results in slightly more ejected flits at the L2 cache than the baseline. The authors explain this is due to timeout-triggered unicasts and multicasts arriving at cores that do not yet need the data. This contradicts the narrative of pure bandwidth efficiency; the system reduces upstream request traffic but can increase downstream data traffic, including useless traffic that pollutes private caches.

Questions to Address In Rebuttal

1. The SPM design seems to be a cascade of fixes: the Timeout mechanism fixes the slow-leader problem, and the Leader Switching mechanism fixes the chronic slow-leader problem. Can the authors justify this layered complexity, and why is it superior to a simpler mechanism that embraces asynchrony, such as allowing followers to issue their own requests to be coalesced at the LLC?

2. Please provide a detailed analysis of the performance degradation seen in backprop (Figure 13, page 8). Specifically, quantify the average stall time introduced by the Waiting Strategy in follower caches and explain why this penalty is more severe than the latency savings from multicasting in this particular workload.

3. Regarding overheads:

   • Please justify the claim that ~67KB of distributed state SRAM for a 64-core system is "light."
   • Please quantify the network traffic (in flits) and latency of the Configuration Stage for a 16-sharer group in the 64-core system. How does this overhead scale as the number of distinct sharer groups in an application increases?
   • What is the estimated performance impact (in cycles) of a single context switch, including the full de-allocation and re-configuration sequence described in Section 3.7 (page 6)?

4. The Timeout Threshold is a critical performance parameter. How do you propose this value be set in a real system where application behavior is not known a priori? Would this require a complex hardware/software runtime tuning mechanism, adding even more complexity to the design?

5. Given that SPM can increase the amount of data traffic ejected to private caches (Figure 17, page 9), some of which may be unused, how does the system ensure that the net effect is a reduction in energy consumption, not just a shift in where bandwidth is consumed?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Software Prefetch Multicast (SPM), a software-hardware co-design aimed at mitigating the on-chip network (NoC) and last-level cache (LLC) bandwidth bottleneck in manycore processors. The core contribution is a mechanism that allows software (i.e., the compiler or programmer) to explicitly communicate the complete set of sharing cores ("sharer groups") to the hardware. This is achieved through new sharer-exposed prefetching instructions.

The hardware uses this information to perform precise, bandwidth-efficient multicasting. To handle the practical challenge of thread-to-thread performance variation, the authors propose a robust leader-follower model. In this model, only one designated "leader" thread issues the prefetch for the group, while "follower" threads block their own redundant prefetches and await the multicast data. Crucially, the system includes a timeout and dynamic leader-switching mechanism to ensure laggard leaders do not stall the entire group, adding a layer of resilience to the design. The evaluation demonstrates significant NoC bandwidth savings (42-50%) and substantial speedups (geomean of 1.28x-1.38x) on 16- and 64-core systems.

Strengths

1. Elegant and Direct Solution to an Information Problem: The fundamental strength of this work lies in its diagnosis of the core problem: hardware, on its own, has an incomplete picture of application-level data sharing. Speculative approaches like Push Multicast [12] are clever but are ultimately constrained by the limitations of historical predictors (e.g., directory capacity, silent evictions, first-access misses). SPM elegantly sidesteps this by creating a direct channel for the software, which possesses ground-truth knowledge of sharing patterns, to inform the hardware. This shifts the paradigm from speculative prediction to explicit direction, which is a powerful and promising approach.

2. Pragmatic Handling of System Dynamics: A purely static leader-follower model would be too brittle for real-world execution. The paper's most impressive design feature is its clear-eyed acknowledgement and handling of thread variation (as motivated in Section 2.3, page 3). The timeout mechanism combined with the dynamic leader-switching algorithm (Section 3.6, page 6) provides the necessary resilience to make the co-design practical. This elevates the work from a simple "what if" idea to a well-considered system. The ablation study in Figure 22 (page 11) effectively validates the necessity of these components.

3. Strong Placement within the Research Landscape: The authors successfully position their work as a synthesis of several long-standing research threads. It leverages the precision of software prefetching, applies the bandwidth-saving principles of multicasting (seen in early work like the NYU Ultracomputer [10]), and embodies the modern trend of software-hardware co-design. It provides a compelling alternative to purely hardware-based coherence prediction schemes [18, 19, 23] by trading hardware complexity and speculation for software hints and ISA support.

4. Well-Scoped and Insightful Future Work Discussion: The discussion in Section 6 (page 12) is a significant strength, as it demonstrates a broad understanding of the system-level implications. The authors thoughtfully consider the interaction with hardware prefetchers, the path toward compiler automation, and the potential impact of new instructions like AMX. This shows maturity and provides a clear roadmap for how this foundational idea could be expanded and integrated into future systems.

Weaknesses

While the core idea is strong, its applicability and system-level integration could be further explored.

1. Software Scope and Generality: The current evaluation relies on manual analysis and insertion of SPM instructions into workloads with regular, statically analyzable sharing patterns (e.g., dense linear algebra). The true test of such a co-design is its generality. It remains an open question how effective SPM would be for applications with more dynamic, input-dependent, or irregular sharing (e.g., graph analytics, sparse solvers, or certain pointer-intensive data structures). While compiler support is noted as future work, the fundamental recognizability of sharer groups is a prerequisite.

2. Configuration and Context-Switch Overhead: The configuration stage (Section 3.4, page 4) involves broadcasting requests to establish sharer groups and leader/follower roles. The paper asserts this overhead is low, but in a workload characterized by many frequent, short parallel regions, this setup/teardown cost could become significant. Similarly, the process for handling context switches (Section 3.7, page 6) involves de-allocation and re-configuration messages, which could add non-trivial overhead in a heavily multi-programmed environment. The impact of these transient states on performance is not fully characterized.

3. Interaction with the Coherence Protocol: The paper focuses primarily on the data movement and bandwidth aspects of the design. However, the interaction with the underlying cache coherence protocol (e.g., MESI) is not fully detailed. For example, what happens if a follower thread attempts to issue a store (a Request for Ownership) to a cache line that is currently in-flight as a read-only multicast prefetch triggered by the leader? This could introduce complex races or require additional logic in the cache controllers that is not discussed.

Questions to Address In Rebuttal

1. Beyond the evaluated workloads, could the authors comment on the applicability of SPM to applications with more dynamic or opaque sharing patterns, such as graph processing? Is the mechanism fundamentally limited to statically-determinable sharing, or is there a path to supporting more irregular workloads, perhaps through runtime profiling as hinted at in Section 6?

2. Could the authors elaborate on the scalability of the configuration stage? While the latency for a single configuration is low, can they provide analysis or data on the potential performance impact in a scenario with thousands of small, distinct parallel regions, where the configuration overhead might become a dominant factor?

3. Can the authors clarify the interaction with the base MESI coherence protocol? Specifically, how are potential write-after-read races handled if a follower thread issues a store to an address that is currently being prefetched via multicast for the group? Does the blocked prefetch request in the follower's private cache also stall subsequent stores to the same address until the multicast data arrives?

4. The Timeout Threshold appears to be a sensitive tuning parameter (Figure 19, page 10). Does this parameter require per-application tuning for optimal performance, or have the authors identified heuristics that would allow for a robust, system-wide default value?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes Software Prefetch Multicast (SPM), a software-hardware co-design to improve bandwidth efficiency in manycore processors by optimizing the handling of shared data. The core claim is that by exposing software-level knowledge of data sharers to the hardware, a more precise and timely multicast can be achieved, overcoming the limitations of prior hardware-only prediction or request coalescing schemes.

The authors identify three primary contributions as novel: 1. A new ISA extension in the form of a "sharer-exposed" prefetching instruction that carries a Group ID, and a corresponding software-hardware interface for configuring these sharer groups. 2. A microarchitecture centered on a "leader-follower" model, where a single designated "leader" thread issues the multicast-triggering prefetch on behalf of the entire group. 3. A dynamic leader-switching mechanism, based on timeouts, to adapt to thread execution variation and prevent stalls.

My analysis concludes that while the individual concepts (prefetching, multicasting, leader election) are not new in isolation, their specific synthesis into a coherent software-directed multicast framework represents a novel and meaningful contribution. However, the degree of this novelty is evolutionary, not revolutionary, and comes at the cost of significant mechanism complexity that warrants scrutiny.

Strengths

1. Novelty of the Core Abstraction: The central novel idea is the "sharer-exposed prefetch" instruction (Section 3.2, page 4). This fundamentally shifts the paradigm from hardware inferring sharers to software declaring them. This is a significant delta from the closest prior art, Push Multicast [12], which relies on historical sharer information stored in the directory. SPM uses a priori knowledge from the program structure, which is inherently more accurate for predictable sharing patterns and does not suffer from limitations like directory capacity or the need for silent eviction protocols. Similarly, it is a clear step beyond generic hints like Intel's MOVRS [14], which indicates data is "read-shared" but crucially does not identify the specific set of sharers. The SPM instruction’s Group ID provides this missing link.

2. Novel Solution to a Known Problem: The paper correctly identifies thread variation as a fundamental limiter for passive multicast/coalescing techniques (Section 2.3, page 3). The proposed dynamic leader-switching mechanism (Section 3.6, page 6) is a novel microarchitectural solution tailored specifically to their leader-follower model. While dynamic adaptation is a common design pattern, its application here—using follower timeouts to trigger a leader re-assignment at the LLC—is a new and well-motivated part of the overall design. It directly addresses a primary weakness of more static or predictive approaches.

3. Clear Articulation of the "Delta": The authors have done a commendable job of positioning their work against existing solutions. The introduction and related work sections (Section 1, page 1 and Section 5, page 11) clearly delineate the conceptual differences between SPM and techniques like GPU packet coalescing [22], Stream Floating [29], and especially Push Multicast [12]. This demonstrates a strong awareness of the prior art and helps isolate their specific novel claims.

Weaknesses

1. Evolutionary, Not Revolutionary, Novelty: The overall concept, while new in its specific implementation, can be viewed as the logical synthesis of several existing ideas. We have software prefetching [6], multicast for shared data [10], and software hints for hardware [14]. SPM combines these into a more powerful, explicit mechanism. The leader/follower model also has conceptual overlaps with helper threading for prefetching [20, 24], where one thread does work (prefetching) on behalf of others. While the SPM leader also performs application work, the functional similarity reduces the perceived novelty of this aspect of the design. The contribution is in the engineering of the combined system, not in a singular, groundbreaking new concept.

2. Significant Complexity for the Achieved Benefit: The proposed mechanism is substantially complex. It requires ISA extensions, new configuration-stage messages (config_req, config_rsp), and new state-holding structures in both the L2 private cache (Leader/Followers Lookup Map, Follower Waiting Table) and the shared LLC (Share Map) (Figures 9 and 10, page 5). This is in addition to the timeout counters, comparators, and logic for the dynamic leader-switching protocol. The evaluation shows geomean speedups of 1.28x-1.38x. While respectable, it is critical to question whether this level of performance gain justifies the introduction of such a multifaceted and invasive hardware mechanism. The novelty here comes with a high complexity tax.

3. The Software Oracle Assumption: The entire premise of SPM's novelty and effectiveness rests on the ability of the compiler or programmer to correctly and comprehensively identify sharer groups statically. The paper focuses on applications with regular, explicit sharing patterns. The novelty of the approach is less clear for applications with dynamic, input-dependent, or pointer-chasing sharing patterns where static analysis is intractable. The proposed mechanism provides no novel way to handle these cases, falling back to standard behavior.

Questions to Address In Rebuttal

1. Complexity Accounting vs. Prior Art: The authors critique Push Multicast [12] for requiring in-network filtering. However, SPM introduces a multi-step configuration protocol, three new tables across L2 and LLC, and a timeout/leader-switching mechanism. Could the authors provide a reasoned, apples-to-apples comparison of the hardware complexity (e.g., estimated storage overhead in KB, logic gate count) of SPM versus the in-network filtering and directory modifications required by Push Multicast?

2. Justification for Group IDs over Simpler Hints: The core ISA novelty is the Group ID. Could the authors elaborate on why a simpler mechanism would not suffice? For example, an enhanced prefetch_shared instruction broadcast from one core, with other cores using a lightweight hardware mechanism to snoop and suppress their own redundant prefetches for a short time window. Please defend why the explicit configuration and management of numbered Group IDs is essential and justifies its complexity over a less stateful design.

3. Overhead of Configuration: The configuration stage is presented as a prerequisite for the multicast operation. For programs with many fine-grained parallel regions, this configuration protocol (broadcast config_req from all sharers, LLC determination, multicast config_rsp) must be invoked repeatedly. At what frequency of parallel region invocation would the overhead of this novel configuration stage begin to negate the benefits of the subsequent multicast?

4. Robustness of the Leader/Follower Model: The paper mentions that context switches require wiping the sharer state and re-configuring (Section 3.7, page 6). This seems to be a significant vulnerability of the proposed stateful model. How does the performance of this novel mechanism degrade in a multi-programmed environment with frequent context switching, compared to a more stateless predictive scheme like Push Multicast?

Review Form:

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The paper proposes RICH, a hardware prefetcher designed for memory systems with high capacity and latency. RICH combines spatial prefetching across multiple region granularities (2 KB, 4 KB, 16 KB). To manage the metadata for larger regions, it employs a hierarchical storage mechanism, keeping frequently used patterns on-chip and offloading less frequent ones to main memory. The design uses a multi-offset trigger mechanism to improve accuracy for large regions and a priority-based arbitration scheme to select the best prefetch size. The authors evaluate RICH against several spatial prefetchers, claiming a 3.4% performance improvement over the state-of-the-art Bingo prefetcher in a conventional system, and an 8.3% improvement in a simulated high-latency system.

Strengths

1. The paper addresses a relevant and forward-looking problem: how to design prefetchers for future memory systems where capacity and bandwidth are plentiful, but latency is a major bottleneck.
2. The initial motivation for exploring multiple prefetch region sizes is well-founded, as demonstrated by the analysis in Figure 1 (page 3).
3. The evaluation is extensive, utilizing a wide range of benchmarks (SPEC06, SPEC17, Ligra, Parsec) and comparing against multiple relevant prior works (Bingo, PMP, SMS, SPP-PPF).

Weaknesses

The paper’s central claims rest on a complex design whose trade-offs are not rigorously analyzed or justified. The performance benefits appear modest relative to the introduced complexity and potential hidden costs.

1. Insufficient Analysis of Off-Chip Overheads: The core premise of RICH is to "store rich information in memory." However, the cost of this strategy is inadequately quantified. The paper fails to report the most critical overhead metric: the amount of main memory bandwidth consumed by its own metadata reads and writes. This traffic must compete with demand accesses and the prefetched data itself. The analysis in Figure 14 (page 11), which shows performance under varying total bandwidth, is insufficient as it doesn't isolate the contribution of this self-inflicted overhead. Without this data, the claim of "strategically consuming" bandwidth is unsubstantiated.

2. Unconvincing Latency Tolerance Argument for Off-Chip Metadata: The justification for moving 16 KB patterns off-chip hinges on Figure 5 (page 5), which claims that a 50ns additional latency results in less than 15% degradation in "prefetch opportunities." This analysis is flawed for two reasons:

   • The y-axis is labeled "Late Prefetches," which is not the same as lost opportunities and is not formally defined. A 15% increase in late prefetches could represent a significant performance impact.
   • The central argument of the paper is that RICH excels in high-latency systems (Section 5.3). However, the analysis in Figure 5 is not connected to the high-latency system evaluation in Figure 13. If system memory latency is already high (e.g., baseline + 120ns), the latency to fetch off-chip metadata will also be substantially higher, likely far exceeding the 50ns tested in Figure 5 and invalidating its conclusion.

3. Arbitrary Design Choices and Unjustified Heuristics: The design of RICH is replete with magic numbers and heuristics presented without sufficient justification.

   • Trigger Mechanism: Why were (PC, 5 offsets) for 16 KB and (PC, 3 offsets) for 4 KB chosen (Section 3.1)? Figure 3 (page 4) shows a clear trade-off between accuracy and coverage. The paper provides no evidence that these specific points are optimal or robust across workloads.
   • Arbitration Priority: The fixed priority scheme in the Region Arbitration unit (Section 4.2, Step P3) is presented as a given. Why is giving the 2 KB (PC, address) prefetch the highest priority optimal? This could potentially block a more beneficial, albeit slightly later, 16 KB prefetch. No alternative arbitration schemes are discussed or evaluated.
   • Prefetch Count Threshold: The threshold of 30 prefetches before offloading a 16 KB pattern to memory (Section 4.1) is stated to be based on "experiments," but no data or sensitivity analysis is provided to support this specific value (beyond the brief analysis in Figure 17).

4. Questionable Iso-Storage Comparison: The iso-storage comparison in Section 5.6 (page 11) aims to show RICH is more storage-efficient. However, the "Enhanced Bingo" baseline is constructed by increasing the PHT's associativity. This is not necessarily the most effective way to utilize a larger storage budget for Bingo; increasing the number of entries may have been more beneficial. This choice of a potentially weak baseline undermines the claim that RICH is better at converting storage to performance.

5. Mismatched Complexity and Benefit: In the conventional system, RICH provides a modest 3.4% geomean speedup over Bingo. This marginal gain comes at the cost of a significantly more complex design, involving three parallel lookup paths, multi-offset tracking FIFOs, a complex arbitration unit, and the entire machinery for managing on-chip/off-chip metadata. The engineering cost and potential for critical path elongation are non-trivial and are not justified by such a small improvement.

Questions to Address In Rebuttal

1. Please provide a quantitative breakdown of main memory bandwidth utilization. Specifically, what percentage of total bandwidth is consumed by RICH's off-chip metadata reads and writes across the evaluated workloads? How does this overhead traffic impact performance, especially in bandwidth-limited scenarios?

2. The paper argues for RICH's strength in high-latency systems (Figure 13, page 10). Please reconcile this with the off-chip metadata access analysis in Figure 5 (page 5). How does the performance of the off-chip metadata mechanism hold up when the baseline memory latency is already increased by 120ns?

3. Please provide a rigorous justification for the choice of 5 and 3 offsets for the 16 KB and 4 KB region triggers, respectively. A sensitivity analysis showing why these specific values are optimal across a range of workloads is required.

4. In the iso-storage comparison (Section 5.6, page 11), please justify why increasing the PHT associativity was chosen as the method to scale Bingo's storage budget, as opposed to other methods like increasing the number of entries.

5. Table 3 (page 6) shows that for some workloads (e.g., roms-294, roms-1070), the 200-entry on-chip PHT cache for 16 KB patterns has poor coverage (32.8% and 38.7%). What is the performance impact on these specific workloads, which must frequently stall for high-latency off-chip metadata accesses? Do these cases suffer a performance degradation compared to the baseline?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents RICH, a novel hardware prefetcher designed to address the growing challenge of high memory latency in modern and future computer systems. The core contribution is a philosophical shift in prefetcher design: instead of being constrained by scarce on-chip resources, RICH strategically leverages abundant off-chip memory capacity and bandwidth to store "rich" prefetching metadata. This enables more powerful prefetching techniques that would otherwise be infeasible.

Specifically, RICH makes two key contributions. First, it implements a multi-scale spatial prefetching mechanism that operates on 2 KB, 4 KB, and 16 KB regions, using a sophisticated multi-offset trigger and arbitration system to balance timeliness, coverage, and accuracy. Second, it introduces a hierarchical on-chip/off-chip metadata storage architecture. Latency-sensitive training and control data, along with patterns for small regions and a cache for large-region patterns, are kept on-chip. The bulk of the large-region (16 KB) patterns, which are shown to be more tolerant to retrieval latency, are offloaded to main memory. The evaluation demonstrates that RICH outperforms state-of-the-art spatial prefetchers like Bingo and PMP, with its performance advantage becoming more pronounced as memory latency increases, validating the paper's central thesis.

Strengths

1. Timely and Highly Relevant Thesis: The paper's fundamental premise is exceptionally well-aligned with dominant trends in the memory subsystem landscape. The authors correctly identify that technologies like CXL-based memory pooling, NVM, and even generational shifts in DDR standards are prioritizing capacity and bandwidth at the expense of latency (Section 1 and 2.1, page 1-2). RICH is not just another prefetcher; it is an architectural response to this macro trend. This makes the work significant and forward-looking.

2. Novel Architectural Pattern for Prefetching: The hierarchical on-chip/off-chip metadata storage is the paper's most compelling contribution. While temporal prefetchers have previously used off-chip memory to store long access streams, applying this concept to the metadata of a spatial prefetcher is a powerful idea. The analysis in Section 3.2 (page 5) that carefully partitions metadata based on latency tolerance, access frequency, and size is insightful and forms the foundation for a practical design. This demonstrates a thoughtful co-design between the algorithm and its physical implementation.

3. Effective Use of "Rich" Metadata: The paper successfully avoids the trap of simply scaling up existing designs. The multi-region prefetching strategy (Insight 1, page 3) and the use of multi-offset triggers to improve accuracy for large regions (Insight 2, page 4) are clever mechanisms that directly convert the availability of more metadata into tangible performance gains. The region arbitration logic (Section 4.2, page 7) elegantly balances the high-performance potential of large regions with the low misprediction cost of small ones.

4. Strong Supporting Evaluation: The experimental results, particularly the sensitivity studies, provide strong evidence for the authors' claims. The analysis of performance under increased memory latency (Figure 13, page 10) is crucial, as it directly validates that RICH is well-suited for the future systems it targets. Similarly, the performance breakdown analysis (Figure 15, page 11) effectively demonstrates that each component of the design—from the multi-offset trigger to the region arbitration—contributes meaningfully to the final result.

Weaknesses

1. Understated Relationship with Temporal Prefetching: While the authors correctly differentiate their work from temporal prefetchers like STMS (Section 2.3, page 3), they could strengthen their positioning by more deeply analyzing the conceptual parallels. The idea of using off-chip memory for prefetcher state is a shared principle. A more detailed discussion could highlight the unique challenges of applying this to spatial metadata (e.g., lack of sequentiality, pattern indexing, latency tolerance of pattern fetches) and thus better underscore the novelty of their architectural solution.

2. Limited Exploration of System-Level Implications: The proposal to dedicate a 128 KB region of main memory per core (Section 4.4, page 8) for prefetcher metadata introduces a new, architecturally-visible resource. The paper assumes a static allocation by the OS at boot time (Section 4.1, page 7), which is a practical starting point. However, this opens up a host of system-level questions that are not addressed:

   • Virtualization: How would a hypervisor manage and partition this off-chip PHT space for multiple guest VMs?
   • Security: Could this shared memory structure become a side-channel for inferring memory access patterns between processes or VMs?
   • Dynamic Allocation: Could the OS dynamically resize or page this metadata space based on application needs? Acknowledging these issues would provide a more complete picture of how RICH would integrate into a full system stack.

3. Design and Verification Complexity: The proposed architecture, with its concurrent lookups for three different region sizes, complex arbitration logic, and asynchronous off-chip metadata management, represents a significant increase in design complexity compared to traditional prefetchers. While the performance gains appear to justify this, a brief discussion on the practical challenges of verification and timing closure for such a unit would add a valuable layer of pragmatism to the proposal.

Questions to Address In Rebuttal

1. Could the authors elaborate on the key architectural and algorithmic differences that make offloading spatial pattern metadata to main memory a distinct and more challenging problem than offloading the access stream histories used by temporal prefetchers?

2. The paper focuses on prefetching across 4 KB page boundaries by using virtual addresses. A major trend in systems is the use of huge pages (e.g., 2 MB, 1 GB) to reduce TLB pressure. How might the RICH architecture leverage knowledge of huge pages? It seems that confirming an access is within a 2 MB huge page could make the 16 KB region prefetching even more aggressive and accurate, potentially unlocking further performance.

3. Regarding the off-chip metadata store, have the authors considered the implications for multi-socket systems connected via CXL? If a thread migrates to a core on a different socket, would its RICH metadata need to be migrated as well? Does this present an opportunity for a shared, system-wide metadata store, or would per-core locality remain paramount?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors propose RICH, a hardware prefetcher designed for future memory systems characterized by high capacity and high latency. The central thesis is to trade abundant memory capacity and bandwidth to hide latency by storing a large amount of prefetching metadata. The paper claims novelty through a combination of three core ideas: (1) a multi-scale spatial prefetcher that simultaneously tracks 2 KB, 4 KB, and 16 KB regions; (2) a "multi-offset" trigger mechanism that uses a sequence of memory accesses, rather than a single access, to validate and trigger prefetches for larger regions; and (3) a hierarchical storage system that keeps latency-sensitive and frequently used metadata on-chip while offloading the large, less-frequently-used 16 KB region patterns to main memory. The authors demonstrate that this combination allows RICH to outperform state-of-the-art spatial prefetchers, particularly in high-latency scenarios.

Strengths

The primary strength of this work lies in its synthesis of existing concepts into a genuinely novel architecture for spatial prefetching. While individual components may have conceptual predecessors, their integration here is new and well-motivated.

1. The Multi-Offset Trigger Mechanism: The most significant novel contribution is the use of multiple access offsets to trigger a spatial pattern prefetch (Section 3, Insight 2, page 4). Conventional spatial prefetchers like SMS [11] or Bingo [12] use a single (PC, offset) or (PC, address) pair. By requiring a sequence of offsets (e.g., 5 for a 16 KB region), RICH creates a more robust trigger that significantly improves accuracy for large-region prefetching. This mechanism is conceptually novel as it borrows a validation principle often seen in temporal/stream prefetching (i.e., confirming a sequence) and applies it to trigger a purely spatial pattern lookup.

2. Novel Application of Hierarchical Storage: The idea of offloading prefetcher metadata to main memory is not new; it is the cornerstone of temporal prefetchers like STMS [16] and MISB [15], which the authors correctly cite. However, the novelty in RICH is the specific application of this concept to bit-vector-based spatial patterns and the design of a coherent tiered system around it. The on-chip 16 KB PHT acts as a cache for the off-chip main PHT, complete with mechanisms like the Valid Map and a prefetch count threshold to manage the off-chip traffic. This is a novel architectural pattern for spatial prefetchers.

3. Coherent Multi-Scale Architecture: While the observation that different workloads benefit from different region sizes is not new, the creation of a prefetcher that concurrently tracks, triggers, and arbitrates between multiple fixed region sizes is a novel architectural choice. The arbitration logic, which prioritizes based on accuracy and misprediction cost (Figure 4, page 5) and de-duplicates requests (Section 4.2, Step P3, page 7), represents a clear and novel design for coordinating these different scales.

Weaknesses

The paper's claims of novelty must be carefully contextualized against prior art, and the justification for the substantial increase in design complexity needs to be robust.

1. Incrementalism of Individual Concepts: While the synthesis is novel, some of the foundational ideas are well-established. The core idea of trading memory capacity for performance is a fundamental principle in computer architecture. Off-chip metadata storage has been explored extensively in the temporal prefetching domain. The paper would be stronger if it more explicitly delineated the novel mechanisms required for their spatial implementation from the known general concept of off-loading.

2. Significant Increase in Design Complexity: The proposed RICH architecture is substantially more complex than its predecessors. It effectively runs three parallel training and lookup pipelines for the different region sizes, which feed into a non-trivial Region Arbitration unit (Figure 8, page 8). The management of the off-chip PHT adds further control logic. The 3.4% performance gain over Bingo in a conventional system seems marginal given this complexity. While the 8.3% gain in a high-latency system is more compelling, it is crucial to question if a simpler mechanism could have achieved similar results.

3. Under-explored Implementation Overheads: The paper briefly mentions on page 6 that "virtual addresses are used to ensure pattern continuity" for 16 KB regions that cross 4 KB page boundaries. This is a critical implementation detail with non-trivial consequences. It implies that every prefetch request generated for a 16 KB region might require address translation, potentially putting pressure on the TLB. This overhead is not quantified or discussed in detail, yet it is a direct consequence of a key novel feature (large region support).

Questions to Address In Rebuttal

1. On the Multi-Offset Trigger: The selection of the number of offsets for the triggers (5 for 16 KB, 3 for 4 KB) appears to be an empirical choice. Could the authors provide more insight into the sensitivity of the accuracy/coverage trade-off to this hyperparameter? Is there a principled reason for these specific values, or were they simply the optimal points found in your design space exploration?

2. Distinction from Prior Off-Chip Metadata Schemes: Please elaborate on the key mechanistic differences between RICH's off-chip management for spatial patterns and the scheme used by a temporal prefetcher like STMS [16]. Beyond the difference in metadata content (bit-vectors vs. address streams), what novel challenges did you face and solve? For instance, how critical is the prefetch count thresholding mechanism for managing off-chip bandwidth, and is this distinct from prior work?

3. Quantifying Control Logic Complexity: The paper provides a clear breakdown of storage overheads (Table 4, page 8). However, it does not address the area and power cost of the additional control logic, particularly the three parallel lookup units and the Region Arbitration logic. Can the authors provide an estimate of this overhead to give a more complete picture of the prefetcher's cost?

4. Impact of Virtual Address Translation: Could you clarify the performance impact of using virtual addresses for 16 KB prefetching? How frequently do these prefetches require new TLB lookups, and what is the potential for this to become a performance bottleneck, especially in multi-core scenarios where TLB pressure is higher?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors identify that LLM inference on modern CPUs is bottlenecked by the software-based decompression of quantized and sparsified weight matrices, which precedes their processing by hardware GeMM engines like Intel's TMUL. To address this, they first propose a 3D performance model, "Roof-Surface," to characterize the interaction between memory bandwidth, vector decompression units, and matrix execution units. Based on insights from this model, they propose DECA, a near-core hardware accelerator to offload decompression. Finally, they introduce a new ISA extension, TEPL, to enable efficient, low-latency, out-of-order invocation of DECA. Their simulation results claim up to a 4x speedup on compressed GeMM kernels and a 1.6x-1.9x reduction in next-token latency for large LLMs compared to an optimized software baseline.

Strengths

1. The paper addresses a relevant and timely problem: the performance bottleneck of weight decompression for LLM inference on CPUs.
2. The use of a state-of-the-art software baseline (Intel's libxsmm kernels) provides a strong point of comparison for the proposed hardware solution.
3. The fundamental idea of decomposing the performance problem into three interacting domains (memory, vector, matrix) is a logical approach to bottleneck analysis.

Weaknesses

My primary concerns with this submission revolve around the foundational model's validity, the justification for costly ISA extensions, and the fidelity of the evaluation methodology.

1. The Roof-Surface Model is an Oversimplification that Ignores First-Order Effects: The central thesis rests on the Roof-Surface model providing "clear insights." However, the model is an idealized representation of pipelined throughputs that systematically ignores crucial, real-world system effects. In Table 2 (page 5), the authors' own data reveals significant discrepancies between the model's predictions (R-S) and measured reality (Real). For instance, for BF16_50%, the model predicts 11.8 TFLOPS while the real performance is 9.2 TFLOPS—a 28% overestimation. The authors dismiss this as "non-plotted factors such as memory latency or cache latency." These are not minor factors; they are fundamental to system performance. A model that cannot account for over 25% of the performance limitation is not a sound foundation upon which to base microarchitectural design decisions. The claims of the model's explanatory power are therefore overstated.

2. Insufficient Justification for a New ISA Extension: The introduction of the TEPL instruction is a significant architectural modification that sets a very high bar for justification. The authors motivate TEPL by comparing it to a store-based invocation method that requires explicit memory fences (Figure 8, page 7), which is known to be inefficient for fine-grained core-accelerator communication. This comparison appears to be against a strawman baseline. The paper fails to explore or evaluate more sophisticated software-only or architectural mechanisms that could mitigate this latency without requiring bespoke ISA changes, such as optimized polling, doorbell registers with better scheduler integration, or advanced prefetching schemes. The conclusion that TEPL is necessary is therefore premature and insufficiently supported.

3. Ambiguous Simulation Fidelity: The evaluation is conducted using an "in-house simulator based on Sniper" (Section 7.1, page 9). Sniper is an interval-based simulator, which raises questions about its cycle-level accuracy for modeling the complex out-of-order execution and pipeline interactions that are central to the benefits claimed for TEPL. The paper provides no details on how the simulator models the TEPL Queue, speculative instruction issue, or the squash signaling between the core and DECA. Without rigorous validation or more detailed explanation of the simulation infrastructure, the results concerning TEPL's ability to "hide communication latency" remain unsubstantiated.

4. Comparison Against Unrealistic Alternatives: In Figure 14 (page 11), the authors compare DECA against "More AVX Units" and "Wider AVX Units." This comparison is not based on a realistic microarchitectural model, but on an abstract thought experiment of "removing the dynamic instructions from 3 out of 4 iterations." This completely sidesteps the immense microarchitectural challenges (e.g., L1 cache port scaling, register file bandwidth, instruction issue width) that make such a design infeasible, a point the authors themselves concede in Section 4. This comparison is therefore misleading, as it pits a detailed simulation of the proposed accelerator against a non-physical, idealized abstraction of an alternative.

Questions to Address In Rebuttal

1. Regarding the Roof-Surface model: How can the model be considered a reliable guide for microarchitecture design when it fails to account for system effects (latency, caching) that result in a ~20-30% performance deviation, as shown in your own results (Table 2)? Please justify why these first-order effects were excluded from your "3D performance model."

2. Regarding the TEPL ISA extension: Before proposing a new instruction, what alternative software-only or existing architectural mechanisms for low-latency core-accelerator communication were evaluated and why were they deemed insufficient? Please provide data to show that a fenced, store-based invocation is the strongest possible baseline short of new ISA.

3. Regarding simulation: Can you provide specific details on how your Sniper-based simulator models the out-of-order execution, speculative issue, and retirement of TEPL instructions? Specifically, how are structural hazards on the DECA loaders and dependencies with the core's ROB modeled to ensure the performance claims are not an artifact of an overly optimistic simulation environment?

4. Regarding the DSE in Section 8.2: The selection of the "best" {W,L} pair for DECA appears to be optimized to solve the problem as defined by your own idealized Roof-Surface model. Given the model's discussed inaccuracies, how can you be certain that this configuration is truly optimal for a real system where other factors (e.g., latency) are at play?

5. The abstract claims a next-token generation time reduction of "1.6x-1.9x." However, in Table 6 (page 12), the Llama2-70B result for N=16 and Q8_30% shows a speedup of 116.6 / 75.7 ≈ 1.54x, which is outside the claimed range. Please clarify the precise set of results used to establish this range.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a holistic, model-driven approach to solving the weight decompression bottleneck in Large Language Model (LLM) inference on CPUs. The authors identify that with high-bandwidth memory (HBM) and powerful in-core matrix engines (like Intel's AMX), the software-based decompression of compressed (quantized and sparsified) weights becomes the new performance limiter.

The core contribution is threefold: 1. A Novel Performance Model: The "Roof-Surface," a 3D extension of the classic Roofline model, which elegantly visualizes the performance bounds imposed by the three key interacting resources: memory bandwidth, vector unit throughput, and matrix unit throughput. 2. A Near-Core Accelerator: DECA, a specialized hardware unit designed to offload dequantization and de-sparsification from the CPU's vector units. The design of DECA is explicitly motivated and guided by insights from the Roof-Surface model. 3. An ISA Extension: The Tile External Preprocess & Load (TEPL) instruction, which enables efficient, out-of-order, low-latency communication between the CPU core and the DECA accelerator, effectively hiding communication overheads by overlapping them with computation.

The evaluation, conducted on a simulated 56-core server, demonstrates that this co-designed system can accelerate compressed matrix multiplication kernels by up to 4x over optimized software and reduce end-to-end next-token latency for large LLMs like Llama2-70B by 1.6-1.9x.

Strengths

The primary strength of this paper lies in its principled, top-to-bottom systems thinking. It is a superb example of how insightful performance modeling can directly inform and justify hardware and ISA design.

1. The Roof-Surface Model: This is the paper's most significant and enduring contribution. The classic 2D Roofline model is a cornerstone of performance analysis, but it falls short when a third resource becomes a first-order performance limiter. The authors' extension to a 3D "surface" (Section 4, page 5) is both intuitive and powerful. It provides a clear, quantitative language to explain why the software-only approach fails on HBM-equipped systems (it becomes "VEC-bound") and precisely what level of acceleration is needed to overcome this bottleneck without over-provisioning hardware. This model has the potential for broad applicability beyond this specific problem.

2. Holistic Co-Design: The paper does not simply propose an accelerator in a vacuum. Instead, it presents a complete solution. The Roof-Surface model identifies the problem, the DECA accelerator provides the hardware muscle, and the TEPL instruction provides the necessary low-level communication primitive to make the hardware effective. This tight integration of modeling, microarchitecture, and ISA is the hallmark of a high-quality computer architecture paper. The motivation for TEPL, contrasting the inefficient store-based invocation (Figure 8, page 7) with the streamlined out-of-order TEPL approach (Figure 9, page 7), is particularly compelling.

3. Timeliness and Practical Impact: The work addresses a critical, real-world bottleneck for a major workload. As CPUs continue to integrate HBM and more powerful matrix engines to stay competitive with GPUs for AI inference, the "software glue" problem will only become more acute. This paper provides a clear, well-reasoned blueprint for how CPU architects can solve this problem, potentially making high-performance, low-latency LLM inference more accessible and cost-effective on general-purpose servers. The comparison in Table 7 (page 12), which shows the proposed system closing a significant portion of the performance gap with a contemporary GPU, underscores this practical relevance.

4. Excellent Contextualization: The authors do a commendable job of situating their work within the broader landscape of in-core and near-core acceleration. The taxonomy presented in Table 8 (page 13) is particularly useful, clearly differentiating DECA from prior work by its unique combination of support for flexible compression, cooperation with matrix units (rather than replacing them), and fine-grained, low-overhead interleaving with the core.

Weaknesses

The weaknesses of the paper are minor and mostly relate to the boundaries of its exploration rather than fundamental flaws in the core idea.

1. Limited Scope of Evaluated Compression Schemes: The evaluation focuses on established but relatively simple schemes (unstructured sparsity, BF8, MXFP4). The field of model compression is evolving rapidly, with more complex methods like product quantization, non-uniform quantization, and structured sparsity patterns gaining traction. While the LUT-based design of DECA suggests flexibility, it is not immediately clear how it would handle schemes that require more complex arithmetic than a simple lookup and scaling operation.

2. Deeper Architectural Implications of TEPL: The TEPL instruction is well-motivated, but its introduction has ripple effects on the core's front-end, scheduler, and ROB. While the paper describes the high-level mechanism (TEPL Queue, speculative issue), a more in-depth discussion of the complexity and area/power cost of these changes to a modern out-of-order core would strengthen the proposal.

3. Unexplored Generality of the Roof-Surface Model: The authors correctly claim in Section 9.2 (page 13) that the Roof-Surface model can be generalized. This is one of the most exciting aspects of the work. However, this claim would be significantly bolstered by briefly applying the model to another, different problem domain to demonstrate its broader utility beyond LLM decompression (e.g., a bioinformatics pipeline, a graphics rendering stage, or a data analytics query).

Questions to Address In Rebuttal

1. The LUT-based dequantization in DECA is flexible for existing formats. How would the architecture adapt to future, more complex schemes that may not be easily captured by a lookup table, such as those requiring on-the-fly arithmetic to reconstruct weights? Does the DECA pipeline have extensibility points for such cases?

2. The paper's strongest contribution is arguably the Roof-Surface model. Can the authors provide another concrete example of a workload with a three-way (or more) dependency chain where their generalized modeling approach from Section 9.2 would yield non-obvious insights that a traditional Roofline analysis would miss?

3. Regarding the TEPL ISA extension: What alternative, less invasive core-accelerator communication mechanisms were considered? For instance, could a system of memory-mapped FIFOs combined with intelligent core-side prefetching and synchronization instructions achieve a similar level of latency hiding without the complexity of a new, blocking, out-of-order instruction? What was the deciding factor that made TEPL the superior choice?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents three distinct but interconnected contributions aimed at accelerating compressed Large Language Model (LLM) inference on CPUs equipped with in-core matrix engines. The core problem—that software-based decompression of quantized/sparsified weights becomes a bottleneck—is well-established. The authors' proposed solution consists of:

1. A novel performance analysis framework called the Roof-Surface model, a 3D extension of the traditional 2D roofline model. This model aims to correctly identify bottlenecks in a pipelined system involving memory, vector units, and matrix units.
2. A near-core decompression accelerator named DECA, which offloads de-quantization and de-sparsification from the CPU's vector units to feed the in-core matrix engine.
3. A new ISA extension, Tile External Preprocess and Load (TEPL), designed for efficient, out-of-order invocation of the DECA accelerator, hiding communication latency.

My analysis concludes that while the constituent hardware components of the accelerator are based on known techniques, the specific combination of the performance model, the cooperative accelerator architecture, and the tight ISA integration represents a significant and novel system-level contribution. The novelty lies not in a single isolated idea, but in the synergistic design where each component is justified by and enhances the others.

Strengths

The primary strength of this paper is the introduction of multiple layers of novelty that build upon each other cohesively.

1. Novelty of the Performance Model: The Roof-Surface model (Section 4, page 5) is a genuinely novel extension of performance modeling for a specific, important class of problems. While multi-dimensional performance models are not unheard of, the specific formalization of a three-stage, pipelined dependency (Memory -> Vector -> Matrix) with corresponding arithmetic intensities (AIXM and AIXV) is new. It correctly identifies the vector-decompression bottleneck where the traditional 2D roofline model fails (as shown in Figure 3b, page 4). This model is not just a theoretical exercise; the authors effectively use it to motivate the need for an accelerator and to perform a quantitative design space exploration for it (Section 8.2, page 11).

2. Novelty in Architectural Division of Labor: The concept of a near-core accelerator is not new. However, DECA's novelty lies in its specific role as a cooperative pre-processing unit for an existing, powerful in-core engine (the TMUL). Much prior work on in/near-core accelerators for sparse workloads (e.g., VEGETA [39], RASA [40]) proposes replacing or heavily augmenting the core's compute units to handle compressed formats directly. In contrast, DECA decouples the decompression task from the GeMM task, offloading the former to a specialized unit while leaving the latter to the highly-optimized TMUL. This specific architectural partitioning is a novel approach to the problem.

3. Novelty of the ISA Integration: The TEPL instruction (Section 5.3, page 7) is a sophisticated and novel solution to the command-and-control problem for tightly-coupled accelerators. The naive approach using memory-mapped stores and loads is correctly identified as inefficient due to serialization and exposed latency. TEPL's design as a single instruction that both triggers the accelerator and receives the result, while being fully integrated into the core's out-of-order engine (using renaming and speculative execution), is a clean and powerful abstraction. This goes significantly beyond the prior art of loosely-coupled accelerators or those requiring no core modifications (e.g., SPADE [19], as noted in Table 8, page 13). The integration is the key innovation here.

Weaknesses

From a novelty perspective, the weaknesses are minor and relate to the granularity of the claims.

1. Constituent Components are Not Fundamentally New: The microarchitecture of the DECA pipeline itself (Section 6, page 8) is a synthesis of well-understood components. It uses Look-Up Tables (LUTs) for dequantization, POPCNT and Parallel Prefix Sum circuits for bitmask processing, and a crossbar for expansion. None of these blocks are, in isolation, novel inventions. The paper's novelty claim rests on their specific arrangement and dimensioning, which is guided by the Roof-Surface model. This is a system-level novelty, not a circuit-level one.

2. Limited Exploration of Non-ISA Alternatives: The paper makes a strong case for TEPL by demonstrating the flaws in a simple store-based invocation mechanism. However, the design space of command-and-control without new instructions is broader. For example, a system with dedicated command queues managed by the accelerator, which the core polls, could potentially offer better performance than the baseline store-with-fence approach. While I suspect the authors' TEPL solution is superior, a more thorough dismissal of advanced non-ISA-modifying alternatives would strengthen the novelty claim of requiring an ISA extension.

Questions to Address In Rebuttal

1. The paper's related work mentions NVIDIA's TMA [52] as a mechanism for supplying data to Tensor Cores and suggests augmenting it with DECA-like features could be an interesting direction (Section 9.1, page 13). Could the authors more precisely articulate the novel "delta" between the proposed TEPL instruction and the existing TMA mechanism? TMA is also, in essence, an accelerator for managing tile movement. How is TEPL's tight integration with the CPU's speculative, out-of-order core fundamentally different from the integration of TMA within an SM?

2. The generality of the Roof-Surface model is briefly discussed (Section 9.2, page 13) by proposing a generalized equation (3). Beyond this abstract formulation, could the authors provide one concrete example of another existing, real-world architecture or application (outside of CPU LLM decompression) where this 3-domain (or n-domain) pipelined model would provide insights that a traditional 2D roofline model would miss?

3. Regarding the TEPL design, was an alternative that relied on a dedicated, hardware-managed command FIFO between the core and DECA considered? Such a design might avoid the full complexity of a new, renamed instruction class while still allowing for out-of-order dispatch and avoiding memory fences. What would be the performance and complexity trade-offs of such a design compared to TEPL?

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents StreamTensor, a compiler framework designed to automate the generation of stream-based dataflow accelerators for LLMs from PyTorch models. The core contributions are an "iterative tensor" (itensor) type system for encoding stream layouts, a hierarchical design space exploration methodology, and an LP-based algorithm for FIFO sizing. The authors evaluate their framework on an FPGA, claiming significant latency and energy efficiency improvements over prior FPGA-based works and contemporary GPUs.

However, the work is undermined by questionable experimental comparisons, unsubstantiated claims regarding the systematic nature of its optimizations, and a concerning lack of critical hardware implementation details. While the proposed abstractions are interesting, the evidence provided is insufficient to validate the claimed performance superiority.

Strengths

1. Formalism for Stream Layouts: The itensor type system (Section 3.1, page 3) provides a structured and verifiable way to represent streamed tensor data. The ability to analytically infer the need for and the minimum size of a layout converter (Algorithm 1, page 9) based on type information is a sound concept.
2. Analytical FIFO Sizing: The formulation of the FIFO sizing problem as a linear programming task (Section 5.3.4, page 11) is a clear and defensible analytical contribution. Modeling token production/consumption with piecewise linear functions (Figure 8, page 10) provides a formal basis for optimizing inter-kernel delays.
3. End-to-End Automation: The demonstrated pipeline from a high-level model in PyTorch down to a hardware bitstream (Figure 4, page 4) represents a substantial engineering effort and addresses a key productivity challenge in hardware acceleration.

Weaknesses

1. Fundamentally Flawed Experimental Comparisons: The central claims of performance superiority are built on unsound comparisons.

   • Quantization Mismatch: The authors' implementation uses a W4A8 quantization scheme. However, the primary FPGA baseline, DFX [29], uses FP16 (Table 6, page 12). A comparison across such drastically different numerical precisions is invalid. W4A8 designs are inherently smaller and faster, but this is an advantage of the quantization scheme, not necessarily the compiler framework. The performance gains cannot be cleanly attributed to StreamTensor's contributions.
   • Hardware Mismatch: The authors use an AMD U55C FPGA, while both Allo [15] and DFX [29] use the U280. While from the same family, these are different parts with different resource counts and characteristics, further confounding the comparison.
   • Misleading GPU Comparison: The results in Table 5 (page 12) show that for Time-To-First-Token (TTFT), the A100 GPU is between 3.97x and 31.99x faster. The authors focus on total latency, but for many LLM applications (e.g., interactive chat), TTFT is the critical metric. Framing the overall result as a win obscures a significant performance deficit in a key area.

2. Missing Essential Hardware Metrics: For a paper proposing an FPGA accelerator framework, the complete omission of post-place-and-route resource utilization data (LUTs, FFs, BRAM, DSPs) is a critical flaw. Without this data, it is impossible to assess the actual efficiency of the generated designs. The memory reduction shown in Figure 10a (page 13) is only for intermediate results and offers no insight into the total on-chip resource cost, which is essential for judging feasibility and scalability.

3. Overstated Claims of "Systematic Exploration": The paper repeatedly claims to "systematically explore" the design space (Abstract, Section 1.4). However, the methods described in Section 5.1 (page 8) are a collection of heuristics: "naive tiling," "intensity-aware algorithm" for unrolling, and a "heuristic that moves reduction loops outward" for permutation. These are reasonable heuristics, but they do not constitute a systematic exploration. The term implies a more exhaustive or provably optimal search, which is not what is being performed.

4. Fragile Assumptions in FIFO Sizing Model: The LP model for FIFO sizing relies on static kernel latencies obtained from profiling (Section 5.3.1, page 10). This assumes a deterministic execution environment. The model's sensitivity to deviations between profiled and actual run-time behavior is not analyzed. Furthermore, the claim that the "memory utilization of stream FIFOs is negligible" (Section 5.3.4, page 11) is a strong assertion that is not backed by data. In a complex graph with hundreds of inter-kernel connections, the aggregate BRAM usage of these FIFOs could become significant.

5. Weak Handling of Dynamic Control Flow: The framework's approach to dynamicism (Section 5.3.5, page 11) is to either fall back to host execution or rely on "shape hints" to bound dynamic tensors. This is not a solution but rather an avoidance of the core challenge of compiling dynamic workloads to a static dataflow architecture. This severely limits the practical applicability of the framework to models with any data-dependent control flow.

Questions to Address In Rebuttal

1. Please provide a justification for comparing your W4A8 implementation against an FP16 baseline (DFX). How can the performance gains be attributed to your compiler framework rather than the fundamentally less complex arithmetic of the 4-bit quantization scheme?
2. Provide detailed post-place-and-route resource utilization reports (LUTs, BRAMs, DSPs, etc.) for each of the evaluated LLM designs on the U55C FPGA. How close are these designs to the resource limits of the target device?
3. Please reconcile the claim of "systematically exploring" the design space with the described use of separate, non-exhaustive heuristics for tiling, unrolling, and permutation.
4. What is the performance degradation of your FIFO sizing solution if the kernel latencies measured during profiling differ from their real runtime values by 10%, 20%, or 50% due to runtime variance?
5. Provide data to support the claim that FIFO memory utilization is "negligible." For the most complex model evaluated (e.g., Llama), what is the total on-chip BRAM consumed by all stream FIFOs combined, and what percentage of the total available BRAM does this represent?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents StreamTensor, an end-to-end compiler framework designed to automate the mapping of PyTorch-based Large Language Models (LLMs) onto stream-based dataflow accelerators, with a specific evaluation on FPGAs. The central challenge addressed is the immense difficulty and error-prone nature of manually designing efficient dataflow hardware, particularly in managing inter-kernel data streaming to overcome memory bottlenecks.

The authors' core contribution is the introduction of a novel iterative tensor (itensor) type system. This abstraction is the linchpin of their entire framework. By explicitly encoding the layout, access pattern, and iteration space of a data stream, itensor elevates the compiler's understanding from a simple memory-mapped tensor to a structured, flowing data entity. This formalization enables a series of powerful, automated optimizations that were previously ad-hoc or intractable: seamless kernel fusion, automatic generation of minimal-cost layout converters, and systematic resource allocation. The paper demonstrates the effectiveness of this approach by achieving significant latency and energy efficiency improvements over state-of-the-art FPGA solutions and competitive GPUs for LLM inference.

Strengths

The true strength of this paper lies in its synthesis of ideas from compiler theory, high-level synthesis, and computer architecture to create a cohesive and powerful automation framework.

1. The itensor as a Unifying Abstraction: The most significant contribution is the itensor type system (Section 3.1, page 3). For decades, compilers for spatial and dataflow architectures have struggled to bridge the semantic gap between imperative code (or static dataflow graphs) and the physical reality of streaming data. Traditional tensor types represent a block of memory; itensor represents a protocol for accessing that data over time. This is a profound and elegant conceptual leap. It provides the formal underpinning necessary for the compiler to reason about stream compatibility, a problem that has historically plagued HLS tools and required manual intervention. This is what allows StreamTensor to confidently fuse any two kernels, inserting a provably correct and minimal converter if their stream protocols (itensor types) do not match.

2. Systematic Exploration of the Design Space: The paper wisely decomposes the notoriously complex accelerator design problem into a hierarchy of three distinct but interconnected spaces: Linalg Tiling, Kernel Fusion, and Resource Allocation (Figure 4, page 4). This structured approach transforms what is often an unmanageable, holistic design challenge into a series of more constrained, solvable optimization problems. For instance, the token behavior model and subsequent LP formulation for FIFO sizing (Section 5.3, page 10) is an excellent example of applying formal methods to a specific sub-problem (Pitfall 4) that is often solved with heuristics or over-provisioning. This brings a much-needed rigor to the field.

3. Bridging the Gap Between AI Frameworks and Dataflow Hardware: This work sits at a critical intersection. On one side, you have the immense productivity of frameworks like PyTorch. On the other, you have the potential performance and efficiency of dataflow architectures like FPGAs, CGRAs (e.g., AMD Versal [24]), and custom DSAs (e.g., SambaNova [43], Groq [1]). The bridge between them has been a rickety, manual, and expert-driven process. StreamTensor represents one of the most serious and complete attempts to build a robust, automated highway. By starting from a high-level model and generating hardware, it lowers the barrier to entry for utilizing these powerful but esoteric architectures.

4. Excellent Problem Motivation and Positioning: The authors do a superb job in Section 1.3 ("Pitfalls," page 2) of articulating the precise, thorny issues that have limited prior work. They correctly identify inter-kernel correlations, memory management, fusion compatibility, and FIFO sizing as the key hurdles. Their entire framework is then built to systematically knock down each of these barriers. This clear problem-solution mapping makes the paper's contributions easy to understand and appreciate.

Weaknesses

The weaknesses of the paper are primarily related to its current scope and the assumptions it makes, which are understandable for a pioneering work but important to acknowledge.

1. Hardware Abstraction and Portability: While built on the general MLIR framework, the implementation and evaluation are tightly coupled to a specific HLS flow (Vitis for AMD FPGAs). The true potential of a framework like StreamTensor is its ability to target a class of dataflow architectures. It is not yet clear how the concepts, particularly the resource allocation and cost models (e.g., fusion cost in Algorithm 2, page 9), would translate to a more structured CGRA with a dedicated network-on-chip versus the "soft" logic of an FPGA. This limits the generality of the claims, though the future work section (Section 8, page 14) rightly identifies this as a key next step.

2. Handling of Dynamicism: The framework's strength lies in analyzing and optimizing statically determined dataflow graphs. The handling of dynamic behavior (data-dependent control flow, dynamic tensor shapes) is pragmatically offloaded to the host CPU (Section 5.3.5, page 11). This is a common and reasonable strategy, but it sidesteps the core challenge that many real-world applications, beyond the autoregressive decoding of LLMs, possess. The framework's performance is predicated on a largely static view of the computation.

3. Scalability to Multi-Device Systems: The current model for kernel fusion and partitioning is implicitly single-chip. The paper sets the maximum fusion cost Cmax to the on-chip memory of a single FPGA. However, deploying large LLMs requires partitioning across multiple accelerators. While the paper notes this is out of scope, the lack of a clear story for multi-chip partitioning and communication scheduling is a major limitation for practical, large-scale deployment. The itensor concept could potentially be extended to describe inter-chip streams, but this is a non-trivial research problem.

Questions to Address In Rebuttal

The authors are encouraged to use the rebuttal to comment on the broader implications and future trajectory of this work.

1. Extensibility of itensor: The itensor type system is beautifully suited for the dense, regular streaming patterns found in LLM transformers. How would this abstraction need to evolve to capture more complex or irregular dataflow patterns, such as those in Graph Neural Networks (GNNs) or models with heavy use of sparse tensors? Could it, for example, encode streams of indices and values separately and describe their relationship?

2. Path to Architectural Portability: Beyond the mention in future work, could you elaborate on the key changes required to retarget StreamTensor from an FPGA HLS backend to a more structured dataflow architecture like a CGRA or a dataflow ASIC? What are the primary architectural parameters that would need to be exposed to the compiler's resource allocation and cost models?

3. Resilience to Inaccurate Profiling: The LP-based FIFO sizing model (Section 5.3.4) relies on kernel latencies and throughputs obtained from a profiling step. In real systems, these values can fluctuate due to data-dependent execution paths, thermal management, or resource contention. How sensitive is the generated solution to inaccuracies in these profiled values? Is there a path toward a hybrid compile-time/run-time approach where buffer sizes could be adapted based on observed behavior?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents StreamTensor, a compiler framework designed to automate the generation of stream-based dataflow accelerators for LLMs, starting from a PyTorch model. The authors identify several key pitfalls in existing dataflow design paradigms, including inter-kernel correlation, external memory management, and FIFO sizing.

The central claims of novelty appear to be twofold: 1. The introduction of an "iterative tensor" (itensor) type system (Section 3.1, page 3), which explicitly encodes the temporal streaming layout of a tensor through an iteration space and an affine map. This type system is used to enable automated kernel fusion, stream layout converter generation, and type-based verification. 2. A piecewise linear function-based model for token behavior (Section 5.3, page 10) that captures both transient and steady-state dynamics of dataflow kernels. This model is used to formulate FIFO sizing as a linear programming (LP) problem, providing an analytical solution to avoid deadlocks and minimize resource utilization.

The paper builds an end-to-end compilation pipeline around these core ideas, transforming high-level Linalg IR into a dataflow IR and ultimately to hardware.

Strengths

The primary strength of this work lies in the conceptual novelty of its core abstractions, which address long-standing challenges in dataflow compilation with new formalisms.

1. Novelty of the itensor Type System: The core contribution, the itensor type, is genuinely novel. While prior tensor compilers and scheduling languages (e.g., Halide [46], TVM [16]) provide mechanisms to describe tiled and permuted access patterns procedurally through a schedule, StreamTensor elevates this description to a first-class, declarative type. This is a significant conceptual shift. By encoding the stream's access pattern directly into the type, the compiler can perform static verification of stream compatibility between a producer and consumer (as illustrated in Figure 5, page 4). This is a more powerful and scalable approach than relying on procedural analysis of two separate kernel schedules. The ability to analytically derive the minimal required stream layout converter and its buffer size (Algorithm 1, page 9) directly from the type mismatch is a direct and elegant consequence of this novel abstraction. This goes beyond prior type systems like Graphene's [28], which, as the authors correctly note, focus on static memory layout rather than the dynamic streaming order.

2. Novelty in FIFO Sizing Formulation: The problem of buffer sizing is not new; it is a foundational topic in dataflow modeling [10, 26, 38]. However, much of the classic work focuses on steady-state analysis in models like Synchronous Dataflow (SDF). The proposed model in Section 5.3 is novel in its explicit and analytical modeling of both the initial transient phase (initial delay, D) and the steady-state phase (pipeline II, L) using piecewise linear functions. This provides a more accurate representation of the behavior of coarse-grained, deeply pipelined hardware kernels than steady-state models alone. Furthermore, framing the global FIFO sizing problem as an LP problem that minimizes inter-kernel delays (a proxy for buffer size) subject to satisfying data dependencies across all graph paths is an elegant and powerful formulation. This analytical approach is a clear advancement over prior automated methods that rely on time-consuming simulation [30].

3. A Coherent, Hierarchical Abstraction: The framework demonstrates a well-conceived hierarchy, using the itensor type at a high level to orchestrate complex dataflow transformations like kernel fusion, and then lowering this abstraction to a more conventional stream/buffer representation for code generation. This separation of concerns allows the novel type system to be maximally effective at the right level of abstraction.

Weaknesses

While the core ideas are novel, the evaluation of their specific benefits and the discussion relative to the closest conceptual prior art could be strengthened.

1. Insufficient Differentiation from Scheduling DSLs: The paper contrasts itensor with traditional tensor types but does not sufficiently discuss the delta between its declarative, type-based approach and the procedural approach of powerful scheduling DSLs like Halide or TVM's TensorIR. A Halide schedule also contains all the information needed to determine a stream's layout. The key difference is that in Halide, this is an attribute of the computation (Func), whereas here it is an attribute of the data (itensor). The authors should more explicitly argue why the latter is superior for the task of composing pre-existing dataflow kernels, which is the central challenge in this domain. The novelty is in the representation, but its superiority over alternative representations is not fully established.

2. Novelty of Design Space Exploration (DSE) Algorithms: In Section 5, the paper describes the exploration of three design spaces. However, the algorithms employed for this exploration (e.g., intensity-driven unrolling, heuristic permutation) are themselves standard practice. The novelty here is that the itensor framework enables a more systematic exploration, but the exploration techniques themselves do not appear to be novel contributions. The paper should be clearer in distinguishing between the novel framework and the standard optimization heuristics applied within it.

3. Complexity vs. Benefit Justification: The proposed framework, particularly the itensor type and the multi-level IR, introduces considerable compiler complexity. The paper demonstrates strong end-to-end results, but it does not isolate the benefit of its novel contributions. For example, the LP-based FIFO sizing is elegant, but how much better is it in practice (in terms of area and performance) than the simpler "Conservative" strategy mentioned in Section 5.3.3? Without an ablation study directly comparing the outcome of the novel LP algorithm to simpler heuristics, it is difficult to judge if the added complexity provides a marginal or a transformative benefit.

Questions to Address In Rebuttal

1. The central novelty is the itensor type. Could the authors elaborate on the fundamental advantages of a declarative, type-based representation of stream layout compared to deriving the same information from a procedural schedule, as is done in compilers like Halide or TVM? Specifically, how does the itensor type simplify or enable optimizations that would be difficult or impossible otherwise?

2. Regarding the novel FIFO sizing model in Section 5.3.4, could the authors provide an ablation study comparing the FIFO sizes and resulting performance/latency from their LP formulation against the simpler "Conservative" strategy and a naive heuristic (e.g., sizing based on peak throughput difference)? This would help quantify the concrete benefit of this novel analytical model.

3. The itensor type system appears well-suited for dense, affine access patterns. What are its limitations? How would the system be extended to support more dynamic or irregular streaming patterns, such as those arising from sparse tensor operations or data-dependent control flow? Is the proposed formalism extensible to these cases?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents Chameleon, an LLM serving system for multi-adapter workloads, aiming to improve latency and throughput over existing systems like S-LoRA. The authors identify two primary bottlenecks: the overhead of loading LoRA adapters from host to GPU memory and head-of-line (HoL) blocking caused by workload heterogeneity. To address these, Chameleon introduces two main components: (1) a dynamic, software-managed cache for LoRA adapters in otherwise idle GPU memory, managed by a cost-aware eviction policy, and (2) a non-preemptive, multi-queue scheduler that classifies requests based on a "Weighted Request Size" (WRS) to prioritize smaller requests without starving larger ones. The evaluation claims significant reductions in P99 TTFT latency (80.7%) and a 1.5x throughput improvement over S-LoRA under high load.

Strengths

1. The paper provides a solid characterization of the problem space in Section 3. The analysis effectively demonstrates that adapter loading contributes significantly to latency, especially with tensor parallelism (Figure 5), and that PCIe bandwidth becomes a bottleneck with many unique adapters (Figure 4). This motivation is clear and well-supported by their initial experiments.
2. The core concept of caching adapters in GPU memory is a logical and powerful solution to the identified loading overhead. The ablation study (Figure 11, "ChameleonNoSched") clearly shows that this caching mechanism is responsible for the majority of the performance gains, confirming the validity of this approach.
3. The evaluation is extensive in scope, covering multiple loads, LLM sizes, GPU memory configurations, and a multi-GPU tensor parallelism setup. The scalability analysis in Section 5.5 and 5.6 provides useful data points on the system's behavior under different hardware constraints.

Weaknesses

My primary concerns with this paper relate to the potential for over-tuning on the evaluation workload, the questionable justification for the scheduler's complexity, and a failure to address critical resource trade-offs.

1. Generalizability of "Adaptive" Policies is Unsubstantiated: The system's core policies rely on magic numbers derived from offline profiling of a single trace, which fundamentally undermines the claim of being adaptive.

   • The cost-aware eviction policy (Section 4.2, page 6) uses the formula Score = FxFrequency+R×Recency+S×Size. The coefficients are statically set to F=0.45, R=0.10, S=0.45 based on "offline profiling of industrial traces [41]". This is a classic case of overfitting the policy to the evaluation data. There is no evidence that these weights are optimal, or even effective, for workloads with different characteristics (e.g., the WildChat or LMSYS traces also used in the paper).
   • Similarly, the Weighted Request Size (WRS) formula (Section 4.3, page 7) uses coefficients A=0.4 and B=0.6. The authors state these are based on "sensitivity studies and on profiling", which is vague and irreproducible. This critical component, which governs the entire scheduling process, appears to be manually tuned to the specific workload and hardware used in the evaluation.

2. Scheduler Complexity is Not Justified by Gains: The paper introduces significant complexity with its dynamic, K-Means-based multi-queue scheduler. However, the evidence shows this complexity yields marginal benefits.

   • In Section 5.4.5 and Figure 22 (page 12), the authors compare their dynamic queue organization against a simple, static 4-queue setup. The dynamic approach provides only a 10% reduction in P99 TTFT at high load. This minor improvement does not seem to justify the overhead and complexity of periodically running K-Means clustering to reconfigure queues and quotas.

3. Baseline Comparisons May Be Unfair: The paper's claims of superiority are predicated on comparisons that may not be rigorous.

   • The comparison against a Shortest-Job-First (SJF) scheduler (Figure 8 and Figure 15) highlights starvation of long requests. However, any production-grade SJF scheduler for latency-sensitive systems incorporates an aging mechanism to mitigate starvation. The paper cites µServe [46], which uses such a mechanism, but it is unclear if the baseline they implemented is this robust version or a naive, strawman SJF. Without aging, the reported starvation is an expected artifact, not a novel finding.
   • The ablation study in Figure 11 shows that the scheduler ("ChameleonNoCache") provides a very small improvement over S-LoRA. The vast majority of the gains come from the cache. This suggests the scheduling contribution is minor.

4. Critical Resource Trade-Off is Ignored: The fundamental premise is to use "idle GPU memory" for the adapter cache. This memory is in direct contention with the KV cache, which is a primary determinant of system throughput via batch size. The paper fails to analyze this trade-off. It is entirely plausible that allocating this "idle" memory to expand the KV cache in the S-LoRA baseline would allow for larger batch sizes, yielding a throughput improvement that could rival or exceed that of Chameleon. By not evaluating this alternative configuration, the paper presents an incomplete and potentially misleading picture of performance.

Questions to Address In Rebuttal

1. Regarding the eviction policy coefficients (F, R, S) and WRS weights (A, B): Please provide evidence of their robustness. How do the results change when applying the exact same coefficients (derived from the Azure trace) to the WildChat and LMSYS traces? If they must be re-tuned for each trace, the claim of adaptivity is weak.
2. The dynamic queue management shows only a 10% benefit over a static configuration (Fig. 22) at high load. Can the authors provide a compelling justification for this added complexity? What is the computational overhead of the periodic K-Means clustering and how does it affect the system, especially during load spikes?
3. Please clarify the implementation of the SJF baseline used in Section 5.3. Does it include an aging mechanism to prevent starvation, as is standard practice and described in the work [46] you cite? If not, why is this considered a fair comparison?
4. The adapter cache competes for GPU memory with the KV cache. Please provide an analysis comparing Chameleon's use of idle memory for adapter caching against an alternative scenario: S-LoRA is configured to use the same amount of "idle" memory to expand its total KV cache capacity, which would enable larger effective batch sizes. How does S-LoRA's throughput compare under that configuration?

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Chameleon, an LLM inference serving system designed for environments with a large number of fine-tuned adapters, a scenario popularized by techniques like Low-Rank Adaptation (LoRA). The authors identify two primary bottlenecks that emerge in these "many-adapter" settings: (1) the I/O overhead of frequently loading adapter weights from host memory to GPU memory, which contends for PCIe bandwidth, and (2) the increased workload heterogeneity introduced by adapters of varying ranks (sizes) and popularity, which exacerbates tail latency issues like head-of-line blocking.

Chameleon's core contribution is to treat the adapters themselves as a first-class resource to be managed. It introduces a synergistic two-part solution: an adaptive adapter cache that utilizes otherwise idle GPU memory to store popular or costly-to-load adapters, and an adapter-aware multi-queue scheduler that classifies requests based on a weighted size (including input, predicted output, and adapter rank) to provide a fast lane for small requests while preventing starvation for large ones. The work is positioned as a significant enhancement over state-of-the-art systems like S-LoRA, which load adapters on-demand. The evaluation demonstrates substantial improvements in P99 latency (80.7% reduction) and throughput (1.5x increase) under high load.

Strengths

1. Excellent Problem Formulation and Characterization: The paper's primary strength lies in its clear identification and meticulous characterization of an important, emerging problem. Before presenting their solution, the authors dedicate Section 3 to demonstrating why existing systems are insufficient. Through targeted experiments, they convincingly show that adapters are a non-trivial source of performance heterogeneity (Figures 2 and 3), that adapter loading is a significant and scalable bottleneck (Figures 4 and 5), and that the opportunity to solve this (idle GPU memory) exists but is dynamic (Figure 6). This foundational analysis is what makes the proposed solution so compelling.

2. Elegant Application of Classic Systems Principles: The core ideas of Chameleon—caching and multi-level feedback queue scheduling—are not new to computer science, but their application to this specific domain is novel and elegant. The paper effectively recasts the adapter management problem into a familiar resource management paradigm. The adapter cache is a clever use of a transient resource (idle GPU memory), and its cost-aware eviction policy (Section 4.2) correctly recognizes that not all cache misses are equal, drawing a parallel to classic web and database caching systems. Similarly, the multi-queue scheduler (Section 4.3) is a well-understood technique for balancing responsiveness and fairness, which the authors have adapted skillfully to the unique sources of heterogeneity in LLM inference.

3. Holistic and Synergistic System Design: The two main components of Chameleon are not independent add-ons; they are designed to work together. The scheduler's awareness of adapter rank informs the cache manager's decisions about which adapters are valuable to keep resident. For example, scheduling a request with a large adapter that is already cached is much cheaper than scheduling one with a small adapter that needs to be loaded. This synergy between scheduling and caching is a hallmark of a well-thought-out system design. The architecture shown in Figure 9 clearly illustrates this interplay.

4. Significant and Practical Impact: The work addresses a real-world challenge. As organizations increasingly rely on fine-tuning for personalization and task-specialization, efficiently serving thousands of adapters will become a critical cost and performance issue. Chameleon offers a practical, software-only solution that could be integrated into popular serving frameworks (like vLLM, TGI, etc.). The reported performance gains, especially the dramatic reduction in P99 tail latency, are highly significant and would translate directly to improved user experience and lower operational costs in production environments.

Weaknesses

While the paper is strong, there are areas where the broader context and system dynamics could be explored more deeply.

1. Interaction Dynamics with the KV Cache: The paper's central premise for the adapter cache is the existence of "idle GPU memory." However, in a heavily loaded serving system, the primary consumer of dynamic GPU memory is the KV cache. The paper treats these two as simply competing for space. A deeper analysis of their interaction would be beneficial. For instance, a burst of long requests could cause the KV cache to expand rapidly, forcing the eviction of many adapters from the Chameleon cache. This could, in turn, increase the latency of the next batch of requests, which now must reload those adapters, creating a potential for performance oscillations or thrashing between the two memory consumers. A discussion of this dynamic would strengthen the paper.

2. Static Nature of the Eviction Policy: The cost-aware eviction policy (Section 4.2) uses a linear combination of frequency, recency, and size, with coefficients (F=0.45, R=0.10, S=0.45) set via offline profiling. While this is a reasonable approach, it raises questions about its robustness across different workloads. A workload with high temporal locality might benefit from a higher weight on recency (R), while a workload with stable popularity might benefit from a higher weight on frequency (F). The paper could be improved by either demonstrating the policy's robustness or discussing a more adaptive mechanism for tuning these weights online.

3. Limited Discussion on Predictor Accuracy: The scheduler relies on a BERT-based proxy model to predict output length, which is a key component of the Weighted Request Size (WRS). The sensitivity analysis in Section 5.4 (Figure 19) shows the system is reasonably robust, but the discussion could be expanded. How does poor prediction accuracy impact fairness? Could it lead to requests being systematically misclassified into the wrong queue, effectively undermining the scheduler's design?

Questions to Address In Rebuttal

1. Regarding the interaction between the adapter cache and the KV cache: Could you elaborate on the potential for thrashing between these two memory consumers under volatile loads? Does Chameleon implement any mechanism to coordinate between the KV cache manager and the adapter cache manager to prevent such negative feedback loops?

2. The eviction policy coefficients are tuned offline. How sensitive are the overall performance gains to these specific values (F=0.45, R=0.10, S=0.45)? Have you explored how these optimal weights might change for workloads with different characteristics (e.g., streaming vs. request-response, different adapter popularity distributions)?

3. The WRS formula in Section 4.3 weights the normalized output size more heavily (B=0.6) than the input size (A=0.4). Could you provide more intuition for this choice? Is it primarily because the decode phase, which depends on output length, typically dominates total execution time?

4. Your work is a significant step forward for node-level scheduling in many-adapter environments. How do you see these ideas integrating with cluster-level schedulers? For example, could a cluster scheduler use information about which adapters are cached on which nodes to make more intelligent request routing decisions?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present Chameleon, an LLM inference serving system designed for multi-task environments that leverage LoRA adapters. The paper claims two primary novel contributions: 1) an adaptive cache for LoRA adapters in GPU memory to mitigate loading overheads, and 2) an adapter-aware multi-queue scheduler to reduce head-of-line blocking and improve tail latency. The system is implemented on top of S-LoRA and evaluated against it, demonstrating significant improvements in P99 TTFT latency and throughput.

While the engineering effort is commendable and the performance results are strong, my review finds that the core ideas presented as novel are, in fact, direct applications of long-established principles from computer systems, operating systems, and networking. The novelty is confined to the specific application of these principles to the domain of LoRA adapter serving, not in the invention of new caching or scheduling paradigms.

Strengths

1. Problem Identification: The paper does an excellent job characterizing (Section 3, pages 3-4) the specific performance bottlenecks in many-adapter serving environments, correctly identifying adapter loading overhead and increased workload heterogeneity as critical issues.
2. System Integration: The integration of the caching and scheduling components is well-executed. The synergy between the two, where the scheduler's decisions inform the cache manager, demonstrates solid system design.
3. Domain-Specific Heuristic: The Weighted Request Size (WRS) formula (Section 4.3, page 7) is a logical, domain-specific heuristic. Including adapter rank alongside input and output size is a sensible extension to prior request-sizing metrics.

Weaknesses

My evaluation is focused exclusively on conceptual novelty. In this regard, the paper's claims are significantly overstated.

1. Adapter Caching Lacks Conceptual Novelty: The proposed "adapter cache" is a standard memory cache. The idea of keeping frequently-used, read-only data in a faster tier of the memory hierarchy (GPU memory) to avoid fetching it from a slower tier (host memory via PCIe) is a foundational concept in computer architecture and systems.

   • Prior Art on Policy: The "Cost-Aware Eviction Policy" (Section 4.2, page 6) is a linear combination of frequency, recency, and size/cost (Score = F*Frequency + R*Recency + S*Size). This is a classic formulation for cost-aware caching heuristics. In fact, the paper itself cites GDSF [5] (page 11), a well-known algorithm from 1998 that uses a Frequency * Cost / Size heuristic. The proposed policy is a variation on this theme, not a new invention. The novelty is in the tuning of weights (F, R, S), not the approach itself.
   • Prior Art on Mechanism: The dynamic sizing of the cache based on available memory is a standard practice in software-managed caches where memory is shared with an application (in this case, the KV cache). The claim of introducing "the first cache design for LoRA adapters" (page 2) is only true in the most literal sense; it is the first application of a standard cache to this specific data type, which does not constitute a novel contribution to the field of computer systems.

2. Multi-Queue Scheduling is Well-Established Prior Art: The proposed "adapter-aware multi-queue scheduler" is conceptually identical to decades of work on preventing Head-of-Line (HoL) blocking in schedulers for operating systems, routers, and web servers.

   • Prior Art on Structure: The core idea of partitioning tasks by size into different queues to provide an "express lane" for short jobs is not new. The paper itself acknowledges this by citing Size-Interval Task Assignment (SITA) [7, 15] and Q-Zilla [35] in the related work section (page 13). SITA, proposed in 1999, is the direct intellectual ancestor of this scheduling approach.
   • Incremental Heuristic: The only new element is the WRS formula, which adds AdapterSize to the sizing metric. Prior work like µServe [46] already uses predicted output length. This is an incremental, though logical, extension of an existing heuristic, not a fundamentally new scheduling algorithm. Applying K-Means to dynamically determine queue boundaries is also an application of a standard clustering algorithm, not a novel scheduling concept.

3. Complexity vs. Benefit: The paper introduces significant machinery (output length prediction, K-Means clustering, dynamic quota calculation, cost-aware cache management) to implement these known concepts. While the performance gains are substantial compared to a simple FIFO baseline (S-LoRA), it is unclear how much of this gain comes from the well-understood benefits of multi-queue scheduling vs. the specific "adapter-aware" component. The contribution is better framed as a comprehensive engineering effort to apply best practices to a new domain, rather than a source of new foundational ideas.

Questions to Address In Rebuttal

1. Please clarify the conceptual novelty of the adapter cache beyond it being a standard software-managed, cost-aware cache. What fundamentally distinguishes the eviction policy from the family of algorithms represented by GDSF [5], which also balances frequency, cost, and size?

2. The paper's scheduler is structurally and functionally analogous to SITA [7, 15]. Given this, can the authors articulate the core novel contribution in their scheduler beyond the domain-specific WRS sizing heuristic? Is the claim one of a new scheduling paradigm, or a new and effective application of an existing one?

3. To better isolate the novelty, could the authors compare Chameleon's scheduler not just to FIFO and SJF, but to a SITA-like scheduler that uses a simpler sizing heuristic (e.g., only predicted output tokens)? This would help quantify the specific benefit of making the scheduler "adapter-aware," which appears to be the primary delta over extensive prior art.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present Coruscant, a co-designed software kernel and hardware architecture extension aimed at accelerating LLM inference by exploiting unstructured sparsity in the 30-70% range. The work is composed of three main contributions: (1) a bitmap-based sparse format designed to improve compression ratios over existing formats like CSR in this moderate sparsity regime; (2) a corresponding GPU SpMM kernel that uses this format to reduce data movement; and (3) a proposed hardware modification to the Tensor Core, the "Coruscant Sparse Tensor Core," which integrates a "Bitmap Decoder" to operate directly on the compressed format. The authors claim significant speedups over cuBLAS and Flash-LLM, and further gains with their proposed hardware.

Strengths

1. Problem Motivation: The paper correctly identifies a critical gap in the literature. While many works focus on extreme sparsity (>90%) or rigidly structured sparsity (2:4), the moderate, unstructured sparsity regime (30-70%) common in accuracy-preserving LLM pruning is underserved by existing hardware and software. Figure 1 provides a standard but clear motivation for focusing on the memory-bound nature of the decode stage.

2. Identification of Format Weakness: The analysis in Section 2.1 and Figure 3 effectively demonstrates that conventional index-based sparse formats (CSR, COO) are inefficient and can even lead to memory expansion at the target sparsity levels. This provides a solid rationale for exploring alternative representations like bitmaps.

Weaknesses

My analysis reveals several significant methodological and analytical flaws that call the paper's central claims into question.

1. Highly Questionable Hardware Simulation Methodology: The evaluation of the proposed Coruscant Sparse Tensor Core (STC) is fundamentally weak. In Section 4, the authors state they "simulate the behavior...by removing bitmap decoding and shared memory access writes...and retaining the standard HMMA instructions." This is not a simulation; it is a simplistic analytical projection. This approach completely ignores potential microarchitectural side effects, such as pipeline stalls caused by the new decoder logic, increased register file pressure, or contention on operand buses. It assumes the proposed "Bitmap Decoder" is a zero-cost, zero-latency oracle. Consequently, all performance results for the Coruscant STC (e.g., in Figures 11b, 15, 19) are speculative at best and likely overly optimistic.

2. Inadequate Comparison to Hardware-Accelerated Baselines: The comparison against NVIDIA's 2:4 sparsity in Section 5.3.4 and Figure 21 is telling. The authors' software kernel is slower than the 2:4 kernel, and even their proposed hardware (Coruscant STC) only "nearly matches" the 2:4 kernel's latency at 50% sparsity. They attribute the baseline's advantage to higher warp occupancy, but this is not an excuse—it is a critical performance factor. Their proposed dataflow and kernel implementation appear to be less efficient at utilizing the GPU's resources than the industry standard. The paper fails to make a convincing case for unstructured sparsity when it cannot clearly outperform the existing, hardware-accelerated structured sparsity solution at its home turf of 50% sparsity.

3. Discrepancy Between Theoretical and Actual Compression: There is a clear disconnect between the ideal compression ratios presented in Figure 3 and the actual memory footprint reduction achieved by the kernel, as discussed in Section 5.2.2. The authors admit this is due to padding added "to fully utilize the GPU memory bandwidth." This overhead is non-trivial and weakens the core premise that their format leads to superior memory efficiency in practice. The paper lacks a rigorous quantification of this padding overhead across different matrix sizes and sparsities, making the real-world benefit of their format ambiguous.

4. Superficial Hardware Overhead Analysis: The area and power analysis in Section 5.3.2 and Table 2 is not credible. The authors report synthesizing the Bitmap Decoder in a 45nm process and then scaling the results to 7nm using generic equations from Stillmaker et al. [48]. This methodology is widely understood to be inaccurate for modern nodes, as it fails to account for the dominance of wire delay, leakage power, and complex physical design rules. A proper hardware analysis would require synthesis in a modern PDK or a detailed, bottom-up analysis of the circuit. The presented numbers appear to be a back-of-the-envelope calculation designed to minimize the perceived cost of their hardware addition.

5. Weak Link Between Performance Claims and Accuracy Motivation: The paper begins by arguing that unstructured sparsity is superior for maintaining model accuracy (Table 1, Figure 2). However, the core end-to-end evaluation in Section 5.1 and Figure 15 only reports performance metrics (tokens/sec). It is never explicitly shown that the models pruned to 30/50/70% for this performance evaluation actually maintain their accuracy. A fair comparison would require evaluating performance against a dense baseline of equivalent accuracy, which may involve techniques other than pruning. The paper implicitly compares a lower-accuracy pruned model to a full-accuracy dense model, which inflates the perceived efficiency gains.

Questions to Address In Rebuttal

The authors must provide clear and convincing responses to the following points:

1. On Hardware Methodology: Justify the use of an analytical model for the Coruscant STC performance evaluation. Provide evidence, such as from a detailed microarchitectural simulator (e.g., GPGPU-Sim), that your model accurately captures pipeline dynamics and that the Bitmap Decoder does not introduce new performance bottlenecks. Without this, the STC results are unsubstantiated.

2. On Baseline Performance: Explain, with detailed profiling data (e.g., from Nsight Compute), the precise reasons for the Coruscant kernel’s lower performance and resource utilization compared to the 2:4 cuSPARSELt kernel at 50% sparsity. Why should the community adopt a more complex unstructured approach if it fails to outperform the simpler structured alternative in a direct comparison?

3. On Compression Overhead: Provide a new table or figure that directly compares the "ideal" compression ratio of your format (from Figure 3) against the "actual" memory footprint of the kernel (including all padding and metadata overheads) for every data point presented in the evaluation (all sparsities, matrix sizes, and batch sizes).

4. On Accuracy in Evaluation: For the end-to-end results in Figure 15, what are the measured perplexity scores for the pruned Llama 2 models at 30%, 50%, and 70% sparsity? How do these scores compare to the dense baseline? The claimed speedups are meaningless without the context of the accuracy trade-off.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Coruscant, a compelling co-design of a GPU kernel and a sparse tensor core architecture aimed at making unstructured sparsity practical for Large Language Model (LLM) inference. The authors identify a critical gap in the current landscape: state-of-the-art pruning techniques naturally produce unstructured sparsity in the 30%-70% range, yet modern hardware, notably NVIDIA's 2:4 semi-structured sparsity, cannot efficiently exploit it, forcing a trade-off between model accuracy and hardware performance.

Coruscant's core contribution is a full-stack solution to this problem. It introduces: 1. A memory-efficient, bitmap-based sparse format designed for this moderate sparsity regime. 2. A software-only GPU kernel that leverages this format on existing commercial GPUs to reduce memory footprint and outperform dense (cuBLAS) and other sparse (Flash-LLM) kernels. 3. A minimal, pragmatic extension to the GPU's Sparse Tensor Core (STC) that directly decodes the bitmap format in hardware, eliminating the software decompression overhead for even greater performance gains.

The work is well-motivated by the memory-bound nature of the LLM decode phase and provides a clear pathway for the machine learning and computer architecture communities to reconcile the benefits of unstructured pruning with the demands of efficient hardware execution.

Strengths

The primary strength of this paper is its insightful positioning and holistic approach to a significant, real-world problem.

1. Excellent Problem Formulation: The authors have correctly identified a crucial point of friction between two advancing fields. On one hand, LLM pruning research (e.g., SparseGPT, Wanda) is pushing towards more flexible, unstructured methods to preserve accuracy. On the other, the hardware community has converged on rigid, structured solutions like 2:4 sparsity for efficiency. Coruscant builds a much-needed bridge between these two worlds. The analysis in Section 2.1 and Figures 2 & 3 effectively establishes the need for a solution in the 30-70% sparsity range, a zone where prior formats are shown to be inefficient.

2. Pragmatic Full-Stack Co-Design: This is not just a theoretical hardware proposal. The authors provide an immediately useful software kernel for today's GPUs and a well-defined, minimally invasive hardware modification for tomorrow's. This two-pronged approach dramatically increases the work's potential impact. It allows the community to adopt the ideas in software now while providing a clear, low-overhead blueprint for hardware vendors. The proposed "Bitmap Decoder" is a small, targeted addition, not a complete redesign, which makes it far more plausible for adoption.

3. Strong Contextualization and Evaluation: The paper does an admirable job of placing itself within the vast literature of sparse computation and LLM optimization. The comparisons are not just against the standard dense baseline (cuBLAS) but also against relevant, cutting-edge sparse kernels like Flash-LLM, SparTA, and Sputnik (Figure 16). Furthermore, the architectural comparison against prior academic STC designs like DSTC and RM-STC (Section 5.3.3) is particularly insightful, correctly arguing that Coruscant's simpler, memory-focused design is better suited for the "skinny matrix" SpMM workloads of LLM inference, as opposed to the compute-bound workloads those prior works targeted.

4. Connecting Pruning to System-Level Benefits: The work successfully translates the benefits of its approach from kernel-level speedups to tangible system-level advantages. The end-to-end evaluation (Section 5.1, Figure 15) shows not only increased token throughput but also the ability to run larger batch sizes by freeing up VRAM, directly addressing the "Out of Memory" errors that plague LLM serving. This demonstrates a deep understanding of the end-user's problem.

Weaknesses

The weaknesses of the paper are minor and primarily relate to missed opportunities to further broaden its context and impact.

1. Limited Discussion on Interaction with Other Compression Techniques: The paper is laser-focused on sparsity. However, in practice, sparsity is almost always combined with quantization. There is no discussion of how the Coruscant format would interact with weight quantization schemes (e.g., 8-bit or 4-bit integers). Would the bitmap overhead become more significant relative to the compressed data? Does the STC design need modification to handle quantized data types? Acknowledging this interaction is crucial for a complete system-level solution.

2. Focus on Weight Sparsity Only: The work is entirely centered on static, unstructured weight sparsity. Another emerging area of interest is activation sparsity, which is dynamic and data-dependent. While this is clearly outside the paper's primary scope, the core idea of a hardware-accelerated bitmap decoder could potentially be relevant there as well. A brief mention of this as a future direction would strengthen the paper's long-term vision.

3. The "Why Now?" Argument Could Be Sharpened: The motivation is good, but it could be even more powerful. The paper explains what the problem is, but it could spend more time on why solving it is becoming existential for the future of LLMs. As models grow and context windows expand, the memory footprint of weights (even without the KV cache) becomes a fundamental limiter. Framing Coruscant not just as an optimization but as an enabling technology for future, more powerful models on commodity hardware would elevate its perceived importance.

Questions to Address In Rebuttal

1. Synergy with Quantization: Could the authors comment on the synergy or potential conflicts between their bitmap-based format and popular quantization schemes (e.g., INT8, FP8, or INT4)? How would the memory footprint and hardware design change if Coruscant were to support pruned and quantized models?

2. The Next Bottleneck: Coruscant convincingly argues for a solution to the weight memory bandwidth bottleneck in sparse LLM inference. With this addressed by the proposed STC, what do the authors foresee as the next major bottleneck? Does this approach simply shift the performance problem to another part of the system, perhaps the instruction frontend or the register file bandwidth, especially given the more complex decoding logic?

3. Generality Beyond LLMs: While the work is excellently motivated by LLM inference, the proposed format and hardware seem more general. Have the authors considered its applicability to other domains that feature moderate, unstructured sparsity, such as graph neural networks (GNNs) or certain scientific simulations? How would the "skinny matrix" assumption hold up in those domains, and would the design still be effective?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents "Coruscant," a co-designed system comprising a bitmap-based sparse format, a corresponding GPU SpMM kernel, and a minimal extension to the GPU Tensor Core, all aimed at accelerating LLM inference with unstructured weight sparsity. The authors identify a critical gap in existing solutions, which are either inefficient for moderate sparsity levels (30-70%) or enforce restrictive semi-structured patterns like 2:4 sparsity. The core proposal is to use a simple bitmap to represent sparsity within tiles, which improves compression for this specific sparsity range, and then to design a software kernel and a hardware "Bitmap Decoder" to process this format efficiently for the memory-bound, skinny matrix multiplications characteristic of LLM decode steps.

Strengths

1. Specialization as a Novel Contribution: The primary strength and novel aspect of this work is not the invention of a new primitive, but the highly effective specialization of existing concepts for a specific, important workload. The authors correctly identify that prior sparse tensor core designs (e.g., DSTC, RM-STC) were targeted at general-purpose, compute-bound problems, leading to complex and area-intensive hardware. Coruscant’s novelty lies in its insight that the LLM inference problem (sparse-weight, dense-activation SpMM) allows for a significant simplification:

   • It only requires single-operand sparsity, eliminating the need for complex scatter-gather and merge logic for the output matrix that plagues dual-operand sparse designs.
   • It targets a memory-bound regime, correctly prioritizing compression efficiency (via larger tile sizes) over maximizing computation skipping. This is a crucial and well-justified trade-off.

2. Architectural Elegance and Simplicity: The proposed hardware modification, the "Bitmap Decoder" (Figure 14, page 7), is minimalistic. It reuses the existing accumulation and data paths of a dense tensor core, adding only the logic to decode the bitmap and select the appropriate non-zero values from registers. This stands in stark contrast to the significant architectural rework proposed in prior art for unstructured sparsity. The claimed low area overhead (Table 2, page 10) makes this a practical and economically viable proposal, which is a form of novelty in itself.

3. Kernel-Level Innovation: The co-designed GPU kernel contains a subtle but important novel element: the use of column-wise tiling to mitigate shared memory bank conflicts during the software-based decompression stage (Figure 9, page 6). This is a non-obvious implementation detail that directly addresses a key performance bottleneck for this class of algorithm and demonstrates a deep understanding of the GPU execution model.

Weaknesses

1. Fundamental Primitives are Not Novel: The primary weakness from a novelty perspective is that the core building blocks are well-established.

   • Bitmap-based Sparse Formats: Representing sparse data with bitmaps is a classic technique. It is not a novel concept.
   • Hardware Decoders for Bitmaps: The idea of a hardware unit that processes a bitmap to gate or select data is also not fundamentally new.
   • Sparse Tensor Cores for Unstructured Sparsity: The most direct prior art, DSTC [55] and RM-STC [21], which the authors thankfully cite and compare against in Section 5.3.3, have already proposed the use of bitmap-based formats for unstructured sparsity in tensor cores.

   Therefore, the claim to novelty must rest entirely on the specific architectural choices and co-design for the LLM workload, not on the invention of the underlying mechanisms. The paper's framing could be clearer on this point; it sometimes implies the format itself is the primary innovation, when in fact the true innovation is the simplified architecture it enables for this specific problem.

2. Limited Scope of Novelty: The contribution is sharply focused on skinny SpMM. While this is the paper's stated goal, it means the proposed architecture is less general than prior work. The novelty is derived from removing generality, which is a valid but narrow path for contribution. The paper would be strengthened by more explicitly defining the boundaries where its approach is superior and where more general (but complex) approaches like DSTC would be preferable.

Questions to Address In Rebuttal

1. Given that DSTC [55] and RM-STC [21] have already proposed bitmap-based sparse tensor cores for unstructured sparsity, please articulate precisely what the core architectural novelty of the Coruscant STC is, beyond a change in tile size and specialization for single-sided sparsity. The rebuttal should focus on why this specialization constitutes a significant inventive step over these prior works, rather than an incremental engineering optimization.

2. The column-wise tiling strategy (Figure 9, page 6) to avoid shared memory bank conflicts is a key part of the software kernel's performance. Is this technique novel in the context of GPU-based sparse matrix decompression, or are there precedents for this specific approach in prior literature on SpMM or other sparse kernels?

3. The paper's core trade-off is prioritizing memory compression over computation skipping, which is well-suited for the target memory-bound workload. Can you characterize the crossover point (e.g., in terms of batch size 'N' or a machine's operational intensity) where the computation-skipping benefits of a more complex architecture like DSTC would begin to outweigh the compression benefits of Coruscant? A clear analysis of this trade-off boundary would better situate the novelty of your contribution.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes MNM, a heterogeneous computing architecture integrating Processing-In-Memory (PIM) on the HBM core die and Processing-Near-Memory (PNM) on the logic die to accelerate Retrieval-Augmented Language Models (RALMs). The authors claim that by offloading memory-bound GEMV operations of the language model to PIM and retrieval-specific tasks (vector search, sorting) to PNM, their architecture can overcome memory bandwidth bottlenecks inherent to conventional GPU-based systems. They supplement this hardware with a novel scheduling strategy, "selective batching and early generation," which purportedly overlaps retrieval and generation to reduce idle cycles. The authors report significant performance speedups (up to 29.2x) and energy savings (up to 71.5%) over an NVIDIA H100 GPU baseline.

Strengths

1. Problem Characterization: The initial workload analysis in Section 3 (Figure 4, page 5) is thorough and correctly identifies the memory-bound nature of key RALM components. The roofline model analysis effectively motivates the need for a non-conventional architectural solution by demonstrating that both the language model's MHA kernels and the retriever's PQ code scan/top-k selection are far from the compute-bound regime on a state-of-the-art GPU.

2. Task Partitioning Rationale: The architectural decision to partition tasks—assigning simple, regular GEMV operations to PIM units and more complex, specialized retrieval logic to PNM—is well-reasoned (Section 4.1, page 6). This approach correctly acknowledges the fabrication and area constraints of logic on a DRAM die versus the flexibility of a logic die.

Weaknesses

My primary concerns with this work lie in the realism of the evaluation methodology, the downplaying of critical trade-offs, and the lack of rigor in overhead analysis.

1. Unjustified Quality-Performance Trade-off: The proposed "Early Generation" scheduling scheme is presented as a key innovation, yet it comes at the cost of model accuracy. Figure 8 (page 9) explicitly shows an increase in perplexity as batch size and nprobe scale. While the authors frame this as a smaller degradation compared to a GPU-based equivalent, it is a degradation nonetheless. The paper fails to adequately justify this trade-off. An architecture that achieves speedup by compromising the correctness of the model's output is fundamentally flawed. The core premise of RALM is to improve generation quality with retrieved data; this scheduler actively works against that goal by using stale data.

2. Questionable Area and Power Overheads: The overhead analysis in Section 6.6 (page 13) relies on highly suspect assumptions. The area of PIM logic is synthesized for a 14nm process and then scaled by a factor of "10x" to approximate a DRAM process node. This scaling factor is a crude approximation based on a single reference and lacks the necessary justification for a rigorous hardware paper. Consequently, the claim of a "15.0% overhead" on the core die is built on a weak foundation. This figure is not minor; a 15% reduction in memory cell area is a commercially prohibitive cost that the authors dismiss too readily. The power figures in Table 3 (page 10) also appear optimistic, and it is unclear if they account for all sources of leakage and activity under realistic concurrent operation.

3. Oversimplified Thermal Modeling: The thermal analysis presented in Section 4.1 (page 6) to justify PNM logic placement is superficial. The use of a simple 1D compact layered-conduction model is inadequate for a complex 3D-stacked device like HBM. This model ignores lateral heat spreading, the cumulative effect of hotspots from underlying PIM logic, and the thermal impact of the high-density TSV arrays. Presenting a temperature rise of less than 0.5°C based on such a model is unconvincing and potentially misleading. Real-world thermal throttling could easily negate the claimed performance benefits.

4. Unquantified System-Level Complexity: The proposed architecture introduces significant complexity that is not accounted for in the evaluation.

   • The MNM Controller (Figure 6, page 7), which translates GPU instructions into MNM commands, is a critical component whose design, latency, and area/power overheads are completely ignored.
   • The scheme to reorder PQ codeword IDs to solve memory alignment issues requires a host-side mapping table. The memory footprint and lookup latency of this table are never quantified. For a large database, this could be a non-trivial overhead.

5. Speculative Scalability Claims: The model and database scaling analysis in Section 6.5 (page 13) is based entirely on projection, not simulation or measurement. While such analyses can be illustrative, the claims made here are presented with undue confidence. The argument that MNM suffers from lower communication overhead than a baseline multi-GPU system is unsubstantiated, as the authors fail to quantify the command and data traffic their own scaled-up system would require between the host, GPUs, and multiple MNM-enabled HBM stacks.

Questions to Address In Rebuttal

1. Provide a rigorous justification for the 10x area scaling factor used for PIM logic. How does this factor account for differences in cell libraries, routing density, and process design rules between a mature logic process and a modern DRAM process?

2. The "Early Generation" scheduler trades model accuracy (perplexity) for latency. At what point does this degradation become unacceptable for a production-level RALM? Please provide a sensitivity analysis showing how perplexity scales with more extreme retrieval latencies and characterize the point at which the model's output quality is critically compromised.

3. Elaborate on the cost of the 15.0% PIM area overhead on the HBM core die. Quantify this in terms of lost memory capacity per die and per stack. How does this impact the commercial viability of such a memory product compared to standard HBM?

4. Address the limitations of the 1D thermal model. Acknowledge the potential impact of 3D thermal effects and explain why the risk of thermal throttling from the proposed PNM logic is negligible, especially when placed adjacent to high-activity PHY and TSV regions.

5. Provide a detailed architectural design for the "MNM Controller" on the GPU side, including its projected area, power, and the latency it adds to the command path. Furthermore, quantify the memory footprint and access overhead of the host-side mapping table required for PQ codeword ID reordering for the largest dataset used.

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

---

Summary

This paper presents MNM, a heterogeneous computing architecture designed to accelerate Retrieval-Augmented Language Models (RALMs). The core contribution is the insightful co-design of a hardware system and a scheduling policy that addresses the two distinct, memory-bound bottlenecks inherent in RALMs. The authors propose integrating both Processing-In-Memory (PIM) on the HBM core die and Processing-Near-Memory (PNM) on the HBM logic die. The PIM units are leveraged to accelerate the GEMV-heavy attention operations in the language model, while the more flexible PNM logic is dedicated to the lookup- and sort-intensive tasks of the vector search-based retriever. This hardware architecture is complemented by a novel "selective batching and early generation" scheduling strategy that exploits the hardware's capabilities to maximize the overlap between token generation and retrieval, mitigating the idle cycles common in batched RALM inference.

---

Strengths

1. Excellent Problem Characterization and Motivation: The authors perform a thorough workload analysis (Section 3, page 5) that correctly identifies the fundamental challenge in RALM acceleration: it is not a monolithic problem. They astutely recognize that both the language model and the retriever are memory-bound, but for different reasons. The language model's attention is limited by bandwidth for GEMV operations, while the IVF-PQ retriever is constrained by irregular memory access patterns (LUTs) and sorting. This clear diagnosis provides a strong foundation for their proposed solution.

2. Holistic, System-Level Approach: This work is a superb example of system-level co-design. Rather than focusing on a narrow hardware optimization, the authors consider the entire RALM pipeline. They propose a hardware solution (MNM) and a software scheduling policy that are synergistic. The hardware enables more efficient, concurrent operations, and the scheduling policy is explicitly designed to exploit this new capability. This holistic view is a significant strength and reflects a mature understanding of the problem domain.

3. Elegant and Well-Justified Architectural Mapping: The core architectural idea—mapping the two distinct RALM workloads to different parts of an HBM stack—is elegant and compelling. Placing simple, massively parallel MAC units (PIM) in the core die for the regular GEMV operations is a natural fit. Simultaneously, using the logic die for a more specialized PNM accelerator for retrieval tasks is equally wise, as it allows for more complex logic (e.g., sorters, arbiters) without the process technology constraints of the DRAM die. This "right tool for the right job" approach is the paper's central technical insight.

4. Strong Contextual Fit within Current Research Trends: This work is exceptionally well-positioned within the broader landscape of computer architecture and AI systems. It directly engages with several critical, contemporary research threads:

   • The Memory Wall: It is a direct attack on the memory wall for a critical emerging workload.
   • Heterogeneous Computing: It moves beyond the traditional CPU-GPU dichotomy and embraces a more specialized, heterogeneous model integrating PIM and PNM.
   • System-Level AI Acceleration: It recognizes that accelerating AI is about more than just speeding up matrix multiplies; it involves optimizing the entire data-to-answer pipeline, including data retrieval. This paper serves as a valuable case study for the future of AI system design.

---

Weaknesses

While the core idea is strong, the paper could be improved by broadening its contextual discussion to better situate its specific design choices.

1. Limited Discussion of Alternative Heterogeneous Designs: The proposed PIM/PNM integration within a single HBM stack is a compelling design point. However, the academic landscape includes other heterogeneous proposals. For instance, the recent work on HeterRAG [60] proposes using HBM-PIM for generation (low latency) and DIMM-based PIM for retrieval (high capacity). A discussion contrasting the MNM approach (all-in-HBM, maximizing bandwidth) with an approach like HeterRAG's would strengthen the paper by exploring the fundamental trade-off between retrieval database capacity and access latency. This would help readers understand the specific part of the design space that MNM is optimizing for.

2. Clarity on the Interdependence of Hardware and Software Contributions: The "Early Generation" scheduling scheme appears highly effective in conjunction with the MNM hardware. However, its performance on conventional systems is less clear. The perplexity analysis in Figure 8 (page 9) is a good start, but the paper could more explicitly articulate whether the scheduling strategy is a general contribution applicable to any RALM system, or if its benefits are primarily unlocked by the dramatically reduced retrieval latency provided by the MNM architecture. Clarifying this relationship would help delineate the impact of the hardware and software components of the work.

---

Questions to Address In Rebuttal

1. The choice to integrate both PIM and PNM accelerators within the HBM stack prioritizes bandwidth and low-latency communication between the two components. Could the authors elaborate on the trade-offs of this approach compared to a disaggregated design, such as the one proposed in HeterRAG [60], which might offer much larger retriever database capacity by using DIMM-based memory for retrieval? In what scenarios or scales of RALM deployment would the MNM design be most advantageous?

2. The proposed "Early Generation" scheduling is shown to be highly effective with MNM. Could you further elaborate on its applicability to conventional GPU-based systems? Given that GPUs would have a much longer retrieval latency, would the window for "early generation" shrink to the point of being ineffective, or does the selective batching component still provide significant benefits on its own?

3. The current PNM design is tailored for IVF-PQ retrieval. Looking forward, retrieval methods are evolving to include more complex logic, such as multi-hop reasoning or graph traversal. How extensible is the proposed PNM architecture to support these more compute-intensive retrieval tasks? Would it require a more general-purpose programmable core on the logic die, and what would be the implications for area and power?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper proposes MNM, a heterogeneous PIM/PNM architecture integrated within a High Bandwidth Memory (HBM) stack to accelerate Retrieval-Augmented Language Models (RALMs). The core idea is to partition the RALM workload according to its distinct computational characteristics and map each part to a specialized memory-centric compute unit. Specifically, the authors propose using Processing-In-Memory (PIM) on the HBM core die to accelerate the GEMV-heavy attention operations of the language model, while using Processing-Near-Memory (PNM) on the HBM logic die to accelerate the lookup- and sort-intensive tasks of the vector search retriever. This architecture is coupled with a novel scheduling policy, "Early Generation," which builds upon selective batching to overlap retrieval and generation phases, aiming to maximize throughput.

Strengths

The primary strength of this work lies in its elegant synthesis and co-design of existing concepts into a cohesive and well-motivated architecture.

1. Novelty in Integration: The central novel claim is the tight, heterogeneous integration of both PIM and PNM accelerators within a single HBM stack, with each component tailored to a different part of the RALM workload. While PIM and PNM have been proposed for these tasks separately, their co-location and co-design within one memory device to serve a single application is a distinct architectural proposition.
2. Workload-Specific Partitioning: The motivation for the heterogeneous design is well-argued. The paper correctly identifies the distinct bottlenecks of RALMs—GEMV-bound attention in the language model and irregular memory access patterns in the retriever (Section 3, page 5). The proposed mapping of these workloads to PIM (for its high internal bandwidth) and PNM (for its more complex logic capability) is logical and architecturally sound.
3. Hardware-Software Co-Design: The proposed "Early Generation" scheduling scheme appears to be a direct consequence of the underlying hardware's capabilities. The ability to perform concurrent PIM and PNM operations, enabled by features like the dual row buffer (Section 4.1, page 6), allows for a more aggressive overlapping than would be possible in systems with physically separate accelerators. This synergy between the proposed hardware and software is a notable strength.

Weaknesses

My primary concern is that while the integration is novel, the fundamental building blocks and conceptual approaches are largely derivative of prior art. The paper could do a better job of positioning its contribution against these closely related works.

1. Constituent Ideas are Not New: The core ideas, taken in isolation, are well-established.

   • PIM for Attention: Using HBM-PIM to accelerate GEMV operations in transformer attention is not a new idea. Prior works like AttAcc [80] and NeuPIM [24] have already established this approach. The PIM component of MNM appears functionally similar to these proposals.
   • PNM for Vector Search: Accelerating vector search using near-memory processing is also a known technique. Works like Chameleon [34] and FAANS [32] have proposed dedicated accelerators (on FPGAs or DIMMs) for IVF-PQ and other approximate nearest neighbor search algorithms. The PNM component of MNM addresses the same problem with a similar approach.

2. Conceptual Overlap with Prior Heterogeneous Systems: The concept of a heterogeneous PIM architecture for RALMs has been recently proposed. HeterRAG [60] specifically proposes combining HBM-based PIM for generation with DIMM-based PIM for retrieval. MNM's core conceptual contribution—partitioning the RALM workload across different PIM/PNM technologies—is therefore not entirely novel. The delta between MNM and HeterRAG is primarily in the implementation: MNM integrates both units within a single HBM stack, whereas HeterRAG uses separate memory systems. The paper fails to discuss this crucial piece of prior art and articulate the specific benefits of its tightly-coupled approach over HeterRAG's disaggregated one.

3. Incremental Novelty in Scheduling: The proposed scheduling scheme is an intelligent adaptation of existing techniques. The idea of overlapping retrieval and generation was explored in PipeRAG [35]. The concept of dynamically managing requests to improve utilization is the basis of continuous/selective batching, as seen in systems like Orca [99]. The authors' "Early Generation" scheme is a clever combination of these ideas, but its fundamental novelty is tied exclusively to its co-design with the MNM hardware rather than being a new scheduling paradigm in itself.

Questions to Address In Rebuttal

The authors should address the following points to clarify the novelty and significance of their contribution.

1. Comparison with HeterRAG [60]: Please provide a detailed, quantitative comparison against HeterRAG. What are the specific architectural and performance advantages of integrating both PIM and PNM within a single HBM stack versus HeterRAG's approach of using separate HBM-PIM and DIMM-PIM systems? Does the tight coupling provide benefits beyond reduced physical distance, such as a more unified programming model or lower-latency coordination?

2. Clarifying Scheduling Novelty: The "Early Generation" scheduling is enabled by the MNM architecture. Can the authors pinpoint the exact architectural feature(s) that make this scheduling scheme uniquely effective or even possible? For instance, is it the dual row buffer that allows for contention-free concurrent access, which would not be possible with other PIM designs? How would a PipeRAG-style scheduling perform on the MNM architecture, and vice-versa?

3. Generality of the PNM Accelerator: The PNM design is highly optimized for IVF-PQ-based retrieval. How adaptable is this design to other, more modern vector search algorithms like HNSW? A truly novel contribution would ideally show a path toward broader applicability. Is the PNM component a fixed-function unit for IVF-PQ, or is it a more general-purpose programmable engine for near-memory retrieval acceleration?

---

Review Form:

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors propose HEAT, a heterogeneous NPU-NDP architecture designed to accelerate inference for Transformer-empowered Graph Neural Networks (TF-GNNs). The architecture assigns the compute-intensive Transformer frontend to a Neural Processing Unit (NPU) and the memory-intensive Graph Neural Network (GNN) backend to a DIMM-based Near-Data Processing (NDP) unit. The paper introduces three primary optimizations: (1) a topology-aware encoding scheme that quantizes vertex features based on their degree; (2) a two-phase subgraph scheduling strategy to reduce DRAM bank conflicts and redundant memory accesses; and (3) a decoupled dataflow to enable parallel execution of the NPU and NDP. The authors claim significant speedup and energy efficiency gains over a high-performance GPU and state-of-the-art specialized accelerators.

Strengths

1. Problem Formulation: The paper correctly identifies a relevant and challenging problem. The performance bottleneck created by the sequential, two-phase nature of TF-GNNs—combining a compute-bound Transformer with a memory-bound GNN—is well-established. The roofline analysis in Figure 3(b) (page 4) provides clear motivation for a heterogeneous approach.

2. Architectural Concept: The high-level decision to map the Transformer to a compute-centric NPU and the GNN to a memory-centric NDP is logical and well-aligned with the computational characteristics of each model component.

Weaknesses

The paper's claims rest on a foundation of overly simplistic assumptions, questionable experimental design, and insufficiently justified heuristics. The reported performance gains appear to be a product of these weaknesses rather than a demonstrated architectural superiority.

1. Under-justified Topology-Aware Encoding: The central premise of the topology-aware encoding (Section 4, page 5) hinges on the simplistic assumption that vertex degree is a sufficient proxy for importance. The authors provide only a coarse masking experiment (Figure 4, page 4) as justification. This neglects more nuanced centrality metrics (e.g., PageRank, betweenness centrality) that often provide a more accurate measure of a node's influence in a graph. The choice of degree feels arbitrary and is not rigorously defended against alternatives, making the entire optimization seem superficial. Furthermore, the selection of the hyperparameter α (Section 6.4.2, page 11) is presented as a simple trade-off, but no sensitivity analysis is provided to show how this choice interacts with different graph structures or model architectures.

2. Questionable Optimality of Scheduling Heuristics: The proposed subgraph scheduling strategy (Section 5.3, page 7) is based on a series of greedy heuristics. While the intra-SG scheduling algorithm aims to balance bank accesses, its optimality is not discussed. The paper fails to quantify how close the resulting schedule is to an optimal or even a lower-bound solution for bank conflict minimization. Similarly, the inter-SG scheduling relies on a heuristic that "subgraphs sampled from neighboring starting points tend to have more overlapped vertices." The generalizability of this observation across diverse and complex graph structures is asserted, not proven. The entire scheduling component lacks theoretical grounding.

3. Unfair and Potentially Misleading Baselines: The primary weakness lies in the evaluation methodology (Section 6.1, page 9).

   • The "FACT+MEGA" baseline appears to be a strawman. The authors have constructed a baseline by serially chaining two independent, state-of-the-art accelerators. This naive composition is inherently limited by the sequential data dependency that HEAT is specifically designed to break. It does not represent a genuinely co-designed heterogeneous competitor and is therefore an unfair point of comparison. HEAT's outperformance of this baseline is expected and fails to demonstrate superiority over a competently designed alternative.
   • The performance comparison against the A100 GPU lacks critical details. It is unclear what level of software optimization (e.g., CUDA graph fusion, optimized libraries) was applied to the GPU baseline. A poorly optimized GPU baseline can artificially inflate the speedup of a custom accelerator.

4. Insufficient Modeling of System-Level Overheads: The evaluation relies on simulation (ONNXim, DRAMsim3), but the paper provides insufficient detail on the modeling of the interconnect between the host, NPU, and NDP. The "NPU Memory Bus" in Figure 7 (page 6) is a black box. The practical overheads of communication, synchronization, and data coherency in a real system with such a decoupled dataflow are non-trivial and could significantly erode the claimed performance gains. The paper seems to assume an idealized communication fabric.

5. Neglect of Dynamic Graph Scenarios: The paper dismisses the cost of its offline scheduling algorithms by comparing it to the one-time cost of model training (Table 5, page 9). This is a critical omission for inference workloads. In many real-world applications (e.g., social networks, recommendation systems), graphs are dynamic and evolve over time. The proposed offline approach would require frequent and costly re-computation, rendering it impractical. The paper completely fails to address the applicability of its methods to dynamic graphs.

Questions to Address In Rebuttal

1. Please provide a more rigorous justification for using vertex degree as the sole metric for importance in your topology-aware encoding. Have you evaluated other centrality metrics, and if so, how do they compare in the accuracy-performance trade-off?
2. Can the authors provide an analysis of the optimality of the greedy subgraph scheduling algorithm? For instance, how does it compare to a theoretical upper bound or a more complex scheduling method like simulated annealing for a small, representative graph?
3. Please defend the fairness of the FACT+MEGA baseline. Why is a simple sequential combination of two distinct accelerators a valid point of comparison for your co-designed architecture, rather than a hypothetical but more fairly integrated heterogeneous baseline?
4. Provide more details on the communication model assumed between the NPU and NDP. What bandwidth and latency are assumed for the "NPU Memory Bus," and how do these assumptions impact the overall performance, especially in the decoupled dataflow? Please conduct a sensitivity analysis on these parameters.
5. How would the proposed offline scheduling strategies adapt to dynamic graphs where the topology changes over time? What is the overhead of re-computing the schedule, and at what rate of graph change does this overhead nullify the benefits of the scheduling?

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

The authors present HEAT, a heterogeneous hardware architecture designed to accelerate Transformer-empowered Graph Neural Networks (TF-GNNs). The paper identifies a key challenge in this emerging model class: the front-end Transformer is compute-bound, while the back-end GNN is memory-bound, and existing accelerators typically focus on only one of these domains. HEAT addresses this by co-designing a Neural Processing Unit (NPU) for the Transformer and a Near-Data Processing (NDP) unit for the GNN.

The central contribution is not merely the hardware itself, but a set of co-design principles that exploit the coupling between the Transformer front-end and the GNN back-end. These include: (1) a topology-aware encoding scheme that uses vertex importance (degree) from the graph to apply variable precision quantization in the Transformer, reducing its computational load; (2) a subgraph scheduling strategy for the GNN that bundles and reorders subgraphs to mitigate DRAM bank conflicts and improve data locality; and (3) a decoupled dataflow that breaks the strict sequential dependency between the two stages, enabling parallel execution on the NPU and NDP. The paper demonstrates significant speedup and energy efficiency gains over a high-performance GPU and state-of-the-art specialized accelerators.

Strengths

1. Novel Cross-Domain Co-Optimization: The key intellectual contribution of this paper is the idea of using information from one domain (graph topology) to optimize computation in another (Transformer encoding). The topology-aware encoding scheme (Section 4, page 5) is an elegant insight. It recognizes that not all nodes in a graph are equally important and that this importance, relevant to the GNN, can be back-propagated to guide the precision of the feature extraction in the Transformer. This is a powerful co-design principle that moves beyond optimizing individual components in isolation.

2. A Holistic, System-Level Perspective: The authors have successfully avoided the pitfall of local optimization. Instead of just bolting a Transformer accelerator onto a GNN accelerator, they have re-architected the entire execution flow. The decoupled dataflow (Section 5.4, page 8), enabled by their heterogeneous hardware split, is a prime example of this. It directly addresses the pipeline bubble that would otherwise form due to the inherent dependency, showing a deep understanding of the full system's performance limiters.

3. Addresses a Timely and Forward-Looking Problem: This work is situated at the confluence of several important trends: the rise of large language models (Transformers), their integration into structured data analysis (GNNs), and the persistent need for specialized hardware. TF-GNNs represent a class of complex, multi-stage AI models that are becoming more prevalent. This paper provides not just a point solution, but a potential architectural template for accelerating future composite AI systems.

4. Strong Supporting Optimizations: The subgraph scheduling strategy (Section 5.3, page 7) is a well-reasoned and effective technique for tackling the notorious memory access challenges of GNNs. The distinction between intra-SG scheduling (for bank parallelism) and inter-SG scheduling (for temporal locality via a Hot Buffer) is particularly clever and demonstrates a nuanced approach to memory system optimization.

Weaknesses

My critiques are less about fundamental flaws and more about the boundaries and future implications of the proposed ideas.

1. Limited Exploration of "Importance" Heuristics: The entire topology-aware encoding scheme hinges on using vertex degree as a proxy for importance. While the authors validate this heuristic empirically (Figure 4, page 4), vertex degree is a relatively simple centrality measure. In more complex graphs, other metrics like PageRank, betweenness centrality, or learned attention scores might be better indicators of importance. The work would be strengthened by a discussion on the sensitivity of the architecture to this choice and its generalizability beyond degree-based importance.

2. Static Nature of the Optimizations: The proposed optimizations, particularly the topology-aware encoding and subgraph scheduling, appear to be performed offline based on a static graph structure. This is suitable for many inference scenarios, but the paper does not address how HEAT would perform in dynamic settings, such as social networks with evolving connections or real-time recommendation systems where graph structures change frequently. This static assumption limits the applicability to a subset of real-world problems.

3. Potential Programmability and Abstraction Challenges: The co-designed nature of HEAT, while powerful, implies a tight coupling between the algorithm and the hardware. It is unclear how a developer would program such a system for a new or slightly different TF-GNN model. The paper would benefit from a brief discussion on the software stack or compiler support required to map new models onto this heterogeneous architecture and manage its unique dataflow.

Questions to Address In Rebuttal

1. Could the authors comment on the sensitivity of their topology-aware encoding scheme to the choice of vertex importance metric? Have they considered or run preliminary experiments with other metrics (e.g., PageRank) and how might that impact the trade-off between computation reduction and accuracy?

2. The decoupled dataflow is a key enabler of parallelism. Could the authors provide more insight into the overhead and management of the "Encode Table" (Section 5.4, page 8)? Specifically, how does the system handle synchronization between the NPU and NDP to ensure vertex features are ready when needed without introducing significant control overhead?

3. How would the proposed subgraph scheduling strategy adapt to a streaming or dynamic graph scenario where the graph topology changes over time? Would the intra-SG and inter-SG schedules need to be recomputed frequently, and what would be the associated overhead?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper proposes HEAT, a heterogeneous NPU-NDP architecture designed to accelerate Transformer-empowered Graph Neural Networks (TF-GNNs). The authors identify the performance challenges arising from the compute-bound Transformer front-end and the memory-bound GNN back-end. HEAT's contributions are presented as a three-pronged optimization strategy: (1) A topology-aware encoding scheme that uses GNN vertex importance (proxied by node degree) to apply variable-precision quantization to the front-end Transformer. (2) A two-phase subgraph scheduling strategy for the GNN back-end that bundles subgraphs to mitigate DRAM bank conflicts and reorders execution to improve data locality for "hot" vertices. (3) A decoupled dataflow that enables pipelined execution by having the Transformer on the NPU generate only the vertex features currently required by the GNN on the NDP.

My analysis concludes that the work contains one core, genuinely novel idea—the co-design linking graph topology to Transformer quantization. The other two contributions, while well-engineered and effective, are applications of well-established principles from the systems and architecture communities to the specific TF-GNN workload. The novelty in those areas lies in the specific heuristics and implementation, rather than in the concepts themselves.

Strengths

1. Novel Cross-Domain Optimization: The most significant and novel contribution is the topology-aware encoding algorithm (Section 4, page 5). The idea of using a structural property from the back-end graph computation (node degree) to guide the precision of the front-end text-encoding Transformer is a genuinely new co-design principle. Prior work on Transformer quantization is typically data-driven (e.g., ANT), while GNN quantization focuses on features within the GNN domain (e.g., MEGA). Linking the two is a clever insight specific to the TF-GNN model class and represents a clear conceptual advance.

2. Holistic System Co-design: The authors present a complete system that addresses bottlenecks across the entire TF-GNN pipeline. Rather than focusing on just the Transformer or just the GNN in isolation, the architecture and its accompanying algorithms consider the interplay between the two, which is commendable.

3. Well-Defined Problem and Solution: The paper does an excellent job of motivating the problem with clear profiling data (Figure 3, page 4) and systematically proposing solutions for each identified challenge. The connection between an observation (e.g., irregular memory access) and the proposed solution (e.g., subgraph scheduling) is logical and well-articulated.

Weaknesses

My critique focuses on the degree of conceptual novelty of the latter two contributions, which appear to be expert applications of existing ideas rather than the creation of new ones.

1. Incremental Novelty in Subgraph Scheduling: The subgraph scheduling strategy (Section 5.3, page 7), while effective, is conceptually an extension of known techniques. The problem of optimizing GNN memory access by reordering computations to improve locality and mitigate bank conflicts is well-trodden ground in GNN accelerator literature (e.g., I-GCN [12], GraNDe [59]). The authors' contribution is a specific two-phase greedy heuristic: bundling subgraphs based on complementary Bank_MAX/Bank_MIN access patterns is a new algorithm, but the underlying principle of co-scheduling tasks to balance resource utilization is a classic systems problem. Similarly, reordering based on neighboring starting points to exploit "hot vertices" is a direct application of the principle of temporal locality. The novelty here is algorithmic and heuristic, not conceptual.

2. Decoupled Dataflow as Applied Pipelining: The proposed decoupled dataflow (Section 5.4, page 8) is a direct application of producer-consumer pipelining to the TF-GNN workload. The "naive dataflow" (Figure 11b) is a classic bulk-synchronous process. The "decoupled dataflow" (Figure 11c) introduces fine-grained pipelining by enabling the consumer (NDP) to start as soon as the producer (NPU) provides the first piece of necessary data. This is a fundamental optimization in computer systems. The use of an "Encode Table" to avoid re-computation is a form of memoization, another standard technique. While this is excellent system engineering and crucial for performance on their heterogeneous architecture, it does not represent a fundamentally new concept in dataflow or execution models.

3. Complexity vs. Benefit Justification: The system introduces significant offline and online complexity (topology analysis, two-phase scheduling, inter-module control for the decoupled dataflow). The performance gains are substantial. However, much of this gain comes from applying known systems principles. The primary novel idea (topology-aware quantization) provides a 1.5x speedup in the ablation study (Figure 17, page 11, base to v1). The subsequent, more conventional optimizations (scheduling and pipelining) provide the remaining boost to the final performance. This suggests that while the system is effective, its performance is built upon a single core novel idea augmented by strong, but standard, engineering.

Questions to Address In Rebuttal

1. On Subgraph Scheduling: Could the authors please clarify the conceptual novelty of the intra-SG scheduling algorithm beyond being a new greedy heuristic for the well-known problem of DRAM bank-conflict reduction in irregular applications? What is the core insight that distinguishes it from prior work on data rearrangement or task scheduling for GNNs or other graph algorithms?

2. On the Generality of the Core Novelty: The paper's primary novel claim hinges on using node degree as a proxy for vertex importance to guide Transformer quantization. This is compelling for the social network and citation network graphs evaluated. How does this principle generalize to graph types where high-degree nodes are not necessarily the most informative (e.g., hubs in protein-protein interaction networks, or in knowledge graphs where relation types are more important than connectivity)? Is there a risk of this optimization being a domain-specific heuristic rather than a general principle for TF-GNNs?

3. On System-Level Conflicts: The decoupled dataflow necessitates fine-grained, on-demand feature generation from the Transformer on the NPU. This contrasts with conventional high-performance Transformer inference, which relies heavily on batching large numbers of inputs to maximize PE utilization. Does this on-demand execution model introduce significant throughput degradation or efficiency loss on the NPU, and if so, how does HEAT mitigate this? Is there a trade-off between latency reduction from pipelining and the NPU's overall throughput?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose RayN, a near-memory computing (NMC) architecture to accelerate ray tracing, specifically Bounding Volume Hierarchy (BVH) traversal, by placing specialized RT units on the logic layer of 3D-stacked DRAM. The paper identifies memory latency from pointer-chasing as the primary bottleneck and argues that NMC is a suitable solution. It explores three memory controller designs (JEDEC-compatible, Unified, Hybrid), a BVH partitioning heuristic to mitigate load imbalance, and evaluates the performance, energy, and area implications.

While the problem is well-motivated, the proposed solutions rest on several questionable assumptions. The most performant architectural proposals require non-standard memory interfaces with no clear path to adoption. The core technical contribution of load balancing is based on a weak heuristic, which is demonstrated to be ineffective by the authors' own results. Finally, the energy and area claims are undermined by a flawed evaluation methodology that omits key components from the analysis.

Strengths

1. Problem Identification: The paper correctly identifies and quantifies that BVH traversal is highly sensitive to memory latency (Figure 1, page 1). The limit study (Figure 3, page 3) provides a reasonable, albeit optimistic, upper bound on the potential for improvement.
2. Architectural Exploration: The consideration of three different memory controller architectures (Section 3.1, pages 4-5) demonstrates a thoughtful approach to the design space, even if the conclusions drawn from this exploration are problematic.
3. Simulation Infrastructure: The use of a detailed, cycle-level simulator (Vulkan-Sim) integrated with a memory simulator (Ramulator) is appropriate for this type of architectural study.

Weaknesses

1. Impractical Architectural Assumptions: The highest-performing "Unified" and "Hybrid" architectures (Section 3.1.2, 3.1.3, page 5) are explicitly not compliant with the HBM JEDEC standard. They require fundamental changes to memory protocols, pins, and host-side controllers. Such proposals are largely academic exercises without a convincing argument for how the entire GPU and memory manufacturing ecosystem would adopt these changes. The "JEDEC-compatible" design is a poor alternative, requiring full BVH duplication and disabling memory channel interleaving, making it a performance strawman.

2. Ineffective Load Balancing Heuristic: The paper claims that a key challenge is partitioning the BVH to mitigate load imbalance. However, the proposed solution is demonstrably weak.

   • The load estimation metric proposed in Equation 1 (page 8) is Volume(root node) * depth(partition). The authors' own analysis in Figure 10 (page 8) shows a very weak correlation between this metric and the actual intersection count, with an average correlation of approximately 0.6 and much lower for several scenes. A heuristic with such low predictive power is fundamentally flawed.
   • Figure 14 (page 11, bottom) provides direct evidence of this failure. It shows the ratio of maximum to minimum intersection tests across memory modules. A perfectly balanced system would have a ratio of 1. The authors report an average ratio of 3.14 for their BLAS Breaking method. This indicates a severe load imbalance remains, directly contradicting the paper's claims of effective partitioning.

3. Unsubstantiated Energy Savings: The claim of a 70% average energy reduction (Figure 17, page 11) is based on a critical methodological omission. As stated in Section 5 (page 9), the "Power usage of near-memory RT units is not measured." The analysis only accounts for the reduction in data movement energy, while completely ignoring the power consumption of the newly added compute units. Placing active logic on the HBM die is a significant thermal challenge, and to ignore its power contribution makes the entire energy analysis unreliable and misleading.

4. Questionable Area Overhead Analysis: The area estimation for the near-memory RT unit (Section 6.3, page 11) is derived by scaling a 65nm mobile GPU architecture (SGRT [51]) to a 12nm process. Technology scaling laws are notoriously unreliable across such a vast gap in process nodes, design goals (mobile vs. high-performance logic), and transistor types. The resulting claim of a trivial 0.78% area overhead is likely a significant underestimate and lacks sufficient rigor.

5. Ambiguous Performance Claims: The abstract claims a "3.0x speedup on average," but Figure 11 (page 9) shows this is the result of their best-performing, non-standard architecture (H+BB) compared to the baseline. The speedup over the most relevant prior work, Treelets [21], is closer to 2.0x. This represents a form of "resultsmanship" that inflates the contribution.

Questions to Address In Rebuttal

1. Regarding the Unified and Hybrid architectures: Beyond stating that they are non-standard, what is the realistic path to industry adoption for a proposal that requires coordinated changes from GPU vendors, memory manufacturers, and standards bodies like JEDEC?
2. Given the poor correlation shown in Figure 10 and the severe measured imbalance in Figure 14, how can the authors justify that their load estimation heuristic (Equation 1) is a valid or meaningful contribution? Why were more sophisticated, possibly runtime-informed, balancing strategies not considered?
3. Please provide a detailed justification for omitting the power consumption of the near-memory RT units. Can the authors provide a sensitivity study or a first-order estimation of this power and re-evaluate their energy savings claims, accounting for the strict thermal design power (TDP) constraints of an HBM logic die?
4. Can the authors provide a more robust justification for their area scaling methodology? For instance, by comparing their scaled estimates to known block sizes from more modern, publicly available die shots of high-performance logic.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents RayN, a near-memory computing architecture designed to accelerate ray tracing by tackling the latency bottleneck of Bounding Volume Hierarchy (BVH) traversal. The core contribution is a holistic system co-design that places specialized, but simple, Ray Tracing (RT) units within the logic layer of 3D stacked DRAM (e.g., HBM). This approach is supported by two key pillars: 1) a thoughtful exploration of memory controller architectures (JEDEC-compatible, Unified, and Hybrid) to manage concurrent access between the host GPU and the near-memory units, and 2) a novel software partitioning algorithm, "BLAS Breaking," that subdivides the scene geometry's acceleration structure to improve load balancing across memory modules. The authors demonstrate through simulation that RayN can achieve an average speedup of 3.0x over a conventional GPU baseline and a 2.2x speedup over a state-of-the-art prefetching technique, all while reducing energy consumption by 70% and incurring a negligible area overhead of ~0.78%.

Strengths

This is a well-executed and compelling piece of work that sits at the intersection of computer graphics, architecture, and the growing field of near-memory computing. Its primary strengths are:

1. A Holistic and Pragmatic System View: The authors' most significant contribution is not just proposing "PIM for ray tracing," but thoughtfully considering the entire system stack. The exploration of three different memory controller designs in Section 3.1 (page 5) is a standout feature. It demonstrates a deep awareness of the practical challenges of deploying near-memory solutions, weighing the trade-offs between standard compliance (JEDEC), raw performance (Unified), and a practical compromise (Hybrid). This elevates the work from a purely academic exercise to a plausible architectural proposal.

2. Connecting Two Converging Trends: The paper effectively synthesizes two major trends in high-performance computing: the specialization of hardware for graphics (dedicated RT cores) and the push towards processing-in-memory/near-memory computing (PIM/NMC) to overcome the memory wall. By identifying BVH traversal as a memory-latency-bound, pointer-chasing problem (as strongly motivated by Figure 1), the authors find a "killer app" for NMC in the graphics domain, which has historically driven memory system innovation (e.g., GDDR).

3. Novel and Well-Motivated Algorithmic Co-design: The "BLAS Breaking" partitioning scheme described in Section 4.1 (page 7) is an elegant solution. Instead of reinventing the wheel, it cleverly adapts the existing TLAS/BLAS dichotomy found in modern graphics APIs like Vulkan and DirectX. By breaking down large BLASes into smaller, more numerous partitions, the system can achieve better load balancing across the distributed near-memory RT units, as shown in Figure 9. This synergy between the hardware proposal and the software algorithm is a key strength.

4. Strong and Clearly Attributed Results: The reported 3.0x speedup is substantial. Crucially, the authors provide excellent analysis to explain why this speedup is achieved. Figure 12 clearly shows the latency reduction for near-memory accesses, while Figure 13 demonstrates a massive (78%) reduction in memory transactions issued by the GPU's main RT units. This detailed breakdown makes the performance claims highly credible and provides valuable insight into the system's behavior. The minimal area overhead and significant energy savings further solidify the proposal's value.

Weaknesses

The paper is strong, but its potential impact could be further clarified by addressing a few points where the current analysis feels incomplete.

1. Oversimplified Load Balancing Heuristic: The core of the memory placement strategy relies on the load estimation heuristic in Equation 1 (page 8). As the authors honestly show in Figure 10, the correlation between this estimate and the actual work is modest. While the sensitivity study suggests the system is robust even in a worst-case imbalance, this heuristic remains the Achilles' heel of the software design. The performance gains could be highly dependent on camera paths and scene layouts that happen to align well with this simple volume-times-depth metric. The work would be stronger if it explored or at least discussed more sophisticated alternatives.

2. Limited Scope of Dynamic Workloads: The evaluation is based on rendering multiple frames of a scene by changing the camera position, which simulates some level of dynamism. However, real-time gaming and interactive applications feature far more complex dynamics, including object deformation, destruction, and streaming assets. These scenarios would necessitate frequent refitting or rebuilding of BLASes. The paper does not analyze the overhead of re-partitioning and re-distributing these dynamic BLASes across memory modules, which could potentially erode the performance gains in highly fluid scenes.

3. Positioning Relative to Cache-Centric Solutions: The paper positions itself against a strong prefetching baseline (treelets). However, it could better contextualize its contribution by briefly discussing how RayN compares, conceptually, to a brute-force approach of simply scaling up on-GPU caches (L2/L3). While NMC is likely the superior path for this kind of irregular access pattern, explicitly stating why—for instance, the inefficiency of caching vast, sparsely accessed BVH trees versus moving the computation—would strengthen the argument that RayN represents a fundamentally better architectural direction, not just an alternative one.

Questions to Address In Rebuttal

1. Regarding the load estimation heuristic (Equation 1, page 8): Have the authors considered alternative or complementary metrics? For example, in a real-time context, could profiling data from the N-1 frame be used to dynamically adjust the load estimates for the Nth frame, creating a more adaptive and accurate placement strategy over time?

2. The paper's design partitions the BVH tree geographically. Could it also be partitioned based on ray type? For instance, in many scenes, a small number of complex objects are responsible for most reflection/refraction effects. Could BLASes for these "hero" objects be handled differently (e.g., duplicated or prioritized) to optimize the traversal of secondary rays, which often exhibit less coherence than primary rays?

3. How does the proposed system handle the interaction between BVH traversal and programmable shaders (e.g., intersection shaders or any-hit shaders)? The paper mentions support for them (Section 4.3, page 8), but running complex, arbitrary shader code on the simple near-memory RT units seems challenging. Could you elaborate on the limitations of the programmable logic in the near-memory units and the mechanism for handing off to the main GPU shader cores if a complex shader is encountered?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper proposes RayN, a near-memory computing architecture designed to accelerate ray tracing. The core idea is to place dedicated ray tracing hardware units (RT units), functionally similar to those on a modern GPU die, directly into the logic layer of 3D-stacked memory modules (e.g., HBM). The stated goal is to mitigate the high memory latency inherent in the pointer-chasing nature of Bounding Volume Hierarchy (BVH) tree traversal.

To support this architectural proposal, the authors introduce three distinct memory controller configurations (JEDEC-compatible, Unified, Hybrid) to manage concurrent memory access from the host GPU and the near-memory RT units. Furthermore, they propose a novel BVH partitioning algorithm, "BLAS Breaking," which leverages the existing TLAS/BLAS API structure to divide the BVH tree across multiple memory modules. This partitioning is guided by a new load-balancing heuristic. The authors claim their Hybrid configuration with BLAS Breaking achieves an average speedup of 3.0x over a baseline GPU.

Strengths

The primary strength of this paper lies in the novel synthesis of two existing, but separate, technology trends: specialized hardware for ray tracing and near-memory computing.

1. Novelty of Architectural Synthesis: While near-memory computing (NMC) is a well-explored field, and on-die RT units are now industry standard, the proposal to place specialized RT accelerator logic directly on the memory logic die appears to be a genuinely novel concept. Prior work on NMC for GPUs has typically focused on offloading kernels to general-purpose cores (e.g., Hsieh et al., "TOM," ASPLOS 2016, ref [37]) or accelerating different domains like graph processing (e.g., Ahn et al., ISCA 2015, ref [7]). This paper correctly identifies a key workload (BVH traversal) that is an excellent candidate for specialized NMC and proposes a specific, bespoke hardware solution.

2. Novelty in Problem-Specific Algorithms: The architectural idea is supported by a novel partitioning algorithm tailored specifically for BVH trees. The "BLAS Breaking" method (Section 4.1, page 7) is a clever adaptation that leverages domain knowledge of how graphics APIs already structure scenes. The load estimation heuristic presented in Equation 1 (page 8), Volume(root node) × depth(partition), is a simple but new contribution for predicting ray tracing workload distribution without runtime camera information. This demonstrates a complete system view, moving beyond just the hardware placement.

Weaknesses

While the central synthesis is novel, several of the supporting components are derivative of prior art, and the evaluation does not sufficiently defend the necessity of the proposed specialized hardware against alternative NMC approaches.

1. Incremental Novelty of Memory Controller Designs: The three controller architectures presented in Section 3.1 (pages 4-5) are not fundamentally new paradigms for managing shared memory access in an NMC system. The "JEDEC-compatible" design, which partitions memory channels, is a standard technique to avoid structural hazards in heterogeneous systems. The "Unified" controller is conceptually similar to architectures where the main memory controller is moved off-host. The "Hybrid" model is a pragmatic engineering compromise between the two. The problem of enabling concurrent access for near-data accelerators and a host is well-known, with prior work like Cho et al. (ISCA 2020, ref [20]) exploring similar challenges. The novelty here is in the application and trade-off analysis, not in the underlying controller concepts.

2. Limited Novelty and Efficacy of the Partitioning Heuristic: The proposed load-balancing heuristic (Equation 1, page 8) is acknowledged to have a "small" correlation with the actual measured load (Figure 10, page 8). The results confirm this, showing a remaining load imbalance with a max/min ratio of 3.14 on average (Figure 14, page 11). While the heuristic itself is new, it is a very simple model. The field of workload prediction and cost modeling for tree traversal is mature, particularly in the database domain (e.g., query plan optimization). The paper fails to justify why a more sophisticated model was not explored, which weakens the contribution of this novel, but seemingly ineffective, algorithm.

3. Failure to Contrast with Software on General-Purpose NMC: The paper's core claim rests on the need for specialized near-memory RT units. However, it fails to provide a comparison against a functionally similar system that uses general-purpose near-memory cores, such as those proposed in TOM (ref [37]) or implemented in commercial products like UPMEM's PIM system (ref [4]). The authors state that prior NMC graph architectures "lack the specialized accelerator hardware required" (Section 1, page 1), but this is an assertion, not a quantified conclusion. Without data showing that a software-based BVH traversal on a general-purpose near-memory core is insufficient, the central novel claim—that specialized hardware is the right solution—remains unsubstantiated. The delta between this work and a software-based NMC approach is unclear.

Questions to Address In Rebuttal

1. Regarding the memory controller designs (Section 3.1), please clarify the precise delta between your proposed solutions and the state-of-the-art in arbitrating between a host processor and a near-memory accelerator. Beyond the application to ray tracing, what is the fundamental novel contribution in the controller logic or protocol itself?

2. The proposed load-balancing heuristic (Equation 1) shows weak correlation to the actual load. Can the authors justify the decision not to explore more advanced predictive models? Have you considered prior art in cost estimation for tree-based data structures from other domains (e.g., databases) or even simple machine learning models trained offline on representative camera paths?

3. The central thesis is that specialized near-memory hardware is required. To justify this novel hardware proposal, please provide a quantitative comparison or a well-reasoned estimate of the performance of a software-based BVH traversal algorithm running on an array of general-purpose, low-power cores in the memory logic die (a configuration proposed by prior work such as ref [37] or ref [71]). Without this, it is difficult to assess whether the complexity of adding new, specialized RT units is justified over a more flexible software-based approach.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes PIMBA, a Processing-in-Memory (PIM) accelerator designed to serve both transformer and "post-transformer" Large Language Models (LLMs), such as those based on State Space Models (SSMs). The authors' central thesis is that a common "state update" operation in post-transformer models is memory-bandwidth-bound, similar to the attention mechanism in transformers. PIMBA's architecture is based on two primary ideas: (1) a State-update Processing Unit (SPU) shared between two DRAM banks using an interleaved access pattern to improve utilization, and (2) the use of MX8 quantized arithmetic to achieve a better accuracy-area tradeoff compared to other low-precision formats. The authors claim significant throughput improvements over GPU and GPU+PIM baselines with minimal accuracy loss.

Strengths

1. The paper correctly identifies a relevant and timely research direction: accelerating the emerging class of post-transformer LLMs.
2. The workload characterization in Section 3, which identifies the state update operation as a potential bottleneck under batched inference, provides a reasonable starting point for the investigation.
3. The analysis considers the critical trade-off between quantization-induced accuracy degradation and hardware area overhead (Section 4.2), which is a necessary component of any proposal involving low-precision arithmetic.

Weaknesses

1. Unsupported Foundational Motivation: The paper's motivation hinges on the claimed superiority of post-transformer models. Figure 1(a) presents a comparison where Mamba-2 achieves 4.5% higher accuracy than a baseline transformer. This result is cited from an external source [15] without providing the necessary context to validate its fairness (e.g., equivalent training compute, data, and model tuning). Without this validation, the entire premise that the community should invest in specialized hardware for these models is built on a potentially specious claim.

2. Oversimplification of the "State Update" Primitive: The paper generalizes the core operations of diverse architectures (SSMs, linear attention, RNNs) into a single "state update" primitive, formalized in Equation 2 (Page 4). This abstraction is a significant simplification. It glosses over key differences, such as the scalar decay in Mamba-2 versus the vector-based gating in GLA. The paper provides no evidence or sensitivity analysis to demonstrate that this single, generalized hardware implementation can efficiently serve these varied computational patterns without significant performance penalties or architectural compromises for one model type versus another.

3. Insufficient Justification for the Choice of MX8: The argument for using MX8 over int8 rests on the claim that int8 incurs "substantial area overhead" due to the need for dequantization and requantization for addition operations (Section 4.2, Page 6). This claim is asserted but not rigorously substantiated. The paper fails to provide a quantitative, apples-to-apples area comparison between the components of their proposed MX Adder (Figure 9b), which includes shifters and comparison logic, and a standard int8 datapath for the same function. Without this direct comparison, the claim that MX8 is Pareto-optimal remains an unproven assertion.

4. Weak and Potentially Biased Baselines: The experimental comparison relies on baselines that appear to be deliberately weakened. The GPU+PIM baseline is described as a "time-multiplexed design" that explicitly lacks the "access interleaving technique of PIMBA" (Section 6.1, Page 10). This constructs a strawman; a fair comparison would be against a more aggressively pipelined PIM baseline, allowing the reader to properly assess the incremental benefit of PIMBA's specific design choices. By comparing against a seemingly self-crippled baseline, the reported speedups of up to 2.1x are likely inflated.

5. Questionable Architectural Novelty: The core architectural proposal of sharing a processing unit between two banks using "access interleaving" (Section 5.2, Page 7) is presented as a key innovation. However, both resource sharing to amortize area cost and interleaving to mitigate memory bank contention are foundational techniques in computer architecture and parallel systems. The paper fails to adequately position this contribution with respect to prior art, thereby overstating its novelty.

Questions to Address In Rebuttal

1. Regarding the core motivation in Figure 1: Can you provide concrete evidence that the 2.7B parameter transformer and Mamba-2 models are fairly compared in terms of training data, total training FLOPs, and hyperparameter tuning? If not, how does this uncertainty affect the premise of your work?

2. Regarding the generalized state update primitive (Equation 2): Please provide a quantitative analysis of the performance and area efficiency of your proposed SPU when executing the specific, non-generalized operations of RetNet, GLA, and HGRN2. How much efficiency is lost by forcing these distinct operations onto your unified hardware?

3. Regarding the choice of MX8 over int8: Please provide a detailed, post-synthesis area breakdown of the MX Adder and MX Multiplier versus an equivalent int8 datapath that includes the necessary dequantization/requantization logic. This data is required to substantiate the claim of Pareto optimality made in Figure 6.

4. Regarding the GPU+PIM baseline: Please justify the decision to use a non-interleaved, "time-multiplexed" design as your primary PIM baseline. Why was a more competitive, pipelined PIM design not used for comparison? How much of your reported speedup is attributable to simply having a better pipeline structure versus the baseline you designed?

5. Regarding the accuracy results in Table 2: Was any form of quantization-aware training or post-quantization fine-tuning used to achieve the reported near-zero accuracy degradation? If only post-training quantization was used, the results are unusually strong and require further explanation and validation.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents PIMBA, a Processing-in-Memory (PIM) accelerator designed for serving both existing Transformer-based Large Language Models (LLMs) and the emerging class of "post-transformer" models (e.g., State Space Models like Mamba-2, linear attention).

The authors' core contribution is founded on a crucial unifying insight: despite their algorithmic differences, the performance of both model classes during batched inference is fundamentally bottlenecked by memory bandwidth. For Transformers, this is the well-known attention mechanism; for post-transformers, it is a newly identified bottleneck in the "state update" operation.

Based on this, the authors propose a unified PIM architecture that can efficiently execute both types of operations. The design is guided by two key principles derived from their analysis: (1) The varied primitives in state update operations make per-bank PIM logic area-inefficient, motivating a shared "State-update Processing Unit" (SPU) that interleaves access between two banks. (2) Post-transformer models are sensitive to quantization, and the authors identify Microsoft's MX format (specifically MX8) as a Pareto-optimal choice for balancing accuracy and hardware area in a PIM context. The resulting system, PIMBA, demonstrates significant throughput improvements over GPU and GPU+PIM baselines while maintaining model accuracy and adhering to practical area constraints.

Strengths

1. Timeliness and Important Vision: This work is exceptionally timely. As the research community actively seeks alternatives to the quadratically scaling Transformer, the question of how to build hardware that supports this transition is critical. The paper's vision of a unified serving system that bridges the gap between today's Transformers and tomorrow's post-transformers is both ambitious and highly valuable. It provides a practical roadmap for evolving our hardware infrastructure.

2. Excellent Unifying Abstraction: The paper's primary strength lies in its abstraction of the core performance problem. By identifying the "state update" as the conceptual parallel to "attention" and demonstrating through roofline analysis (Figure 1b, page 2) and latency breakdowns (Figure 3, page 4) that both are memory-bound, the authors distill a complex landscape into a single, addressable hardware challenge. This insight forms a powerful foundation for their entire work.

3. Principled, Data-Driven Hardware Design: The design of PIMBA is not arbitrary; it follows directly from well-articulated principles and thorough analysis.

   • The decision to share an SPU between two banks is a clever solution to the area-throughput tradeoff identified in their analysis of pipelined vs. time-multiplexed PIM designs (Section 4.1, page 6).
   • The selection of the MX8 format is convincingly justified through a detailed accuracy-area tradeoff analysis (Figure 6, page 6), which clearly shows its superiority over other formats like int8 (too much area) and low-precision floating point (poor accuracy) for this specific workload. This is an excellent piece of co-design.

4. Comprehensive Scope and Evaluation: The authors evaluate their proposal against a broad set of modern architectures—four distinct post-transformer models, a hybrid model (Zamba2), and a traditional Transformer (OPT). The evaluation across multiple scales (up to 70B parameters) and on a wide range of metrics (throughput, latency, energy, accuracy, and area) lends significant credibility to their claims. The minimal accuracy degradation shown in Table 2 (page 11) is particularly compelling evidence for the viability of their chosen quantization strategy.

Weaknesses

My concerns are less about flaws in the existing work and more about the boundaries of its claims and its integration into the broader systems landscape.

1. Robustness of the "State Update" Generalization: The authors unify several post-transformer operations into a single generalized state update form (Equation 2, page 4). This is elegant and effective for the models studied. However, the post-transformer field is nascent and evolving rapidly. It is conceivable that a future dominant architecture might introduce primitives (e.g., complex non-linearities, different data dependencies) that do not map cleanly to this structure. This could limit the future-proofing of the PIMBA design.

2. System-Level Integration Challenges: The paper effectively addresses the accelerator microarchitecture but is lighter on its integration into a full-fledged, dynamic serving system. Modern LLM serving schedulers (like those in vLLM or Orca) use sophisticated techniques like continuous batching and preemption to maximize GPU utilization. The paper acknowledges that PIMBA operates in a "blocked manner" with the GPU (Section 8, page 13), which inherently leads to utilization gaps. While the authors suggest leveraging techniques from NeuPIMs [28], a more detailed discussion of how PIMBA's deterministic, command-based execution model would coexist with the highly dynamic and asynchronous nature of a production-level scheduler would strengthen the paper's system-level contribution.

3. The Software Hurdle: As with all novel hardware, the software and programmability aspect is a major barrier to adoption. The paper mentions extending CUDA and defining custom DRAM commands (Section 5.1, page 7), which is a non-trivial engineering effort. A deeper contextualization of the required software changes would provide a more complete picture of the path to real-world deployment.

Questions to Address In Rebuttal

1. Regarding the generalized state update operation (Equation 2): Could the authors comment on the potential limitations of this formulation? Have they considered any emerging post-transformer algorithms that might challenge this abstraction, and how might PIMBA be adapted to accommodate them?

2. Could the authors elaborate on the system-level scheduling vision? How would a system using PIMBA-enabled memory efficiently manage the pipeline bubbles created during the hand-offs between GPU and PIM execution, especially in a dynamic, multi-user environment managed by a sophisticated scheduler?

3. The Pareto-optimality of the MX8 format was compellingly demonstrated for the Mamba-2 model (Figure 6). Was this detailed tradeoff analysis also performed for the other SU-LLMs (e.g., RetNet, GLA)? Confirming that MX8 remains the optimal choice across this diverse set of models would further strengthen this key design decision.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents PIMBA, a Processing-in-Memory (PIM) accelerator designed to serve both transformer and, more critically, "post-transformer" Large Language Models (LLMs) like State Space Models (SSMs) and their variants. The authors first identify a common, memory-bound "state update" operation that unifies various post-transformer architectures. They then propose a novel PIM architecture to accelerate this operation alongside standard attention. The core architectural claims of novelty are (1) a State-update Processing Unit (SPU) that is shared between two memory banks, using an "access interleaving" technique to mitigate read/write hazards and maximize throughput within a constrained area budget, and (2) a State-update Processing Engine (SPE) within the SPU that uses custom microarchitecture for element-wise multiplication and addition on the MX low-precision data format, extending its use beyond its original dot-product design.

Strengths

The primary strength of this paper lies in its well-defined and well-defended novel contributions at multiple levels of abstraction.

1. Novel Workload Abstraction: The identification and generalization of the "state update" operation (Equation 2, Section 3.1, page 4) across a diverse set of emerging post-transformer models (Mamba-2, GLA, RetNet, HGRN2) is a significant conceptual contribution. While prior work has analyzed individual models, this paper is the first I have seen to propose a unified hardware target based on this common algorithmic pattern. This insight alone provides a strong foundation for the work.

2. Novel Architectural Technique: The "access interleaving" mechanism where a single SPU is shared between two banks (Section 5.2, page 7) is an elegant and novel solution to a practical PIM design problem. The authors correctly identify the area-throughput tradeoff between naive time-multiplexed and fully-pipelined per-bank designs (Section 4.1, page 6). Their proposed solution, which pipelines operations by alternating reads and writes between two banks into a single SPU, effectively achieves the throughput of a per-bank design with roughly half the area overhead. This is a clever application of pipeline hazard mitigation in a new domain.

3. Novel Microarchitectural Design: The paper's novelty extends to the microarchitecture of the SPE (Section 5.3, page 8). While the MX format itself is not new [16], its application has been predominantly for GEMM/dot-product operations. The authors’ contribution is the design of bespoke MX Multiplier and MX Adder units for element-wise operations, which are critical for the "state update" primitive. This extension of the MX format's utility is a non-trivial and novel piece of engineering that is directly justified by their compelling area-accuracy tradeoff analysis (Figure 6, page 6).

4. Novel Empirical Insights: The quantization analysis (Section 3.2, page 5) provides a genuinely new finding. The observation that post-transformer state updates are highly susceptible to the "swamping effect" with standard low-precision floating-point formats (e.g., e4m3, e5m2) is a crucial insight that distinguishes this workload from transformer KV cache quantization. The subsequent identification of MX8 with stochastic rounding as a Pareto-optimal solution in the context of PIM area constraints is a strong, data-backed novel claim.

Weaknesses

My criticisms are primarily focused on contextualizing the novelty and identifying elements that are more evolutionary than revolutionary.

1. Incremental System Integration: The overall system design, particularly the software stack and host interface, heavily leverages prior art. The authors explicitly state their system architecture is "similar to the existing PIM-based LLM serving systems [28, 54, 55, 67]" and that the software stack is "based on a prior work, HBM-PIM [40]" (Section 5.1, page 6). The proposed custom DRAM commands (ACT4, REG_WRITE, etc., in Section 5.5, page 8) are logical extensions of command-based PIM interfaces seen in prior works and do not represent a fundamental shift in the PIM execution model. The novelty is clearly in the PIM logic itself, not its system-level integration.

2. Application of a Known Principle: The concept of interleaving accesses between memory banks to improve functional unit utilization is a classic computer architecture principle. While its application to solve the state update read/write hazard in PIM is novel and well-executed, the underlying idea of hiding latency by finding parallelism between independent resources is not fundamentally new. The paper would be strengthened by acknowledging this principle and more clearly articulating how its constraints and implementation differ in the PIM context.

Questions to Address In Rebuttal

1. On the Novelty of Access Interleaving: The proposed SPU shared between two banks is a cornerstone of your architectural contribution. Can you clarify if any prior work in the broader near-data processing or PIM literature has employed a similar shared-unit, interleaved-bank access scheme to resolve read-after-write or write-after-read hazards, even if for a different application (e.g., database operations, graph processing)?

2. On the Generality of the "State Update" Primitive: Your design is predicated on the "state update" abstraction (Equation 2). This holds for the current crop of post-transformer models. However, this field is evolving rapidly. How robust is your architecture to potential future post-transformer models that may deviate from this specific pattern of decay ⨀ state + outer(k, u)? Is the SPE's functionality general enough, or is PIMBA's novelty tightly coupled to this specific formulation?

3. On the Microarchitectural Delta of the MX SPE: The paper proposes custom MX multipliers and adders (Figure 9, page 8). Beyond the logic for handling the shared exponents and microexponents, how much of the core datapath (i.e., the mantissa computation) differs from standard integer multiplication/addition? Please quantify the "delta" in hardware complexity between your proposed units and a hypothetical design that dequantizes MX to a shared internal format (e.g., FP16), performs standard FP operations, and then re-quantizes. This would help solidify the claimed area benefits.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present "GateBleed," a purported timing side channel in Intel's Advanced Matrix Extensions (AMX) on 4th Gen Xeon CPUs. The central claim is that an undocumented, staged power gating mechanism creates distinct, reuse-distance-dependent latencies for AMX instructions. The paper attempts to build upon this primitive to demonstrate three classes of attacks: AI privacy leaks (membership inference, MoE expert routing), a generic microarchitectural magnifier, and a high-bandwidth remote covert channel for Spectre-style attacks. The authors claim their attack is both high-performance and stealthy, evading state-of-the-art hardware attack detectors.

While the initial characterization of the timing anomaly is notable, the paper's primary claims regarding the attack's real-world viability, novelty of impact, and stealthiness are insufficiently supported by the provided evidence. The work suffers from a combination of threat model contrivances, overstated conclusions, and a lack of rigorous, controlled comparisons against established baselines.

Strengths

1. Systematic Characterization of a Timing Anomaly: The authors have performed a detailed characterization of AMX instruction latency as a function of idle duration (Figure 1, page 2). The investigation in Section 4.3 to rule out confounding factors like DVFS, C-states, and value dependency is methodical and represents a solid piece of reverse engineering.
2. Demonstration of a Primitive: The paper successfully identifies a timing differential and demonstrates that it can be triggered. The core phenomenon—that an AMX instruction can take ~50 cycles or ~20,000 cycles depending on recent activity—is clearly established on the tested hardware.
3. Breadth of Exploration: The authors apply their primitive to multiple domains (AI privacy, remote channels), showing ambition in exploring the potential impact of the vulnerability.

Weaknesses

1. Insufficient Proof of Root Cause: The central pillar of this paper is that the timing variance is caused by "undocumented power gating." The primary evidence for this is the correlation between latency states and package-level power consumption shown in Figure 5 (page 8). While suggestive, this is not definitive proof. Package-level power is a coarse metric, and the observed drop could be correlated with, but not directly caused by, the same mechanism responsible for the latency. Without more direct evidence (e.g., from on-chip sensors, thermal analysis, or official documentation), attributing this exclusively to power gating is an unsubstantiated leap. The mechanism could be a more complex, undocumented power/clock management state machine.

2. Contrived Threat Models for AI Attacks: The demonstrated AI attacks rely on scenarios that lack clear justification for their real-world prevalence.

   • The Mixture-of-Experts (MoE) attack (Section 6.2) achieves its claimed 100% accuracy only with a "layer gap ≥ 8" between experts. This is an extreme case of architectural heterogeneity. The authors provide no evidence that such architecturally imbalanced MoE models are common or practical in production. The attack's effectiveness degrades significantly as the experts become more similar (Table 4, page 11), which is the more likely scenario.
   • The Membership Inference Attack (MIA) (Section 6.3) claims its 81% accuracy "rivals or exceeds prior attacks." This claim is made without a crucial baseline. The authors should have implemented a standard, confidence-score-based MIA on the exact same early-exit Transformer model to provide a direct comparison. Without this control, it is impossible to assess whether this complex timing-based approach offers any practical advantage over well-established, and potentially simpler, methods.

3. Overstated Claims of Performance and Stealth:

   • Remote Channel Performance: The claimed 70,000x leakage rate improvement over NetSpectre (Section 6.4) is sensationalized. The term "production network" is used without any quantitative characterization of its properties (i.e., baseline latency, jitter, packet loss). Network noise is the single most important variable for remote timing attacks. Without these statistics, the comparison is meaningless and the results are not reproducible or generalizable. The presented violin plots in Figure 9 (page 12) show clear separation, but this could easily be a result of a low-noise, low-contention network path.
   • Stealthiness: The claim of evading SOTA detectors (Section 6.7) is not rigorously proven. The authors state that including the EXE.AMX_BUSY performance counter "did not improve the models' performances." This is a weak dismissal. A thorough analysis would require detailing how this feature was incorporated into the detectors' models and presenting the resulting detection accuracy. It is plausible that a detector specifically designed to look for sparse, high-latency AMX executions could be effective. The current analysis is insufficient to support the strong claim of being "effectively invisible" (page 13).

4. Limited Scope of Hardware Evaluation: The experiments were conducted on a single CPU model (Intel Xeon Gold 5420+). The paper makes broad claims about "Intel AMX," but provides no evidence that this specific timing behavior is present across the entire Sapphire Rapids family, let alone subsequent generations like Emerald Rapids. The findings may be specific to a particular stepping or microcode version of a single product.

Questions to Address In Rebuttal

1. Root Cause: Beyond the correlation with package power, what further evidence can you provide to definitively attribute the observed latency stages to a power gating mechanism, as opposed to another undocumented microarchitectural state change (e.g., clock gating, firmware-managed idle state)?

2. MoE Attack Realism: Please provide justification or citations from deployed systems showing that MoE models with significant architectural heterogeneity (e.g., an 8-layer difference between experts) are a realistic threat model, rather than a constructed best-case scenario for your attack.

3. MIA Baseline: Please provide results from a direct, controlled comparison of your timing-based MIA against a standard confidence-score-based MIA performed on the identical early-exit Transformer model and dataset. This is essential to substantiate the claim that your attack "rivals or exceeds" prior work.

4. Remote Channel Characterization: Please provide quantitative metrics for the "production network" used in Section 6.4, including mean latency, standard deviation (jitter), and packet loss rates over the experiment's duration. How does GateBleed's performance degrade as jitter increases?

5. Stealth Analysis: Please provide a more detailed analysis of the attempt to retrain SOTA detectors. Specifically, describe the feature engineering for the EXE.AMX_BUSY counter and present the full confusion matrix for the retrained detectors. Why, precisely, does this feature fail to enable detection?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces GATEBLEED, a novel and potent timing side channel rooted in the aggressive power gating mechanism of Intel's on-core AI accelerator, Advanced Matrix Extensions (AMX). The core contribution is the discovery that the time required to "wake up" the AMX unit from various power-gated states creates a massive and easily measurable timing discrepancy (up to 20,000 cycles).

The authors masterfully demonstrate that this is not merely an interesting microarchitectural quirk but a versatile and powerful attack primitive with two significant applications: 1. A new vector for AI privacy attacks: They show how GATEBLEED can be used to perform high-accuracy membership inference and infer routing decisions in Mixture-of-Experts (MoE) models. Crucially, these attacks operate purely on timing, without needing access to model outputs like logits or confidence scores, thereby bypassing a whole class of traditional defenses. 2. A generic, high-performance, and stealthy attack magnifier: They repurpose GATEBLEED as a transmission channel for a remote Spectre attack that is robust to network noise, and as a magnifier that can amplify subtle microarchitectural events to bypass timer coarsening defenses.

The work provides a thorough characterization of the vulnerability, identifies exploitable gadgets in major ML libraries, demonstrates end-to-end attacks, and evaluates potential mitigations.

Strengths

The true strength of this paper lies in its synthesis of several research domains and its demonstration of a fundamental vulnerability with far-reaching implications.

1. Bridging Hardware Architecture and AI Security: The most significant contribution is the direct line it draws from a low-level hardware power optimization to high-level AI privacy risks. While prior works have attacked AI models via side channels (e.g., Cache Telepathy [144]), GATEBLEED exposes a new, more fundamental leakage source. By linking the conditional execution inherent in modern models (like MoEs and early-exit networks) to the conditional activation of a hardware accelerator, the paper reveals that architectural choices in ML now have direct, exploitable hardware security consequences. This is a timely and important connection.

2. Exceptional Signal-to-Noise Ratio and Practicality: The vulnerability's defining characteristic is the sheer magnitude of the timing delta. A 20,000-cycle gap is orders of magnitude larger than those seen in previous power/frequency-based attacks like Hertzbleed [135] (~200 cycles). This is not an incremental improvement; it is a categorical shift. This massive signal makes the attack resilient to real-world noise (as demonstrated in the remote Spectre attack in Section 6.4, page 11) and resistant to standard defenses like timer coarsening, lending the attacks a degree of practicality rarely seen in academic side-channel research.

3. Stealth and Evasion of Existing Defenses: The paper convincingly argues that GATEBLEED is exceptionally stealthy. Because the attack can be triggered by a single instruction following a period of natural idleness, it leaves minimal footprint and evades state-of-the-art microarchitectural attack detectors (as shown in Table 5, page 13). This is a critical finding, as it suggests that current defense paradigms, which often look for anomalous patterns like high cache miss rates or TLB flushing, are blind to this class of vulnerability.

4. Broad Applicability as a Generic Primitive: Beyond the novel AI attacks, the positioning of GATEBLEED as a generic magnifier and covert channel primitive (Section 5.2, page 8 and Section 6.4, page 11) significantly broadens the paper's impact. It provides the security community with a powerful new building block that could enable or enhance a wide range of microarchitectural attacks, especially in constrained or noisy environments where they were previously infeasible.

Weaknesses

The paper's core ideas are strong, and the execution is thorough. The weaknesses are less about flaws and more about opportunities to further explore the context and boundaries of the findings.

1. Limited Discussion on Threat Model Practicality in Cloud Environments: The most powerful local attacks rely on the "AMX Usage" threat model, where an attacker process is co-located on the same physical core as the victim. While possible, modern hypervisor and OS schedulers in multi-tenant cloud environments are increasingly sophisticated and may actively work to isolate workloads, potentially making same-core co-residency a rare or difficult-to-achieve condition. A deeper discussion of the real-world probability and techniques for achieving this co-residency would strengthen the paper's claims about practical risk.

2. The Scope of Generality: The paper demonstrates GATEBLEED's capability as a magnifier by amplifying a cache hit/miss timing difference. This is a clear and effective proof of concept. However, to truly cement its status as a generic magnifier, it would be beneficial to discuss or demonstrate its application to other, more subtle microarchitectural events, such as contention on execution ports or scheduler queues.

3. A Singular Focus on AMX: The work is an excellent deep dive into Intel's AMX. However, AMX is just one example of a broader trend toward on-core, specialized accelerators (e.g., NPUs in consumer chips, Google's TPUs). The paper would be even more impactful if it briefly contextualized its findings within this trend, speculating on whether similar design principles (i.e., aggressive power gating for high-power, intermittently-used units) might lead to analogous vulnerabilities in other vendors' hardware.

Questions to Address In Rebuttal

1. Regarding the threat model for the co-resident attacks: Could the authors comment on the prevalence of same-core scheduling in major public cloud environments (e.g., AWS, GCP, Azure)? Are there known techniques an attacker could use to increase the likelihood of being scheduled on the same core as a target victim process?

2. The paper compellingly demonstrates GATEBLEED as a magnifier for a cache timing delta. Could the authors elaborate on its potential to amplify other, more subtle microarchitectural events? Is the 20,000-cycle "cliff" sensitive enough to be tipped by phenomena smaller than an L3 cache miss, such as contention for a specific execution port?

3. Given the industry-wide trend towards integrating specialized accelerators directly onto the CPU die, do the authors believe that power gating mechanisms in other on-core accelerators (e.g., NPUs, integrated GPUs) represent a likely new frontier for GATEBLEED-style vulnerabilities? Are there fundamental architectural reasons why this vulnerability might be unique to AMX, or is it likely a more generalizable problem?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present GateBleed, a timing side channel vulnerability found in the power gating mechanism of Intel's Advanced Matrix Extensions (AMX) on-core AI accelerator. The core of the work is the discovery and characterization of an undocumented, multi-stage power management feature that introduces reuse-distance-dependent latency variations of up to 20,000 cycles.

The authors leverage this new primitive in three primary ways: 1. As a novel vector for AI privacy attacks (Membership Inference and Mixture-of-Experts routing leakage) that relies solely on timing and not on model outputs like confidence scores. 2. As a generic, single-instruction "magnifier" to amplify subtle microarchitectural state differences, making them observable even with coarse timers. 3. As a high-bandwidth covert channel for remote attacks like Spectre, which is shown to be resilient to network noise where prior art fails.

The paper claims novelty in being the first to exploit accelerator power optimizations for AI privacy attacks and in creating a uniquely stealthy and powerful microarchitectural magnifier.

Strengths

The primary strength of this paper is the discovery of a genuinely new and potent microarchitectural primitive. My analysis of prior art confirms the following points of novelty:

1. A New Primitive, Not Just a New Target: The core discovery—a multi-stage, stepped power gating mechanism local to the AMX unit (as detailed in Section 4, pages 6-8 and Figure 1, page 2)—is fundamentally different from previously known power-related side channels.

   • It is distinct from DVFS-based channels like Hertzbleed [135], as the authors demonstrate the effect persists at fixed frequencies (Section 4.3, page 7).
   • It is distinct from whole-core sleep state channels like IdleLeak [105], as it is independent of core C-states.
   • Crucially, it is distinct from the closest related work, Thor [29], which also targets AMX. Thor describes an operand-dependent timing variation in a single low-power state. GateBleed describes a reuse-distance-dependent timing variation across five distinct power-gating stages. The root cause is different, and the latency magnitude of GateBleed appears to be two orders of magnitude larger, making it a far more powerful primitive.

2. Novel Advancement in Magnifier Design: The formulation of GateBleed as a single-instruction magnifier (Section 5.2, page 8) is a significant conceptual advance over prior art. Existing magnifiers like Hacky Racers [143] require complex, carefully crafted instruction sequences to exploit Instruction-Level Parallelism, leaving a large and detectable footprint. Microscope [117] requires privileged OS-level access. The GateBleed magnifier is unprivileged, requires only a single instruction following a passive wait, and its mechanism (hardware power-up latency) is entirely new in this context. This represents a qualitative leap in simplicity and stealth.

3. New Vector for AI Privacy Attacks: While hardware side-channel attacks on AI models are known (e.g., Cache Telepathy [144]), they typically target static model artifacts (recovering weights or architecture). GateBleed introduces a novel attack surface: leaking private, input-dependent dynamic decisions (e.g., early-exit paths, expert routing) by observing the side effects of accelerator power management. This is a new conceptual link between hardware power optimization and AI privacy that has not been explored before.

Weaknesses

My critique is not that the work lacks novelty, but that its novelty could be framed with greater precision and its implications explored more broadly.

1. Positioning: A Potent Instance or a New Class? The paper convincingly presents a new vulnerability. However, it is an instance of a broader, known principle: power state transitions incur latency penalties. The exceptional novelty here stems from the undocumented, multi-stage, and high-latency nature specific to AMX. The authors should more clearly position their contribution: is this a one-off implementation flaw in AMX, or is it the first documented example of a new class of vulnerabilities we should expect in future tightly-integrated, aggressively power-managed accelerators?

2. Limited Generalizability of the Primitive: The discovered primitive is, by definition, specific to a particular microarchitecture (Intel AMX on Sapphire Rapids and newer). While the applications are more general, the root cause is narrow. The paper would be strengthened by a discussion on the architectural trends that led to this design choice and whether similar vulnerabilities are likely to exist in other on-core accelerators (e.g., NPUs in consumer SoCs, GPU Tensor Cores) that also employ aggressive power gating. Without this, the novelty risks being perceived as narrow, even if it is deep.

Questions to Address In Rebuttal

1. The authors cite Thor [29] as a related work that also finds a timing channel in AMX. Could the authors dedicate a paragraph to explicitly contrasting the root cause of GateBleed (reuse-distance-dependent staging) with that of Thor (operand-dependent leakage)? This would further solidify the novelty of the discovered primitive.

2. In Section 6.7 (page 12), the authors claim GateBleed evades state-of-the-art detectors. Is this evasion simply because current detectors are not instrumented to monitor AMX-specific performance counters, or is there a more fundamental reason why this primitive is inherently difficult to detect? For example, does a single AMX instruction after a long, passive wait period fall below the anomaly detection threshold of any conceivable event-based detector?

3. The core mechanism is tied to AMX. Could the authors speculate on the likelihood of finding similar multi-stage, high-latency power gating channels in other classes of on-core accelerators? What architectural pressures (e.g., thermal density, power budget sharing) would incentivize designers to create such leaky optimizations? This would help frame the work's conceptual contribution beyond a single-vendor implementation.

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The paper proposes "Athena," a framework and co-designed hardware accelerator for executing quantized convolutional neural networks (CNNs) under fully homomorphic encryption (FHE). The core methodological deviation from prior work is the rejection of the conventional CKKS scheme for approximate arithmetic in favor of an integer-based FHE scheme (akin to BFV). Non-linear operations (i.e., activation functions) are handled via a "Functional Bootstrapping" (FBS) mechanism, which is effectively a polynomial evaluation of a Look-Up Table (LUT). The authors claim this approach allows for much smaller cryptographic parameters, resulting in significant speedups (1.5x-2.3x) and EDP improvements (3.8x-9.9x) over state-of-the-art CKKS-based FHE accelerators, with negligible accuracy degradation relative to a quantized plaintext baseline.

Strengths

1. Problem Simplification: The fundamental premise of leveraging model quantization to simplify the underlying cryptographic problem is sound. Moving from the complexities of approximate arithmetic in CKKS to integer arithmetic is a valid research direction that could plausibly reduce overhead.
2. Hardware-Software Co-design: The authors have clearly considered the interplay between their proposed framework and the hardware needed to execute it. The design of specialized units like the FRU to accelerate the specific bottlenecks of their framework (namely FBS) demonstrates a coherent design philosophy.

Weaknesses

My primary concerns with this submission revolve around the rigor of the analysis, the fairness of the experimental comparisons, and the overstatement of the framework's generality and novelty.

1. Insufficient Noise Analysis: The noise analysis presented is superficial and unconvincing. In Section 3.2.2 (p. 5), the authors introduce a rounding noise ems during modulus switching. They claim it has "minimal impact" because it contaminates LSBs. Figure 4 (p. 7) shows that this "minimal" impact results in data error ratios of up to 11% in some layers of ResNet-56. A claim that this level of error has no significant cumulative effect on a deep network is extraordinary and requires extraordinary proof, which is not provided. The analysis lacks any formal treatment of how this error propagates and accumulates across dozens of layers. Relying on final accuracy metrics for just a few models is insufficient to validate the robustness of this noise-handling approach.

2. Misleading Performance Comparisons: The performance evaluation in Section 5.2.2 (p. 10) constitutes a severe apples-to-oranges comparison. The baseline accelerators (CraterLake, ARK, SHARP) are designed to handle the significantly more complex and general-purpose CKKS scheme, which includes computationally intensive operations for bootstrapping and complex number arithmetic. Athena, by design, targets a much simpler cryptographic problem (integer-only arithmetic). It is therefore unsurprising that a specialized accelerator for a simpler problem outperforms a general-purpose accelerator for a harder one. Figure 8 (p. 11), which shows that CKKS accelerators are ill-suited for the Athena workload, does not support the authors' claims of superiority; it merely states the obvious. A fair comparison would involve implementing the Athena framework on a configurable platform against a CKKS implementation on the same platform, or comparing against a hypothetical CKKS accelerator that is also specialized for quantized workloads.

3. Questionable Novelty and Scalability of "Functional Bootstrapping" (FBS): The FBS mechanism, described in Section 3.2.3 (p. 6), is presented as a key innovation. However, it is fundamentally a known technique: evaluating a LUT via polynomial interpolation. The complexity analysis in Table 3 (p. 7) states the complexity of FBS is O(t), where t is the plaintext modulus. For t = 65537, this is computationally massive. While the BSGS optimization (Algorithm 2) reduces ciphertext multiplications to O(√t), the number of scalar multiplications and additions remains a direct function of t. The paper's impressive performance results seem to be achieved not by an algorithmically superior method, but by applying brute-force hardware (a 16-block FRU array) to this high-complexity problem. This raises serious questions about scalability.

4. Limited Generality and Fragile Parameterization: The entire framework hinges on the plaintext modulus t being large enough to contain all intermediate inner product results. The authors select t = 65537, which Figure 4 shows is just sufficient for the tested benchmarks. The paper provides no methodology for selecting t for an arbitrary new model, nor does it analyze the sensitivity of the framework to this choice. If a deeper or wider network architecture requires a larger dynamic range, t would need to increase, causing the complexity of FBS to explode and likely rendering the approach impractical. The claim that Athena can "support any type of activation functions" is thus unsubstantiated, as any function requiring a larger LUT or higher precision would break the current parameterization.

Questions to Address In Rebuttal

The authors must address the following points directly to make a case for this paper's acceptance:

1. Regarding Noise Propagation: Please provide a formal analysis of the propagation and accumulation of the ems error introduced during modulus switching. Demonstrate, either through formal proof or extensive empirical evaluation on significantly deeper networks (e.g., >100 layers), that this error remains bounded and does not catastrophically degrade accuracy.
2. Regarding Baselines: Please justify the fairness of comparing your specialized, integer-only accelerator against general-purpose CKKS accelerators. To make a convincing case, provide a comparison against a more appropriate baseline, for instance, an implementation of your framework on a general-purpose FHE accelerator or a theoretical analysis against a CKKS flow optimized for quantized inference.
3. Regarding FBS Scalability: Given the O(t) complexity of the underlying FBS operation, how does the system's performance and hardware cost scale as t increases? What is the practical upper bound on t before the FBS latency and area make the accelerator infeasible?
4. Regarding Generality: How would a user of the Athena framework determine the required value for the plaintext modulus t for a new, arbitrary CNN model? What happens if this required t is larger than the 17-bit value used in this work?
5. Regarding Complex Functions: The paper briefly mentions a three-step process for Softmax. This appears to involve multiple FBS evaluations and a costly ciphertext-ciphertext multiplication. Please provide a detailed breakdown of the latency, noise growth, and complexity for this operation, as it is a critical component in many classification networks.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces "Athena," a novel co-designed framework and hardware accelerator for performing Convolutional Neural Network (CNN) inference under Fully Homomorphic Encryption (FHE). The core contribution is a paradigm shift away from the prevailing approach of using approximate arithmetic schemes like CKKS on floating-point models. Instead, Athena leverages the mature field of Quantized CNNs (QCNNs), whose integer-based arithmetic is a natural fit for integer-based FHE schemes like BFV.

This fundamental pivot allows for significantly smaller cryptographic parameters, which in turn leads to dramatically reduced ciphertext sizes and on-chip memory requirements. The framework handles the critical non-linear activation functions not with inaccurate polynomial approximations, but with an exact "functional bootstrapping" (FBS) mechanism. This mechanism, reminiscent of TFHE's programmable bootstrapping, performs a lookup-table operation that can implement any arbitrary activation function and simultaneously handles the re-quantization (remapping) step. The authors present a full-stack solution, from the five-step cryptographic loop (Section 3.1, page 4) to a specialized accelerator architecture designed to handle the framework's unique computational bottlenecks, particularly the FBS step. The result is a system that claims near-plaintext accuracy with significant performance and efficiency gains over state-of-the-art CKKS-based accelerators.

Strengths

1. Elegant Core Idea and Problem Reframing: The primary strength of this work lies in its insightful connection between two distinct domains: Quantized Neural Networks and Integer-based FHE. Instead of trying to force the square peg of floating-point neural networks into the round hole of approximate FHE (CKKS), the authors correctly identify that the integer-only nature of QCNNs is a perfect match for schemes like BFV. This reframing of the problem is the paper's most significant intellectual contribution and the source of all subsequent benefits.

2. Practicality and Feasibility: The consequences of this pivot are profound. As shown in Table 1 (page 3) and Table 8 (page 12), the required ciphertext and key sizes are drastically reduced, leading to a >4x reduction in on-chip scratchpad memory compared to leading FHE accelerators like CraterLake and ARK. This is not an incremental improvement; it is a step-change in hardware feasibility and potential cost, making private ML inference a much more tangible reality.

3. Generalized and Accurate Non-Linearity Handling: The use of functional bootstrapping (FBS) is a powerful choice that solves a major pain point in FHE-based ML. The reliance on Taylor/Chebyshev polynomial approximations in CKKS-based systems is a notorious source of error and requires expert tuning (as shown in Figure 1, page 3). Athena's FBS approach is general—it can implement ReLU, Sigmoid, and even complex pooling operations with perfect precision within the quantized domain. Merging the activation function and the remapping step into a single LUT operation is a particularly clever co-design choice.

4. Strong Full-Stack Co-Design: The paper presents a convincing end-to-end solution. The software framework's five-step loop is directly reflected in the hardware design. The identification of FBS as the new performance bottleneck (as opposed to NTT in traditional designs) and the subsequent design of the versatile FRU and the pipelined two-region dataflow (Section 4.3, page 9) demonstrates a deep understanding of the entire stack. The results in Figure 8 (page 11), where the Athena framework is simulated on prior hardware, effectively argue for the necessity of this specialized accelerator.

Weaknesses

While the work is strong, its focus is sharp, leaving some broader contextual questions open. These are not so much flaws as they are opportunities for strengthening the paper.

1. Limited Discussion on the Quantization Process: The paper assumes the availability of a pre-trained QCNN. However, the process of creating a high-accuracy QCNN, often through Quantization-Aware Training (QAT), is non-trivial. More importantly, the choices made in the FHE framework (e.g., the plaintext modulus t in Section 3.3, page 7) are deeply intertwined with the quantization strategy (e.g., the bit-width and range of intermediate values). A brief discussion on this interplay would strengthen the paper by showing how the two domains must be co-designed, not just cascaded.

2. Architectural Generality: The work is presented and evaluated exclusively on CNNs. While this is a critical workload, the field of deep learning is rapidly expanding, with Transformers becoming dominant in many areas. Transformers also rely heavily on quantization for efficient inference. How would the Athena framework and accelerator adapt to the different computational patterns of a Transformer (e.g., large matrix multiplications and the Softmax function)? Addressing this would elevate the contribution from a "solution for CNNs" to a more general "framework for quantized models."

Questions to Address In Rebuttal

1. Could you elaborate on the interplay between the quantization strategy and the selection of FHE parameters? Specifically, how does the choice of the plaintext modulus t (65537 in your work) constrain or inform the quantization process (e.g., the 7-bit w7a7 scheme)? Is there a systematic way to co-optimize these parameters?

2. The paper's evaluation focuses entirely on CNNs. Could the authors comment on the applicability of the Athena framework to other quantized architectures like Transformers? The Softmax operation, in particular, seems like a perfect candidate for the FBS mechanism. Would the accelerator's dataflow, designed for convolutions, be efficient for the large matrix multiplications in a Transformer's attention and feed-forward layers?

3. In the performance comparison in Table 6 (page 10), the paper compares against baselines running CKKS-based models. Could you provide more detail on how the "computational complexity of other benchmarks" was normalized to that of ResNet-20 for a fair comparison, particularly for ResNet-56 which was not reported by all baselines?

4. Step 4 of the Athena framework ("Packing") involves a "homomorphic decryption" of LWE ciphertexts. This is a powerful but potentially costly primitive. In the execution time breakdown (Figure 9, page 11), this operation does not appear to be explicitly separated. Could you clarify where its cost is accounted for and its relative significance to the overall latency?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

This paper introduces "Athena," a framework and co-designed hardware accelerator for performing inference on quantized convolutional neural networks (QCNNs) under fully homomorphic encryption (FHE). The central claim of novelty lies not in the creation of a new cryptographic primitive, but in the formulation of a new end-to-end framework that departs from the dominant CKKS-based paradigm for FHE-based machine learning.

The core idea is to build a processing loop around integer-based FHE (specifically, BFV-like operations) tailored for QCNNs. This loop systematically manages computation and noise through five steps: (1) a linear operation using coefficient encoding, (2) modulus switching to reduce noise, (3) ciphertext conversion from RLWE to LWE, (4) homomorphic decryption and packing back into RLWE, and (5) a "Functional Bootstrapping" (FBS) step. The most significant aspect of this proposed framework is that the FBS step unifies three traditionally separate operations: the noise-clearing bootstrap, the non-linear activation function evaluation, and the integer re-quantization/remapping. The authors claim this synthesis allows for smaller cryptographic parameters and higher accuracy compared to prior CKKS-based approaches, and they present a hardware architecture specifically designed to accelerate the bottlenecks of this new workflow.

---

Strengths

1. A Novel Conceptual Framework: The primary strength and most significant novel contribution is the deliberate departure from the CKKS-based paradigm for deep learning inference. While prior work has focused on approximating real-number arithmetic with CKKS, Athena embraces the discrete nature of quantized networks and builds a native integer-based FHE pipeline. This represents a distinct and valuable alternative direction in the design space.

2. Unification of Operations in Functional Bootstrapping: The application of functional bootstrapping to simultaneously perform bootstrapping, non-linear activation, and remapping (as described in Section 3.2.3, Page 6) is a highly novel synthesis of existing ideas. Prior works have used bootstrapping to manage noise and separate techniques (e.g., polynomial approximation) for activations. Merging these, along with the crucial remapping step required for quantization, into a single LUT-based operation is a clever and powerful simplification of the inference flow. This is the paper's most compelling technical insight.

3. A Coherent, Self-Contained Workflow: The proposed five-step loop (Figure 2, Page 4) is a well-defined and repeatable process for executing QCNN layers under FHE. It presents a structured methodology for managing noise and evaluating functions that is distinct from the ad-hoc parameter tuning often required in leveled CKKS schemes. This structured workflow itself can be considered a novel contribution to the field.

4. Novelty in Hardware Co-Design: The accelerator architecture is not merely a collection of standard FHE components. The design of the versatile "FBS and RNS Base changing unit" (FRU) and the two-region dataflow (Section 4.3, Page 9) is a direct and novel consequence of the proposed software framework. The design correctly identifies FBS as the new system bottleneck (a shift away from NTT in many prior works) and dedicates significant, specialized resources to it.

---

Weaknesses

1. Novelty is in Synthesis, Not Primitives: The paper's novelty is almost entirely based on the clever combination and application of pre-existing techniques. The authors appropriately cite prior art for the core FBS primitive ([29]), coefficient encoding for convolutions ([16, 21]), and RLWE-to-LWE conversions ([12]). While the synthesis is novel, the paper could be more explicit in delineating what is being adopted versus what is being created. The contribution is the recipe, not the ingredients.

2. Incremental Novelty in Encoding: The paper contrasts its coefficient encoding scheme with Cheetah [16] (Table 2, Page 5). The claimed improvement appears to be a different batching strategy (prioritizing output channels) to improve data locality for subsequent steps, rather than a fundamentally new encoding method. The novelty here is an optimization tailored to their specific framework, which, while valuable, is an incremental advancement over prior art.

3. Complexity vs. Benefit Justification: The proposed five-step loop introduces significant complexity, involving two forms of ciphertext (RLWE, LWE) and multiple conversions between them (Steps 3, 4, and the final S2C step). While the results are impressive, the justification for this specific complex pathway over a simpler, purely BFV-based or TFHE-based approach could be stronger. The benefit is clear, but it comes at the cost of a non-trivial pipeline that requires specialized hardware (like the SE unit) to remain efficient.

---

Questions to Address In Rebuttal

1. The core of your non-linear evaluation rests on functional bootstrapping, which essentially performs a LUT lookup. Prior works such as PEGASUS [30] and TOTA [42] have also proposed using TFHE-style bootstrapping for LUT-based evaluation of non-linear functions in FHE. Please clearly articulate the delta between Athena's approach and these prior works. Is the primary novelty the integration of the remapping step into the LUT, the specific five-step pipeline in which it is embedded, or another factor?

2. Regarding the coefficient encoding for linear layers (Section 3.2.1, Page 5), please clarify the precise novel contribution over the methods used in Cheetah [16] and NeuJeans [21]. Is the novelty in the encoding itself, or is it strictly in the batching strategy that arranges data to benefit the subsequent sample extraction step?

3. The proposed framework unifies activation and remapping within the FBS step. How does the framework handle layers that do not have a non-linear activation (e.g., a convolution followed directly by a pooling or batch normalization layer)? Does this require an "identity-with-remapping" LUT, and if so, what is the performance and complexity overhead compared to a more direct remapping operation? This would help clarify the generality of the proposed novel framework.

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors present ccAI, a hardware-software co-design intended to provide confidential computing for AI workloads on heterogeneous systems with legacy xPUs. The system's architecture is anchored on a hardware module, the PCIe Security Controller (PCIe-SC), which intercepts and secures PCIe traffic between a Trusted VM (TVM) and an xPU. This is complemented by a software component, the "Adaptor," which operates within the TVM to manage security operations without modifying the user application or the xPU's native driver stack. The core proposition is that by operating at the PCIe packet level, ccAI can offer a compatible, transparent, and secure solution for a wide range of xPUs that lack native confidential computing features.

Strengths

1. Problem Motivation: The paper correctly identifies a critical and practical gap in the current ecosystem: the vast majority of deployed xPUs lack the confidential computing capabilities of cutting-edge hardware like the NVIDIA H100, yet they process sensitive AI workloads. Addressing this is a worthwhile endeavor.
2. Architectural Approach: The central idea of leveraging the PCIe interconnect as a universal enforcement point is logical. Since PCIe is the de facto standard, this approach has the potential to be more broadly applicable than solutions tied to specific xPU architectures or TEE designs.
3. Prototyping Effort: The authors have clearly invested significant engineering effort in building a functional prototype. Implementing the PCIe-SC on an FPGA and integrating it with five distinct, real-world xPUs from multiple vendors (NVIDIA, Tenstorrent, Enflame) is a non-trivial accomplishment and lends a degree of credibility to the feasibility of the design.

Weaknesses

My primary concerns with this submission relate to the strength of its claims regarding compatibility, security rigor, and the representativeness of the performance evaluation.

1. Overstated Compatibility and Transparency: The claim of "no xPU SW changes" (Table 2, page 10) is a significant overstatement. The paper details the introduction of a new kernel module, ccAI_adaptor, within the TVM (Section 7.1, page 8). This module is not part of the original xPU software stack and creates a new, non-trivial dependency. It interacts with the PCIe-SC, allocates memory, and handles encryption. Any update to the native xPU driver that alters its memory access patterns, DMA semantics, or MMIO interactions could break the assumptions made by the Adaptor, requiring a corresponding update. This is not seamless transparency; it is shifting the modification burden from the driver itself to an adjacent, tightly-coupled kernel module. The compatibility is therefore fragile.

2. Insufficient Scrutiny of the TCB and Security Guarantees: The security of the entire system hinges on the correctness of the PCIe-SC and its configuration.

   • Hardware Complexity: The PCIe-SC implementation consumes 218.6K ALUTs and 195.7K logic registers (Table 3, page 11). This is a substantial and complex piece of hardware, effectively a sophisticated NIC/firewall on the PCIe bus. A hardware design of this complexity is a major attack surface in itself. The paper provides no evidence of formal verification or rigorous testing to ensure the hardware is free of critical bugs that could be exploited to bypass security policies.
   • Packet Filter Brittleness: The security enforcement relies on a set of L1/L2 table rules (Figure 5, page 6). How are these rules generated? The paper glosses over this critical process. For any new xPU or even a new driver version, this rule set must be perfectly defined to distinguish between benign and malicious traffic. An incorrect or incomplete rule set is tantamount to an open door. This suggests a high operational burden and a high risk of misconfiguration, which undermines the practical security guarantees.
   • Unaddressed Threat Vectors: The paper dismisses side-channel attacks as orthogonal (Section 2.2, page 4). However, the PCIe-SC itself introduces new potential side channels. For instance, packet processing time within the PCIe-SC could vary depending on the security action taken (e.g., pass-through vs. decryption). An attacker monitoring PCIe bus traffic timing could potentially infer information about the operations being performed. This is not an orthogonal issue; it is a new vulnerability created by the proposed architecture.

3. Unconvincing Performance Evaluation: The reported low overhead figures (0.05% - 5.67%) appear to be the result of testing under favorable, compute-bound conditions, which do not sufficiently stress the ccAI architecture.

   • Non-Representative Workloads: The majority of the LLM evaluations, particularly in Figure 9 (page 11), are performed with a batch size of 1. While useful for latency measurements, this configuration is heavily compute-bound and minimizes the relative impact of I/O overhead. In real-world cloud inference scenarios, throughput is maximized by using larger batch sizes, which dramatically increases the volume of data traversing the PCIe bus. The evaluation fails to demonstrate how ccAI performs under such I/O-intensive, high-throughput conditions where the per-packet overhead of the PCIe-SC would be most pronounced.
   • Insufficient Stress Testing: The stress test in Section 8.6 (page 12) artificially limits PCIe bandwidth but does not present a workload that saturates the link. The crucial test is one where the application is bottlenecked by PCIe bandwidth in the baseline configuration. Only then can the true overhead of ccAI's cryptographic and filtering operations be accurately measured. The current test does not provide this insight.

Questions to Address In Rebuttal

1. Please reconcile the claim of "no software changes" with the necessity of the ccAI_adaptor kernel module. How do the authors guarantee that the Adaptor will remain compatible across future, potentially significant updates to the proprietary xPU drivers it must coexist with?

2. Given the substantial complexity of the PCIe-SC hardware (Table 3), what specific steps were taken to verify its functional correctness and security properties? Was any form of formal methods or exhaustive simulation employed to prove it is not a source of vulnerabilities itself?

3. The paper's performance claims hinge on benchmarks that are not I/O-bound (e.g., batch size 1). Please provide performance evaluation data for a high-throughput scenario where the workload is designed to saturate the PCIe bus (e.g., large-batch inference on a model with significant weight loading, or a data-intensive training workload).

4. Please elaborate on the generation and management process for the Packet Filter rules (Figure 5). Is this a manual, per-device, per-driver process? If so, how does this scale and how do you prevent human error from introducing critical security flaws? What is the performance penalty for rule lookup as the rule set grows in complexity?

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents ccAI, a novel system designed to provide confidential computing for a wide range of AI accelerators (xPUs). The authors identify a critical gap in the current landscape: while cutting-edge hardware like the NVIDIA H100 offers built-in confidentiality, the vast majority of deployed accelerators lack such features, and existing academic solutions often suffer from poor compatibility or require significant application/driver modifications.

The core contribution of ccAI is its architectural choice to enforce security at the PCIe interconnect level. This is achieved through a hardware-software co-design comprising two key components: 1) a PCIe Security Controller (PCIe-SC), a hardware module that sits between the host and the xPU to intercept, filter, and process all PCIe packets, and 2) a software "Adaptor" running within a Trusted VM (TVM) on the host, which transparently manages the secure workflow without altering the user application or the native xPU drivers. By treating the PCIe packet stream as a universal interface, ccAI aims to deliver a compatible, transparent, and secure solution for heterogeneous AI computing environments.

Strengths

1. A Pragmatic and Powerful Architectural Choice: The single most important idea in this paper is the decision to place the security boundary on the PCIe bus. This is a brilliant move that reframes the problem. Instead of trying to secure the infinitely complex internals of various xPUs or modify proprietary driver stacks, the authors treat the accelerator as a black box and secure its sole communication channel with the host. This abstraction is the key to achieving the paper's primary goal of compatibility. It elegantly sidesteps the vendor-specific details that have plagued many previous approaches.

2. Excellent Contextualization and Problem Framing: The authors demonstrate a clear and panoramic understanding of the confidential computing landscape. Figure 1 (Section 3, page 3) is particularly effective, providing a concise taxonomy of existing approaches (TEE-based, HW-based, TDISP, etc.) and clearly positioning ccAI as a distinct and complementary solution. The paper correctly identifies that while standards like TDISP are the long-term future, there is a pressing, immediate need for a solution that can secure the massive installed base of legacy hardware. ccAI is presented not just as a research project, but as a practical answer to a real-world market and operational gap.

3. Strong System-Oriented Design: The work goes beyond a simple conceptual model. The authors have considered the full lifecycle of a secure workload, including a secure boot process for the PCIe-SC, remote attestation protocols (Section 6, page 7), and key management. The design of the Packet Filter and Packet Handlers (Section 4, page 5) shows a thoughtful approach to balancing security and performance by categorizing packet types and applying tailored protection policies. This level of detail suggests a mature and well-considered system design.

4. Comprehensive and Convincing Evaluation: The experimental validation is a significant strength. By implementing a prototype and testing it across five distinct xPUs from three different vendors (NVIDIA GPUs, a Tenstorrent NPU, and an Enflame GPU, as detailed in Section 7, page 8), the authors provide strong evidence for their central claim of compatibility. The low performance overheads reported across a wide range of Large Language Models (LLMs) further bolster the argument for the system's practicality.

Weaknesses

While the core idea is strong, the paper could be improved by addressing the practical implications of its proposed hardware.

1. The Practicality of the PCIe-SC: The entire system hinges on the existence and deployment of the PCIe-SC hardware module. While an FPGA prototype is excellent for academic validation, the path to real-world deployment is a significant hurdle. Is this envisioned as a standalone PCIe card that an xPU plugs into? An integrated component on future motherboards? An offering from a third-party hardware security company? The paper lacks a discussion of the form factor, cost, and supply chain implications, which are crucial for assessing its potential for widespread adoption in cloud data centers.

2. Scalability and Performance Ceiling: The evaluation demonstrates low overhead on current hardware. However, the bandwidth requirements of next-generation accelerators are growing exponentially. The prototype is based on an Intel Agilex 7 FPGA. A critical question is whether this architecture can scale to saturate the PCIe Gen5/Gen6 links of future flagship GPUs without becoming the primary performance bottleneck. While an ASIC implementation would be faster, some analysis of the architectural throughput limits would strengthen the paper's claims of future viability.

3. The Trusted Computing Base (TCB) of the Intermediary: By introducing the PCIe-SC, the authors have created a new, critical piece of hardware that becomes part of the TCB. The security analysis in Section 8.2 (page 10) is good, but it could further explore the threat model against the PCIe-SC itself. What is the mechanism for securely updating its firmware? How is its physical integrity maintained beyond the proposed chassis sealing? While the TCB is small compared to a full driver stack, its criticality warrants a more in-depth discussion.

Questions to Address In Rebuttal

1. Could the authors elaborate on the envisioned deployment model for the PCIe-SC in a typical cloud environment? What would be the most likely path to market for such a device, and what are the primary barriers to its adoption by cloud providers?

2. The paper compares ccAI's performance overhead to the NVIDIA H100's confidential mode. Could you discuss the architectural trade-offs? While ccAI is more compatible, does the H100's tight integration of security features provide fundamental performance or security advantages that an external PCIe device can never fully match?

3. How do the authors see ccAI co-existing with the emerging TDISP standard in the long term? Is ccAI primarily a bridging solution for the next 5-10 years until TDISP is ubiquitous, or does the packet-level inspection and policy enforcement of the PCIe-SC offer complementary security guarantees that would remain valuable even in a TDISP-enabled ecosystem?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes ccAI, a system for retrofitting confidential computing capabilities onto legacy, general-purpose xPUs (GPUs, NPUs, etc.) within a heterogeneous cloud environment. The authors' central claim of novelty rests on their specific architectural approach, which aims to provide strong security guarantees with high compatibility and user transparency, a combination they argue is lacking in prior art.

The system is composed of two primary components: (1) a hardware module, the PCIe Security Controller (PCIe-SC), which is physically interposed between the host's PCIe bus and the target xPU, and (2) a software component, the Adaptor, which resides in a Trusted VM (TVM) and coordinates with the PCIe-SC. The core mechanism involves the PCIe-SC intercepting all PCIe traffic to and from the xPU at the packet level. It uses a set of filtering rules and handlers to enforce security policies, such as encrypting/decrypting data payloads for DMA and validating MMIO operations, while remaining transparent to the unmodified xPU driver and user application.

The core novelty is not in the individual ideas of hardware-based security or using TEEs, but in the specific architectural synthesis: using an external, packet-level PCIe security module to create a confidentiality boundary for unmodified, legacy hardware and software stacks. This positions ccAI as a solution for the vast ecosystem of existing accelerators that lack the built-in features of NVIDIA's H100 or do not comply with emerging standards like TDISP.

Strengths

1. Novel Architectural Niche: The primary strength of this work is its novel architectural arrangement. While prior art has explored confidential accelerators, they typically fall into three categories that ccAI cleverly sidesteps:

   • Device-Integrated Hardware (e.g., NVIDIA H100 [50]): These solutions are vendor-specific, proprietary, and only available on the latest, most expensive hardware. ccAI's approach of externalizing the security module makes it, in principle, vendor-agnostic and applicable to legacy devices.
   • TEE-based Software Modifications (e.g., Cronus [40], CAGE [85]): These systems often require significant and complex modifications to the xPU driver stack to partition it and run critical parts within a TEE. ccAI's proposed hardware/software co-design aims for a much smaller software footprint (the "Adaptor") and claims to leave the complex driver stack untouched, a significant delta in terms of transparency and compatibility.
   • Forthcoming Standards (e.g., TDISP): These require compliance from the CPU, platform, and the xPU device itself. The novelty of ccAI is that it provides a concrete solution for the present-day reality where such compliant hardware is not widely deployed.

2. Packet-Level Abstraction: Anchoring the security mechanism at the level of the PCIe packet (as detailed in Section 4, page 5) is a powerful and novel choice for this specific problem domain. PCIe is the lingua franca of host-device communication. By operating at this level, ccAI establishes a uniform enforcement point that is conceptually independent of the specific xPU architecture (e.g., GPU vs. NPU), thus providing a more generalizable solution than prior works that are often tailored to a specific accelerator's command submission workflow.

Weaknesses

1. The "In-line Security Appliance" Precedent: While the application to confidential xPU computing is novel, the fundamental concept of an in-line hardware appliance on a communication bus (be it PCIe, Ethernet, etc.) that filters, modifies, and secures traffic is not entirely new. The paper would be strengthened by more clearly positioning its novelty against this broader class of "bump-in-the-wire" security devices. The novelty is in the details of the packet handlers and the co-design with the TVM-side Adaptor, not the general idea of intercepting traffic.

2. Generalization Claim vs. Inherent Specificity: The paper's core claim is broad compatibility. However, the true novelty of a general solution is tested by its ability to handle diversity without becoming a collection of special cases. The "Packet Filter" (Section 4.1, page 6) relies on rules based on address spaces and requester IDs. Different xPUs and their drivers use MMIO and DMA regions in vastly different, sometimes idiosyncratic, ways. The paper does not provide sufficient evidence that a simple, generalizable rule set can be defined for truly disparate devices (e.g., an NVIDIA GPU vs. a Tenstorrent NPU) without requiring extensive, device-specific reverse engineering and tuning. The delta between ccAI and a device-specific solution may be smaller in practice than claimed.

3. Handling of Proprietary Sidebands and Packets: The architectural novelty assumes all critical communication happens over standard, well-documented PCIe packets. However, many complex devices use vendor-defined message types or sideband communication channels that may not be visible or interpretable as standard DMA/MMIO. The brief mention of "Customized packets" in the Discussion (Section 9, page 12) acknowledges this but doesn't fully address the threat to the novelty of a universal solution. If the PCIe-SC cannot parse or secure these proprietary flows, the security guarantees are incomplete, reducing the significance of the advancement.

Questions to Address In Rebuttal

1. The novelty of the "Adaptor" component hinges on it being a lightweight, minimally intrusive module. For a completely new xPU not evaluated in the paper (e.g., an AMD GPU or a Google TPU), what would be the precise engineering effort required to develop the necessary Adaptor hooks and Packet Filter rules? Please provide a concrete, step-by-step process. This will help clarify whether the solution is genuinely general or just a framework for creating bespoke drivers.

2. The packet filtering mechanism (Figure 5, page 6) is key to the design. Can the authors provide a concrete example of a non-trivial security policy that differentiates between two distinct operations on the same GPU? For example, how would the L1/L2 table rules distinguish between a legitimate kernel launch command write and a malicious MMIO write attempting to access a configuration register that could compromise isolation?

3. The proposed PCIe-SC is a novel hardware component. Its viability depends on its ability to scale with technology. The prototype is evaluated on a PCIe 4.0 system. What are the architectural bottlenecks in the PCIe-SC design (e.g., table lookup logic, cryptographic engine throughput) that would need to be overcome to support the line rates and lower latencies of future PCIe 6.0 or 7.0 interfaces? Is the proposed architecture fundamentally scalable?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose "Ironman," a Near-Memory Processing (NMP) architecture to accelerate Oblivious Transfer Extension (OTE), a critical component in many Privacy-Preserving Machine Learning (PPML) frameworks. The proposal involves a hardware/software co-design approach: for the compute-bound Single-Point Correlated OT (SPCOT) sub-protocol, they introduce an m-ary GGM tree expansion using a ChaCha-based PRG instead of the standard AES. For the memory-bound Learning Parity with Noise (LPN) sub-protocol, they propose an NMP architecture with a memory-side cache and a pre-sorting algorithm to improve data locality. While the paper identifies the correct bottlenecks, its central claims are predicated on a number of strong, and in some cases, unrealistic, hardware assumptions, and the reported performance gains appear to be misleadingly framed.

Strengths

1. The fundamental diagnosis of the performance bottlenecks in PCG-style OTE is sound. The identification of SPCOT as compute-bound and LPN as memory-bandwidth-bound (Section 1, page 2, Figure 1(c)) correctly motivates the need for distinct optimization strategies.
2. The algorithmic optimization for SPCOT, specifically the combination of m-ary tree expansion with a ChaCha8-based PRG, is a logical approach to reducing the total number of PRG calls. The ablation study presented in Figure 13(a) (Section 6.2, page 11) provides clear evidence that this combined approach is superior to applying either optimization in isolation.
3. The proposed index sorting algorithm for the LPN phase (Section 5.3, page 9) is an intelligent technique to mitigate the performance degradation from irregular memory accesses. The use of offline, compile-time sorting for a fixed matrix is a valid optimization strategy in principle.

Weaknesses

1. Fundamentally Unrealistic Hardware Model: The entire proposal hinges on the ability to place a custom, high-throughput "ChaCha8 Core" and other specialized logic (e.g., the Unified Unit) on the buffer chip of a DRAM DIMM (Section 5.1, page 7, Figure 9). The authors themselves concede in Section 5.1.3 (page 8) that deploying this on existing commercial NMP hardware like UPMEM or HBM-PIM would present "certain challenges" and would ultimately "require replacing the existing hardware with our custom ASIC." This admission relegates the work to a purely theoretical exercise. It is not an acceleration on near-memory processing as it exists, but on a hypothetical, full-custom memory system that is not commercially viable or available.
2. Misleading Presentation of Performance Gains: The abstract and headline results prominently feature a "39.2-237.4× improvement in OT throughput." However, the end-to-end application speedup for actual PPML frameworks is a far more modest 2.1-3.4× (Table 5, Section 6.5, page 11). While solving one bottleneck to reveal another (communication) is a common outcome, framing the work around a component speedup that is orders of magnitude larger than the real-world application benefit is misleading to the reader. The true impact is much smaller than suggested.
3. Insufficiently Rigorous Baseline Comparisons: The paper compares against a "full-thread CPU implementation" and a GPU implementation. The GPU baseline, an NVIDIA A6000, is a powerful accelerator. However, its implementation is described in a single sentence (Section 6.1, page 10), lacking any detail on the level of optimization. Without evidence of a highly optimized CUDA implementation that effectively leverages GPU architecture for this specific task, the 40.31x speedup claimed over the GPU baseline is suspect and likely inflated by comparing against a strawman implementation.
4. Unsupported Claims of Negligible Overhead: The authors claim in Section 5.1.3 (page 8) that the cost of offloading data to the host CPU "becomes negligible" because generation can be overlapped with transmission. This is an oversimplification that ignores the complexities of host-device synchronization, instruction dispatch overhead, and potential bus contention. This critical system-level cost is dismissed without any quantitative analysis or simulation data to support the claim.
5. Limited Applicability of LPN Optimization: The index sorting algorithm, which is key to the LPN speedup, requires the index matrix A to be fixed and known at compile time (Section 5.3, page 9). This is a strong assumption that may not hold for all PPML scenarios or future cryptographic protocols. The paper fails to discuss the implications or performance impact if this matrix were dynamic or generated on-the-fly.

Questions to Address In Rebuttal

1. Regarding the hardware model: Can you justify the practicality of your proposed architecture? Please provide a clear pathway for implementing this design on any existing or near-future NMP platform without requiring a full-custom ASIC on the DIMM buffer chip. If no such pathway exists, the contribution must be re-framed as purely theoretical.
2. Regarding the performance claims: Please reconcile the 237.4x component speedup highlighted in the abstract with the ~3x end-to-end application speedup from Table 5. Why is the most prominent number in the paper not representative of the actual system-level impact?
3. Regarding the GPU baseline: Please provide concrete details of your GPU implementation. Specifically, what CUDA kernels were designed, how was work distributed across thread blocks, and what memory optimizations (e.g., shared memory usage, coalesced access patterns) were employed? This is necessary to validate that your comparison is fair.
4. Regarding the offloading cost: Please provide a quantitative breakdown (e.g., from your ZSim simulation) of the overhead associated with the host processor dispatching NMP instructions and receiving the final COT correlations. How does this overhead scale with the number of correlations, and under what specific conditions is it truly "negligible"?
5. Regarding the LPN sorting algorithm: Please clarify the assumptions under which the index matrix A can be considered static and sorted offline. What is the performance impact on the LPN stage if this assumption is violated and the matrix must be processed without prior sorting?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents "Ironman," a specialized hardware accelerator designed to address a critical performance bottleneck in Privacy-Preserving Machine Learning (PPML): the Oblivious Transfer Extension (OTE) protocol. The authors correctly identify that as other parts of PPML frameworks are optimized, the cryptographic machinery for handling non-linear functions—which heavily relies on OTE—becomes the dominant cost.

The core contribution is a holistic hardware/software co-design. The authors cleverly partition the OTE protocol into its two main components: the computation-bound SPCOT and the memory-bound LPN. For SPCOT, they propose replacing the standard binary AES-based tree expansion with a more hardware-friendly 4-ary tree using a custom ChaCha8 core, significantly reducing the number of primitive operations. For LPN, they leverage a Near-Memory Processing (NMP) architecture with memory-side caches and a novel offline index sorting algorithm to mitigate the performance degradation from LPN's irregular memory access patterns. The design is thoughtfully unified to support the switching of sender/receiver roles inherent in many MPC protocols. The simulation-based results demonstrate a substantial 39x-237x speedup for the OTE protocol itself and a compelling 2.1x-3.4x end-to-end latency reduction for complex CNN and Transformer models within modern PPML frameworks.

Strengths

1. Excellent Problem Identification and Contextualization: The paper is situated at the crucial intersection of computer architecture, applied cryptography, and secure AI. The authors have correctly diagnosed that OTE is the next major frontier for optimization in making large-scale PPML practical. By profiling real frameworks (Figure 1, page 2), they provide strong motivation that this is not a theoretical problem, but a real-world engineering challenge for the deployment of private AI.

2. Sophisticated HW/SW Co-Design: This is the paper's most significant strength. Rather than naively accelerating the canonical OTE algorithm, the authors re-evaluate the algorithmic components from first principles with hardware in mind. The decision to move from a standard binary GGM tree with AES to a 4-ary tree with a custom ChaCha8 primitive (Section 4, page 6) is an outstanding example of co-design. It showcases a deep understanding of both the cryptographic requirements and the hardware implementation trade-offs, leading to a 6x performance improvement for the SPCOT component (Figure 13, page 11).

3. Sound and Modern Architectural Approach: The use of Near-Memory Processing (NMP) for the memory-bound LPN component is a well-justified and modern architectural choice. The design insight that LPN's random access patterns can be regularized via offline sorting (Section 5.3, page 9) and serviced efficiently by distributed memory-side caches and rank-level parallelism is very compelling. This correctly maps the problem's characteristics (low compute intensity, high memory bandwidth demand) to an appropriate architectural solution.

4. Enabling Practical Private AI for Complex Models: By achieving significant end-to-end speedups, this work pushes the boundary of what is considered feasible for PPML. The evaluation on large Transformer models (ViT, BERT, GPT-2) is particularly important, as these models are often considered prohibitively expensive to run in a secure context. This work provides a credible pathway to deploying such state-of-the-art models with strong privacy guarantees.

Weaknesses

While the core ideas are strong, the paper could be improved by addressing the following points, which are more about completeness than fundamental flaws:

1. Positioning of Novelty: The paper's novelty lies in the synthesis and application of existing ideas to a new, important domain. NMP, custom cryptographic cores, and index sorting for sparse operations are established techniques. The paper would be stronger if it more explicitly framed its contribution not as the invention of these components, but as the first successful synthesis of them into a coherent accelerator for the PCG-style OTE bottleneck.

2. Practicality of NMP Integration: The paper relies on a simulated NMP environment. While this is standard for architectural research, a brief discussion on the practical path to deployment would be valuable. For instance, what modifications would be needed in the OS/memory controller to manage the NMP units? How would the Ironman software stack integrate with existing PPML frameworks like PyTorch or TensorFlow, which are often the front-ends for these secure back-ends?

3. Overhead of Offline Pre-processing: The index sorting algorithm for LPN is performed offline. The paper states this cost is amortized because the LPN matrix is fixed. While true for many inference scenarios, it would be useful to quantify this one-time cost. Furthermore, a discussion on scenarios where the matrix might not be fixed (e.g., certain types of secure training or dynamic systems) would add nuance and scope the applicability of this specific optimization.

Questions to Address In Rebuttal

I am broadly supportive of this work and believe it makes a valuable contribution. I encourage the authors to use the rebuttal to address the following to strengthen the final paper:

1. Regarding the LPN optimization (Section 5.3, page 9): Could you quantify the one-time computational cost of the offline column and row sorting algorithm? How does this cost scale with the size of the LPN matrix, and at what point might it become a consideration in the overall workflow?

2. Regarding the architectural scalability: Your evaluation shows strong scaling up to 16 ranks. As you scale to a larger number of DIMMs/ranks, do you foresee the final XOR reduction in the DIMM-NMP module (Figure 9b, page 8) becoming a new bottleneck, and if so, how might your design address this?

3. Regarding the algorithmic co-design (Section 4.1, page 6): You make a compelling case for using a 4-ary tree with ChaCha8. Could you provide a little more architectural insight into why ChaCha8 is so well-suited for a pipelined hardware implementation compared to, for example, a pipelined AES implementation? Is it primarily the longer output, or are there also advantages in the simplicity of its core operations (Add-Rotate-XOR) that lead to a more area- or power-efficient pipeline?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents "Ironman," a hardware accelerator architecture for PCG-style Oblivious Transfer Extension (OTE), targeting the performance bottleneck in modern Privacy-Preserving Machine Learning (PPML) frameworks. The authors identify two key components of OTE: the computation-bound Single-Point Correlated OT (SPCOT) and the memory-bandwidth-bound Learning Parity with Noise (LPN). To address these, they propose a two-pronged solution: 1) a novel "hardware-aware" m-ary GGM tree expansion algorithm for SPCOT that uses a ChaCha-based PRG instead of AES, and 2) a Near-Memory Processing (NMP) architecture with an index sorting algorithm to improve data locality for LPN.

My primary concern is the degree of conceptual novelty. While the paper claims to be the first customized accelerator for PCG-style OTE, which appears to be true, the underlying techniques employed are largely adaptations of well-known principles from other domains. The primary contribution is therefore the synthesis and application of these existing ideas to a new problem, rather than the invention of fundamentally new architectural or algorithmic concepts.

Strengths

1. First-Mover on a Relevant Problem: To the best of my knowledge, this is the first work to propose a dedicated, end-to-end hardware architecture for accelerating PCG-style OTE. Previous work on OT acceleration (e.g., POTA [94]) has focused on other variants like IKNP-style OT, which have different performance characteristics. Identifying and targeting this specific, computationally intensive protocol is a timely contribution.

2. Hardware/Algorithm Co-Design for SPCOT: The most novel element of this work is the proposed co-design for SPCOT in Section 4 (Page 6). The idea of replacing the standard 2-ary AES-based GGM tree with an m-ary ChaCha-based tree is a clever, hardware-centric optimization. While m-ary trees have been explored for V-OLE [88], the explicit analysis and motivation for coupling this with a specific, hardware-friendly PRG like ChaCha8 to reduce operator count and improve area/power efficiency (Table 2, Page 5) constitutes a legitimate, albeit specialized, engineering novelty.

Weaknesses

1. Limited Conceptual Novelty in LPN Acceleration: The proposed solution for the memory-bound LPN component lacks fundamental novelty. The core ideas are:

   • Applying NMP: Using Near-Memory Processing to accelerate a memory-bandwidth-bound problem with low computational intensity is the foundational premise of NMP research. The application here is a straightforward, albeit effective, use case. It does not introduce a new NMP paradigm.
   • Index Sorting: The "Index Sorting Algorithm for Memory-side Cache" described in Section 5.3 (Page 9) is conceptually identical to decades of work on improving data locality for Sparse Matrix-Vector Multiplication (SpMV). The authors themselves formulate LPN as an SpMV problem. Techniques such as column and row reordering to cluster non-zero elements and improve cache utilization are standard practice in the high-performance computing (HPC) and compiler communities. The application to LPN is new, but the technique itself is not.

2. "Unified Architecture" is Standard Design Practice: The claim of novelty for a unified architecture supporting both sender and receiver roles (Section 5.2, Page 8) is overstated. Designing a datapath with shared resources (like an XOR tree) that can be reconfigured through control logic to perform slightly different but related functions is a standard, fundamental principle of efficient hardware design to minimize area. While a necessary feature for a practical implementation, it does not constitute a novel research contribution.

3. The "Delta" is Primarily Application-Specific: The paper's main strength—being the first accelerator for PCG-style OTE—is also the source of its weakness from a novelty perspective. The contributions are highly specific to this protocol. The core takeaways for a general hardware architect are "use NMP for memory-bound problems" and "reorder memory accesses to improve locality," both of which are already known. The paper does an excellent job of system integration and engineering, but the set of new, generalizable concepts is small.

Questions to Address In Rebuttal

The authors should focus their rebuttal on clarifying the conceptual advances beyond the direct application to PCG-style OTE.

1. On LPN Acceleration: The authors frame the LPN operation as an SpMV problem (Section 5.3, Page 9). Please explicitly differentiate your proposed "Column Swapping + Row Looking-ahead" sorting algorithm from existing graph partitioning or matrix reordering algorithms used to optimize SpMV performance in the HPC domain. What is fundamentally new about your sorting approach that is uniquely tailored to the structure of LPN and not just an application of a known technique?

2. On Conceptual Contribution: Beyond being the first architecture for this specific task, what is the single most important conceptual and generalizable contribution of this work? Is there a new principle of cryptographic acceleration or NMP design that a future architect, working on a different protocol, could learn and apply from this paper?

3. On the "Hardware-Aware" m-ary Tree: The co-design of the m-ary tree with ChaCha is presented as a key contribution. Could the authors elaborate on whether this principle is more profound than a simple substitution of a more parallelizable PRG? For instance, does the structure of the ChaCha output influence the optimal choice of m in a way that would not apply to other long-output PRGs? This would help solidify the "co-design" aspect as more than just a component swap.

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors attempt to reverse engineer the core microarchitecture of modern NVIDIA GPUs (Turing, Ampere, and Blackwell) through a series of microbenchmarks. Based on their inferences, they propose a new core model for the Accel-sim framework. The paper claims this new model significantly improves simulation accuracy, reducing the Mean Absolute Percentage Error (MAPE) from over 34% to approximately 13.5% for an Ampere GPU when compared to the baseline Accel-sim. The primary contributions center on elucidating the function of compiler-managed control bits for dependency tracking, a new issue scheduler policy, and refined models for the register file and memory pipeline.

Strengths

1. The ambition of the work is commendable. Tackling the opaque nature of modern commercial GPU architectures is a difficult and labor-intensive task.
2. The paper provides a wealth of specific, quantitative data in its tables (e.g., Table 2, Memory instruction latencies) which, if accurate, could serve as a useful reference.
3. The authors have clearly invested significant effort in creating and running numerous microbenchmarks to generate the timing data that forms the basis of their hypotheses.

Weaknesses

My primary concerns with this manuscript are the opacity of the methodology, the logical leaps made from limited evidence, and the potential for an unfair baseline comparison, which may inflate the significance of the reported results.

1. Methodological Rigor and Reproducibility: The reverse engineering methodology described in Section 3 is presented anecdotally. The authors provide two illustrative examples (Listing 1, scheduler policy) but fail to describe the systematic process and full scope of their investigation. How many microbenchmark variants were run to derive each conclusion? What was the process for ruling out alternative hypotheses? Without a rigorous account of the methodology, the findings appear to be a collection of observations from hand-picked scenarios rather than the result of a comprehensive, scientific dissection. The work is therefore difficult to validate or trust.

2. Overstatement of Inferred "Discoveries": The paper presents its hypotheses as definitive facts. For instance, the proposed "Compiler Guided Greedy Then Youngest (CGGTY)" issue policy (Section 5.1.2) is a strong claim based on insufficient evidence. Figure 4 shows only three specific scenarios with homogeneous, independent instructions. This is hardly a comprehensive evaluation required to definitively characterize a complex scheduler. How does the policy behave under heavy memory contention, with long-latency instructions, or around synchronization primitives? The evidence supports CGGTY as a plausible hypothesis for a limited case, not as a confirmed mechanism. This pattern of over-claiming permeates the paper (e.g., register file organization, front-end policy).

3. Questionable Baseline for Comparison: The validation in Section 7 hinges on a comparison against "the Accel-sim simulator." It is well-known that the public Accel-sim model is largely based on the much older Tesla/Fermi architecture. Comparing a new model tailored for Ampere against a decade-old architectural model and then claiming a >20% reduction in MAPE is not a fair or insightful comparison. It proves that Ampere is different from Fermi, which is already known. The authors have not demonstrated that their model is superior to a state-of-the-art academic model properly configured for a more recent architecture. This appears to be a "strawman" comparison.

4. Unsupported Claims Regarding the Blackwell Architecture: The claim to have modeled the very recent Blackwell architecture with high accuracy (17.41% MAPE, Table 4) is not adequately substantiated. In Section 6, the authors mention only supporting new SASS instructions and extending the L2 hashing function. These are minor adjustments and it is highly improbable that they capture the full microarchitectural evolution from Ampere to Blackwell. This claim feels premature and potentially misleading.

5. Contamination of Validation Results: A critical detail is buried at the end of Section 6: for some kernels, where SASS code was unavailable, a "hybrid mode" using "traditional scoreboards" was employed. This is a major confounding variable. The paper’s central thesis is about the accuracy of a new model based on compiler-set control bits. However, the final accuracy numbers are an amalgam of this new model and an entirely different, traditional dependency model. The authors do not state for which of the 128 benchmarks this hybrid mode was used, making it impossible to assess the true accuracy of their primary contribution.

Questions to Address In Rebuttal

The authors must provide clear and concise answers to the following questions to alleviate the concerns raised.

1. Regarding your methodology (Section 3), can you provide a quantitative summary of the reverse engineering effort? For example, for the issue scheduler policy alone, how many distinct microbenchmark scenarios (beyond the three in Figure 4) were constructed and tested to validate the CGGTY policy against other plausible alternatives (e.g., LRR, GTO)?

2. Regarding the baseline (Section 7.2), please specify the exact version and configuration of Accel-sim used as the baseline. Justify why this configuration, which primarily models an older architecture, is considered a fair and relevant baseline for comparison against modern Ampere/Turing hardware.

3. Regarding the "hybrid mode" for dependency tracking mentioned in Section 6, please provide a list of the benchmarks from your validation suite (Table 3) that required this mode. Furthermore, present a separate MAPE analysis for the subset of benchmarks that ran purely on your proposed control-bit model versus the subset that used the hybrid scoreboard model.

4. Regarding your Blackwell model, please provide a comprehensive list of all microarchitectural changes implemented beyond the Ampere model. How do you justify that this limited set of changes is sufficient to claim an accurate model of the Blackwell core architecture, rather than simply an "Ampere-plus" model that coincidentally performs well on your chosen benchmarks?

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a comprehensive reverse-engineering study of modern NVIDIA GPU core microarchitectures, from Turing to the latest Blackwell generation. The authors' primary contribution is bridging the significant and growing gap between the aging architectural models used in academic simulators (like Accel-sim) and the reality of contemporary commercial hardware. Through meticulous micro-benchmarking using hand-written SASS code, the work uncovers crucial, previously undocumented design details. These include the compiler-driven mechanism for managing data dependencies (replacing traditional hardware scoreboards), a novel "Compiler Guided Greedy Then Youngest" (CGGTY) issue policy, and detailed models of the multi-banked register file, its associated cache, and the memory pipeline. The authors integrate these findings into a new, publicly available simulator model, demonstrating a dramatic improvement in accuracy—reducing the Mean Absolute Percentage Error (MAPE) from over 34% to 13.45% on an Ampere GPU compared to the baseline Accel-sim. This work serves as both a significant contribution to the scientific understanding of modern GPU design and a vital update to the community's research infrastructure.

Strengths

1. High-Impact Contribution to Research Infrastructure: The most significant strength of this work is its direct and potent address of the "relevance crisis" in academic GPU simulation. For over a decade, much of the community's research has been predicated on simulators modeling architectures from the Tesla era (circa 2006). This paper performs the herculean task of updating our collective understanding and providing a tangible, validated tool that will elevate the quality and relevance of future research in areas like scheduling, memory systems, and compiler optimizations for GPUs. The validation across three major architectural generations (Turing, Ampere, Blackwell) underscores its immediate and likely future utility.

2. Unveiling a Fundamental Architectural Paradigm Shift: The paper's detailed exposition of the software-hardware co-design for dependency management (Section 4, page 3 and Section 7.5, page 13) is a profound insight. The move away from complex, power-hungry hardware scoreboards towards compiler-managed Stall counters and Dependence counters represents a major philosophical shift in GPU design. This finding contextualizes modern GPUs within a broader architectural trend towards compiler-led complexity management, reminiscent of VLIW principles. This discovery alone is a major contribution to computer architecture literature.

3. Methodological Depth and Rigor: The authors' methodology of using carefully crafted, low-level SASS microbenchmarks to probe the hardware's behavior is commendable. This is a non-trivial undertaking that requires deep expertise. The detailed examples provided (e.g., Listing 1 for register file conflicts, the analysis in Figure 4 for the scheduler policy) lend significant credibility to their inferred models. This empirical, bottom-up approach is exactly what is needed to demystify these otherwise black-box systems.

4. Holistic and Coherent Model: Unlike prior works that often focused on reverse-engineering a single component (e.g., a cache, a specific unit), this paper presents a coherent model of the entire core pipeline. It successfully connects the dots between the front-end fetch/decode, the issue logic, the register file, and the memory subsystems, showing how they interact. The discovery of the CGGTY issue policy and its interaction with the compiler-set Yield bit is a perfect example of this holistic view.

Weaknesses

As a contextual analyst, I view these less as flaws and more as inherent limitations or opportunities for deeper discussion.

1. Inherent Ambiguity of Reverse Engineering: The work constructs a highly plausible and well-validated model, but it is ultimately an inferred model. The authors are commendably transparent about this (e.g., acknowledging in Section 5.1, page 6, that they "could not find a model that perfectly fits all the experiments"). The paper could benefit from a brief, consolidated discussion on the limitations of this approach and the confidence bounds on their conclusions. While the MAPE reduction is impressive, it's important to frame the resulting model as a powerful and accurate approximation, not necessarily ground truth.

2. Limited Exploration of the "Why": The paper excels at detailing the "what" (the mechanisms) and the "how" (their operation). However, it offers little speculation on the "why"—the architectural trade-offs that likely motivated these design choices. For example, why did NVIDIA pivot so heavily to compiler-managed dependencies? Was the primary driver area savings, power reduction, clock speed improvements, or simpler hardware verification? Adding a short discussion section to hypothesize on these design rationales would elevate the paper from a descriptive masterpiece to a more complete architectural analysis.

3. NVIDIA-Centric Focus: The paper's deep dive is exclusively on NVIDIA architectures. This is a practical and understandable choice given NVIDIA's market position and the sheer scope of the work. However, it implicitly positions the NVIDIA way as the modern GPU way. While a brief comparison to AMD's waitcnt is included (Section 5, page 5), the work would be even more valuable to the broader community if it could contextualize its findings by more explicitly contrasting them with the known design philosophies of other major GPU vendors.

Questions to Address In Rebuttal

1. The core of your work rests on a complex, inferred model of the GPU pipeline. Beyond the aggregate MAPE scores, were there any specific micro-architectural behaviors or corner-case instruction sequences that your final model still struggled to accurately predict? Discussing these outliers could provide valuable clues for future reverse-engineering efforts.

2. Could you elaborate on the likely architectural motivations behind NVIDIA's shift to the compiler-managed dependency system? What are the primary trade-offs (e.g., in terms of area, power, performance, and compiler complexity) compared to the traditional hardware scoreboard approach that your own results in Section 7.5 (page 13) so clearly favor?

3. This work represents a monumental effort to re-synchronize academic tools with a fast-moving industry target. Looking forward, how sustainable is this manual, intensive reverse-engineering process? Does your experience suggest any potential avenues for semi-automating the discovery of such microarchitectural properties for future GPU generations, to ensure the community's research tools do not fall so far behind again?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present a work of microarchitectural reverse engineering and modeling for modern NVIDIA GPU cores (Turing, Ampere, and Blackwell). The central claim is that existing academic simulators are based on antiquated architectural assumptions (dating back to the Tesla architecture) and are therefore inaccurate. This paper seeks to remedy this by discovering and modeling key features of modern designs. The core novel claims center on the detailed mechanics of a compiler-hardware co-design for managing data dependencies, a specific issue scheduler policy ("Compiler Guided Greedy Then Youngest"), the internal structure of the register file (RF) and its cache, and the absence of an operand collector stage. These findings are integrated into the Accel-sim framework, and the resulting model is shown to be significantly more accurate than the baseline.

Strengths

The primary strength of this paper is the novelty of its empirical findings. While many papers propose new architectural ideas, this work provides a rare and valuable service by reverse-engineering and documenting a complex, proprietary, state-of-the-art commercial architecture. The novelty is not in inventing new mechanisms, but in revealing existing ones for the first time in the public domain.

1. Novel Semantics of Dependence Management: The most significant novel contribution is the detailed elucidation of the software-based dependence management system (Section 4, pages 3-4). The existence of compiler-inserted "hints" is not new; this was observed in the Kepler architecture and noted by Jia et al. [46, 47] for Volta/Turing. However, prior work has not provided a functional, mechanistic model. This paper's detailed breakdown of the Stall counter for fixed-latency hazards and the system of six Dependence counters (SBx registers) with producer-increment/consumer-decrement semantics for variable-latency hazards is a genuinely new contribution to public knowledge. The explanation of the DEPBAR.LE instruction and the Yield bit provides a complete, plausible model that has not been described elsewhere.

2. Refutation of the Operand Collector Assumption: The paper makes a strong claim that modern NVIDIA GPUs do not use an operand collector unit (Section 5.3, page 8). This is a direct and novel refutation of a core assumption in the widely-used GPGPU-Sim and Accel-sim models. The authors' reasoning—that variable latency from an operand collector would break the compiler's ability to calculate static stall cycles—is logical and compelling. Disproving a long-held assumption in a dominant model is a significant form of novelty.

3. Specific Characterization of the Issue Scheduler: While greedy scheduling policies are well-known, the specific "Compiler Guided Greedy Then Youngest" (CGGTY) policy (Section 5.1.2, page 6) is a novel finding. This moves beyond the canonical "Greedy Then Oldest" (GTO) policy and demonstrates a tight coupling between the scheduler's fallback mechanism (picking the youngest warp) and the compiler's explicit instructions to yield the pipeline (Yield bit). The detailed experimental timeline in Figure 4 provides strong evidence for this specific, previously undocumented policy.

4. Timeliness of Blackwell Modeling: The claim of being the first work to provide an accurate, validated model for the NVIDIA Blackwell architecture (Section 1, page 1 and Section 7.2, page 12) is a strong point of novelty. Given the recent release of this architecture, this contribution is at the cutting edge of academic GPU modeling.

Weaknesses

The weaknesses of the paper, from a novelty perspective, lie in areas where the contributions are more confirmatory or incremental rather than fundamentally new concepts.

1. Incremental Novelty of the RF Cache: The concept of a compiler-managed register file cache for GPUs is not a new idea. The paper's model is explicitly and correctly identified as being "similar to the work of Gebhart et al. [34]" (Section 5.3.1, page 8). While the reverse-engineered parameters—such as one entry per bank and specific software management via a "reuse bit"—are new empirical details for NVIDIA's implementation, the core architectural concept has been established in prior art for over a decade. The delta here is one of implementation-specifics, not of fundamental mechanism.

2. Assumed Prefetcher Design: The front-end model relies on a simple stream buffer for instruction prefetching (Section 5.2, page 7). This is a classic mechanism, and the authors state that they "suspect it is a simple scheme" and "assume" its size is 8 entries. While this assumption is validated by the model's overall accuracy, the contribution lacks the rigor of the reverse engineering applied elsewhere. As a contribution, proposing a standard, well-known prefetcher is not novel.

3. Known Trade-off Analysis: The analysis in Section 7.5 (page 13) comparing the discovered software-based dependence management to a traditional hardware scoreboard is an evaluation of a known design trade-off. The conclusion that a software-managed approach has lower area overhead is expected. The value is in quantifying this trade-off with realistic parameters, but this does not represent a new conceptual insight into computer architecture.

Questions to Address In Rebuttal

1. On the Fetch Policy's Novelty: The front-end fetch policy is assumed to mirror the issue scheduler's greedy logic (Section 5.2, page 7). This is presented as a plausible assumption rather than a direct finding. What experiments were performed to rule out other well-known fetch policies (e.g., round-robin, ICOUNT)? How much of the model's accuracy hinges on this specific assumption, which appears less rigorously proven than other claims in the paper?

2. Clarifying the Delta vs. Gebhart et al.: The paper acknowledges the similarity of its RF cache model to that of Gebhart et al. [34]. Beyond the parametric differences and the lack of a two-level scheduler, could the authors more sharply define the novel architectural principle, if any, that their findings reveal? Is NVIDIA's implementation merely a modern instantiation of the 2011 concept, or is there a more fundamental conceptual difference that this review has missed?

3. On the Generality and Longevity of Findings: The detailed semantics of the control bits and dependence counters are a key contribution. These were reverse-engineered across three recent architectures. Based on this, do the authors have evidence to suggest that this specific mechanism is a stable, long-term feature of NVIDIA's design philosophy, or is it an artifact of a particular architectural era? How much risk is there that the next generation invalidates this detailed model, thereby limiting the durable novelty of these specific findings?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper introduces BITGEN, a code generator that compiles multi-pattern regular expressions into bitstream programs for execution on GPUs. The central thesis is that a "sequential block-wise" execution of these programs is inefficient due to poor data reuse and high memory traffic. The authors propose an "interleaved execution" model, fusing all bitstream instructions into a single GPU kernel loop. To address the primary technical challenge of cross-block data dependencies introduced by shift operations, they present three techniques: 1) Dependency-Aware Thread-Data Mapping (DTM) which resolves dependencies via selective recomputation, 2) Shift Rebalancing (SR) which restructures the dataflow graph to reduce synchronization, and 3) Zero Block Skipping (ZBS) which adds conditional checks to avoid computation on sparse, zero-filled blocks. The authors claim their system achieves a geometric mean speedup of 19.5x over ngAP, a state-of-the-art GPU regex engine.

Strengths

1. Correct Problem Identification: The paper correctly identifies a critical performance bottleneck in the naive GPU implementation of bitstream-based regex matching—namely, the memory traffic and lack of data reuse inherent in executing instructions in separate, sequential loops (as described in Section 3.2, page 4).
2. Sound Core Concept: The proposed interleaved execution model is a logical approach to improving data locality. Fusing operations into a single loop to keep intermediate results in registers is a well-established optimization principle, and its application here is well-motivated.
3. Comprehensive Technical Approach: The authors demonstrate a thorough understanding of the challenges arising from their proposed execution model. The three primary contributions (DTM, SR, ZBS) form a cohesive set of solutions that systematically address the problems of cross-block dependencies, synchronization overhead, and redundant computation.

Weaknesses

My primary concerns with this work relate to the robustness of the core methodology, the fairness of the experimental comparison, and the practical scalability of the proposed system.

1. Unresolved Failure Mode in Core Technique: The Dependency-Aware Thread-Data Mapping (DTM) relies on recomputing data from an overlapping window of the previous block. The paper itself acknowledges a critical limitation: the required overlap distance can exceed the size of a single block, making dependencies unresolvable with the current method (Section 8.2, page 12, "Discussion: Limits of Overlap Distance in DTM"). The authors propose a "fallback mechanism" but state, "We leave the fallback mechanism as future work." This is a significant flaw. The system, as presented, is not robust. The evaluation may have succeeded only because the chosen benchmarks (Table 1, page 10) do not contain patterns (e.g., /.*/ on long lines, or large bounded repetitions) that trigger this known failure condition. The claims of the paper rest on a technique that is demonstrably incomplete.

2. Questionable Comparison to ngAP: The headline claim of a 19.5x speedup over ngAP is staggering and requires extraordinary evidence. However, the comparison appears to be between two fundamentally different execution paradigms. BITGEN uses a bit-parallel model derived from Parabix, while ngAP uses an automata-based model. The performance of these models is highly dependent on the specific characteristics of the regex set and input data. The paper offers a brief explanation that ngAP suffers from irregular memory access and limited worklist size (Section 8.1, page 10), but fails to provide a rigorous analysis of why the chosen benchmarks are so particularly unsuited to an automata-based approach and so perfectly suited to a bitstream approach. Without this analysis, the 19.5x speedup seems less like a fundamental improvement and more like an artifact of benchmark selection that favors the authors' paradigm.

3. Overstated Novelty of Zero Block Skipping: The paper claims that interleaved execution "enables" Zero Block Skipping (ZBS), implying that it is not possible in a sequential model (Section 6, page 9). This is an overstatement. A sequential block-wise implementation could readily check if an intermediate bitstream block is all zeros after one kernel finishes and, if so, skip the launch of subsequent kernels that operate on it. While the authors' fused approach may implement this check more efficiently (avoiding kernel launch overhead), it does not uniquely enable it. The claim should be tempered to reflect a benefit of efficiency rather than capability.

4. Inherent Scalability Limitation: The execution model assigns a regex group to a single Cooperative Thread Array (CTA) to "simplify synchronization" (Section 9, page 13). The authors admit this comes "at the cost of per-regex scalability." This is not a minor trade-off; it is a primary architectural bottleneck. For any workload dominated by one or a few highly complex regexes, the system has no mechanism to parallelize the work beyond a single SM. The impressive throughput numbers reported may only be achievable on workloads with thousands of simple regexes that can be evenly distributed. This severely limits the applicability of the system in more challenging, real-world scenarios.

Questions to Address In Rebuttal

1. Regarding the unhandled failure mode of DTM: Please provide experimental results for regex patterns known to generate large overlap requirements (e.g., (.*){N} or [a-z]{16384,} on matching input). How does the system perform, and at what point does it fail? Justifying the system's correctness requires addressing this worst-case behavior, not just the average cases presented.

2. Regarding the 19.5x speedup over ngAP: Can you provide a more detailed, quantitative analysis of the benchmark regexes to demonstrate that the performance gap is not merely an artifact of comparing two disparate execution models on a dataset that strongly favors one? For instance, what is the ratio of literal-heavy patterns versus patterns with complex control flow (alternation, Kleene stars), and how do the two systems' performances correlate with these features?

3. Regarding the single-CTA execution model: Please quantify the performance limitation of this design. For example, what is the throughput when matching only the single most complex regex from the Snort or ClamAV benchmarks across the entire GPU, and how does this compare to the throughput of Hyperscan on a single CPU core? This would provide a more honest assessment of the system's scalability limitations.

4. Regarding the recomputation overhead of DTM: Table 5 (page 12) shows the average-case overhead is low. What is the maximum observed recomputation overhead for any single iteration in your experiments, and under what conditions does it occur? The average can hide pathologically expensive iterations that may impact tail latency.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents BITGEN, a code generator that fundamentally rethinks how bit-parallel regular expression programs are executed on GPUs. The authors identify a critical performance bottleneck in the straightforward approach: executing bitstream instructions sequentially across the entire input (instruction-wise) leads to poor data reuse and massive memory traffic for intermediate results.

The core contribution is a shift to an interleaved (block-wise) execution model, where a block of data is processed through all bitstream instructions before moving to the next block. This model is enabled by a clever technique called Dependency-Aware Thread-Data Mapping (DTM), which resolves the cross-block data dependencies inherent in SHIFT operations not by complex state-forwarding mechanisms, but by selectively recomputing a small number of overlapping bits. This core idea is further enhanced by two well-motivated compiler optimizations: Shift Rebalancing to shorten critical dependency paths and Zero Block Skipping to exploit data sparsity.

The result is a system that achieves a remarkable 19.5x geometric mean speedup over a state-of-the-art GPU automata-based engine (ngAP) and significantly outperforms highly optimized multi-threaded CPU engines like Hyperscan on many workloads.

Strengths

1. A Powerful and Elegant Core Idea: The central concept of switching from an instruction-wise to a block-wise execution model is a paradigm shift for this problem on GPUs. The decision to resolve dependencies via recomputation is particularly insightful. It trades a small amount of redundant computation—a resource GPUs have in abundance—for a massive reduction in memory traffic and synchronization overhead, which are often the true bottlenecks. This is a beautiful example of architecture-aware algorithm design. The clarity of this idea is well-illustrated in the contrast between Figure 1(a) and Figure 1(b) on page 1.

2. Exceptional Performance and Empirical Validation: The performance gains reported are not merely incremental; they represent an order-of-magnitude improvement over prior GPU art. The thorough evaluation against established CPU and GPU baselines (Section 8, pages 10-13) convincingly demonstrates the effectiveness of the proposed approach. The performance breakdown analysis (Figure 12, page 11) is a particular strength, as it clearly isolates the contribution of each proposed technique, confirming that the entire system is well-designed, not just a single trick. The analysis of DRAM traffic reduction (Table 4, page 12) directly validates the paper's initial hypothesis about data reuse.

3. Excellent Synthesis of Existing Concepts into a Novel Framework: This work does not exist in a vacuum, and its strength lies in how it connects disparate fields. It builds upon the bitstream program representation from the CPU-focused world of Parabix [24, 50]. It then applies classic compiler optimization principles, such as dependency graph restructuring (Shift Rebalancing) and value-based optimization (Zero Block Skipping), to this representation. Finally, it tailors the execution model specifically for the GPU architecture (data-parallelism, memory hierarchy, CTA-local synchronization). The result is a system that is greater than the sum of its parts and successfully ports a powerful but previously CPU-bound paradigm to the GPU with resounding success.

4. Opens a New Avenue for GPU Stream Processing: While the paper is framed around regex matching, the core pattern of "interleaved execution with selective recomputation" has broader implications. It provides a template for executing any data-parallel stream processing algorithm with local, sliding-window dependencies (e.g., 1D stencils, certain sequence alignment tasks) on a GPU without suffering from the "halo exchange" problem at the global memory level. The paper effectively introduces a new, powerful pattern for GPU computing.

Weaknesses

1. Limited Discussion of Scalability Beyond a Single CTA: The current model simplifies synchronization by assigning a group of regexes to a single Cooperative Thread Array (CTA), as mentioned in Sections 3.1 and 9. While this is a pragmatic choice for the multi-pattern problem, it sidesteps the challenge of scaling a single, extremely complex regex pattern across multiple CTAs. Such a scenario would require inter-CTA synchronization to handle cross-block dependencies, a notoriously difficult problem on GPUs. A more thorough discussion of the architectural implications and potential solutions for this would strengthen the paper's positioning.

2. Understated Generality of the Core Technique: The authors frame their work tightly around regex matching. However, as noted in the strengths, the underlying technique of resolving sliding-window dependencies via recomputation is potentially much more general. The paper could have a greater impact if it dedicated a short section to discussing how this execution model could be abstracted and applied to other stream-processing domains that have similar dependency patterns. This would elevate the contribution from a "fast regex engine" to a "novel parallel execution strategy for stream computations."

3. Potential for Pathological Cases in Dynamic Overlap: The paper handles dynamic overlap distances arising from while loops (Figure 7, page 7), which is a non-trivial accomplishment. However, the analysis of its limits feels brief (Discussion in Section 8.2, page 12). Regular expressions can contain nested Kleene stars (e.g., /(a*b*)*/), which could potentially lead to data-dependent overlap requirements that grow very rapidly. A deeper discussion on the theoretical bounds of this overlap and the performance cliff that might occur if it exceeds the practical limit (e.g., the size of a block) would provide a more complete picture of the technique's robustness.

Questions to Address In Rebuttal

1. Regarding the single-CTA execution model: What are the primary challenges you foresee in extending the Dependency-Aware Thread-Data Mapping to a multi-CTA model for a single, large bitstream program? Would it require global synchronization, or could a more asynchronous, pipelined model between CTAs be devised?

2. The core idea of trading recomputation for memory locality is powerful. Do you view this as a general-purpose parallel programming pattern for GPUs? Could the BITGEN compiler framework be extended to support other stream-processing DSLs that exhibit similar local dependency structures?

3. Could you elaborate on the worst-case behavior of the dynamic overlap distance calculation? Are there specific classes of regular expressions or input data patterns that would cause the required overlap Δ to become prohibitively large, and if so, how does your system currently handle such a scenario? You mention a "fallback mechanism" as future work; could you briefly sketch what that might entail?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present BITGEN, a code generator for compiling multi-pattern regular expressions into efficient GPU kernels. The central claim of novelty lies in a shift from the standard sequential, instruction-by-instruction execution of bitstream programs to an "interleaved" execution model. In this model, all bitstream instructions are fused into a single block-wise loop. To make this non-trivial transformation correct and efficient, the authors introduce three key techniques: 1) Dependency-Aware Thread-Data Mapping (DATDM), which resolves cross-block data dependencies (primarily from SHIFT operations) via selective recomputation; 2) Shift Rebalancing, a compiler pass that restructures the dataflow graph to shorten dependency chains; and 3) Zero Block Skipping, which exploits bitstream sparsity within the interleaved model to avoid redundant work. The authors claim this new execution model and its enabling techniques yield significant performance improvements over state-of-the-art CPU and GPU regex engines.

Strengths

My evaluation is based exclusively on the novelty of the proposed ideas relative to prior art.

1. Novel Execution Model for Bitstream Programs: The core concept of interleaving the execution of a complete bitstream program on a per-data-block basis is a genuine departure from the established model described in works like Parabix [24, 50], which processes one instruction at a time across all data. While loop fusion is a classic compiler optimization, its application here is non-trivial due to the unique cross-block, data-dependent control flow inherent to bitstream programs. The novelty is not "loop fusion" itself, but the creation of a viable framework to apply it to this specific, challenging domain on GPUs.

2. Principled Resolution of Cross-Block Dependencies: The primary obstacle to the interleaved model is resolving dependencies from SHIFT operations across block boundaries. The authors' proposed solution, Dependency-Aware Thread-Data Mapping (DATDM), is novel. Instead of attempting complex state-forwarding mechanisms, they opt for selective recomputation. The method for systematically analyzing the dataflow graph to calculate the required overlap distance (Δ), including handling dynamic distances from while loops (Section 4.2, page 6), appears to be a new and well-defined technique. It provides a principled way to manage the recompute-vs-communicate trade-off for this problem.

3. Novel Compiler Optimization (Shift Rebalancing): The concept of operand rewriting (e.g., (A >> n) & B -> (A & (B << n)) >> n) is based on fundamental algebraic properties. However, the systematic application of this identity in a dedicated compiler pass to rebalance a bitstream program's dataflow graph for improved GPU instruction-level parallelism (Section 5, page 7) is a novel contribution in compilation engineering. It addresses a specific performance bottleneck (long dependency chains) created by the bitstream abstraction.

Weaknesses

My concerns relate to the boundaries and fundamental nature of the claimed novelty.

1. Limited Discussion of the Novelty's Generality: The core enabling technique, DATDM, relies on recomputing a finite number of bits from an adjacent block. As the authors themselves briefly acknowledge in "Discussion: Limits of Overlap Distance in DTM" (Section 8.2, page 12), regex constructs like /.*/ on long lines can create overlap requirements that exceed the block size, breaking the model. The authors dismiss this by stating their benchmarks do not trigger this and suggesting a "fallback mechanism" as future work. For a core contribution, this is a significant limitation. The novelty is presented as a general model, but its applicability appears bounded in a way that is not fully explored.

2. Incremental Nature of Secondary Optimizations: While Shift Rebalancing and Zero Block Skipping are novel in their specific application, the underlying concepts are familiar. Operand rewriting for balancing expression trees is a known compiler technique, and exploiting sparsity is fundamental in many high-performance domains. The novelty here lies in the synergy with the interleaved model. However, if a different method were to solve the cross-block dependency problem, the contribution of these secondary optimizations would be diminished. Their novelty is contingent on the novelty of the DATDM approach.

3. Overlapping Concepts with Prior Art in Speculative Execution: The paper differentiates itself from Qiu et al. [59] by stating they handle data dependencies via recomputation, whereas Qiu et al. handle control dependencies via speculation on CPUs (Section 9, page 13). This distinction is valid but narrow. Both works address the challenge of breaking sequential dependencies in a parallelized stream-processing context. The choice of recomputation over speculation for GPUs is a key engineering decision, but the fundamental problem being tackled—enabling non-sequential processing of a dependent stream—shares significant conceptual overlap. The paper could do more to argue why recomputation is fundamentally a more novel or superior approach in the GPU context, beyond empirical results.

Questions to Address In Rebuttal

1. Regarding the limitations of DATDM: Can the authors provide a more concrete analysis of which specific, real-world regex constructs (beyond /.*/) would break the proposed model by requiring an unbounded or excessively large overlap distance? How would the proposed "fallback to sequential execution" interact with the rest of the fused kernel, and what would its performance implications be?

2. The paper distinguishes its dependency resolution from Qiu et al. [59]. Could the authors elaborate on whether speculation and recomputation are mutually exclusive solutions for this problem on GPUs? Could a hybrid approach exist? Please provide a more fundamental argument as to why recomputation is the superior and more novel paradigm for resolving data dependencies from SHIFT operations in a massively parallel, high-latency GPU environment compared to a speculative model.

3. The Shift Rebalancing optimization is presented as an iterative, greedy algorithm. Is there any theoretical basis to suggest this approach nears an optimal DFG balance for this problem, or is it purely a heuristic? The novelty of the engineering would be strengthened if its theoretical properties were better understood.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present SoftWalker, a framework that offloads GPU page table walks from dedicated hardware PTWs to software threads, termed "Page Walk Warps" (PW Warps), running on SMs. The central thesis is that for irregular applications, the primary bottleneck in address translation is not the latency of a single page walk, but the severe queueing delay caused by contention for a limited number of hardware PTWs. By leveraging "idle" GPU cycles to execute page walks in software, the authors claim to achieve massive parallelism, thereby eliminating queueing delay. The proposal is supported by two main architectural additions: dedicated PW Warps to isolate translation from computation, and "In-TLB MSHRs" to expand the capacity for tracking outstanding misses by repurposing L2 TLB entries. The authors claim an average speedup of 2.24x (3.94x for irregular workloads).

While the diagnosis of the problem (PTW contention) is sound, the proposed solution appears to be a costly and complex hardware/software co-design masquerading as a simple software framework. The evaluation relies on a favorable baseline and assumptions that seem to minimize the very real overheads introduced by the software-based approach, and the security implications are not sufficiently addressed.

Strengths

1. Problem Characterization: The paper provides a compelling analysis motivating the work. The identification in Section 3.2 (Page 5) that queueing delay constitutes up to 95% of the total page walk latency for irregular workloads is a strong and clear insight.
2. Empirical Motivation: The microbenchmark results from a real NVIDIA A2000 GPU (Figure 4, Page 3), which demonstrate a 4x increase in memory access latency with 256 concurrent walks, provide a solid, real-world grounding for the contention problem.
3. Clear Presentation of Core Concept: The conceptual overview in Figure 1 and the latency comparison in Figure 9 (Page 6) effectively communicate the paper's core trade-off: accepting a modest increase in per-walk latency to drastically reduce total latency by eliminating queueing.

Weaknesses

1. Mischaracterization as a "Software" Solution: The proposal is repeatedly framed as shifting work "from fixed-function hardware to software execution." This is misleading. SoftWalker requires significant and non-trivial hardware modifications.

   • Dedicated PW Warp Contexts (Section 4.2, Page 7): The paper states that SoftWalker provisions "dedicated architectural slots for the PW Warp, including an instruction buffer entry, scoreboard entries, and SIMT stack entries." This is not a "software" change; it is a modification to the core SM pipeline and resource allocation hardware. The claim of "minimal hardware overhead" is therefore unsubstantiated. The overhead analysis in Section 5.2 (Page 9) only quantifies storage (1470 bits), ignoring the cost and complexity of the required control logic modifications in the warp scheduler and pipeline.
   • Required ISA Extensions (Section 4.3, Page 7): The proposal mandates four new instructions (LDPT, FL2T, FPWC, FFB). Adding and decoding new instructions, particularly privileged ones like LDPT which bypasses the TLB, represents a fundamental change to the processor's ISA and hardware, not a simple software routine.

2. Unconvincing Security Model: The security implications of the new ISA are not rigorously handled.

   • The LDPT instruction, which loads a page table entry using a physical address and bypasses the TLB, is extremely powerful. The authors' security argument in Section 5.1 (Page 9) relies entirely on the premise that only the isolated PW Warp can execute it. However, the paper fails to describe the hardware mechanism that enforces this restriction. What prevents a malicious user-space kernel from discovering the opcode for LDPT and using it to read arbitrary physical memory? A robust security model would require hardware-level privilege checks, which are not discussed.

3. Flawed Performance Trade-offs and Evaluation: The evaluation appears to be constructed to amplify the benefits of SoftWalker while obscuring its costs.

   • The "In-TLB MSHR" is TLB Pollution: The mechanism described in Section 4.5 (Page 8) repurposes valid L2 TLB entries to store miss metadata. For any application that is not pathologically irregular—i.e., any application with some degree of spatial or temporal locality—this will lead to the eviction of useful translations, increasing the overall TLB miss rate. The authors tacitly admit this weakness by proposing a "Hybrid Approach" (Section 5.4) for regular applications, which confirms the pure software walker is detrimental in those cases. The evaluation lacks experiments on mixed regular/irregular workloads that would expose this critical flaw.
   • Unrealistic Communication Latency Modeling: The performance of SoftWalker is highly sensitive to the communication latency between the L2 TLB and the SM executing the page walk. In Section 6.1 (Page 11), the authors state this is modeled as "equal to the L2 TLB access latency." This seems optimistic. The process involves the Request Distributor, the SoftWalker Controller, and instruction fetch/execution on the SM pipeline, which likely incurs additional cycles beyond a simple L2 access. The sensitivity study in Figure 22 (Page 13) shows performance degrading with higher latency, underscoring the criticality of this assumption, which itself lacks rigorous validation.
   • Weak Baseline: The baseline architecture uses 32 hardware PTWs. While this is a plausible configuration for some GPUs, high-end designs may feature more. By comparing against a relatively constrained baseline, the severity of the queueing problem is maximized, making the gains from SoftWalker appear larger than they might be against a more robust hardware alternative.

Questions to Address In Rebuttal

1. Please provide a detailed hardware cost analysis (in terms of area and design complexity) for the "dedicated architectural slots" required for a PW Warp in each SM. How does this compare to the cost of simply adding more conventional hardware PTWs (e.g., doubling them to 64)?
2. What specific hardware mechanism prevents a user-level thread from executing the LDPT instruction? Is there a new privilege level or hardware flag, and if so, what is its cost and how is it managed?
3. The In-TLB MSHR mechanism evicts potentially useful L2 TLB entries. Please provide an evaluation of a workload that mixes frequent, localized memory accesses (which benefit from the TLB) with sporadic, irregular accesses. I expect this to demonstrate performance degradation due to TLB pollution from In-TLB MSHRs.
4. Please provide a justification for modeling the L2-to-SM request distribution and software walker initiation latency as being equal to a single L2 TLB access. A more detailed cycle breakdown of this critical path is needed to validate the performance claims.
5. How does the performance of SoftWalker compare against a baseline with 128 PTWs? Your own area analysis in Figure 15 (Page 10) suggests that 128 PTWs is a comparable design point in terms of area cost. A direct performance comparison is necessary for a fair evaluation.

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

---

Summary

This paper introduces SoftWalker, a novel framework that fundamentally shifts GPU page table walking from fixed-function hardware to a scalable, software-based execution model. The authors identify that for emerging irregular applications (e.g., graph analytics, sparse linear algebra), the primary bottleneck in address translation is not the latency of a single page walk, but the massive queueing delay caused by contention on a small, fixed number of hardware Page Table Walkers (PTWs).

The core contribution is to leverage the GPU's inherent massive thread-level parallelism and abundant stall cycles to perform these walks in software. SoftWalker dynamically dispatches specialized, lightweight "Page Walk Warps" (PW Warps) on the SMs to handle TLB misses concurrently. To support this high degree of parallelism, the paper also introduces "In-TLB MSHRs," a clever mechanism that repurposes underutilized L2 TLB entries to track outstanding misses, thereby overcoming the limited capacity of dedicated hardware MSHRs. The evaluation demonstrates significant performance improvements, particularly for irregular workloads, by effectively eliminating translation queueing delays.

---

Strengths

1. Elegant and Foundational Re-imagination of a Core Problem: The central idea of SoftWalker is exceptionally strong. It revisits the classic architectural debate between hardware and software-managed address translation, but brilliantly re-contextualizes it for the unique characteristics of a GPU. The authors correctly observe that while software page walks are too costly in traditional CPUs due to expensive traps and context switches, they are a natural fit for GPUs, where near-zero-cost warp switching is a foundational design principle. This transforms what is typically a liability for irregular workloads—long-latency memory stalls—into an opportunity to perform productive, parallel work. The conceptual diagram in Figure 1 (page 2) powerfully illustrates this conversion of stall cycles into progress.

2. Excellent Problem Analysis and Motivation: The paper does an outstanding job of motivating the work. The analysis in Section 2.2 and 3.2 is compelling. In particular, Figure 7 (page 5), which breaks down page walk latency, provides the "smoking gun" for the entire paper: queueing delay constitutes up to 95% of the total latency for irregular workloads. This clear, data-driven insight makes the authors' subsequent design choices feel not just logical, but necessary.

3. Holistic and Practical System Design: The proposal is not a single-trick pony. The authors demonstrate a deep, system-level understanding by identifying and addressing the next bottleneck. After proposing thousands of software walkers, they rightly identify that the limited number of hardware MSHRs would become the new point of contention. Their solution, In-TLB MSHRs (Section 4.5, page 8), is a clever and resource-efficient technique that again leverages an underutilized resource (the L2 TLB itself) to solve the problem. This foresight strengthens the entire proposal.

4. Awareness of Broader Architectural Context: SoftWalker fits neatly into a broader research theme of using "helper threads" or "assist warps" to accelerate system-level tasks (e.g., CABA [93] for compression). This work provides one of the most compelling use cases for this paradigm by applying it to the fundamental process of address translation. Furthermore, the inclusion of a hybrid approach (Section 5.4, page 10) demonstrates pragmatism. The authors acknowledge that their software-first approach might penalize latency-sensitive regular workloads and propose a practical path to deployment that retains existing hardware, making the idea far more compelling for real-world adoption.

---

Weaknesses

While the core idea and execution are strong, the paper could be strengthened by a deeper exploration of its broader implications.

1. Security Implications of Privileged Software Execution: Section 5.1 (page 9) provides a solid initial discussion of resource isolation. However, moving a privileged operation—one that deals directly with physical page table addresses—into a software-like execution flow on the SM represents a significant shift in the GPU's security and trust model. While direct access between warps is prevented, the discussion could benefit from considering more subtle side-channel attacks (e.g., through timing, cache, or memory controller contention) that could arise from co-locating these privileged PW Warps with untrusted user warps on the same SM resources.

2. Interaction with Future Heterogeneous Memory Systems: The paper positions itself in the context of current GPU architectures. However, the field is rapidly moving towards more complex, heterogeneous systems enabled by interconnects like CXL. In these environments, address translation may become more complex, potentially spanning multiple memory domains with different latency characteristics. The paper would be more forward-looking if it discussed how the SoftWalker model might extend to these multi-hop translation scenarios, where a single page walk could involve traversing structures across a CXL link.

3. Software Ecosystem Complexity: The proposal relies on a host driver to pre-load the PW Warp code and orchestrate its execution. While architecturally sound, this introduces new complexity into the driver and runtime stack. A brief discussion on the anticipated software engineering effort and the interface between the hardware controller and the driver would add a valuable layer of practical analysis.

---

Questions to Address In Rebuttal

1. Regarding security, can the authors elaborate on the threat model they considered? Beyond direct register/shared memory access, what is their assessment of potential timing or contention-based side channels between a privileged PW Warp and co-resident user warps? Does giving the PW Warp highest scheduling priority introduce any predictable timing patterns that could be exploited?

2. How does the SoftWalker framework adapt to future memory systems? Specifically, in a CXL-enabled system with tiered memory, a page walk might require traversing page tables located in remote, high-latency memory. How would the fixed instruction sequence of a PW Warp handle such variable and potentially very long latencies? Does this scenario weaken the benefits of the software approach?

3. The hybrid model is a key practical feature. Could the authors provide more detail on the policy within the Request Distributor that chooses between hardware PTWs and software PW Warps? For instance, how does it handle the transition when hardware PTWs become fully saturated? Is there a risk of creating bubbles or inefficiencies in the hand-off process itself?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper, "SoftWalker," proposes a framework for handling GPU page table walks in software rather than with fixed-function hardware Page Table Walkers (PTWs). The core idea is to leverage the GPU's own massive thread-level parallelism by dispatching dedicated, lightweight software threads ("Page Walk Warps" or PW Warps) on Streaming Multiprocessors (SMs) to resolve TLB misses. To support this high degree of parallelism, the authors also introduce "In-TLB MSHRs," a mechanism that repurposes underutilized L2 TLB entries to track outstanding misses when the dedicated MSHR hardware is saturated. The authors claim this software-defined approach fundamentally addresses the scalability limitations of hardware PTWs, especially for irregular applications with high TLB miss rates.

My analysis concludes that the fundamental concept of software-managed address translation is not new. However, the paper's primary novel contribution is the adaptation and architectural integration of this concept into the unique, massively parallel execution model of modern GPUs. The novelty lies not in the "what" (software page walks) but in the "how" and "where" (using the GPU's own compute fabric at scale).

Strengths

The main strength of this work is its core insight: the trade-offs that made software-managed translation unattractive for latency-sensitive CPUs are inverted in the context of a throughput-oriented GPU. The authors correctly identify (Section 3.3, page 5) that the high context-switching overhead of CPU exception handling is a key impediment, whereas GPUs are designed for near-zero-cost switching between thousands of threads. Exploiting this architectural feature to parallelize a fundamental OS-level task is a conceptually novel application.

The supporting architectural mechanisms, while drawing from existing concepts, are integrated in a novel way to solve this specific problem:

1. PW Warp Isolation: The concept of a specialized, privileged warp is a necessary and non-trivial extension of the "assist warp" pattern seen in prior work (e.g., CABA [93] for compression). Applying this pattern to a privileged and security-sensitive task like page table walking, with the required resource partitioning, represents a significant delta.
2. In-TLB MSHRs: While the paper honestly cites prior art for "In-Cache MSHRs" ([21] Farkas and Jouppi, 1994), its application to a TLB to solve the secondary bottleneck created by their own high-throughput walker is a clever and contextually novel engineering step. Without this, the primary contribution would be less effective.

Weaknesses

The primary weakness from a novelty perspective is that the paper's foundational ideas are adaptations of well-established concepts from other domains. A critical reader with deep knowledge of prior art will immediately recognize the parallels.

1. Software-Managed TLBs: The concept of a software handler for a TLB miss is decades old, dating back to architectures like MIPS and Alpha. The paper acknowledges this for CPUs in Section 3.3 (page 5) but could do more to frame its contribution as a novel re-imagining of this old idea for a new architectural paradigm, rather than presenting software page walks as a fundamentally new idea in and of itself.
2. Use of "Assist Warps": The idea of co-opting GPU execution resources for system-level or helper tasks is not entirely new. Prior work such as CABA [93] for data compression and CUDA-DMA [10] for memory copies established the pattern of warp specialization. SoftWalker's novelty is in the target application (address translation) and the required privilege level, which is a significant distinction, but the underlying mechanism of using a specialized warp is an extension of this existing pattern.
3. In-TLB MSHRs: As noted, this is a direct adaptation of the "In-Cache MSHR" concept. The novelty is in the application, not the core mechanism. The paper is transparent about this, but it must be recognized that this is an incremental, albeit important, innovation.

The complexity of the proposed ISA extensions (Section 4.3, page 7) and new hardware components (SoftWalker Controller, Request Distributor) is non-trivial. While the performance gains for irregular workloads are substantial (average 3.94x speedup), the justification for this added complexity rests entirely on the importance of this specific workload class. For regular workloads, the system introduces a performance regression, which is mitigated by a hybrid model that essentially retains the old hardware path. This suggests the novel mechanism is a specialized accelerator rather than a universal replacement, somewhat narrowing the scope of the innovation's impact.

Questions to Address In Rebuttal

1. The concept of software-managed address translation has a long history in CPU architectures. Please elaborate further on the specific architectural features of GPUs (beyond fast context switching) that make this old idea newly viable. Conversely, what undiscovered challenges arise when porting this model from a single-core, deep-pipeline CPU to a massively parallel, wide-pipeline SM?
2. Please position the "PW Warp" concept more directly against the "assist warp" pattern from prior work like CABA [93]. What are the fundamental architectural differences required to support a privileged, system-critical task like address translation compared to an application-level helper task like compression? Specifically, how is security and isolation enforced at the hardware level beyond what was proposed in previous assist warp schemes?
3. The novelty of "In-TLB MSHRs" lies in its application to a new structure. Were there any non-obvious technical challenges in adapting the In-Cache MSHR idea to a TLB? For instance, do the tag/data organization, replacement policies, or interactions with page table updates in memory introduce complexities not present in a traditional data cache?

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors propose LATPC, a hardware mechanism to accelerate GPU address translation by exploiting intra-warp VPN regularity. It comprises three main components: a "Regularity Detector" to identify strided VPN access patterns within a warp, Locality-Aware MSHR Compression (LATC) to merge multiple TLB miss requests into a single compressed MSHR entry, and Locality-Aware TLB Prefetching (LATP) to batch page table walks for PTEs residing in the same page table. The authors claim a 1.47x geometric mean speedup over a baseline system, asserting superiority over existing prefetchers and state-of-the-art speculative translation mechanisms like Avatar.

While the proposed mechanism demonstrates significant speedup on paper, the evaluation rests on several questionable assumptions, particularly regarding the implementation of prior work and the practicality of the proposed hardware. The claims of effectiveness are potentially inflated due to an uncharitable comparison against the state-of-the-art.

Strengths

1. Problem Identification: The paper correctly identifies TLB MSHR contention and page table walker (PTW) occupancy as key bottlenecks in modern GPU address translation, as substantiated by the initial analysis in Figure 1. This motivation is sound.
2. Core Insight: The observation that VPNs within a warp often exhibit strong regularity with respect to thread index rather than temporal access order (Section 4.1, Figure 7) is a valuable insight. Shifting the frame of reference for pattern detection from temporal to spatial (within the warp) is a logical approach.
3. Component-wise Analysis: The evaluation effectively isolates the performance contributions of the prefetching (LATP) and compression (LATC) components (Figure 17a), which provides clarity on the source of the claimed performance gains.

Weaknesses

1. Unconvincing Comparison to Prior Art: The evaluation of prior work, particularly Valkyrie [14], appears to be based on a configuration that does not represent its full potential. The authors' own discussion (Section 7, page 12) reveals that simply adjusting the L2 TLB MSHR configuration improves Valkyrie's GMean speedup from 1.03x to 1.20x. This is a critical admission. It suggests the main evaluation in Figure 17 significantly understates the performance of a key state-of-the-art competitor, thereby inflating the relative gains of LATPC. A fair comparison would use an optimized configuration for all evaluated schemes.

2. Overly Optimistic Hardware Cost and Complexity: The practicality and cost of the per-SM Regularity Detector are questionable. The working model (Figure 12) processes one unique VPN per cycle. For a warp with high page divergence (e.g., 32 unique pages), this implies a serialization latency of up to 32 cycles before the full access pattern is determined and can be acted upon. The assumption of a single-cycle latency for this entire detection process (Section 5.5, page 8) is not justified for this clear worst-case scenario and seems entirely unrealistic. This un-accounted for latency could erode a significant portion of the claimed performance benefits.

3. Ambiguous and Contradictory "Accuracy" Metric: The claims regarding prefetch accuracy are confusing and poorly defined. In Section 6.5 (page 11), the authors state that for several workloads, LATPC issues no incorrect prefetches, resulting in "100% accuracy," but then choose to report this as 0% "to avoid overstating the benefits." This is methodologically unsound. If no prefetches are issued, accuracy is an undefined or irrelevant metric. If prefetches are issued and all are correct, the precision is 100%. Reporting 0% is not conservative; it is incorrect and obfuscates the true behavior of the prefetcher on those workloads. The standard metrics of coverage and precision should be used clearly and consistently.

4. Limited Architectural Scope of Evaluation: The evaluation is based on a simulated configuration modeled after the NVIDIA Turing architecture (RTX 2060-like, Section 6.1, page 9). It is unclear how the findings generalize to more recent architectures like Ampere or Hopper, which feature significantly larger L2 TLBs (e.g., 4,096 entries, as mentioned in Section 6.7) and different memory subsystem characteristics. The sensitivity study in Figure 22b shows that simply increasing the baseline's L2 TLB entry count closes the performance gap. This suggests that the problem LATPC solves may be less severe on newer architectures, potentially making this complex hardware solution less impactful. The paper's conclusions about the general necessity of LATPC are therefore not fully supported.

5. Unsubstantiated Claim on Regularity in "Irregular" Workloads: The paper claims that even "irregular" workloads exhibit an "almost-strided pattern" within a warp (Section 4.1, page 4). However, the evidence provided in Figure 8 is weak. It shows that there are, on average, 2.49 unique strides. While less than 32, this is far from "almost-strided" and suggests a more complex pattern than a single stride can capture. The Regularity Detector as designed (Figure 12) appears to only capture a single dominant stride at a time, potentially leaving performance on the table for these more complex patterns.

Questions to Address In Rebuttal

1. Please provide a revised performance comparison (revising Figure 17) where Valkyrie is evaluated using the configuration that yields a 1.20x speedup, as discussed in Section 7. Justify why the original, lower-performing configuration was chosen for the primary evaluation.
2. Provide a detailed timing analysis of the Regularity Detector for a worst-case scenario (e.g., a warp with 16 or 32 unique VPNs that do not form a simple stride). How is the multi-cycle detection latency modeled in the simulator? Justify the single-cycle latency assumption used for the hardware cost analysis in Section 5.5.
3. Please clarify the prefetcher evaluation metrics. Re-plot Figure 19 using the standard definitions of prefetch coverage (prefetched misses that become hits / total misses) and prefetch precision (correct prefetches / total prefetches issued). Explain how cases with zero issued prefetches are handled.
4. Given that newer architectures like Ampere have 4,096 L2 TLB entries, and your own sensitivity study (Figure 22b) shows that LATPC with 2,048 entries only marginally outperforms a baseline with 4,096 entries, please discuss the relevance and expected performance benefit of LATPC on such modern or future GPUs.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces LATPC, a mechanism to accelerate GPU address translation by exploiting the often-regular spatial patterns of memory accesses within a single warp instruction. The authors correctly identify that frequent TLB misses and contention for limited Miss-Status Holding Register (MSHR) entries are primary bottlenecks in modern GPU virtual memory systems.

Instead of adopting traditional CPU-style temporal prefetching (predicting the next access for a thread), LATPC makes the key insight that the collection of accesses within a single warp instruction often forms a predictable, strided pattern. It proposes a holistic, two-part solution to leverage this insight: 1. Locality-Aware TLB Prefetching (LATP): A "Regularity Detector" identifies strided VPN patterns within a warp's unique memory requests. This information is used to batch page table walks, fetching multiple related Page Table Entries (PTEs) in a single, efficient operation that leverages DRAM row buffer locality. 2. Locality-Aware TLB MSHR Compression (LATC): The same regularity information is used to compress what would be multiple individual TLB miss requests into a single, compact MSHR entry, dramatically reducing contention on this critical resource.

The work is evaluated using a cycle-level simulator across 24 workloads, demonstrating a significant 1.47x geometric mean speedup over a baseline system, outperforming several existing prefetching schemes and the state-of-the-art speculative approach, Avatar.

Strengths

The paper's value lies in its elegant re-framing of the GPU address translation problem and its well-designed, synergistic solution.

1. A Fundamental and GPU-Native Insight: The primary strength is its core conceptual contribution: shifting the focus of TLB prefetching from temporal prediction (what will this thread do next?) to spatial regularity (what are this thread's neighbors doing right now?). This is a profoundly important distinction. While prior work has struggled to apply CPU prefetching concepts to the chaotic temporal access streams of GPUs (as shown well in Figures 4 and 5, page 3), this paper embraces the GPU's fundamental execution primitive—the warp—as the source of predictability. The data presented in Figure 8 (page 4), showing that even "irregular" workloads exhibit few unique VPN strides within a warp, provides strong evidence for this foundational premise.

2. Holistic and Synergistic Design: LATPC is not a single trick; it is a well-conceived, holistic solution that attacks the problem from two critical angles. LATC directly addresses the resource contention bottleneck (MSHRs), while LATP addresses the latency bottleneck (page table walks). The two components are powered by the same underlying insight and mechanism (the Regularity Detector), making the design elegant. Figure 11 (page 5) provides an excellent timeline visualization of how these two components work together to resolve queueing delays that plague the baseline system.

3. Strong Grounding in the Current Research Landscape: The authors do an excellent job of situating their work. They not only compare against a gamut of traditional prefetchers but also against Avatar [84], a very recent and philosophically different (speculative) approach. The analysis in Section 6.5 (page 11), which shows that LATPC is not only competitive with Avatar but can be combined with it for even greater gains, is particularly valuable. This demonstrates a mature understanding of the field, positioning LATPC not just as a replacement for other techniques, but as a powerful, orthogonal component in the architect's toolkit.

4. Connecting Hardware Principles: The work effectively connects two well-understood hardware principles in a novel way. It takes the concept of memory access coalescing, a cornerstone of GPU performance, and applies it to the metadata of memory access—the address translations themselves. Furthermore, by batching page table walks, it recognizes that the page table itself is just another data structure in memory and that accesses to it can benefit from DRAM locality, a concept explored in other contexts but applied very effectively here.

Weaknesses

The weaknesses are less about fatal flaws and more about the boundaries of the proposed idea and areas that could be explored more deeply.

1. Characterizing the Limits of "Regularity": While the paper demonstrates benefits even for workloads classified as "irregular," the true strength of the "Regularity Detector" seems tied to almost-strided access patterns. The paper would be strengthened by a more explicit discussion of the mechanism's behavior in the face of truly pathological, pointer-chasing workloads where intra-warp VPNs might have no discernible stride or structure. How gracefully does LATPC degrade to the baseline performance in such scenarios?

2. Interplay with the Upstream Coalescer: The Regularity Detector is situated after the TLB coalescer, which provides it with a set of unique VPNs. The paper's discussion in Section 7 (page 12) on the sorting requirement feels a bit like an addendum but is, in fact, central to the mechanism's real-world efficacy. The ability to detect a consistent stride depends heavily on the order in which the unique VPNs are processed. A deeper analysis of the sensitivity of the Regularity Detector to the output order and behavior of a realistic hardware coalescer would lend more robustness to the claims.

3. Overhead vs. Simpler Alternatives: The hardware overhead analysis in Section 5.5 (page 8) is thorough and shows the cost is modest. However, the overall design complexity (a new detector unit, modified MSHR tags, and modified PTW logic) is non-trivial. The sensitivity study in Section 6.7 (page 11) convincingly shows that LATPC can outperform a baseline with more resources. Still, a more direct comparison would be valuable: for the same transistor budget as the entire LATPC mechanism, how much would performance improve by simply building a larger, more associative L2 TLB or adding more standard PTWs? This would help contextualize LATPC within the broader space of architectural trade-offs.

Questions to Address In Rebuttal

1. Could the authors better characterize the performance of LATPC on workloads with highly unstructured, pointer-chasing memory access patterns (e.g., graph analytics on sparse, high-degree graphs), where intra-warp VPNs may have no discernible stride? At what point does the overhead of the Regularity Detector fail to provide a benefit?

2. The authors briefly mention the impact of sorted vs. unsorted VPNs from the coalescer in Section 7. Could they elaborate on the sensitivity of the Regularity Detector to the output order of the coalescer? How much of the claimed 1.47x speedup depends on the coalescer producing a thread-index-ordered list of unique VPNs, and what is the expected performance if this order is not guaranteed?

3. From a designer's perspective, is there a crossover point where simply investing the area and power budget of LATPC into more conventional resources (e.g., doubling the L2 TLB entries or adding 50% more PTWs) would yield equivalent or better performance across this set of workloads? The sensitivity study shows LATPC scales well, but a direct iso-area comparison would be insightful.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces LATPC, a hardware mechanism designed to accelerate GPU address translation. The authors' central claim of novelty rests on a specific insight: while the temporal sequence of TLB accesses in a GPU is often chaotic, there exists significant spatial regularity in the Virtual Page Numbers (VPNs) requested by threads within a single warp instruction. LATPC is a three-part system built around this insight: 1) a Regularity Detector that identifies stride-based patterns among the unique VPNs of a warp; 2) a Locality-Aware TLB MSHR Compression (LATC) scheme that represents multiple strided TLB misses in a single MSHR entry using a <Base VPN, Stride, Valid Mask> format; and 3) a Locality-Aware TLB Prefetching (LATP) mechanism that leverages this information to issue batched page table walks, exploiting DRAM row buffer locality at the final level of the page table.

My review will focus exclusively on the novelty of this core idea and its implementation.

Strengths

1. Novel Core Insight: The primary strength of this paper is its shift in analytical perspective. The vast majority of prior art in TLB prefetching, particularly work adapted from CPUs, analyzes a temporal stream of misses from a single execution context (e.g., Sequential, Stride, and Distance prefetchers, as correctly identified by the authors in Section 3.1, page 3). The authors convincingly argue and demonstrate (most effectively in Figure 7, page 4) that for GPUs, this temporal view is noisy and lacks predictable patterns. Their proposed alternative—a spatial analysis across the threads of a single warp instruction—is a genuinely novel approach for the problem of TLB prefetching. It recasts the problem from time-series prediction to spatial pattern recognition at the instruction level.

2. Novel Hardware Mapping of the Insight (LATC): The proposed modification to the L1 TLB MSHR structure is a direct and elegant implementation of the core insight. While miss coalescing is a known technique, it typically applies to multiple requests for the same resource (e.g., the same cache line). LATC, in contrast, coalesces requests for different but predictably related resources (strided VPNs). The use of a <Base VPN, Stride, Valid Mask> representation within a single MSHR entry (Section 5.3, page 7) is a novel mechanism in the context of TLB miss handling. It effectively creates a compressed representation of a "miss stream" that exists spatially within the warp, not temporally.

3. Cohesive, End-to-End System: The novelty is not confined to a single component. It flows logically from the initial observation (Section 4) to the pattern detection (Regularity Detector, Section 5.2), miss status handling (LATC, Section 5.3), and finally to the page walk optimization (LATP, Section 5.4). This tight integration, where the prefetch generation directly informs a specialized page walk batching strategy to exploit DRAM characteristics, constitutes a complete and novel system design.

Weaknesses

1. Constituent Concepts are Not Fundamentally New: While the synthesis is novel, the underlying concepts are not. Stride detection is a classical technique. The idea of representing a regular region of memory with a base and bounds/stride has appeared in various forms, such as in stream buffers and region-based prefetching for data caches. The concept of batching requests to improve memory-level parallelism is also well-established. The paper's novelty hinges entirely on the application of these concepts to the specific problem of warp-level, inter-thread TLB misses. The authors should more clearly position their work as a novel application and synthesis of existing principles rather than the invention of fundamentally new ones.

2. Simplicity of the Regularity Detector: The proposed Regularity Detector (Figure 12, page 6) only identifies a single, constant stride. While the data in Figure 8 (page 4) suggests this is often sufficient, it is a very simple pattern. This mechanism is not novel from a hardware complexity standpoint. The contribution is its purpose, not its implementation. The work could be perceived as less groundbreaking because it does not attempt to identify more complex or multiple interleaved stride patterns, which have been explored in other prefetching domains. The "delta" over a simple stride detector is zero.

3. Insufficient Differentiation from Conceptual Prior Art: The authors do an excellent job differentiating from the specific CPU/GPU prefetchers they evaluate. However, the conceptual link to stream prefetching is strong. A stream prefetcher identifies an access stream (base address + stride), reads ahead, and stores data in a buffer. LATPC identifies a VPN stream (base VPN + stride), "pre-misses" ahead, and stores the status in a compressed MSHR entry. The paper would be stronger if it acknowledged this conceptual parallel and clearly articulated the key differences in the problem constraints (e.g., handling misses vs. prefetching data, interacting with the page walk mechanism vs. the data cache).

Questions to Address In Rebuttal

1. On MSHR Compression (LATC): The core of your hardware novelty appears to be the <Base VPN, Stride, Valid Mask> MSHR entry. Can you confirm if this exact representation for compressing multiple, distinct, in-flight misses has been proposed in prior art for any miss-handling structure (e.g., data cache MSHRs, L2 TLB MSHRs, etc.)? Please clarify the precise delta between LATC and prior work on miss coalescing or MSHR compression.

2. On the Regularity Detector: The decision to detect only a single stride per pattern appears pragmatic. What is the estimated performance impact of this limitation? What percentage of warp memory instructions exhibit more complex, yet still regular, patterns (e.g., multiple interleaved strides) that your current detector cannot capture? This will help quantify the sufficiency of your proposed novel, but simple, detector.

3. On the Novelty of Batched Page Walks (LATP): The paper cites prior work on enhancing page table walkers (e.g., [60], [97], [98]). Please articulate more sharply the novelty of LATP's batching mechanism compared to these works. Is the novelty simply that the batch is generated by your prefetcher, or is the mechanism for exploiting DRAM row buffer locality during the walk itself fundamentally different from prior proposals for batching page walks?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present S-DMA, a software-hardware co-design framework intended to accelerate sparse Diffusion Models (DMs). The core contributions are threefold: 1) A "Spatiality-Aware Similarity" (SpASim) method that reduces the complexity of sparsity prediction from O(N²) to O(N) by assuming local similarity; 2) A "NAND-based Similarity" computation that approximates cosine similarity using bitwise operations on sign or most significant bits (MSBs) to reduce hardware overhead; and 3) A "Dimension-Adaptive Dataflow" designed to unify sparse convolution and GEMM operations for processing on a dedicated PE array. The authors claim significant speedup and energy efficiency improvements over a baseline GPU and state-of-the-art (SOTA) DM accelerators.

Strengths

1. Problem Identification: The paper correctly identifies a critical bottleneck in prior work: the computational overhead of the sparsity prediction step itself can negate the benefits of sparse computation (Challenge 1 and 2, Figure 2, page 3). Focusing on reducing this overhead is a valid and important research direction.

2. Comprehensive Co-design: The proposed solution is a full-stack effort, spanning from algorithmic heuristics (SpASim, NAND-similarity) to microarchitectural implementations (SP²U, reduction network). This holistic approach is commendable.

3. Operator Unification: The dimension-adaptive dataflow (Section 3.3, page 6; Figure 8, page 7) is a technically sound approach to homogenize the dataflow for different sparse operator types (convolution and GEMM), which is a non-trivial challenge in hardware design.

Weaknesses

My primary concerns with this manuscript revolve around the fragility of its core assumptions, the justification for its approximation methods, and the soundness of its experimental comparisons.

1. Unjustified Locality Heuristic (SpASim): The central premise of SpASim—that semantically similar tokens are spatially proximal—is a strong heuristic that lacks robust validation.

   • The evaluation is performed on standard datasets (COCO, GSO) which are dominated by images with clear subjects and backgrounds, naturally favoring such a locality assumption. The work presents no evidence of how SpASim performs on adversarial inputs designed to violate this assumption (e.g., complex textures, abstract patterns, or images with fine-grained, distributed details).
   • The "adaptive" selection of the window size K (Algorithm 1, page 6) is performed offline. This is a misnomer; the system is not adaptive at runtime. A single, pre-calibrated K value is used for all inference tasks, which is brittle and may perform poorly on out-of-distribution inputs.

2. Extreme and Under-analyzed Similarity Approximation: The NAND-based similarity is a radical simplification of cosine similarity.

   • For SeS, using only the sign bit discards all magnitude information. For SpS, the paper claims to use MSBs because the distribution is non-negative (Section 3.2, page 6), but provides no analysis on how many bits are used or a sensitivity analysis of quality vs. the number of bits. Is a single MSB truly sufficient to capture similarity in a "long-tail positive distribution"? This seems highly improbable and is not substantiated.
   • The hardware savings reported in Figure 7 (page 6) are against an XNOR-based design, not the full MAC and normalization pipeline of cosine similarity. This presents the savings in an overly favorable light.

3. Unsupported "Zero-Latency" Claims: The paper repeatedly makes claims of "no additional inference latency" or "fully overlapped" operation for its auxiliary hardware components.

   • For the SP²U's sorting mechanism (Section 4.2, page 7), the claim that sorting is "fully overlapped" and introduces "no additional inference latency" is unsubstantiated. A formal analysis of potential pipeline hazards or stalls is required. It is difficult to believe there are no conditions under which the main PE array would have to wait for the SP²U.
   • Similarly, the Sparsity-Aware Reduction Network (Section 4.4, page 8) claims its accumulation can be "fully overlapped with PE computation, introducing no additional inference delay." Re-accumulating partial results from different PE lines based on dynamic sparsity masks is a complex routing and synchronization problem. Without cycle-level simulation data or a detailed pipeline diagram, this claim is not credible.

4. Fundamentally Flawed Baseline Comparisons: The experimental evaluation, particularly against SOTA accelerators, is misleading.

   • As acknowledged in Section 5.3 (page 10), the chosen baselines (Cambricon-D, Ditto) primarily exploit inter-step temporal sparsity. S-DMA exploits intra-step spatial/semantic sparsity. These are orthogonal, not competing, optimization strategies. Claiming a 7.05x speedup over them is an apples-to-oranges comparison and does not represent a legitimate scientific advance over their techniques. A proper comparison would be against an accelerator designed for intra-step sparsity or a system that integrates both approaches.
   • The reported speedups over the NVIDIA A100 GPU (up to 51.11x, Figure 12, page 10) are suspiciously high. This typically indicates that the GPU baseline is not sufficiently optimized. The authors provide no details on the implementation of "GPU+Sparsity." Was this implemented using highly-optimized CUDA kernels and libraries like cuSPARSE, or a naive high-level implementation? Without these details, the GPU baseline appears to be a strawman.

Questions to Address In Rebuttal

The authors must address the following points to establish the validity of their work:

1. On SpASim's Robustness: Provide evidence that the SpASim method is robust. This should include evaluation on a dataset specifically curated to challenge the spatial locality assumption. Furthermore, justify the use of an offline-tuned, fixed K and discuss the performance degradation when an input image violates the characteristics of the tuning set.

2. On NAND-based Similarity: Please provide a detailed sensitivity analysis for the NAND-based similarity metric. Specifically for SpS, clarify precisely how many MSBs are used and show how generation quality (e.g., FID, CLIP) degrades as the number of bits is reduced. Justify why this coarse approximation does not lead to catastrophic failures in semantic understanding.

3. On Architectural Latency Claims: Substantiate the claims of "no additional latency" for the SP²U sorter and the reduction network. Provide pipeline diagrams and/or cycle-level performance data demonstrating the absence of stalls across a range of sparsity patterns, including highly irregular ones.

4. On Experimental Baselines: Justify the direct comparison of S-DMA to accelerators (Cambricon-D, Ditto) that optimize for a completely different and orthogonal type of sparsity. Acknowledge that these are not competing approaches and re-frame the results accordingly. Provide comprehensive details on the implementation and optimization level of the GPU baselines to prove they are not strawmen.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents S-DMA, a comprehensive software-hardware co-design framework for accelerating sparse diffusion models (DMs). The authors correctly identify that while sparsity offers a promising path to reduce the immense computational cost of DMs, existing methods are critically hampered by two second-order effects: 1) the significant computational overhead of predicting where sparsity exists, and 2) the hardware inefficiency of processing the diverse and irregular sparsity patterns that emerge across different operators, namely convolutions (CONV) and general matrix multiplications (GEMM).

To address this, S-DMA proposes a holistic solution. On the software side, it introduces a "Spatiality-Aware Similarity" (SpASim) method that leverages the inherent local correlations in image data to reduce the complexity of sparsity prediction from O(N²) to O(N). It further proposes a hardware-friendly, NAND-based similarity computation to replace expensive multiply-accumulate operations. On the hardware side, the work designs a dedicated accelerator featuring a novel "Dimension-Adaptive Dataflow." This key architectural contribution unifies the execution of sparse CONV and sparse GEMM operations into a single, efficient GEMM-based pipeline, overcoming a major challenge in accelerating hybrid models. The architecture is supported by a lightweight sparsity prediction unit (SP2U) and a sparsity-aware reduction network. The authors demonstrate significant speedup and energy efficiency gains over both high-end GPUs and other state-of-the-art DM accelerators.

Strengths

The true strength of this paper lies in its holistic and deeply integrated approach to a complex problem. It moves beyond simply applying known sparsity techniques and instead re-evaluates the entire sparse inference pipeline from first principles.

1. Excellent Problem Formulation: The authors' core insight is that the cost of finding sparsity can negate its benefits. By framing the problem around the three challenges in Figure 2 (page 3)—prediction complexity, prediction overhead, and low PE utilization—they provide a clear and compelling motivation for their work. This demonstrates a mature understanding of the practical barriers to deploying sparse acceleration.

2. Elegant Algorithmic and Architectural Synergy: The proposed solutions are not independent optimizations but a tightly coupled set of ideas. The SpASim algorithm is motivated by the spatial locality of the target domain (images), and the NAND-based similarity is a direct consequence of designing an algorithm with hardware implementation costs in mind. The "Dimension-Adaptive Dataflow" is the centerpiece of this synergy, providing a hardware substrate that can efficiently execute the sparse workloads created by the software-side prediction. This transformation of sparse convolution into a structured sparse GEMM (Section 3.3, page 6) is a particularly clever contribution that avoids the well-known overheads of traditional im2col-based approaches.

3. Strong Contextualization within the Field: The paper does an excellent job of placing itself within the broader landscape of AI acceleration. It correctly identifies the limitations of prior work, such as the unsuitability of sign-based similarity from ViT accelerators (like AdapTiV) for the non-negative attention maps in DMs (Section 2.2, page 4). It also distinguishes itself from other DM accelerators (like Cambricon-D and Ditto) by tackling a different and complementary form of sparsity (semantic/spatial vs. temporal/value). This demonstrates a nuanced understanding of the research frontier.

4. Significant Potential Impact: Diffusion models are a dominant workload in generative AI, and their computational demands are a major bottleneck. S-DMA provides a compelling blueprint for future specialized hardware. By making sparsity practical and efficient, this work could significantly reduce the latency and energy cost of DM inference, enabling their deployment in a wider range of applications, from on-device editing to real-time content generation. The core ideas—especially the unified dataflow—could also prove influential for accelerating other hybrid CNN-Transformer architectures.

Weaknesses

The weaknesses of the paper are primarily related to its scope and the exploration of its boundaries. The core contribution is sound, but its context could be further enriched.

1. Limited Discussion on Generalizability: The framework is highly optimized for the U-Net-based architectures common in DMs. While this specialization is a strength, the paper would benefit from a discussion on how the core concepts might generalize. For instance, could the dimension-adaptive dataflow be applied to other models that mix attention and convolution, such as vision transformers with convolutional stems or mobile-friendly hybrid networks? A brief exploration of this could broaden the paper's perceived impact.

2. Sensitivity of the SpASim Method: The performance of the SpASim method relies on the window size K, which is determined offline (Algorithm 1, page 6). The evaluation in Section 5.2 (page 9) shows this works well for the tested benchmarks, but a deeper analysis of its robustness would be welcome. How sensitive is the performance to this hyperparameter? For example, would a model trained on a different data distribution or a highly unusual user prompt require re-profiling to find a new optimal K?

3. Lack of Comparison with Structured Sparsity: The work focuses on exploiting dynamic, fine-grained sparsity. It would be valuable to briefly contrast this with structured sparsity approaches (e.g., block or vector sparsity). Structured methods are often considered easier to support in hardware and could present a different trade-off between compression rate and hardware complexity. Including this discussion would provide a more complete picture of the design space.

Questions to Address in Rebuttal

1. The core architectural contribution is the dimension-adaptive dataflow that unifies sparse CONV and GEMM. Could the authors comment on the applicability of this technique to other hybrid CNN-Transformer architectures outside the diffusion model space, such as those used in object detection or semantic segmentation?

2. The selection of the local window hyperparameter K is performed offline. How robust is a pre-selected K value to variations in input prompts or generation tasks? For instance, does an image edit focusing on a tiny detail versus a large global change affect the optimal K, and if so, how does S-DMA handle such dynamic variation?

3. The paper's evaluation focuses on its sparsity-centric approach in isolation. How do the authors envision S-DMA synergizing with orthogonal acceleration techniques for DMs, such as quantization, knowledge distillation, or the differential/temporal computing exploited by competitors like Cambricon-D? Are the performance gains expected to be additive?

4. The NAND-based similarity is a creative and effective hardware-aware approximation. Have the authors considered if this low-cost hardware primitive could be adapted for other similarity-based tasks in machine learning beyond sparsity prediction, such as in retrieval or clustering algorithms?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces S-DMA, a software-hardware co-design framework to accelerate Diffusion Models (DMs) by exploiting semantic and spatial sparsity. The authors propose three core contributions: (1) a "Spatiality-Aware Similarity" (SpASim) algorithm that reduces the complexity of sparsity prediction from O(N²) to O(N) by leveraging local similarity; (2) a "NAND-based Similarity" computation method that replaces expensive multipliers with bitwise logic for both symmetric (SeS) and non-negative (SpS) activation distributions; and (3) a "Dimension-Adaptive Dataflow" and corresponding hardware that unifies sparse convolution and sparse GEMM operations into a dense GEMM format.

While the paper presents a well-integrated and high-performing system, my analysis concludes that the foundational ideas behind each of the core contributions are not new. Instead, they represent effective adaptations and combinations of well-established principles from the fields of efficient Transformers and sparse hardware acceleration, applied specifically to the domain of Diffusion Models. The novelty is therefore in the application and system-level integration, not in the core concepts themselves.

Strengths

1. System-Level Co-Design: The work is a comprehensive example of software-hardware co-design, connecting algorithmic optimizations directly to bespoke hardware units (SP²U, reduction network).
2. Problem Formulation: The authors correctly identify and articulate the key challenges (Section 1, page 2, Figure 2) in accelerating sparse DMs: the overhead of sparsity prediction and the difficulty of handling heterogeneous sparse operators.
3. Holistic Sparsity Support: The framework's ability to handle both semantic sparsity (token merging) and spatial sparsity (image editing) within a unified architecture is a notable engineering achievement.

Weaknesses

The primary weakness of this paper, from the perspective of novelty, is that its core contributions are derivations of prior art.

1. Spatiality-Aware Similarity (SpASim) is an application of Local Attention: The central idea of SpASim (Section 3.1, page 5) is to reduce a quadratic O(N²) similarity computation to linear O(N) by restricting comparisons to local windows. This concept is the cornerstone of numerous efficient Transformer models developed over the past several years to overcome the exact same bottleneck. Architectures like the Swin Transformer (windowed attention) or methods like Longformer (sliding window attention) are built on this exact principle of exploiting locality to make attention tractable. While the authors apply this to the sparsity prediction step for DMs, the algorithmic principle of trading global comparison for local comparison to achieve linear complexity is not a novel contribution.

2. NAND-based Similarity is an incremental extension of Bitwise Similarity Proxies: The proposal to use cheap bitwise operations as a proxy for expensive cosine similarity (Section 3.2, page 6) is not new. The authors themselves reference AdapTiV [48], which uses XNOR-based sign similarity for this purpose in Vision Transformers. The authors' claim to novelty rests on adapting this for DMs, where SpS activations are non-negative, by using the Most Significant Bit (MSB) instead of the sign bit. Using MSBs as a low-cost proxy for magnitude is a standard technique in approximate computing. The move from XNOR to NAND gates (Figure 7, page 6) is a minor circuit-level optimization. Therefore, this contribution is a small, albeit clever, delta over existing work, adapting a known technique to a slightly different data distribution.

3. Dimension-Adaptive Dataflow is a form of Sparse Data Compaction: The proposed dataflow (Section 3.3, page 6, Figure 8) aims to unify sparse convolution and GEMM by transforming sparse convolution into a dense GEMM operation. This is achieved by gathering active tokens/channels and permuting them into a dense block. The concept of converting sparse operations into dense ones by gathering non-zero elements to feed a dense systolic array or PE array is a foundational technique in sparse accelerator design. While this method avoids the memory overhead of the classic im2col transformation for sparse inputs, the "gather-compute-scatter" pattern is not a new architectural paradigm. The novelty lies in the specific permutation strategy for DMs, not in the fundamental approach of data compaction for efficient hardware utilization.

Questions to Address In Rebuttal

1. Regarding SpASim: The authors must explicitly differentiate their contribution from the large body of existing work on local and windowed attention in the Transformer literature. Beyond applying a known technique to a new problem (sparsity prediction in DMs), what is the fundamental algorithmic novelty?

2. Regarding NAND-based Similarity: Can the authors argue that the extension from sign-based similarity (as in AdapTiV [48]) to a hybrid sign/MSB-based approach is a non-obvious conceptual leap? Given that using MSBs as magnitude comparators is a common heuristic, the rebuttal should clarify why this specific adaptation constitutes a significant novel contribution.

3. Regarding the Dimension-Adaptive Dataflow: Please contrast the proposed dimension permutation technique with other structured sparsity or data compaction schemes in the hardware accelerator literature. How does this approach differ fundamentally from prior "gather-compute-scatter" architectures designed to handle sparse activations? The defense of novelty should focus on the architectural concept, not just its specific tuning for DM workloads.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper proposes repurposing video codecs, specifically components of H.264/H.265, as a general-purpose method for compressing tensors in large language models (LLMs). The authors coin this method "LLM.265" and claim it is effective for weights, KV cache, activations, and gradients, for both inference and training. They leverage existing hardware video encoders/decoders (NVENC/NVDEC) on GPUs for implementation and further propose a custom, optimized "three-in-one" hardware codec design based on their findings. The central thesis is that the statistical properties of tensors are sufficiently analogous to those of natural images to make video compression techniques highly effective.

Strengths

1. Novel Application of Existing Hardware: The idea to leverage the perpetually idle NVENC/NVDEC hardware for a new, relevant workload (tensor compression) is resourceful and pragmatic from a systems perspective.
2. Ambitious Scope: The authors attempt to create a unified compression framework that applies to a wide variety of tensor types across different stages of the LLM lifecycle (inference and training). This contrasts with existing methods that are typically specialized for one tensor type (e.g., weights only).
3. Empirical Breadth: The paper presents experiments across multiple models (LLaMA-2/3, Pythia, T5, ViT), tasks (reasoning, classification, training), and tensor types, demonstrating a commendable effort to validate the proposed method's versatility.

Weaknesses

The paper's claims rest on a foundation of weak analogy, questionable experimental comparisons, and a critical omission of performance metrics that are essential to its core value proposition.

1. The Core Analogy is Flawed and Overstated: The central claim "Video Codecs are Secretly Tensor Codecs" is a dramatic overstatement. The authors' own analysis in Section 3.1 (Figure 2, p. 4) shows that Inter-Frame Motion Prediction—a critical component responsible for the high efficiency of modern video codecs—is actively harmful, increasing the required bitrate. The method's success thus relies on cherry-picking specific components (Intra-Prediction, DCT, Entropy Coding) from the video codec pipeline while discarding others. The paper should be reframed to argue that intra-frame image compression techniques are effective for tensors, which is a far less sensational but more accurate claim.

2. Critical Omission of Latency Overheads: The paper's thesis is predicated on improving efficiency by reducing data movement. However, it completely fails to report the latency of the compression (encoding) and decompression (decoding) operations. This is a fatal flaw. For communication-bound scenarios, the time saved by transferring less data can be easily nullified by the computational overhead of the codec itself. Without wall-clock time comparisons for end-to-end steps (e.g., time per training step, total inference latency), the claims of improved performance are unsubstantiated and potentially misleading.

3. Contradictory and Unconvincing Training Results: In Section 5.1 (p. 9), the authors claim their compressed training method achieves a "final validation perplexity is 36.7, which is lower than that of full-precision training." This is in direct contradiction with their own plot in Figure 9(b), where the uncompressed baseline clearly converges to a much lower perplexity of ~24. A perplexity of 36.7 is a significant degradation in model quality, not an improvement. This erroneous claim severely undermines the credibility of the training-related results.

4. Unfair Experimental Comparisons: The weight compression experiments in Section 4.1 (Figure 5, p. 6) compare LLM.265 against baselines like GPTQ and AWQ. However, the authors employ a "variable bit-width" search for their method, effectively performing a fine-grained hyperparameter optimization of bit allocation across layers. The baselines are typically evaluated at fixed, uniform bitrates. This is not an apples-to-apples comparison; LLM.265 is given an optimization advantage that the baselines are not. The performance gap may be attributable to this search strategy rather than the inherent superiority of the codec itself.

5. Insufficient Justification for Efficacy: The explanation for why the method works (Section 3.1, p. 4) relies heavily on qualitative arguments and visual inspection. The claim that the "channel-wise distribution property" creates "edges and planar blocks that are similar to real-world images" is a weak, non-rigorous analogy. A formal statistical analysis comparing the properties of tensor sub-blocks to the assumptions underpinning intra-prediction and DCT would be required to make this a convincing argument.

6. Speculative and Overreaching Hardware Proposal: The paper makes a significant leap from empirical results on existing hardware to a detailed proposal for a new "three-in-one" codec (Section 7). The evaluation of this proposed hardware is based on synthesis of open-source RTL and an analytical model, not a physical implementation. The performance and energy claims derived from this model (Figure 16, p. 13) are therefore highly speculative and should be presented with much greater caution.

Questions to Address In Rebuttal

1. Please provide end-to-end wall-clock timing results for your key experiments. Specifically:

   • For the inference scenario in Section 4.2, what is the total latency per generated token, including KV cache compression/decompression, when compared to an uncompressed baseline?
   • For the training scenarios in Section 5, what is the time-per-step (including codec latency and data transfer) for your method versus the uncompressed and baseline methods?

2. Please clarify the statement in Section 5.1 (p. 9) that a final perplexity of 36.7 is "lower than that of full-precision training." Your own Figure 9(b) clearly shows the uncompressed baseline achieves a perplexity of ~24. Is this a typo, or a misinterpretation of the results?

3. Can you justify the fairness of comparing your variable bit-width compression scheme against fixed bit-width quantization baselines in Figure 5? Please provide an ablation study where LLM.265 is constrained to a fixed bitrate across all layers to enable a more direct comparison.

4. Beyond the visual analogy presented in Section 3.1, can you provide a more rigorous, quantitative analysis demonstrating that the statistical properties of LLM tensors align with the assumptions made by the H.265 intra-prediction modes?

5. Given that a core component of video codecs (inter-frame prediction) is detrimental to tensor compression, do you agree that the paper’s primary claim should be narrowed from "video codecs" to "intra-frame image compression techniques"?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a novel and intriguing approach to tensor compression for large language models (LLMs), proposing that standard video codecs (like H.264/H.265) can serve as effective, general-purpose "tensor codecs." The core thesis is that the underlying principles of video compression—specifically transform coding (DCT), intra-frame prediction, and entropy coding—are surprisingly well-suited for compressing the various tensors encountered in LLM workloads, including weights, activations, KV caches, and gradients.

The authors introduce a framework, LLM.265, which leverages the existing, and often idle, hardware video encoders and decoders (NVENC/NVDEC) on modern GPUs. This approach is positioned as a "general-purpose" and "versatile" alternative to the current landscape of specialized, data-dependent quantization techniques. The paper provides extensive empirical evidence showing that LLM.265 achieves state-of-the-art compression rates across both inference and distributed training scenarios. Building on this insight, the authors conclude with a compelling architectural proposal for a customized, high-throughput "three-in-one" codec optimized for tensors, videos, and images, demonstrating its potential for significant area and energy savings in future accelerators.

Strengths

The primary strength of this work is its beautiful and non-obvious central idea. The connection between video signal processing and tensor compression is a fantastic example of lateral thinking that bridges two seemingly disparate domains. It shifts the conversation from designing bespoke numerical formats to repurposing a mature, highly optimized technology stack.

1. A Unifying Framework for Compression: The most significant contribution is the "general-purpose" nature of the proposed solution. The current state of LLM compression is fragmented: one uses techniques like GPTQ/AWQ for weights, different methods for KV cache, and yet another set (e.g., 1-bit Adam) for gradients during training. LLM.265 offers a single, unified mechanism for all of them, as clearly illustrated in Figure 1 (page 1). This simplification of the software and hardware stack is a massive advantage for system design and deployment.

2. Pragmatic Use of Existing Hardware: The decision to leverage the on-chip NVENC/NVDEC hardware is architecturally astute. These are powerful, fixed-function accelerators that are typically idle during the compute-heavy phases of LLM execution. Tapping into this underutilized resource provides a low-cost, immediately applicable pathway for accelerating data movement without requiring new hardware. This is a classic architectural win.

3. Versatility and Robustness: The authors convincingly argue that their method is "versatile" because it is data-independent, requiring no calibration or warm-up periods (Section 4, page 6). This is a critical advantage over methods like GPTQ that depend on a calibration set, whose quality can affect final model performance. The ability to achieve fractional bitrates is another subtle but powerful feature that distinguishes it from standard integer quantization.

4. Excellent Forward-Looking Architectural Analysis: The paper goes beyond a mere software proposal and provides a thoughtful vision for future hardware. The analysis in Section 6 (page 10), particularly the die area comparison in Figure 12, effectively argues for the cost-efficiency of integrating high-throughput tensor codecs. The proposed "Three-in-one codec" in Section 7 (page 11) is a well-reasoned design that balances the needs of multimedia and AI workloads, demonstrating a clear path from the paper's core insight to next-generation silicon. This is precisely the kind of content that elevates a paper at a top-tier architecture conference.

Weaknesses

While the core idea is compelling, the paper could be strengthened by addressing a few key points that are currently underexplored.

1. The Bottleneck of Existing Hardware: The authors rightly acknowledge that the throughput of current NVENC/DEC units (~1.1 GB/s, as stated in Section 6.1, page 10) is a limitation. However, the paper does not sufficiently contextualize how severe this bottleneck is. Modern intra-node interconnects like NVLink can exceed 900 GB/s. A 1.1 GB/s codec throughput would be a significant bottleneck for communication over NVLink, potentially negating the benefits of compression. The current implementation seems most viable for inter-node communication over slower networks like Ethernet, but this trade-off is not explicitly discussed.

2. Opacity of the Initial Quantization Step: The method requires converting FP16 tensors to 8-bit integers before they can be processed by the hardware video codec (Section 3.2, page 5). This initial quantization step is itself a form of compression. The paper lacks a clear ablation study that disentangles the effects of this initial FP16-to-INT8 conversion from the subsequent H.265 encoding. It leaves the reader wondering: how much of the final compression-accuracy trade-off is attributable to the simple 8-bit quantization, and how much additional benefit is truly provided by the more complex video codec pipeline?

3. Lack of Broader Context on General-Purpose Compression: While the work is well-positioned against domain-specific quantization methods, it would benefit from comparison with other hardware-accelerated, general-purpose lossless/lossy compressors (e.g., Zstd, Blosc, etc.). This would help to establish whether the primitives in video codecs are uniquely suited for this task, or if other compression schemes could offer similar benefits. The comparison in Section 7.1 (page 12) is excellent for the custom hardware proposal but is missing from the main evaluation of LLM.265.

Questions to Address In Rebuttal

1. Could the authors provide an ablation study that isolates the impact of the initial FP16-to-INT8 quantization? For example, what is the accuracy when compressing tensors using only 8-bit quantization versus the full LLM.265 pipeline at a comparable bitrate? This would clarify the unique contribution of the video codec algorithms.

2. Can you elaborate on the performance implications of the ~1.1 GB/s NVENC throughput? In which specific distributed training/inference scenarios (e.g., inter-node vs. intra-node communication, specific network fabrics) does the communication saving from compression outweigh the latency introduced by the codec itself?

3. The finding that "Inter-Frame Motion Prediction Does not Work" (Section 3.1, page 5) is fascinating. Could you speculate on the deeper implications of this? Does the lack of inter-layer correlation in LLM weights suggest something fundamental about the learned representations, for instance, that layers learn relatively independent features? This could be a valuable insight for the broader deep learning community.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents the core claim that standard video codecs (H.264/H.265) can be repurposed as highly effective, general-purpose tensor codecs for Large Language Models (LLMs). The authors demonstrate this through three primary contributions: (1) An empirical investigation showing that the stages of a video codec's intra-frame pipeline (specifically entropy coding, DCT, and intra-prediction) are surprisingly effective at compressing various LLM tensors. (2) A system, "LLM.265," that leverages the existing, and typically idle, hardware video encoders/decoders (NVENC/NVDEC) on commercial GPUs for this task. (3) A proposal for a future "three-in-one" hardware codec that enhances a video codec's architecture for tensor compression while retaining video capabilities. My review will assess the novelty of these claims against prior art.

Strengths

The primary strength of this paper is its significant conceptual novelty.

1. A Genuinely New Systems-Level Insight: The central idea of repurposing on-chip hardware video codecs for tensor compression is, to my knowledge, entirely new. While tensor compression itself is a crowded field (quantization, pruning, etc.), this work sidesteps the conventional software- or custom-hardware-centric approaches. The insight that a significant, specialized hardware unit on the GPU die is sitting idle during LLM workloads and can be co-opted for a critical bottleneck (data movement) is a powerful and novel systems contribution. This is not merely a new algorithm but a new paradigm for resource utilization in existing hardware.

2. Novel Explanation of Mechanism: The paper does not simply state "it works" but provides a novel analysis of why it works in Section 3.1. While the use of Discrete Cosine Transform (DCT) for compressing data with spatial locality is not new (it's the basis of JPEG), its application here is justified with a novel lens: mitigating outliers (Figure 3). This is a distinct and more modern justification than the traditional signal-processing argument of energy compaction. Furthermore, the identification that intra-frame prediction successfully captures the channel-wise structure of weight tensors (Figure 4) while inter-frame prediction fails is a new and valuable empirical finding that informs their entire approach.

3. Novel Architectural Proposal Grounded in Evidence: The proposed "three-in-one" codec in Section 7 is a novel hardware design. Its novelty does not come from inventing a new compression algorithm from scratch, but from the principled modification and augmentation of a known, highly-optimized architecture. By identifying the uselessness of inter-frame prediction for tensors, the authors propose excising it to save area and power, and then re-investing those resources to boost throughput for the shared intra-frame pipeline. This "renovate, don't rebuild" approach is a novel design philosophy in the space of hardware accelerators for compression and stands in contrast to ground-up designs.

Weaknesses

My critiques are not to claim the work is derivative, but to more precisely circumscribe the boundaries of its novelty.

1. Constituent Components are Not New: The novelty lies in the synthesis, not the ingredients. The paper should be careful not to overstate the novelty of the underlying algorithms. DCT, context-adaptive entropy coding (CABAC), and predictive coding are all decades-old pillars of compression. The paper’s contribution is the discovery that this specific combination, designed for natural images, is unexpectedly effective for the statistical distributions found in LLM tensors. The title's use of "Secretly" is rhetorical; this is an empirical discovery of an emergent property, not the uncovering of a hidden, intentional design.

2. Novelty of Hardware Proposal vs. Prior Art Could Be Sharpened: In Section 7, the paper proposes a new hardware design. While I assess this design as novel, its positioning relative to other hardware compression accelerators could be more explicit. For example, Atalanta [40] proposes hardware for CABAC-based compression. The key novelty delta here is that this paper's proposal reuses the entire video codec frontend (prediction, transform) and is presented as a multi-purpose unit, whereas Atalanta is a more focused, from-scratch tensor-only block. A more direct discussion of these differing design philosophies would better highlight the specific novelty of their approach.

Questions to Address In Rebuttal

1. The use of DCT for neural network weight compression is not, in itself, a new idea; prior work has explored compressing weights in the frequency domain. Can the authors precisely articulate what novel benefit is gained by using the entire intra-frame video pipeline (i.e., prediction -> transform -> quantization -> entropy coding) over a simpler, custom pipeline that might only use DCT and entropy coding? Is there a synergistic effect between the stages that is critical to the observed performance?

2. The "zero-cost" hardware argument for using NVENC/NVDEC is compelling. However, the data path requires converting tensors from FP16/BF16 to 8-bit integers on the CUDA cores before sending them to the hardware unit. At what point (e.g., for smaller tensors or lower communication bandwidths) does the overhead of this data conversion and the API latency negate the benefits of using the "free" hardware? A characterization of this trade-off would strengthen the novelty claim by defining its practical boundaries.

3. Regarding the proposed three-in-one codec (Section 7), the novelty stems from adapting an existing architecture. This path implies accepting certain legacy design constraints from the original video codec. What are these constraints, and how do they compare to the freedom of a "clean slate" design for a tensor-only codec? In essence, what is the fundamental trade-off between the efficiency gained from reuse and the potential performance lost from not designing a purely optimal tensor compression pipeline from first principles?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes HLX, a unified hardware accelerator designed for Hybrid Transformer-Mamba language models. The authors identify performance bottlenecks in the two primary kernels, FlashAttention-2 (FA-2) and State-Space Duality (SSD), when executed on modern GPUs. To address these, they introduce two novel fine-grained pipelined dataflows: PipeFlash, to hide non-MatMul operational latency in attention, and PipeSSD, a fused and pipelined execution for Mamba-2's core computation to reduce memory traffic and pressure. The paper presents a unified hardware architecture (URSC) to execute these dataflows and evaluates it via a cycle-level simulator against GPU (A100, H100) and TPU baselines. The authors claim significant improvements in compute utilization, kernel-level speedup, end-to-end latency, and area/power efficiency.

Strengths

1. Well-Defined Problem: The performance analysis in Section 3 is competent. The paper correctly identifies known limitations of GPUs for these workloads: inter-operation dependencies in FA-2/FA-3 and the severe memory-bound nature of SSD. The identification of excessive on-chip memory requirements (642KB for a fused SSD block) as a primary blocker for performant GPU execution is a crucial and valid insight.

2. Sound Core Concepts: The proposed dataflows, PipeFlash and PipeSSD, are logical responses to the identified problems. Employing fine-grained pipelining to hide latency (PipeFlash) and to manage intermediate data size (PipeSSD) are well-established hardware acceleration principles. The block-level fusion of SSD operations is a direct and appropriate strategy to counter its low arithmetic intensity.

3. Methodologically Sound Evaluation Framework: The use of a cycle-level simulator and comparison against strong, contemporary baselines (A100, H100) using optimized kernels (FA-2, FA-3, provided SSD) is appropriate. The scaling of the proposed architecture's specifications (HLX30/60) to match the theoretical throughput and memory bandwidth of the baselines provides a reasonable basis for comparison.

Weaknesses

1. Optimistic Pipeline Balancing Claims: The paper claims its pipeline balancing scheme is robust, maintaining high utilization with less than 2% variation across model configurations (Section 4, page 9). This is highly suspect. The proposed method of adjusting the number of processed rows works cleanly only when model dimensions (block_size, d_head, d_state) are integer multiples of one another and the pipeline depth. Real-world models often use dimensions that would break this harmony (e.g., d_head = 96, not 64 or 128). The paper provides no analysis of pipeline efficiency, stall cycles, or utilization degradation under these more realistic, non-ideal conditions. The claim of "nearly 100% compute utilization" is an idealized best-case scenario presented as a general property.

2. Understated Overhead of "Unification": The analysis in Table 3 (Section 6, page 12) claims a mere 3-4% area and power overhead for supporting both Transformer and Mamba-2 kernels compared to a specialized, single-purpose design. This figure seems implausible. The Reconfigurable Vector Processing Engine (RVPE) and Update Engine (UpE) must contain significant, distinct datapath and control logic for operations that are not shared (e.g., softmax vs. cumsum/softplus, reciprocal for attention vs. none for state updates). The cost of muxing, expanded microcode, and control flow logic to manage two fundamentally different dataflows is likely far greater than stated. The analysis lacks the necessary detail to substantiate this claim.

3. Diminishing Returns on Batching: The results in Figure 17 (page 11) reveal a critical weakness that is not sufficiently emphasized: the speedup advantage of HLX over GPUs decreases as the batch size increases. The paper attributes this to GPUs leveraging increased parallelism, but this framing downplays the issue. It indicates that HLX is primarily optimized for low-batch scenarios and its architectural advantage erodes significantly in high-throughput inference settings, which are economically critical. This is a fundamental limitation of the architecture's scalability.

4. Complete Omission of Decode-Phase Performance: The paper focuses exclusively on the prefill stage of inference. A key motivation for using Mamba-based models is their efficient O(1) state update during the auto-regressive decode phase. The paper claims HLX is "well-suited" for this (Section 4, page 9) but provides zero evidence, simulation data, or analysis. The architectural requirements for efficient single-token decoding (low latency, high occupancy with minimal work) are vastly different from those for parallel prefill. Without this analysis, the evaluation of a "Hybrid Transformer-Mamba" accelerator is critically incomplete.

5. Narrow Scope of Attention Variants: The evaluation is confined to standard multi-head attention (as implemented in FA-2/FA-3). The brief mention of applicability to GQA/MLA in Section 6 is speculative and insufficient. Variants like Grouped-Query Attention fundamentally alter the K and V tensor shapes relative to Q, which would directly impact the assumptions made in the PipeFlash datapath and the pipeline balance calculations. A claim of general applicability requires empirical validation, not a hand-waving assertion.

Questions to Address In Rebuttal

1. Provide quantitative data on the pipeline utilization of HLX when running a model with non-ideal dimensions (e.g., d_head = 96, block_size = 192, or a non-power-of-two value). What is the performance degradation compared to the idealized cases presented?

2. Present a detailed area and power breakdown of the RVPE and UpE components, clearly delineating the logic dedicated solely to attention, solely to Mamba-2, and shared between them. Justify how the total overhead for unification amounts to only 3-4%.

3. Address the performance trend with increasing batch size. Is there a batch size at which the H100 GPU baseline would match or exceed the performance of HLX for end-to-end inference? If so, what is it?

4. Provide a thorough analysis of the HLX architecture's performance during the auto-regressive decode phase for a representative sequence length. What is the single-token latency, and how does the architecture's utilization hold up in this serial, memory-latency-bound phase?

5. Demonstrate the claimed flexibility by providing performance results (speedup and utilization) for HLX running an attention layer with Grouped-Query Attention (GQA). How is the pipeline balancing in Figure 13 affected when the number of K/V heads is a fraction of the Q heads?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces HLX, a unified and pipelined hardware accelerator specifically designed for the emerging class of Hybrid Transformer-Mamba language models. The authors correctly identify a key challenge in this domain: these models exhibit heterogeneous computational patterns, with performance bottlenecks shifting between the attention kernel (FlashAttention-2) and the state-space model kernel (State-Space Duality, SSD) depending on the sequence length.

The core contribution is a holistic, algorithm-hardware co-design solution. The authors propose two novel, fine-grained pipelined dataflows: "PipeFlash" to hide non-MatMul latency in attention, and "PipeSSD" to fuse the disparate operations of the SSD algorithm, thereby increasing data reuse and reducing memory traffic. These dataflows are instantiated on a unified hardware architecture, the "Unified Reconfigurable Streamlined Core" (URSC), which is capable of efficiently executing both computational patterns. The paper presents compelling simulation results demonstrating significant improvements in compute utilization, latency, and power/area efficiency compared to high-end GPUs like the A100 and H100.

Strengths

The true strength of this paper lies in its insightful contextualization and response to a clear and present trend in large language model architecture.

1. Exceptional Timeliness and Problem Formulation: The research community is rapidly converging on hybrid architectures as a pragmatic solution for long-context modeling, blending the recall of attention with the efficiency of SSMs. This paper is not just chasing a trend; it is one of the first to deeply analyze the systems-level performance implications of this architectural synthesis and propose a dedicated hardware solution. The analysis in Section 1 and Figure 1 perfectly frames the problem of shifting bottlenecks, which is the central motivation for a unified architecture.

2. Strong Algorithm-Hardware Co-Design Philosophy: The paper's most significant contribution is not merely the hardware, but the co-designed dataflows, PipeFlash and PipeSSD. Instead of accelerating the existing kernels as-is, the authors re-architect the computation to be pipeline-friendly. PipeSSD, in particular (Section 4.1, Figure 10), is an excellent example of this. It takes the five distinct GPU kernels of the baseline SSD and fuses them into a single, streamlined, multi-stage pipeline, fundamentally changing the execution model to favor on-chip data movement and reuse. This demonstrates a deep understanding of where the true inefficiencies lie.

3. A Unified Architecture for a Hybrid Future: The design of the URSC is a direct and elegant answer to the problem statement. By creating a flexible core with a Dot-Product Engine (DPE), a Reconfigurable Vector Processing Engine (RVPE), and an Update Engine (UpE), the authors provide a substrate that can be configured to map both the PipeFlash and PipeSSD pipelines (as shown beautifully in Figure 12). This moves beyond the typical dichotomy of accelerators for either attention or SSMs (as seen in prior work like SOFA and MARCA, which they correctly position in Section 6) and instead provides a blueprint for accelerating composite models.

4. Connecting Algorithmic Theory to Hardware Reality: The work successfully bridges the gap between the theoretical properties of models and their practical performance. It correctly identifies why FA-2/FA-3 saturate in utilization (inter-operation dependency) and why SSD is memory-bound on GPUs (high intermediate data volume, low reuse across kernels). The proposed solutions directly target these identified root causes.

Weaknesses

The weaknesses of the paper are primarily related to the scope of its evaluation and its forward-looking positioning, rather than fundamental flaws in the core idea.

1. Prefill-Centric Evaluation: The entire evaluation focuses on the "prefill" or "encoding" phase, where a long context is processed in parallel. While this is a critical part of long-context inference, the paper completely omits an analysis of the autoregressive "decode" phase, where tokens are generated one at a time. This phase is notoriously memory-bandwidth bound and has very different performance characteristics. A truly "unified" solution for inference must be efficient in both regimes. The authors claim in Section 5.1 (page 9) that HLX is "well-suited" for this, but without data, this remains an unsubstantiated claim.

2. The Moving Target of GPU Architectures: The comparisons to A100 and H100 are fair contemporary baselines. However, GPU architectures are not static. The very limitations HLX exploits (e.g., rigid SIMT execution for heterogeneous warps, coarse-grained memory movers) are areas of active research and development by GPU vendors. The paper would be strengthened by a discussion of how its architectural advantages would hold up against a future GPU that might incorporate more flexible pipeline support or more powerful asynchronous execution primitives.

3. Generalizability to Future Model Variants: The authors briefly touch upon the applicability of PipeFlash to variants like GQA and MLA (Section 6, page 13), arguing that the core computation remains the same. While this is likely true, this work opens up the question of how such a pipelined architecture would handle more radically different future models, such as those with highly dynamic data-dependent routing (e.g., Mixture-of-Experts) or fine-grained sparsity. The reconfigurable nature of the RVPE suggests potential, but this is an unexplored frontier.

Questions to Address In Rebuttal

1. The evaluation is centered on the prefill phase. Could the authors provide some analysis, even if qualitative or theoretical, on how the HLX architecture and its fine-grained pipeline would perform during the memory-bandwidth-bound autoregressive decoding phase? How would pipeline stalls be managed when processing a single token, and what would the expected utilization be?

2. The paper makes a compelling case against current GPU architectures. However, what are the fundamental architectural advantages of the proposed URSC that could not be reasonably integrated into a next-generation GPU? In other words, is the proposed execution model a specialized one-off, or does it offer general principles that could inform the evolution of commercial accelerators?

3. Could the authors elaborate on how the proposed PipeSSD dataflow, which is tailored for Mamba-2's SSD, would need to be adapted to handle other structured SSM variants, such as the original Mamba or newer models like Zamba2 (Ref [15, 16]) which may have different internal operations or data dependencies? This would help clarify the robustness of the proposed architectural template.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present HLX, a unified, pipelined hardware accelerator designed for Hybrid Transformer-Mamba language models. The central thesis is that the heterogeneous computational patterns of the two core kernels—FlashAttention-2 (FA-2) for the Transformer portion and State-Space Duality (SSD) for the Mamba-2 portion—create shifting bottlenecks that limit performance on general-purpose hardware like GPUs.

The paper’s novel claims are encapsulated in two proposed dataflows and one unified architecture: 1. PipeFlash: A fine-grained, row-level pipelined dataflow for attention computations, designed to hide the latency of non-matrix multiplication (non-MatMul) operations by mitigating inter-operation dependencies present in block-level approaches like FA-2. 2. PipeSSD: A novel dataflow for Mamba-2’s SSD kernel that first fuses the distinct computational steps (chunk cumsum, chunk state, etc.) into a single conceptual block-level operation and then applies a fine-grained, dependency-aware pipeline to this fused kernel. 3. A Unified Hardware Architecture (URSC): A specialized hardware core designed explicitly to execute both PipeFlash and PipeSSD efficiently, bypassing the limitations the authors identify in GPU SIMT execution models for this type of heterogeneous pipelining.

Strengths

From a novelty perspective, the paper’s primary strengths are:

1. A Novel Strategy for the SSD Kernel: The most significant novel contribution is the approach to accelerating the SSD kernel. While operator fusion is a known technique (e.g., FlashAttention), its application to the five distinct and memory-intensive kernels of SSD (as shown in Figure 5, page 4) appears to be new. The subsequent proposal of a fine-grained pipeline (PipeSSD) to manage the complex row-wise and column-wise dependencies within this newly fused kernel is a non-trivial and novel contribution. The analysis in Section 3.2 (page 6) correctly identifies that naively fusing SSD on a GPU fails due to on-chip memory constraints, which provides a strong motivation for their novel hardware/software co-design approach.

2. Architectural Specialization for Fine-Grained Pipelining: The closest prior art for pipelining attention is FlashAttention-3 [47], which introduces a 2-stage asynchronous pipeline using warp specialization on NVIDIA's Hopper architecture. PipeFlash differentiates itself by proposing a much finer granularity (row-level, as shown in Figure 9, page 6) and a multi-stage pipeline implemented on a specialized, non-SIMT architecture (the URSC). This represents a novel architectural path, departing from the "make the GPU do it" approach and instead arguing for specialized hardware to overcome fundamental GPU limitations (cited in Section 3.3, page 6) for this workload.

3. The Unified Nature of the Accelerator: The current landscape of accelerators for large language models has bifurcated, with works focusing on Attention (e.g., SOFA [52]) or SSMs (e.g., MARCA [26], VGA [25]) separately. The proposal of a single, unified architecture that natively supports both computational patterns of an emerging and important class of hybrid models is a timely and novel contribution. The overhead analysis in Table 3 (page 12) suggests this unification is achieved with high efficiency, which strengthens the novelty claim.

Weaknesses

My concerns are focused on precisely delineating the boundaries of the novelty against existing concepts:

1. Conceptual Proximity of PipeFlash to FA-3: The conceptual foundation of PipeFlash—overlapping non-MatMul computation with MatMul computation in attention—is identical to that of FlashAttention-3. FA-3 uses a producer-consumer model with specialized warps. PipeFlash uses a multi-stage pipeline on a specialized datapath. While the implementations are worlds apart (GPU vs. ASIC), the core algorithmic insight is the same. The paper’s novelty here is less about a new pipelining idea and more about a new, specialized hardware implementation of that idea. This distinction should be made clearer.

2. "Fusion" as an Application of a Known Principle: The claim that "no research has yet fused SSD" (Section 2, page 2) may be accurate for published literature, but the principle of fusing memory-bound operators to improve arithmetic intensity is a cornerstone of performance engineering. The novelty lies not in the idea of fusion itself, but in the specific method for managing the complex dependencies of the SSD algorithm post-fusion. The paper's contribution is the design of the PipeSSD dataflow that makes fusion practical, not the abstract idea of fusion.

3. Under-explored Comparison to Reconfigurable Architectures: The URSC is described as a "unified reconfigurable streamlined core." The field of reconfigurable computing has a long history. While the paper compares HLX to GPUs and other LLM accelerators, it does not situate its reconfigurable datapath (RVPU in Figure 11c, page 8) within the context of prior work on reconfigurable dataflow architectures. It is unclear if the reconfigurability itself contains novel mechanisms or if it simply uses standard techniques to switch between the dataflows required for PipeFlash and PipeSSD.

Questions to Address In Rebuttal

1. Please elaborate on the fundamental novelty of the PipeFlash dataflow when compared to the asynchronous pipelining in FlashAttention-3. Is the contribution a new pipelining concept for attention, or is it a more efficient hardware implementation of the known producer-consumer pipelining principle, made possible only by a specialized, non-SIMT architecture?

2. Regarding the fusion of SSD kernels: Is the novelty in the idea of fusing these kernels, or is it in the specific dependency analysis and pipeline design (PipeSSD) that overcomes the on-chip memory barriers that prevent this "obvious" optimization from working on GPUs? Clarifying this would sharpen the paper's claimed contribution.

3. Could the authors contrast the architectural novelty of the URSC with prior work in reconfigurable dataflow computing? Specifically, what is the key architectural innovation within the RVPU's "local NoC" and associated units that enables it to efficiently handle the distinct requirements of both the softmax-centric PipeFlash and the cumsum-centric PipeSSD with minimal overhead, beyond simply instantiating the necessary functional units?

Review Form:

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors propose ORCHES, a heterogeneous GPU-PIM system designed to accelerate Test-Time-Compute (TTC) based LLM reasoning on edge devices. The paper identifies three primary challenges in TTC workloads: (1) variable parallelism complicating scheduling, (2) inter-step dependencies hindering pipelining, and (3) memory fragmentation from branch pruning. To address these, ORCHES introduces three corresponding techniques: (1) an adaptive workload assignment strategy, (2) a branch prediction mechanism to enable speculative pipelining, and (3) a memory management scheme to mitigate fragmentation. The system is evaluated via simulation, and the authors claim significant speedups (4.16× for text, 3.10× for vision) over a baseline GPU implementation.

Strengths

1. Problem Formulation: The paper provides a clear and structured breakdown of the unique computational challenges posed by TTC-based reasoning pipelines (Section 3, page 4). The identification of variable parallelism, branch dependencies, and memory fragmentation as key barriers is logical and well-articulated.

2. Comprehensive Solution: The proposed ORCHES framework is comprehensive, with each of its three core techniques (T1, T2, T3) directly targeting one of the identified challenges. This demonstrates a thorough approach to system design.

3. Detailed Mechanisms: The paper details the mechanisms for its proposed techniques, including the analytical models for workload partitioning (Section 4.2, page 6) and the history alignment strategy for the candidate predictor (Section 4.3.1, page 8).

Weaknesses

My primary concerns with this manuscript center on the evaluation methodology, the lack of crucial performance-cost analysis for the proposed techniques, and the potential for an overstated problem definition.

1. Questionable Evaluation Baseline and Methodology:

   • The performance claims are based on a simulation framework extended from AttAcc [25]. While leveraging existing simulators is standard practice, the complexity of the proposed scheduling and memory management in ORCHES raises concerns about the fidelity of a simulation. Real-world overheads from the OS, memory controller contention, and interconnect latency in a tightly-coupled heterogeneous system are notoriously difficult to model accurately.
   • The comparison against AttAcc [25] and Duplex [40] in Figure 11 is fundamentally flawed. These systems are designed for general LLM inference, not the highly specialized, multi-step, branch-intensive workloads of TTC. Comparing a purpose-built system (ORCHES) to systems not designed for the target workload inflates the perceived benefits. The most critical baseline—a highly optimized software-only implementation of the same TTC reasoning pipeline on the baseline GPU (NVIDIA AGX Orin)—appears to be missing. Without this, it's impossible to discern how much of the speedup comes from the novel hardware and how much could be achieved through superior software scheduling on existing hardware.

2. Unquantified Misprediction Penalty:

   • Technique 2 relies on a candidate verification predictor to enable speculative execution. Table 4 shows the predictor achieves ~78% accuracy after applying the "history alignment" mechanism. While an improvement, a 22% misprediction rate is substantial. The paper states that on a misprediction, the system must "roll back to the correctly selected candidate and regenerate the output" (Section 4.3.1, page 8). However, the latency cost of this rollback and regeneration process is never quantified. Without knowing the misprediction penalty, the entire benefit of the pipelining technique is unsubstantiated. A high penalty could easily negate the gains from the 78% of correct predictions. The case studies in Figure 13 show only the ideal scenario and are not sufficient evidence.

3. Under-substantiated Overhead Claims for Memory Management:

   • Technique 3 introduces a complex memory management system involving an address cache, dynamic reorganization, and a controller-side buffer. The authors claim in Section 5.5 (page 12) that the average runtime overhead of this reorganization is "only 0.12%, which is negligible in practice." This figure seems extraordinarily low for a process that involves tracking fragmentation and physically moving KV cache segments in memory. The paper provides no breakdown of how this 0.12% was calculated, what operations it includes (e.g., data movement, metadata updates), or how frequently the reorganization is triggered. This claim lacks credibility without a detailed analysis.

4. Problem Framing May Be an Artifact of a Specific Setup:

   • Challenge 2, "Branch Dependencies Hinder Pipeline Execution," is primarily motivated by the scenario where the Process Reward Model (PRM) is significantly larger than the policy model (e.g., an 8B PRM verifying a 1B policy model, as shown in Figure 6, page 5). While this may be a valid configuration from a specific paper [18], it is not a fundamental, immutable property of TTC. The performance bottleneck is a direct consequence of an algorithmic choice. The paper frames this as a general hardware challenge, but one could just as easily argue it is a software problem that could be mitigated by using more balanced model sizes. The generalizability of this "challenge" is therefore questionable.

Questions to Address In Rebuttal

The authors must address the following points to establish the technical soundness of their work:

1. Misprediction Penalty: What is the precise latency cost, in cycles or milliseconds, of a single branch misprediction event in your proposed system? Please provide a detailed breakdown of the rollback and regeneration overhead. How does this penalty affect the overall speedup when factoring in the ~22% misprediction rate?

2. Memory Reorganization Overhead: Provide a detailed breakdown of the 0.12% runtime overhead claimed for Technique 3. This breakdown should include the cost of data movement (reads and writes), metadata management for the address cache, and the computational cost of the reorganization logic itself. How was this measured in the simulator?

3. Baseline Justification: Please justify the choice of AttAcc and Duplex as primary comparison points, given they are not optimized for TTC workloads. More importantly, provide performance data comparing ORCHES against a state-of-the-art, software-only TTC implementation (using, for example, optimized kernels, batching, and scheduling) running on the standalone baseline AGX Orin GPU.

4. Analytical Model Fidelity: The offline and online scheduling strategies (Technique 1) depend on an analytical performance model (Equations 1-7). What evidence is there that this simplified model accurately predicts the performance of complex operators on real heterogeneous hardware, accounting for factors like cache contention and memory interference?

5. Sensitivity to Model Configuration: How do the reported speedups change if the policy model and PRM are similarly sized (e.g., 3B policy and 3B PRM)? Does "Challenge 2" cease to be a significant bottleneck, and if so, how does that impact the contribution of Technique 2 to the overall performance?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents ORCHES, a heterogeneous GPU and Processing-in-Memory (PIM) system co-designed to accelerate a specific and increasingly important class of workloads: multi-step, Test-Time-Compute (TTC) based LLM reasoning. The authors' core contribution is not merely the application of PIM to LLMs, but the insightful identification of a new set of system-level challenges unique to these reasoning pipelines. They astutely observe that TTC workloads are fundamentally different from standard, single-step LLM inference.

The authors categorize these new challenges into three key barriers: 1. Variable Parallelism (C1): The workload dynamically shifts between compute-bound (e.g., verification/prefilling) and memory-bound (e.g., policy model decoding), complicating static scheduling. 2. Branch Dependencies (C2): The sequential nature of the reasoning steps (generation followed by verification) creates pipeline stalls that hinder throughput. 3. Memory Fragmentation (C3): The pruning of unsuccessful reasoning "branches" leads to sparse and fragmented memory, which degrades the performance of memory-sensitive architectures like PIM.

In response, ORCHES proposes a tightly integrated set of three corresponding techniques: adaptive workload assignment (T1), speculative branch-aware pipelining (T2), and fragmentation-aware memory structuring (T3). The work positions itself as a forward-looking solution for enabling complex AI reasoning on resource-constrained edge devices, demonstrating significant speedups over state-of-the-art baselines in simulation.

Strengths

1. Excellent Problem Formulation and Contextualization: The primary strength of this paper lies in its clear and compelling problem definition. The authors do an exceptional job of explaining why existing LLM acceleration techniques, which are optimized for monolithic inference, are insufficient for the emerging paradigm of TTC. The breakdown of the problem into the three challenges (C1, C2, C3) in Section 3 (pages 4-5) is insightful and provides a strong foundation for the proposed solutions. This work successfully frames TTC not just as another LLM task, but as a distinct algorithmic workload with unique system-level implications.

2. Elegant Synthesis of Cross-Disciplinary Concepts: The proposed solutions are a thoughtful synthesis of ideas from different domains of computer science. Technique 2 (Branch Prediction Facilitating Pipelining, Section 4.3, page 8) is a particularly clever adaptation of a cornerstone concept from classical CPU architecture—speculative execution—to hide the latency of inter-step dependencies in a reasoning pipeline. Similarly, Technique 3 (Memory Structuring, Section 4.4, page 9) draws parallels to memory management and garbage collection strategies from operating systems. This cross-pollination demonstrates a deep understanding of systems design and elevates the work beyond a simple application-specific accelerator.

3. A Forward-Looking Perspective on AI Systems: This research is timely and significant. The field of AI is rapidly moving beyond simple text generation towards more complex, multi-step reasoning, as seen in agentic systems, Chain-of-Thought, and Tree-of-Thoughts. This paper is one of the first to tackle the systems-level challenges of these algorithms head-on. By treating the entire reasoning process as the target for optimization, ORCHES provides a blueprint for a new class of "reasoning accelerators." Its focus on enabling compact models to achieve the performance of much larger ones through efficient computation is a critical direction for deploying advanced AI on the edge.

4. Systematic and Coherent Solution: The one-to-one mapping of the proposed techniques (T1, T2, T3) to the identified challenges (C1, C2, C3) results in a very coherent and compelling narrative. The system feels thoughtfully architected rather than being a collection of disparate optimizations. The ablation studies presented in the evaluation (e.g., Figure 12, page 11) effectively demonstrate that each component contributes meaningfully to the overall performance, reinforcing the validity of the initial problem analysis.

Weaknesses

While the core ideas are strong, the paper could be strengthened by addressing the following points, which are more about depth and potential limitations than fundamental flaws.

1. Reliance on Simulation: The evaluation is conducted entirely within a simulated environment. While this is standard practice for novel architecture proposals, the significance of the results hinges on the fidelity of the underlying performance and power models, especially in a complex heterogeneous system. A deeper discussion on the calibration of the simulator against real hardware (beyond referencing prior work) would build more confidence in the reported speedup and energy figures.

2. Scope of TTC Generalizability: The paper focuses on a specific TTC structure involving a policy model and a process reward model (PRM). However, the landscape of reasoning algorithms is evolving. It is unclear how well the ORCHES design principles would map to other structures, such as Monte Carlo Tree Search (MCTS) in AlphaCode-style generation, or agentic workflows that involve external tool use and dynamically change the nature of the computation at each step. The current design is tightly coupled to the generate-verify loop.

3. Overhead Analysis of Memory Management: Technique 3 is presented as a highly effective solution to memory fragmentation, with the authors stating the average runtime overhead is a "negligible" 0.12% (Section 5.5, page 12). This figure seems exceptionally low for a process that involves tracking, buffering, and reorganizing memory. A more detailed cost-benefit analysis is warranted. For example, what is the latency of the reorganization process itself, and how does its trigger policy (e.g., after 3-5 steps) impact performance under different reasoning depths and branch widths? There might be corner cases where this overhead becomes more significant.

Questions to Address In Rebuttal

1. The proposed system is expertly tailored to the generate-verify structure of the evaluated TTC pipelines. Could the authors comment on the applicability of the ORCHES framework to other multi-step reasoning paradigms like Tree-of-Thoughts (ToT), which involves more complex state evaluation and backtracking, or agentic systems that might call external APIs, introducing unpredictable latency? Does the core principle of separating and speculating on distinct computational steps still hold?

2. Regarding Technique 2 (Branch Prediction), Table 4 (page 11) shows that the history alignment mechanism significantly improves prediction accuracy. Could you provide more insight into the performance trade-offs? Specifically, what is the misprediction penalty in terms of latency or wasted work, and how does this penalty interact with the predictor's accuracy to determine the overall speedup from speculation?

3. Could the authors provide a more detailed breakdown of the 0.12% runtime overhead claimed for Technique 3? Specifically, what is the latency of a single memory reorganization operation, and what is the typical frequency of this operation in your benchmarks? Understanding these two factors would help clarify how the overhead remains so low across different workloads.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces ORCHES, a heterogeneous GPU-PIM system designed to accelerate Test-Time-Compute (TTC) based Large Language Model (LLM) reasoning. The authors first identify a set of challenges unique to TTC workloads that are not present in standard single-step LLM inference: C1) variable parallelism complicating scheduling, C2) inter-step branch dependencies hindering pipelining, and C3) branch pruning inducing memory fragmentation. To address these, the authors propose a system integrating three primary techniques: T1) adaptive workload assignment between GPU and PIM, T2) a speculative, branch-aware pipelining mechanism, and T3) a fragmentation-aware memory structuring scheme.

My analysis concludes that while the system-level integration and the specific application to the TTC reasoning problem are well-executed, the novelty of the core underlying techniques is limited. Many of the proposed solutions are adaptations of well-established concepts from heterogeneous computing, speculative execution, and memory management. The primary contribution of this work is therefore not the invention of new primitives, but rather the insightful characterization of the TTC workload and the synthesis of existing ideas into a cohesive system to solve that specific problem.

Strengths

1. Novel Problem Characterization: The paper's most significant novel contribution is its in-depth analysis and characterization of the TTC-based LLM reasoning workload in Section 3 (Pages 4-5). The identification of dynamically evolving compute patterns due to the changing ratio of shared-to-unique KV caches (Section 3.1.2) is a sharp and valuable insight that clearly distinguishes this workload from standard LLM serving. This analysis provides a strong motivation for a specialized solution.

2. System-Level Synthesis: The authors have assembled a coherent system by integrating techniques from different domains. The novelty lies in this synthesis—recognizing that a combination of adaptive scheduling, speculation, and custom memory management is required to holistically address the TTC problem on a GPU-PIM architecture.

3. Refinement of an Existing Idea: Within the broader "branch prediction" technique (T2), the proposed "history alignment" strategy (Section 4.3.1, Figure 9c, Page 8) is a clever and potentially novel refinement. Using the more accurate historical scores from the large verification model to condition the lightweight prediction model is a non-obvious mechanism to improve the accuracy of a speculative process.

Weaknesses

My primary concerns relate to the novelty of the core technical contributions when evaluated individually against prior art.

1. T1: Adaptive Assignment is a Known Concept: The core idea of partitioning workloads between heterogeneous processors (GPU and a co-processor like PIM) based on their arithmetic intensity (Figure 4, Page 5) is a foundational principle of heterogeneous computing. This methodology has been explored for decades in the context of CPU-GPU systems. While the online compensation model (Section 4.2.2) adds a dynamic element, the fundamental approach of mapping compute-bound kernels to the GPU and memory-bound kernels to a memory-centric accelerator is not new.

2. T2: "Branch Prediction" is Conceptually Indistinguishable from Speculative Decoding: The proposed "branch prediction" mechanism (Section 4.3, Page 8) is a direct application of the "draft-then-verify" paradigm, which is the cornerstone of speculative decoding in LLMs. The body of work on speculative decoding is extensive (e.g., Chen et al., 2023, "Accelerating large language model decoding with speculative sampling"; Leviathan et al., 2023, "Fast inference from transformers via speculative decoding"). The authors' mechanism uses a smaller model (a subset of the PRM layers) to "draft" a likely path, which is then "verified" by the larger model. This is functionally identical to speculative decoding, merely applied to reasoning branches instead of token sequences. The paper acknowledges this in Related Work (Section 6, Page 12) but does not sufficiently differentiate its core mechanism as a novel contribution. The renaming of the technique to "branch prediction" does not create novelty.

3. T3: Memory Structuring Leverages Standard Techniques: The techniques proposed for memory management (Section 4.4, Page 9) are a combination of well-known solutions.

   • Memory Compaction: The process of reorganizing memory to eliminate fragmentation ("holes") is a classic technique used in garbage collectors and memory management units for decades.
   • Caching and Buffering: The use of an address cache and a controller-side buffer are standard architectural optimizations to reduce latency and manage data movement.
   • Overlap with PageAttention: The problem of managing a dynamic and sparse KV-cache has been famously addressed by PageAttention (Kwon et al., 2023). PageAttention uses a virtual-to-physical mapping akin to OS page tables to handle non-contiguous memory blocks. ORCHES instead appears to enforce contiguity via periodic reorganization. While the implementation differs, the high-level problem it solves is not new, and the paper should provide a more direct and rigorous comparison to this state-of-the-art baseline. The claim that T3 "achieves both the elimination of memory waste and the contiguous storage" (Section 6, Page 13) is the key delta, but it comes at the cost of reorganization overhead, a trade-off that is not fully explored against prior art.

Questions to Address In Rebuttal

1. Regarding T2 (Branch Prediction): Please articulate the fundamental conceptual novelty of your proposed "branch prediction" mechanism (Section 4.3) compared to the existing body of work on speculative decoding. Beyond the application context (reasoning steps vs. output tokens), what makes the core "draft-then-verify" process presented here different and novel? Is the "history alignment" technique the sole point of novelty in this contribution?

2. Regarding T3 (Memory Management): Please provide a more detailed comparison of your memory reorganization approach (Section 4.4) with PageAttention. Specifically, can you quantify the performance trade-offs between your approach (which incurs runtime overhead for compaction to maintain contiguity) and the PageAttention approach (which avoids compaction overhead but may incur latency penalties from non-contiguous memory access patterns)? Why is your chosen approach superior for a collaborative GPU-PIM system?

3. Regarding Complexity vs. Benefit: The proposed system introduces significant complexity with three distinct optimization techniques running concurrently. The online scheduling compensation in T1, for example, relies on an analytical model that may have its own inaccuracies. Could the authors demonstrate that this combined complexity provides a benefit that is substantially greater than applying just one or two of the more novel refinements (e.g., only the history-aligned speculation)? Is it possible that a simpler, static partitioning scheme combined with your memory manager would yield a large fraction of the benefits with much lower complexity?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose LoopFrog, a hardware/software co-design for speculative in-core loop parallelization on wide out-of-order processors. The scheme uses compiler-inserted hint instructions (detach, reattach, sync) to delineate parallelizable loop regions, which are then executed on lightweight, OS-transparent "threadlets." Data versioning and conflict detection are managed by a new microarchitectural unit, the Speculative State Buffer (SSB), and a conflict detector. The authors claim a geometric mean speedup of 9.5% on SPEC CPU 2017 over a strong 8-wide baseline, with what they characterize as "modest" overheads.

However, the work's central claims are predicated on several critical idealizations in the experimental methodology, most notably a perfect conflict detection mechanism and an oracle-based loop selection strategy. These assumptions sidestep the most challenging practical aspects of thread-level speculation, calling into question the validity and achievability of the reported results.

Strengths

1. Strong Baseline: The evaluation is performed against a convincing, aggressive 8-wide out-of-order CPU model (Table 1, page 9). This provides a challenging baseline and ensures that the reported speedups are not merely due to deficiencies in the baseline architecture.
2. ISA Design: The hint-based ISA extension is a reasonable approach. It offloads the difficult problem of dynamic task detection to the compiler, which is the appropriate place for it, while maintaining backward compatibility.
3. Detailed Analysis: The paper provides a breakdown of performance gains into sub-categories (Table 2, page 11), attempting to explain the sources of speedup. This analysis, particularly the identification of prefetching effects, is insightful, even if it simultaneously weakens the paper's main thesis.

Weaknesses

1. Idealized Conflict Detection: The single greatest flaw in this study is the idealization of the conflict detector. The authors state in Section 6.1 (page 9) that their simulation models "No false positives" for the conflict check, which they propose to implement with Bloom filters. They dismiss the impact of false aliasing as a "second-order effect" (page 8). This is fundamentally incorrect. A single false positive in a conflict detector can cause the erroneous squash of an entire epoch of potentially useful work. The resulting performance degradation is a first-order effect. Without modeling a realistic conflict detector with a non-zero false-positive rate, the performance results are unreliable at best.
2. Unrealistic Compilation and Loop Selection: The compiler's role is severely overstated. The study relies on manual #pragma annotations to simulate "perfect static loop selection" (Section 5.1, page 8). This completely avoids the notoriously difficult problem of identifying profitable loops automatically. Furthermore, the hint insertion pass is naive; as stated in Section 5.3 (page 8), it "does not consider through-memory LCDs." This limitation excludes a vast and important class of loops, meaning the technique is only applicable to loops that were largely parallel to begin with. The study is therefore evaluating a best-case scenario that is unlikely to be realized by a real-world compiler.
3. Insufficient Coherence and Multi-Core Analysis: The mechanism for preserving the memory model described in Section 4.1.4 (page 6) is hand-waved. The SSB is said to "send coherence messages" to acquire lines and is squashed if another core requests a line in an incompatible state. The entire evaluation is performed in a single-core context. This is a critical omission. In any realistic multi-threaded application running on a multi-core system, coherence traffic is constant. The rate of squashes due to external invalidations could easily overwhelm any benefit from speculation. The lack of any multi-core evaluation renders the memory model claims unsubstantiated.
4. Conflation of Parallelism with Prefetching: The analysis in Section 6.4.2 (page 11) reveals that a significant portion of the performance gain (35% of the total, combining "Branch conditions" and "Data values") comes from the prefetching side-effects of failed speculation. This raises a critical question: is LoopFrog an effective parallelization technique, or is it an exceedingly complex and expensive hardware prefetcher? A rigorous study would compare these gains against a state-of-the-art stride or stream prefetcher, which could potentially achieve similar benefits with a fraction of the complexity.

Questions to Address In Rebuttal

1. Please provide a sensitivity study showing the impact of a realistic Bloom filter-based conflict detector on the geometric mean speedup. What is the performance degradation at plausible false-positive rates (e.g., 0.1%, 1%)? Justify the claim that this is a "second-order effect."
2. The compiler ignores memory-carried dependencies (Section 5.3). What percentage of total execution time in the SPEC benchmarks is spent in loops that are disqualified by this constraint? How does this limitation affect the overall applicability of your technique?
3. How does the LoopFrog mechanism behave in a multi-core context running a workload with true sharing (e.g., a parallel benchmark like PARSEC)? Specifically, what is the frequency of speculation squashes caused by coherence requests from other cores, and what is the resulting performance impact?
4. Please provide a more direct comparison to justify the complexity of LoopFrog. How does the 35% of your speedup attributed to prefetching effects compare to the gains from enabling or enhancing a state-of-the-art hardware prefetcher in your baseline system?
5. The area overhead of 12-17% relative to a non-SMT core (Section 6.8) is substantial. The cost analysis via CACTI appears to neglect the control logic complexity for the SSB, the multi-versioned read logic, and the integration with the core's coherence protocol. Can you provide a more comprehensive estimate of these logic overheads?

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents LoopFrog, an in-core, hint-based microarchitectural technique for speculative loop parallelization. The work is motivated by the diminishing returns and resource underutilization seen in modern, wide out-of-order processors. The core idea is to revive Thread-Level Speculation (TLS) in a modern context by using compiler-inserted hints to spawn lightweight, OS-transparent "threadlets" within a single core. These threadlets speculatively execute future loop iterations, "leapfrogging" beyond the main thread's instruction window to expose medium-grained parallelism.

The authors propose a set of hardware extensions, most notably a Speculative State Buffer (SSB) that manages speculative memory state in a way that is contained within the core and respects the architectural memory model. Using a modified LLVM compiler and gem5 simulation of an aggressive 8-wide baseline, the authors demonstrate a geometric mean speedup of 9.5% on SPEC CPU 2017 with what they argue are modest hardware overheads. The work represents a compelling synthesis of classic speculative parallelization ideas with the realities of modern CPU design.

Strengths

1. High Conceptual and Contextual Relevance: The paper tackles one of the most significant challenges in high-performance computing today: the stagnation of single-thread performance. It correctly identifies that simply widening cores is becoming inefficient (as shown in their motivational Figure 1, page 2). By targeting the "medium-granularity" parallelism that falls between traditional ILP and coarse-grained TLP, LoopFrog addresses a well-known and valuable performance gap.

2. An Elegant Synthesis of Established Concepts: The true strength of this work lies not in a single novel mechanism, but in its masterful integration of ideas from several research domains. It draws from:

   • Classic TLS/SpMT (e.g., Multiscalar, STAMPede) for the core concept of speculative tasking.
   • Simultaneous Multithreading (SMT) for the underlying microarchitectural substrate of sharing resources between thread contexts.
   • Hardware Transactional Memory (HTM) for the principles of speculative state buffering and conflict detection, which are clearly reflected in the design of the SSB.
   • Compiler-Architecture Co-design (e.g., Tapir) for the use of lightweight, semantics-preserving hints as the interface between software and hardware.

   This synthesis results in a design that is far more practical and less disruptive than many of its historical predecessors. By confining speculation entirely within the core and making it transparent to the OS and memory system, the authors have significantly lowered the barrier to potential real-world adoption.

3. Strong and Well-Analyzed Empirical Results: A geometric mean speedup of 9.2-9.5% on SPEC CPU suites is a significant result that would be highly attractive to processor designers. The authors go beyond simply presenting the final number. The detailed breakdown of performance sources in Table 2 (page 11) is particularly insightful, revealing that the benefits arise not just from "true parallelism" but also from powerful prefetching side-effects that resolve hard-to-predict branches. This level of analysis provides a deep understanding of why the mechanism works and builds confidence in the results.

4. Thoughtful Design for a Modern System: The design carefully considers key requirements for modern architectures, such as preserving the memory consistency model (Section 4.1.4, page 6). This is a critical detail that was often a stumbling block for earlier TLS systems that exposed speculative state more widely. The granular conflict checking (as opposed to cache-line level) is another pragmatic choice that, as their sensitivity study shows, is key to avoiding false conflicts.

Weaknesses

1. The Compiler's Crucial Role is Underdeveloped: The evaluation relies on manual #pragma annotations to select loops for parallelization, which the paper describes as simulating "perfect static loop selection." While the hint insertion is automated, the far more challenging problem of identifying profitable loops automatically and robustly is left as future work. The entire system's practical success hinges on a compiler that can make intelligent decisions about which loops to annotate, avoiding the slowdowns the authors themselves mention (up to 10%). The paper would be stronger if it explored heuristics for this process or showed sensitivity to a non-perfect selection.

2. Overhead and Complexity Analysis is High-Level: The area and power analysis in Section 6.8 (page 12) is based on high-level models (CACTI) and analogies to existing SMT overheads. While a reasonable first-order approximation, the complexity of the proposed structures, particularly the SSB, may be understated. The logic for performing parallel, multi-versioned reads and snooping coherence traffic could have non-trivial implications for timing, verification effort, and power consumption that are not fully captured by this analysis.

3. Lack of Comparison to an "Iso-Area" Alternative: The paper provides a strong performance evaluation against its own baseline. However, to fully contextualize the efficiency of LoopFrog, it would be beneficial to compare it against an alternative design that uses a similar increase in hardware resources. For example, how does a 4-threadlet LoopFrog core compare to a baseline core that is simply made wider (e.g., 9- or 10-wide) or equipped with a much larger re-order buffer or a next-generation hardware prefetcher, assuming a similar transistor budget? This would help answer whether speculative threadlets are the most efficient use of that additional silicon.

Questions to Address In Rebuttal

1. The reliance on manual loop selection is a significant limitation on the path to a practical system. Could the authors elaborate on the primary challenges a fully automated compiler would face in making these decisions? Based on your analysis, what static or dynamic features (e.g., trip count, body size, memory access patterns, branch predictability) are the best indicators of a loop being profitable for LoopFrog, and how might a compiler collect and use this information?

2. The analysis in Table 2 (page 11) is excellent and reveals that 35% of the total speedup comes from "Prefetching" effects (primarily resolving branch conditions faster). This suggests that a significant benefit comes from running ahead, even if the speculation ultimately fails. How does this implicit prefetching capability compare to what could be achieved with a state-of-the-art, dedicated hardware prefetcher (e.g., one that can prefetch down complex pointer chains or recognize indirect branch patterns)? Is it possible that a more advanced but less complex prefetcher could capture a large fraction of this particular benefit?

3. The Speculative State Buffer (SSB) is the heart of the hardware proposal. The read path requires a parallel lookup across all active threadlet slices plus the L1D, followed by logic to merge the results into the correct version for the reading threadlet (Section 4.1.3, page 6). Could you comment on the potential timing implications of this logic? Is there a risk that it could extend the critical path of a load-to-use dependency and impact the core's clock frequency?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present LoopFrog, an in-core speculative execution framework for parallelizing loops within a single, wide out-of-order processor core. The stated goal is to utilize the spare execution resources evident in modern superscalar designs, thereby tapping into "medium-granularity" parallelism that is too small for traditional thread-level parallelism (TLP) across cores but too large to be fully captured by the instruction window of a single thread (ILP).

The proposed mechanism relies on compiler-inserted ISA hints (detach, reattach, sync) to delineate loop iterations. The microarchitecture uses these hints to spawn lightweight, OS-transparent "threadlets" that execute future iterations speculatively. A key component is the Speculative State Buffer (SSB), which buffers speculative memory writes, versions data, and detects inter-threadlet dependency violations. The entire mechanism is designed to be contained within the core, hiding the speculative state from the broader memory system and other cores. The authors report a geometric mean speedup of 9.5% on SPEC CPU 2017.

My analysis concludes that while the engineering and evaluation are sound, the core concepts are a synthesis of well-established prior art. The novelty is not in the invention of a new mechanism, but rather in the specific integration and application of existing techniques (SMT-based TLS, HTM-like memory versioning, hint-based ISAs) to the context of modern, wide superscalar cores. The "delta" over prior art is tangible but evolutionary, not revolutionary.

Strengths

1. Clear Problem Definition: The paper effectively frames its motivation around the diminishing returns of widening superscalar processors, clearly illustrated by the divergence of IPC and commit utilization in Figure 1 (page 2). This provides a compelling, modern context for revisiting speculative execution techniques.

2. Architectural Containment: The decision to confine all speculative state and management logic within a single core (the "in-core" aspect) is a significant and novel design choice compared to many prior TLS systems (e.g., STAMPede [28], Swarm [12]) that require modifications to the multi-core cache coherence protocol. This self-containment greatly improves the feasibility of deployment in commercial systems.

3. Granular Dependency Tracking: The analysis in Section 6.6 (page 11) demonstrating the performance advantage of sub-cache-line granularity for conflict detection is a valuable contribution. Many historical TLS/HTM systems operate at cache-line granularity, which is known to suffer from false sharing. By designing and evaluating a system with 4-byte granules, the authors directly address this known limitation and show it is a key enabler for their performance.

Weaknesses

1. Synthesis of Existing Concepts: The primary weakness from a novelty standpoint is that LoopFrog is a composite of previously published ideas.

   • In-Core SMT-based TLS: The core execution model of using multiple hardware contexts within a single core to run speculative threads is not new. It was a central idea in early work like the Dynamic Multithreading Processor (DMP) [1] and Implicitly-Multithreaded Processors (IMT) [21]. LoopFrog's "threadlets" are functionally identical to the speculative SMT threads in these proposals.
   • Speculative Memory Buffering: The Speculative State Buffer (SSB) is functionally analogous to the memory versioning systems proposed in countless TLS and Hardware Transactional Memory (HTM) papers. Its role in buffering writes, detecting read-after-write hazards across threads, and versioning data is a foundational concept in speculative parallelization. The description in Section 4.1 strongly echoes the logic of Log-based or Eager versioning HTM systems.
   • Hint-Based ISA: The use of architectural hints to guide speculative parallelization has also been explored. The detach/reattach hints are conceptually similar to the spawn/sync primitives in Tapir [23] (which the authors cite) and the fork/join model used to delineate tasks in ordered parallelism work like Swarm [12]. The innovation here is in the target of the hints (an in-core microarchitecture) rather than the concept of the hints themselves.

2. Marginal Delta Over Prior Art: The authors argue in Section 7 (page 12) that the gains from prior SMT-based TLS have been "superseded by progress in CPU core design." While plausible, the paper does not sufficiently articulate the specific architectural "delta" that makes LoopFrog succeed where its predecessors may have faltered. Is it simply the availability of more idle resources, or is there a fundamental architectural innovation in LoopFrog beyond scaling up old ideas? The novelty claim rests heavily on this distinction, which needs to be sharpened.

3. Performance Gains vs. Complexity: The implementation of the SSB, conflict detector, and checkpointing logic introduces significant design complexity, regardless of the final silicon area. A crucial finding in the paper's own analysis (Section 6.4.2, page 11) is that a large fraction of the total speedup (32% + 3% = 35%) comes from the prefetching side-effects of (often failed) speculation. This raises a critical question: could a significant portion of the 9.5% geomean speedup be achieved with a much simpler, dedicated hardware prefetcher aware of loop structures, without the full complexity of speculative execution, state buffering, and squash logic? This potential for a simpler alternative dilutes the perceived value of the proposed complex mechanism.

Questions to Address In Rebuttal

1. Differentiation from SMT-TLS: Please articulate the specific, fundamental microarchitectural differences between LoopFrog and prior SMT-based TLS proposals like IMT [21] and DMP [1]. Beyond the argument that baseline cores are now wider, what core mechanism in LoopFrog is novel and essential to its success that was absent in this prior art?

2. Comparison to HTM: The SSB's functionality closely mirrors that of an Eager versioning Hardware Transactional Memory system. Could the authors contrast their detach/reattach/sync model for loops against a hypothetical implementation where each loop iteration is simply wrapped in an HTM transaction? What are the fundamental performance and complexity advantages of the LoopFrog model that would justify it over leveraging an existing HTM implementation?

3. Justification of Speculative Execution for Prefetching: Given that over a third of the performance benefit derives from prefetching effects (as detailed in Section 6.4.2), please justify the necessity of the full speculative execution and state versioning framework. Could a simpler "scout thread" or a runahead execution scheme, which executes instructions only for their prefetching side-effects and discards all results, achieve a comparable performance gain with substantially lower hardware complexity than the proposed SSB and conflict detector?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose "Multi-Stream Squash Reuse," a microarchitectural technique to recover useful work from multiple, non-contiguous previously squashed instruction streams. This extends prior work like Dynamic Control Independence (DCI) and Register Integration (RI), which are typically limited to reusing work from only the most recent squashed path. The central mechanism is the "Rename Mapping Generation ID" (RGID), a versioning tag for architectural-to-physical register mappings, intended to track data dependencies across disjoint execution contexts. The authors present an implementation integrated into the Fetch and Rename stages and report average IPC improvements of 2.2% on a subset of SPECint2006, 0.8% on SPECint2017, and 2.4% on GAP benchmarks.

Strengths

1. Sound Motivation: The paper correctly identifies a limitation in existing squash reuse schemes. The analysis in Section 2.2.5 (Figure 4) provides empirical evidence that a non-trivial fraction of reconvergence opportunities (15-43% in some SPEC benchmarks) involves streams other than the immediately preceding one. This quantitatively justifies the exploration of a multi-stream approach.

2. Plausible Core Mechanism: The RGID concept is a conceptually straightforward approach to tracking the temporal state of register mappings. By versioning the mappings themselves, it avoids the complexities of reconstructing and comparing full dependency graphs across streams.

Weaknesses

My primary concerns with this work center on the practical viability of the proposed solution, the significance of the results relative to the induced complexity, and an insufficient analysis of critical corner cases, particularly memory dependencies.

1. Marginal Performance Gains for Significant Hardware Complexity: The headline results are underwhelming. An average IPC gain of 0.8% on SPECint2017 and 2.2% on SPECint2006 is deep in the territory of diminishing returns. Yet, the hardware required to achieve this is substantial: two-dimensional Wrong-Path Buffers (WPBs), Squash Logs, per-architectural-register RGID counters, extensions to the RAT and ROB to store RGIDs, and non-trivial reconvergence detection logic (Section 3.4, Figure 7) and reuse test logic (Section 3.5, Figure 8). The authors have not made a convincing case that this complexity is justified for such a minor performance uplift on standard workloads.

2. Superficial Treatment of Memory Order Violations: The handling of memory dependencies, a notoriously difficult problem for speculative reuse techniques, is a critical weakness. In Section 3.8, the authors propose two options: a Bloom filter or re-executing all reused load instructions. They state, "In our evaluation, we choose to implement the latter mechanism for simplicity." This is a significant concession that undermines the entire premise of "reuse." Re-executing all loads is not reuse; it is a "verification" that consumes execution ports and energy, and its performance cost could easily negate the gains from reusing ALU instructions. The paper provides no data on what percentage of reused instructions are loads, nor the performance penalty incurred by this re-execution policy. This is not a minor implementation detail; it is fundamental to the correctness and performance of the scheme.

3. Unconvincing Critical Path Analysis: The authors claim their modifications do not affect the processor's critical path, but the evidence is thin. The reuse test logic presented in Section 3.5 and Figure 8 introduces a dependency for the Nth instruction's reuse test on the reuse test results of the preceding N-1 instructions in the same cycle. While they argue this is overshadowed by the existing register dependency check (Reg CMP), adding any serial dependency chain to the rename stage is hazardous. The post-synthesis results in Table 4 report up to 41 logic levels for an 8-wide machine. For a target 2 GHz clock (a 500ps cycle time), this path depth (~12 ps/level) is extremely aggressive and likely on the critical path, contrary to the authors' assertions.

4. RGID Management is Under-specified: The paper mentions a "global RGID reset" when counters overflow or other conditions are met (Section 3.4). This is a coarse-grained, disruptive event that disables the entire mechanism. The authors provide no analysis on the frequency of these resets. If overflows are common, the actual achievable performance will be lower than what is simulated, as the mechanism will be periodically unavailable. This is a crucial parameter for understanding the real-world efficacy of the RGID system.

5. Selective Evaluation: The authors state in Section 4 that they "select benchmarks from the SPECint2006, 2017 suites that have a branch misprediction rate of more than 3%." This constitutes a form of selective reporting. While justified for studying the mechanism, it inflates the perceived average benefit. The impact on the full, unmodified SPEC suites would provide a more honest assessment of the technique's value to a general-purpose processor and would almost certainly be much lower than the already marginal numbers reported.

Questions to Address In Rebuttal

The authors must address the following points to substantiate their claims:

1. Cost/Benefit Justification: Given the added hardware complexity (WPBs, Squash Logs, RAT/ROB extensions), how do you justify an average IPC improvement of less than 2.5% on SPEC and GAP suites? Provide a detailed area and power breakdown versus a baseline core.

2. Memory Dependencies: Please provide a quantitative analysis of your chosen memory hazard solution (re-executing all loads). Specifically:

   • What percentage of instructions that passed the RGID reuse test were loads?
   • What is the performance impact of re-executing these loads versus truly reusing their results? Please provide data showing the IPC gain before and after accounting for load re-execution.
   • Why was a proper analysis of a Bloom filter approach, including its false-positive rate and performance impact, omitted?

3. RGID Overflow: What is the frequency of the "global RGID reset" event in your SPEC and GAP simulations? What is the performance cost associated with the periods where the squash reuse mechanism is disabled awaiting re-synchronization?

4. Critical Path: Can you provide a more rigorous analysis comparing the timing of your full reuse-test logic path (41 levels for 8-wide) against the critical path of a comparable, baseline high-frequency rename stage? The claim that this logic is "overshadowed" requires stronger evidence than is provided.

5. Full Suite Results: Please provide performance results for the entire SPECint2006 and SPECint2017 suites, not only the subset with high misprediction rates, to demonstrate the technique's overall value.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper addresses the long-standing problem of branch misprediction penalties in high-performance processors. The authors observe that existing techniques for "squash reuse"—recovering useful work from a mispredicted path—are overly constrained. They typically only consider reconvergence between the current correct path and the single, most recent incorrect path, often limiting them to simple if-else control structures.

The core contribution of this work is the concept of Multi-Stream Squash Reuse, a mechanism that generalizes reuse to enable the current instruction stream to reconverge with any of several previously squashed streams. To achieve this, the authors introduce an elegant and novel mechanism called Rename Mapping Generation ID (RGID). Each time an architectural register is mapped to a new physical register, the mapping is tagged with a unique, incrementing ID. By comparing the RGIDs of an instruction's source operands on the current path with those on a past squashed path, the processor can quickly and robustly verify data-flow integrity without complex dependency tracking.

The authors implement this scheme with modest hardware extensions to the fetch (Wrong-Path Buffers) and rename (Squash Log) stages and demonstrate average IPC improvements of 2.2% on SPECint2006 and 2.4% on GAP benchmarks, with significant gains on select workloads.

Strengths

1. Excellent Problem Formulation and Motivation: The paper's primary strength lies in its clear and compelling motivation. The authors effectively argue that the traditional model of reconvergence is too simplistic for modern complex control flow. The distinction between "software-induced" and "hardware-induced" multi-stream reconvergence (Section 2.2.2, page 3) is particularly insightful, highlighting that out-of-order execution itself can create complex reconvergence scenarios that simpler models would miss. The profiling data in Figure 4 (page 5), which shows that non-neighboring stream reconvergence constitutes a significant fraction (15-43%) of opportunities, provides strong evidence that this is a problem worth solving.

2. Elegant and Scalable Core Mechanism (RGID): The RGID concept is the technical heart of the paper and its most significant contribution. It is a beautifully simple solution to the complex problem of tracking data-flow equivalence across multiple divergent speculative paths. It neatly sidesteps the major issues of prior table-based schemes like Register Integration (RI), such as table conflicts and the overhead of transitive invalidations (as well-argued in Section 3.7, page 10). It also appears more scalable than extending queue-based schemes like Dynamic Control Independence (DCI), which would require managing complex poison vectors across multiple stream segments. The RGID is a powerful abstraction for representing dynamic data versions.

3. Contextualization within the Field: This work fits perfectly within the lineage of research on Control Independence and squash reuse, pioneered by works from Sohi, Smith, and Rotenberg. It can be seen as a direct and logical evolution of foundational papers like Register Integration [24] and Dynamic Control Independence [5]. Where RI used physical register names and DCI used a single shadow ROB, this work introduces a more general versioning system (RGIDs) to create a more powerful and flexible framework. The authors do an excellent job of positioning their work relative to these predecessors in Section 3.7.

4. Thorough and Credible Evaluation: The experimental methodology is solid. The use of gem5 with detailed modeling, combined with SPEC and GAP benchmarks, is appropriate. Crucially, the inclusion of a direct comparison against a re-implementation of Register Integration (Figure 12, page 13) strengthens their claims. Furthermore, the post-synthesis complexity analysis for the critical logic components (Table 4, page 13) adds a layer of practicality and credibility, showing that the proposed hardware is feasible within a reasonable area and power budget.

Weaknesses

While the core idea is strong, its practical realization as presented has a few points that could be strengthened:

1. Handling of Memory Dependencies: The proposed solution for handling memory order violations (Section 3.8, page 10) feels underdeveloped compared to the elegance of the RGID mechanism for registers. The authors evaluate a simplified approach of re-executing all reused load instructions, which seems to partially defeat the purpose of reuse. While this is acknowledged as a choice for simplicity, it leaves a significant question mark over the true potential of the technique. The performance gains might be considerably higher if a more sophisticated memory dependency mechanism, such as the proposed Bloom filter, were implemented and evaluated.

2. Practicality of RGID Overflow and Reset: The paper briefly mentions a global reset mechanism to handle RGID counter overflows (Section 3.4, page 8). However, the performance implications of this are not explored. Halting the acceptance of new squashed streams, even temporarily, could introduce performance jitter or bubbles that negate some of the gains, especially in programs with very long-running phases of high branch misprediction rates. Some data on the frequency of these resets and their performance cost would be valuable.

3. Modest Gains on Newer Benchmarks: The average IPC improvement on SPECint2017 is notably lower (0.8%) than on SPECint2006 (2.2%). This suggests that either the control-flow patterns in the newer suite offer fewer multi-stream reconvergence opportunities, or that other bottlenecks (e.g., memory dependencies, cache misses) are more dominant, limiting the impact of this optimization. A deeper analysis of this discrepancy would strengthen the paper.

Questions to Address In Rebuttal

1. On Memory Dependencies: The decision to re-execute all loads is a major simplification. Could the authors provide data on what fraction of the total reuse opportunities come from load instructions in their key benchmarks? This would help the committee understand how much performance is being left on the table by the current evaluation model and assess the importance of developing a more sophisticated memory hazard detection scheme.

2. On RGID Overflow: Could the authors provide statistics on the frequency of RGID overflows and the subsequent mechanism resets during the evaluated benchmark runs? What is the performance sensitivity to the "fixed number of instructions" that must be committed before the mechanism is re-enabled?

3. On SPECint2017 Performance: Could the authors offer a more detailed hypothesis for the lower average gains on SPECint2017? Is this an artifact of the specific benchmarks chosen, or does it reflect a broader trend in modern software where this class of optimization provides diminishing returns?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

The paper proposes "Multi-Stream Squash Reuse," a technique to recover useful work from multiple, non-contiguous, previously squashed instruction streams following a branch misprediction. This extends the scope of traditional squash reuse, which typically only considers reconvergence with the single most recent mispredicted path. The core enabling mechanism is a novel concept called Rename Mapping Generation ID (RGID), a versioning tag applied to each architectural-to-physical register mapping. By comparing RGIDs between a currently fetching instruction and its counterpart in an old squashed stream, the system can verify data integrity and reuse previously computed results.

My assessment is that the core idea of extending squash reuse to multiple streams, enabled by the novel RGID mechanism, represents a genuine contribution to the field. The mechanism avoids known pitfalls of prior art. However, the demonstrated performance benefits appear modest relative to the proposed hardware complexity, raising questions about the practical utility of this novel concept.

---

Strengths

1. Novelty of Scope (The "Multi-Stream" Concept): The primary novelty lies in identifying and targeting reconvergence opportunities beyond the immediate predecessor stream. Prior art, most notably Dynamic Control Independence (DCI) [5], established the queue-based approach but limited its scope to a single squashed stream. This paper convincingly argues in Section 2.2 that both software control-flow structures and hardware phenomena (out-of-order branch resolution) create scenarios where the correct path might reconverge with an "ancestor" squashed stream. To my knowledge, this is the first work to systematically design a mechanism to exploit this.

2. Novelty of Mechanism (The "RGID" Concept): The proposed RGID mechanism is an elegant and novel solution to the data dependency tracking problem in this complex multi-stream context. It differs fundamentally from prior approaches:

   • It avoids the table-based structure of Dynamic Instruction Reuse (DIR) [30] and Register Integration (RI) [24], thereby sidestepping the issues of table conflicts and transitive invalidations, as correctly articulated in Section 3.7.
   • It offers a more scalable approach than naively extending DCI's poison-vector method. As the authors argue in Section 2.2.3, tracking dependency segments across N streams with poison vectors would lead to significant management complexity. RGIDs provide a decentralized check: two execution states are equivalent with respect to a register if their RGIDs match. This is a clever way to compare states without needing to reconstruct the full path between them.

3. Clear Articulation of the "Delta" from Prior Art: The authors demonstrate a strong command of the literature. Section 3.7 provides a direct and accurate comparison against DIR, RI, and DCI, clearly isolating what makes their approach different and, in their view, superior. This clarity is commendable.

---

Weaknesses

1. Marginal Performance Gains vs. Significant Complexity: The central weakness is the trade-off between the novelty and its impact. The proposed architecture introduces non-trivial hardware: Wrong-Path Buffers (WPBs), a multi-stream Squash Log, and additional storage in the RAT and ROB for RGIDs (summarized in Table 2). The post-synthesis results in Table 4 confirm this, showing thousands of µm² in area and a critical path for the reuse test that scales with pipeline width (e.g., 41 logic levels for an 8-wide machine). In return for this complexity, the average IPC gains are 2.2% on SPECint2006, 0.8% on SPECint2017, and 2.4% on GAP. While maximum gains on specific benchmarks like astar are notable (8.9%), the average improvements across standard suites are low for a mechanism of this complexity. A truly innovative idea should ideally provide a more compelling performance-per-transistor argument.

2. Unexplored Practicalities of RGID Management: The RGID concept, while novel, has potential failure modes that are not fully characterized. The paper mentions in Section 3.4 that a "global RGID reset" is triggered on overflow or after repeated overflow events, causing a "temporary halt" in accepting new squashed streams. This sounds disruptive. The frequency of these events and the precise performance penalty are not quantified. If RGID counters are small, this reset mechanism could frequently disable the entire benefit of the multi-stream reuse, undermining the proposal's value.

3. The Opportunity is Niche: The authors' own profiling in Figure 4 shows that for a majority of benchmarks (especially in GAP), "simple reconvergence" — the kind that can be handled by a single-stream DCI-like scheme — constitutes the vast majority of opportunities. The combined software- and hardware-induced multi-stream opportunities are significant in only a handful of the SPEC benchmarks shown (omnetpp, astar). This suggests that the novel problem the paper solves, while real, may not be prevalent enough to justify a general-purpose hardware solution.

---

Questions to Address In Rebuttal

1. Complexity vs. Benefit Justification: Can the authors provide a more direct analysis of the efficiency of their proposal? For instance, what is the IPC-per-area (mm²) or IPC-per-watt (mW) improvement of your technique over the baseline? A 2% IPC gain for a 5% area/power increase might be acceptable, but the current presentation makes this trade-off difficult to assess.

2. RGID Overflow Analysis: Please provide data on the frequency of RGID overflows and the resulting global resets during the benchmark runs. How much performance is lost due to the "temporary halt" described in Section 3.4? This is critical to understanding if the mechanism is robust in practice.

3. Characterization of Ideal Workloads: The novelty of this work lies in addressing multi-stream reconvergence. Could you provide a more detailed characterization of the program structures (e.g., deeply nested loops with data-dependent exits, complex recursive functions) that generate these opportunities? This would help justify the design by clearly defining the domain where its novel capabilities provide a substantial advantage over single-stream approaches.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper posits that state-of-the-art LLC replacement policies, such as Hawkeye and Mockingjay, are sub-optimal in many-core systems with sliced LLCs. The authors identify two primary deficiencies: 1) "myopic" reuse predictions resulting from access streams being scattered across slices, and 2) "underutilized" sampled sets where randomly selected sets for monitoring receive too few misses to provide useful training data. To address this, the paper proposes "Drishti," a set of two enhancements: a per-core, globally-accessible reuse predictor to create a non-myopic view, and a dynamic sampled cache (DSC) that intelligently selects high-miss-rate sets for monitoring. The authors claim that these enhancements significantly improve the performance of existing policies, with Mockingjay's geomean speedup over LRU on a 32-core system increasing from 6.7% to 13.2%.

Strengths

1. The paper correctly identifies the "myopic" prediction problem in sliced LLC architectures as a potential performance limiter. The scattering of a single PC's access stream across multiple per-slice predictors is a valid architectural concern.
2. The motivation for the proposed enhancements is supported by some initial data analysis (e.g., Figures 2, 3, and 5), which illustrates the access scattering and skewed miss distribution that form the basis of the authors' arguments.
3. The authors provide an ablation study (Figure 17) that attempts to isolate the performance contributions of each proposed mechanism (the global predictor and the dynamic sampled cache).

Weaknesses

1. Critical Dependency on Unrealistic Hardware: The central and most significant weakness is the proposal's complete reliance on a dedicated, low-latency, three-cycle interconnect (NOCSTAR). This is not a simple policy tweak but a fundamental alteration of the on-chip network fabric. The authors' own data in Figure 11a demonstrates that without this idealized network, their proposal results in a significant performance slowdown (up to 9% on average for 32 cores). This makes NOCSTAR a hard requirement, not an optimization. The practicality, area, power, and complexity costs of adding a second, dedicated network are non-trivial and are not sufficiently justified against the performance gains. The proposal's viability hinges entirely on this assumption.

2. Selective Scope and Inconsistent Baseline Comparison: The authors explicitly state in the introduction (Section 1, Page 1) that they "do not consider machine learning (ML) [55] and reinforcement learning (RL) [38]-based LLC replacement policies." This is a critical omission, as these policies represent the current state-of-the-art frontier. This selective exclusion creates a simplified problem space where Drishti's enhancements may appear more effective than they would against stronger, more adaptive baselines. This exclusion is then directly contradicted in Section 6 (Page 12), where the authors claim applicability to and provide results for CHROME [38] (RL-based) and Glider [55] (deep learning-based). This inconsistency suggests either a flawed initial premise or a post-hoc attempt to broaden the paper's applicability without a rigorous, head-to-head comparison in the main evaluation.

3. Fragile Justification for Dynamic Sampled Cache (DSC): The justification for the DSC relies heavily on workloads with highly skewed miss distributions (e.g., mcf in Figure 5a). The authors concede that for workloads with uniform distributions (lbm in Figure 5c), this mechanism is ineffective and must be disabled via a detection heuristic (Section 4.2, Page 7). This admission reveals the DSC is not a universally applicable improvement but a mode-based optimization that adds the complexity of phase/behavior detection logic. It is unclear how this mechanism performs on the wide spectrum of workloads between these two extremes. The claim of a net hardware saving (Table 3) is also tenuous; while the sampled cache structure is smaller, the proposal adds k-bit saturating counters for every LLC set, selection logic, and the aforementioned NOCSTAR interconnect. The overall system complexity is demonstrably higher.

4. Insufficient Detail in Motivational Analysis: In the motivational analysis (Section 3.1, Figure 3), the methodology for simulating the "global view" predictor is not sufficiently detailed. It appears to be an idealized oracle with perfect and instantaneous access to all sampled set information across all slices. This sets an unachievably high bar and may inflate the perceived potential of a global predictor, making the authors' practical implementation seem closer to the ideal than it actually is. The performance gap between this oracle and the authors' NOCSTAR-based implementation is not quantified.

Questions to Address In Rebuttal

1. Please provide a detailed analysis of the performance of Drishti using the existing on-chip mesh interconnect, assuming realistic contention and latency derived from the increased predictor traffic. Given that your own data (Figure 11) shows a performance loss without NOCSTAR, how can this proposal be considered a practical enhancement to replacement policies rather than a coupled co-design of a policy and a new interconnect?

2. Please justify the exclusion of ML/RL-based policies like CHROME or Glider from the main evaluation (Section 5), especially given that you test against them in Section 6. To properly situate Drishti's contribution, a full comparison against these true state-of-the-art policies is necessary in the main results tables and figures.

3. How does the Dynamic Sampled Cache (DSC) perform on workloads where the miss distribution across sets is relatively flat but not perfectly uniform (i.e., not as skewed as mcf, but not as flat as lbm)? What is the performance and hardware overhead of the workload detection logic that is required to enable/disable the DSC?

4. Clarify the precise simulation setup for the idealized "global view" predictor used in Figure 3. Is this a contention-free model with zero-cycle access latency? Please provide data comparing this ideal oracle to your implemented per-core global predictor with the three-cycle NOCSTAR latency to properly frame the implementation's efficiency.

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper, "Drishti," addresses a timely and important disconnect between the design of state-of-the-art Last-Level Cache (LLC) replacement policies and their deployment in modern many-core systems. The authors correctly observe that while advanced policies like Hawkeye and Mockingjay show significant promise, their evaluation has largely been confined to monolithic LLC models. In contrast, commercial processors utilize sliced, non-uniform cache architectures (NUCA).

The core contribution is not a new replacement policy, but rather a set of two well-motivated architectural enhancements that make existing policies "slice-aware." The authors first identify the problem of "myopic predictions," where per-slice predictors have an incomplete view of a program's global reuse behavior. Their solution is a clever compromise: a local (per-slice) sampled cache that feeds into a per-core, yet globally-aware, reuse predictor. Second, they identify that randomly selected sampled sets are often "underutilized," receiving too few misses to effectively train the predictor. They propose a dynamic sampled cache that intelligently selects sets with high miss rates (MPKA) to maximize learning efficiency. The paper demonstrates that these enhancements, when applied to Hawkeye and Mockingjay, can substantially boost their effectiveness, for instance, nearly doubling the performance gains of Mockingjay on a 32-core system.

Strengths

1. Excellent Problem Formulation: The paper's greatest strength is its premise. It tackles a practical and increasingly relevant issue that is often overlooked in academic studies. By grounding their work in the reality of commercial sliced LLCs (as mentioned in Section 1, page 1), the authors immediately establish the significance of their investigation. This is a crucial step in bridging the gap between theoretical microarchitectural improvements and real-world processor design.

2. Clear, Data-Driven Motivation: The motivation presented in Section 3 (page 3) is compelling. Figure 2, which quantifies how many Program Counters (PCs) are confined to a single slice, provides a powerful and intuitive argument for why per-slice predictors are inherently myopic. Furthermore, the analysis of underutilized sampled sets (Section 3.2, page 4), culminating in the simple yet powerful experiment in Table 1, perfectly justifies the need for a more intelligent set selection mechanism.

3. Elegant and Generalizable Solutions: The proposed enhancements are not overly complex and demonstrate a deep understanding of the design trade-offs. The "per-core yet global" predictor is a pragmatic solution that balances the need for a global view against the communication overhead of a fully centralized structure. The dynamic sampled cache uses a well-understood technique (saturating counters) to solve the problem in a simple and effective manner. Most importantly, as shown in Section 6 (page 12), these ideas are not limited to Hawkeye and Mockingjay. The authors demonstrate their applicability to a wider class of prediction-based policies (including SHiP++, CHROME, and Glider), elevating the work from a specific optimization to a more general architectural principle.

4. Contextualization within the Field: This work fits beautifully into the ongoing evolution of cache management. It follows the trajectory from simple heuristics (LRU), to predictive policies (RRIP, SHiP), to emulating optimality with large-scale tracking (Hawkeye, Mockingjay). Drishti represents the next logical step: adapting these sophisticated predictors to the physical and distributed reality of modern hardware. It addresses the "systems" aspect of a microarchitectural problem.

Weaknesses

1. Framing as an "Enhancement": While technically accurate, framing the work solely as an enhancement to existing policies may undersell its conceptual contribution. The core ideas—decoupling the scope of sampling from the scope of prediction and making the sampling process adaptive—are fundamental principles that could inform the design of future replacement policies from the ground up.

2. Reliance on a Dedicated Interconnect: The proposal for a dedicated interconnect (NOCSTAR, Section 4.1.4, page 6) to handle predictor traffic is a practical solution but also introduces non-trivial design complexity and area/power overhead, however small. While the authors justify its low cost, it represents an additional system component that must be validated. An exploration of alternative approaches, such as leveraging quality-of-service (QoS) mechanisms on the main network-on-chip (NoC), would have strengthened this aspect of the proposal.

3. Trace-Driven Simulation Limitations: The use of a trace-driven simulator (ChampSim) is standard practice and perfectly acceptable for this type of study. However, it inherently cannot capture complex feedback loops where changes in cache behavior might alter the application's execution path, memory access timing, or prefetching behavior. This is a minor limitation but worth noting, as the true system-level impact could differ slightly in an execution-driven environment.

Questions to Address In Rebuttal

1. The utility of each enhancement is analyzed in Figure 17 (page 10). It appears the move to a global predictor provides the largest performance jump, with the dynamic sampled cache (DSC) providing a further, significant boost. Could the authors confirm this interpretation and comment on the synergy between the two proposals? Are the gains largely additive, or is there a super-linear effect where the DSC is particularly effective because it is training a more powerful global predictor?

2. Regarding the NOCSTAR interconnect, could the authors elaborate on why a dedicated network is preferable to using a high-priority virtual channel on the existing NoC? While the paper argues for low latency, could a QoS approach provide "good enough" latency without the cost of dedicated wiring and arbiters?

3. The paper effectively shows scalability up to 128 cores. As we look toward future systems with hundreds of cores, does the "per-core yet global" predictor model begin to face new scalability challenges? For example, does the storage for the per-core predictors become prohibitive, or does the aggregate traffic to these distributed predictors start to congest the dedicated interconnect?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper identifies a valid and often overlooked problem: state-of-the-art Last-Level Cache (LLC) replacement policies, such as Hawkeye and Mockingjay, which were designed for monolithic caches, suffer from degraded performance on the sliced LLC architectures common in modern many-core processors. The authors attribute this degradation to two primary factors: "myopic predictions" from per-slice predictors that lack a global view of an application's reuse behavior, and "underutilized sampled sets" where randomly chosen sets for monitoring see too few misses to provide a useful training signal.

To address this, the paper proposes "Drishti," a set of two hardware enhancements. The first is a "per-core yet global" reuse predictor architecture, which aims to provide a global view of a core's access patterns without the bottleneck of a single centralized predictor. This is paired with a local, per-slice sampled cache. The second enhancement is a "dynamic sampled cache," which eschews random set selection in favor of a mechanism that periodically identifies and samples LLC sets with the highest miss rates (MPKA), thereby focusing the monitoring effort where it is most impactful.

Strengths

1. Problem Identification: The paper correctly identifies a significant gap between academic research on LLC replacement policies and the reality of commercial hardware. The analysis in Section 3.1 (page 3), particularly Figures 2 and 3, provides a clear and compelling demonstration of the "myopic" prediction problem in sliced LLCs. This is a valuable insight.

2. System Integration: The authors propose a complete system solution. The two enhancements are designed to work in concert, addressing two distinct but related weaknesses of existing policies in a sliced environment. The consideration of interconnect traffic and the proposal to use a dedicated interconnect (NOCSTAR) shows a thoroughness in engineering the solution.

Weaknesses

My evaluation is centered on the fundamental novelty of the proposed ideas. While the engineering and integration are competent, the core concepts appear to be reformulations or applications of pre-existing principles.

1. "Per-Core yet Global" Predictor Lacks Conceptual Novelty: The core idea here is to overcome the limitations of distributed, uncoordinated decision-making by introducing a shared, global perspective. This is a foundational concept in distributed systems and parallel computing, not a new one. The paper itself acknowledges that a fully centralized predictor is a "trivial solution" (Abstract, page 1). The proposed "per-core yet global" architecture is an engineering trade-off to manage the scalability of this known solution. It is essentially a form of state replication (one predictor instance per core) where each replica is updated by a single logical stream (the core's accesses). While this specific arrangement might be new in the context of LLC predictors, the architectural pattern itself is not novel. The problem of balancing global state with distributed access is well-trodden ground. The novelty lies in the application, not the invention.

2. "Dynamic Sampled Cache" is an Application of a Known Principle: The insight that random sampling can be inefficient and that monitoring should focus on "hotspots" or high-activity regions is not new. The principle of dynamically identifying high-pressure resources to guide policy is seen in other areas of cache management. For instance:

   • Utility-Based Cache Partitioning (UCP): This class of techniques monitors the marginal utility (e.g., miss rate reduction) of allocating cache resources to different applications. This inherently involves identifying which applications/cores are creating the most memory pressure.
   • Adaptive Set Management: Prior work like The V-Way Cache [47] and The Set Balancing Cache [50] dynamically adjust cache associativity or line placement based on monitoring pressure within individual sets. The underlying principle is the same: identify high-contention sets and act upon them.

   Drishti's contribution is to apply this principle of "focus on the hotspots" to the specific problem of selecting which sets to sample for training a reuse predictor. The mechanism proposed—using per-set saturating counters to track MPKA—is a straightforward heuristic implementation of this principle. The idea is an incremental refinement of sampling strategy, not a fundamentally new concept.

3. Overall Contribution is Systematization, Not Invention: The primary contribution of this paper is the careful identification of a real-world system issue and the competent engineering of a solution by combining and adapting existing architectural principles. It is a work of system integration. While valuable, it does not present a novel algorithmic or architectural paradigm for cache replacement. The "delta" over prior art is in the specific application and combination of ideas, which, while effective, is not large from a conceptual standpoint.

Questions to Address In Rebuttal

1. The concept of creating a global view to overcome myopic local decisions is a classic problem. Please articulate the fundamental architectural novelty of the "per-core yet global" predictor beyond it being a point in the design space between fully distributed and fully centralized state. What makes this specific arrangement conceptually distinct from other forms of replicated, coherent state in parallel architectures?

2. Please contrast the core principle of your dynamic sampled cache with prior work in adaptive cache management (e.g., adaptive associativity, cache partitioning) that also dynamically identify high-pressure sets/regions to guide policy decisions. Is the novelty in the principle itself, or solely in its application to selecting predictor training sets?

3. The proposed hardware changes (a dedicated NOCSTAR interconnect, per-set MPKA counters) are significant. Given that the underlying concepts are adaptations of existing principles, could a substantial fraction of the performance gain be achieved through a less complex approach, perhaps by leveraging existing coherence traffic or performance counters to approximate a global view and identify high-miss sets? Please justify why this level of hardware complexity is necessary for the claimed advance.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose TRRIP, a software-hardware co-design for instruction cache replacement. The core mechanism leverages Profile-Guided Optimization (PGO) to classify code into "hot," "warm," and "cold" temperatures. This temperature information is stored in page table entries (PTEs) via an OS interface and subsequently passed to the L2 cache controller with each memory request. The hardware then uses this hint to modify a baseline RRIP replacement policy, prioritizing the retention of "hot" instruction lines. The paper claims this approach reduces L2 instruction MPKI by 26.5%, leading to a 3.9% geomean speedup on mobile-like workloads, with minimal hardware modifications.

Strengths

1. Problem Identification: The motivation presented in Section 2 is well-founded. Figure 3 effectively demonstrates that even after PGO-based layout optimizations, hot instruction code still suffers from high reuse distances, clearly identifying a remaining performance gap that pure software or conventional hardware policies fail to address.

2. Pragmatic Hinting Mechanism: The proposal to pass software hints to hardware via existing, implementation-defined PTE bits (as detailed in Section 3.1, page 5) is a practical design choice. It correctly identifies and avoids the significant barrier of modifying the Instruction Set Architecture (ISA), which has doomed many similar co-design proposals.

3. Baseline Policy: The choice to build upon RRIP is sensible. RRIP is a strong and widely recognized baseline, making the claimed improvements more credible than if they were compared against a weaker policy like pure LRU.

Weaknesses

My primary concerns with this work center on the validity of its evaluation methodology and the overstatement of its practical applicability.

1. Lack of Representativeness in Benchmarks: The paper's central premise is to solve a problem observed in "modern mobile system software" (Section 2.1, page 2), citing components like UI frameworks, renderers, and interpreters (Figure 1). However, the evaluation in Section 4 is conducted on a suite of "proxy benchmarks" (e.g., clang, gcc, deepsjeng, omnetpp). There is no evidence provided to substantiate the claim that the instruction access patterns and memory behavior of these proxy applications are representative of the actual mobile system components they are meant to mimic. This creates a fundamental disconnect between the problem being motivated and the problem being solved.

2. Simulation Fidelity and Key Omissions: The evaluation relies on the Sniper simulator, which is trace-based. As the authors themselves concede in Section 4.1 (page 7), this means the simulation does not model wrong-path execution. For a study focused on the CPU frontend, this is a critical omission. Instruction prefetchers, especially aggressive ones, frequently operate on wrong paths, polluting the cache. The interaction between a replacement policy and wrong-path prefetches is a first-order effect, and its absence calls the accuracy of the MPKI and speedup results into serious question. The "pseudo-FDIP" prefetcher model further weakens the setup.

3. Unfair Comparison to Prior Art: The implementation of competing state-of-the-art techniques, particularly Emissary, is described as being done "to the best of our ability" (Section 4.3, page 8). This phrasing suggests a potential lack of fidelity to the original proposal. Emissary's mechanism is tightly coupled to a specific microarchitecture's stall signals. It is unclear if the authors' implementation on a different simulation infrastructure accurately captures its behavior. Consequently, the performance comparison may be unfairly skewed in TRRIP's favor due to a suboptimal implementation of the competition.

4. Superficial Analysis of Practical Limitations:

   • PGO Brittleness: The entire system hinges on the quality and representativeness of PGO profiles. The paper leverages an existing PGO flow but fails to discuss the significant engineering challenges of maintaining profile freshness and coverage for complex, rapidly evolving system software. An application's behavior might drift from the profile, rendering TRRIP's temperature hints incorrect and potentially degrading performance.
   • Page Size Issues: The analysis in Section 4.9 (page 11) acknowledges that larger page sizes can cause a single page to contain code of multiple temperatures, corrupting the hint. The proposed solutions—"adding padding" or "disable marking"—are hand-wavy and their performance implications are not evaluated. Adding padding increases the code footprint, while disabling marking negates the benefit of TRRIP on those pages. This is a non-trivial practical issue that is inadequately addressed.
   • The "Zero-Cost" Fallacy: While the paper claims minimal hardware changes by reusing PTE bits, it glosses over the significant, cross-stack engineering cost. Coordinating changes between the compiler, OS, and multiple hardware teams to correctly implement and validate this feature is a monumental undertaking. Claiming this is "practical and adoptable" without discussing this process is misleading.

Questions to Address In Rebuttal

1. Please provide quantitative evidence (e.g., analysis of instruction stream deltas, call graph similarity, cache access patterns) to demonstrate that the chosen proxy benchmarks are indeed representative of the real mobile system software components whose problems motivate this work (as shown in Figure 1).

2. The authors' implementation of Emissary is stated as a best-effort port. Can you elaborate on the specific microarchitectural signals used to drive your Emissary implementation and justify why this is a fair and faithful representation of the original work, whose performance is heavily dependent on those signals?

3. TRRIP is fundamentally a static, profile-based optimization. How does the system handle code for which no PGO profile exists (e.g., dynamically loaded third-party libraries, JIT-compiled code) or situations where the runtime behavior significantly deviates from the training profile? Does TRRIP not risk pessimizing the cache for hot code in these common scenarios?

4. Given that your simulation is trace-based and does not model wrong-path execution, how can you be confident in your results? A key function of a replacement policy is to mitigate cache pollution from sources like overly aggressive or inaccurate prefetching, which predominantly occurs on wrong paths. Please justify why this omission does not invalidate your conclusions regarding MPKI reduction and speedup.

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents TRRIP, a software-hardware co-design for improving instruction cache performance on mobile platforms. The core contribution is a pragmatic, end-to-end system that leverages existing Profile-Guided Optimization (PGO) infrastructure to classify code into "temperature" tiers (hot, warm, cold). This temperature information is then passed from the compiler to the hardware through the operating system, using existing, implementation-defined bits in the Page Table Entries (PTEs). The hardware cache controller uses this simple hint to modify the baseline RRIP replacement policy, giving strong priority to hot instruction lines to prevent their premature eviction. The authors demonstrate that this lightweight approach yields a 3.9% geomean speedup by reducing L2 instruction MPKI by 26.5% on PGO-optimized mobile proxy benchmarks, with negligible power and area overhead.

Strengths

The true strength of this paper lies in its elegant integration of existing, mature technologies into a novel and highly practical system. It stands as an excellent example of a co-design that respects the constraints of real-world product development.

1. Pragmatism and Adoptability: The authors' primary design pillar—"No additions/modifications to ISA"—is a crucial one. Many academic proposals in this space fail to gain traction because they require fundamental, costly changes. By instead utilizing existing architectural features like the implementation-defined bits in ARM PTEs (as mentioned in Section 3.3, page 6), TRRIP presents a solution with a remarkably low barrier to adoption. This focus on practicality is the paper's most significant contribution.

2. Clear and Compelling Motivation: The background analysis in Section 2 is thorough and effectively builds the case for TRRIP. The authors don't just state that frontend stalls are a problem; they demonstrate it with data on real mobile system software (Figure 1). Crucially, the analysis in Section 2.4 and Figure 3 isolates the core issue: even after PGO, frequently executed hot code suffers from high re-reference intervals, making it vulnerable to eviction. This insight provides a sharp, focused target for their solution.

3. Elegant System-Level Integration: The proposed flow, beautifully illustrated in Figure 4 (page 5), connects disparate parts of the system stack—the compiler, the object file format, the OS loader, and the microarchitecture—into a cohesive whole. The use of the OS and page tables as the conduit for compiler-derived information is a clever and efficient communication mechanism. This is co-design done right: not just proposing a new hardware feature in isolation, but architecting the flow of information across the entire system.

4. Strong and Relevant Evaluation: The authors compare TRRIP against a strong suite of modern replacement policies, including CLIP, SHiP, and Emissary. This demonstrates a solid understanding of the state-of-the-art. By outperforming these more complex, purely hardware-based solutions on average, the paper makes a compelling case for its simpler, co-designed approach.

Weaknesses

The paper's weaknesses are less about fundamental flaws and more about the inherent trade-offs of its chosen approach. Exploring these boundaries would strengthen the work.

1. The "Coverage" Limitation: The most significant limitation of TRRIP is that its benefits are confined to code that has been compiled through its specific PGO-enabled toolchain. As the authors themselves astutely analyze in Section 4.6 ("Coverage of Costly Instruction Misses"), costly misses in third-party libraries, dynamically-linked system code, or JIT-compiled code will not be covered. While PGO is widely used for first-party system components, a modern mobile environment is a heterogeneous ecosystem. This static, ahead-of-time dependency is the Achilles' heel of the approach when compared to purely dynamic hardware solutions like Emissary, which can react to any code being executed.

2. Static Profiles vs. Dynamic Behavior: PGO provides a static snapshot of program behavior based on a specific set of training inputs. The paper acknowledges performance degradation can occur due to profile mismatch (footnote 1, page 3) but does not fully explore the system's robustness. How gracefully does TRRIP handle significant application phase changes or workloads that deviate substantially from the profiling runs? Its static hints could become counter-productive in such scenarios.

3. Interaction with Other Frontend Mechanisms: The paper reasonably claims that instruction prefetching is an orthogonal technique. However, the interaction is likely more complex. An aggressive hardware prefetcher could be a primary source of cache pollution that evicts the very "hot" lines TRRIP is trying to protect. A deeper analysis of the interplay between TRRIP's priority scheme and prefetcher-induced cache pressure would provide a more complete picture of its system-level impact.

Questions to Address In Rebuttal

1. Quantifying the Coverage Gap: Regarding the "coverage" limitation, could the authors provide an estimate of what percentage of total execution cycles in a typical, interactive mobile usage scenario (e.g., app launch, web browsing) is spent in code that would not be visible to the TRRIP toolchain (e.g., third-party SDKs, JIT code from web engines)? This would help contextualize the real-world impact of the approach.

2. Robustness to Profile Mismatch: Have the authors performed experiments to measure the sensitivity of TRRIP to profile quality? For instance, evaluating the benchmarks using a profile generated from a completely different input set could reveal how TRRIP performs under sub-optimal conditions and whether it risks significant performance degradation compared to a baseline RRIP.

3. Synergy with Prefetching: Instead of being merely orthogonal, could TRRIP's temperature hints actively improve prefetching? For example, could the hardware use the "hot" hint to protect a cache set from polluting prefetches, or perhaps use the "cold" hint to identify regions where prefetching should be more aggressive?

4. Generalizability of "Temperature": The conclusion suggests applying the TRRIP philosophy to other structures like the BTB and TLB. What would be the analogous PGO-derived metric for "temperature" for these structures? Is it simply execution frequency, or would a more nuanced metric (e.g., branch misprediction rate for BTB entries, TLB miss rate for pages) be required?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

The paper proposes TRRIP, a software-hardware co-design for instruction cache replacement. The core idea is to leverage Profile-Guided Optimization (PGO) in the compiler to classify code into "temperature" categories (hot, warm, cold). This temperature information is then communicated to the hardware via implementation-defined bits in the page table entries (PTEs), a mechanism supported by modern architectures like ARM. The hardware cache replacement policy, based on RRIP, uses these temperature hints to prioritize hot instruction lines, inserting them with a higher priority (Immediate re-reference) and demoting them more slowly, aiming to reduce frontend stalls from instruction cache misses. The authors claim a 3.9% geomean speedup and a 26.5% reduction in L2 instruction MPKI over a baseline RRIP policy on PGO-optimized mobile workloads.

---

Strengths

1. Practicality of the Communication Mechanism: The most significant aspect of this work is its focus on a practical, non-intrusive communication path between software and hardware. By leveraging existing, implementation-defined page table attribute bits (e.g., ARM's PBHA), the authors sidestep the need for ISA extensions, which is a notoriously high barrier to adoption for co-designed techniques. This makes the proposal more plausible for real-world implementation than many of its predecessors.

2. Low Implementation Overhead: The proposed hardware modification is minimal, essentially adding a small amount of conditional logic to an existing RRIP controller based on the temperature hint (Algorithm 1, page 6). As shown in Table 4 (page 9), the area and power overheads are negligible. The software complexity is also low, as it builds upon existing PGO infrastructure.

3. End-to-End System Integration: The paper presents a complete, vertically integrated solution spanning the compiler, OS, and microarchitecture. The flow from PGO analysis to ELF sectioning to loader/OS page table population to hardware action is clearly articulated (Figure 4, page 5).

---

Weaknesses

1. Limited Conceptual Novelty: The core concept of using compiler-generated hints to guide cache replacement is not new. The work is an evolutionary step that combines known concepts into a new, practical package. The primary novel element is not the idea itself, but the specific implementation path.

   • Prior Art: The most relevant prior work is Ripple [39], which also uses PGO to guide instruction cache replacement in data centers. Ripple's goal is nearly identical to TRRIP's: use offline profiles to identify and protect important instruction cache lines. The key difference—and TRRIP's main contribution over Ripple—is the communication mechanism. Ripple proposes new ISA instructions (crm.set, crm.unset), whereas TRRIP uses PTE bits. While this difference is critical for practicality, it means the fundamental concept of "PGO-guided I-cache replacement" has been previously established.

2. Granularity of Hints: The proposed mechanism provides hints at the granularity of an OS page (typically 4KB or 16KB). As the authors acknowledge in Section 4.9 (page 11), a single page can contain code of mixed temperatures, especially as page sizes increase. This can lead to coarse-grained, potentially inaccurate prioritization, where cold code on a "hot" page is prioritized, or hot code on a "warm" page is not. While the paper suggests mitigation strategies, this appears to be a fundamental limitation of the chosen communication mechanism compared to more fine-grained, instruction-based hints.

3. Static Nature of Hints: The approach is entirely dependent on a static, offline PGO profile. It cannot adapt to dynamic phase changes in application behavior where the "hot" code paths might shift. In such scenarios, the static hints could become stale and potentially degrade performance by protecting code that is no longer critical. Purely hardware-based adaptive schemes, such as Emissary [45] (which tracks frontend stalls at runtime), do not suffer from this limitation, though they come with their own hardware overheads. The novelty of TRRIP's approach does not address this long-standing issue with static optimization.

---

Questions to Address In Rebuttal

1. Differentiation from Ripple [39]: The authors should more directly and explicitly contrast their work with Ripple. Beyond the acknowledged difference in the software-hardware interface (PTE bits vs. ISA extension), are there any other fundamental, conceptual, or algorithmic differences in how the PGO data is used to inform the replacement policy? The contribution would be stronger if it were framed as a more practical and lightweight implementation of the principle established by Ripple, rather than a wholly new concept.

2. Impact of Page-Level Granularity: The analysis in Section 4.9 (page 11) and Table 5 shows the number of pages used but does not quantify the performance impact of mixed-temperature pages. Can the authors provide data on how much of the hot code resides on pages that are not marked as hot? What is the performance loss when a truly hot cache line resides on a page that is classified as warm or cold due to the surrounding code? This is key to understanding the trade-off made for the practical communication mechanism.

3. Dynamic Behavior and Stale Profiles: How does TRRIP's performance hold up when the execution profile deviates significantly from the training profile used for PGO? A sensitivity study showing performance against varying inputs would help quantify the robustness of this static approach. How does it compare to a purely dynamic hardware scheme like Emissary [45] under such workload shifts?

4. Interaction with Code Layout Optimizations: PGO is already used for hot-cold splitting and basic block reordering, which primarily improve spatial locality to reduce I-cache misses. TRRIP aims to improve temporal locality. Is the 3.9% speedup measured on top of a baseline that already includes aggressive PGO-based code layout? Could the authors clarify if there is a synergistic or potentially overlapping effect between these optimizations?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose NetZIP, an algorithm/hardware co-design for in-network lossless compression to accelerate distributed large model training. The approach consists of two parts: NetZIP-algorithm, which applies bit/byte grouping and delta-value transformations to make gradients and activations more compressible, and NetZIP-accelerator, a proposed bump-in-the-wire hardware block within a NIC to perform these operations with low latency. The central claim is that this co-design achieves superior compression ratios compared to standard lossless algorithms and results in a 35% reduction in total training time by mitigating communication bottlenecks. However, the evaluation's heavy reliance on simulation for its primary system-level claims, combined with a hardware prototype that is an emulation rather than an integrated system, raises significant questions about the validity and practical achievability of the reported end-to-end performance gains.

Strengths

1. Problem Motivation: The paper correctly identifies a critical bottleneck in distributed training and provides a clear motivation for exploring lossless compression, detailing the shortcomings of both lossy approaches (convergence issues) and standard lossless methods on commodity hardware (high latency overhead).
2. Data Characterization: The analysis of the bfloat16 representation of gradients and activations in Section 5.1 (page 6, Figure 5) is sound. Identifying the low entropy in exponent bits versus the high entropy in mantissa bits provides a solid, data-driven foundation for the proposed byte- and bit-grouping techniques.
3. Baseline Evaluation: The experimental results in Section 4, which quantify the performance of standard lossless algorithms (LZ4, Zstd, etc.) on commodity CPU, GPU, and SNIC platforms, are thorough. Table 3 (page 6) effectively establishes a crucial baseline, demonstrating that a naive application of these algorithms increases, rather than decreases, total communication latency. This provides strong justification for the need for a specialized solution.

Weaknesses

1. Evaluation Methodology Relies on Unvalidated Simulation: The headline claim of a 35% reduction in training time is not derived from a real-world, at-scale hardware deployment. Instead, it is the output of the SimAI simulator (Section 6.3, page 12). The authors provide no evidence validating SimAI's accuracy against a physical cluster for this specific workload. More critically, the simulator is fed latency values obtained from a separate FPGA emulation (Section 6.1, page 9). This creates a fragile, multi-layered abstraction from reality, making the final end-to-end numbers speculative rather than empirically proven.
2. Hardware Claims are Based on Emulation and Projection, Not Integration: The "NetZIP-accelerator" as tested is not an integrated NIC ASIC. It is an FPGA connected externally to a standard NIC (Figure 9, page 9). This "bump-in-the-wire" setup is a proof-of-concept at best and fails to capture the realities of on-chip resource contention, memory bandwidth, and power envelopes within a real NIC ASIC. Furthermore, the claims regarding ASIC area and power in Section 5.2 (page 9) are merely projections based on a methodology from 2010, which is insufficient evidence for a hardware-centric claim in 2025.
3. Limited Applicability of Delta Compression: The evaluation is conducted on fine-tuning tasks using the Alpaca dataset (Section 6.1, page 10). The core assumption of delta compression is that values change incrementally between iterations. While this may hold during fine-tuning, it is unlikely to be true during the volatile, early stages of pre-training a model from scratch, where gradients can experience large fluctuations. The paper makes a general claim about "distributed large model training" but only provides evidence for a specific, and arguably more favorable, sub-domain.
4. Insufficient Justification for the Delta Base Value Heuristic: The paper concedes that true delta compression is infeasible due to memory constraints and instead proposes using a single base value (the minimum value in a layer) for subtraction (Section 5.1, "Delta Value Compression", page 7). This is a critical simplification. There is no theoretical or empirical justification provided for why the minimum value is a suitable or optimal choice over other statistics like the mean, median, or a learned scalar. The impact of this heuristic on compression effectiveness across different model architectures and training phases is left unexplored.
5. Strawman Comparison to Lossy Compression: The comparison with lossy compression in Section 6.4 (Figure 14, page 12) is weak. The authors compare NetZIP only against top-K sparsification. This ignores a vast body of work on more sophisticated lossy techniques, such as adaptive quantization, error feedback (e.g., EF-SGD), and gradient-norm-based methods, which are known to achieve high compression ratios with minimal impact on convergence. To claim superiority, a comparison against the state-of-the-art in lossy compression is required, not a baseline method.

Questions to Address In Rebuttal

1. How have you validated the accuracy of the SimAI network simulation for this specific communication pattern against a physical testbed? Please provide data comparing simulated and real-world communication times on a smaller-scale cluster (e.g., 8-16 nodes) to justify its use for the 512-node extrapolation.
2. The hardware evaluation uses an external FPGA. How would the performance (latency, throughput) and resource costs (area, power) be affected by a true integration into a modern NIC ASIC, considering shared resources such as on-chip memory controllers and PCIe interfaces? Why should the projected ASIC figures be considered reliable?
3. Please provide compression ratio data for NetZIP's delta compression algorithm during the initial epochs of pre-training a large model from a random initialization. How does its effectiveness compare to the fine-tuning results presented?
4. Please provide an ablation study justifying the choice of the layer's minimum value as the base for delta compression. How does this heuristic compare, in terms of compression ratio, against using the layer's mean or median value?
5. The comparison to lossy compression is limited to top-K. Please provide a time-to-accuracy comparison between NetZIP and a state-of-the-art lossy compression algorithm (e.g., QSGD with error feedback) to demonstrate the practical advantage of your lossless approach.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents NetZIP, an algorithm/hardware co-design for in-network lossless compression to accelerate distributed large model training. The authors identify a critical gap: lossy compression techniques can harm model convergence, especially for activations, while existing lossless compression methods are too slow on current platforms (CPU/GPU/SNIC) to overcome their own latency overhead.

The core contribution is a two-part solution. First, NetZIP-algorithm is a data transformation technique that analyzes and restructures gradients and activations at the bit and value level (via byte/bit grouping and delta compression) to make them significantly more compressible by standard algorithms. Second, NETZIP-accelerator is a lightweight, "bump-in-the-wire" hardware accelerator integrated into a NIC that implements this transformation alongside a simple, fast compressor like LZ4. This co-design approach aims to deliver both high compression ratios and extremely low latency, thereby reducing the end-to-end communication time that dominates training. The authors demonstrate through comprehensive experiments and large-scale simulation that NetZIP can reduce total training time by up to 35% for models like Llama-3 and GPT-3.

Strengths

1. Excellent Problem Scoping and Motivation: The paper does a superb job of positioning its contribution. The authors clearly establish the limitations of the two dominant alternatives. In Section 3 (pages 3-4), they show that lossy compression, while reducing per-iteration time, can increase total training time due to accuracy degradation. Then, in Section 4 (pages 5-6), they convincingly demonstrate that off-the-shelf lossless compression on existing hardware platforms actually increases communication time. This framing creates a clear and compelling need for the proposed solution.

2. Insightful Data-Driven Algorithm Design: The strength of NetZIP-algorithm lies in its foundation of careful data analysis. The insights from Figure 5 (page 6)—that exponent bits are structured while mantissa bits are random, and that value distributions are narrow—directly motivate the byte/bit grouping strategy. Similarly, the intuition that values change slowly between iterations motivates the delta compression scheme, which is validated in Figure 7 (page 7). This is not a brute-force approach; it's an elegant solution derived from understanding the unique properties of the target data.

3. Strong Systems-Level Co-design Thinking: This work is a prime example of successful algorithm/hardware co-design. The authors recognize that heavy algorithms like Zstd/Deflate, while offering better compression, are too costly for a low-latency hardware implementation. By designing an algorithm that makes data highly compressible even for a simple method like LZ4, they enable a hardware design that is fast, efficient, and practical to integrate into a NIC ASIC. The design space exploration in Section 5.2 (page 8) and the subsequent accelerator architecture in Figure 8 are logical consequences of this co-design philosophy.

4. High Potential for Practical Impact: The work addresses a real, expensive, and worsening bottleneck in AI infrastructure. By focusing on a lossless approach, NetZIP sidesteps the complex and often unpredictable effects of lossy compression on model convergence. Its ability to compress activations is particularly significant, as this has been a major challenge for prior work. If integrated into future NICs, this technology could substantially reduce the cost and time of training large models, especially for users relying on public cloud infrastructure with limited network bandwidth, as motivated in Figure 3 (page 4).

Weaknesses

1. Limited Engagement with Modern Parallelism Strategies (FSDP): While the paper evaluates a standard DP/TP/PP setup, the related work section (Section 7, page 13) acknowledges but then sidesteps Fully Sharded Data Parallelism (FSDP). FSDP is now the dominant strategy for training very large models, and it fundamentally changes communication patterns from the AllReduce of gradients to ReduceScatter and AllGather of parameters. This will change the size, structure, and timing of the data chunks being sent over the network. The paper would be significantly stronger if it included an analysis, even speculative, of how NetZIP's data assumptions and performance benefits would translate to an FSDP environment.

2. Scope of Data Analysis (Fine-tuning vs. Pre-training): The experiments are based on collecting gradients and activations during fine-tuning (Section 6.1, page 10). During this phase, model weights and activations are expected to change incrementally, making the delta compression scheme particularly effective. However, during the initial stages of pre-training from scratch, gradients can be much larger and more chaotic. The core assumptions about small deltas may not hold as strongly, potentially reducing the effectiveness of the algorithm. A brief analysis of data from early-stage pre-training would help generalize the paper's claims.

3. Hardware Implementation Realism: The hardware evaluation relies on an FPGA-based prototype connected externally to a standard NIC, with performance plugged into a simulator. This is a common and reasonable methodology. However, the true challenge lies in the tight integration of such a "bump-in-the-wire" accelerator into a real, high-performance NIC data path, especially one supporting RDMA. A deeper discussion of the practical challenges—such as managing buffer pressure when compression ratios vary, handling packet reordering, and interacting with the NIC's transport-level logic without adding significant latency—would add valuable context to the ASIC projection claims (Section 5.2, page 9).

Questions to Address In Rebuttal

1. Regarding FSDP: Could the authors speculate on how the communication patterns of FSDP (specifically, the AllGather of sharded parameters) would affect the performance of NetZIP? Would the data chunks still exhibit the same compressibility properties observed in the paper's AllReduce-centric evaluation?

2. Regarding Pre-training: The paper's analysis is based on fine-tuning. Have the authors examined data from early-stage pre-training? How does the effectiveness of the delta compression scheme, in particular, change when gradients and activations are more volatile?

3. Regarding Hardware Integration: Could the authors elaborate on the potential challenges of integrating the NETZIP accelerator directly into a modern NIC's data path alongside an RDMA engine? Specifically, how would the system handle situations where the compressed payload size exceeds the original packet buffer, and how is flow control managed between the DMA engine and the compressor?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present NetZIP, a hardware/software co-design aimed at reducing communication latency in distributed large model training through in-network lossless compression. The core proposal consists of two parts: (1) NetZIP-algorithm, a set of pre-processing techniques (byte/bit grouping and delta compression) designed to transform gradient and activation data to be more amenable to standard lossless compression; and (2) NetZIP-accelerator, a "bump-in-the-wire" hardware architecture integrated into a NIC that implements these algorithms and a lightweight compressor (LZ4) to minimize overhead.

My analysis concludes that while the constituent algorithmic components have clear and strong precedents in prior art, their specific synthesis and application to the lossless compression of both gradients and activations for large model training, coupled with the proposed in-network hardware architecture, represents a novel system-level contribution. However, the paper overstates its algorithmic novelty and fails to properly contextualize its methods against existing, functionally analogous techniques.

Strengths

1. Novel Architectural Proposal: The "bump-in-the-wire" accelerator architecture (Section 5.2, page 8, Figure 8) is a key novel contribution. By placing the compression/decompression logic directly in the NIC's datapath between the DMA engine and protocol engines, the design compellingly addresses the overhead of data movement to/from host CPU, GPU, or even a PCIe-attached accelerator on a SmartNIC, which the authors correctly identify as a major performance bottleneck (Section 5.2, page 7). This architectural insight is well-argued and significant.

2. Application to Activations: The paper's focus on compressing not only gradients but also activations is a noteworthy distinction from the bulk of prior work in training communication reduction, which has myopically focused on gradients. The analysis showing that activations constitute a significant portion of communication traffic (21-49% in their models, Section 4, page 4) validates this focus and represents a novel problem framing.

3. Co-design Synergy: The primary strength of the work lies in the co-design itself. The choice of a lightweight algorithm (LZ4) is justified not on its standalone compression ratio (which is poor) but on its hardware efficiency, which enables a high-throughput, low-latency implementation. This efficiency is then leveraged by the algorithmic pre-processing, which specifically enhances the compressibility for LZ-style algorithms. This tight coupling is the essence of the proposed system's novelty.

Weaknesses

My singular focus is novelty, and on this front, the claimed algorithmic contributions are substantially weaker than presented.

1. Bit/Byte Grouping is Not New: The core idea of reorganizing data by grouping bits or bytes of similar entropy to improve compressibility is not novel. This technique is functionally analogous to the "Lane Compression" method proposed by Ko et al. [23], which the authors cite in their related work (Section 7, page 13) but do not compare against. Lane Compression also groups bit positions into different "lanes" based on entropy to aid subsequent compression. While the application domain differs (model parameters for inference vs. gradients/activations for training), the fundamental algorithmic principle is identical. The paper must acknowledge this prior art directly and significantly temper its claims of algorithmic novelty in this area.

2. Delta Compression is a Standard Technique: The proposed "Delta Value Compression" (Section 5.1, page 7) is a straightforward application of delta encoding, one of the oldest and most fundamental techniques in data compression. It is used ubiquitously in version control, video codecs, and data backup systems. While its application to gradients in training is logical and effective, as shown by the authors' analysis, it does not represent a new algorithmic concept. The adaptation of using a minimum value per layer as a base instead of the previous iteration's full tensor is a practical engineering choice to manage memory, not a fundamental innovation.

3. Overstated Algorithmic Contribution: Due to the points above, the paper's narrative of proposing new algorithms is misleading. The novelty is not in the invention of these techniques, but in their application and hardware co-design. The contribution would be stronger if it were framed as "a novel system architecture that effectively applies and accelerates known data transformation techniques for in-network training communication," rather than implying the creation of new compression algorithms.

Questions to Address In Rebuttal

The authors should use the rebuttal to clarify the precise boundaries of their novel contributions.

1. Please explicitly differentiate the proposed "bit/byte grouping" from the "Lane Compression" method in [23]. Beyond the difference in application domain (training vs. inference), what is the fundamental algorithmic distinction in how data is transformed to improve compression? Why was this highly relevant work not discussed in the main body of the paper (e.g., in Section 5.1)?

2. Given that delta compression is a well-established technique, could the authors refine their claim regarding the novelty of "Delta Value Compression"? Is the novelty simply its application in this context, or is there a more subtle algorithmic innovation that I have missed?

3. The paper notes that FSDP is a key distributed training paradigm that it could not evaluate (Section 7, page 13). FSDP fundamentally changes communication by sharding parameters and optimizer states, potentially altering the iterative similarity of communicated data. How do the authors expect the effectiveness of their delta compression scheme to be impacted in an FSDP context, where a given worker may not see the same tensor slice in consecutive iterations? Does this limit the novelty of the approach to specific parallelism strategies (DP/TP/PP)?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present a characterization study of distributed Large Language Model (LLM) training. They evaluate performance, power, and thermal behavior across three modern GPU platforms (NVIDIA H100/H200, AMD MI250) using several dense and sparse models. The study examines the effects of different parallelism strategies (Tensor, Pipeline, Data, Expert) and common software optimizations like activation recomputation and compute-communication overlap. The authors conclude with several insights, including the nuanced trade-offs between scale-up and scale-out systems, inefficiencies in hybrid parallelism schemes, the limits of microbatch scaling, and the performance impact of thermal imbalances. Finally, they attempt to project their findings to datacenter-scale systems via simulation.

Strengths

1. Experimental Testbed: The study is conducted on an impressive and relevant set of modern hardware (H200, H100, MI250). Access to such platforms for a systematic study is a notable strength.
2. Breadth of Experiments: The authors have undertaken a significant experimental effort, covering multiple models, parallelism configurations, and optimization techniques. This provides a broad, albeit thin, survey of the design space.
3. Inclusion of Thermal Analysis: The focus on thermal imbalance (Section 6) and its concrete impact on clock throttling (Figure 17) is a valuable contribution. It highlights a practical, physical-layer constraint often overlooked in purely algorithmic performance studies.

Weaknesses

The paper's ambition is commendable, but its execution lacks the necessary rigor, leading to shallow analyses and insufficiently substantiated claims.

1. Insufficient Methodological Detail: The foundation of any characterization study is its measurement methodology, which is inadequately described here.

   • The authors state they use a "modified version of Zeus" (Page 4, Section 3.1) for energy and telemetry. The nature and impact of these modifications are not specified. What is the measurement overhead? How was the modified tool validated against ground truth? Without this information, the fidelity of all power, thermal, and utilization data is questionable.
   • For the AMD MI250 platform, the paper states that "smaller versions of GPT-3 and Llama-3" were used due to memory constraints (Page 5, Section 3.2). The process of scaling down these models is not detailed. Architectural changes during scaling (e.g., number of layers vs. hidden dimension) can drastically alter the compute-to-communication ratio, making the cross-platform comparison to the full-size models on NVIDIA hardware potentially invalid. The claim of providing "valuable cross-platform insights" is therefore weakened.

2. Superficial Analysis and Overstated Insights: The paper identifies several well-known phenomena but fails to provide deep, quantitative explanations for their root causes.

   • Scale-up vs. Scale-out (Section 4.1): The conclusion that the optimal strategy "depends on model size, sparsity, and parallelism strategy" is not a novel insight. The analysis attributes performance differences to "communication locality" and "inter-node traffic" (Page 5), but fails to provide a quantitative breakdown. For instance, in the GPT3-175B case where H200 excels, what precise percentage of the performance gain is due to avoiding inter-node AllReduce versus exploiting higher intra-node NVLink bandwidth for Tensor Parallelism? The kernel breakdowns in Figure 3 are a start, but the narrative connecting them to the high-level claims is tenuous.
   • Limits of Microbatch Scaling (Section 5): The paper correctly observes that increasing microbatch size can harm performance but attributes this to vague causes like "communication bandwidth saturation" and "pipeline-induced stalls" (Page 10, Insight box). Which specific communication fabric is saturating (PCIe, InfiniBand)? Figure 15 shows that AllReduce and SendRecv time increases, but provides no evidence of why. Is this due to increased message size leading to network congestion, or is it a tail-latency effect from stragglers? The analysis stops short of identifying the true bottleneck.

3. Unsupported Extrapolation and Speculation:

   • The projection to 8K-GPU systems in Section 7.1 is the paper's most significant flaw. The authors switch from empirical measurement on at most 64 GPUs to simulation with Astra-Sim. The paper provides zero detail on how the simulator was parameterized or calibrated against their real-system measurements. Network simulators are notoriously difficult to configure accurately, and without a rigorous calibration methodology, the results presented in Figure 22 are purely speculative and cannot be considered a valid extension of the paper's empirical findings. This section undermines the work's grounding in real-world characterization.

4. Inclusion of Underdeveloped and Contradictory Results:

   • The "thermal-aware pipeline parallelism strategy" presented at the end of Section 6 and in Figure 21 is a premature and distracting inclusion. It is presented as a solution, yet the results are mixed: a meager 4% efficiency gain for Llama3 is contrasted with a 7% efficiency degradation for GPT3-175B. The paper glosses over this negative result. Such a preliminary and inconclusive experiment does not belong in a characterization paper and weakens its focus and credibility.

Questions to Address In Rebuttal

The authors must provide precise, data-driven answers to the following questions:

1. Regarding your "modified version of Zeus" (Page 4): What specific modifications were made? Provide data on the validation of this tool and quantify its measurement overhead on the system.
2. Regarding the scaled-down models for the MI250 (Page 5): Detail the exact methodology used to scale down GPT-3 and Llama-3. Provide evidence that these smaller variants maintain the same fundamental bottleneck characteristics (e.g., compute-bound vs. memory-bound, communication patterns) as their full-sized counterparts, thereby justifying the cross-platform comparison.
3. Regarding the claim of "communication bandwidth saturation" with larger microbatches (Page 10): Provide specific data from your telemetry (e.g., PCIe bus utilization, InfiniBand NIC throughput) that directly demonstrates saturation of a specific hardware resource. Correlate this saturation point with the observed performance degradation.
4. Regarding the thermal-aware scheduling experiment (Page 12, Figure 21): Explain the 7% performance degradation observed for GPT3-175B. Given this negative result, what is the justification for including this experiment as a positive contribution?
5. Regarding the 8K GPU extrapolation (Page 12, Section 7.1): Provide a complete and detailed account of the calibration process for Astra-Sim. How were kernel latencies, network parameters, and collective communication models from your 32/64-GPU empirical measurements translated to the simulator to ensure the validity of the 8K-GPU projections?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a comprehensive, multi-faceted characterization of distributed training for large language models (LLMs). The authors move beyond traditional performance metrics (e.g., throughput) to provide a holistic analysis that incorporates power consumption, hardware utilization, and thermal behavior. By conducting experiments on a diverse set of modern hardware platforms (NVIDIA H100/H200, AMD MI250) and across various models and parallelism strategies (TP, PP, DP, EP), the authors empirically investigate the complex, second-order effects that arise at scale.

The core contribution is the assertion, backed by extensive data, that optimizing large-scale training requires a "full-stack" perspective. The paper demonstrates that software-level decisions—such as the choice of parallelism strategy, microbatch size, and optimizations like activation recomputation—have profound and often non-intuitive interactions with the physical realities of the underlying hardware, including network topology, power limits, and thermal dissipation. Key findings include the nuanced trade-offs between scale-up and scale-out systems, the communication inefficiencies of certain parallelism combinations (TP+PP), the performance pitfalls of excessive microbatch scaling, and the significant impact of thermal imbalance on system reliability and throughput.

Strengths

This is an excellent and timely study that provides a much-needed bridge between the domains of ML systems software and computer architecture/datacenter operations. Its primary strengths are:

1. Holistic and Grounded Perspective: The single most important strength of this paper is its commitment to analyzing the entire stack. While many papers focus on optimizing parallelism strategies in the abstract (e.g., Megatron-LM, DeepSpeed), and others focus on datacenter efficiency, this work is one of the first to rigorously connect the two. It moves the conversation from idealized performance models to the messy, physical realities of running these workloads, which is where the next set of bottlenecks lies. The focus on thermal throttling and power draw (Sections 5 & 6, pages 9-11) is particularly novel and significant for the training domain.

2. Methodological Rigor and Relevance: The experimental setup is state-of-the-art and highly relevant. The use of H100, H200, and MI250 GPUs covers the most important accelerator architectures in the market today. The choice of workloads, from dense models like GPT-3 and Llama3 to sparse MoE models like Mixtral, ensures the findings are broadly applicable. The fine-grained telemetry collection provides a solid empirical foundation for the paper's claims.

3. Revealing Non-Obvious Interactions: The paper excels at uncovering insights that challenge conventional wisdom. For example:

   • The finding that higher-memory "scale-up" systems (32xH200) can outperform "scale-out" systems (64xH100) in communication-heavy regimes (Section 4.1, page 5) highlights that raw aggregate compute is not the only factor in performance.
   • The demonstration that increasing microbatch size can harm performance due to communication saturation and thermal stress (Section 5, page 9, Figure 13) provides a crucial, practical guideline that contradicts the simplistic "bigger is better" assumption.
   • The clear visualization of thermal imbalance due to physical node layout (Section 6, page 10, Figure 17) and its direct link to performance throttling is a powerful demonstration of how physical constraints impact distributed algorithms.

4. Connecting to Broader Research Agendas: This work provides foundational data that can inform multiple research communities. It implicitly challenges automated parallelism frameworks (e.g., Alpa) to incorporate physical constraints into their cost models. It provides concrete motivation for work on topology-aware communication collectives (e.g., TACCL, TopoOpt). Finally, it extends the investigation of power- and thermal-aware scheduling, previously explored for inference (e.g., TAPAS, DynamoLLM), into the synchronous, long-running, and highly-coupled domain of LLM training.

Weaknesses

The weaknesses of the paper are primarily related to its framing and the scope of its conclusions, rather than fundamental flaws in the methodology or results.

1. Characterization vs. Solution: The paper is, at its heart, a characterization study. It does an excellent job of identifying and quantifying problems. While the brief exploration of thermal-aware pipeline scheduling is a step in the right direction (Section 7, page 12), it feels preliminary compared to the depth of the problem analysis. The paper would be strengthened if the authors were more explicit in framing their work as foundational characterization that motivates the need for new solutions, rather than presenting a complete solution itself.

2. Generalizability of Topology-Specific Findings: The results are derived from specific cluster topologies (e.g., NVIDIA HGX nodes connected via InfiniBand). While these are common, the authors could do more to discuss how their findings might change in systems with different network topologies (e.g., Dragonfly, optical circuit switches) or cooling systems (e.g., direct liquid cooling). For instance, the severe thermal imbalance shown in Figure 17 might be less pronounced in a liquid-cooled system, which would alter the trade-off calculus for different parallelism strategies.

3. Depth of Causal Analysis: The paper establishes strong correlations between software choices and physical effects (e.g., PP-heavy configurations lead to higher peak power). However, a deeper microarchitectural analysis explaining why these patterns emerge would be beneficial. For example, what specific resource contention (e.g., on memory controllers, PCIe bus) during compute-communication overlap leads to the observed thermal stress? While likely outside the primary scope, adding some discussion here would further solidify the paper's contribution.

Questions to Address In Rebuttal

1. The thermal-aware pipeline stage placement experiment (Section 7, page 12) is very promising. Could you elaborate on the limitations of this approach? For instance, how does the strategy adapt if the "hot" and "cold" GPUs change during a long training run, and how much of the performance gain is tied to the specific model architecture (e.g., number of layers being divisible by the number of stages)?

2. Your work compellingly demonstrates the importance of physical node layout and network topology. How do you foresee your key insights—particularly regarding scale-up vs. scale-out and TP+PP inefficiencies—translating to future disaggregated or chiplet-based systems where memory, compute, and networking resources may be composed in more flexible ways?

3. The industry is increasingly moving towards advanced cooling solutions like direct liquid cooling to manage the thermal density of modern accelerators. How would such a technology alter the conclusions of your study? Would it eliminate the thermal bottleneck entirely, or would it simply shift the bottleneck to another system component (e.g., power delivery, interconnect bandwidth)?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present a comprehensive characterization study of distributed Large Language Model (LLM) training. The work's central thesis is that traditional performance-centric analysis is insufficient, and a holistic perspective incorporating power consumption and thermal behavior is necessary to understand system efficiency. The authors evaluate various parallelism strategies (Tensor, Pipeline, Data, Expert) and common optimizations (activation recomputation, compute-communication overlap) across modern hardware platforms (NVIDIA H100/H200, AMD MI250).

The paper does not propose a new algorithm, hardware architecture, or training technique. Its claim to novelty rests entirely on the insights derived from its multi-faceted characterization. Specifically, it claims to be the first to systematically and quantitatively link specific software-level parallelism and optimization choices to their physical, second-order consequences, such as thermal imbalance, power excursions, and clock throttling, in the context of state-of-the-art LLM training systems.

Strengths

The primary contribution of this work is the rigorous documentation of system behaviors that, while perhaps suspected by practitioners, have not been systematically studied and published in an academic venue. The novelty is not in the invention of a new method, but in the elucidation of previously unquantified, non-obvious system interactions.

Specific novel insights include:

1. The Counter-Intuitive Limits of Microbatch Scaling: While conventional wisdom suggests larger microbatches are better if memory allows, this paper provides concrete evidence that beyond an optimal point, they create bursty execution patterns that lead to higher peak power and worsened thermal throttling, ultimately degrading performance (Section 5, page 9, Figure 13). This is a significant finding that challenges common heuristics.

2. Quantification of Thermal Imbalance Impact: The paper moves beyond acknowledging thermal issues to showing a direct causal link between server airflow design (Figure 16, page 10), GPU placement, the chosen parallelism strategy (high-PP configurations), and persistent clock throttling on specific GPUs (Figure 17, page 11). This demonstrates that applying uniform software optimizations to physically non-uniform hardware is a flawed strategy.

3. Inefficiency of Combined Parallelism Strategies: The analysis revealing that the combination of Tensor Parallelism and Pipeline Parallelism (TP+PP) leads to underutilization of PCIe bandwidth due to sparse, uncoordinated SendRecv calls is a specific and novel finding (Section 4.2, page 7). This identifies a concrete inefficiency in current software frameworks that was not previously highlighted.

Weaknesses

My main critique stems from the definition of novelty. This is fundamentally a characterization study, an established research methodology. The work synthesizes existing measurement tools and techniques to analyze existing software on existing hardware.

1. Lack of a Constructive Contribution: The paper is diagnostic, not prescriptive. It identifies and quantifies numerous problems but stops short of proposing and evaluating a novel mechanism to solve them. For example, after demonstrating the negative impact of thermal imbalance, the authors offer recommendations but do not implement a novel thermal-aware scheduler. The brief experiment on thermal-aware placement in the discussion (Section 7, page 12) is a simple heuristic (placing heavy stages on cold GPUs) and is presented more as a proof-of-concept than a fully-fledged novel algorithm. The core of the paper remains observational.

2. Conceptual Overlap with Prior Art: While the authors focus on LLM training, the core concept of co-designing software with an awareness of power and thermal constraints is not new. Prior work in HPC and datacenter management has long explored thermal-aware job scheduling. Google's technical blog post, "Balance of power: A full-stack approach to power and thermal fluctuations in ML infrastructure" [16], which the authors cite, discusses this exact problem space at a high level. This paper's contribution is the granular, quantitative analysis for specific LLM parallelism strategies, which makes it an incremental, albeit valuable, advancement over the conceptual state-of-the-art rather than a paradigm shift.

3. Dependence on Established Optimizations: The techniques analyzed—activation recomputation [28], compute-communication overlap [75], FSDP [82]—are all well-established. The paper's contribution is to map their secondary effects, not to introduce a new optimization.

In essence, the paper provides a new, high-resolution map of a known territory. The map is useful and reveals details not seen before, but it is not the discovery of a new continent.

Questions to Address In Rebuttal

1. The paper's core novelty lies in its insights. Could the authors consolidate their findings and explicitly state which of their documented system interactions they believe were truly unknown to the community (both academic and industrial) prior to this work, versus those that were "common knowledge" but previously unquantified?

2. The work expertly diagnoses the sub-optimality of applying uniform optimizations to thermally heterogeneous hardware. The lack of a proposed novel mechanism to address this is a significant limitation. Can the authors justify why they chose not to propose and evaluate a new scheduling algorithm based on their findings? Does the brief thermal-aware placement experiment (page 12) contain a novel algorithmic component that was not fully elaborated upon?

3. How does this work's contribution differ fundamentally from the full-stack power/thermal analysis described in prior industrial reports, such as Google's blog post [16]? Please precisely articulate the delta beyond being a more detailed, academic study. Is the novelty simply the level of granularity, or is there a conceptual advance?

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

---

Summary

The authors propose SkipReduce, a technique to accelerate distributed training by deterministically omitting entire communication steps within the Ring AllReduce algorithm's Reduce-Scatter phase. The central idea is that by skipping steps, the communication latency is proportionally reduced. To mitigate potential accuracy degradation from biased gradient skipping, the authors introduce "Random SkipReduce," which varies the skipped gradient slices across iterations, and "Selective SkipReduce," which avoids skipping gradients from pre-determined "important" layers. The authors claim that SkipReduce provides up to a 1.58x speedup in time-to-accuracy over baseline AllReduce and other communication reduction techniques on an 8-GPU system.

Strengths

1. Simplicity and Low Overhead: The core mechanism of skipping communication steps is conceptually simple and, as implemented, introduces negligible computational overhead compared to compression-based methods like top-k sparsification or low-rank approximation, which require significant pre/post-processing.
2. Direct Integration: The proposed method is implemented directly within NCCL, demonstrating a plausible path for integration into existing deep learning frameworks without requiring major architectural changes.
3. Problem Formulation: The paper correctly identifies the growing problem of communication overhead in distributed training and explores a direction—coarse-grained communication omission—that is distinct from fine-grained compression.

Weaknesses

The paper's claims rest on a foundation that appears methodologically fragile and insufficiently validated. My primary concerns are as follows:

1. Fundamentally Flawed Baseline Comparison: The central performance claims are benchmarked against a fundamentally flawed and non-functional baseline, "Ideal Top-k." In Section 5.1 (page 9), the authors explicitly state, "we implement an idealized version of top-k, in which the indices and values of the selected gradients are communicated as if they were dense tensors... this implementation is not functionally correct as it does not support actual sparse reductions." This is not a valid scientific comparison. It compares a real, working implementation (SkipReduce) against a hypothetical best-case scenario for a competitor that ignores the very real-world overheads (e.g., COO format conversion, irregular memory access, sparse reduction kernels) that make top-k methods challenging. The reported speedups in Figure 11 and Figure 12 are therefore highly misleading, as they do not reflect a comparison against a viable, state-of-the-art alternative.

2. Fragile Heuristic for Selective SkipReduce: The concept of "Selective SkipReduce" relies on a fragile and poorly justified heuristic. In Section 4.4 (page 8), "important gradients" are identified as those in the global top-25% by magnitude, sampled at the end of the first epoch. The gradient distribution of a model is known to be highly dynamic, shifting significantly from early-stage training to convergence (a point the authors themselves make in Figure 2). A decision based on a single snapshot after the first epoch is unlikely to remain optimal for the entire duration of training. No evidence is provided to support the robustness of this one-time decision. This introduces an ad-hoc hyperparameter (the sampling point) and undermines the generality of the selective approach.

3. Insufficient Justification for "Skipping" over "Dropping": The authors differentiate between "skipping" (effectively zeroing out a gradient's contribution) and "dropping" (removing the gradient and adjusting the divisor for the average). In Figure 15 (page 10), they show "DropReduce" performs worse but offer a weak explanation: "by completely removing the gradients, the contribution of the remaining gradients is amplified, possibly creating bias." This is a hand-wavy argument. Adjusting the divisor is a mathematically sound way to compute a correct average over the non-dropped elements. The claim that zeroing is superior requires a more rigorous theoretical or empirical analysis, as it is non-obvious why introducing zeros (a specific value) is inherently better than re-normalizing.

4. Inadequate Scale of Evaluation: The empirical evaluation is conducted exclusively on an 8-GPU system (Section 5.1, page 8). While useful for initial validation, this scale is insufficient to substantiate claims about accelerating large-scale machine learning. The communication-to-computation ratio, and thus the potential benefit of techniques like SkipReduce, changes dramatically when scaling to hundreds or thousands of workers. The "analytical evaluation" for other topologies in Section 6.1 (page 10) is not a substitute for empirical results and offers no proof that the method's accuracy/performance trade-off holds at scale.

5. Post-Hoc Modifications to the Algorithm: The introduction of a "warm-up" period for Sharded Data Parallel (SDP) training (Section 6.3, page 12) appears to be an ad-hoc modification required to make the technique work in that context. This was not presented as a core component of the SkipReduce algorithm. Its necessity complicates the method, adds another hyperparameter (the warm-up duration), and raises questions about whether similar "fixes" are needed for other models or training regimes not tested in the paper.

Questions to Address In Rebuttal

1. Please justify the use of a non-functional "Ideal Top-k" baseline. To make a credible claim, the authors must either compare against a real, functional top-k implementation (e.g., from [25, 42]) or significantly temper their performance claims and acknowledge the comparison is only against a theoretical, unachievable lower bound.
2. Provide evidence that selecting "important" layers based on the gradient distribution after a single, initial epoch is a robust heuristic. For example, show how the set of important layers evolves throughout training and demonstrate that the initial choice remains valid.
3. Provide a more rigorous mathematical or empirical justification for why "skipping" (zeroing) gradients results in better accuracy than "dropping" (re-normalizing the average). The current explanation is insufficient.
4. How can the results from an 8-GPU experiment be extrapolated to large-scale systems where communication patterns and bottlenecks are vastly different? Please provide a more compelling argument for the scalability of SkipReduce's benefits.
5. Is the "warm-up" period for SDP a general requirement for SkipReduce when applied to weight communication, or a specific fix for the LLaMA model? How should a user determine the necessity and duration of such a warm-up?

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces "SkipReduce," a novel and pragmatic approach to accelerate collective communication in distributed deep learning. The core contribution is the idea of coarse-grained gradient skipping, implemented by intentionally omitting a fraction of the communication steps in a standard ring-based AllReduce algorithm. This method directly reduces communication time by shortening the critical path of the collective operation.

The authors astutely identify that while prior gradient compression techniques (e.g., top-k sparsification, low-rank approximation) also reduce data transfer, they often introduce significant computational overhead for selection, compression, and reconstruction. This overhead can negate the benefits, especially on modern systems with high-bandwidth interconnects. SkipReduce's primary advantage is its near-zero computational overhead, allowing it to scale effectively with improving hardware.

The paper further refines this core idea by introducing two key enhancements: 1) Random SkipReduce, which randomizes the skipped gradient slices in each iteration to mitigate the bias and accuracy loss of naively skipping the same slices repeatedly, and 2) Selective SkipReduce, which leverages the insight that gradient importance is non-uniform across model layers, allowing for targeted skipping of less critical layers to preserve model accuracy. The authors demonstrate through comprehensive experiments that SkipReduce provides a significant speedup in time-to-accuracy for various models, positioning it as a powerful and practical alternative to existing communication acceleration techniques.

Strengths

1. Elegant and Practical Core Idea: The central concept of skipping collective communication steps is both simple and powerful. It reframes the problem of gradient sparsification from a data-centric view (which gradients to send?) to a system-centric one (which communication steps can be omitted?). This leads to an implementation with minimal overhead, which is a critical differentiator from much of the prior work. The argument presented in Section 3 and validated in Figures 3 and 4 (pages 4-5) is particularly compelling, showing how the computational overhead of techniques like PowerSGD and top-k becomes a performance limiter on high-bandwidth networks, a limitation that SkipReduce elegantly sidesteps.

2. Excellent Contextualization and Exploration of Broader Implications: This work stands out for its panoramic view. The authors do not just present a performance optimization; they explore its deeper connections within the field.

   • Connection to Regularization: The analysis in Section 6.4 (page 12), which positions SkipReduce as a form of coarse-grained gradient dropout, is insightful. Demonstrating that it is complementary to traditional Dropout (Figure 20) elevates the contribution from a mere systems trick to a technique with potential benefits for model generalization.
   • Applicability to Advanced Parallelism: The extension to Sharded Data Parallelism (SDP) in Section 6.3 (page 11) is forward-looking and demonstrates the versatility of the core idea. The discussion of the challenges (aligning skipped weights and gradients) and the novel implications (creating an ensemble of subnetworks) opens up fascinating new research avenues.
   • Analysis of Alternative Topologies: The analytical exploration of how SkipReduce would behave with tree and halving-doubling topologies in Section 6.1 (page 10) shows a thorough understanding of the collective communication landscape and strengthens the paper's claims of generalizability.

3. Thorough and Methodologically Sound Evaluation: The experimental validation is robust. The use of time-to-accuracy (TTA) as the primary metric is the correct choice, as it holistically captures the trade-off between per-iteration speedup and convergence behavior. The comparison against a well-implemented "ideal top-k" baseline and PowerSGD across a diverse set of workloads (CNN, and large-scale Transformers like BERT and LLaMA) provides a convincing case for SkipReduce's effectiveness, especially in the high-bandwidth regime common in modern GPU clusters. The ablations, such as comparing Static vs. Random SkipReduce (Figure 7, page 7), are well-designed and clearly justify the design choices.

Weaknesses

1. Limited Discussion on Hyperparameter Tuning: SkipReduce introduces a new critical hyperparameter: the skipping ratio. The paper primarily evaluates a fixed ratio (e.g., 50%). However, there is little guidance on how a practitioner should select this value. The observation in Figure 2 (page 4) that gradient sparsity changes dynamically throughout training suggests that a static skipping ratio may be suboptimal. A discussion on the sensitivity to this parameter or the potential for an adaptive skipping schedule would significantly enhance the practical utility of the work.

2. Theoretical Intuition Could Be Deepened: The paper's motivation rests on the empirical observation of gradient sparsity. While it correctly cites prior work [46] on the convergence of random gradient compressors, the connection could be more deeply explored. Why is skipping coarse-grained, contiguous slices in a ring AllReduce so effective? There seems to be an implicit assumption that information is sufficiently redundant or evenly distributed such that dropping entire slices is tolerable. A more formal or intuitive explanation of the interplay between the ring algorithm's specific data flow and gradient information structure would strengthen the paper's foundations.

Questions to Address In Rebuttal

1. Could the authors elaborate on the sensitivity of SkipReduce's performance (both TTA and final accuracy) to the skipping ratio? Is there a clear "sweet spot," and does it vary significantly across different model architectures or datasets? Could you provide guidance on how a practitioner might approach tuning this hyperparameter?

2. The connection to regularization is compelling. The finding that SkipReduce and Dropout are complementary (Figure 20, page 12) suggests they might be regularizing the model in different ways. Could you speculate on the mechanisms? For instance, does Dropout prevent co-adaptation of individual neurons, while SkipReduce's coarse-grained nature prevents co-adaptation of larger, structurally-defined groups of parameters (i.e., those grouped into a slice)?

3. The analysis of Sharded Data Parallelism (SDP) in Section 6.3 is fascinating. In this setting, skipping steps in the weight AllGather phase is framed as training a unique subnetwork on each GPU. This seems conceptually similar to ensemble methods. Did you observe if this ensembling effect led to benefits beyond what was seen in the data-parallel case, such as improved model calibration or robustness, even if final accuracy was slightly lower?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

This paper introduces SkipReduce, a novel method for accelerating collective communication in distributed deep learning. The core idea is to modify the standard ring AllReduce algorithm by intentionally skipping a subset of the communication steps in the Reduce-Scatter phase. This action is functionally equivalent to dropping entire slices of gradients in a coarse-grained manner, which directly reduces the total communication time. The authors augment this core idea with two key refinements: 1) "Random SkipReduce," which randomizes the skipped slices across iterations to prevent systemic bias, and 2) "Selective SkipReduce," which applies the skipping mechanism only to model layers deemed less important based on gradient magnitudes. The work positions itself as a low-overhead alternative to existing communication reduction techniques like top-k sparsification and gradient quantization, particularly for high-bandwidth systems where the computational overhead of those methods can negate their benefits.

Strengths

1. Novelty of the Core Mechanism: The central contribution—modifying the communication protocol itself by truncating its steps—is genuinely novel. Prior art has extensively focused on modifying the data being sent (e.g., sparsification via top-k, compression via quantization or low-rank factorization). SkipReduce, in contrast, modifies the protocol execution. Instead of sending modified data for 2(N-1) steps, it sends unmodified data for 2(N-1)-S steps. This shift in approach from data manipulation to protocol manipulation is a significant and clever conceptual leap.

2. Elegant Simplicity: The proposed modification is remarkably simple. As described in Algorithm 2 (page 6), the implementation appears to only require changing the starting index of a loop within the Reduce-Scatter phase. This stands in stark contrast to the significant engineering complexity of efficient top-k selection, sparse format handling (and the associated "fill-in" problem), or the computational cost of low-rank factorization (e.g., PowerSGD). The low complexity is a major strength.

3. Clear Articulation of the Problem Niche: The authors correctly identify a critical gap in existing work. As shown in their motivational analysis (Section 3, Figures 3 and 4), the computational overhead of many compression techniques makes them less effective on modern, high-bandwidth interconnects. By proposing a method with virtually zero computational overhead, the paper targets a relevant and increasingly important regime of distributed training.

Weaknesses

1. Conceptual Overlap with Fault-Tolerant Collectives: While the motivation is novel (proactive acceleration), the effect (an incomplete gradient reduction) bears a strong conceptual resemblance to fault-tolerant or resilient collective communication algorithms designed for unreliable networks (e.g., MLT [52], OptiReduce [53]). These works also deal with dropped packets/gradients and proceed with an incomplete result to avoid tail latency. The paper mentions these works as orthogonal (Section 2.3, page 4), but the distinction is not sufficiently sharp. A more direct comparison explaining why intentionally skipping steps is fundamentally different from reactively handling dropped steps would strengthen the novelty claim.

2. The Novelty is Tightly Bound to a Specific Protocol: The entire mechanism and its elegant simplicity are described almost exclusively in the context of Ring AllReduce. While Section 6.1 (page 10) provides a high-level analytical discussion of other topologies (tree, halving-doubling), it lacks the concrete, algorithmic detail needed to establish the novelty of the idea beyond the ring structure. For a tree-based AllReduce, it is not immediately obvious what "skipping a step" means and whether it would yield a similarly clean reduction in communication time without significant algorithmic changes. The novelty, as presented, is somewhat narrow.

3. Incremental Novelty of Refinements: The refinements, while valuable, are applications of well-known principles to the new SkipReduce mechanism. Randomization to mitigate bias ("Random SkipReduce") is a standard technique in machine learning. Similarly, using gradient magnitude as a proxy for importance ("Selective SkipReduce") is the foundational heuristic for all top-k sparsification methods. While the application of these ideas is new in this specific context, the ideas themselves are not. The paper's primary novelty lies squarely in the core SkipReduce protocol modification, not these extensions.

Questions to Address In Rebuttal

1. Please elaborate on the novelty of SkipReduce in contrast to fault-tolerant collective communication schemes (e.g., MLT [52], OptiReduce [53]) that also result in incomplete reductions. While the motivation (performance vs. resilience) differs, the functional outcome has strong similarities. What is the fundamental distinction that makes SkipReduce a novel contribution beyond a change in intent?

2. The core mechanism is elegantly demonstrated on Ring AllReduce. Can the authors provide a more concrete algorithmic sketch of how SkipReduce would be implemented on a non-ring topology, such as a tree or recursive doubling algorithm? The current analysis in Section 6.1 (page 10) is primarily theoretical and does not sufficiently demonstrate that the core novel idea is generalizable.

3. The concept of selectively skipping less important layers (Section 4.4, page 7) relies on gradient magnitude, a well-established heuristic borrowed from the gradient sparsification literature. Could the authors clarify the novel contribution of this aspect beyond applying a known heuristic to the new SkipReduce primitive?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present a suite of algorithms for All-to-All collective communication on torus networks, targeting both fault-free and fault-tolerant scenarios. For the fault-free case, they propose HalfRing for single-dimension communication and DimRotation for multi-dimensional scheduling. For scenarios with link failures, they introduce FoldedRing as a basic recovery mechanism and MATE/MATEe as an acceleration technique that leverages links from other dimensions. The paper claims significant performance speedups over a baseline Ring algorithm and Google's state-of-the-art routing on TPUv4 clusters.

However, the work rests on a series of questionable assumptions and methodological choices that undermine the validity of its core claims. The comparison against state-of-the-art is fundamentally flawed, the reported real-machine performance directly contradicts the simulation results for the fault-tolerant case, and crucial algorithmic details for complex scenarios are omitted entirely.

Strengths

1. The paper addresses a timely and relevant problem: the All-to-All bottleneck in large-scale distributed training, particularly for torus networks where contention is a primary concern. The added focus on fault tolerance is also well-motivated.
2. The core ideas of HalfRing (utilizing bidirectional links for shortest paths) and DimRotation (balancing load across dimensions) are conceptually straightforward and intuitive for improving upon a simplistic baseline.

Weaknesses

1. Fundamental Methodological Mismatch in SOTA Comparison: The authors' primary comparison is against Google's DOR/WFR routing on TPUv4. Their proposed methods are based on a fine-grained, hop-by-hop, store-and-forward scheduling model. In contrast, modern interconnects like Google's ICI utilize hardware-based, low-latency wormhole or virtual-cut-through routing. Comparing a contention-avoiding store-and-forward algorithm against a contention-prone but low-latency wormhole routing algorithm is a category error. The performance characteristics are entirely different; the former trades higher per-hop latency for algorithmic simplicity and contention avoidance, while the latter optimizes for latency at the cost of requiring complex hardware-level flow control. The paper makes no attempt to justify this apples-to-oranges comparison or to model the latency and resource overheads of its own store-and-forward approach fairly against a hardware-based one.

2. Selection of a Weak Baseline: The claimed speedups (e.g., 2.28x in Figure 11) are presented relative to a "Ring algorithm with pipeline scheduling." This appears to be a strawman. While the basic Ring algorithm is standard, more sophisticated pipeline scheduling schemes that mitigate bubbles more effectively than the simple one implied in Figure 7a exist in the literature. By selecting a potentially weak baseline, the performance gains of HalfRing and DimRotation are likely inflated.

3. Contradictory and Alarming Real-Machine Results: The most significant flaw is the stark contradiction between simulation and real-world measurement. For the fault-tolerant case, simulations claim MATE achieves a ~1.37x speedup over the fault-free baseline. However, the real-machine experiments reported in Section 5.8 and Figure 19 show that MATE achieves a speedup of 0.77x—a 23% slowdown compared to the baseline. The authors briefly attribute this to "greater complexity" and "frequent interruptions," but this explanation is insufficient. This result invalidates the central claim that MATE is an effective acceleration technique in practice; on the contrary, it suggests the overheads of the proposed scheme are so severe that they negate any theoretical benefits.

4. Oversimplified and Underdeveloped Fault-Tolerant Mechanisms:

   • The MATE algorithm's core mechanism relies on "offline performance analysis" (Section 4.2, page 8) to allocate data for acceleration. This critical procedure is never detailed. How is this analysis performed? Is it a heuristic or an optimal solver? Without this information, the algorithm is not reproducible or verifiable.
   • The analysis of multiple faults in Section 5.7 is superficial. For Type 1 failure (two faults on one ring), the paper claims a "two-acceleration-phase MATEe scheme" is used, but provides zero detail on how this works, how it is scheduled, or how it avoids conflicts. This is a critical omission for a paper claiming robust fault tolerance.

5. Unsubstantiated Claims of Generality: The paper claims DimRotation handles mixed-radix torus networks well (Section 3.2, page 6), but provides no analysis or experiments to support this. In mixed-radix systems, communication time per dimension is inherently unbalanced. It is not self-evident that simply rotating the dimension order would be optimal, as the longest-latency dimension would remain the bottleneck regardless of its position in the schedule.

Questions to Address In Rebuttal

1. Please provide a rigorous justification for comparing your software-based, store-and-forward scheduling model against Google's hardware-based, wormhole-routing model (DOR/WFR). Address the fundamental differences in latency, buffer requirements, and contention management.

2. Defend your choice of "Ring algorithm with pipeline scheduling" as the primary baseline. Provide evidence that this baseline is representative of common practice and not an artificially weak competitor.

3. The central contradiction: Your simulations show MATE providing a >1.3x speedup under failure, yet your own real-machine tests show it causes a >20% slowdown (0.77x). Please provide a detailed, quantitative explanation for this discrepancy. Why should the community trust the simulation results when they are invalidated by physical measurement?

4. Provide the full algorithm and a conflict-freedom analysis for the "two-acceleration-phase MATEe scheme" used to handle multiple link failures on a single ring, as mentioned in Section 5.7.

5. Detail the exact "offline performance analysis" procedure used to schedule data transfers in MATE. What are the inputs, the objective function, and the algorithm used to determine the communication schedule? How is the fraction parameter for MATEe determined?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a suite of contention-free algorithms and scheduling policies to optimize the All-to-All collective communication primitive on torus networks, a topology critical to modern large-scale ML systems like Google's TPU clusters. The authors address this well-known bottleneck from two angles: a fault-free scenario and a fault-tolerant one.

For the fault-free case, they propose the HalfRing algorithm, which optimizes single-dimension communication by using bidirectional links for shortest-path routing, and DimRotation scheduling, which orchestrates communication across multiple dimensions to eliminate pipeline bubbles and maximize bandwidth utilization.

For the fault-tolerant case, they introduce FoldedRing to handle a single link failure within a ring, and more importantly, the MATE scheduling strategy. MATE's core insight is to leverage healthy links from other, parallel dimensions to accelerate the necessarily slower communication on the faulty ring. The work is evaluated through extensive simulation, showing significant speedups over baseline ring algorithms and, notably, over the sophisticated hardware routing schemes used in Google's TPUv4 clusters.

Strengths

1. Significant and Timely Problem: The paper tackles a problem of immense practical importance. As acknowledged in the introduction (Section 1, page 1), All-to-All communication is a dominant performance bottleneck for training and inference of massive Mixture-of-Experts (MoE) and Deep Learning Recommendation Models (DLRM). Optimizing this primitive on widely-deployed torus networks is a high-impact endeavor.

2. Elegant Core Contribution in MATE: While the fault-free optimizations (HalfRing, DimRotation) are solid, principled improvements on existing ideas, the MATE scheduling concept for fault tolerance is the paper's most significant and novel contribution. The idea of "borrowing" bandwidth from healthy dimensions to compensate for a localized failure in another (as visualized in Figure 9, page 7) is an elegant solution. It transforms the problem from merely circumventing a failure to actively mitigating its performance impact using the network's inherent multi-dimensional resources. This is a powerful new perspective on fault-tolerant collective design.

3. Principled, Contention-Free Design: The authors' choice to decompose multi-hop transfers into a sequence of orchestrated single-hop, store-and-forward steps is a classic and robust approach to designing collective algorithms. By doing so, they guarantee contention-free communication, which sidesteps the complex and often unpredictable congestion issues that can plague hardware-based, multi-hop routing schemes, even sophisticated ones. This principled design is a key reason for their strong performance.

4. Strong and Contextually-Aware Evaluation: The evaluation is comprehensive and compelling. The authors don't just compare against a weak baseline; they go head-to-head with Google's dimension-order routing (DOR) and wild-first routing (WFR), which are state-of-the-art industrial solutions for the exact same problem on the same hardware topology (Section 5.4, page 10). Demonstrating a 1.57x-1.61x speedup over this baseline is a very strong result. Furthermore, the analysis of real model performance (Section 5.5, page 11) and non-uniform communication patterns (Section 5.6, page 12) grounds the work in practical, real-world scenarios.

Weaknesses

While the paper is strong, there are areas where its context and limitations could be further explored. My points are less about flaws and more about understanding the work's boundaries.

1. Practical Overheads of Store-and-Forward: The store-and-forward approach, while eliminating network contention, can introduce its own overheads, such as per-hop latency, memory buffer pressure, and CPU/local controller costs for managing the fine-grained steps. The paper's analytical model (Table 1, page 5) simplifies this, and the real-machine experiments (Figure 19, page 12) hint at this complexity, showing non-trivial "Startup Time" and "Interruption" costs. While the net result is still a win, a deeper discussion of these practical trade-offs would strengthen the paper.

2. Specialization to Torus Networks: The proposed solutions are exquisitely tailored to the properties of a torus network (specifically, its orthogonal dimensions and wrap-around links). This specialization is a strength, as it allows for high performance. However, it also means the direct contributions are not applicable to other popular large-scale topologies, such as fat-trees or Clos networks, which are common in GPU-based clusters. This is not a flaw, but a boundary condition worth acknowledging more explicitly when discussing the work's impact.

3. Complexity of MATE Scheduling: The implementation of MATE requires an offline analysis to allocate data volumes across the healthy "acceleration planes" (Section 4.2, page 7-8). For a single, known link failure, this seems tractable. However, in scenarios with multiple or dynamic failures, this scheduling problem could become significantly more complex. The paper touches on multiple faults (Section 5.7, page 12), but the scalability and complexity of the scheduler itself is an important practical consideration that could be discussed further.

Questions to Address In Rebuttal

1. Your real-machine performance breakdown (Figure 19b, page 12) shows that MATE has a noticeably higher "Startup Time" and overall "Interruption" time compared to the baseline. Could you elaborate on the source of this overhead? Does it stem from the increased complexity of the multi-path scheduling logic in the PyTorch Distributed backend, and do you see avenues for optimizing this control-plane aspect of your proposal?

2. The MATE scheduling strategy relies on an offline calculation to partition the workload. How does the complexity of this calculation scale with the number of network dimensions and the number/pattern of link failures? Is there a point where the scheduling becomes computationally prohibitive or where finding an optimal partition is an NP-hard problem?

3. The field has seen a growing interest in automated synthesis of collective algorithms (e.g., SCCL, TACOS, as mentioned in Section 6.1). How do you see your manually designed, principled algorithms co-existing with this trend? Could the core insight of MATE—borrowing inter-dimensional bandwidth for fault tolerance—be used as a heuristic or a core principle to guide a future automated synthesizer for resilient collectives?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents a suite of algorithms and scheduling strategies for optimizing All-to-All collective communication on torus networks, considering both fault-free and fault-tolerant scenarios. For the fault-free case, the authors propose the HalfRing algorithm for single-dimension communication and DimRotation for multi-dimensional scheduling. For scenarios with link failures, they introduce the FoldedRing algorithm and the MATE scheduling strategy, which leverages links from other dimensions to accelerate communication on a faulty ring.

While the paper presents a comprehensive and well-engineered system, the core novelty of its constituent components varies significantly. The proposed fault-tolerant scheduling strategy, MATE, appears to be a genuinely novel contribution to collective communication design. However, the remaining components—HalfRing, DimRotation, and FoldedRing—are largely adaptations or specific implementations of well-established principles in parallel algorithms and network routing. The primary contribution of this work thus lies in the clever MATE concept and the integration of these techniques into a cohesive, contention-free framework for All-to-All.

Strengths

1. Novel Fault-Tolerant Scheduling (MATE): The most significant and novel contribution of this paper is the MATE scheduling strategy (Section 4.2, page 7). The idea of dynamically constructing parallel communication paths for nodes on a faulty ring by "borrowing" links from orthogonal dimensions is a clever departure from simple packet-level rerouting (like Google's WFR [78]). This rescheduling of a collective's sub-problem onto entirely different physical resources is a new and powerful concept for algorithmic fault tolerance.
2. Comprehensive System Design: The authors have successfully integrated their proposed techniques into a complete, end-to-end, contention-free solution for All-to-All on a torus. The combination of intra-dimension algorithms and inter-dimension scheduling is systematically handled for both fault-free and faulty cases.
3. Strong Baseline Comparison: The evaluation against Google's DOR and WFR routing schemes on TPUv4-like configurations provides a strong and relevant point of comparison, lending credibility to the performance results.

Weaknesses

The central weakness of this paper is the limited novelty of several of its core algorithmic claims. While presented as new proposals, they represent specific instances of long-standing concepts.

1. HalfRing Lacks Algorithmic Novelty: The HalfRing algorithm (Section 3.1, page 5) is described as leveraging bidirectional links for shortest-path communication. This is the fundamental principle behind any optimal All-to-All algorithm on a ring. The idea of splitting the message for the diametrically opposite node is a textbook method for achieving optimal bandwidth utilization and has been implicitly or explicitly part of optimal ring algorithms for decades. For example, the work of Lam et al. [44] on optimal personalized communication on tori already established the theoretical performance limits that such an approach would achieve. The contribution here is an implementation, not a new algorithm.

2. DimRotation is an Incremental Scheduling Pattern: The DimRotation scheduling scheme (Section 3.2, page 6) is an elegant way to avoid pipeline bubbles. However, the core concept of staggering or rotating the order of operations across parallel units to improve resource utilization is a well-known technique in parallel scheduling. While it is an improvement over the simple pipeline shown in Figure 7a, it is not a fundamentally new scheduling paradigm. The novelty is limited to the specific cyclic assignment of starting dimensions to chunks.

3. FoldedRing is a Reapplication of a Known Concept: The FoldedRing algorithm (Section 4.1, page 7) repairs a broken link by using the reverse channel to form a longer, logical ring. This concept is functionally identical to fault-tolerance strategies seen in other contexts. For instance, Google's "AltRing" algorithm for All-Reduce [42] employs a similar strategy of rerouting to maintain a logical ring in the presence of node failures on a 2D mesh. The idea of "folding" a path back on itself to bypass a fault is a classic routing technique. The authors' contribution is applying this known method to their specific store-and-forward All-to-All implementation, which does not constitute a novel algorithmic discovery.

Questions to Address In Rebuttal

The authors should use the rebuttal to clarify the precise "delta" between their proposals and existing art, and to defend the significance of their most novel contribution.

1. On HalfRing and FoldedRing: The HalfRing algorithm appears to be an implementation of the theoretically optimal strategy for ring All-to-All. Similarly, FoldedRing seems analogous to existing fault-tolerant ring repair mechanisms. Could the authors please clarify what is conceptually novel about these two algorithms beyond their application within the paper's specific store-and-forward scheduling framework?

2. On the Novelty of MATE: The MATE scheduler is the most compelling contribution. However, its efficacy depends on the availability of otherwise idle links in orthogonal dimensions. How would MATE's performance and implementation complexity be affected in a scenario where multiple collective operations are overlapped, potentially creating contention for the "borrowed" links?

3. On Complexity vs. Benefit: The proposed methods rely on fine-grained, software-driven, store-and-forward scheduling, which is inherently more complex to orchestrate than the hardware-based, dimension-order routing (DOR) used as a baseline. For the fault-free case, the combined HalfRing+DimRotation approach yields a ~1.57x speedup over DOR on a single TPUv4 pod (Section 5.4, page 11). Is this performance gain substantial enough to justify the significant increase in software complexity and scheduling overhead compared to simpler, hardware-managed routing?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present Titan-I (T1), a parameterizable, out-of-order (OoO), lane-based RISC-V vector core generator. The paper introduces several microarchitectural techniques aimed at improving scalability and performance, including a floor-planning solver, a dedicated permutation unit, and a shadow cache for mask registers. The evaluation section presents performance comparisons against high-end GPUs (NVIDIA GA102/GB202) and contemporary CPU vector architectures (HiSilicon KP920, SpacemiT X60).

While the ambition of the project is noted, the paper’s central claims of superior performance are predicated on a narrow and highly favorable set of benchmarks. Several key architectural decisions appear to trade correctness and generality for performance in specific scenarios, and the quantitative analysis of the design's own scaling properties is questionable. The comparisons to other architectures, particularly GPUs, are fundamentally flawed, raising serious doubts about the validity of the conclusions drawn.

Strengths

1. Open-Source Contribution: The commitment to providing an open-source RTL generator is a significant contribution that enables community verification and research.
2. Area Efficiency: The reported area efficiency against the HiSilicon KP920 core (Section 6.2.1, page 12) is impressive, assuming the area measurement methodology for the baseline is sound. Achieving comparable performance in 19% of the area is a noteworthy engineering result.
3. Focus on Scalability: The paper correctly identifies critical bottlenecks in scaling vector architectures (permutation, masking, scheduling) and makes a concerted effort to address them, even if the proposed solutions have weaknesses.

Weaknesses

1. Fundamentally Flawed GPU Comparison: The central claim of outperforming NVIDIA GPUs is based on an inappropriate comparison. The authors benchmark T1 against a single Streaming Multiprocessor (SM) on two integer-only cryptographic workloads (NTT, MMM) (Section 6.1, page 11). These workloads are known to be pathologically ill-suited for the SIMT execution model of GPUs and are perfectly suited for a wide-vector architecture. This is a clear case of cherry-picking benchmarks to maximize the perceived advantage. The comparison completely ignores workloads where GPUs excel, such as dense floating-point linear algebra or highly divergent codes. The claim of "outperforming" a GPU is meaningless without a balanced and representative benchmark suite.

2. Questionable Area Scaling Analysis: The area scaling results presented in Figure 4 (page 6) are highly suspect. Specifically, Figure 4c suggests that increasing the LaneScale (i.e., making individual lane datapaths wider) leads to a decrease in the total area of T1. This is counter-intuitive and physically implausible. While the authors attribute this to sharing control logic, such a dramatic reduction in total area (nearly 40% when moving from LaneScale=1 to 4) is an extraordinary claim that requires a much more detailed explanation and justification than is provided. It suggests a potential flaw in the area model or an omission of critical information.

3. Unsafe Architectural Shortcuts for Exception Handling: The mechanism for handling long-latency indexed memory operations relies on a "chicken bit" in a CSR to suppress access-fault exceptions (Section 4.2.1, page 7 and Section 4.2.6, page 9). This is not a robust architectural solution; it is a hardware hack that offloads the burden of ensuring memory safety entirely to software. For a core intended for high-performance, general-purpose workloads, this is a critical design flaw. It renders the core unsuitable for environments where precise exceptions are required for memory management, debugging, or security.

4. Misleading Comparison to In-Order Cores: The performance comparison against the SpacemiT X60 (Section 6.2.3, page 12) results in a claimed 8.05x speedup. However, the X60 is a simpler, in-order core. It is neither surprising nor particularly insightful that a complex OoO architecture significantly outperforms an in-order one. This comparison serves more to inflate T1's performance numbers than to provide a meaningful benchmark against a peer competitor.

5. Insufficient Detail on Novel Contributions: The "coarse-grained floor-planning solver" (Section 4.1.1, page 6) is presented as a key innovation. However, the paper provides no details on the heuristic algorithm itself. Without this information, it is impossible to assess its novelty or effectiveness beyond the single, potentially cherry-picked example in Figure 5. It is unclear how this differs from standard P&R scripting. Similarly, the "shadow mask" cache (Section 4.1.2, page 6) is described, but the mechanism for ensuring correctness and handling coherence with pending writes to v0 is glossed over, despite this being a potential serialization point.

Questions to Address In Rebuttal

1. On GPU Benchmarking: Can the authors justify their claim of GPU superiority by providing performance comparisons on workloads where GPUs are traditionally strong, such as FP32/FP16 SGEMM, stencil computations, or graph analytics? If not, the claims regarding GPU performance should be significantly moderated to reflect the narrow, integer-only context.

2. On Area Scaling: Please provide a detailed breakdown and justification for the claim in Figure 4c that total core area decreases as LaneScale increases. What specific components are shrinking, and why does this effect overwhelm the expected area increase from wider datapaths and crossbars within the lane?

3. On Exception Handling: Please defend the use of a "chicken bit" for indexed memory operations. How does this design support robust, general-purpose software that relies on precise exceptions for virtual memory paging, memory protection, or runtime error handling?

4. On the Floorplan Solver: Please provide sufficient detail on the heuristic algorithm used in your floorplan solver to allow the reader to understand its novelty and distinguish it from a trivial script that drives a standard placement tool. What are the constraints and the objective function it optimizes?

5. On the X60 Comparison: Please clarify if the SpacemiT X60 core used for comparison is an in-order design. If so, please justify why this is presented as a meaningful comparison for an OoO architecture, rather than an expected outcome.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Titan-I, an open-source, highly parameterized generator for an out-of-order (OoO) RISC-V vector (RVV) processor. The work's core contribution is a suite of microarchitectural innovations designed to holistically address the long-standing challenge of scaling both data-level parallelism (DLP), via wide vector datapaths and long vector lengths (VLEN), and instruction-level parallelism (ILP), via fine-grained, OoO execution.

The authors identify that traditional superscalar OoO techniques do not scale well to the massive state of wide vector registers, while traditional vector machines often sacrifice ILP. Titan-I tackles this gap with several key techniques: a coarse-grained floor-planning solver to manage routing complexity in wide designs, a datapath-wide permutation unit to efficiently handle shuffle-heavy workloads, a shadow cache for mask registers to reduce broadcast traffic, and a novel "issue-as-commit" mechanism to decouple the scalar and vector pipelines. Central to its ILP capabilities is a fine-grained chaining microarchitecture that operates at the sub-register (lane) level.

The authors validate their architecture with strong empirical results, including two ASIC tape-outs. Their evaluations show Titan-I outperforming high-end NVIDIA GPUs on specific cryptographic kernels and demonstrating competitive performance-per-area against a high-performance ARM SVE core (HiSilicon KP920) in HPC workloads. The work is presented not as a single point design, but as a flexible generator, positioning it as a significant contribution to the open-source hardware ecosystem.

Strengths

1. Addresses a Fundamental Architectural Trade-off: The paper targets the difficult, yet crucial, intersection of CPU and GPU design philosophies. The quest to unify high single-thread performance (ILP) with massive data throughput (DLP) is a central theme in modern computer architecture. Titan-I offers a compelling and well-reasoned "vector-first" approach to this problem, standing as a modern successor to the philosophy of classic vector supercomputers like the Cray-1.

2. Holistic, System-Level Design: The strength of this work lies not in a single trick, but in the co-design of multiple solutions to solve the wider problem of scalability. The authors correctly identify that simply widening a datapath creates cascading problems in routing, control logic, and data movement. Their solutions—the floorplanner for physical layout (Section 4.1.1, page 6), the dedicated permutation unit for data shuffling (Section 4.1.3, page 7), and the mask register cache (Section 4.1.2, page 6) for predication—demonstrate a deep understanding of both the logical and physical barriers to scaling.

3. Credible and Impressive Empirical Validation: The authors provide a robust evaluation against relevant and powerful commercial counterparts. Comparing against NVIDIA GPUs for crypto and a flagship ARM server CPU for HPC is ambitious and gives the results significant weight. The fact that the project has yielded two physical tape-outs (Section 5.3, page 10) lends enormous credibility to the claimed performance and area results, moving it beyond a purely academic simulation study.

4. Contribution as an Open-Source Generator: Perhaps the most significant aspect of this work is that its deliverable is a highly parameterized generator. This elevates its potential impact substantially. Instead of a single, static design, the authors provide a framework that can be adapted to different application domains and PPA (Power, Performance, Area) targets, from edge devices to data centers. This is a powerful enabler for the RISC-V ecosystem and the broader hardware community.

Weaknesses

While the microarchitectural contributions are excellent, the paper could be strengthened by providing more context on its place within a complete system, particularly regarding software and memory.

1. The Software Abstraction Challenge: The paper rightly celebrates its hardware's performance, but this performance is unlocked via hand-tuned assembly or a custom MLIR-based toolchain (Section 5.1, page 10). A significant advantage of competitors like NVIDIA is the maturity and accessibility of the CUDA ecosystem. For Titan-I to have broader impact, the path from high-level code (e.g., C++, Python) to high-performance vectorized execution needs to be more thoroughly explored. The current presentation leaves the impression that achieving these results requires expert-level programming effort, which could limit its adoption.

2. The Scalar Core as a Potential Bottleneck: The architecture is presented as a vector co-processor that relies on a scalar core for control flow, address generation, and non-vector instructions. The "issue-as-commit" policy (Section 4.2.1, page 7) is a clever way to decouple the pipelines, but the performance of many real-world vector applications (e.g., sparse matrix operations) is often limited by the efficiency of the scalar code that prepares the data for the vector unit. The paper provides little detail on the assumed capabilities of the scalar core or the potential for it to become an "Amdahl's Law" bottleneck.

3. System-Level Memory Integration: The paper demonstrates impressive memory latency tolerance (Section 6.2.2, page 12) and discusses a Memory Management Unit (MMU) as future work (Section 7, page 13). However, in a real system, the interaction with virtual memory—handling page faults, TLB misses, and maintaining coherence with other cores in an SoC—is a first-order design constraint, not just a feature to be added later. A deeper discussion of how the massive, long-latency memory accesses would be managed within a virtual memory system would contextualize the design's practicality for general-purpose computing.

Questions to Address In Rebuttal

1. Could the authors elaborate on the maturity and usability of their MLIR-based software toolchain? For the HPC benchmarks, how much of the performance was achieved through fully automatic auto-vectorization versus manual intervention or the use of compiler intrinsics?

2. The "issue-as-commit" mechanism effectively decouples the scalar and vector units. However, what are the key performance considerations for the scalar core that pairs with Titan-I? Are there specific workload characteristics (e.g., complex address generation, frequent loop-carried dependencies) where the scalar front-end is likely to become the primary performance limiter?

3. The paper's discussion on memory latency tolerance is a highlight. Could the authors comment on how the design's philosophy of handling long-latency operations would be extended to manage system-level complexities like page faults? Would a page fault on one element of a VLEN=16384 operation require stalling the entire vector instruction for its duration, and what would be the performance implications?

4. Section 7 mentions the possibility of adopting a Multiple Streaming Processor (MSP) approach, akin to the Cray X-MP, to improve TLP. Given the generator's flexibility, have the authors explored configurations that might partition a wide Titan-I core into several narrower, independent vector contexts? This seems like a natural architectural evolution to bridge the gap between this powerful single-thread vector model and the many-thread GPU model.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present Titan-I (T1), a generator for an open-source, out-of-order (OoO), lane-based RISC-V Vector (RVV) core. The central thesis is that a combination of novel microarchitectural techniques can overcome the traditional scaling challenges of vector processors, enabling simultaneous scaling of both Data-Level Parallelism (DLP) and Instruction-Level Parallelism (ILP). The proposed contributions are a collection of solutions targeting specific bottlenecks: a coarse-grained floor-planning solver and a dedicated permutation unit for DLP scaling, and a fine-grained chaining mechanism, a scalar-vector decoupling policy ("issue-as-commit"), and specialized memory scheduling for ILP. The authors provide an open-source RTL generator and validate their design with two ASIC implementations and extensive performance comparisons.

My review focuses exclusively on the novelty of the proposed ideas relative to the vast body of prior work in vector and parallel architectures.

Strengths

The primary strength of this work lies not in a single, revolutionary concept, but in the clever synthesis and specific implementation of multiple techniques, some of which represent a genuine delta over prior art.

1. Fine-Grained, Configurable Chaining: The concept of chaining dates back to the Cray-1. However, its application in modern, highly-laned architectures presents new challenges. The closest academic prior art cited, Berkeley's Saturn [51], implements chaining at a coarse "DLEN-granular" level. T1's proposal for configurable, fine-grained chaining that can operate down to the element (ELEN) level (Section 4.2.2, page 8) is a significant and novel advancement. It directly addresses the need for maximizing pipeline utilization in the presence of wide, partitioned datapaths, a key limitation of prior academic designs like Ara [9, 35], which lacks chaining entirely.

2. Physical Design-Aware Microarchitecture: The introduction of a "coarse-grained floor-planning solver" (Section 4.1.1, page 6) is a noteworthy contribution. While floorplanning is a standard part of physical design, explicitly incorporating a heuristic solver into the architectural design flow of a generator to minimize cross-lane routing latency is novel. It acknowledges that at the scale the authors are targeting, physical realities can no longer be an afterthought for the microarchitect. This is a commendable step towards a true hardware-software-physical co-design methodology.

3. Specific Bottleneck Alleviation: The "shadow-cache for mask registers (v0)" (Section 4.1.2, page 6) is an elegant solution to a well-known and painful bottleneck in lane-based RVV designs. While caching is a fundamental computer science concept, the creation of a specialized, dedicated cache within the permutation unit to solve the v0 broadcast problem is a specific, novel, and practical microarchitectural innovation.

Weaknesses

While the paper contains novel elements, several of the core ideas are clever engineering applications of well-established architectural concepts. The paper would be stronger if it more precisely delineated its contributions from this foundational work instead of presenting them as entirely new pillars.

1. Derivative ILP Concepts: Several techniques presented to enhance ILP are modern implementations of classic ideas.

   • "Issue-as-commit" (Section 4.2.1, page 7): This is a form of scalar-vector decoupling. The idea that a scalar core can run ahead of a long-latency vector unit as long as there are no dependencies is conceptually similar to Decoupled Access/Execute (DAE) architectures and the function of scoreboards in early machines like the CDC 6600. The contribution here is a specific, low-overhead scoreboard implementation for RVV, not a fundamentally new execution model. The novelty is in the implementation, not the concept.
   • "Memory Interleaving" and "Memory Delay Slot" (Sections 4.2.5 and 4.2.6, page 9): Overlapping load and store operations and scheduling independent instructions to hide memory latency are canonical compiler and architecture techniques. The paper presents a robust implementation for RVV (e.g., the Conflict Region Table), but the underlying principles are not new.

2. Conflation of Contributions: The paper aggregates a large number of techniques. This makes it difficult to assess the novelty and benefit of each individual contribution. The impressive final results are a product of the entire system, but the value of, for instance, the floor-planning solver is not isolated from the value of the fine-grained chaining. An ablation study would be necessary to truly weigh the merit of each proposed idea against its complexity. The performance gains are substantial, but it's unclear if they come from one or two key breakthroughs or the aggregation of many marginal improvements.

Questions to Address In Rebuttal

1. Clarification on Chaining Novelty: The claim of "fine-grained chaining" is compelling. Could the authors please elaborate on the delta between their linked-list scoreboard approach (Section 4.2.4, page 8) and other forms of fine-grained dependency tracking in prior SIMD or vector architectures, academic or industrial? Is the novelty in the data structure, the configurability, or both?

2. Comparison to Classic Decoupling: Please contrast the "issue-as-commit" mechanism with classic scoreboard-based designs and more formal Decoupled Access/Execute architectures. What is the precise, novel contribution beyond applying a known decoupling principle to the specific scalar-vector interface of RVV?

3. Quantifying Individual Contributions: The paper presents a complex system with multiple interacting optimizations. To better assess the significance of each novel idea, can the authors provide any data (even from simulation) that isolates the performance benefit of the key contributions? For example, what is the performance of T1 with fine-grained chaining disabled (i.e., DLEN-granular like Saturn)? What is the performance impact of using a naive, grid-like floorplan instead of the solver's output?

4. Generality of the Floor-Planner: The floor-planning solver is an interesting design-time contribution. Is the solver's heuristic specifically tuned for the permutation patterns of RVV, or does it represent a more generalizable approach for optimizing communication in tiled accelerator architectures?

Recommendation: Accept with minor revisions.

The paper presents a powerful and well-engineered vector core. While some of its ILP-enhancing concepts are derived from established principles, the work contains several genuinely novel and significant contributions, particularly in its approach to fine-grained chaining and the integration of physical design constraints into the microarchitecture generator. The rebuttal should focus on more precisely situating their work in historical context and, if possible, providing data to deconvolve the benefits of their many contributions.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present SHADOW, an asymmetric Simultaneous Multi-Threading (SMT) architecture that concurrently executes out-of-order (OoO) and in-order (InO) threads on a single core. The stated goal is to dynamically balance Instruction-Level Parallelism (ILP) and Thread-Level Parallelism (TLP) to better suit workloads with irregular memory access patterns, such as sparse matrix multiplication. The core mechanism involves partitioning core resources (e.g., the register file) between a small number of heavyweight OoO threads and a larger number of lightweight InO threads, with a standard software work-stealing mechanism distributing tasks. While the paper identifies a valid and well-known problem, the proposed solution's evaluation, claims of efficiency, and novelty are not rigorously substantiated, and critical design aspects such as security are inadequately addressed.

Strengths

1. Problem Formulation: The paper correctly identifies the performance challenges of sparse, memory-bound workloads on conventional CPU architectures and the inherent tension between deep ILP extraction and wide TLP scaling.
2. Core Concept: The high-level idea of combining OoO and InO execution contexts within a single core to adapt to dynamic workload phases is conceptually plausible.
3. Detailed Case Study: The analysis of Sparse Matrix Multiplication (SpMM) across a range of sparsities (Section 5.3, pages 10-11) provides the paper's most compelling, albeit narrow, evidence for the architecture's potential benefits under specific memory-constrained conditions.

Weaknesses

1. Unsubstantiated and Implausible Overhead Claims: The central claim of a "just 1% area and power overhead" (Abstract, Section 5.4 page 12) is based on a "modified McPAT" model. This is insufficient evidence. The paper fails to detail the modifications made to the tool, nor does it justify them. The architecture introduces non-trivial hardware: per-thread InO FIFO queues, a multi-ported scoreboard for InO dependency checking, and additional multiplexing/demultiplexing logic in the fetch, decode, and issue stages (Table 4, page 11). To claim these structures collectively contribute only 1% overhead without a detailed, verifiable analysis is not credible.

2. Grossly Inadequate Security Analysis: The security implications of this new SMT design are dismissed in a single, hand-waving paragraph (Section 3.11, page 7). In the current landscape of microarchitectural attacks, proposing a novel resource-sharing scheme without a thorough analysis of its vulnerability to contention-based side- and covert-channels is a critical omission. Stating that comprehensive protection is "beyond this paper's scope" is unacceptable. The design directly creates new shared resources (e.g., InO issue logic, scoreboards) that could be exploited.

3. Flawed and Unconvincing Competitive Analysis: The comparison to FIFOShelf is not based on a faithful implementation but on a "roof-lined" model whose parameters seem arbitrary (Section 5.3.1, page 10; Figure 13, page 10). The authors claim this provides an "optimistic upper bound," but the justification for the chosen parameters (e.g., "a doubled ROB dedicated to the OoO path") is absent. Without a principled or direct comparison, the claims of SHADOW's superiority over related speculative-steering approaches are unsupported.

4. Overstatement of Novelty in Work Distribution: The paper presents the "dynamic work distribution mechanism" (Section 3.9, page 7) as a key feature. However, Algorithm 1 is a textbook implementation of a work-stealing queue using a shared counter protected by a mutex. The "emergent" property where faster threads take more work is a fundamental, long-understood characteristic of this pattern, not a novel co-design. The contribution is merely the application of a standard software library pattern on their hardware, not a novel hardware-software mechanism.

5. Contradictory Performance Rationale: Table 3 (page 10) claims the best configuration for Backprop (1 OoO + 3 InO) provides a 3.16x speedup by alleviating "RF, ROB and cache contention." While InO threads do not consume ROB entries or OoO PRF entries, the baseline is a single OoO core. The critical comparison should be against symmetric SMT configurations (e.g., 2-OoO, 3-OoO). Figure 4 shows that 3-OoO performs worse than 1-OoO+4-InO, but the text needs to more rigorously prove that the resource savings from using InO threads outweighs the performance loss from their simpler pipelines, especially compared to a well-provisioned symmetric 2-OoO SMT core, which is the industry standard.

Questions to Address In Rebuttal

1. Overhead: Please provide a detailed breakdown of the modifications made to McPAT. Justify the area and power models used for the new structures (InO FIFOs, scoreboards, multiplexers). How can these additions be credibly contained within a 1% total core overhead budget?
2. Security: Given that SMT security is a first-order design concern, provide a detailed analysis of potential new contention channels introduced by SHADOW. How would the shared fetch policy, InO/OoO RS arbitration, and the InO scoreboard be secured against malicious threads?
3. Comparison: Justify the specific parameters chosen for the FIFOShelf roofline model. Why is this configuration a fair and representative upper bound for a state-of-the-art speculative instruction steering architecture? A more principled comparison is required.
4. Work Stealing: Please clarify the novelty of the work distribution mechanism. Is there any hardware support for this beyond providing distinct thread contexts, or is the contribution entirely the use of a standard software Pthreads library?
5. Critical Path: Section 3.10 (page 7) cites MorphCore's 2.5% frequency impact and claims a similar effect for SHADOW due to multiplexers. Can you provide a more rigorous analysis of the critical path impact? Specifically, how does the logic for selecting between ready OoO and InO instructions affect the issue stage's timing?
6. OS/Runtime Interaction: The shdw_cfg instruction (Section 3.3, page 5) implies OS modification and a reconfiguration process. What is the latency of this context switch and reconfiguration? How does this latency impact workloads with frequent phase changes or fine-grained multitasking?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents SHADOW, a novel asymmetric Simultaneous Multi-Threading (SMT) architecture. The core contribution is the ability to execute both traditional high-ILP out-of-order (OoO) threads and lightweight high-TLP in-order (InO) threads concurrently within a single processor core. By dynamically balancing the workload between these asymmetric thread types via a software work-stealing mechanism, SHADOW aims to adapt to applications with shifting parallelism characteristics, particularly memory-bound and sparse workloads that suffer from underutilized resources on conventional core designs. The authors demonstrate significant performance gains (up to 3.16x, with a 1.33x average) over a baseline OoO core with minimal (1%) area and power overhead.

Strengths

The true strength of this work lies in its elegant synthesis of several established architectural concepts into a coherent and compelling new design point.

1. A Novel and Elegant Approach to ILP-TLP Balancing: The problem of balancing Instruction-Level Parallelism (ILP) and Thread-Level Parallelism (TLP) is a classic challenge in computer architecture. Historically, designs have polarized towards one extreme (e.g., wide OoO cores for ILP) or the other (e.g., Sun's Niagara for TLP). More recent approaches have explored heterogeneity at the core level (e.g., ARM's big.LITTLE) or temporal mode-switching within a core (e.g., MorphCore). SHADOW’s contribution of enabling simultaneous, intra-core heterogeneity is a genuinely new perspective. It avoids the OS scheduling complexity of inter-core migration and the potential stalls of mode-switching by allowing both execution styles to coexist and dynamically share resources. This is a very clever architectural solution.

2. Excellent Positioning within the Design Space: The paper does a good job of situating SHADOW relative to its closest relatives. The distinction from MorphCore (simultaneous execution vs. mode-switching) and from speculative steering approaches like FIFOShelf (partitioning at the thread-level vs. the instruction-level) is clearly articulated in Section 2.2 (page 3). This highlights the pragmatism of the SHADOW design, which avoids the complexities of speculative recovery across different execution paths. It finds a sweet spot in the design space that was previously unoccupied.

3. Pragmatic Hardware-Software Co-Design: The architecture does not rely on exotic hardware or a new ISA. Instead, it makes modest, well-motivated modifications to a conventional OoO pipeline. Critically, it leverages a well-understood software work-stealing paradigm (Algorithm 1, page 8) for load balancing. This emergent balancing, where faster (high-IPC OoO) threads naturally claim more work, is an efficient and decentralized control mechanism. It connects the hardware architecture to decades of research in parallel runtime systems (e.g., Cilk, TBB, OpenMP) and makes the design more readily programmable.

4. Strong Problem Motivation: The choice to focus on sparse and memory-bound workloads is timely and important. These workloads are prevalent in critical domains like machine learning, graph analytics, and scientific computing, and they are notoriously difficult for conventional architectures to accelerate efficiently. By demonstrating significant speedups on benchmarks like SpMM, Backprop, and APSP, the paper makes a strong case for its real-world relevance.

Weaknesses

While the core idea is strong, the evaluation and discussion are focused primarily on the microarchitecture, leaving its broader system-level implications less explored.

1. Limited System-Level Scope: The current model is constrained to either multiple single-threaded applications or a single multi-threaded application (as stated in Section 3.3, page 5). This is a significant limitation for a general-purpose processor operating in a modern multi-tasking OS. The paper does not adequately explore the challenges of resource partitioning, scheduling, and ensuring fairness or QoS if multiple SHADOW-aware applications were to run concurrently. This is the biggest barrier between the current concept and a deployable system.

2. Under-explored Cache and Prefetcher Dynamics: The paper reports that cache-sensitive kernels can slow down due to contention (Section 5.1, page 9, Figure 12). This is a crucial interaction. The lightweight InO threads, while not adding ROB pressure, will still aggressively issue memory requests. How do their access patterns interact with the (potentially more regular) access patterns of the OoO thread? It seems likely that the mix of memory streams could confuse conventional stride or stream prefetchers, potentially degrading performance for both thread types. A deeper analysis of this cross-thread resource contention, especially on the memory subsystem, would strengthen the paper.

3. Security Implications Acknowledged but Not Addressed: Section 3.11 (page 7) correctly identifies that resource sharing in SMT creates security vulnerabilities. While it notes that SHADOW's InO lanes reduce some attack surfaces by eliminating speculation, it does not explore the new, potentially subtle channels that arise from the interaction between OoO and InO threads sharing the L1 cache, execution units, or other structures. Given the intense focus on SMT security post-Spectre, this aspect warrants more than a brief mention and a pointer to prior work.

Questions to Address In Rebuttal

The authors have presented a compelling architectural idea. To better understand its potential, I would appreciate their thoughts on the following:

1. OS Scheduling Interaction: Beyond the "delegate thread" for configuration, how do you envision a modern OS scheduler (like the Linux CFS) interacting with a SHADOW core? Would the OS need to be aware of the OoO/InO asymmetry to make intelligent placement decisions, for example, by prioritizing latency-sensitive threads for the OoO slots?

2. Hardware-Software Interface for Work Stealing: The current work-stealing mechanism is purely software-based. Could the hardware provide performance counters or hints to the runtime—for instance, an indicator of the OoO thread's ROB/LSQ occupancy or recent IPC—to help the software make more informed decisions about work chunk size or stealing strategy?

3. Applicability to Other Domains: While the paper focuses on sparse workloads, the core idea of balancing ILP and TLP seems broadly applicable. Could you comment on how SHADOW might perform on other types of workloads, such as those with producer-consumer patterns, where one thread might be compute-bound (ideal for OoO) while another is I/O-bound (potentially benefiting from a lightweight InO thread)?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The central novel claim of this paper is the design of an asymmetric Simultaneous Multi-Threading (SMT) core that concurrently executes both full-fledged out-of-order (OoO) software threads and lightweight in-order (InO) software threads. The proposed architecture, SHADOW, partitions front-end resources (e.g., renaming logic, reorder buffer) to cater to these two distinct thread types while allowing them to share back-end execution units. The stated goal is to create a single core that can dynamically and efficiently balance Instruction-Level Parallelism (ILP), extracted by the OoO thread(s), and Thread-Level Parallelism (TLP), provided by the numerous InO threads. This adaptability is presented as a solution for workloads, such as sparse matrix multiplication, that exhibit shifting parallelism characteristics.

My assessment, based on an extensive review of prior art in computer architecture, is that the core architectural concept—the simultaneous co-existence and execution of distinct OoO and InO software threads within a single SMT core—is a novel contribution. While constituent ideas (SMT, heterogeneous architectures, lightweight threads) are well-established, their synthesis into this specific microarchitecture represents a new and unexplored design point.

Strengths

1. Clear Novel Contribution in SMT Design: The paper successfully differentiates its core idea from the closest prior art. The key differentiator is simultaneity and thread-level granularity.

   • Unlike MorphCore [56], which performs whole-core mode-switching between OoO and InO execution, SHADOW allows both modes to operate concurrently. This is a fundamental architectural distinction that enables the handling of workloads with mixed-parallelism phases without incurring mode-switch penalties.
   • Unlike FIFOShelf [53] and FIFOrder [6], which speculatively steer instructions from a single thread down different paths, SHADOW partitions entire software threads non-speculatively. This is a much coarser-grained approach that avoids the significant hardware complexity of cross-path dependency tracking, speculative wakeup, and misprediction recovery, making the design arguably more practical.

2. Elegant Synthesis of Existing Concepts: The novelty here is not in the invention of a brand-new mechanism from scratch, but in the clever synthesis of established principles. The idea of adding lightweight, non-speculative execution capabilities to a complex OoO core is a direct and logical approach to tackling the underutilization of back-end resources during memory stalls. The proposed implementation, which leverages per-thread FIFO queues for InO instructions to bypass the complex rename/ROB pipeline (Figure 5, page 4), is a clean and low-overhead design.

3. Low-Complexity Architectural Delta: The authors claim the modifications result in only a 1% area and power overhead (Section 5.4, page 12). From a novelty standpoint, this is crucial. The proposal does not require a radical redesign of the core; rather, it augments an existing OoO pipeline with simple, parallel structures. This demonstrates that a significant gain in adaptability can be achieved with a minimal and non-disruptive change, which enhances the value of the new idea.

Weaknesses

1. The Software Mechanism Lacks Novelty: While the hardware architecture is novel, the mechanism for workload distribution—software-based work stealing (Algorithm 1, page 8)—is a standard, widely-used technique in parallel programming (e.g., Intel TBB, Cilk). The paper should be more explicit that the novelty lies exclusively in the hardware that makes this standard software pattern highly effective, rather than in the pattern itself. The dynamic adaptation is an emergent property of existing software running on new hardware, not a feature of a new co-designed algorithm.

2. Under-explored Connection to Intra-Core Heterogeneity: The paper correctly distinguishes its approach from inter-core heterogeneous systems like ARM's big.LITTLE. However, the novelty could be framed more powerfully by situating it within the broader landscape of "intra-core heterogeneity." The introduction focuses on the ILP-TLP trade-off but could benefit from a clearer articulation of how SHADOW presents a new path to achieving heterogeneity inside the core boundary, contrasting it more sharply with prior academic concepts like Core Fusion [28] which composed cores, rather than decomposing thread types.

3. Static Nature of the Asymmetry: The novelty is the flexible mix of OoO and InO threads. However, the mechanism to configure this mix, the shdw_cfg instruction (Section 3.3, page 5), appears to be a one-time setup at the start of an application or context switch. This feels like a missed opportunity. A truly dynamic architecture might allow for the promotion/demotion of threads between OoO and InO modes during execution, which would represent an even greater delta over the prior art. As presented, the "dynamic" balancing is in the work distribution, not in the hardware's configuration itself post-spawn.

Questions to Address In Rebuttal

1. Dynamic Reconfiguration: The shdw_cfg instruction (Section 3.3, page 5) appears to set the core's asymmetric configuration for an application's lifetime or until the next context switch. Is there any fundamental architectural barrier to reconfiguring the OoO/InO mix dynamically within a single application run without a full OS context switch? For example, could a user-level runtime library trigger a reconfiguration if it detects a persistent phase change in the application? This speaks to the true dynamism and novelty of the design.

2. Comparison to Fine-Grained Heterogeneity: Could the authors further elaborate on the trade-offs between their coarse-grained, thread-level asymmetry and the finer-grained, instruction-level asymmetry of FIFOShelf/FIFOrder? While SHADOW is clearly simpler, are there classes of workloads with very rapidly changing ILP characteristics where an instruction-level approach, despite its complexity, might prove superior? A deeper analysis would better solidify the novelty and contribution of the specific design point SHADOW occupies.

3. Novelty Beyond a Single OoO Thread: The paper's most compelling results often come from a 1 OoO + N InO configuration. How does the novelty and benefit of the architecture hold up when scaling to multiple OoO threads (e.g., 2 OoO + 2 InO)? My concern is that contention between two complex OoO threads on shared resources (ROB partitions, LSQ, register file) could negate the benefits of the InO threads, making the novel concept less effective beyond the single "main thread" use case.

Review Form:

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors propose a register renaming technique, "ATR," that aims to reduce physical register file pressure by releasing registers early. The central concept is the "atomic commit region," defined as a sequence of instructions containing no branches or potential exceptions. The authors claim this allows for safe, out-of-order register release without the need for checkpointing or complex recovery mechanisms associated with fully speculative techniques. They present analysis suggesting a significant portion of registers (17% in SPECint) are allocated within these regions and show IPC improvements, particularly for small register file configurations.

Strengths

1. Problem Motivation: The paper does an excellent job of motivating the problem. The introductory discussion and Figure 1 clearly and effectively illustrate that physical register file pressure is a first-order performance limiter in modern wide-issue superscalar processors. The analysis is sound and sets a clear context for the work.

2. Opportunity Analysis: The lifecycle analysis of a physical register in Section 3.1 is well-structured and provides a useful taxonomy ("In-use," "Unused," "Verified-unused"). Figure 4, which quantifies the time spent in each state, effectively frames the performance opportunity that early release schemes, including the one proposed, aim to capture.

3. Conceptual Framework: The core idea of identifying an "atomic commit region" as a basis for a safer form of early release is a logical middle ground between overly conservative commit-order release and aggressive, complex speculative release. Conceptually, it presents a potentially valuable point in the design space.

Weaknesses

My primary concerns with this work center on the fundamental safety claims, the practical implications of the design, and the significance of the results when placed in context.

1. The Definition of "Atomic" is Fundamentally Unsafe Regarding Exceptions: The paper's central claim of safety hinges on identifying "atomic commit regions" at the rename stage. These regions must exclude "exception-causing instructions" (Abstract, page 1). However, the paper fails to address how it can possibly know, at rename, whether a memory instruction (e.g., a load or store) will cause an exception like a page fault. A memory access is always potentially faulting until it has been executed and its address translated by the memory subsystem. By classifying all memory instructions as potentially exception-causing and thus breaking atomic regions, the length of these regions would become trivial, likely consisting of only a few ALU instructions. Conversely, if the authors are not considering memory instructions as exception-causing at rename, their technique is no longer safe or non-speculative with respect to exceptions, directly contradicting their claims (e.g., "our approach is safe providing precise exceptions," Section 1, page 2). This appears to be a critical, unaddressed flaw in the core premise.

2. Impractical Interrupt Handling: The proposed handling for interrupts in Section 4.1 (page 5) is concerningly simplistic and heavyweight. The authors suggest either draining the ROB (introducing potentially unbounded latency) or flushing the entire pipeline and re-executing. While they argue this "does not violate correctness," it sidesteps the immense performance implications. A high-priority interrupt in a real-time system or even a standard OS timer tick could trigger a catastrophic performance loss with the flush-based approach. The paper provides zero evaluation of the performance overhead of this mechanism, which is a major omission for a technique intended for high-performance processors.

3. Optimistic Hardware Implementation and Timing: The hardware described in Section 4.2.2 (page 7) for bulk-marking ptags as "no-early-release" appears complex for the rename stage. The logic must read all current architectural-to-physical mappings from the SRT upon renaming any branch or exception-causing instruction. The authors propose pipelining this critical path, but the analysis in Section 5.5 that dismisses the impact of a 2-cycle delay is unconvincing. It relies on averages from Figure 14, which can easily hide worst-case scenarios where short-lived registers are redefined within the pipeline delay, completely negating the benefit of ATR for those instances. The claim that this complex logic can be pipelined to run at 4GHz+ after synthesis at 2.6GHz seems optimistic.

4. Marginal Gains Over a Stronger Baseline: While the speedups for a 64-entry register file are large, this is an artificially constrained configuration for the simulated Golden Cove-like core, which in reality has a 512-entry ROB and would be paired with a much larger register file. In the more realistic 224-entry configuration (Figure 10), the proposed "atomic" scheme provides only a 1.48% speedup. More importantly, when combined with a non-speculative early release (nonspec-ER) scheme, the additional benefit of ATR is a mere 0.37% for SPECint. This suggests that a well-implemented conventional early release scheme already captures most of the available benefits, making the significant added complexity of ATR difficult to justify for such a negligible incremental improvement.

Questions to Address In Rebuttal

1. Please clarify your precise criteria for identifying an instruction as "non-exception-causing" at the rename stage. Specifically, how are memory access instructions handled? If they are treated as potentially causing exceptions (as they should be), please provide data on the resulting (likely much smaller) size of atomic regions and the impact on your reported 17%/13% opportunity in Figure 6. If they are not, please justify how your mechanism can be considered "safe" with respect to precise exceptions.

2. The proposed interrupt handling mechanisms (ROB drain or flush) introduce significant, unevaluated performance penalties. Please quantify the performance impact of your chosen mechanism under a realistic interrupt workload (e.g., a 1ms timer tick) and justify its viability in a general-purpose processor.

3. The performance impact of the N-stage pipeline delay for the bulk-marking logic was dismissed as "negligible" based on average register lifetimes. What percentage of atomic early-release opportunities are lost specifically due to this 2-cycle delay? How does this impact registers with lifetimes shorter than this delay?

4. For the realistic 224-entry register file configuration, the "combined" scheme shows only a 0.37% improvement over the "nonspec-ER" baseline. Given the added hardware complexity for atomic region detection, consumer counting, and the intricate flush-recovery logic (Section 4.2.4), how do you justify the value proposition of ATR for such a marginal gain over an existing technique?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces "ATR" (ATomic register Release), a novel technique for improving the efficiency of physical register file (PRF) utilization in modern out-of-order processors. The core problem it addresses is the conservative nature of conventional register release, where a physical register is only freed after a subsequent instruction that redefines the same architectural register commits. Existing "early release" solutions are often either speculatively unsafe (requiring complex recovery mechanisms like shadow register files) or non-speculatively safe but still overly conservative (requiring the redefining instruction to be past all unresolved branches and potential exceptions).

The key insight of ATR is to identify and exploit "atomic commit regions"—sequences of instructions guaranteed to contain no conditional branches or exception-causing instructions. The authors posit that within such a region, a physical register can be safely released out-of-order as soon as its last consumer has executed and it has been redefined, even if older, unresolved branches exist outside the region. This safety is guaranteed because any misprediction of an older branch would flush the entire atomic region, making the early release moot, while a correct prediction ensures the region will eventually commit, validating the release. The authors propose a low-overhead hardware mechanism to identify these regions and track consumer counts, demonstrating significant IPC improvements on resource-constrained PRFs or, alternatively, substantial reductions in required PRF size for equivalent performance.

Strengths

1. Elegant Core Concept: The central idea of using "atomic commit regions" as a basis for safe, out-of-order release is both insightful and elegant. It carves out a well-defined space between the aggressive-but-complex speculative approaches (e.g., checkpointing) and the safe-but-conservative non-speculative ones. It's a clever microarchitectural observation that directly translates into a practical optimization.

2. Strong Contextualization and Motivation: The paper does an excellent job positioning itself within the broader landscape of PRF management techniques. The background (Section 2) and related work (Section 6) sections clearly delineate how ATR differs from and improves upon prior art. The motivation provided in the introduction, supported by Figure 1, effectively communicates the criticality and timeliness of the problem.

3. Pragmatic and Plausible Implementation: The proposed hardware mechanism (Section 4.2) appears practical and low-cost. Augmenting the physical register table with a small consumer counter and adding logic to the rename stage to detect the boundaries of atomic regions is a far lighter-weight solution than implementing a full shadow register file or complex checkpoint-and-restore logic. The overhead analysis in Section 4.4, including the synthesis results, further strengthens the claim of practicality.

4. Demonstrated Orthogonality: A key strength of this work is the demonstration that ATR is not a mutually exclusive alternative but a complementary technique. The evaluation in Figure 10, which shows that combining ATR with a traditional non-speculative early release scheme yields the best results, highlights its value. This suggests that ATR could be integrated into existing high-performance cores as another tool in the architect's toolbox for managing register pressure.

Weaknesses

1. Limited Scope of "Atomic Regions": The definition of an atomic region is necessarily strict to ensure safety (no branches, no loads/stores, etc.). While the analysis in Figure 6 shows a respectable opportunity (13-17% of registers), it also implicitly highlights that over 80% of registers are outside the scope of this specific optimization. The paper could be strengthened by a discussion on the potential for, or challenges of, relaxing this definition. For instance, could branches with extremely high confidence or memory operations with predictable latency (e.g., L1 hits) be incorporated into a "quasi-atomic" region concept? This feels like a natural and important direction for future work that is worth acknowledging.

2. Brief Treatment of Interrupts: The handling of precise interrupts is addressed in Section 4.1 by suggesting either draining the ROB or a full flush after the active atomic regions are resolved. While functionally correct, this could introduce significant and unpredictable interrupt latency. In many domains (e.g., real-time systems, high-frequency trading), this latency is a critical design parameter. A more in-depth analysis of the performance implications of this design choice, perhaps quantifying the frequency of interrupts and the resulting pipeline drain cycles, would make the proposal more robust.

3. Missed Connection to Trace-Based Mechanisms: The concept of identifying and optimizing branch-free regions of code is reminiscent of work on trace caches and other trace-based processors (e.g., Rotenberg et al. [28]). While the goal is different (instruction supply vs. register release), the underlying principle of leveraging linear code sequences is similar. A brief discussion situating ATR in relation to these concepts could provide a richer context, exploring potential synergies or design trade-offs if both mechanisms were present in a core.

Questions to Address In Rebuttal

1. The core contribution hinges on the definition of an atomic region. Could the authors elaborate on the sensitivity of their results to this definition? For instance, what is the impact on the opportunity space (the percentage of "atomic registers") if load instructions that are guaranteed L1D hits are allowed within a region? Is there a path toward a more flexible, dynamic definition of atomicity?

2. Regarding interrupt handling (Section 4.1), the proposal to drain the pipeline could be a performance concern. Could you provide data on the frequency of interrupts in the SPEC workloads and estimate the average number of stall cycles this mechanism might introduce per interrupt event? How does this compare to the baseline architecture's interrupt handling latency?

3. The combined scheme (ATR + non-speculative ER) shows the most promise. This suggests a hybrid approach is optimal. Do the authors envision a scenario where the processor could dynamically choose which release policy to apply based on runtime characteristics, or is a static, combined implementation the most logical endpoint for this line of research?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper introduces "ATR: Out-of-Order Register Release Exploiting Atomic Regions," a technique aimed at alleviating physical register file pressure. The central claim is that by identifying "atomic commit regions"—sequences of instructions guaranteed to contain no branches or exception-causing instructions—it is possible to safely release a physical register out-of-order, even before the redefining instruction has been pre-committed. This carves out a new design point between aggressive but unsafe speculative early release schemes (which require complex checkpointing and recovery) and safe non-speculative schemes (which are conservative, waiting for all prior branches to resolve). The authors implement this idea and show modest IPC speedups for constrained register files (5.13% for 64-entry) and a more significant reduction in register file size (27.1%) for a minimal performance loss.

Strengths

The primary strength of this paper lies in its core conceptual contribution. My analysis confirms that the central idea is, in fact, novel.

1. Novel Condition for Safe Release: The prior art in safe, non-speculative early release, such as Monreal et al. [19], hinges on the redefining instruction becoming pre-committed (i.e., all older control flow instructions have resolved). This is a global property of the instruction stream. ATR replaces this global condition with a local one: whether the producer, all its consumers, and the redefiner exist within a dynamically identified atomic region. This allows release to occur even in the presence of an older, unresolved, long-latency branch, something existing safe techniques cannot do. This "local atomicity" as a sufficient condition for safe out-of-order release is a new insight.

2. Clear Delimitation from Prior Art: The paper does an admirable job of positioning its contribution relative to the vast body of work on register management. It correctly identifies the unsafe nature of purely speculative techniques (e.g., Moudgill [22], Ergin [6]) and the conservative nature of existing non-speculative techniques [19]. ATR cleverly exploits a property of instruction sequences to achieve safety without the conservatism of waiting for pre-commit.

3. Thorough Analysis of the Opportunity: The analysis in Section 3, particularly Figure 6 (page 5), is commendable. Quantifying the fraction of registers that fall within non-branch, non-exception, and fully atomic regions provides a solid theoretical foundation for the work and demonstrates that the opportunity ATR targets is non-trivial.

Weaknesses

While the core idea is novel, its practical realization and significance are open to critique.

1. Marginal Impact for Realistic Configurations: The novelty's impact diminishes as the machine configuration scales. A 1.48% speedup for a 224-entry register file is a very marginal gain. While the authors pivot to a register-file-size reduction argument (Figure 15, page 11), this reframes the contribution from a performance enhancement to a cost-saving measure. The novelty is not in question, but its ability to drive significant performance in modern, well-provisioned cores is. A truly groundbreaking idea should ideally provide more substantial benefits.

2. Non-Trivial Implementation Complexity: The proposed mechanism for identifying atomic regions and managing register state is not simple. The "bulk no-early-release" logic described in Section 4.2.2 (page 7) requires setting the state of numerous ptags in parallel whenever a branch or exception-causing instruction is renamed. The authors themselves note this may require pipelining, which introduces latency into the redefinition signal—the very signal that enables early release. While their sensitivity study suggests a 2-cycle delay has minimal impact, this adds a non-trivial piece of timing-sensitive logic to the already critical rename stage. Furthermore, the "Double-Free Avoidance" mechanism (Section 4.2.4, page 7) adds state (two bits per architectural register) and complex logic to the flush recovery path. The novelty comes at the cost of tangible complexity.

3. Restrictive Definition of "Atomic Region": The definition of an atomic region is extremely strict: no conditional branches, no indirect jumps, and no exception-causing instructions. The exclusion of exception-causing instructions effectively bars all memory operations (loads/stores) from these regions. This severely limits the length and prevalence of qualifying atomic regions, capping the potential of the proposed technique. The novelty is confined to very specific, short instruction sequences.

Questions to Address In Rebuttal

1. Significance of the Contribution: Given the modest IPC gains (1.48%) on the 224-entry register file configuration, which is closer to modern core designs, what is the most compelling argument for a processor architect to adopt the added hardware complexity of ATR over existing, simpler non-speculative schemes? Is the primary benefit area/power reduction rather than performance?

2. Boundaries of the "Atomic Region" Concept: The definition of an atomic region seems overly restrictive by excluding all loads and stores due to the possibility of page faults. Have the authors considered relaxing this definition? For instance, could loads that are provably non-faulting (e.g., stack-relative accesses within the mapped stack frame) be permitted within an atomic region? Such a relaxation would significantly increase the applicability of ATR and strengthen the novelty of the contribution.

3. Scalability of the Invalidation Logic: The bulk invalidation logic (Figure 9, page 7) must check and potentially set the no-early-release status for all ptags referenced in the SRT when a branch is renamed. For an 8-wide x86 machine, this is a substantial number of ptags. Could the authors elaborate on the scalability and timing implications of this logic for machines wider than the 6-wide Golden Cove core modeled? Does the fan-in/fan-out of this logic create a potential timing bottleneck in the rename stage for future, wider designs?

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present "Vegapunk," a software-hardware co-design framework for decoding quantum Low-Density Parity-Check (qLDPC) codes. The core proposal involves an offline, SMT-based strategy to transform a given check matrix into a partially block-diagonal form, purportedly to mitigate quantum degeneracy. This is followed by an online, hierarchical greedy algorithm, implemented on a custom FPGA accelerator, to perform the decoding under a real-time (<1µs) latency constraint. The authors claim that this approach achieves decoding accuracy on par with the state-of-the-art BP+OSD decoder while meeting the strict latency requirements for fault-tolerant quantum computing.

Strengths

1. The work addresses the critical and well-understood trade-off between accuracy and latency in qLDPC decoding, a central challenge for the field.
2. The software-hardware co-design approach is a logical direction for tackling this problem, correctly identifying that algorithmic improvements must be paired with dedicated hardware to meet real-time constraints.
3. The use of an offline pre-computation step to simplify the online decoding problem is a sound principle.

Weaknesses

My primary concerns with this submission relate to the feasibility of the offline step, the significant limitations of the online greedy heuristic, and the overstatement of the empirical results.

1. Computational Feasibility of SMT-based Decoupling: The paper's proposal hinges on an offline SMT optimization to find suitable transformation (T) and permutation (P) matrices (Section 4.2, page 5). The authors claim this "only needs to be performed once per check matrix" (page 6) but provide no data on the computational cost of this step. The search space for these matrices is combinatorially vast, and solving such an SMT problem is likely NP-hard. It is entirely plausible that this offline step is computationally intractable for the very codes for which this decoder is intended. Without reporting the wall-clock time required by the Z3 solver for the benchmarked codes (especially [[784,24,24]]), the entire method's practicality is unsubstantiated.

2. Fundamentally Limited Greedy Heuristic: The online hierarchical algorithm is a greedy search that is hard-capped at a maximum iteration count of M=3 (Section 6.4, page 12). This explicitly limits the search for the "right error" (r) to a maximum Hamming weight of 3. This is a severe and arbitrary constraint. While low-weight errors are most probable, error correction codes are defined by their ability to correct errors up to a certain weight ((d-1)/2). By design, this decoder will fail on any valid correctable error pattern that requires a right-part error of weight 4 or more. Claiming accuracy "on par with BP+OSD" (a far more exhaustive search method) is therefore a significant overstatement. The justification provided in Figure 13 (page 13) merely shows diminishing returns for their specific heuristic; it does not prove that higher-weight errors are irrelevant.

3. Unsupported Complexity Claims vs. Implementation: The complexity analysis (Section 4.4, page 8) claims logarithmic scaling with sufficient parallelism, suggesting P > n*K parallel processing units. For the [[784,24,24]] code, this implies P > 3920 * 14 ≈ 55,000 units. It is highly doubtful that the FPGA implementation reported in Table 4 (page 12) instantiates this many parallel cores. The paper fails to state how many parallel HDUs were actually implemented, creating a disconnect between the theoretical scaling argument and the physical reality of the accelerator that produced the results. The reported latency is likely for a configuration with far less parallelism, rendering the O(log n + S) complexity claim moot.

4. Inconsistent Evaluation and Misleading Comparisons:

   • Accuracy: The LER plots in Figure 10 (page 10) show that Vegapunk is frequently outperformed by the BP+OSD-CS(7) baseline, especially at lower physical error rates which are the target regime for fault tolerance (e.g., plots a1, a3). The claim of being "on par" is not supported by the authors' own data; in several key instances, there is a clear accuracy penalty.
   • Noise Models: The evaluation employs a circuit-level noise model for BB codes but a less realistic phenomenological noise model for HP codes (Section 6.1, page 10). This inconsistency undermines the generality of the conclusions and prevents a fair comparison of performance across the different code families. A rigorous evaluation requires a consistent and realistic noise model for all tested codes.

Questions to Address In Rebuttal

1. Regarding the offline SMT decoupling (Section 4.2): What is the actual wall-clock time required by the Z3 solver for the largest codes presented, namely [[784,24,24]] and [[1488,30,7]]? Please provide evidence that this offline step is computationally feasible for codes relevant to fault-tolerant computing.

2. The online algorithm is limited by M=3, capping the searchable Hamming weight of the "right error" to 3. Please provide a rigorous justification for why this hard limit does not unacceptably compromise decoding accuracy. What is the fraction of correctable errors (within the code's distance) that are missed due to this constraint, and how does this affect the error floor?

3. The parallel complexity analysis (Section 4.4) relies on a number of parallel units (P) that appears unrealistic for an FPGA implementation. Please clarify the exact number of parallel Hierarchical Decoding Units (HDUs) instantiated in the evaluated FPGA design for each code. How does the latency scale if the number of HDUs is fixed while the code size n increases?

4. In several plots in Figure 10 (e.g., a1 for [[72,12,6]] and a3 for [[108,8,10]]), Vegapunk's Logical Error Rate is visibly higher than that of BP+OSD-CS(7). How do the authors reconcile this observable accuracy gap with the claim of being "on par" with the state-of-the-art?

5. Please justify the use of two different noise models (circuit-level for BB codes, phenomenological for HP codes). Would the conclusions for HP codes remain the same if a more realistic circuit-level noise model were applied?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents "Vegapunk," a comprehensive software-hardware co-design framework aimed at solving the critical accuracy-latency tradeoff in the decoding of quantum Low-Density Parity-Check (qLDPC) codes. The central challenge in this domain is that fast decoders like Belief Propagation (BP) are inaccurate due to issues like quantum degeneracy, while accurate decoders like BP with Ordered Statistics Decoding (BP+OSD) are far too slow for the real-time requirements of fault-tolerant quantum computers (typically < 1µs).

Vegapunk's core contribution is a novel two-pronged strategy:

1. Offline SMT-Optimized Decoupling: The authors reframe the problem by performing a one-time, offline transformation of the qLDPC check matrix. Using a Satisfiability Modulo Theories (SMT) solver, they find a transformation that converts the wide, unstructured check matrix into a more manageable form: a series of smaller, independent diagonal blocks and a sparse residual matrix. This pre-computation directly mitigates the quantum degeneracy issue that plagues BP, fundamentally simplifying the online decoding task.

2. Online Hierarchical Greedy Algorithm & Accelerator: The structured matrix produced by the offline step enables a much simpler online decoding algorithm. Instead of complex message passing, Vegapunk uses a hierarchical, greedy search to find the most likely error pattern. The authors then design a dedicated, sparse hardware accelerator that fully exploits the parallelism and sparsity inherent in this new algorithmic structure, allowing it to meet the stringent real-time deadline.

Experimental results demonstrate that Vegapunk achieves decoding latencies under 1µs for significant qLDPC codes (up to [[784,24,24]]) while maintaining a logical error rate comparable to the highly accurate but slow BP+OSD, effectively breaking the existing tradeoff.

Strengths

1. High Conceptual Significance and Impact: This work addresses what is arguably one of the most significant architectural bottlenecks for next-generation quantum computers. The field has largely acknowledged that surface codes have poor scaling overhead, and qLDPC codes are the leading contender to replace them. However, the lack of a practical, real-time decoder has been the primary barrier to their adoption. By providing a plausible and well-evaluated solution that achieves both speed and accuracy, this work could fundamentally alter the trajectory of quantum architecture research, shifting focus toward the practical implementation of these more efficient codes.

2. Novel and Elegant Problem Reframing: The most intellectually compelling aspect of this paper is the offline decoupling strategy (Section 4.2, page 5). The application of SMT solvers—a tool from the formal methods and verification community—to restructure a coding theory problem is a brilliant piece of interdisciplinary thinking. It transforms a difficult, recurring online problem into a (potentially very hard) one-time offline problem and a much simpler online one. This "pre-computation" approach is an elegant way to manage complexity and is a powerful design pattern that could inspire solutions in other areas.

3. Holistic Full-Stack Approach: The authors demonstrate a mature understanding of the problem by providing a full-stack solution. They do not merely propose a new algorithm in isolation; they consider its hardware mapping from the outset. The co-design of the hierarchical algorithm with the sparse accelerator (Section 5, page 8) is crucial. This demonstrates a deep appreciation for the fact that in the realm of quantum computer architecture, algorithms and hardware are inextricably linked. This end-to-end perspective, from abstract matrix transformation to FPGA implementation details, makes the work far more credible and impactful.

4. Strong and Well-Contextualized Empirical Evaluation: The experimental results are thorough and convincing. The authors benchmark against the correct state-of-the-art baselines (BP for speed, BP+OSD for accuracy) using relevant and challenging qLDPC code families (BB and HP codes). The demonstration of sub-microsecond latency for a large [[784,24,24]] code is a headline result that will capture the community's attention. The LER curves in Figure 10 (page 10) clearly show that they have achieved their goal of matching BP+OSD's accuracy, solidifying their central claim.

Weaknesses

While the core idea is powerful, the paper could be strengthened by addressing the broader implications and potential limitations of its approach.

1. Scalability and Cost of the Offline Step: The SMT-based decoupling is the "secret sauce," but the paper provides limited discussion on the practical cost of this step. SMT problems can have exponential complexity. While performing a difficult computation offline is acceptable, it is crucial to understand its limits. How long did the Z3 solver take for the [[784,24,24]] code? How will this scale to the even larger codes required for full-scale fault tolerance? A discussion of the computational complexity and resource requirements of the offline step would provide a more complete picture of the framework's practicality for future codes.

2. Generality of the Decoupling Strategy: The method is demonstrated successfully on BB and HP codes. The authors provide some intuition for why a block-diagonal structure can be found for these specific code families (Section 4.2, "Caveats of Check Matrix Decomposition," page 6). However, the broader applicability of this SMT formulation to any arbitrary qLDPC code remains an open question. Is there a risk that for some code families, the SMT solver might fail to find a useful, sparse decomposition? A more general discussion on the conditions under which this method is expected to succeed would be valuable.

3. Limited Intuition on the Online Heuristic: The online algorithm is a greedy search with a very small maximum iteration count (M=3). The ablation study in Figure 13 (page 13) empirically justifies this choice by showing diminishing returns. However, the paper could offer a deeper theoretical or intuitive explanation for why such a simple heuristic is so effective. Is it purely that most correctable errors have low weight, or does the SMT decoupling process structure the problem in such a way that the residual syndrome becomes "easy" to solve greedily? Connecting the success of the online algorithm more explicitly to the properties of the offline transformation would strengthen the narrative.

Questions to Address In Rebuttal

1. Could the authors please provide data on the runtime and resource usage of the offline SMT decoupling step for the largest codes tested? What are their projections for how this offline cost will scale as qLDPC codes grow in size in the future?

2. Can the authors comment on the expected generality of their SMT-based decoupling? Are there known properties of qLDPC check matrices (e.g., related to their construction) that would make them more or less amenable to this transformation?

3. The success of the simple, low-iteration greedy search is a key enabler for the framework's speed. Could the authors provide more intuition as to why this approach is sufficient? Does the offline decoupling effectively "pre-solve" the hardest parts of the problem, leaving a residual search space that is easily navigable by a greedy algorithm?

4. On a lighter note, is there a specific inspiration behind the name "Vegapunk"? It is a distinctive and memorable choice.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper, "Vegapunk," presents a software-hardware co-design framework for decoding quantum Low-Density Parity-Check (qLDPC) codes. The authors claim to achieve both high accuracy (comparable to BP+OSD) and low, real-time latency (< 1µs), addressing a well-known trade-off in the field. The central thesis is that a computationally intensive offline pre-processing step can transform the qLDPC check matrix into a structure that enables a very simple, fast, and parallelizable online decoding algorithm.

The core of the proposed novelty lies in using Satisfiability Modulo Theories (SMT) solvers to perform this offline matrix transformation. This "decoupling" yields a matrix with a block-diagonal structure and a sparse remainder. The online decoding algorithm is a hierarchical, greedy search that first guesses the error components corresponding to the sparse part of the matrix and then directly solves for the errors related to the block-diagonal part. A dedicated accelerator is designed to be a direct hardware mapping of this online algorithm.

While the constituent concepts (greedy algorithms, matrix transformations, hardware acceleration) are not new in isolation, their synthesis, particularly the use of an SMT solver to find an optimal matrix structure for decoding, appears to be a genuinely novel approach in this domain.

Strengths

1. Novel Offline Formulation: The primary and most significant novel contribution is the offline, SMT-optimized decoupling strategy (Section 4.2, page 5). While SAT/SMT solvers have been explored in classical coding theory for tasks like finding minimum-weight codewords, their application here is fundamentally different and new. The authors are not using the SMT solver to decode, but rather to restructure the decoding problem itself. Formulating the search for optimal transformation (T) and permutation (P) matrices as a constrained optimization problem for an SMT solver (Figure 5, page 5) is a clever and, to my knowledge, unprecedented method for preparing a qLDPC code for hardware-accelerated decoding.

2. Synergistic Algorithm-Architecture Co-Design: The novelty is reinforced by the tight coupling between the offline transformation and the online algorithm. The online hierarchical greedy algorithm (Section 4.3, page 6) is specifically enabled by the structure created by the SMT solver. While greedy search is a standard heuristic, its application here—guessing the "right error" r to simplify the solution for the "left error" l—is a direct consequence of the D' = ([diag(D_i)], A) decomposition. This demonstrates a strong system-level novelty where the offline step creates a problem that the simple online step is uniquely suited to solve.

3. Complexity Shifting as an Architectural Principle: The work presents a clear and novel architectural trade-off: shifting immense computational complexity from the time-critical online path to a one-time, offline pre-computation. This is a powerful design pattern, and its application to the qLDPC decoding problem appears to be a new and insightful contribution.

Weaknesses

My concerns are not that the core idea has been done before, but rather with the characterization and evaluation of the novel components themselves.

1. Under-explored Scalability of the Core Novelty: The central claim hinges on the offline SMT step, yet the paper provides zero data on its computational cost. SMT is NP-hard. The "Caveats of Check Matrix Decomposition" section (page 6) briefly discusses the process but completely omits any discussion of the SMT solver's runtime. How long did it take to find the transformation matrices for the [[784,24,24]] BB code? Hours? Days? Weeks? Does the solver time scale polynomially or exponentially with the code size? Without this information, the practicality of the entire approach for future, much larger qLDPC codes remains a major open question. The novelty is significant, but its viability is not established.

2. Limited Generalizability of the Novel Method: The SMT formulation is evaluated only on highly structured Bivariate Bicycle (BB) and Hypergraph Product (HP) codes. As noted on page 6, the structure of these codes provides hints for the decomposition (e.g., K = max(min(l, m), l x m) for BB codes). It is unclear whether the SMT-based approach is a general tool or if its success is an artifact of the inherent algebraic structure of these specific code families. A truly novel method should demonstrate robustness on less structured or more generic qLDPC code constructions.

3. Incremental Novelty of the Online Algorithm: The online hierarchical algorithm, when viewed in isolation, is a simple greedy search. Its novelty is almost entirely derived from the pre-transformed matrix it operates on. Conceptually, the strategy of guessing a small number of error bits to resolve a syndrome is related to the post-processing in Ordered Statistics Decoding (OSD) or decimation strategies used in other decoders (e.g., BPGD). The paper should more clearly articulate the fundamental delta between its "guess-and-solve" method and these prior conceptual frameworks, beyond simply stating that it operates on a different matrix structure.

Questions to Address In Rebuttal

The authors should address the following points to solidify the significance of their novel contributions:

1. Scalability of the SMT Solver: Please provide the runtime of the Z3 SMT solver for the largest BB code ([[784,24,24]]) and the largest HP code ([[1488,30,7]]) presented in the paper. Can you provide any data or theoretical arguments regarding how this runtime scales with n (number of qubits)?

2. Generalizability: Have you attempted to apply your offline SMT decoupling strategy to other families of qLDPC codes that lack the clean algebraic structure of BB or HP codes? If the method failed or was too slow, this would be crucial information for understanding the boundaries of this novel technique.

3. Algorithmic Distinction: Please provide a more detailed comparison between the online hierarchical algorithm and decimation-based approaches like BPGD [48]. Both involve iteratively making high-confidence "guesses" to simplify the remaining problem. What is the fundamental algorithmic innovation in your online approach beyond the fact that it leverages the pre-computed matrix structure?

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose OneAdapt, a compilation framework for measurement-based quantum computing (MBQC) on photonic hardware. The work introduces a new intermediate representation (IR) that extends the prior FlexLattice IR by (1) enforcing a bound on the length of temporal edges and (2) allowing "skewed" temporal edges between nodes at nearby 2D locations on different layers. The compiler includes two main optimization passes: a "dynamic refresh" mechanism to manage node lifetime and a "2D-bounded temporal routing" algorithm to leverage the new skewed edges. The authors claim significant reductions in execution time (1D depth) compared to both a modified version of the OnePerc compiler and a more rigid cluster-state-based approach. The framework's adaptability to various hardware constraints is evaluated, along with a preliminary extension to fault-tolerant quantum computing (FTQC) using surface codes.

Strengths

1. Problem Motivation: The paper correctly identifies a critical gap in existing MBQC compilers for photonic systems. The tension between the rigidity of cluster states and the potentially unbounded resource requirements of the more flexible FlexLattice IR is a well-articulated and important problem.
2. IR Design: The proposed IR features—bounded-length and skewed temporal edges—are well-grounded in the physical realities and capabilities of fusion-based photonic architectures. Capturing these hardware characteristics at the IR level is a sound design choice.
3. Core Algorithm Concept: The central idea of "dynamic refresh" (Section 4.3) is a logical and more granular alternative to the "periodic refresh" strategy from prior work. Performing refreshes on an as-needed basis, prioritized by computational relevance, has clear potential to avoid the overhead of dedicated refresh layers.

Weaknesses

My primary concerns with this paper relate to the justification for key claims, the strength of the experimental baselines, and the oversimplification of critical hardware effects.

1. Unsupported Claims Regarding Skewed Edge Implementation: The paper makes the strong claim that skewed edges can be implemented with "a slight algorithm modification, without requiring additional hardware capabilities" (Section 3.1, page 5) and that the associated overheads are "negligible" (Section 4.5, page 9). This is insufficiently substantiated.

   • PL Ratio: The analysis in Section 5.5 (Figure 12) assesses the Physical-to-Logical (PL) ratio overhead by randomly selecting skewed edges. This is not representative of a real compilation scenario where structured algorithms create deterministic and potentially conflicting routing patterns (i.e., routing hotspots). The conclusion that the overhead is negligible is therefore unconvincing.
   • Fidelity: The authors acknowledge that skewed edges may require longer fusion paths, which degrades fidelity. They then hand-wave this concern away by arguing that the reduction in 1D depth reduces the total edge count, thus "improving the overall fidelity" (Section 4.5, page 9). This is a qualitative argument without any quantitative backing. A paper focused on hardware-adaptive compilation must include a concrete fidelity model to properly evaluate such trade-offs. The absence of one is a major flaw.

2. Potentially Weak or Unfair Baselines: The impressive performance gains reported (e.g., 3.48x in Table 1) are questionable due to the construction of the baselines.

   • Baseline 1 (OnePerc): The authors modify OnePerc by forcing it to perform periodic refreshes and restricting its scheduling (Section 5.1, page 9) to make it "more comparable." This appears to be a post-hoc modification that forces the baseline into a regime it was not designed for, potentially crippling its performance. The data in Table 1 (page 10) shows OnePerc's compiled temporal edge length (Df (compiled)) far exceeds the target Df = 20, while OneAdapt meets it. This does not demonstrate that OneAdapt is 3.48x faster; it demonstrates that OneAdapt can satisfy a constraint that the authors' modified version of OnePerc cannot. This is a much weaker claim.
   • Baseline 2 (Qiskit to Cluster State): Compiling a circuit to a rigid 3D cluster state is a known-to-be-inefficient method that flexible compilers like OnePerc and OneAdapt are explicitly designed to outperform. While useful as a sanity check, the large 6.7x improvement over this baseline is expected and does not constitute a strong result on its own.

3. Over-reliance on Empirically Tuned Heuristics: The compiler's performance hinges on several heuristics. The refresh prioritization scheme (Section 4.3, page 7) is reasonable but not deeply analyzed for failure modes. More critically, the "Refresh Percentage Tuning" (Equation 1, page 8) relies on a parameter p, which is empirically set to 0.4 based on the data in Table 3 (page 12). This suggests the system is sensitive and has been tuned for the specific benchmarks tested, raising questions about its generalizability.

4. Superficial FTQC Analysis: The extension to FTQC (Section 5.6, page 12) is underdeveloped. The baseline is a "static strategy that interleaves QEC patches uniformly." This appears to be a strawman. The field of FTQC compilation has more sophisticated resource management schemes. Without comparing against a stronger, state-of-the-art dynamic scheduling baseline, the claimed 3.33x improvement is not credible.

Questions to Address In Rebuttal

1. On Skewed Edges: Please provide a quantitative analysis of the physical overheads of skewed edges.

   • a) Re-evaluate the PL ratio using the specific, deterministic routing patterns generated for the benchmark circuits in your evaluation, not random sampling.
   • b) Introduce a simple, explicit fidelity model (e.g., constant depolarizing error per fusion) and show how the final logical fidelity is affected by the trade-off between longer paths for skewed edges and a lower total 1D depth.

2. On Baselines: Please provide a stronger justification for your choice and modification of the OnePerc baseline. Is it not possible to configure OnePerc in a different manner to more effectively handle temporal edge constraints, even if not explicitly bounded? A fair comparison requires demonstrating that you are comparing against the baseline operating in a reasonable, if not optimal, configuration.

3. On Heuristics: With respect to the refresh percentage bound p, how sensitive is the compiler's performance to this parameter? Please provide data showing how the performance improvements hold up across a wider range of p values and justify why p=0.4 is a robust choice and not simply an artifact of overfitting to the selected benchmarks.

4. On FTQC: Please provide citations and justification that your "static strategy" baseline for FTQC scheduling is representative of a state-of-the-art approach. If it is not, please explain why a more advanced dynamic scheduling algorithm was not used for comparison.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces OneAdapt, a resource-adaptive compiler for Measurement-Based Quantum Computing (MBQC) specifically tailored for photonic hardware. The work identifies critical limitations in prior Intermediate Representations (IRs), such as the FlexLattice IR, which lack adaptivity to realistic hardware constraints. The core contribution is a novel, more hardware-aware IR and two associated optimization passes designed to bridge this gap.

The new IR extends FlexLattice in two significant ways: 1. It enforces a bound on the length of temporal edges, directly modeling the physical constraint of finite photon delay lines. 2. It introduces skewed temporal edges, which connect nodes at nearby but different 2D spatial locations across time steps, exploiting a latent capability of fusion-based architectures.

To manage and leverage this new IR, the authors propose two key compiler optimizations: 1. Dynamic Refreshing: An intelligent, on-demand node refresh mechanism that prevents temporal edge lengths from exceeding the hardware-imposed limit, in contrast to less efficient periodic refresh schemes. 2. 2D-Bounded Temporal Routing: A routing algorithm that utilizes the new skewed edges to achieve more efficient mappings, reducing both the required 2D hardware area and the 1D temporal depth of the computation.

The paper evaluates OneAdapt against both the state-of-the-art photonic compiler (OnePerc) and a circuit-model-based compiler (Qiskit), demonstrating significant improvements in execution depth while respecting hardware constraints. The framework is also extended to the fault-tolerant setting, showing substantial depth reduction for surface code implementations.

Strengths

The primary strength of this paper is its thoughtful and pragmatic approach to co-designing a quantum compiler IR with the realities of a promising hardware platform. It successfully moves the compilation stack for photonic MBQC from a realm of idealized assumptions toward practical implementation.

1. Clear Context and Problem Formulation: The paper does an outstanding job of contextualizing its contribution. Figure 1 (page 2) is particularly effective, providing a concise visual history of MBQC IRs and clearly positioning this work as the next logical step in balancing expressive power with hardware feasibility. The motivation is compelling and grounded in real physical limitations (finite delay lines) and opportunities (underutilized fusion pathways).

2. Significant Technical Contributions: The two core ideas are both novel and impactful.

   • The concept of dynamic refresh is an elegant solution to the bounded-delay problem. By refreshing nodes based on their individual "time-to-live" in a delay line rather than through rigid, periodic cycles, the compiler can minimize the overhead associated with maintaining quantum states over time.
   • The introduction and exploitation of skewed edges is a keen architectural insight. Recognizing that the underlying fusion-based hardware can support these connections without modification and then building a compiler pass to leverage them for routing is a prime example of effective hardware/software co-design. It unlocks a new dimension for optimization that was previously ignored.

3. Demonstrated Resource Adaptivity: The paper's central claim of "resource-adaptivity" is well-supported by the evaluation (Section 5, starting page 9). The experiments showing trade-offs between 2D hardware size and the available delay line length (varying Df in Figure 10, page 11) are crucial. This capability elevates the compiler from a mere translator to a vital tool for architects exploring the vast design space of future photonic systems.

4. Forward-Looking Scope (FTQC): The inclusion of a study on fault-tolerant quantum computing (FTQC) using surface codes (Section 5.6, page 12) is a significant strength. It demonstrates that the proposed techniques are not limited to the NISQ era but provide a scalable path toward fault tolerance, which is the ultimate goal. The reported 3.33x depth reduction in this context is highly promising.

Weaknesses

The weaknesses of the paper are primarily related to the scope of its analysis and potential missed connections, rather than fundamental flaws in the core ideas.

1. Hardware Model Fidelity: The paper's model of the photonic hardware, while more realistic than its predecessors, is still an abstraction. The crucial PL Ratio (Physical-to-Logical layer ratio) is treated as a fixed parameter based on prior simulations. However, the effectiveness of the proposed routing strategies, especially for skewed edges, likely depends heavily on the actual connectivity of the physical layer after probabilistic fusions. A discussion on how the compiler's performance degrades as the fusion success rate drops (and the physical graph becomes sparser) would add significant depth and realism. Section 5.5 touches on this but could be more integrated with the main results.

2. Narrow Focus on Fusion-Based MBQC: The work is situated entirely within the fusion-based MBQC paradigm. While this is a leading approach for photonics (e.g., PsiQuantum), it is not the only one. Other paradigms, such as continuous-variable (CV) cluster state generation or direct circuit-model implementations on different photonic architectures, exist. A brief acknowledgment of these alternatives in the introduction or related work would help to better delineate the specific domain of the paper's contribution and strengthen its overall positioning within the broader landscape of photonic quantum computing.

3. Fidelity vs. Resource Costs: The evaluation focuses exclusively on architectural metrics: 1D depth, 2D size, and temporal edge length. However, the introduction of skewed edges, as the authors briefly note in Section 4.5 (page 9), may require longer physical fusion paths, potentially leading to lower fidelity per logical edge. The paper lacks a quantitative analysis or even a qualitative discussion of this critical trade-off. A 3x reduction in depth is less compelling if it comes at the cost of a 10x increase in the logical error rate.

Questions to Address In Rebuttal

1. The effectiveness of 2D-bounded temporal routing relies on finding connected paths in the physical substrate. How sensitive are the reported depth and size improvements to the fusion success rate? Is there a percolation threshold below which the advantage of skewed edges diminishes because the required skewed paths are rarely available?

2. The dynamic refresh mechanism is governed by a refreshing bound br, which is tuned by the parameter p (Equation 1, page 8). The paper identifies p=0.4 as a "sweet spot." Could the authors elaborate on the methodology used to determine this value? Furthermore, how sensitive are the results to variations in p, and could an adaptive scheme that dynamically tunes p based on program characteristics yield even better performance?

3. Regarding the implementation of skewed edges (Section 4.5), the paper argues that they can lead to longer fusion paths and fidelity degradation. Can the authors provide a more concrete analysis of this trade-off? For instance, for a given reduction in 1D depth, what is the estimated increase in the total number of physical fusions required, which could serve as a proxy for fidelity cost? This would provide a more holistic view of the optimization's overall benefit.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents OneAdapt, a compiler for Measurement-Based Quantum Computing (MBQC) targeting photonic hardware. The core contribution is twofold: 1) an extension of the Intermediate Representation (IR) from prior work, and 2) two new compilation passes designed to leverage this extended IR. Specifically, the authors extend the FlexLattice IR, introduced in OnePerc [74], by incorporating two new features: a hard constraint on the length of temporal edges and the allowance of "skewed" temporal edges that connect nodes at nearby, but not identical, 2D locations on different time-like layers. The new compilation passes, "Dynamic Refreshing" and "2D-bounded Temporal Routing," are designed to enforce the length constraint and exploit the skewed edges, respectively. The authors claim this new framework leads to significant reductions in program depth and hardware requirements compared to prior art.

Strengths

The paper's primary strength lies in its clearly articulated and well-justified novelty. The contributions are not presented in a vacuum but as a direct and intelligent evolution of a specific, state-of-the-art predecessor (OnePerc [74]).

1. Novelty of the IR Extension: The introduction of "skewed edges" is a genuinely novel concept in the context of MBQC compilation for photonic systems. While routing is a general concept, embedding this specific form of spatially-offset temporal connectivity directly into the IR is a clever architectural insight. It correctly identifies a latent capability in the underlying fusion-based hardware model—that path-finding between resource states is not strictly limited to a vertical "stack"—and proposes a formal IR feature to exploit it. This is a significant conceptual leap beyond the strictly columnar temporal connections in the original FlexLattice IR.

2. Algorithmic Novelty: The proposed "Dynamic Refreshing" mechanism (Section 3.2, page 6) is a substantial improvement over the "periodic refreshing" from OnePerc. The latter is a brute-force, synchronous method, whereas the proposed dynamic approach is an asynchronous, needs-based, and more granular scheduling algorithm. The use of a feedback mechanism based on the number of constraint-driven refreshes (Section 4.4, page 8, equation 1) to tune the refresh-to-computation ratio is a sophisticated and novel heuristic in this domain. This represents a significant increase in the algorithmic elegance for managing a critical resource constraint (photon storage time).

3. Synergistic Design: The two primary novelties (the extended IR and the new passes) are not independent but work in synergy. The skewed edges are not merely an addition; they are the feature that enables the more powerful 2D-bounded temporal routing. This tight coupling between the IR design and the compiler optimizations demonstrates a mature co-design approach, which itself is a valuable contribution. The resulting performance gains are not marginal; a 3.48x average depth reduction over OnePerc (Table 1, page 10) is substantial and directly validates the benefit of the novel ideas presented.

Weaknesses

My concerns are not with the existence of novelty, but with the rigor used to justify the practicality and scope of the novel claims.

1. Under-substantiated Feasibility of Skewed Edges: The entire premise of the skewed edge novelty rests on the claim that its implementation requires minimal overhead. Section 4.5 (page 9) states that a skewed IR edge "can be formed easily by a skewed fusion path" and that the only required change is "to allow path searching between qubits corresponding to IR nodes at nearby 2D locations." This assertion is the lynchpin for the paper's most innovative feature, yet it is treated with surprising brevity. Allowing paths to deviate from a straight column could significantly increase the complexity and runtime of the (2+1)-D path searching algorithm used for renormalization. It could also increase the likelihood of routing conflicts between different logical edges, potentially increasing the physical-to-logical layer ratio (PL Ratio). While the paper claims in Section 5.5 (page 11) that this effect is "negligible" for a skew distance of 1, this feels more like an empirical observation than a rigorous justification. The novelty is strong, but its practical foundation feels shaky.

2. Arbitrary Limitation on Novelty: The skewed edges are restricted to a Hamming distance of 1. While this is a practical choice that delivers good results, the paper does not explore the reasoning behind this specific limit. Is this a fundamental constraint imposed by the physics or connectivity of the hardware, or is it merely the first parameter value the authors tried? The novelty of the skewed edge concept would be significantly strengthened by a characterization of the design space. An analysis of the trade-offs (e.g., impact on PL Ratio, path-finding complexity, potential for 1D depth reduction) for a skew distance > 1 would provide a much deeper understanding of this new IR feature. As it stands, the innovation feels like a single point-solution rather than the introduction of a new, well-understood architectural knob.

Questions to Address In Rebuttal

1. The feasibility of skewed edges is central to this work. Can the authors elaborate on the algorithmic complexity of the modified (2+1)-D path searching? Does searching a larger volume for paths for each logical edge measurably increase the runtime of the renormalization step, which is a critical part of the real-time hardware operation? Can you provide stronger evidence that path congestion and the PL Ratio are not adversely affected in denser, more complex programs than those in the benchmark suite?

2. Regarding the scope of the skewed edge concept: Please justify the choice of a Hamming distance of 1. Is this limit based on a physical constraint in the fusion-based architecture model, or is it an empirical choice? A brief discussion on the projected overheads and potential benefits of allowing a larger skew distance would help establish the generality of this novel contribution.

3. The dynamic refresh algorithm is an interesting scheduling solution that prioritizes nodes based on their computational relevance and storage time. This bears a conceptual resemblance to deadline-driven scheduling in real-time systems or advanced cache/register management in classical compilers. Was this novel approach inspired by prior art in other domains of computer science? Placing this quantum compilation technique in a broader context could help clarify its fundamental contribution.

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose a compilation framework, MUSS-TI, tailored for a hypothetical trapped-ion architecture they term EML-QCCD. This architecture segregates trap regions into specialized zones (storage, operation, optical). The compiler employs a multi-level scheduling heuristic, analogous to classical memory hierarchies, to manage qubit movement between these zones. The primary claims are significant reductions in shuttle operations and execution time compared to existing QCCD compilers, leading to improved final state fidelity.

Strengths

1. The paper identifies a relevant problem: compiling for modular, zoned trapped-ion architectures is a critical next step for scalability.
2. The inclusion of an ablation study (Section 5.4, Figure 8) to dissect the performance contribution of different components (mapping vs. SWAP insertion) is a methodologically sound practice.

Weaknesses

1. Unfair Baseline Comparison: The central weakness of this work is the comparison of a specialized compiler (MUSS-TI) on its target specialized architecture (EML-QCCD) against baseline compilers [13, 55, 70] designed for generic QCCD grids. The baselines are not zone-aware and are thus fundamentally disadvantaged. The reported performance gains are therefore unsurprising and likely exaggerated. A rigorous evaluation would require adapting the baseline heuristics to be zone-aware or demonstrating MUSS-TI's superiority on a standard, non-zoned architecture.
2. Oversimplified and Potentially Biased Fidelity Model: The conclusions regarding fidelity are heavily dependent on the chosen model (Section 4, page 7). The quadratic decay of two-qubit gate fidelity with ion number (1 - €N^2) directly drives the conclusion of an "optimal" trap size in Section 5.3. The fixed fidelity for remote entanglement (0.99) is highly optimistic. The heating model (-kn) is simplistic and lacks detailed justification for the chosen parameter k=0.001. The paper fails to provide a sensitivity analysis of its results with respect to these crucial, and debatable, modeling assumptions.
3. Heavily Parameterized Heuristics: The SWAP insertion strategy (Section 3.3, page 6) relies on manually-tuned "magic numbers" (k=8, T=4). The paper provides scant justification for these specific values and does not explore how performance changes with different parameter choices. The claim that k can be adjusted based on "an understanding of the locality of the input circuits" is unsubstantiated and not demonstrated.
4. Hypothetical Architecture: The work is predicated on the EML-QCCD architecture, which, while plausible, is presented without a detailed hardware analysis. Claims that it is "more achievable" than other proposals like TITAN (page 4, sec 2.3) are asserted rather than proven. The compiler's performance is inextricably linked to this architecture's specific layout, making the results less generalizable.
5. Strained Analogy: The framing of the problem as analogous to "multi-level memory scheduling" (Section 3, page 4) is more of a narrative convenience than a technically rigorous mapping. The physical realities of ion shuttling—high latency, induced heating, and connectivity constraints—differ fundamentally from data movement in classical memory hierarchies, and the analogy may obscure more than it clarifies.

Questions to Address In Rebuttal

1. How would you justify the fairness of comparing your zone-aware compiler against zone-unaware baselines? Can you provide results for an experiment where the baseline algorithms are modified to be aware of the EML-QCCD zones, or where MUSS-TI is benchmarked against them on a standard QCCD grid architecture?
2. Please provide a sensitivity analysis for the key parameters in your fidelity model (€, k) and your SWAP insertion heuristic (k, T). How robust are your claimed improvements to variations in these assumptions? Specifically, how do the results change if the fiber entanglement fidelity is lowered from the optimistic 0.99?
3. The ablation study (Figure 8) suggests that the SABRE-style mapping provides the vast majority of the performance improvement. Can you quantify the individual contributions of the LRU replacement policy and the multi-level scheduling heuristic, independent of the SABRE mapping? This would clarify whether the core "MUSS" concept is truly the main driver of performance.
4. Can you elaborate on the claim that the EML-QCCD architecture is "more readily achievable" (page 3)? What specific fabrication or control challenges present in architectures like TITAN are circumvented by this design, and what new challenges does it introduce?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces MUSS-TI, a compilation framework designed for a promising class of large-scale trapped-ion quantum computers: Entanglement Module Linked Quantum Charge-Coupled Devices (EML-QCCD). The core and most compelling contribution of this work is the application of a classical computer architecture concept—the multi-level memory hierarchy—to the complex problem of quantum circuit compilation. The authors intelligently map the distinct functional zones of the EML-QCCD architecture (storage, operation, and optical zones) to different levels of a memory hierarchy. By doing so, they can leverage well-established scheduling policies, such as Least Recently Used (LRU), to manage the movement (shuttling) of qubits. This elegant analogy allows their compiler to make informed decisions about where and when to move qubits, with the primary goal of minimizing the costly shuttling operations that introduce latency and decoherence. The paper provides a comprehensive evaluation showing significant reductions in shuttle counts and execution time, particularly for medium- and large-scale applications, thereby making a strong case for the viability of both the proposed compiler and the underlying EML-QCCD architecture.

Strengths

1. Powerful and Elegant Central Analogy: The single greatest strength of this paper is its central idea: framing the qubit scheduling problem as a memory hierarchy management problem. This is a beautiful piece of conceptual synthesis. It takes a notoriously complex, multi-variable optimization problem in the quantum domain and maps it to a problem space that has been deeply studied for decades in classical computer architecture. This not only provides an intuitive framework for reasoning about qubit locality and movement but also unlocks a rich toolbox of existing scheduling heuristics (like LRU, as demonstrated here). This reframing is a significant intellectual contribution.

2. Architectural Foresight and Relevance: The work is not performed in a vacuum; it directly addresses the software challenges of a highly plausible next-generation hardware architecture. The EML-QCCD model, as described in Section 1 and Figure 2 (page 3), represents a credible path toward scaling trapped-ion systems by modularizing them. By developing a compiler specifically for this architecture, the authors are engaging in crucial hardware-software co-design. This work anticipates the needs of hardware experimentalists and provides a foundational software layer that will be necessary to make such systems programmable and efficient. It connects directly to the trend of building distributed and modular quantum systems, as seen in works like TITAN [11] and various experimental demonstrations of photonic links.

3. Comprehensive and Scalable Evaluation: The evaluation is thorough and persuasive. The authors benchmark MUSS-TI against several relevant prior works across small, medium, and large-scale applications. The results presented in Table 2 (page 8) and Figure 6 (page 8) consistently show dramatic improvements in shuttle count and execution time. Furthermore, the analysis extends beyond simple metrics to include crucial investigations into the impact of trap capacity (Section 5.3, page 9) and an ablation study of the compiler's own techniques (Section 5.4, page 9), lending significant weight to their conclusions. This demonstrates that the benefits are not just theoretical but are robust across different system parameters and application types.

4. Bridging Disciplinary Gaps: This paper serves as an excellent bridge between the fields of quantum computing and classical computer architecture. It demonstrates that the challenges emerging in scalable quantum systems are not entirely alien; rather, they are new instantiations of fundamental computer science problems related to locality, communication, and resource management. This work can inspire further cross-pollination of ideas, which will be essential for building the full quantum computing stack.

Weaknesses

While the core idea is strong, the work could be further contextualized and its underlying assumptions explored more deeply. These are not fatal flaws but rather opportunities for strengthening the work.

1. Simplification of the Cost Model: The fidelity model presented in Section 4 (page 7) is a necessary and reasonable simplification for a compiler-level study. However, the costs of the different "memory accesses" are highly complex. For instance, the error mechanisms of an intra-QCCD MS gate (in the "operation zone") are very different from those of a photonically mediated remote entanglement gate (in the "optical zone"). The current framework appears to treat the hierarchy as a linear progression of cost/speed, but the reality might be more nuanced. The paper would be strengthened by a discussion of how the framework might adapt if, for example, the fidelity of remote gates improved dramatically, potentially changing the optimal scheduling strategy.

2. Limited Exploration of the Analogy's Full Potential: The authors successfully apply the concept of a memory hierarchy and an LRU replacement policy. However, the analogy has much deeper potential. Classical systems employ sophisticated techniques like prefetching, different write-back/write-through policies, and dynamic cache partitioning. While beyond the scope of this initial work, a discussion of how these more advanced concepts might translate to the qubit scheduling problem would elevate the paper's vision and impact. For instance, could static analysis of the quantum circuit's dependency graph (DAG) enable predictive "qubit prefetching"?

3. Scalability of the Classical Compilation: The paper focuses on the performance scalability of the quantum circuit execution. The analysis of the compiler's own classical runtime (Section 5.6, page 10) is present but brief. The observed spikes in compilation time in Figure 10 suggest that for extremely large circuits, the classical overhead of making these sophisticated scheduling decisions could become non-trivial. This is a common challenge in advanced compilation, but it warrants a more detailed discussion about the trade-offs and potential bottlenecks in the classical control system.

Questions to Address In Rebuttal

1. The memory hierarchy analogy is the paper's most significant contribution. Could the authors elaborate on how this analogy might be extended? For instance, could data-flow analysis of the circuit's DAG be used to implement a form of "qubit prefetching," where qubits are speculatively moved to higher-level zones (e.g., from storage to operation) in anticipation of their use, potentially hiding shuttle latency?

2. The principles of MUSS-TI are developed for the EML-QCCD architecture. How generalizable is this multi-level scheduling concept? Could a similar framework be applied to other emerging modular architectures, such as networked superconducting processors with different tiers of coupler speeds, or neutral atom arrays with physically separate "storage" and "interaction" zones?

3. The current work prioritizes minimizing shuttling, which is a key bottleneck. However, the cost landscape is dynamic. How would the MUSS-TI framework adapt if future hardware advancements dramatically changed the relative costs of operations—for example, if fiber-based remote entanglement (Level 2) became significantly faster or higher fidelity than local MS gates (Level 1)? Does the framework allow for flexible cost functions to re-prioritize scheduling decisions based on evolving hardware realities?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present MUSS-TI, a compiler framework designed to optimize shuttle scheduling for a large-scale, modular trapped-ion architecture termed Entanglement Module Linked Quantum Charge-Coupled Device (EML-QCCD). The central claim to novelty lies in the application of a classical multi-level memory hierarchy analogy to the problem of qubit movement. Specifically, the authors map the distinct functional zones of the EML-QCCD architecture—storage, operation, and optical zones—to different levels of a memory hierarchy (L0, L1, and L2/CPU, respectively). This conceptual framework is then used to drive a scheduling algorithm that employs policies directly inspired by classical cache management, such as a Least Recently Used (LRU) policy for qubit "eviction" from high-value zones. This is complemented by a tailored, lookahead-based SWAP gate insertion heuristic to manage qubit placement across different QCCD modules. The paper claims this new approach significantly reduces shuttle operations and improves overall circuit fidelity compared to existing compilers.

Strengths

1. Novelty of the Core Conceptual Framework: The primary strength of this paper is the successful and novel application of a well-understood concept from classical computer architecture (multi-level memory/cache hierarchies) to a fundamentally different domain (quantum ion shuttling). While prior works [13, 55, 69] have developed heuristics for shuttle optimization, they have not, to my knowledge, explicitly formalized the problem using this powerful analogy. The mapping of physical zones to memory levels, as detailed in Section 3 (page 4), provides a new and intuitive lens through which to structure the optimization problem. This conceptual bridge is the paper's most significant and original contribution.

2. Effectiveness of the Novel Heuristics: The new perspective is not merely a semantic relabeling; it directly inspires effective algorithmic choices. The use of an LRU replacement policy for managing limited space in the operation/optical zones is a direct and logical consequence of the cache analogy. This appears to be a genuinely new approach for qubit management in this context. The substantial performance gains reported in Table 2 and Figure 6 (page 8), which are far from marginal, serve as strong evidence that this novel conceptual framework leads to a superior class of scheduling heuristics.

Weaknesses

1. The Architectural Premise is Evolutionary, Not Revolutionary: The EML-QCCD architecture, which underpins the entire scheduling model, is itself not a wholly new invention. The concept of functionally distinct zones (storage, interaction, readout) has been a cornerstone of QCCD proposals since the original work by Kielpinski, Monroe, and Wineland [30]. Similarly, the use of photonic interconnects to link modules is a widely explored avenue for scaling [1, 33, 61]. The paper is compiling for a specific, plausible, and well-motivated instantiation of these existing ideas. The novelty lies in the compiler, not the hardware concept, and this distinction could be made clearer. The contribution is a novel solution for an evolved architecture, not the invention of the architecture itself.

2. Supporting Techniques are Adaptations of Prior Art: While the synthesis is novel, several key components of the algorithm are direct adaptations of existing techniques.

   • LRU Policy: The qubit replacement scheduler (Section 3.2, page 5) is explicitly an LRU policy, a foundational algorithm in classical OS and architecture. Its application here is novel, but the algorithm itself is not.
   • SWAP Insertion: The use of a lookahead-based search to determine the utility of inserting SWAP gates is conceptually similar to established techniques in qubit routing for other platforms, most notably SABRE [37], which the authors also adapt for initial mapping. The novelty in their SWAP insertion method (Section 3.3, page 6) is confined to the specific cost metric (W(qi, cj)) tailored for their inter-module architecture, which is an incremental, albeit useful, advancement.

The paper’s main achievement is the integration of these ideas under a new, cohesive framework, rather than the invention of each individual part from first principles.

Questions to Address In Rebuttal

1. The central novelty is the mapping of the EML-QCCD architecture to a classical memory hierarchy. Beyond providing an intuitive vocabulary (e.g., 'cache miss', 'eviction'), what fundamental scheduling advantages does this analogy unlock that could not be achieved through a more conventional graph-based heuristic cost model that simply assigned higher costs to shuttles into/out of specific zones? Please elaborate on the specific algorithmic choices that are uniquely enabled by this perspective.

2. The SWAP insertion policy described in Section 3.3 (page 6) relies on a lookahead window of k=8 layers in the DAG. How was this value determined? Section 5.5 (page 9) suggests that the optimal k is application-dependent. How sensitive is the performance of MUSS-TI to this hyperparameter, and does this sensitivity undermine the claim of a generally applicable, robust framework?

3. The proposed multi-level scheduling is tightly coupled to the three-zone (storage, operation, optical) EML-QCCD architecture. How would the MUSS-TI framework generalize to a different trapped-ion architecture with, for instance, only two distinct zone types (e.g., storage and a combined operation/optical zone) or a more complex hierarchy with four or more levels? Does the novelty of the approach persist if the underlying architecture does not map as cleanly to a three-level memory system?

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose "Rasengan," a variational quantum algorithm for constrained binary optimization. The core idea deviates from traditional VQAs by expanding the search space from a single known feasible solution rather than shrinking it from a global superposition. This expansion is driven by a proposed "transition Hamiltonian," derived from the homogeneous basis of the problem's linear constraints. To make the algorithm deployable, the authors introduce several optimization techniques: Hamiltonian simplification/pruning, a "segmented execution" strategy to manage circuit depth, and a "purification" step for error mitigation. The paper claims significant improvements in both accuracy (ARG) and circuit depth over existing methods like penalty-term QAOA and commute-Hamiltonian-based QAOA (Choco-Q).

While the paper presents a novel approach built on a sound linear algebra foundation, its claims of superiority rest on a series of methodological choices and dramatic optimizations that warrant severe scrutiny. The core algorithm appears undeployable in its pure form, and its claimed practical success is almost entirely dependent on a segmented, measurement-heavy execution model that challenges its classification as a cohesive quantum algorithm.

Strengths

1. Conceptual Foundation: The core concept of building the solution space from a known feasible solution (xp) and the null space (u) of the constraint matrix (C) is sound from a linear algebra perspective. Framing this expansion in the language of a "transition Hamiltonian" is a clear and direct way to map the classical concept to a quantum operator.

2. Problem Formulation: The paper is well-structured and clearly identifies a key weakness in existing VQAs for constrained problems—namely, the inefficiency of searching a vast, mostly infeasible space. The proposed "expand-from-feasible" approach is a logical alternative.

3. Ablation Study: The inclusion of an ablation study (Section 5.6, Figures 15 and 16) is commendable, as it provides some insight into the relative impact of each proposed optimization. However, the interpretation of these results is debatable, as I will detail below.

Weaknesses

1. The "Segmented Execution" Breaks Coherent Quantum Evolution: The paper's claim to deployability hinges critically on the "segmented execution" strategy (Section 4.2). This technique partitions the sequence of Hamiltonians, executes each segment, measures the result, and uses the output probability distribution to initialize the next segment. This is not a continuous, coherent quantum evolution. It is a sequence of distinct quantum-classical steps. By collapsing the wavefunction to a classical probability distribution between segments, any quantum coherence and entanglement that might have been built up across segments is destroyed. This raises a fundamental question: is Rasengan a true quantum algorithm, or a classical heuristic that uses a quantum computer as a parameterized sampler at each step? The claim that it "preserves the probability information" (Section 4.2, Page 7) is insufficient; the crucial aspect of a quantum algorithm is the evolution of amplitudes in a superposition, which is explicitly broken here.

2. Misleading Baseline Comparisons: The primary claims of algorithmic superiority are based on potentially unfair comparisons. In Table 2, the baselines (HEA, P-QAOA, Choco-Q) are run with a fixed five layers. However, the paper's own analysis in Figure 9 demonstrates that the performance of Choco-Q improves dramatically with more layers, approaching Rasengan's accuracy at 14 layers. Therefore, the headline claim of a 4.12x accuracy improvement (from the abstract) is derived from comparing Rasengan against under-parameterized and non-optimized baselines. A rigorous comparison would benchmark algorithms against a fixed resource budget (e.g., total circuit depth or number of CNOTs), not a fixed, arbitrary layer count.

3. Grossly Overstated Real-Hardware Performance: The 379x improvement claim on real hardware (Section 5.4, Figure 11) is profoundly misleading. The data shows that the baseline algorithms produce ARGs greater than 90, which indicates a complete failure to find any meaningful solution—they are effectively outputting noise. Rasengan's achievement is not outperforming them in a meaningful way, but simply not failing as catastrophically. Attributing a "379x improvement" to this is statistically questionable. This success is likely due to the segmented, shallow-depth nature of the execution and the aggressive post-selection, which makes it more resilient to decoherence, rather than inherent algorithmic superiority.

4. "Purification" is Trivial Post-Selection: The "Error Mitigation by Purification" (Section 4.3) is presented as a novel technique. It is not. It is simply checking if a measurement result satisfies the classical constraints (Cx = b) and discarding it if it does not. This is standard post-selection. While necessary, its impact is misrepresented. The 100% in-constraints rate reported in noisy experiments (Figure 11b) is not an achievement of the algorithm in handling noise, but a direct artifact of this filtering process. Furthermore, the massive 303x ARG improvement on hardware attributed to Opt 3 (which includes purification) in Figure 16 reveals that this filtering step is responsible for the vast majority of the claimed performance, masking the poor quality of the raw quantum output.

5. Contradictory Circuit Complexity Narrative: The paper initially presents a transition Hamiltonian that requires "34k CX gates" where k is the number of non-zero elements in the basis vector (Section 3.2, Page 5). For non-trivial problems, this is undeployable on any NISQ device. The paper only achieves a deployable depth (~50, Section 4) via the "segmented execution" method, which, as argued in point #1, fundamentally changes the nature of the algorithm. The narrative presents a single algorithm, "Rasengan," but its theoretical formulation and its practical implementation are two vastly different procedures.

Questions to Address In Rebuttal

1. Please provide a theoretical justification for how "segmented execution" (Section 4.2), which involves intermediate measurements and classical handoffs, preserves the quantum computational advantage of superposition and entanglement across the entire problem evolution. How does this method differ from a classical search heuristic that uses a quantum device as a parameterized sampler at each step?

2. Given that Figure 9 shows Choco-Q's ARG improves significantly with more layers, how can the authors justify the fixed 5-layer comparison in Table 2 and the resulting 4.12x improvement claim? A fair comparison would require comparing both algorithms under an equivalent resource budget (e.g., total circuit depth or execution time).

3. Can the authors defend the 379x hardware improvement claim (Figure 11) when the baselines are performing at near-random levels (ARG > 90)? Does this result demonstrate genuine algorithmic superiority, or merely that the baselines are more fragile to noise and Rasengan's segmented, post-selected approach is less so?

4. Please clarify the contribution of the "purification" technique (Section 4.3). Since this is equivalent to post-selecting valid solutions, how does it address the underlying noise in the quantum computation itself, rather than simply filtering the final, noisy output distribution? Does this post-selection not risk biasing the final solution, especially if noise affects different basis states non-uniformly?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces Rasengan, a novel variational quantum algorithm (VQA) for solving constrained binary optimization problems. The work's essential contribution is a fundamental inversion of the typical VQA search strategy. Instead of starting with a superposition of all possible states (both feasible and infeasible) and attempting to "shrink" the search space towards an optimal solution, Rasengan begins with a single, classically-found feasible solution and uses a specially constructed "transition Hamiltonian" to "expand" the search space to cover all other feasible solutions.

This transition Hamiltonian is elegantly derived from the homogeneous basis vectors (the null space) of the problem's linear constraint matrix, ensuring that every step of the quantum evolution remains strictly within the feasible subspace. The authors complement this core algorithmic idea with a suite of pragmatic algorithm-hardware co-design techniques—including Hamiltonian simplification, pruning of redundant transitions, segmented execution to manage circuit depth, and a purification step for error mitigation. These optimizations make the conceptually powerful idea deployable on near-term quantum hardware. The paper presents a comprehensive evaluation demonstrating significant improvements in both accuracy and circuit depth over existing state-of-the-art methods like penalty-based and commute-Hamiltonian QAOA.

Strengths

1. Powerful Conceptual Shift: The most significant strength of this work is its core idea of "expanding" the feasible space rather than "shrinking" the total space (beautifully illustrated in Figure 2, page 4). This directly addresses a primary source of inefficiency in many quantum optimization algorithms, which expend vast computational resources navigating a huge, mostly infeasible search space. By constraining the search to the relevant subspace from the very beginning, the algorithm promises a much more efficient use of quantum resources.

2. Elegant Connection Between Linear Algebra and Quantum Dynamics: The formulation of the transition Hamiltonian (Section 3.1, page 4) based on the homogeneous basis vectors of the constraint system Cx=b is a particularly insightful contribution. It provides a rigorous mathematical bridge between the classical, algebraic structure of the problem's constraints and the quantum evolution needed to explore the solution space. This is a far more structured and problem-aware approach than the generic mixers used in standard QAOA or the adaptive layers of a Hardware Efficient Ansatz (HEA).

3. Pragmatic Co-Design for NISQ-era Reality: The authors do not stop at a purely theoretical proposal. The optimization techniques presented in Section 4 (page 6) show a mature understanding of the practical limitations of current quantum computers. Segmented execution, in particular, is a clever strategy to execute deep logical circuits on hardware with short decoherence times. This combination of a strong theoretical idea with practical, hardware-aware engineering is a major strength and is what makes the impressive real-hardware results (Figure 11, page 10) believable and impactful.

4. Strong Empirical Validation: The experimental evaluation is extensive and convincing. The authors test their method against relevant baselines across five different problem domains (Table 2, page 8), analyze scalability (Figure 10, page 9), and demonstrate performance on real IBMQ hardware. The reported 4.12x accuracy improvement over the state-of-the-art Choco-Q and the staggering 379x improvement on a real quantum device are compelling evidence of the method's potential.

Weaknesses

My critiques are less about fundamental flaws and more about clarifying the boundaries and assumptions of the proposed method, which will help contextualize its applicability.

1. Dependence on an Initial Feasible Solution: The entire framework is predicated on the ability to classically and efficiently find at least one feasible solution to initialize the algorithm. The authors state this can often be done in linear time (Section 5.1, page 7), which is true for the benchmarks chosen (e.g., facility location, set cover). However, for other classes of problems, particularly those with very tight or complex constraints (e.g., certain integer linear programs), finding any feasible solution can be an NP-hard problem in itself. This assumption represents a key boundary on the applicability of Rasengan.

2. Limited Scope of Constraints: The methodology is presented for problems with linear equality constraints (Cx=b). While the paper notes that inequalities can be handled by introducing slack variables, this standard technique can significantly increase the number of required qubits, which is the most precious resource in the NISQ era. The work would be strengthened by a more thorough discussion of its applicability and potential performance degradation when applied to problems dominated by inequalities or those with non-linear constraints.

3. Implicit Assumptions on Basis Sparsity: The performance of the algorithm, especially the complexity of the transition Hamiltonian circuit, depends on the number of non-zero elements in the homogeneous basis vectors. While the Hamiltonian simplification technique (Section 4.1, page 6) is designed to mitigate this, the paper does not fully explore the worst-case scenario where the basis vectors are pathologically dense. This could lead to circuits that remain prohibitively deep even after optimization, representing another potential boundary condition.

Questions to Address In Rebuttal

1. Could the authors comment on the applicability of Rasengan to problems where finding an initial feasible solution is itself computationally difficult? Does the framework offer any recourse in such scenarios, or would it be considered out-of-scope for this method?

2. Regarding the handling of inequality constraints, could the authors provide some analysis or experimental data on how the introduction of slack variables impacts Rasengan's performance? Specifically, how does the increase in qubit count affect the circuit depth and overall accuracy compared to methods that handle inequalities more natively?

3. While the Hamiltonian simplification and pruning are shown to be effective on the chosen benchmarks, could the authors elaborate on the theoretical limitations of these techniques? Are there known problem structures or constraint topologies that would result in dense homogeneous basis vectors for which these optimizations would provide limited benefit?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces "Rasengan," a variational quantum algorithm (VQA) for constrained binary optimization problems with linear constraints. The central claim of novelty lies in its core methodology: instead of starting with a superposition of all possible states and "shrinking" the search space towards a feasible optimum (the standard QAOA paradigm), Rasengan starts from a single, classically-determined feasible solution and "expands" the search space to cover all other feasible solutions.

This expansion is achieved through a novel construct the authors term the "transition Hamiltonian," which is derived directly from the homogeneous basis of the problem's linear constraint matrix. The algorithm iteratively applies these transition Hamiltonians to navigate exclusively between basis states corresponding to feasible solutions. The paper also presents a set of co-design optimizations (Hamiltonian simplification/pruning, segmented execution, and purification) to make this approach viable on NISQ-era hardware.

Strengths

My evaluation identifies the following genuinely novel contributions:

1. Core Algorithmic Paradigm Shift: The primary strength and most significant novel contribution is the conceptual reversal from a "shrinking" to an "expanding" search space. To my knowledge, existing VQAs for this problem class, such as penalty-based QAOA or commute-Hamiltonian QAOA (like Choco-Q [43]), operate by either penalizing infeasible states or designing mixers that confine the evolution within the feasible subspace. Rasengan's approach of structured, generative exploration from a single point using the algebraic properties of the constraints is a fundamentally new and distinct VQA design pattern. This is clearly articulated in Figure 2 (page 4).

2. The "Transition Hamiltonian" Construct: The specific formulation of the transition Hamiltonian H(u) in Section 3.1 (Equation 7, page 5) is a novel piece of technical machinery. It translates a vector from the constraint's null space (u) directly into a quantum operator that performs a transition between two specific feasible solutions. This creates a direct, programmable link between the algebraic structure of the problem and the quantum evolution, which is a more tailored approach than the generic mixers used in standard QAOA.

3. Decoupling of Objective Function from Quantum Evolution: A subtle but profound novelty is that Rasengan’s quantum circuit is entirely agnostic to the objective function. Standard QAOA requires encoding the objective function into a (potentially complex) phase-separating Hamiltonian. Here, the quantum evolution's sole purpose is to navigate the feasible space. The objective function is evaluated purely classically on the measurement outcomes. This offloading of complexity from the quantum to the classical component represents a new way of structuring a hybrid quantum-classical workflow for optimization.

Weaknesses

My analysis identifies areas where the novelty is either overstated or not sufficiently demarcated from existing concepts:

1. Limited Novelty of Optimization Techniques in Isolation: The three optimization techniques presented in Section 4 (page 6) are novel in their specific integration but are conceptually derivative of prior art.

   • Segmented Execution (Section 4.2, page 7): The idea of breaking a deep circuit into shallower ones is well-established in the field, often under names like "circuit cutting" or in the context of dynamic circuits. The paper does not adequately compare its "probability-preserving" approach to this body of work to clearly isolate its novel contribution.
   • Error Mitigation by Purification (Section 4.3, page 7): This is a form of post-selection based on constraint satisfaction. Post-selection is a standard, albeit costly, error mitigation technique. Its application between segments is a specific implementation choice, but the underlying concept is not new.
   • Hamiltonian Simplification (Section 4.1, page 6): The greedy search for a sparser basis for the null space is a classical heuristic. While its application to reducing quantum circuit complexity is clever, the novelty lies in the application, not the technique itself.

2. Dependence on Classical Pre-Processing: The algorithm's novelty is purely in the quantum search phase. The entire framework is predicated on the ability to classically and efficiently find an initial feasible solution and the homogeneous basis ({u}). For problems where this classical pre-computation is itself NP-hard, the practical novelty of the quantum speedup is diminished. The paper should more clearly define the boundary of its contribution, acknowledging this reliance.

3. Constrained Scope of Novelty: The formulation of the transition Hamiltonian is explicitly tied to linear equality constraints (Cx=b). The paper makes no claims about extending this core novel idea to problems with non-linear or inequality constraints, thus limiting the generality of the proposed paradigm.

Questions to Address In Rebuttal

The authors should use the rebuttal to clarify the precise boundaries of their novel contributions:

1. Regarding "Segmented Execution" (Section 4.2, page 7): Can you elaborate on the key difference between your proposed method and established circuit cutting techniques? Is the primary novelty simply the dynamic re-allocation of shots based on intermediate probabilities, or is there a more fundamental distinction?

2. The decoupling of the objective function from the quantum circuit is a significant architectural change from QAOA. To your knowledge, is Rasengan the first VQA for optimization that uses the quantum evolution solely for state space navigation while leaving the objective evaluation entirely classical? Please position this contribution with respect to prior art.

3. The proposed method is applicable to problems with linear constraints. How would the core novel idea—the construction of a "transition Hamiltonian" from the problem structure—be formulated for problems with non-linear constraints? Does the paradigm break down, or is there a conceivable path to generalization?

4. The complexity of finding the homogeneous basis is a classical pre-processing cost. For which important problem classes is this step guaranteed to be polynomial, and are there classes where it is not? How does this affect the overall novelty of the quantum algorithm's contribution to solving the end-to-end problem?

---

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes Chasoň, an HBM-based streaming accelerator for sparse matrix-vector multiplication (SpMV). The central contribution is a novel scheduling scheme, Cross-HBM Channel out-of-order Scheduling (CrHCS), designed to mitigate the processing element (PE) underutilization found in prior work like Serpens. CrHCS enables the migration of non-zero data elements across HBM channels to fill pipeline stalls that would otherwise occur. The authors implement Chasoň on an AMD Alveo U55C FPGA and evaluate it against Serpens, high-end GPUs, and a CPU, claiming significant improvements in performance and energy efficiency stemming from improved PE utilization.

Strengths

1. The paper correctly identifies a significant and well-known limitation in state-of-the-art HBM-based sparse accelerators: PE stalls due to workload imbalance across rows mapped to a single HBM channel. The motivation presented in Section 2 is clear and logically sound.
2. The proposed solution, migrating non-zeroes between channels, is a conceptually logical extension to the existing intra-channel scheduling schemes.
3. The experimental evaluation is broad, utilizing a large set of 800 matrices and comparing against multiple relevant platforms (FPGA, GPU, CPU).

Weaknesses

My primary concerns with this work center on the validity of its core assumptions, the fairness of its experimental comparison, and the analysis of its performance claims.

1. Unsubstantiated Core Scheduling Assumption: The entire efficacy of CrHCS rests on a critically optimistic and unproven assumption. In Section 3.3, the authors state, "Our experiments have shown that CrHCS never fails to find a RAW dependency-free value to migrate." This is an exceptionally strong claim presented without any supporting data, theoretical proof, or statistical analysis. The motivational example in Figure 2c conveniently shows a perfect pipeline with 0% underutilization, which is predicated on the constant availability of suitable non-zero candidates from the neighboring channel. This appears to be a best-case scenario presented as a general capability. The paper fails to characterize the conditions under which a suitable candidate would not be available, which would break the core premise of the work.

2. Flawed and Unfair Baseline Comparison: The comparison to the primary baseline, Serpens, is questionable.

   • The authors report their implementation of Serpens achieves only 223MHz, while Chasoň achieves 301MHz on the same U55C platform (Section 5.2). A ~35% frequency advantage for Chasoň is a major confounding variable. The authors attribute this to "reduced logic congestion," but this is a vague justification. A more rigorous analysis is required to prove that their port of Serpens is faithful and optimized, and not simply a poor implementation used to inflate Chasoň's relative performance.
   • The resource comparison in Table 1 shows that Chasoň consumes significantly more resources than Serpens (+58% LUT, +66% FF, +57% DSP). The authors dismiss the need for an iso-area comparison by asserting that Serpens could not benefit from more PEs (Section 4.5). This is an unsubstantiated claim and sidesteps a fundamental requirement for fair hardware accelerator comparisons. The authors are comparing a larger, more complex, and higher-frequency design to a smaller, simpler, and slower one. The performance gains are therefore not solely attributable to the CrHCS scheduling algorithm.

3. Misleading Attribution of Performance Gains: The paper attributes its speedup over Serpens to "data transfer reduction" (Figure 15, Section 6.2). This is a confusing and misleading framing. Both Chasoň and Serpens must process the exact same number of non-zero elements. The issue with Serpens is not that it "transfers zeros," but that its pipeline stalls for lack of available work. Chasoň avoids these stalls by transferring useful non-zero data from another channel. The gain comes from increased temporal utilization of the datapath (i.e., stall reduction), not from reducing the total volume of data transferred from HBM. This mischaracterization obscures the true mechanism of improvement and suggests a misunderstanding of the performance dynamics.

4. Introduction of New, Unanalyzed Bottlenecks: The architectural components required to support CrHCS (Router, Reduction Unit, Re-order Unit) introduce non-trivial latency and complexity. The authors' own analysis provides evidence of this. On page 12, they note that the C5 matrix, despite having a larger "data transfer reduction" factor, achieves lower speedup than the MY matrix because its larger column dimension creates contention and latency in the ScUGs and Reduction Unit. This is a critical finding: the overhead of the proposed architecture can itself become the primary performance bottleneck, potentially negating the benefits of CrHCS. This trade-off is not systematically explored or quantified.

Questions to Address In Rebuttal

1. Regarding the core assumption of CrHCS: Please provide statistical data from your 800-matrix evaluation that quantifies the probability of finding a RAW-dependency-free non-zero element in the neighboring channel for any given stall cycle. How does this probability change with matrix structure and sparsity? On what basis can the authors claim this "never fails"?

2. Regarding the baseline comparison: Please provide a detailed justification for the 35% frequency advantage over your implementation of Serpens. Furthermore, please provide a detailed argument for why a rigorous iso-resource comparison is not necessary, or alternatively, provide such a comparison. For instance, what would the performance of Serpens be if it were scaled to consume the same resources as Chasoň?

3. Regarding performance attribution: Please clarify the "data transfer reduction" claim. Is it not more accurate to state that the improvement comes from reducing pipeline stall cycles? Please re-frame the argument in terms of pipeline occupancy or throughput.

4. Regarding new bottlenecks: The paper admits that the reduction and re-ordering logic can become a bottleneck (C5 vs. MY matrix example). Please provide a sensitivity analysis that characterizes how the latency of these new architectural units impacts the overall speedup as a function of matrix properties (e.g., number of columns, non-zero distribution). At what point do the overheads of Chasoň's architecture outweigh the benefits of stall reduction?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces Chasoň, an HBM-based streaming accelerator for Sparse Matrix-Vector Multiplication (SpMV). The central contribution is a novel out-of-order scheduling scheme, "Cross-HBM Channel OoO Scheduling" (CrHCS), which enables dynamic data migration across HBM channels. The authors identify a key limitation in state-of-the-art accelerators like Serpens: while they parallelize work across HBM channels, their scheduling is confined within a channel ("intra-channel"). This leads to significant PE underutilization when the matrix rows assigned to a particular channel are sparse.

Chasoň's CrHCS addresses this by allowing a PE group associated with one channel to "borrow" non-zero elements from a neighboring channel's data stream, effectively filling its own pipeline stalls. The paper presents the necessary architectural support for this scheme, including data tagging, a routing mechanism within the PEs, and dedicated on-chip memory structures to manage results from both "private" and "shared" (migrated) data. The authors implement Chasoň on an AMD Alveo U55C FPGA and demonstrate significant improvements in PE utilization, leading to substantial performance and energy efficiency gains over Serpens, as well as modern GPU and CPU platforms.

Strengths

This is a strong paper with a clear, compelling, and well-executed core idea.

1. Excellent Problem Formulation and Motivation: The paper does an outstanding job of contextualizing its contribution. It clearly explains the evolution from row-based to PE-aware scheduling (Section 2.2, pages 2-3) and uses a simple, effective diagram (Figure 2, page 3) to illustrate the limitations of the current state-of-the-art. The motivation is powerfully reinforced by Figure 3 (page 4), which quantifies the PE underutilization problem across a large dataset, showing that ~70% underutilization is common. This makes the need for a better solution undeniable.

2. A Novel and Elegant Core Concept: The idea of inter-channel data migration is a genuinely new perspective in the design space of HBM-based sparse accelerators. The prevailing wisdom has been to treat the HBM channels as independent data streams feeding isolated processing element groups (PEGs). Chasoň cleverly breaks this isolation to solve the fundamental problem of load imbalance. This is a microarchitectural solution to a classic parallel computing challenge, and it feels both innovative and intuitive. It's a natural evolution of out-of-order principles applied at a coarser, inter-channel granularity.

3. Thorough and Credible Implementation: This is not merely a conceptual or simulation-based study. The authors provide a detailed architectural design (Section 4, pages 5-7) and a full implementation on a modern FPGA platform. The design considers the practical details required to make CrHCS work correctly, such as the pvt and PE_src flags for data provenance and the Rearrange Unit to ensure final results are correctly ordered. Achieving a 301MHz clock frequency on the Alveo U55C with this level of complexity lends significant credibility to the work.

4. Strong Connection to the Broader Landscape: The work is well-positioned within the literature. It correctly identifies Serpens [71], Sextans [72], and other HBM-based accelerators as its direct lineage. By building upon and significantly improving a known, open-source architecture (Serpens), the authors make their contribution easy to understand and appreciate. The principle of dynamically sharing work to smooth out irregularities connects to broader themes in parallel processing, systolic array design (e.g., [35]), and even processing-in-memory concepts where data locality and movement are paramount.

Weaknesses

The weaknesses are minor and relate more to exploring the full design space of the proposed idea rather than fundamental flaws in the presented work.

1. Limited Exploration of Migration Policies: The implementation restricts data migration to only the immediate next channel (as mentioned in Section 6.1, page 10). While justified by on-chip resource constraints, this feels somewhat arbitrary from a conceptual standpoint. A richer discussion on the design space of migration policies would strengthen the paper. For instance, what are the trade-offs of bidirectional migration (borrowing from channel i-1 and channel i+1), or a wider but shallower "neighborhood" of channels? This seems like a critical design parameter that is fixed here without much exploration.

2. Interplay with Static Data Partitioning: The effectiveness of the dynamic migration scheme is likely coupled with the initial static assignment of matrix rows to HBM channels. The paper does not discuss this relationship. For example, if a "poor" static partitioning creates a pattern where a sparse channel is always followed by another sparse channel, the "next-channel" policy would be ineffective. A brief analysis of this interplay would add depth.

3. Analysis of Latency Overhead: While the paper successfully argues that reducing stalls and data transfers yields a large net performance gain, the latency overhead of the new architectural units (Router, Reduction Unit, Reorder Unit) is not explicitly quantified. The authors state these units are "fundamental" (Section 6.2.2, page 12), but a more direct analysis—perhaps in terms of additional pipeline stages or critical path impact—would provide a more complete picture of the architectural trade-offs.

Questions to Address In Rebuttal

1. The current CrHCS implementation limits data migration to the immediate "next" channel in a unidirectional manner. Could the authors elaborate on the design trade-offs here? Would a bidirectional (i.e., borrowing from both previous and next channels) or a wider neighborhood migration scheme be feasible, and what would be the expected impact on resource usage (especially URAMs) and performance?

2. How sensitive is the proposed scheme to the initial static mapping of matrix rows to HBM channels? For example, if the workload distribution across channels is highly correlated (e.g., alternating blocks of dense and sparse regions), could the fixed "next channel" migration policy become a bottleneck?

3. The paper's core idea seems highly generalizable. Beyond the successful application to SpMV, have the authors considered its potential for other sparse kernels that are often accelerated on HBM platforms, such as SpMM (Sparse-Matrix Dense-Matrix Multiplication) or SpGEMM (Sparse-Matrix Sparse-Matrix Multiplication), where load imbalance across parallel units can be even more severe? (The authors briefly touch on SpMM in Section 7.1, but I would be interested to hear more on the potential challenges).

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces "Chasoň," an HBM-based streaming accelerator for sparse matrix-vector multiplication (SpMV). The central claim to novelty lies in its non-zero scheduling scheme, "Cross-HBM Channel out-of-order Scheduling" (CrHCS). This scheme is designed to address the problem of processing element (PE) underutilization, which the authors identify as a key weakness in prior art like Serpens [71]. Where Serpens and similar works use intra-channel out-of-order scheduling, CrHCS extends this to inter-channel scheduling. This is achieved by allowing non-zero data elements scheduled for one HBM channel to be migrated to an adjacent channel to fill stalls in its data stream. To support this, the authors propose an architecture with specific hardware for routing, re-ordering, and reducing partial sums originating from these migrated, or "shared," data elements.

Strengths

1. Clear Articulation of the Novel "Delta": The paper does an excellent job of isolating its contribution relative to the immediate state-of-the-art. It correctly identifies that HBM-based accelerators like Serpens [71] and Sextans [72] treat HBM channels as independent data streams feeding dedicated PE groups. The core novel idea of Chasoň is to break this strict channel-to-PEG isolation at the data scheduling level. The shift from an intra-channel to an inter-channel scheduling scope is a specific and well-defined advancement.

2. Novel Application of a Known Concept: The underlying principle of CrHCS is a form of work stealing or dynamic load balancing—transferring work from a source with available tasks to an idle resource. While work stealing is a classic concept in parallel computing, its application in this specific context is novel. The authors have adapted this general principle to the unique constraints of a streaming HBM-FPGA architecture, where data is pre-scheduled into fixed-width streams. This is not a trivial application and represents a new approach for this class of accelerators.

3. Enabling Architectural and Data Structure Novelty: To realize the CrHCS scheme, the authors introduce necessary and novel modifications. The inclusion of the pvt (private) and PE_src (source PE) flags in the 64-bit data element (Section 3.2, page 4) is a simple but clever mechanism to handle the bookkeeping of migrated data without requiring complex control logic. The architectural additions—specifically the Router within the PE and the system-level Reduction and Re-order Units (Section 4, pages 5-7)—are a direct and logical consequence of the scheduling scheme and are new in their composition and purpose for this problem.

Weaknesses

1. Conceptual vs. Foundational Novelty: While the application of inter-channel data migration is novel for HBM-based sparse accelerators, the paper could be stronger by explicitly positioning CrHCS within the broader literature of work stealing and dynamic load balancing. The current framing presents the idea as wholly new, whereas its foundational concept is well-established. Acknowledging this and then detailing why their implementation is non-trivial and unique to the HBM architecture would more accurately place the contribution.

2. Incremental Nature of Architectural Components: The hardware units proposed to support CrHCS (Router, Reduction Unit, Re-order Unit) are, in isolation, standard digital design building blocks. The Router is a multiplexer-based structure, the Reduction Unit is an adder tree, and the Re-order Unit is a form of sorter or stream aligner. The novelty is not in the components themselves, but entirely in their synthesis to support the new scheduling algorithm. The paper is clear about their function, but it's important to recognize that the architectural novelty is compositional, not elemental.

3. Limited Scope of the Novel Mechanism: The implementation of CrHCS is constrained to migrating data only from the immediate next channel (e.g., channel i+1 to i). The authors justify this by citing on-chip memory limitations on the target FPGA (Section 6.1, page 10). While this is a practical engineering decision, it significantly limits the generality and novelty of the proposed data migration scheme. A truly novel interconnection scheme might support more flexible migration patterns (e.g., any-to-any, or from the two nearest neighbors). The current implementation is a 1D, unidirectional work-stealing approach, which is the simplest possible form of inter-resource cooperation.

Questions to Address In Rebuttal

1. On the Generality of CrHCS: The decision to migrate data only from the immediate next channel appears pragmatic but limits the solution's generality. Could the authors elaborate on the architectural complexity (e.g., interconnect, URAM scaling for partial sums, and Re-order Unit complexity) of a more flexible "any-to-any" or "bi-directional nearest-neighbor" channel migration scheme? What is the anticipated crossover point where the hardware overhead and control complexity would outweigh the benefits of filling more stalls?

2. Comparison to Classic Work Stealing: Please contrast CrHCS with classic work-stealing algorithms (e.g., Cilk). What specific constraints of the HBM/FPGA streaming architecture (e.g., fixed AXI widths, offline scheduling, lack of a shared task queue) prevent a more traditional implementation, and how does CrHCS's offline, stream-injection approach uniquely address them?

3. Overhead of the Novel Scheduler: The paper focuses on the hardware implementation, but CrHCS requires a more sophisticated offline scheduler compared to the simpler round-robin assignment in Serpens. What is the computational overhead of this new scheduling step? While this cost is likely amortized over many runs, a quantification would help assess if the novelty in scheduling introduces a new, non-trivial bottleneck in the overall workflow before execution can even begin.

---

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents D2T2, a framework for optimizing static tiling schemes for sparse tensor algebra. The core idea is to move beyond conservative, worst-case tiling by building a probabilistic model based on high-level statistics extracted from the input tensors. This model is used to predict memory traffic for various tile shapes and sizes, enabling a search for a traffic-optimal configuration that does not require specialized hardware. The authors evaluate D2T2 against state-of-the-art methods like Tailors and DRT, claiming significant performance improvements.

However, the work rests on a series of questionable assumptions regarding the statistical independence of tensor structures, and its experimental validation against prior art is methodologically compromised. While the goal is commendable, the rigor of the modeling and evaluation is insufficient to substantiate the paper's primary claims.

Strengths

1. Hardware-Agnostic Approach: The primary conceptual strength is the aim to improve static tiling without mandating specialized hardware support (unlike DRT's dynamic aggregation or Tailors' overflow buffers). This makes the proposed technique potentially more generalizable.
2. Data-Driven Philosophy: The fundamental motivation to use the statistical properties of the input data to inform tiling is sound. Moving away from worst-case (dense) assumptions is a necessary step for efficient sparse computation.
3. Identification of a Correlation Proxy: The introduction of the Corrs statistic (Section 4.4, page 6) to model output reuse is a necessary attempt to patch the model's core assumption of independence. Figure 8 demonstrates a clear, if simplistic, relationship between this metric and optimal tile shape.

Weaknesses

1. Fundamentally Flawed Modeling Assumption: The entire probabilistic model is built on a demonstrably false premise. In Section 4.2 (page 5), the authors state: "We estimate this probability... by assuming that the nonzero structures of A and C are uncorrelated." This assumption of independence is incorrect for a vast majority of real-world sparse computations. For example, in graph analytics (A x A^T), the structures of the input and output are highly correlated. Banded matrices, diagonally-dominant systems, and tensors from physical simulations all exhibit strong structural correlations. The Corrs proxy introduced in Section 4.4 is a limited, pairwise correction that is insufficient to capture the full, complex nature of these dependencies. The authors' own validation in Section 5.3 (page 8) and Figure 5 reveals this weakness, where they admit that for the A x A^T kernel, "the tiles may be correlated, leading to under-estimation of valid intersections." A model that is systematically biased for such a common kernel is not robust.

2. Methodologically Unsound Comparison with DRT: The comparison against DRT in Section 6.2 (page 10) is invalid. The authors use two different backends for the evaluation: DRT's performance is measured using its dedicated simulator, while D2T2's is measured using a TACO-based backend. The justification provided is that "the DRT simulator was unable to map D2T2-generated configurations to micro-tiles for most matrices." This is not a justification; it is a critical confounding variable. The claim that the backends produce comparable results (within <5%) is only validated for the Conservative tiling scheme. This says nothing about their relative performance on the highly non-uniform, rectangular tiles that D2T2 generates. In fact, the DRT simulator's failure to map these tiles suggests a fundamental incompatibility in the execution models, making any direct comparison of traffic numbers meaningless. The reported 1.13x traffic reduction over DRT is therefore an unsubstantiated claim.

3. Insufficient Validation on Higher-Order Kernels: The evaluation of TTM and MTTKRP in Section 6.3 (page 10) is weak. While the paper uses real-world tensors (from Frostt, Facebook), the other operands are random matrices with uniform sparsity. The text states, "we use a random matrix with dimensions T3 × max (T1, T2)". Real-world tensor operations often involve operands that share structural properties. Using random matrices fails to test the model against the structured correlations that are common in these applications and which the model is likely to mis-predict.

4. Misleading Overhead Analysis: The overhead analysis in Section 6.5 (page 11) reports the overhead of statistics collection (9.3%) and optimization (7.9%) relative to the initial tiling time. Tiling is typically a very small fraction of the total end-to-end execution time. Framing the overhead in this manner minimizes its perceived cost. A more transparent analysis would present this overhead as a percentage of the total application runtime for a set of representative problems.

Questions to Address In Rebuttal

1. Please provide a rigorous justification for the uncorrelation assumption that underpins your traffic model (Section 4.2). Given that many important sparse kernels (e.g., A x A^T) explicitly violate this assumption, how can the model be considered general? How does the simple pairwise Corrs statistic (Section 4.4) account for non-local or higher-order correlations present in real-world data?

2. How can the performance claims against DRT (Section 6.2) be considered valid when entirely different execution backends were used? Please provide evidence that the performance characteristics of the TACO and DRT backends are identical for the non-uniform, rectangular tiles generated by D2T2, not just for simple square tiles. Absent this evidence, the comparison must be removed.

3. In Figure 5, the model systematically underestimates traffic for A x A^T. Can the authors provide a more rigorous analysis of this model error and explain why relative accuracy is sufficient for optimization when the absolute error is significant and biased?

4. Why were random matrices used for the evaluation of TTM and MTTKRP in Table 4, rather than leveraging multiple real-world tensors which possess complex, correlated structures that would more rigorously test the model's assumptions?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

The authors present D2T2 (Data-Driven Tensor Tiling), a framework for generating data-aware, static tiling schemes for sparse tensor algebra computations. The core contribution is a probabilistic modeling approach that uses high-level statistics, extracted efficiently from the input tensors' compressed formats, to predict memory traffic for different tiling configurations. By modeling the probability of access and the expected size of tiled data, D2T2 searches for non-uniform tile shapes and sizes that minimize total memory movement. Crucially, this optimization is performed offline during a pre-processing step, allowing the resulting static tiling scheme to be executed on accelerators without requiring specialized dynamic tiling hardware. The paper demonstrates that this approach can achieve performance superior to state-of-the-art dynamic (DRT) and overflow-based (Tailors) tiling schemes, positioning it as a highly practical solution for a broad range of hardware.

Strengths

1. Elegant and Practical Core Idea: The central thesis—that a probabilistic model of data distribution can inform a superior static tiling scheme—is both powerful and pragmatic. It carves out a valuable and previously underexplored design point between two extremes: overly conservative static tiling (which ignores data properties) and complex hardware-intensive dynamic tiling (which is costly and less portable). This "software-first" approach to data-awareness is the paper's most significant strength.

2. Broad Deployability: By obviating the need for specialized hardware like dynamic tile aggregators (DRT) or overflow-capable buffers (Tailors), the D2T2 framework is immediately applicable to a wide array of existing and future sparse accelerators. The evaluation on the real-world Opal CGRA (Section 6.4, page 10) is a compelling demonstration of this, showing significant speedups on a hardware platform not co-designed with this specific tiling strategy in mind. This greatly enhances the potential impact of the work.

3. Strong Contextualization and Clear Problem Framing: The authors do an excellent job of positioning their work within the broader landscape of sparse tensor computation. The introduction clearly articulates the "utilization problem" and the limitations of prior art. The summary in Table 1 (page 3) is particularly effective, immediately clarifying the trade-offs between different tiling schemes and highlighting D2T2's unique value proposition.

4. Comprehensive Experimental Analysis: The evaluation is thorough. The authors not only compare against relevant state-of-the-art systems on their original benchmarks but also validate their predictive model's accuracy (Section 5.3, page 8), analyze the framework's runtime overhead (Section 6.5, page 11), and perform insightful ablation studies on the importance of different statistics (Section 6.7, page 12). This rigorous approach builds significant confidence in the results.

Weaknesses

1. Model Fidelity on Correlated Data: The model's primary simplification appears to be the assumption of uncorrelated inputs when estimating output traffic. The authors rightly note this leads to under-prediction in cases like A x A^T (Figure 5, page 9), where the input structures are perfectly correlated. While they argue the model still captures relative trends correctly, this is a fundamental limitation. For many real-world problems (e.g., graph analytics involving adjacency matrices and their transposes), such correlations are the norm, not the exception. The "Corrs" statistic is a clever proxy to patch this, but it feels like a heuristic layered on top of a model that doesn't natively handle this phenomenon.

2. Generalizability of the Statistical Framework: The paper introduces a set of key statistics (e.g., PrTileIdx, ProbIndex, Corrs). These appear highly effective for the matrix- and 3-tensor-based kernels evaluated. However, it is less clear how this specific set of statistics would generalize to more complex, higher-order tensor expressions with multiple contraction loops. The concept of Corrs as a 1D correlation over the contracted dimension (Section 4.4, page 6) is intuitive for matrix multiplication, but its extension to, for instance, a tensor-times-matrix operation with two shared indices might require a more sophisticated, multi-dimensional correlation model.

3. Exploration of the Tiling Search Space: The shape optimization is guided by varying a "Reorder Factor (RF)" that controls the tile aspect ratio while preserving area (Section 5.2, page 8). This is a reasonable and practical heuristic. However, the true optimization space of tile shapes is vast. A deeper discussion on why this 1D parameter sweep is a sufficient exploration would be beneficial. It's possible that for some tensor structures, more exotic tile shapes not discoverable through this method could yield even better results.

Questions to Address In Rebuttal

1. On the Impact of Model Correlation: Regarding the model's under-prediction for correlated inputs (Section 5.3), could you elaborate on how this might affect tile shape selection for computations where output reuse is the dominant performance factor? Is it possible that the model, even while capturing the relative trend, might still erroneously prefer an outer-product-like shape (minimizing input re-fetches) when a square-like shape (maximizing output reuse) would have been globally optimal for that specific correlated input pair?

2. On the 'Sweet Spot' for This Approach: A key contribution is enabling data-awareness for static tiling. This implies a trade-off. Could you comment on the data distributions or problem types that define the "sweet spot" for D2T2? Conversely, are there specific sparsity structures (e.g., power-law graphs with extreme variance in degree) where the global statistics used by D2T2 might be misleading, making a fully dynamic, fine-grained approach like DRT inherently superior despite its hardware cost?

3. On the Future of Probabilistic Modeling: Your work successfully applies a probabilistic lens to tiling. Looking forward, how could this modeling framework be extended beyond tiling? For instance, could these same data statistics be used to guide other crucial compiler decisions, such as data layout transformations (e.g., choosing between CSF, COO, etc., on a per-tensor basis) or kernel fusion strategies for sparse computations?