Hot Chips 2026: d-Matrix stacks AI accelerator directly on custom DRAM for 100 TB/s per card — TSMC 4nm compute die bonded face-to-face at a 36-micron pitch on top of a custom-designed die
This Tom’s Hardware Premium article is free to read with a Tom’s Hardware account; no payment necessary. We’re offering free access from August 23 to 26 so you can read all of our reporting from Hot Chips.
d-Matrix presented Raptor, which it calls the first 3D DRAM accelerator for generative inference, at Hot Chips 2026 this week, showing a TSMC 4nm compute die bonded face-to-face at a 36-micron pitch on top of a custom-designed DRAM die that delivers 100 TB/s of bandwidth from 32GB per card.
Co-founder and CTO Sudeep Bhoja put the vertical interface’s energy cost at 0.37 pJ/bit against roughly 2.4 pJ/bit for moving data into an HBM4 base die, calling it “a measured number” from working silicon, and the accompanying ISCA 2026 paper, written with the University of British Columbia, projects around 4.7 times higher throughput per card than HBM-based designs. Bhoja, however, didn’t disclose who manufactures the DRAM die.
Latest Videos FromTom’s Hardware
CEO Sid Sheth told CNBC in June that Raptor is slated to launch in 2027, but at Hot Chips the company gave no firm date or information on volume and pricing, and every performance figure shown, including 988 tokens per second per user on the 2.8-trillion-parameter Kimi K3 model at 1M-token context, is a d-Matrix projection built on early silicon.
The custom DRAM die
Raptor inverts the usual 3D stacking arrangement by putting the logic die on top and the DRAM underneath, so a cold plate sits directly on the compute silicon and the DRAM die doubles as the interposer, carrying PCIe and die-to-die signals down through its TSVs. “For the same amount of power, you can drive the bandwidth up,” Bhoja said during the session, “and so we were able to drive the bandwidth up here to 100 terabytes per second.”
The one-high stack achieves a power density of roughly 0.5W per square millimeter, which liquid cooling can handle, but the DRAM is designed for a junction temperature of 105°C, where retention collapses from a standard 32ms to 4ms, and the memory must refresh eight times more often. d-Matrix absorbed that penalty by shrinking each microbank to 1,366 rows and about 5.33MB, so a full refresh sweep costs only 1.37% of overall bandwidth.
The die carries 840 banks per chiplet, of which 72 (around 9%) are spares wired into a two-level mux chain the company calls bank chaining, letting any two failed banks anywhere on the die be switched out while channels stay symmetric. A [132,128] Reed-Solomon code on the logic die corrects two symbol errors per 128 bytes, with a CRC behind it. The interface has no PHY, no burst structure, and no sideband pins, so conventional data-bus inversion was impossible; d-Matrix instead compares each 128-byte flit to the previous one and stores a 1-bit inversion tag alongside the ECC metadata, recovering roughly 20% of the I/O power DBI would have saved. At full tilt, the vertical interface still burns 296W of the 422W per-package budget the ISCA paper discloses.
The bank geometry, spare-bank mux tree, refresh behavior, and interleaved ECC columns were all co-designed with the compute die. The 0.37 pJ/bit figure exists only because of that pairing, and no memory maker has anything like this die in its catalog.
Who’s supplying it?
d-Matrix has named TSMC for the N4P logic die and Alchip as its ASIC design and 2.5D/3D packaging partner, but neither company operates a DRAM fab, and across the Hot Chips talk, the ISCA paper, and every public announcement since the Pavehawk 3DIMC test silicon came online last September, the firm has never identified who fabricates its custom DRAM. Only three companies make leading-edge DRAM at volume, and all three are allocating capacity to HBM4 lines that are effectively sold out through 2026.
J.P. Morgan estimates DRAM prices will have risen more than 400% between the start of 2024 and the end of 2026. In addition, analysts have recorded contract price increases of 90% to 95% in Q1 2026 alone, and SK hynix CEO Kwak Noh-jung told Reuters in July that “customer demand will remain higher than our supply capacity even beyond 2030.”
Nvidia, the memory makers’ largest and most leveraged customer, is reportedly testing Rubin Ultra configurations with as little as 192GB because it may not be able to source enough HBM4E. A startup with roughly $450 million raised, asking a memory maker to run a bespoke die with non-standard bank geometry on capacity that could otherwise print HBM, is negotiating from a far weaker position than that, and until the supplier is named, Raptor’s 2027 volume plan rests entirely on an unknown, undisclosed dependency.
The die’s 11.4 MB/mm2 density is roughly half of HBM4’s 21.9 to 26.3 MB/mm2, Bhoja acknowledged during the Q&A: “A lot of the drop for us was also because we used a not-so-advanced DRAM. And so if we used a more mainline DRAM, just like the HBM4 guys are doing, we would be able to push that up almost all the way to the HBM4 numbers.” The density penalty therefore tracks back to whatever foundry arrangement d-Matrix currently has.
32GB per card
Raptor’s 32GB per card stands against 192GB to 288GB for HBM4-equipped accelerators, so d-Matrix sizes deployments at rack scale instead: 72 cards carry 2.3TB, enough to hold Kimi K3‘s weights at 4-bit precision with headroom for around 54 concurrent users at 1M context, by its own calculations. “Even with 32 gigabytes of memory capacity, we are able to solve SOTA models in a scale-up network, so no bits are wasted,” Bhoja said.
KV cache growth works against that arithmetic over time, and co-presenter Aayush Ankit, who led Raptor’s SoC architecture at d-Matrix before joining Meta’s MTIA team, explained the failure mode while dismissing SRAM alternatives: “We are making this unit of compute blazingly fast. Communication becomes a bottleneck soon enough.” Once models and context no longer fit a single rack, inference spills into the inter-card synchronization overhead that the vertical bandwidth was meant to eliminate, which the ISCA paper concedes.
Cerebras claimed 969 tokens per second on Llama 3.1 405B in November 2024, with third-party firm Artificial Analysis verifying the figure on live hardware, and that remains the closest published reference point for Raptor’s numbers. d-Matrix’s 988 tokens per second per user on a model seven times larger would be a step change if it holds water, but no third party has measured Raptor, and the comparison points in the ISCA paper are simulations anchored to early silicon characterization.
Asked by an Nvidia employee about multi-layer stacking plans, Bhoja said: “Our roadmap is still a work in progress, and we have a hard enough time trying to get one-high to work and work around all of the thermal issues of that.” Samsung brought its own version of the idea to Hot Chips with zHBM, a concept that stacks HBM directly on the processor and carries a claimed 70% power-efficiency gain over an HBM4E setup, with no production timeline before HBM5. The problem for d-Matrix is that Samsung runs its own DRAM fabs; d-Matrix doesn’t. Whether it can get wafers at volume remains to be seen.
Full d-Matrix Hot Chips 2026 presentation
