Summary:
d-Matrix introduces the Corsair PCIe card for fast LLM inference.
Corsair boasts 10PFLOPs FP4 compute power and 2GB of SRAM.
Utilizes LPDDR5 memory to reduce costs compared to HBM.
Claims 10x better performance over traditional GPU solutions.
Plans for mass production in Q2 2025 with future developments in memory technology.
Introduction
Silicon Valley startup d-Matrix, backed by Microsoft, has introduced a cutting-edge chiplet-based solution designed specifically for fast, small-batch inference of large language models (LLMs) in enterprise settings. Their innovative architecture leverages a compute-in-memory approach that utilizes modified SRAM cells, focusing on speed and energy efficiency.
Key Features of Corsair
- 10PFLOPs FP4 compute power and 2GB of SRAM performance memory.
- Utilizes LPDDR5 memory, avoiding the high costs associated with HBM memory.
- Capable of handling up to 256GB per card, optimizing performance for larger models or batch inference workloads.
Performance Advantages
d-Matrix claims that Corsair offers 10x better interactive performance, 3x energy efficiency, and 3x better cost-performance compared to traditional GPU solutions like the Nvidia H100.
Overcoming Limitations
Sree Ganesan, head of product at d-Matrix, emphasized that current solutions often struggle with the memory wall, leading to the need for excessive compute and power consumption. d-Matrix’s solution focuses on memory bandwidth and addressing the memory-compute barrier to improve efficiency.
Innovation in Memory Computing
The startup has built a digital in-memory compute core that allows for multiply-accumulate operations to occur directly within memory, achieving 150 terabytes per second bandwidth. This strategic approach is aimed at solving the long-standing memory wall challenge in AI processing.
Future Plans
d-Matrix was founded in 2019, with CEO Sid Sheth noting that they initially took a leap of faith on inference technology. Now, with the rise of AI applications like ChatGPT, their vision is becoming increasingly relevant. Corsair is set to enter mass production in Q2 2025, with plans for a next-generation ASIC, Raptor, that will incorporate 3D-stacked DRAM for enhanced reasoning capabilities.
Conclusion
The advancements showcased by d-Matrix with the Corsair PCIe card represent a significant step forward in the realm of AI processing, promising to bridge the gap between memory and compute power effectively.
Comments