Semiconductor Engineering

Heterogeneous System With Specialized HW For Disaggregated LLM Inference (Princeton Univ., Univ. of Washington)

A new technical paper titled “SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference” was published by researchers at Princeton University and University of Washington. “Large ...
Semiconductor Engineering

Impact Of On-Chip SRAM Size And Frequency On Energy Efficiency And Performance of LLM Inference (Uppsala Univ.)

A new technical paper titled “Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling” was published by researchers at Uppsala University. “Energy consumption ...
Hosted on MSN

A closer look at Dynamo, Nvidia's 'operating system' for AI inference

GTC Nvidia's Blackwell Ultra and upcoming Vera and Rubin CPUs and GPUs dominated the conversation at the corp's GPU Technology Conference this week. But arguably one of the most important ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
Business Wire

Red Hat Launches the llm-d Community, Powering Distributed Gen AI Inference at Scale

Forged in collaboration with founding contributors CoreWeave, Google Cloud, IBM Research and NVIDIA and joined by industry leaders AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI and university ...
Computer Weekly

Red Hat launches llm-d community & project

The latest trends and issues around the use of open source software in the enterprise. Red Hat has announced the launch of llm-d, a new open source project designed to address generative AI’s future ...