Google Cloud, Red Hat and IBM release llm-d 1.0 for distributed LLM inference
Original titleGoogle Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
- Outlet
- MarkTechPost
- Published
- Importance
- 4/5
Summary
Google, Red Hat, and IBM released llm-d 1.0, a Kubernetes-native open-source stack for distributed LLM inference. The platform adds multi-tenant flow control, zero-downtime upgrades, and agent-aware routing across vLLM, SGLang, TPUs, and GPUs.
Original excerpt
Google, Red Hat and IBM released llm-d 1.0, an open-source Kubernetes-native stack for distributed LLM inference. It adds multi-tenant flow control, zero-downtime upgrades and agent-aware routing across vLLM, SGLang, TPUs and GPUs. The post Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets appeared first on MarkTechPost.
#inference#kubernetes#llm-d