The age of AI inference doesn’t care how many GPUs you have. It’s not the size of your GPU cluster that matters, but how you can optimize what you have to work more efficiently and cost-effectively on AI workloads.
Even Nvidia gets that, as Nvidia CEO Jensen Hunag has a made a point in recent months of highlighting the ongoing importance of compute resources and techniques like test time scaling in creating and working with new reasoning and inference-focused AI models. That came after Nvidia’s hyping of AI’s hunger for more and more GPU resources was called on the carpet by China’s DeepSeek and its more optimized approach to AI inference that relied on fewer and less cutting-edge GPUs.
Now, Atlas Cloud is the latest to show that better AI inference can be achieved through optimization techniques. The company, which offers a cloud service that allows AI users to leverage clusters of up to 5,000 GPUs (Nvidia H100s, owned by Atlas), announced the launch of Atlas Inference, an AI inference platform that it says dramatically reduces GPU and server requirements, enabling faster, more cost-effective deployment of large language models (LLMs).
Atlas Cloud co-developed the platform with SGLang, an open-source inference engine, and incorporated into the platform four key technical innovations:
- Prefill/Decode Disaggregation: Separates compute-intensive operations from memory-bound processes to optimize efficiency
- DeepExpert (DeepEP) Parallelism with Load Balancers: Ensures over 90% GPU utilization
- Two-Batch Overlap Technology: Increases throughput by enabling larger batches and utilization of both compute and communication phases simultaneously
- DisposableTensor Memory Models: Prevents crashes during long sequences for reliable operation
The result is a platform that processes more tokens with less hardware, according to Atlas CEO Jerry Tang, who said in a statement, "Our platform's ability to process 54,500 input tokens and 22,500 output tokens per second per node means businesses can finally make high-volume LLM services profitable instead of merely break-even.”
He said Atlas is out to “fundamentally break down the economics of AI deployment," with an approach that reduces infrastructure requirements and operational costs while addressing hardware costs, which represent up to 80% of AI operational expenses, according to a Gartner report.
Atlas claimed that in a comparison to DeepSeek’s published performance results, Atlas Inference’s 12-node cluster beat DeepSeek’s reference implementation of their DeepSeek-V3 model while using only two-third of the servers.
The company also claimed Atlas Inference’s performance exceeds other major players like Amazon, Nvidia, and Microsoft, delivering up to 2.1 times greater throughput using just 12 nodes compared to competitors’ larger setups. It maintains sub-5-second first-token latency and 100-millisecond inter-token latency with more than 10,000 concurrent sessions, ensuring a seamless and superior user experience at scale, Atlas said.
"This platform represents a significant leap forward for AI inference," said Yineng Zhang, Core Developer at SGLang. "What we built here may become the new standard for GPU utilization and latency management. We believe this will unlock capabilities previously out of reach for the majority of the industry regarding throughput and efficiency."