As artificial intelligence transitions from experimental models to practical applications, a new wave of startups is emerging to tackle the critical challenge of making AI systems run efficiently in production. This week, Inferact, the commercial spin-out of the popular open-source project vLLM, announced a massive $150 million seed funding round at an $800 million valuation, co-led by Andreessen Horowitz and Lightspeed Venture Partners.
The funding marks a significant milestone in what industry experts are calling the “inference revolution” – the shift from training AI models to deploying them at scale. Inferact’s technology, originally developed at UC Berkeley’s lab under Databricks co-founder Ion Stoica, helps AI tools run faster and more affordably, with existing users including Amazon’s cloud service and major shopping applications.
The Inference Infrastructure Boom
Inferact’s debut mirrors a broader trend in the AI ecosystem. Just days before, another UC Berkeley-born project, SGLang, spun out as RadixArk with a $400 million valuation led by Accel. These parallel developments highlight how venture capital is pouring into inference optimization technologies as companies struggle with the practical realities of running AI in production.
“Several large tech companies already run their inference workloads using vLLM, and SGLang has also gained significant popularity over the last six months,” noted Brittany Walker, General Partner at CRV. The inference market is exploding with activity, with companies like Baseten recently securing $300 million at a $5 billion valuation and Fireworks AI raising $250 million at a $4 billion valuation in October.
Beyond Software: The Hardware Frontier
While software optimization is crucial, some companies are taking a more radical approach to the inference challenge. Neurophos, an Austin-based photonics startup, recently raised $110 million to develop tiny optical processors that promise to revolutionize AI inferencing. Their technology claims to deliver 235 POPS (peta-operations per second) at just 675 watts, compared to Nvidia’s B200 GPU which delivers 9 POPS at 1,000 watts.
“Modern AI inference demands monumental amounts of power and compute,” explained Marc Tremblay, Corporate Vice President at Microsoft. “We need a breakthrough in compute on par with the leaps we’ve seen in AI models themselves.” Neurophos CEO Patrick Bowen added, “If you want to go fast, you have to solve the energy efficiency problem first. Because if you’re going to take a chip and make it 100 times faster, it burns 100 times more power.”
The Voice AI Dimension
The inference challenge extends beyond traditional AI applications into emerging domains like voice AI. LiveKit, a developer of infrastructure software for real-time AI voice and video applications, recently hit a $1 billion valuation after raising $100 million. The company powers OpenAI’s ChatGPT voice mode and serves customers including xAI, Salesforce, Tesla, and emergency service operators.
Meanwhile, Google DeepMind has reportedly acquired the CEO and several top engineers from voice AI startup Hume AI through a licensing agreement, bringing technology focused on understanding user emotions through voice to improve Gemini’s voice features. “Voice is the only acceptable input mode for wearables,” noted investor Vanessa Larco. “This acquisition will only accelerate the need for voice apps.”
Balancing Innovation with Practical Concerns
As the inference market heats up, questions are emerging about how these technologies will be monetized and integrated into existing business models. Google DeepMind CEO Demis Hassabis recently expressed surprise at OpenAI’s early move to introduce ads in ChatGPT, stating, “I’m a little bit surprised they’ve moved so early into that… In the realm of assistants, and if you think of the chatbot as an assistant that’s meant to be helpful… there is a question about how ads fit into that model?”
The rapid commercialization of inference technologies also raises questions about the future of open-source AI projects. Both vLLM and SGLang began as academic projects before transitioning to venture-backed startups, suggesting a new model for how AI innovation moves from research labs to commercial applications.
What This Means for Businesses
For enterprises looking to implement AI, the inference infrastructure boom represents both opportunity and complexity. On one hand, better inference tools mean lower costs and faster performance for AI applications. On the other, companies must navigate a rapidly evolving landscape of competing technologies and approaches.
The key takeaway for business leaders is clear: the AI race is no longer just about who has the best models, but about who can deploy them most effectively. As Inferact CEO Simon Mo and his team work to commercialize vLLM, they’re not just building a company – they’re helping define how AI will work in the real world.
With billions flowing into inference optimization and related technologies, the coming years will likely see significant innovation in how AI systems are deployed, monitored, and scaled. The question isn’t whether AI will transform business – it’s how efficiently and affordably that transformation will happen.