SRAM inference racks, a $110 billion cheque, and search as a gateway tool

At Hot Chips on 24 August 2026 Nvidia announced that Groq 3 LPX is in full production. The rack is positioned as an extension of Vera Rubin NVL72: NVL72 for general training and high-throughput inference, LPX for interactive token generation — the step that decides whether an agent feels instant. Jensen Huang called inference the growth engine of AI and LPX a way to raise token generation rates for context-heavy agent work.

Nvidia said Artificial Analysis recorded 3,400 output tokens per second on Gemma 4 31B, an open agentic model, at 100,000-token context — the fastest result they cite for that model — and ‘4x faster responsiveness’ than the nearest alternative platform for agents and other latency-sensitive loads. Those are Nvidia-quoted benchmark figures; treat them as vendor-reported until you run your own model.

A technical blog describes a 256-chip LPX with 315 PFLOPS of inference compute, 128 GB total SRAM, 40 PB/s on-chip SRAM bandwidth and 640 TB/s scale-up. Each LPU keeps about 500 MB of SRAM as the working set, with the compiler placing weights, activations and KV state explicitly rather than relying on caches. Chip-to-chip uses 96 links at 112 Gbps (about 2.5 TB/s aggregate bidirectional I/O per device in Nvidia’s account).

Nebius is the first AI cloud named to put LPX into production via Nebius Token Factory, promising the same API developers already use. Groq, the inference cloud, is listed among the earliest adopters after Nebius. LPX sits in a wider Vera Rubin factory story: BlueField-4 DPUs, Vera CPU racks, Spectrum-6 Ethernet.

The split is now productised: GPUs (or NVL72) for prefill, training and broad model support; SRAM machines for decode latency. Builders who only buy ‘a GPU cloud’ will leave interactivity on the table — and will overpay if they use LPX-class hardware for jobs that do not need it.

Why it mattersAgent products are limited by decode latency at long context. A production SRAM rack next to Rubin makes that a buying decision, not a research hope.

OpenAI said on 27 February 2026 that it is taking $110 billion of new investment at a $730 billion pre-money valuation. Amazon is in for $50 billion, Nvidia $30 billion and SoftBank $30 billion. Other financial investors were expected to join as the round progressed. AP and OpenAI use the $730 billion pre-money figure; Reuters described the deal as valuing the company at $840 billion, which matches a simple pre-plus-new-cash reading. Amazon’s cheque is staged: $15 billion first, then $35 billion when conditions are met.

The round is as much infrastructure as equity. OpenAI announced a multi-year Amazon partnership; AP reported AWS as the exclusive third-party cloud in that deal. Nvidia collaboration expands to 3 GW of dedicated inference capacity and 2 GW of training on Vera Rubin systems, on top of Hopper and Blackwell already running across Microsoft, OCI and CoreWeave. OpenAI said the new valuation lifts the OpenAI Foundation’s stake in OpenAI Group to over $180 billion.

CNBC noted the round more than doubled OpenAI’s prior record raise and followed a $500 billion secondary valuation in the previous October. Amazon’s $50 billion is, Bloomberg reported, the largest amount it has put into any company. The competitive backdrop is other labs raising tens of billions (CNBC cited Anthropic’s then-latest $30 billion and xAI’s $20 billion) and Microsoft, still a major OpenAI partner, separately shopping for AI startups according to later Reuters reporting.

Staged Amazon money, gigawatts of Nvidia, and a nonprofit parent with a giant marked-up stake are not the same as a finished IPO. They do set 2026’s bargaining table: model APIs, cloud regions, and chip supply will keep following these three cheques. Customers should assume OpenAI has the cash to keep training — and that Amazon and Nvidia will want distribution and silicon filled in return.

For everyone not named in the press release, the implication is concentration. Independent model companies, inference clouds and application startups will either ride these rails or differentiate on price, openness, or data that never leaves the building.

Why it mattersFrontier-model supply, cloud defaults and chip allocation are being set by a handful of overlapping cheques. Product teams need a second-vendor plan even if they standardise on GPT-6 today.

Cloudflare announced a Web Search API as part of AI Gateway, its control plane for model traffic. Instead of wiring a separate search vendor with its own key, bill and log silo, developers can call search beside their other AI Gateway requests. Launch partners are Ceramic.ai, Exa and Linkup. Cloudflare says it sells those partners’ search at their list API prices with no extra markup.

The integration pitch is operational. Search queries draw down the same AI Gateway credit balance. Requests appear in the same observability logs. Access controls decide who may call which provider. Cloudflare says it will identify partners that support zero data retention so buyers can see whether query text is kept. A standalone REST API is also available, not only the Gateway path.

Cloudflare also said it is building native server tools in AI Gateway so developers do not have to define common tools themselves. Web search is listed among the first. That is a bid to become the default toolbelt for agents: one account, one log, one policy, many providers.

Caveats: this is still someone else’s index. Quality, freshness and geographic coverage will differ by partner. Legal access to content and how answers are grounded remain the caller’s problem. Teams that already have a tightly negotiated Exa or Linkup contract may not need a middle layer; teams drowning in keys and invoices might.

For agent builders the win is boring and large. Grounding is no longer a second vendor onboarding. It is another line in the gateway policy file, with the same spend cap as the model.

Why it mattersAgents that cannot search hallucinate with confidence. Putting search on the same bill and policy as the model makes grounding an infrastructure default instead of a side project.