Nemotron 3.5 Lightning and NeMo Switchyard: How NVIDIA Is Solving the Execution Layer Problem in Agentic AI

When people talk about AI agents, the conversation usually centers on the planning model — the frontier system that reasons, decomposes tasks, and decides what to do next. But in practice, most of an agent's token budget is spent on something far less glamorous: tool calls, result validation, format conversions, and subagent delegation. These are the repetitive, high-volume steps that keep a workflow moving, and they don't need a 550-billion-parameter model to handle them.

NVIDIA's August 11 release of Nemotron 3.5 Lightning and NeMo Switchyard is a direct response to this mismatch. Together, they form a two-part answer to a practical question: how do you build always-on agents that are both capable and affordable?

The Execution Layer Problem

Modern agentic systems increasingly operate as "systems of models" — ensembles where different models handle different parts of a workflow. A frontier reasoning model like Nemotron 3 Ultra or GPT-5.6 might orchestrate a complex plan, but the bulk of the actual work — running git commands, validating tool outputs, answering billing questions, reviewing code diffs — is repetitive and well-defined.

Routing all of that execution traffic through a frontier model is expensive and slow. But routing it through a model that can't reliably follow structured instructions or handle tool schemas is worse. The execution layer needs a model that is fast, accurate on constrained tasks, and cheap enough to run at high volume.

That's the gap Nemotron 3.5 Lightning is designed to fill.

What Nemotron 3.5 Lightning Actually Is

Nemotron 3.5 Lightning is a 30-billion-parameter Mixture-of-Experts (MoE) model with only 3 billion active parameters per token. The MoE architecture means a learned router sends each token to a small subset of specialized "expert" sub-networks, so the model gets the representational capacity of a 30B model at the compute cost of a 3B one.

The model ships with several inference optimizations baked in:

Speculative decoding via multi-token prediction (MTP). During pretraining, the model was trained to predict multiple tokens simultaneously, not just the next one. This enables speculative decoding at inference time, where a draft model proposes several tokens that the main model verifies in parallel — significantly increasing throughput. NVIDIA ships two draft models alongside Lightning: DSpark (optimized for low-concurrency workloads on DGX Spark) and DFlash (better suited for higher-concurrency scenarios).

NVFP4 quantization. The model ships with both BF16 and NVFP4 checkpoints. NVFP4 uses 4-bit floating-point representation with specialized kernels that run across NVIDIA Blackwell, Hopper, and Ampere GPUs, enabling near-lossless performance at a significantly reduced memory footprint.

Harness-optimized training. The model was specifically trained for popular agent harnesses like OpenClaw and Hermes Agent, which means it's tuned to produce the structured outputs those frameworks expect — reducing parsing errors and retry loops in production.

The result, according to NVIDIA's PinchBench benchmarks, is 86% accuracy on agentic tasks while completing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy levels. On the Artificial Analysis Intelligence Index — which aggregates nine evaluations across agentic tasks, coding, scientific reasoning, and general intelligence — Lightning sits on the accuracy-speed Pareto frontier for its class.

NeMo Switchyard: Routing Work to the Right Model

Having a fast execution model is only half the problem. The other half is deciding, at runtime, which requests should go to Lightning and which should go to a frontier model. That's what NeMo Switchyard handles.

Switchyard is an open-source model routing library that integrates with existing agent frameworks — LangChain, LiteLLM, Kong AI Gateway, and others — without requiring developers to rewrite their applications. It analyzes incoming requests and routes them to the most appropriate model based on configurable priorities: quality, latency, or cost.

The routing logic is tunable. A team that needs frontier-level accuracy on planning steps but can tolerate a smaller model for execution can configure Switchyard to reflect that. A team optimizing purely for cost can push more traffic to Lightning. The router itself can be customized with different routing algorithms.

The partner case studies NVIDIA published alongside the release give a concrete sense of what this looks like in practice:

• Ramp used Switchyard to match frontier model performance on their SWE-Bench variant while cutting costs by 58% and runtime by 33%.
• LangChain achieved 74% lower cost across 145 multi-turn Deep Agents tasks by routing only 7% of calls to a frontier model, at a 6% accuracy tradeoff.
• Cognition integrated Switchyard into Devin Desktop, achieving near-frontier performance on FrontierCode Main while reducing mean cost by 28%.
• Boomi achieved 100% domain-routing accuracy, sending 59% of traffic to a 5x faster fine-tuned model and reducing later-turn latency by 21%.

These aren't theoretical projections — they're production measurements from teams that have already deployed the system.

Customization and Deployment

One of the more practical aspects of this release is how accessible customization is. Because Lightning is a small model, LoRA fine-tuning is fast and cheap. NVIDIA also released Nemotron-RL-Agentic-Terminal-Pivot, an open agentic reinforcement learning dataset used to post-train Lightning's coding agent capabilities — giving teams a starting point for domain-specific adaptation.

The model runs on hardware ranging from an NVIDIA RTX 5090 or DGX Spark to full data center deployments. It's available on Hugging Face, ModelScope, OpenRouter, and as an NVIDIA NIM microservice on build.nvidia.com. The weights, training data, and recipes are released under the OpenMDW-1.1 license, which permits commercial use and modification.

NeMo Switchyard is available on GitHub and is being integrated into partner platforms including Kong AI Gateway and LiteLLM's proxy layer.

What This Means for Practitioners

The practical implication of this release is that the "use one model for everything" approach to agentic AI is becoming harder to justify economically. The cost and latency gap between frontier models and well-optimized execution models is large enough that routing matters — and the tooling to do it is now available off the shelf.

For teams building always-on agents, the architecture this points toward is a two-tier system: a frontier model for planning and complex reasoning, and a fast, customizable execution model for the high-volume work that dominates actual token spend. Switchyard makes that division of labor programmable without requiring a custom routing layer.

The broader trend here is worth noting. As agentic workloads mature, the bottleneck is shifting from "can the model do this?" to "can we run this at scale without the cost becoming prohibitive?" Nemotron 3.5 Lightning and NeMo Switchyard are a direct answer to that second question.

Whether this specific combination becomes the standard execution layer for enterprise agents remains to be seen. But the problem it's solving — making high-volume agentic execution fast, accurate, and affordable — is real, and the approach is technically sound.