DEV Community

TechLatest
TechLatest

Posted on • Originally published at Medium on

How to Run NVIDIA Nemotron 3.5 Lightning (Free): 4 Methods from Local GPU to Zero-Code Agent

#nvidiagpu #opensource #nvidia #aimodel

NVIDIA Nemotron 3.5 Lightning Setup Guide: vLLM, Ollama, OpenRouter & Agent IDE (2026)

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 is a hybrid Mixture-of-Experts (MoE) model featuring Mamba-2, MoE, and Attention layers. It has 30B total parameters with only 3B active per token, making it highly efficient for research, post-training (SFT/RL), and customization.

Critical Distinction: This BF16 version is the full-precision reference weight intended for training, fine-tuning, and evaluation. For production inference with optimized latency/throughput, NVIDIA recommends using the NVFP4 quantized variant instead. This guide focuses strictly on deploying this BF16 checkpoint.

Key Specifications

  • Active Params: 3B | Total Params: 30B
  • Context: Up to 1M tokens (256K recommended for single H100)
  • Architecture: Hybrid Mamba-2 + MoE + Attention
  • License: OpenMDW-1.1 (Commercial use allowed)
  • Release Date: August 11, 2026

Benchmarks

Supported GPU Configurations

Multi-GPU Configuration (Full 1M Context)

Key Memory Considerations

Hardware & System Prerequisites

Before installation, verify your system meets these requirements. The BF16 weights require significant VRAM compared to quantized versions.

Step-by-Step Process to Install & Run NVIDIA Nemotron 3.5 Lightning

Step 1: Create an Isolated Python Environment

Begin by creating a dedicated conda environment to prevent dependency conflicts with any existing PyTorch or CUDA installations on your system. Use Python 3.11 as it has the best compatibility with the current vLLM nightly builds and Mamba-2 kernel compilation. After activating the environment, install PyTorch with CUDA 12.6 support from the official PyTorch wheel index, then verify that the installation confirms CUDA availability.

conda create -n nemotron-lightning python=3.11 -y
conda activate nemotron-lightning

pip install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu126

python -c "import torch; print(f'PyTorch: {torch. __version__ }, CUDA: {torch.cuda.is_available()}')"
Enter fullscreen mode Exit fullscreen mode

Step 2: Install vLLM Nightly

This model requires vLLM nightly version 0.27.1 or later because stable vLLM releases do not yet support the hybrid Mamba-2 and MoE architecture used by Nemotron 3.5 Lightning. Install the pre-release build using pip with the extra index URL pointing to the PyTorch nightly CUDA 12.6 wheels, then verify the reported version meets the minimum requirement. If you encounter build errors, ensure your system has GCC 11 or newer, ninja-build, and the CUDA 12.6 toolkit installed, as vLLM nightly compiles custom Mamba kernels at install time.

pip install vllm>=0.27.1 --pre \
    --extra-index-url https://download.pytorch.org/whl/nightly/cu126

python -c "import vllm; print(f'vLLM: {vllm. __version__ }')"
Enter fullscreen mode Exit fullscreen mode

Step 3: Set Model Checkpoint Variables

Define the BF16 checkpoint path as an environment variable so you can reuse it across all deployment commands without retyping the full model identifier. If you plan to use DSpark speculative decoding on GB200 hardware, also define a second variable pointing to the NVFP4-DSpark draft model checkpoint. These variables will be referenced in every serve command throughout this guide.

export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
Enter fullscreen mode Exit fullscreen mode

Step 4: Launch the vLLM Server

Choose the deployment configuration that matches your hardware and launch the vLLM OpenAI-compatible server. For a single H100 or A100 80GB GPU, use the max-throughput configuration with prefix caching, async scheduling, the FlashInfer Mamba backend, and the Mamba SSM cache dtype set to float16 to conserve VRAM. Always include the reasoning parser, tool call parser, and auto tool choice flags regardless of hardware configuration.

vllm serve $MODEL_CKPT \
    --max-num-seqs 128 \
    --enable-prefix-caching \
    --async-scheduling \
    --mamba-backend flashinfer \
    --mamba-ssm-cache-dtype float16 \
    --enable-mamba-cache-stochastic-rounding \
    --mamba-cache-philox-rounds 5 \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice
Enter fullscreen mode Exit fullscreen mode

For eight H100 or H200 GPUs targeting the full 1M token context, enable tensor parallelism size 8, expert parallelism, and set the max model length to 1048576 with the long-context environment variable.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve $MODEL_CKPT \
    --moe-backend flashinfer_cutlass \
    --mamba-backend flashinfer \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --max-model-len 1048576 \
    --enable-expert-parallel \
    --tensor-parallel-size 8 \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice
Enter fullscreen mode Exit fullscreen mode

For GB200 hardware with DSpark speculative decoding, include the speculative config pointing to the DSpark draft model with five speculative tokens and disable prefix caching as required by the DSpark recipe.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve $MODEL_CKPT \
    --max-num-seqs 128 \
    --max-model-len 1048576 \
    --max-num-batched-tokens 10240 \
    --no-enable-prefix-caching \
    --async-scheduling \
    --speculative_config.model $DSPARK_CKPT \
    --speculative_config.num_speculative_tokens 5 \
    --mamba-backend flashinfer \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice
Enter fullscreen mode Exit fullscreen mode

Context Length Tip: If you are memory-constrained or want more KV-cache headroom at high concurrency, lower --max-model-len to match your actual workload and remove VLLM_ALLOW_LONG_MAX_MODEL_LEN=1.

Step 5: Verify the Deployment and Test Inference

Once the server is running, confirm it is healthy by sending a request to the OpenAI-compatible endpoint using the recommended sampling parameters of temperature 1.0 and top_p 0.95. Test basic chat completion first, then verify tool calling by passing a function definition and including the force_nonempty_content chat template kwarg required for coding agents. Finally, test reasoning mode toggling by sending requests with enable_thinking set to both True and False.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "What's the weather in Santa Clara?"}],
    tools=tools,
    max_tokens=16000,
    temperature=1.0,
    top_p=0.95,
    extra_body={"chat_template_kwargs": {"force_nonempty_content": True}},
)

print(response.choices[0].message.tool_calls)
Enter fullscreen mode Exit fullscreen mode

Controlling Reasoning Mode

Reasoning can be toggled at runtime through chat template kwargs:

  • Enable Thinking: extra_body={"chat_template_kwargs": {"enable_thinking": True}}
  • Disable Thinking: extra_body={"chat_template_kwargs": {"enable_thinking": False}}
  • Thinking Budget: extra_body={"chat_template_kwargs": {"thinking_budget": 4096}}

Step 6: Post-Training & Customization Notes

Since this BF16 release is primarily designed as a base for customization:

  • SFT / RL: Use NeMo RL and NeMo Gym frameworks. The model was trained with GRPO across math, code, science, and tool-use environments.
  • Quantization: Produce your own NVFP4, W4A16, or GGUF variants from this checkpoint for edge/mobile deployment.
  • Datasets: Pre-training data (20T+ tokens, cutoff Sep 2025) and post-training data (cutoff May 2026) are available via nvidia/nemotron-pre-training-datasets and nvidia/nemotron-post-training-v3.
  • Evaluation: Reproducible benchmarks are published in NeMo Gym with exact harness configurations.

Method 2: Run Nemotron 3.5 Lightning Using Ollama

NVIDIA has published an official Ollama model entry for Nemotron 3.5 Lightning, making local deployment significantly simpler than the vLLM approach. The model is available as a ready-to-run Ollama package at 25GB with support for up to 1M context length, tool calling, and reasoning modes out of the box. This method is ideal for developers who want to get started quickly without managing complex serving configurations, and it integrates directly with agent harnesses such as Claude Code, OpenCode, Hermes Agent, and OpenClaw via the NVIDIA NemoClaw stack.

Step 1: Install Ollama

Install the latest version of Ollama on your Linux or macOS machine using the official install script. After installation, confirm that the Ollama service is running and that your NVIDIA GPU is detected. Windows users can download the Ollama installer directly from the official website. Ensure your NVIDIA drivers are updated to the latest stable release so Ollama can fully utilize your GPU for inference.

curl -fsSL https://ollama.com/install.sh | sh

ollama --version

nvidia-smi
Enter fullscreen mode Exit fullscreen mode

Step 2: Pull the Official Nemotron 3.5 Lightning Model

Pull the official Nemotron 3.5 Lightning model directly from the Ollama registry. The default tag downloads the 30B parameter variant at 25GB with 1M context support. If you are on Apple Silicon hardware, use the MLX-optimized tag instead, which is 23GB and supports up to 256K context. Ollama will automatically download and cache the model weights in its local library upon completion.

ollama pull nemotron-3.5-lightning

ollama list
Enter fullscreen mode Exit fullscreen mode

For Apple Silicon Macs, use the MLX variant:

ollama pull nemotron-3.5-lightning:30b-mlx
Enter fullscreen mode Exit fullscreen mode

Step 3: Run Interactive Inference

Launch an interactive chat session with the model using a single command. Ollama handles GPU layer offloading, memory management, and streaming responses automatically. Type your prompts directly into the terminal and observe real-time output. The model supports both direct answers and reasoning modes natively through its built-in chat template.

ollama run nemotron-3.5-lightning

>>> What is speculative decoding and how does DSpark work?
>>> Write a Python function that implements binary search
>>> /set parameter temperature 1.0
>>> /set parameter top_p 0.95
Enter fullscreen mode Exit fullscreen mode

Step 4: Launch with Agent Harnesses

Nemotron 3.5 Lightning is purpose-built for always-on AI agents and ships with one-command launch support for popular agent harnesses. Use the ollama launch command to start the model inside Claude Code, OpenCode, Hermes Agent, or OpenClaw directly. Each harness is preconfigured to leverage the model’s tool-calling and reasoning capabilities for autonomous task execution across personal productivity, financial services, cybersecurity, telecom, and retail workflows.

ollama launch claude --model nemotron-3.5-lightning

ollama launch opencode --model nemotron-3.5-lightning

ollama launch hermes --model nemotron-3.5-lightning

ollama launch openclaw --model nemotron-3.5-lightning
Enter fullscreen mode Exit fullscreen mode

Step 5: Use the Ollama API for Application Integration

Ollama exposes an OpenAI-compatible REST API on port 11434 by default. To allow external applications or remote machines to access the model, set the OLLAMA_HOST environment variable before starting the service. Then use any OpenAI-compatible client library in Python, JavaScript, or cURL to send requests, specifying nemotron-3.5-lightning as the model name in each call.

OLLAMA_HOST=0.0.0.0:11434 ollama serve
Enter fullscreen mode Exit fullscreen mode

Python example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="nemotron-3.5-lightning",
    messages=[{"role": "user", "content": "Summarize the key risks in this financial document"}],
    temperature=1.0,
    top_p=0.95,
    max_tokens=8192,
)

print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

cURL example:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron-3.5-lightning",
    "messages": [{"role": "user", "content": "Classify this security alert by severity"}],
    "temperature": 1.0,
    "top_p": 0.95
  }'
Enter fullscreen mode Exit fullscreen mode

Step 6: Verify Performance and GPU Utilization

Confirm that Ollama is fully utilizing your GPU by monitoring nvidia-smi while processing a request. You should see VRAM allocation matching the 25GB model size on your H100, A100, RTX 5090, or GB200 device. Nemotron 3.5 Lightning delivers 4x higher throughput and 30% lower task completion time compared to other leading open models of similar size, so benchmark your specific workload by sending repeated requests and measuring tokens per second. If you observe CPU fallback or slow generation, ensure OLLAMA_NUM_GPU is set high enough to force full layer offloading to your GPU.

OLLAMA_NUM_GPU=99 ollama run nemotron-3.5-lightning

nvidia-smi
Enter fullscreen mode Exit fullscreen mode

When to Use Method 2 vs Method 1: Choose Ollama when you want the fastest path to local inference, need built-in agent harness integration, or are running on Apple Silicon with the MLX variant. Choose vLLM from Method 1 when you need maximum throughput at high concurrency, full 1M context on multi-GPU setups, speculative decoding with DSpark, or production-grade serving with prefix caching and async scheduling.

Method 3: Run Nemotron 3.5 Lightning Free via OpenRouter API

If you do not want to spend money on cloud GPU credits or manage local hardware, you can run Nemotron 3.5 Lightning completely free through the OpenRouter API. NVIDIA hosts this model at zero cost for both input and output tokens with 1M context support, tool calling, and reasoning capabilities included. This is the fastest way to start building with a 30B MoE model without downloading weights or configuring GPUs.

Step 1: Get Your Free API Key

Sign up at OpenRouter, navigate to your dashboard, and create a new API key. No credit card is required. Set it as an environment variable in your terminal so your code can access it securely without hardcoding secrets.

export OPENROUTER_API_KEY=sk-or-v1-your-actual-key-here
Enter fullscreen mode Exit fullscreen mode

Step 2: Make Your First Request

Use the OpenAI-compatible endpoint with the free model identifier nvidia/nemotron-3.5-lightning: free. The optional HTTP-Referer and X-Title headers let your app appear on OpenRouter leaderboards but are not required for functionality.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="nvidia/nemotron-3.5-lightning:free",
    messages=[{"role": "user", "content": "How many r's are in strawberry?"}],
    temperature=1.0,
    top_p=0.95,
)

print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Step 3: Enable Streaming and Reasoning Tokens

Add stream set to true to receive responses as server-sent events for real-time output. To see the model’s step-by-step thinking process, include the reasoning parameter in your request and read the reasoning_details array from the response. When continuing a conversation, always pass the complete reasoning_details back in the message history so the model can resume reasoning from where it left off.

curl -N https://openrouter.ai/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning:free",
    "stream": true,
    "reasoning": {"enabled": true},
    "messages": [{"role": "user", "content": "Solve x^2 - 5x + 6 = 0"}]
  }'
Enter fullscreen mode Exit fullscreen mode

Step 4: Use Tool Calling and Alternative SDKs

The free tier fully supports OpenAI-style tool calling, TypeScript SDKs, and Anthropic Messages API format through dedicated endpoints. Pass your function definitions in the tools array exactly as you would with OpenAI, and the model will return structured tool_calls in the response. For TypeScript projects, install the official @openrouter/sdk package and initialize it with your API key to get typed streaming responses with automatic reasoning token tracking in the usage object.

Endpoint Reference: Chat completions at POST /api/v1/chat/completions, Responses API at POST /api/v1/responses, and Anthropic-compatible messages at POST /api/v1/messages. All three accept the same free model identifier and bearer token authentication.

Use Nemotron 3.5 Lightning in a Zero-Coding Agent IDE

Use the model inside AI coding agents like OpenCode, Claude Code, or Hermes Agent without writing any code. Routes through OpenRouter free tier — zero cost.

Step 1: Launch your agent IDE in the terminal

Step 2: Select OpenRouter as provider in the setup wizard

Step 3: Paste your free OpenRouter API key when prompted

Step 4: Choose model → nvidia/nemotron-3.5-lightning:free

Step 5: Press Enter to save when you see “Ready to connect”

Conclusion

NVIDIA Nemotron 3.5 Lightning represents a significant step forward in open, efficient AI models — delivering 30B total parameters with only 3B active per token through its hybrid Mamba-2 + MoE architecture. With support for up to 1M context tokens, native tool calling, configurable reasoning modes, and a commercially permissive OpenMDW-1.1 license, it is purpose-built for the next generation of always-on AI agents across coding, finance, cybersecurity, telecom, and retail workflows.

Throughout this guide, we covered four distinct deployment paths to match every developer’s needs and infrastructure constraints:

  • Method 1 (vLLM) gives you maximum throughput, full 1M context on multi-GPU setups, speculative decoding with DSpark, and production-grade serving with prefix caching — ideal for data center deployments on H100, H200, A100, or GB200 hardware.
  • Method 2 (Ollama) provides the fastest path to local inference with a single pull command, built-in agent harness integration for Claude Code, OpenCode, Hermes Agent, and OpenClaw, plus Apple Silicon support via the MLX variant — perfect for developers who want privacy and simplicity.
  • Method 3 (OpenRouter API) lets you run the model completely free with zero infrastructure cost, no weight downloads, and instant access from any device — the best choice for prototyping, evaluation, and lightweight integrations.
  • Method 4 (Zero-Coding Agent IDE) connects the free OpenRouter endpoint directly into AI coding agents through a visual provider wizard — enabling non-coders to leverage a 30B MoE model inside their editor without writing a single line of API code.

Whether you are fine-tuning the BF16 reference weights for domain-specific customization, deploying quantized NVFP4 variants for low-latency production inference, or simply experimenting with agentic workflows at zero cost, Nemotron 3.5 Lightning offers a flexible entry point that scales from a single laptop to an eight-GPU data center rack.

The model’s release alongside open training datasets, reproducible NeMo Gym evaluation recipes, and the NVIDIA NemoClaw security stack for always-on agents signals NVIDIA’s commitment to an open ecosystem where developers retain full control over their AI infrastructure. As the model matures and community quantizations, GGUF conversions, and fine-tuned variants proliferate across Hugging Face and Ollama, expect Nemotron 3.5 Lightning to become a default backbone for specialized AI agents throughout 2026 and beyond.

Pick the method that matches your hardware, budget, and use case — and start building with one of the most efficient open 30B models available today.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

Website: https://www.techlatest.net/

Newsletter: https://substack.com/@techlatestnet

Twitter: https://twitter.com/TechlatestNet

LinkedIn: https://www.linkedin.com/in/techlatest-net/

YouTube:https://www.youtube.com/@techlatest_net/

Blogs: https://medium.com/@techlatest.net

Reddit Community: https://www.reddit.com/user/techlatest_net/

Top comments (0)