GPT-OSS in Practice: Native 4‑bit MoE, MXFP4, and the New Inference Stack

Key takeaways

OpenAI’s GPT‑OSS models bring native 4‑bit (MXFP4) MoE checkpoints under a permissive license and a unified chat format (Harmony) that integrates cleanly with modern inference stacks.
MXFP4 is E2M1 block‑floating quantization with 32‑value blocks – slashing ~4× VRAM versus bf16 while maintaining excellent runtime throughput via specialized Triton kernels.
Hugging Face Transformers now ships day‑one support: downloadable kernels (RMSNorm, Flash‑Attn‑3, MegaBlocks MoE), MXFP4, Dynamic KV Cache, Continuous Batching, and built-in Tensor/Expert Parallelism.
You can run 20B on a 16 GB Blackwell‑class GPU (or fallback to bf16 on 48 GB GPUs), and 120B on an 80 GB Hopper. Practical serving is straightforward with vLLM using Harmony prefill and stop‑tokens.
Fine‑tuning native‑4‑bit models requires discipline: prefer MXFP4‑aware paths when your hardware supports it; otherwise consider dequantized bf16 or vendor tooling, knowing the trade‑offs.

The Sanity Check

The open-weight model landscape has been a chaotic spectacle of capability chasing. GPT-OSS feels like a course correction. It’s less about a new peak on a leaderboard and more about a new floor for operational sanity. It arrives with two Mixture-of-Experts (MoE) checkpoints (≈20B and ≈120B) where only a fraction of experts are active per token. Coupled with native 4-bit weights and a standardized format compatible with the OpenAI API ecosystem, this is one of the cleanest on-ramps for teams that want serious capability without maintaining exotic, bespoke infrastructure.

Architecture at a glance

There are no radical inventions here, and that’s the point. This is a consolidation of what works: a sober, pragmatic design for real-world workloads.

Token‑choice MoE with softmax‑after‑top‑k gating.
SwiGLU feed‑forward experts.
Hybrid attention: alternating global and sliding‑window (128‑token) layers.
RoPE for long context (up to 128k tokens).
Learned attention sinks to stabilize long‑context attention.
Harmony chat format + GPT‑4o‑aligned tokenizer for smooth tool‑use and structured outputs.

Here’s the routing math you’ll often see in token‑choice MoE (per token hidden state $h$ ):

$\textrm{scores} = W_{g} h,\quad I = \operatorname{topk}(\textrm{scores}, k),\quad \alpha_{i} = \frac{\exp(\textrm{scores}_{i})}{\sum_{j\in I} \exp(\textrm{scores}_{j})} \;\; (i\in I),$ $\textrm{MoE}(h) = \sum_{i\in I} \alpha_{i} \cdot E_{i}(h),$ where $E_{i}$ is expert $i$ , and routing normalizes after top‑k selection.

MoE flow

MXFP4: Taming the Memory Beast

Memory, not FLOPS, is the final boss of inference. MXFP4 uses a 4‑bit floating layout (E2M1) plus blockwise scaling to attack the VRAM bottleneck.

Group weights into blocks of 32 values.
Store a shared 8-bit scale per block.
Store each value as sign + 2‑bit exponent + 1‑bit mantissa relative to the block scale.

This preserves dynamic range while slashing memory traffic. In practice:

Model	Precision path	Typical VRAM (single device)	Notes
GPT‑OSS‑20B	MXFP4	≈16 GB	Best on Blackwell‑class GPUs; Triton MXFP4 kernels
GPT‑OSS‑20B	bf16 (fallback)	≈48 GB	Use when MXFP4 kernels/CC not available
GPT‑OSS‑120B	MXFP4	≈80 GB	Fits a single H100 80 GB; TP/EP recommended for throughput

Tip: MXFP4 kernels demand recent Triton and compute capability ≥ 7.5. When unavailable, Transformers gracefully degrades to higher precision. No drama.

Running GPT‑OSS with Transformers

The implementation is refreshingly free of boilerplate. The transformers library now possesses the intelligence to select the optimal path – a welcome departure from the days of manual kernel management and arcane flags. The pattern below is minimal and production-ready.

# python>=3.10, torch>=2.7, transformers>=4.55 (or newer)
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "openai/gpt-oss-20b"

# 1) Tokenizer
_tok = AutoTokenizer.from_pretrained(model_id)

# 2) Model: let transformers pick the right dtype & MXFP4 kernels when supported
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",   # spreads layers if multiple GPUs present
    dtype="auto",        # MXFP4 on supported GPUs, otherwise bf16
    # If you explicitly want community kernels for bf16 paths:
    # use_kernels=True,
    # attn_implementation="kernels-community/vllm-flash-attn3",  # if Hopper/driver supports it
).eval()

# 3) Chat template (Harmony‑compatible)
messages = [
    {"role": "system", "content": "Be concise and cite sources when asked."},
    {"role": "user", "content": "Explain KV cache in one paragraph."},
]

inputs = _tok.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=160)

print(_tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Memory sanity check (MXFP4 vs dequantized)

A quick demonstration of the physics involved.

import torch
from transformers import AutoModelForCausalLM, Mxfp4Config

model_id = "openai/gpt-oss-20b"

# Load dequantized (memory‑heavier) for comparison
m_deq = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    quantization_config=Mxfp4Config(dequantize=True),
)
print(f"Dequantized alloc: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

del m_deq; torch.cuda.empty_cache()

# Load the default quantized (memory‑efficient) path
m_q = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", dtype="auto")
print(f"Quantized alloc: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

Expect ~4× lower allocated VRAM in the quantized path on supported hardware. The numbers don’t lie.

Long‑context without the Pain

Long context has been the siren song of recent models, often leading to the rocks of quadratic complexity and OOM errors. GPT-OSS alternates global and 128-token sliding attention, a practical compromise. Transformers’ DynamicCache respects this architecture: KV tensors for sliding layers stop growing past their window, effectively halving KV cache usage for these hybrid stacks.

from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache

model_id = "openai/gpt-oss-20b"
_tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", dtype="auto").eval()

messages = [
    {"role": "system", "content": "Always answer in two sentences."},
    {"role": "user", "content": "Summarize the Gauss–Newton method."},
]

inputs = _tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
cache = DynamicCache(config=model.config)

out = model.generate(**inputs, past_key_values=cache, max_new_tokens=800)
print(_tok.decode(out[0][inputs.shape[-1]:]))

Distributed Inference: The Necessary Evil of Scale

Scaling a model beyond a single chassis is where proofs-of-concept die and production systems are born. TP (sharding matrices across GPUs) and EP (sharding experts across GPUs) are the necessary primitives for this transition.

# torchrun --nproc-per-node 4 run_tp.py
import torch
from transformers import PreTrainedTokenizerFast, GptOssForCausalLM

mid = "openai/gpt-oss-120b"
_tok = PreTrainedTokenizerFast.from_pretrained(mid)
model = GptOssForCausalLM.from_pretrained(
    mid,
    tp_plan="auto",   # built-in sharding recipe
    dtype="auto",
).eval()

msgs = [{"role": "user", "content": "Explain KV caching briefly."}]
inputs = _tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
with torch.inference_mode():
    gen = model.generate(**inputs, max_new_tokens=128)
print(_tok.decode(gen[0][inputs.shape[-1]:]))

For MoE models, enabling Expert Parallelism is non-negotiable.

# torchrun --nproc-per-node 4 run_ep.py
from transformers import GptOssForCausalLM, PreTrainedTokenizerFast
from transformers.distributed import DistributedConfig

mid = "openai/gpt-oss-120b"
_tok = PreTrainedTokenizerFast.from_pretrained(mid)
model = GptOssForCausalLM.from_pretrained(
    mid,
    distributed_config=DistributedConfig(enable_expert_parallel=True),
    dtype="auto",
).eval()

Rules of thumb

Prefer EP+TP for MoE at scale; keep ranks within the same node (NVLink) to avoid network bottlenecks.
device_map="auto" is about placing layers. TP is about sharding computation. Use TP for real throughput.

Serving with vLLM + Harmony

A model is only as useful as its interface. The Harmony format is a crucial piece of standardization – a lingua franca that ends the bespoke parsing hell that plagued early structured generation. It defines channels (assistant, tool calls) and stop tokens that make structured output reliable. vLLM leverages this natively.

# pip install vllm openai-harmony
from vllm import LLM, SamplingParams
from openai_harmony import HarmonyEncodingName, load_harmony_encoding, Conversation, Message, Role

enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
convo = Conversation.from_messages([
    Message.from_role_and_content(Role.SYSTEM, ""),
    Message.from_role_and_content(Role.USER, "Return a JSON with keys: city, temp_c."),
])
prefill_ids = enc.render_conversation_for_completion(convo, Role.ASSISTANT)
stop_ids = enc.stop_tokens_for_assistant_actions()

llm = LLM(model="openai/gpt-oss-20b", trust_remote_code=True)
outputs = llm.generate(prompt_token_ids=[prefill_ids], sampling_params=SamplingParams(max_tokens=96, stop_token_ids=stop_ids))
print(outputs[0].outputs[0].text)

For multi-tenant serving, combine client-side Continuous Batching with vLLM’s scheduler to keep the GPU fed and utilization high.

Fine‑Tuning: Navigating the Minefield

Fine-tuning a quantized model is where the abstractions leak. You are no longer just a model user; you are interacting with the low-level representation of its weights. Tread carefully.

MXFP4‑native FT is the cleanest path, avoiding format conversions, but requires a fully compatible toolchain.
bf16 FT is the universal solvent; just accept the higher VRAM cost and the potential for quantization drift when you save back to MXFP4.
Tooling like Unsloth can brute-force 4-bit training on older hardware by substituting quantization schemes; validate downstream accuracy relentlessly before shipping.

Checklist before you fine‑tune

Verify compute capability ≥7.5 if you intend to train in native MXFP4.
Freeze non‑convertible modules when starting from a quantized state; use LoRA/DoRA to manage memory.
Always A/B test on held‑out tasks that reflect reality, not zero-shot leaderboards. Catch regressions before your users do.

Evaluation: Resisting the Leaderboard Siren Song

Public leaderboards are a form of insight porn – useful for broad calibration, but terrible for making high-stakes decisions. Real evaluation is about testing against the cold, hard reality of your specific use case.

Layer	What to test	Why it matters
Behavioral	Instruction‑following, refusal, tool use	Does it actually follow instructions, or just hallucinate structured gibberish?
Knowledge/Math	MMLU‑style subsets, AIME‑like reasoning	Does it know things or just parrot syntax?
Robustness	Long‑context stability, input perturbations	Does it collapse under pressure?
Latency/Cost	TP/EP scaling, batch‑mix	Can you afford to run it at the required SLA?

Expect rapid improvement as kernels and caches mature. Re-test before forming strong opinions.

Hardware Guidance

This is not a negotiation. These are the tools for the job.

Single‑GPU dev: Blackwell 16–24 GB (20B @ MXFP4) or a 48 GB Hopper/Ada (bf16 fallback).
Serious eval/FT: A pod of L4s/L40Ss, A100s, or H100s, configured for TP/EP.
Throughput: Prioritize NVLink nodes and recent Triton. Enable prefetching and pinned memory in your dataloaders.

What this means for builders

GPT-OSS represents a maturation point. The arms race for parameter counts is giving way to a focus on efficiency, standardization, and deployability. The innovation here isn’t a single architectural miracle, but the careful removal of friction at every layer of the stack: quantization, kernels, caching, parallelism, and a standard format.

If you’ve been on the sidelines, waiting for the pragmatic, affordable LLM stack to emerge from the chaos, this is a strong signal. The era of heroic, bespoke LLM deployment is ending. The era of building durable, scalable, and affordable intelligence on an open foundation is here. Start building.