Building NCM vs LLMs: How the Deep Learning Workflow is Evolving

Over the last couple of weeks I kept seeing posts saying NCM is "the end of LLMs". That got my attention, so I looked into what it actually is. The idea underneath is genuinely interesting, it's a lot simpler than the hype makes it sound, and it isn't the end of anything yet.

In my last post we pictured a model as a box full of dials, and an LLM as that box running in a loop, one token at a time. Keep that picture handy. NCM changes the loop, not the box.

The short version

NCM stands for Next-Concept Modeling. It comes from Stanford's Hazy Research lab, working with Together AI, and was published in October 2026 (paper, code).

An LLM writes the way you type a text message: one word, then the next, then the next. NCM writes the way you'd write an essay in school: rough outline first, then paragraphs, then the exact sentences. And in each of those rounds it fills in every part at the same time, instead of left to right.

Why would anyone want this?

Because of that loop. An LLM has to make one full trip through the box for every token it writes, and trip 50 can't start until trip 49 is done. A 500-token answer is 500 trips, in a strict line. That's why answers stream onto your screen word by word, and it's a big part of why serving LLMs is expensive.

Think of a restaurant kitchen with one rule: the chef can only start the next dish when the last one is plated. You can buy a bigger stove, but you can't cook in parallel. Researchers have been trying to break that rule for years.

The main attempt before NCM is the diffusion language model. It starts with a page of blanks and fills them in over many cleaning passes, a bit like a photo slowly coming into focus. It works in parallel, but it still needs lots of passes (often 128 or more), and every pass processes the whole page at full length.

NCM is the Stanford team's answer to both problems: fewer passes, and the early passes are tiny.

How NCM works: follow one question

The easiest way to get this is to follow one question all the way through. Say you ask:

"What is the capital of France?"

NCM gets you to the answer in two stages. One happens long before you ask anything. The other happens the moment you hit enter.

Stage 1: learn a shorthand (done once, during training)

Before NCM can answer anything, it needs a compact way to write text down. So it first trains a small helper, the tokenizer, which has two halves:

• The encoder takes a sentence like "The capital of France is Paris." and squeezes it into levels of codes. Level 1 is a single code with the gist. Level 2 has a few more codes with a bit more detail. The last level has one code per word.
• The decoder does the reverse: give it those levels, and it rebuilds the sentence.

Training is simple. Squeeze a sentence, rebuild it, check it matches, nudge the dials. Repeat over a huge pile of text. Once it rebuilds text reliably, it's frozen, and the decoder is kept for later.

Think of it like learning shorthand before you take notes in a lecture. You practise once, then use it every time.

Stage 2: answer the question (every time you ask)

Now the main model, the Next-Concept Transformer, does the writing. It never writes words directly. It writes those shorthand levels, one level per round, and it looks at your question and every earlier round while it does:

1. Round 1: it writes the gist, one code that roughly means "the answer is a short fact about a city".
2. Round 2: with that gist in hand, it writes the next level, two codes, both at the same time: roughly "France's capital" and "is Paris".
3. Round 3: it writes the last level, one code per word, all six at the same time.
4. Decode: the frozen decoder from Stage 1 turns those codes into text, and you get "The capital of France is Paris.", the whole sentence in one go.

An LLM would need six trips through the box for those six words, each one waiting for the last. NCM needed three rounds, and the first two were tiny because they only had one or two codes to write.

Two honest notes on this example. The codes are really learned numbers, not readable phrases; I've written them as words so you can see what each level roughly carries. And real setups use 8 to 32 levels, not 3. The released checkpoints also don't include a general question-answering model, so treat this as a picture of the flow, not a demo you can run.

That's also what the "concept" in the name means: a code in one of those early, rough levels. It's a learned compression of text, not a concept in the human sense.

What the paper actually shows

Here's what the authors report, all against other small models of similar size, measured on one NVIDIA H100:

• 10 to 31 times faster than the strongest diffusion models, at similar quality. 21× on one general text benchmark (LM1B), 10× on another (OpenWebText), 31× on news summarization (XSum).
• About 4 to 5 times less training data than those diffusion models, and that count includes training the tokenizer.
• Sudoku is the eye-catching one. A Sudoku board has hard rules everywhere, and writing it left to right is a terrible fit, because a cell in the top row depends on cells in the bottom row.

Because NCM sketches the whole board roughly first, every later level can fix the whole board at once. That's a good sign for problems full of constraints. It is not proof that NCM is a better general reasoner.

Hold on before you call it an LLM killer

The paper itself never says that. It says something more modest, and the gap between the two matters:

• The models are small. Most experiments use around 100 million parameters, and the biggest test goes to 725 million. The chat models you use daily are hundreds of times larger. Nobody knows yet if NCM keeps its advantages at that size.
• It's compared mostly with diffusion models, not with GPT, Claude or Gemini class LLMs.
• Each released checkpoint does one task. There's a Sudoku model, a summarization model, a translation model. No general assistant.
• The structure is fixed. Every output goes through every level, even a short one, and each setup has a fixed output length (128 tokens for some tasks, 1,024 for others).
• There are rough edges. The authors mention occasional glitches where one coarse level hands off to the next.
• These are the authors' own numbers, and nobody has independently reproduced them yet.

One more name clash to avoid: Meta published "Large Concept Models" in 2024. Different idea. Meta's model treats each sentence as a concept and still predicts the next sentence one after another. NCM learns its own levels of codes and writes each level in parallel.

What changes for developers

This is the part I care about most. If NCM-style models do scale, what changes for the people training, shipping and running them? Quite a lot, actually.

The tokenizer becomes a real model. Today the tokenizer is a fixed lookup table (usually BPE) you download and forget. In NCM it's a trained neural network, a VQ-VAE (an encoder plus decoder that outputs codes from a fixed list). It has its own training run, its own quality metric (how well it rebuilds text) and its own bugs. A bad tokenizer caps how good the whole system can ever be.

Training is a two-step pipeline. Train the tokenizer, freeze it, then train the generator on its codes. Two jobs, two sets of checkpoints, and a hard dependency between them. The authors list joint training as future work, so this might collapse back into one step later.

The attention pattern is different. An LLM uses a causal mask: each token can only look at tokens to its left. NCM's generator uses a block mask by level: each level can look at all earlier levels, and everything within a level is produced together. For a prompt (like the text to summarize), it uses a prefix setup where the prompt is visible to everything.

The serving tricks don't carry over. Most of today's LLM serving stack is built around the one-token-at-a-time loop: the KV cache, continuous batching, speculative decoding. vLLM, TGI, llama.cpp and Ollama are all tuned for that loop. None of them run NCM today. What exists is the research repo: PyTorch code plus checkpoints and datasets on Hugging Face. Anyone serving this in production would be writing a lot of that layer from scratch.

Generation has new knobs. Instead of temperature and top-p on each token, you get things like how many candidate drafts to generate and pick from (the paper uses a method called MBR decoding for that) and a guidance scale for conditional tasks.

There's no streaming. The answer appears all at once, at the end. For a chat UI that's a real change: no words trickling in while you wait. In return, latency is far more predictable, because the number of passes is fixed by the level count, not by how long the answer turns out to be.

The ops and security view

Even at research stage, a few things jump out when I look at it as someone who runs this stuff:

• Two artifacts, one version. Every NCM checkpoint ships as a tokenizer.pt and a generator.pt. They only work as a pair. Pin and verify them together, the same way you'd never mix a container image with someone else's config.
• .pt files can run code. PyTorch checkpoints are often pickle files, and loading an untrusted pickle can execute whatever is inside. Verify checksums, load with weights_only=True where you can, and pull only from the official source.
• Moderation gets easier. Since nothing streams, you can check the full output before the user sees a single word. With streaming LLMs, that's harder to get right.
• Cost becomes more predictable. A fixed number of passes means latency and GPU time per request are steadier, which is nicer for capacity planning than "depends how chatty the model feels today".
• Logs need to be clear about the tokenizer version. If you ever debug a bad output, you'll need to know which tokenizer produced the codes, not just which generator.

Key takeaways

• An LLM writes one token at a time, in order. NCM writes a rough draft first, then adds detail, filling every part of a level at once.
• It works in two stages: a learned tokenizer that turns text into levels of codes, then a transformer that generates those levels one at a time.
• At small scale it's 10 to 31 times faster than strong diffusion models and needs 4 to 5 times less training data. It hasn't been shown at the size of today's chat models.
• For developers, the tokenizer becomes a model you train, training becomes a two-step pipeline, streaming goes away, and today's serving tools don't support it yet.

When to care: if you research models, or work on tasks with lots of rules where left-to-right writing struggles. When not to: if you're picking a model for a product this quarter. Keep using LLMs and watch whether NCM holds up at bigger sizes.

Try this today: the released checkpoints are small enough for a single GPU. Clone the repo, pull the checkpoints and data, and run the hard Sudoku eval yourself:

git clone https://github.com/HazyResearch/ncm.git && cd ncm
pip install -r requirements.txt
hf download hazyresearch/ncm-checkpoints --local-dir artifacts
hf download hazyresearch/ncm-datasets --repo-type dataset --local-dir data
python evaluate.py --config config/eval/sudoku-hard.yaml \
  --checkpoint-dir artifacts/sudoku-hard --data-dir data/sudoku-hard

Check the README first for the exact folder names in your download, since the checkpoints are grouped by task. It's a nice way to see a non-LLM text model run on your own hardware instead of reading about it.

Credit for all of this goes to the authors: Hermann Kumbong, Ishan S. Khare and the team at Hazy Research and Together AI. Read their blog post if you want the full picture. If something here didn't land, ping me on LinkedIn or X and I'll pick it up in a future post.