​

We Don't Need a Frontier Model for Every Decision

I already thought smaller models were going to matter.

Not because I expect a 500M parameter model to eventually become GPT-6 if we believe in it hard enough.

Almost the opposite.

For a while now, I have been increasingly unconvinced by the idea that progress in AI should be reduced to:

more parameters
→ more compute
→ more reasoning
→ better model

Frontier models are incredible.

They are also an incredibly expensive answer to a very large number of very small problems.

I think a much larger part of AI adoption will happen when inference becomes cheap enough that we stop thinking about individual calls.

Models running locally.

Models embedded inside ordinary applications.

Models sitting in background processes.

Models making thousands of tiny decisions without every one of those decisions requiring a trip to a frontier model running somewhere inside a very expensive datacenter.

The interesting future, to me, is not necessarily one model doing everything.

It is a lot of smaller models doing specific things extremely well.

Then Jev happened.

And for about a week, a surprisingly large part of the AI developer conversation became interested in a model that cannot write.

A model that deliberately does less

On September 15, 2026, TypeSafe AI introduced Jev, the first model in what the company calls its System One family.

Jev is not trying to be another chatbot.

It does not generate prose.

It does not write code.

It does not generate JSON and then hope your parser agrees with its interpretation of the schema.

You give it some state.

You define typed questions over that state.

It returns decisions and probabilities.

Conceptually:

state
+
typed questions
↓
choice / score / probability

Rather than:

state
+
prompt
↓
generate tokens
↓
extract the useful part
↓
validate it
↓
hope nothing weird happened

TypeSafe describes Jev as a model designed around structured decisions that software can consume directly, trained using what it calls Reinforcement Learning for Calibrated Decisions, or RLCD.

The important part for me was not really the acronym.

It was the decision to remove generation from the problem.

We use language models for a lot of things that do not require language

This becomes slightly strange once you notice it.

A lot of modern AI software looks approximately like this:

something happens
↓
send everything to an LLM
↓
ask what should happen
↓
generate structured output
↓
parse structured output
↓
do the thing

Need to classify a support request?

LLM.

Need to decide which agent gets a task?

LLM.

Need to determine whether a document is relevant?

LLM.

Need to score the output of another model?

LLM.

Need to decide whether that model should retry?

Another LLM.

We have become very good at making language models behave like software components.

Tool calling.

Structured outputs.

JSON schemas.

Constrained decoding.

Agent routers.

Evaluators.

All genuinely useful.

But underneath those abstractions, we are often still using a general-purpose generative model to answer what is fundamentally a bounded question.

Sometimes the required answer is literally:

YES
NO

Or:

A
B
C
D

Using a model capable of writing a novel to answer a multiple-choice question works.

That does not necessarily make it good architecture.

Jev did not change my mind about small models

It gave the idea a much more specific shape.

There is a difference between:

smaller general-purpose models

and:

models that are small because they deliberately do less.

That second category is much more interesting to me.

A 4B model trying to approximate every capability of a frontier model is useful.

A 400M model that owns one particular operation inside a larger system may be even more useful.

That suggests a different kind of AI architecture.

Diagram source: flowchart LR A[Input] --> B{What kind of problem is this?} B -->|Generation| C[Generative model] B -->|Deep reasoning| D[Reasoning model] B -->|Vision| E[Vision model] B -->|Bounded decision| F[Decision model] C --> G[Application] D --> G E --> G F --> G

The frontier model stops being the default answer to every semantic problem.

It becomes one component among several.

That is much closer to how I think mass AI deployment eventually becomes practical.

Then the idea spread very quickly

Jev itself is interesting.

The reaction to Jev may be even more interesting.

Within days, LangChain was experimenting with Jev inside agent workflows, specifically around things like routing and classification.

A week later, LangChain was already writing about using Jev with LangGraph in production-style agent architectures.

Vercel reported that Jev became the fastest-adopted model launch in AI Gateway history, reaching nearly 13% of paid teams within its first 24 hours there.

TypeSafe's own benchmarks claim very large latency and cost advantages over generative LLMs for the kinds of bounded tasks Jev targets.

Those are vendor benchmarks.

I would not treat them as universal laws.

But the developer reaction is interesting regardless.

The question people started asking was not really:

Is Jev better than GPT?

That comparison barely makes sense.

The more interesting question was:

How much of the current LLM stack actually needs a generative LLM?

That is a much better question.

And it immediately made me think about something completely different.

My first thought was data ingestion

I spend a lot of time thinking about data systems.

And data ingestion has a slightly annoying prerequisite that architecture diagrams tend to skip.

Before transforming data, you need to understand what actually arrived.

The clean version looks like this:

known source
↓
known schema
↓
known transformation
↓
destination

Reality is often closer to:

API
CSV
TSV
Excel
JSON
middleware export
customer dataset
mysterious file
        ↓
       ???
        ↓
processing

That ??? can become surprisingly complicated.

What kind of dataset is this?

Which known schema does it resemble?

Which parser should handle it?

Do these columns correspond to internal fields under different names?

Which transformation pipeline should run?

Is a required value genuinely missing?

Or did the customer call it something completely different?

Can this case continue automatically?

Should somebody inspect it?

These are semantic problems.

But most of them are not generation problems.

They are decision problems.

That made the Jev pattern immediately interesting from a data ingestion perspective.

Not as a replacement for the pipeline.

As a semantic layer before the deterministic pipeline.

Diagram source: flowchart TD A[Incoming data] --> B[Extract metadata + representative content] B --> C[Local decision model] C --> D{Known input?} D -->|High confidence| E[Select schema] E --> F[Select parser] F --> G[Select transformation path] G --> H[Deterministic validation] H --> I{Valid?} I -->|Yes| J[Destination] I -->|No| K[Human review] D -->|Ambiguous| K K --> L[Record resolution] L --> M[Improve rules / examples / model]

That is the architecture I actually wanted to explore.

The model does not transform the data.

It does not own the pipeline.

It does not get to invent a new schema because it felt creative that afternoon.

It makes a bounded semantic decision.

Then normal software takes over.

Why this could matter

Rules are still preferable whenever the answer can be determined reliably with rules.

I do not want a neural network deciding whether a CSV has six columns when Python is perfectly capable of counting them.

But deterministic systems become uncomfortable when equivalence becomes semantic.

Imagine receiving:

product_name

from one source.

productDescription

from another.

item_label

from another.

And:

designation

from somebody who apparently wanted to keep things interesting.

You can keep adding mapping rules.

That is often exactly what you should do.

But at some point the system needs to answer:

Does this field mean the same thing as something I already know?

That is where a cheap local semantic decision becomes interesting.

Especially if the system can say:

Known mapping: 0.98

and continue automatically,

while:

Known mapping: 0.54

goes somewhere a human can inspect.

That is a much more constrained use of machine learning than giving an agent complete control of the ingestion process.

Which is precisely why I like it.

But Jev was not the experiment I wanted

There was one obvious problem.

This thread is called Experiments in Local Inference.

I wanted something I could run myself.

Download.

Inspect.

Break.

Benchmark badly.

Fix.

Benchmark slightly less badly.

And more importantly, I wanted to understand whether the interesting part of Jev was Jev itself, or the architectural pattern it represented.

It did not take very long for open implementations to appear.

One of them was Kev.

Enter Kev

Kev describes itself as a family of Jev-like decision models built on Qwen.

The version I ended up testing was Kev-4B.

Its model card describes Qwen3.5-4B-Base with a LoRA adapter and a pointer head.

Like Jev, the important part is what it does not do.

No text generation in the decision path.

A document or state goes in.

Typed questions go in.

A probability distribution over the possible answers comes out.

That was much closer to something I could actually experiment with locally.

And then I found Laya.

And then Laya made the experiment more interesting

Laya approaches the same general idea from a very different size class.

The English Laya checkpoint I tested uses a ModernBERT-large backbone at around 421 million parameters.

It is non-autoregressive.

It accepts state plus typed questions and returns decisions rather than generated text.

So now I had two local models representing two fairly different approaches:

Model	Approximate scale	Approach
Laya	421M	Encoder-based, non-autoregressive decision model
Kev-4B	4B base	Qwen-based decision model with adapter + pointer head

And my original question:

Could a small local decision model help classify incoming data?

temporarily became:

Are these things actually good at classification?

That is how an ingestion experiment became a benchmark.

First sanity check: AG News

I started small.

Very small.

I used AG News as a four-class smoke test:

• World
• Sports
• Business
• Sci/Tech

Three examples per class.

Twelve examples total.

Model	Correct	Accuracy	Runtime
Laya	11 / 12	91.67%	6.20 s
Kev-4B	12 / 12	100.00%	67.15 s

This looked good.

It also meant almost nothing statistically.

At twelve examples, one wrong answer changes accuracy by more than eight percentage points.

There is another reason not to overthink this result: Kev's current model documentation lists AG News among its training datasets.

So I consider this a harness validation, not evidence of generalization.

Both models received the same input.

Both understood the typed-choice setup.

Both returned something my evaluator could score.

Nothing caught fire.

Good enough.

Time for a larger test.

DBpedia 14

For the main benchmark, I used DBpedia 14.

The downloaded dataset contains:

560,000 training examples
70,000 test examples
14 entity classes

I did not run all 70,000 test rows.

Instead, I built a deterministic balanced fixture:

10 examples × 14 classes = 140 examples

The fourteen classes were:

1. Company
2. Educational institution
3. Artist
4. Athlete
5. Office holder
6. Means of transportation
7. Building
8. Natural place
9. Village
10. Animal
11. Plant
12. Album
13. Film
14. Written work

Both models received the same title and body text.

Both received the same class definitions.

Predictions were scored against the dataset label.

Because every class contained exactly ten examples, overall and macro accuracy are equivalent for this fixture.

The result

Model	Correct	Accuracy	Elapsed runtime
Laya	117 / 140	83.57%	65.13 s
Kev-4B	139 / 140	99.29%	1,190.77 s

Kev got 139 out of 140 examples right.

Laya got 117 out of 140.

The accuracy gap is fairly easy to understand.

The runtime gap requires a paragraph of disclaimers.

Which means it is time for charts.

DBpedia 14 accuracy

data view

On this fixture, Kev was simply much more accurate.

Observed DBpedia 14 elapsed runtime

data view

Laya finished in about:

65 seconds

Kev took:

1,190.77 seconds

or approximately:

19 minutes 51 seconds

Divide one by the other and you get:

1190.77 / 65.13 ≈ 18.28×

Which would make an excellent headline.

Unfortunately, I also designed the benchmark.

So I know why I should not use that number as a model-level speed claim.

The 18× difference is real, but the comparison is not fair

Laya processed the workload in batches of ten states.

Kev was queried through its local server one document at a time.

There is another asymmetry.

Laya's elapsed time included loading its cached checkpoint and running prediction.

Kev's measurement started with the server and model already running.

So the benchmark looked approximately like:

Diagram source: flowchart LR A[140 DBpedia examples] --> B[Shared state + class definitions] B --> C[Laya] B --> D[Kev-4B] C --> E[10-state batches] D --> F[Sequential local HTTP requests] E --> G[65.13 s] F --> H[1190.77 s]

This is not normalized throughput.

It is observed elapsed time under the way I actually used each model.

So I am comfortable saying:

Laya completed this workload much faster under my CPU harness.

I am not comfortable saying:

Laya is 18.28 times faster than Kev.

Those are different statements.

Local benchmarking remains a very effective way of generating extremely precise numbers followed by increasingly long explanations of why you should not take them too seriously.

Where things got more interesting

The aggregate accuracy only tells part of the story.

Here are the class-level results:

Class	Laya	Kev-4B
Company	10/10	10/10
Educational institution	10/10	10/10
Artist	5/10	10/10
Athlete	10/10	10/10
Office holder	9/10	9/10
Means of transportation	10/10	10/10
Building	7/10	10/10
Natural place	10/10	10/10
Village	10/10	10/10
Animal	2/10	10/10
Plant	10/10	10/10
Album	8/10	10/10
Film	10/10	10/10
Written work	6/10	10/10

And visually:

Laya accuracy by DBpedia 14 class

data view

This makes something fairly obvious.

Laya was not bad everywhere.

It was perfect on eight of the fourteen classes.

Its errors were heavily concentrated around particular distinctions.

And then there were the animals.

Apparently, animals are plants

Laya's largest observed confusion was:

Animal → Plant: 8

Eight of the ten animal examples became plants.

That accounted for more than a third of all of Laya's errors.

Other repeated confusions included:

Written work → Company: 3
Artist       → Company: 2
Album        → Film:    2

Kev made one mistake:

Office holder → Artist: 1

I would like to tell you that I focused on the animal/plant result because it revealed a fascinating semantic boundary in the representation space of a 421M parameter encoder.

Mostly, eight animals becoming plants is just very difficult to ignore.

But the pattern actually matters.

Because for the ingestion use case I care about, aggregate accuracy is not the only interesting metric.

Predictable failure can be useful.

A model does not necessarily need to answer everything

Imagine that the classes in this benchmark were not:

Animal
Plant
Film
Album

but:

Known customer schema
Legacy schema
Internal export
External API payload
Unknown

If a model is extremely reliable on four classes but consistently uncertain on the fifth, I do not necessarily need a larger model for every input.

I can build a system around that uncertainty.

Diagram source: flowchart TD A[Incoming data] --> B[Small local model] B --> C{Confidence / class} C -->|Known + high confidence| D[Automatic route] C -->|Known + uncertain| E[Second-stage model] C -->|Unknown| F[Human review] D --> G[Deterministic pipeline] E --> H{Resolved?} H -->|Yes| G H -->|No| F F --> I[Resolution recorded] I --> J[Improve routing layer]

This is where the benchmark starts becoming relevant to the original idea.

I do not necessarily need the first model to achieve frontier accuracy.

I need it to handle enough traffic correctly and cheaply that the expensive path becomes the exception.

That changes the economics quite a bit.

The small model does not replace the large model

This is probably the biggest distinction.

I am not arguing that small models replace frontier models.

I think that framing misses the interesting part.

The goal is not:

frontier LLM
↓
replace it with tiny model

It is:

100,000 decisions
↓
small specialized model handles 90,000
↓
larger model sees 9,000
↓
human sees 1,000

Those numbers are hypothetical.

But the architecture is the point.

A specialized model does not need to become the smartest model in the system.

It needs to reduce how often the smartest model has to be invoked.

That is a much easier target.

And potentially a much more consequential one.

The real benchmark I want to run is not DBpedia

DBpedia is useful because evaluation is objective.

A company is a company.

A plant is hopefully not an animal.

The dataset gives me clean labels and lets me compare models without inventing my own definition of success.

But this is not the workload I actually care about.

The real experiment is data ingestion.

Something closer to:

input:
customer_sales_september.xlsx

observed columns:
Customer ID
Date
SKU
Description
Qty
Total ex VAT
Country

questions:
- Which known dataset family does this resemble?
- Which internal schema is the closest match?
- Which parser should handle it?
- Does this require preprocessing?
- Is the mapping confident enough to continue automatically?

Then I want to make the problem progressively worse.

Rename columns.

Remove columns.

Add useless columns.

Mix languages.

Change date formats.

Change decimal separators.

Use poorly named Excel sheets.

Put metadata above the table.

Send something completely unknown.

Basically, reproduce what people already do to production ingestion systems for free.

That is the benchmark that would actually tell me whether this architecture is useful.

And then there is continuous improvement

There is another reason I like the ingestion use case.

Edge cases naturally produce supervision.

Imagine the router sees something unfamiliar.

confidence too low
↓
human reviews input
↓
human selects correct schema + transformation
↓
decision is recorded

That resolution is useful.

It can improve deterministic rules.

It can become another example.

It can help evaluate the classifier.

Eventually, it may become training data.

The system starts accumulating knowledge about the weird data people actually send it.

That is much more interesting to me than building a completely autonomous agent and hoping the prompt eventually becomes good enough.

What this benchmark does not prove

There are several tempting conclusions here that this experiment does not support.

It does not prove that:

• Kev is universally more accurate than Laya.
• Laya is intrinsically 18.28× faster.
• either model would produce these scores across all 70,000 DBpedia test examples.
• these results transfer directly to data ingestion.
• CPU performance predicts GPU performance.
• either model has better production economics.
• either benchmark is free from training-data overlap.
• confidence calibration is equivalent between the two models.

The last point is especially important.

Laya emitted a runtime warning around temperature values for the checkpoint I tested, so I deliberately left confidence calibration out of this benchmark.

That is unfortunate because confidence is probably one of the most important properties for the architecture I actually want to build.

If the system is going to say:

continue automatically

versus:

ask a human

then knowing whether 0.95 actually means something becomes extremely important.

That needs its own experiment.

The dataset is also much larger than what I tested

DBpedia 14 contains:

70,000 test examples

I ran:

140

That makes this a controlled experiment.

It does not make it a leaderboard.

At Kev's observed sequential rate of roughly:

8.50 seconds / request

running the entire test set under the same harness would also be a fairly questionable use of my weekend.

The next sensible step is not immediately throwing another 69,860 examples at it.

It is fixing the harness.

Normalize batching.

Separate cold start from warm inference.

Measure per-request latency.

Test concurrency.

Then scale the fixture.

This is supposed to be an experiment.

I occasionally have to remind myself that brute force is not the same thing as methodology.

So what did I actually learn?

The most obvious answer is:

Kev-4B was dramatically more accurate on this DBpedia fixture.

99.29% versus 83.57% is not subtle.

At the same time:

Laya's runtime behaviour was much more interesting for the kind of high-volume local routing I have in mind.

A 421M parameter decision model that can process batches locally on CPU is a very different architectural object from a general-purpose frontier model.

That does not make one of them the winner.

It makes the next question more interesting.

Can I build a cascade?

Diagram source: flowchart LR A[Incoming item] --> B[Laya / small router] B --> C{Confident?} C -->|Yes| D[Route immediately] C -->|No| E[Kev / stronger decision model] E --> F{Confident?} F -->|Yes| D F -->|No| G[Frontier model or human] D --> H[Deterministic processing]

Small, fast model first.

Stronger local model second.

Frontier model or human only when necessary.

Now the question is no longer:

Which model is best?

It becomes:

How little intelligence can I spend on each decision while still getting the decision right?

That is a much more useful optimization problem.

This is probably becoming a project

I originally started looking at Jev because I wanted to understand what decision models could mean for data ingestion.

Then I found Kev.

Then Laya.

Then I benchmarked them.

Now I have a routing architecture, a calibration problem, a dataset-design problem, a potential human-feedback loop, and several new benchmarks I want to run.

This has crossed the traditional NotFinalDev threshold where an article accidentally becomes software.

So I am going to treat the ingestion side as a separate project.

This article remains part of Experiments in Local Inference because the benchmark itself is exactly that.

The project will answer the question that caused the benchmark in the first place:

Can small local decision models become a useful semantic routing layer for messy real-world data ingestion?

I do not know yet.

Which is considerably more interesting than pretending I do.

---

Reproducibility notes

The DBpedia fixture used:

• the official fancyzhx/dbpedia_14 test split;
• 10 deterministic examples from each of 14 classes;
• 140 total examples;
• identical state fields and human-written class criteria for both models.

The downloaded DBpedia dataset card declares CC-BY-SA-3.0 and also references the GNU Free Documentation License. Applicable attribution and share-alike requirements should therefore be preserved when redistributing the underlying dataset.

Models

• Jev — TypeSafe AI
• Laya — Hugging Face
• Kev — GitHub
• Kev-4B — Hugging Face

Further reading

• TypeSafe — Introducing System One Models & Jev
• LangChain — Building a Harness with Jev
• LangChain — Building Production Agents with Jev and LangGraph
• Vercel — Jev is the fastest-adopted model in AI Gateway history
• AG News — Hugging Face
• DBpedia 14 — Hugging Face
• Character-level Convolutional Networks for Text Classification — Zhang, Zhao, and LeCun