​

One of the things I wanted from NotFinalCode was the feeling that I actually owned the whole platform.

Not just the frontend.

Not just the database.

The little AI features too.

That is why I started experimenting with small local models instead of immediately calling a large hosted API.

Part of it was privacy. Part of it was cost.

Mostly, I wanted to be able to experiment badly without receiving an invoice for the privilege.

A 400M or 800M parameter model running on hardware I control feels very different from sending every little idea to a frontier model.

It feels like part of the application.

The first job: writing briefs

NotFinalCode contains hundreds of programming questions. The raw information is often structured enough for a computer, but not necessarily pleasant for a human to read.

One of my first experiments was using a small Qwen model to help generate coding briefs: historically, Qwen2.5-0.5B-Instruct Q4_K_M.

The model did not need to solve the problem or write production code. It needed to transform structured information into something closer to:

Given an array of numbers, return…

That is a much narrower job.

And narrow jobs are where small models become interesting.

A recorded batch covered 622 questions in roughly 45 minutes. The manifest identifies the service as local-brief-author and pins a model hash; that alias alone is not enough to identify whatever model might be running behind it today.

The timing is pipeline wall time, not a clean model benchmark. It includes orchestration and potentially waiting, retries and other overhead.

More importantly, the entire batch ended in needs-review.

Generation was not publication.

The model could produce material. A separate step still had to decide whether that material was acceptable.

Then I wanted the brief to understand itself

I tend to associate colors with values and concepts.

If a problem mentions numbers, target or capacity, I find it easier to understand when the text visually connects that word to its input definition.

So I started experimenting with semantic highlighting.

If the brief says:

Given an array of numbers…

then that phrase can be associated with the actual numbers input supplied to the function.

It sounds like a tiny detail.

It is.

I like tiny details.

That interest also led to requirement feedback for SQL: crossing out individual phrases when a query demonstrably satisfies them.

The important word is demonstrably. That feature now covers all 45 SQL questions using authored fixtures and deterministic checks. The model does not decide whether a query is correct.

Kev made the idea viable

I used jaredpalmer/kev-0.8b for semantic matching.

Instead of asking:

What does this entire programming problem mean?

I could ask something smaller:

Does this exact phrase refer to this exact input?

That distinction matters.

I built a development evaluation with 32 probes: 23 positive matching cases and nine negative controls.

All three settings use a margin of 0.15.

At threshold 0.70, the accepted matches had 100% precision and 82.61% recall on this development set.

That does not mean Kev is 100% accurate. It means the small set contained no observed false positives among the matches accepted at that threshold.

Lowering the threshold increased coverage, but introduced a wrong match.

That was exactly the trade-off I cared about.

I would rather miss a highlight than confidently highlight the wrong thing.

Then I tried the whole library

The historical Kev semantic pass selected 622 records.

The pass annotated 96.95% of selected records. That is processing coverage, not correctness.

Producing an annotation does not prove the annotation is right.

The recorded wall time was about 1 hour 24 minutes. A later replay finished in seconds, but reused recorded work. It would be misleading to present the replay as the model suddenly becoming hundreds of times faster.

Kev made the idea viable. In practice, I also found its workflow slow enough to start looking for something lighter.

That is a workflow observation, not a controlled hardware benchmark.

Enter Laya

I tried English Laya 421M Q8_0 as a smaller alternative.

The interesting part was not only its size. The task could be expressed as a constrained choice rather than free-form generation.

The helper uses native [CLS], [SEP] and [MASK] formatting. It bounds context and choices within a 512-token total budget. Individual occurrences can be classified independently, with at most four concurrent occurrence-level requests in the documented helper.

Not:

Please understand everything.

More like:

Pick one of these known possibilities, or abstain.

The model file is pinned by revision and SHA-256, so changing the file does not silently become “the same experiment.”

The Laya evaluation was less flattering

The next evaluation contained 25 labeled occurrences drawn from the formerly abstained drafts, plus negative controls.

It was a different development set from Kev's. These numbers cannot rank the models directly.

The pipeline also includes explicit-reference handling, so this is not a pure model-only score.

Again, the margin is 0.15.

At 0.70, precision was 84.62% and recall 57.89%. Raising the threshold to 0.80 reduced recall to 42.11% without eliminating the wrong matches.

That was useful information.

Not good information.

Useful information.

The smaller model was interesting enough to keep experimenting with. Its output was not reliable enough to promote automatically into published annotations.

So I kept the quality gate.

Small models need somewhere to fail

The easiest way to make a small model look impressive is never to let it say “I don't know.”

It is also a great way to create bad data.

There are several steps between receiving a model decision and putting that decision on the website.

Laya giving wrong matches was not a reason to hide the evaluation. It was a reason to keep review artifacts separate from approved content.

I would rather show no highlight than attach a confident color to the wrong concept.

Speed is more complicated than “model A is faster”

I moved away from Kev because the workflow felt too slow for how I wanted to use it.

The records do not give me a clean speed comparison.

One AgentHQ Kev run lasted roughly 20 minutes before interruption. One Laya manifest spans more than 13 hours. Both were interrupted and marked not materializable.

Those intervals include orchestration and may include waiting, retries, failures or idle time. They are not sustained inference timings.

The failed manifest counts do not prove every affected record reached the model and was rejected, either.

I do not have defensible numbers for tokens per second, time to first token, p50/p95 request latency, peak RAM, power consumption or matched-hardware Kev-versus-Laya throughput.

So I am not going to invent them from timestamps.

A proper comparison needs the same verified hardware, the same labeled inputs and the same prompt budgets. It also needs separate cold-load, warm-inference, SSH-overhead and cache measurements.

A smaller parameter count is a reason to investigate. It is not a benchmark result.

The constraint is the feature

The model does not need unrestricted project access. It does not need to chat with the user or decide whether code is correct.

It does not need to invent executable contracts.

For many parts of NotFinalCode, deterministic code is simply better.

The final function-based starters are derived from reviewed contracts. Named inputs, minimal signatures and sample decoding are things the application already knows. There is no reason to ask a model to improvise them.

SQL requirement feedback follows the same principle: run targeted fixtures, inspect actual output, leave insufficient evidence unverified.

The AI is useful where there is ambiguity.

Writing a clearer sentence.

Understanding whether a phrase refers to an input.

Helping structure content that will still be checked before publication.

Those are jobs where a little semantic flexibility is useful.

The models I actually used

Qwen2.5-0.5B-Instruct helped with historical brief authoring.

Kev 0.8B showed that contextual input-family matching could be useful.

English Laya 421M offered a smaller, constrained-choice approach—and another reminder that model output and approved content are different things.

There was also earlier authoring assistance through a host-configured smol model. The exact provider and revision were not recorded well enough for meaningful benchmark claims.

I do not want to fill gaps in the experiment's history with a more impressive story.

Owning the pipeline matters to me

The current policy is to run these authoring jobs on AgentHQ over SSH, not through a hosted provider and not through a silent workstation fallback.

If the intended runtime is unavailable, the operation fails.

I prefer that.

I know where the request went. I can pin a revision, hash the model file and change the pipeline.

The current transport is also much less magical than “AI streaming.”

The request travels through stdin and the result returns through stdout, but the adapter buffers the response. There is no token-by-token browser stream or WebSocket spectacle.

Just JSON going to another computer and eventually coming back.

That is enough.

I do not need the smartest model

This experiment changed how I think about local AI.

For NotFinalCode, I need models that are useful at one sufficiently constrained thing.

Qwen helped write briefs. Kev made contextual matching viable. Laya gave me a smaller approach worth exploring.

The evaluations also showed where the models stop being trustworthy.

That part matters just as much.

The goal is not to pretend a 421M model is secretly a frontier model. The goal is to design a task where 421M parameters might be enough.

Give it explicit choices.

Keep the context bounded.

Validate the output.

Let it abstain.

Never confuse processing coverage with accuracy.

Never confuse a confident probability with truth.

Small models are useful.

You just have to stop asking them to be big ones.