I spent years on platform teams, where the rule was simple: if a check fails, the build fails. You don't ship around it, you don't skip the flaky test, you go and find out why.
When I fine-tuned a small model to write bhai-lang, the toy language with Hindi-slang keywords (hi bhai, bol bhai, jab tak bhai), I ran the dataset the same way. That habit did more for the result than anything in the training config.
The short version, if you haven't seen it: Qwen2.5-Coder-1.5B, trained on my M1 Pro (16 GB) in about 40 minutes with Troy, scored 100/100 on a held-out eval. The full story is here. This post is about how the data got made.
The problem with "generate and filter"
The usual way to make synthetic training data is to ask a big model for examples, then filter out the bad ones. It works, but the filter quietly hides your bugs. If 5% of samples fail, you drop them and move on, and you never learn whether those 5% were noise or a sign that your generator is broken in some systematic way.
I didn't want a filter. I wanted a test suite.
Templates write the programs, an interpreter checks them
Instead of asking a model to write bhai-lang, I wrote 27 parameterised task templates: printing, arithmetic, conditionals, loops, break/continue, strings, booleans, up through GCD, primality, Fibonacci and digit manipulation. Each template takes random numbers or strings and produces three things:
- an instruction, in English or Hinglish
- a bhai-lang program
- the expected output, computed separately in Python
Then every program runs through a small bhai-lang interpreter I wrote in Python, and its stdout is compared to the expected output.
This is where the CI habit comes in. A mismatch fails the build. It doesn't get dropped. Since templates write the programs, not a model, a mismatch can only mean one of two things: a bug in the template, or a bug in the interpreter. Both are things I want to know about before training, not after.
The numbers from the shipped run
With seed 42, the pipeline made:
- 2,016 attempts
- 1,500 kept (1,400 for training, 100 held out)
- 516 rejected, all as duplicate instructions
- 0 execution mismatches
That zero isn't luck. By the shipped run, every mismatch the gate had ever raised had been fixed at the source. It's the same reason a green CI run means something: a failure was never allowed to pass quietly.
The run is also deterministic. Same seed, same dataset, every time. If someone asks "what did you train on?", the answer is a commit and a seed, not "roughly these files".
The eval runs the code too
The eval follows the same rule. I don't string-match the model's output against a reference, because two correct programs can look completely different. Instead:
- take the model's reply
- pull the program out with a regex (
hi bhai[\s\S]*?bye bhai) - run it in the interpreter
- compare stdout with the reference
100/100 on that eval means 100 programs that actually ran and printed the right answer.
Where it broke
A CI pipeline is only honest if it reports the failures, so here are mine.
Novel tasks: 3 out of 5. I tested five task types the model had never seen. It passed three, mostly by combining idioms it knew: step-2 loops for summing odd numbers, and digit counting. It failed two:
- an off-by-one error in a loop boundary
- an invented construct,
nahi to {, for a bare else, which bhai-lang doesn't have
So it learned the language well within its training families and can stretch a bit, but it isn't a general bhai-lang programmer. A 1.5B model trained on 27 templates shouldn't be expected to be one.
Fusing into 4-bit weights wiped out the adapter. Merging the LoRA into the quantised base lost the adapter's changes to quantisation noise. The fix was to fuse and dequantise to fp16 before publishing (troy export --dequantize).
A system prompt made it worse. I trained on bare user turns. Adding a system prompt at inference hurt format adherence. Inference has to look like training.
It doesn't always stop. Generation sometimes runs past bye bhai, which is another reason the eval extracts the program with a regex and throws away the rest.
What I'd tell someone starting out
- If you can execute it, execute it. Code, SQL, config, maths: anything with a checker can have verified training data.
- Make bad data fail the build. Dropping it hides your bugs.
- Pin the seed. Your dataset should be as reproducible as your binary.
- Report your failures. "3 out of 5 on novel tasks, and here's why" is more useful than another 100%.
None of these are ML ideas. They're the things platform engineering already taught me, applied to training data.
Everything is open: the pipeline, the model, and Troy. On an Apple Silicon Mac, you can reproduce the whole thing with python3 gen_dataset.py && troy train.