Vibe-training a small open-weight model to write regular expressions
posted on 17/11/2026An animated language model predicting the next tokenTokenThe piece of text a model works with: a word, part of a word or a symbol. Qwen3 knows about 150,000 of them., generated using the /colorfigures skill.
TL;DR:
- I post-trainedPost-trainingAny training done after pre-training to steer the model: a format, a tone, a task, a domain. Qwen3 (0.6B then 1.7B) to generate JS-flavored regular expressions from English descriptions
- Resulting model runs on a playground locally in-browser (1.43GB model download): https://regex-vibes.netlify.app
- Results are mixed: ~60% of good results (=regex passing 100% of tests) on simple prompts, ~20% on medium ones, ~10% on difficult ones. Still, failing regexes pass ~70% of their tests.
- I made this experiment for learning purposes. I wouldn't recommend using the resulting model for production needs.
Like most software engineers nowadays, I use large language models on a daily basis. While I have a Claude subscription at work (and love it!), I keep an eye on open-weightOpen-weightA model whose weights are published, under a license that allows reusing and modifying them. models that seem to become more and more viable by the week. Moreover, specialized small language models seem to give pretty good results when properly trained.
So I decided to try by myself and I post-trained a small-sized LLMLLM (Large Language Model)A model trained on huge amounts of text that predicts the next piece of text. Chat, code, summaries: everything starts from there. (actually had Opus 5.5 drive me, hence the "vibe-training") to be good at generating (locally in-browser!) JS-flavored regular expressions from an English description of what to match. The resulting model runs in this playground.
I learned a lot doing it: here's the process I followed.
Why regular expressions?
The goal is pretty simple: give a small language model a description in English of what to match, and get a JS regular expression in return.
Plain English
"french phone numbers like 06 12 34 56 78 or 0612345678"
JS regular expression
/^0[1-9](?: ?\d{2}){4}$/
It turns out regular expressions are a good candidate for a first post-training experiment, for several reasons:
- narrow scope: the model only has one thing to be good at
- short output: a single line, no explanation needed
- verifiable output: we can run the generated regex on a list of strings and know for sure if it's right or not, no need for a human (or another LLM) to judge it
- useful: every developer uses them (and nobody likes writing them)
The targeted flavor is ES2018 as executed by Node 22: character classes, quantifiers, lookahead and lookbehind, named groups, \p{β¦}, etc. but not the more recent v flag.
And the starting point was not great: before any trainingTrainingShowing examples to the model, measuring its error, and adjusting its weights so it gets a bit less wrong. In a loop., the candidate models were giving less than 8% of fully correct regular expressions. Not great indeed.
What is post-training?
Large language models are basically machine learning models statistically predicting pieces of text (= tokens) coming after given pieces of text.
The weightsWeights (parameters)The numbers stored in the model, from millions to billions. Training a model means adjusting these numbers., as shown in the schema, are the numerical parameters built into the model (millions to trillions of them) adjusted during the training of the base modelBase model (foundation)A model fresh out of pre-training: it completes text, but can't answer a question or follow an instruction. to produce the right output to a given input.
Post-training is the process of nudging these weights after the initial training of a model to steer it into giving answers in a certain style, tone, format, or on a very specific domain.
Choosing what model to post-train
There are several factors to take into account when it comes to picking a language model for a given task: the "difficulty" of the task, the performance constraints, the budget, the environment it will run in, etc.
For my use case, I identified the following constraints for my model:
The model must be open-weight
It basically means that the weights we talked about in the previous section are public, and that the license of the model allows their reuse.
Here are some useful links to look for these:
- huggingface.co/models: the main hub for open-weight everything
- Ollama Library: a library of popular open-weight models
- xigh/open-weight-models (GitHub repo.): "a curated list of open-weight AI models with commercially exploitable licenses, verified benchmarks, and no geographic restrictions."
It should have an Instruct / it variant available
When talking about language models, a distinction must be made between two things:
- base models (or foundation models) that are only trained to guess the next token of a given text
- instruct modelsInstruct model (it, chat)A base model already post-trained to follow instructions and answer like an assistant. (or chat models) that are base models already post-trained to answer questions, be helpful to end users, and follow instructions
Starting from a model that already has instruct capabilities is valuable in our case as we won't need to train the model for it ourselves.
It should be suitable to be trained on a MacBook Pro with 24GB RAM
Training a machine learning model needs RAM! The reason is that there are several things that need to fit in memory at the same time: the weights of the model to train, training activations, caches, but also other things that are closely tied to the post-training method used:
- Full Fine-TuningFine-tuningContinuing the training of an existing model on your own data. "Full fine-tuning" updates every weight.: update all of the model's original weights during training. This leads to the best results, but at the cost of really high resource needs (rule of thumb is 14 to 16 GB of RAM per billion parameters).
- Low-Rank Adaptation (LoRA)LoRA (Low-Rank Adaptation)The original weights are frozen and only two small matrices added next to them are trained. 5 to 8 times less memory, ~95% of the effectiveness.: freeze all of the model's original weights and instead train two smaller low-rank matrices of weights (called an adapterAdapterThe output of a LoRA training: a small file (a few dozen MB) applied on top of the original model, or merged into it.). This method is much more memory-efficient (rule of thumb is 3 to 4 GB of RAM per billion parameters, see hf-accelerate/model-memory-usage) and leads to results almost as good as full fine-tuning (see Thinking Machine's "LoRA Without Regret" article).
We also need to keep in mind that the model needs to be compatible with MLXMLXApple's machine learning framework, a PyTorch equivalent optimized for Apple Silicon chips., Apple's machine-learning framework (equivalent of PyTorch or JAX, optimized for Apple Silicon chips).
Final model should be usable in-browser
As we want the resulting model to run on a playground in-browser, we need to validate its compatibility with Transformers.jsTransformers.jsHugging Face's JS library for running models in the browser, on WebGPU or WebAssembly. which is the state-of-the-art solution to run machine learning models on web clients.
There's just a quick thing to validate when picking the model: onnx-community/<model> should exist on the HuggingFace hub. If an ONNXONNXA standard model export format, readable by many runtimes, including Transformers.js in the browser. version of the base model we want to train exists (which is the format used by Transformers.js), it means that our post-trained model will also be exportable to ONNX in the end, and thus usable in-browser.
The end model should also be small enough to be downloaded and stored by end-users. My hard limit on this was ~1.5GB (which is already a lot!).
So, what do we pick?
At this point, we still have several candidates matching our needs:
Qwen3-0.6B/1.7B,Qwen3.5-0.8B,Qwen2.5-Coder-0.5B: Alibaba's Qwen family, the most popular small open-weight models (theCodervariant being specialized in code)Gemma3-1b,Gemma3-270m: Google's family of small open-weight modelsLlama-3.2-1B: Meta's smallest text modelGranite-4.0-350M: IBM's tiny model- ... and more
I first measured all of these candidates raw (with a few examples of expected answers in the prompt, aka few-shotFew-shotPutting a few solved examples in the prompt, without training anything. Zero-shot: no examples at all. prompting) on two sets: the held-out set (more on it later) and a set of 400 real-world regex needs gathered from regex101. They all scored between ~1% and 8%: no obvious winner...
| Model | Held-out (few-shot) | regex101 (few-shot) |
|---|---|---|
| Gemma3-1B | 1.4% | 3.3% |
| Granite-4.0-350M | 2.9% | 3.5% |
| Llama-3.2-1B | 4.8% | 5.5% |
| Qwen2.5-Coder-0.5B | 5.9% | 5.3% |
| Qwen3-0.6B | 2.3% | 2.8% |
| Qwen3.5-0.8B | 4.8% | 2.8% |
| Qwen3-1.7B | 5.0% | 3.5% |
Fully correct regular expressions before any training, on the held-out set and on 400 real-world needs (regex101).
After some online research, it seemed like Qwen3-0.6B was one of the most common bases for small-scale post-training: it's strong for its small size, it supports thinking and non-thinking modes, and it fits easily on an average GPU, so it's cheap to iterate on. So I followed the advice and went with Qwen3-0.6B to iterate (~1h per training), and Qwen3-1.7B for the final version (~2h15). I also disabled Qwen3's thinking mode: for a one-line answer, latency matters more than reasoning.
Building the training dataset
Now that we have a model to train, we need data to train it on. It is one of the most important parts of post-training: a good training dataset is what will give you the best results. Let's see what "good" means here:
General guidelines
Quality is worth more than quantity
For a narrow post-training task done with LoRA on a small (~1B parameters) model, hundreds to a few thousands of training examples are sufficient. A few references going in that direction:
- OpenAI's supervised fine-tuning guide reports improvements from fine-tuning on 50 to 100 examples, and recommends starting with 50 and evaluating before adding more.
- AlpaGasus (Chen et al., ICLR 2024): keeping only the 9k best-rated examples of the 52k Alpaca dataset gives a better model than training on all of it.
- My own runs: with only 2k examples,
Qwen3-0.6Balready reached 56.6% on the held-out, against 71.7% with the full 9k. Most of the gain comes from the first thousands of examples, even if more data still helps.
Examples must be verified
As we'll explain further, most post-training techniques rely on imitation, and as models imitate mistakes just as faithfully as good answers, a wrong example is worse than one less example. This is why we want to make sure that our "input -> output" pairs are valid. How to validate is really dependent on the problem we're trying to tackle, but in our case of generating regular expressions, this is pretty straightforward:
For each example, we just keep two lists of strings next to the description and the expected regex: should_match and should_not_match
{
"description":
"french phone numbers like 06 12 34 56 78 or 0612345678",
"regex": "/^0[1-9](?: ?\\d{2}){4}$/",
"should_match": [
"06 12 34 56 78",
"0612345678",
"01 23 45 67 89"
],
"should_not_match": [
"6 12 34 56 78",
"00 12 34 56 78",
"06-12-34-56-78",
"061234567"
]
}
should_match is a list of strings the regex must match and should_not_match is a list of strings the regex must reject. Both are required for the regex to be considered valid: as /.*/ matches every should_match string possibly existing, should_not_match ensures that generated regexes are not too permissive.
Then, we can just validate each example as such (ofc oversimplified code):
// 1. does the regex compile properly?
const re = new RegExp(source, flags);
const ok =
// 2. does it match what it should?
should_match.every(s => re.test(s)) &&
// 3. does it not match what it shouldn't?
should_not_match.every(s => !re.test(s));
If one of the three tests fails, the example is not kept. Only examples that pass all the tests are kept in the dataset.
Diversity of examples taxonomy
Dataset examples must look like real-world usage. To drive the generation of examples, we must define a taxonomy of needs that defines the diversity of the cases we want. The end model will be good at the type of examples it saw during training, so the closer our training dataset resembles what the model will get fed in "production", the better it will perform.
This taxonomy must make explicit the following things about what to generate:
- the regex concepts to cover: character classes, quantifiers, anchors, groups, alternation, lookahead/lookbehind, backreferences, flags (
i,g,m,u,s), etc. - the categories of examples: numbers, money, datetimes, web, phone numbers, logs, text, structured data...
- the difficulties of the examples and their relative proportion in the dataset: simple (30%), medium (50%), hard (20%)
- the "tones" of descriptions:
- Well-written with an example:
"Accept a positive whole number (ex: 4567): One or more digits..." - Relaxed style + spelling mistake:
"six pairs of hexadecimal digits sparated by hyphens" - Terse prompt + mistakes:
"css length w uint px em rem % vh vw no spaces decimal ok" - Just one word:
"email" - ...
- Well-written with an example:
Keep one consistent output format
To be able to properly validate the model's answers, we have to make sure that its output follows a format that is constant and unique: just a JS regular expression literal (/β¦/flags), nothing else. No explanation, no "Here is the regex:", no ```js block around it.
The parsing of the answers is strict on purpose: a chatty answer is considered malformed, because it can't be compiled as is in Node and therefore can't be validated. To achieve that, the system prompt ("write an ES2018 JS regex, only answer with a /β¦/flags literal") is defined once and shared everywhere: when generating the dataset, when training, when evaluating, and in the playground.
It might seem like a detail, but it's actually the first thing raw models fail at: before any training, Qwen3-1.7B answered in a wrong format 60% of the time (pattern without delimiters, /β¦/.test(str), text around the regex...). This is also the first thing post-training fixes: after SFT, malformed answers drop to 1.8% on the held-out (7.5% on regex101).
Generating the dataset with a teacher model
So, where does the data come from? There's no public dataset of "description β regex + tests" that matches our needs, so we have to generate it ourselves. That's where a bigger model comes in: a teacherTeacher modelThe big model that writes the training examples. Here DeepSeek V4.1 Flash, through OpenRouter. model writes the exercises, and Node sorts them out.
- 1. Teacher: given the taxonomy, DeepSeek V4.1 Flash writes the descriptions, the regexes and the tests.
- 2. Guardrails: everything is run in Node, tested, deduplicated...
- 3. Dataset: only what passes every check is kept.
The guardrails are where most of the work is. A few things that turned out to matter:
- tests are written in a separate call, without seeing the regex: otherwise they inherit its bugs
- at least 5 positive and 5 negative tests, including at least 3 "near-miss" negatives (one char too many, wrong anchoring, wrong case, out of range value...)
- trivial or mutated regexes must fail:
/.*/, the reference regex without its anchors, with loosened quantifiers or widened character classes must fail at least one test, otherwise the tests are too weak and the example is rejected
Here are the stats of generating the dataset with DeepSeek V4.1 Flash:
Generated needs
27,460
Kept examples (36%)
9,819
Total cost
$6
Only 36% of the generated needs were kept (guardrails really matter!), and this depends a lot on the category: 9% for logs (long lines with nested rules), 61% for identifiers.
Cleaning and splitting
Once we have our examples, we need to split them, and that's an important step: we never evaluate a model on what it has been trained on.
- train (9348 examples), the exercises: what the model studies.
- held-outHeld-out setA part of the dataset set aside and never shown during training. Same mold as the training set: it measures what the model learned. (471 examples), the mock exam: never studied by the model, but generated from the same mold as the train.
- benchmarksBenchmarkA frozen evaluation set, written by a different source than the training data. It's the exam that counts. (3 of them), the real exam: written by someone other than the teacher. In my case: one written by Claude, one by GPT, and 400 real-world needs from regex101 (based on this JSON dataset, descriptions reworked and tests written by GPT 6.1 Sol).
Why bother having benchmarks and a held-out?
Because the held-out comes from the same teacher as the train: same style, same wording, same way of understanding a need. Evaluating only on it would flatter the model, and we'll see later that it actually does, a lot.
Choosing a post-training method
There are a lot of post-training methods, but they can be grouped in three families, depending on how the model learns.
- Learning by imitation: Continued pretrainingContinued pre-trainingResuming the pre-training objective (predicting the next token) on raw, unlabeled text from a specific domain (code, medicine, lawβ¦), to teach the model its vocabulary and knowledge. No question/answer pairs: just text., SFT (Supervised Fine-Tuning)SFT (Supervised Fine-Tuning)Learning by imitation: the model is shown "input β correct answer" pairs and learns to reproduce the answer., RFT (Rejection sampling Fine-Tuning)RFT (Rejection sampling Fine-Tuning)The model generates several answers, the correct ones are kept, and an SFT is run on them., DistillationDistillationTraining a small model (the student) on the answers of a big model (the teacher)., ...
- Learning by preferences: DPO (Direct Preference Optimization)DPO (Direct Preference Optimization)Learning from preferences: for the same input, one preferred answer and one rejected. The model learns to lean towards the first., KTOKTOA DPO variant: each answer is simply labeled good or bad, no pairs needed., ...
- Learning by trying and being judged: Reinforcement learning (RLHF or RLAIF)RLHF (Reinforcement Learning from Human Feedback)The model generates, a judge (trained on human ratings) scores, and the model reinforces what scores well. RLAIF: the judge is an AI., GRPOGRPOReinforcement without a learned judge: several answers per question, scored by a verifiable rule, compared to the group average., ...
My need was a precise task with a fixed output format, so I went with SFTSFT (Supervised Fine-Tuning)Learning by imitation: the model is shown "input β correct answer" pairs and learns to reproduce the answer.. Strictly speaking, since every answer in the training dataset was written by DeepSeek, it's actually also a form of distillationDistillationTraining a small model (the student) on the answers of a big model (the teacher)..
In practice, SFT almost always comes first when doing post-training, as it puts the model in the right format and on the right domain before any other method can be applied.
Training with SFT and LoRA
How SFT works
SFT is the same loop as pre-training (predict β measure β correct β repeat), but on our data. For each example, the prompt and the beginning of the right answer are given to the model, and it must guess the next token:
Prompt (context, not graded)
"french phone numbers like 06 12 34 56 78 or 0612345678"
Beginning of the right answer
/^0β―
Model's guesses for β―
\d 55%
6 20%
[1-9] 12% β
β¦
- 1. Predict: the model gives
\dat 55%,6at 20%,[1-9]at 12%... - 2. Get surprised: the right answer was
[1-9], at 12% only. The lossLossHow much probability the model gave to the correct token: 90% means a near-zero loss, 1% a big one. It's the model's "surprise" at the right answer, and training drives it down. is high (2.1). - 3. Correct: the gradientGradientFor each weight, the direction and size of the change that lowers the loss. "Gradient descent" follows that direction. adjusts the weights a little bit:
[1-9]β,\dβ based on the loss. - 4. Repeat: next token, next example. Here, 9348 examples over 2 epochsEpochOne full pass over the whole training dataset. Two epochs: every example is seen twice..
Only the answer is graded, the prompt is just context. We don't want to teach the model to rewrite the user's description.
LoRA: train a small patch rather than the whole model
As explained earlier, LoRA freezes the original weights W of the model, and trains two small matrices B and A next to them: the output of a layer becomes W * x + B * A * x.
In our case, with a rankRank (LoRA)The inner size of the two small LoRA matrices: B is (d Γ r) and A is (r Γ d), so r sets how many weights are trained. A higher rank lets the adapter learn more, but costs more memory. Common values: 8 to 64. of 16 on the attention and MLP layers of every block (couldn't have guessed it without Opus), only ~1% of the base model weights are trained, and the result is an adapterAdapterThe output of a LoRA training: a small file (a few dozen MB) applied on top of the original model, or merged into it. of a few dozen MB.
From the Mac to the browser
Once the adapter is trained, a few steps remain to get it running in a browser:
- 1. Fuse: the adapter is merged into the base model weights (made using MLX)
- 2. Export: to ONNX, quantizedQuantizationStoring weights on fewer bits (16 β 4): a model 3 to 4 times lighter, for a small loss in quality. in 4 bits.
- 3. Serve: with Transformers.js, on WebGPUWebGPUThe browser API that gives access to the machine's GPU, for rendering or compute..
For the final model: 2h15 of training, 7.1GB of memory at peak, and 1.43GB once exported. Yes, that's right at the edge of the ~1.5GB limit I gave myself earlier, but I still went with Qwen3-1.7B because it was clearly better in quality, and Qwen3.5-0.8B, which would have fit and was almost as good for almost half the size, was way too slow in-browser (>5s latency, for model architecture reasons).
Measuring the results
How to evaluate
The evaluation follows the same logic as the dataset validation: for each case, the prompt is given to the model, its answer is parsed as a /β¦/flags literal, and the regex is run in Node on hidden tests.
The main metric is full success: all the tests must pass for a regex to be considered valid. Because in real life, a regex that is 90% right is a wrong regex. We also keep an eye on the rate of passed tests and on malformed answers, to understand how the model fails.
The numbers
On the held-out:
Qwen3-1.7B raw (few-shot)
5.0%
0.6B after SFT
72%
1.7B after SFT
77%
From 5.0% to 77% of fully correct regular expressions, that's huge! But the held-out is the mock exam... On requests the model has never seen, it gets 6 out of 10 simple requests right, and only 1 out of 10 harder ones...
Still, when it fails, ~70% of the tests pass anyway: the regexes are almost right. Which is actually a problem: an almost right regex is a silent error.
Outside the teacher's mold
Here's what happens when we compare the held-out to the benchmarks, by difficulty:
| SFT Qwen3-1.7B | Held-out | Claude benchmark | regex101 benchmark |
|---|---|---|---|
| Simple | 88% | 73% | 46% |
| Medium | 77% | 38% | 14% |
| Hard | 57% | 11% | 10% |
The held-out measures what the model learned from the teacher, not how good it is in the real world. Even on simple requests, we lose ~15% of good results as soon as we leave DeepSeek's distribution...
For reference, Claude Opus 5.5 (reasoning, medium effort) scores 100% on every one of these benchmarks: the tests are not ambiguous, the gap really comes from the model.
Retrospective
What worked
- SFT: from 5.0% to 77% on the held-out
- tests as ground truth: no need for an external judge (human or other LLM), Node is enough
- the cost: $6 of data, a Mac, no rented GPU
- scaling: going from 0.6 to 1.7 billion parameters gave +5 points
- in-browser execution: private, free for every request, <500ms latency
What didn't
- real-world requests: ~23% on average on the regex101 needs
- model too small: a model like Qwen3-1.7B might be too small for production use in this case
- silent errors: almost right regexes, ~70% of tests passing
- GRPOGRPOReinforcement without a learned judge: several answers per question, scored by a verifiable rule, compared to the group average. on top of SFT: I tried reinforcement learning on
Qwen3-0.6B, rewarding each answer by the share of tests it passes. It made the model more reliable (malformed answers down from 7.1% to 4.8% on the Claude benchmark, +3 to 4 points of passed tests) but barely more correct: +3.8% of full success, not statistically significant, and nothing on the held-out.
Conclusion
Yes, I know, the final numbers are not that good. But to be honest I did not really expect to get 100% success. My goal was mainly to learn the basics of post-training, and to make sure of something: a small specialized model can give decent results on a narrow task. And as shown by the results, the format and specificities of a teacher model can be learned very well from a few thousand examples.
You can try the result by yourself on the playground (Chromium with WebGPU only for now).
Hope you learned something!
Want to reproduce it?
- Playground: https://regex-vibes.netlify.app
- Code, dataset scripts and training configs: mdubourg001/regex-generator-local-post-training (GitHub)
- Final model (ONNX, 4 bits): maximedubourg/regex-sft-qwen3-1.7b-onnx (Hugging Face)