Vibe-training a small open-weight model to write regular expressions

posted on 17/11/2026

An animated language model predicting the next tokenTokenThe piece of text a model works with: a word, part of a word or a symbol. Qwen3 knows about 150,000 of them., generated using the /colorfigures skill.

TL;DR:

  • I post-trainedPost-trainingAny training done after pre-training to steer the model: a format, a tone, a task, a domain. Qwen3 (0.6B then 1.7B) to generate JS-flavored regular expressions from English descriptions
  • Resulting model runs on a playground locally in-browser (1.43GB model download): https://regex-vibes.netlify.app
  • Results are mixed: ~60% of good results (=regex passing 100% of tests) on simple prompts, ~20% on medium ones, ~10% on difficult ones. Still, failing regexes pass ~70% of their tests.
  • I made this experiment for learning purposes. I wouldn't recommend using the resulting model for production needs.

Like most software engineers nowadays, I use large language models on a daily basis. While I have a Claude subscription at work (and love it!), I keep an eye on open-weightOpen-weightA model whose weights are published, under a license that allows reusing and modifying them. models that seem to become more and more viable by the week. Moreover, specialized small language models seem to give pretty good results when properly trained.

So I decided to try by myself and I post-trained a small-sized LLMLLM (Large Language Model)A model trained on huge amounts of text that predicts the next piece of text. Chat, code, summaries: everything starts from there. (actually had Opus 5.5 drive me, hence the "vibe-training") to be good at generating (locally in-browser!) JS-flavored regular expressions from an English description of what to match. The resulting model runs in this playground.

I learned a lot doing it: here's the process I followed.

Why regular expressions?

The goal is pretty simple: give a small language model a description in English of what to match, and get a JS regular expression in return.

Plain English

"french phone numbers like 06 12 34 56 78 or 0612345678"

JS regular expression

/^0[1-9](?: ?\d{2}){4}$/

It turns out regular expressions are a good candidate for a first post-training experiment, for several reasons:

The targeted flavor is ES2018 as executed by Node 22: character classes, quantifiers, lookahead and lookbehind, named groups, \p{…}, etc. but not the more recent v flag.

And the starting point was not great: before any trainingTrainingShowing examples to the model, measuring its error, and adjusting its weights so it gets a bit less wrong. In a loop., the candidate models were giving less than 8% of fully correct regular expressions. Not great indeed.

What is post-training?

Large language models are basically machine learning models statistically predicting pieces of text (= tokens) coming after given pieces of text.

Schema of a language model: input tokens go through a matrix of weights that outputs probabilities for the next token

The weightsWeights (parameters)The numbers stored in the model, from millions to billions. Training a model means adjusting these numbers., as shown in the schema, are the numerical parameters built into the model (millions to trillions of them) adjusted during the training of the base modelBase model (foundation)A model fresh out of pre-training: it completes text, but can't answer a question or follow an instruction. to produce the right output to a given input.

Post-training is the process of nudging these weights after the initial training of a model to steer it into giving answers in a certain style, tone, format, or on a very specific domain.

Choosing what model to post-train

There are several factors to take into account when it comes to picking a language model for a given task: the "difficulty" of the task, the performance constraints, the budget, the environment it will run in, etc.

For my use case, I identified the following constraints for my model:

The model must be open-weight

It basically means that the weights we talked about in the previous section are public, and that the license of the model allows their reuse.

Here are some useful links to look for these:

It should have an Instruct / it variant available

When talking about language models, a distinction must be made between two things:

Starting from a model that already has instruct capabilities is valuable in our case as we won't need to train the model for it ourselves.

It should be suitable to be trained on a MacBook Pro with 24GB RAM

Training a machine learning model needs RAM! The reason is that there are several things that need to fit in memory at the same time: the weights of the model to train, training activations, caches, but also other things that are closely tied to the post-training method used:

We also need to keep in mind that the model needs to be compatible with MLXMLXApple's machine learning framework, a PyTorch equivalent optimized for Apple Silicon chips., Apple's machine-learning framework (equivalent of PyTorch or JAX, optimized for Apple Silicon chips).

Final model should be usable in-browser

As we want the resulting model to run on a playground in-browser, we need to validate its compatibility with Transformers.jsTransformers.jsHugging Face's JS library for running models in the browser, on WebGPU or WebAssembly. which is the state-of-the-art solution to run machine learning models on web clients.

There's just a quick thing to validate when picking the model: onnx-community/<model> should exist on the HuggingFace hub. If an ONNXONNXA standard model export format, readable by many runtimes, including Transformers.js in the browser. version of the base model we want to train exists (which is the format used by Transformers.js), it means that our post-trained model will also be exportable to ONNX in the end, and thus usable in-browser.

The end model should also be small enough to be downloaded and stored by end-users. My hard limit on this was ~1.5GB (which is already a lot!).

So, what do we pick?

At this point, we still have several candidates matching our needs:

Artificial Analysis Progress Index bar chart for tiny models: MiniCPM5-1B leads with 163, followed by Qwen3.5-0.8B (140), Qwen3-1.7B and LFM2.5-1.2B-Instruct (131), DeepSeek R1 Distill Qwen 1.5B (126), Qwen3-0.6B (107), LFM2-1.2B (99), Gemma 3 1B (98) and Llama 3.2 1B (92)

I first measured all of these candidates raw (with a few examples of expected answers in the prompt, aka few-shotFew-shotPutting a few solved examples in the prompt, without training anything. Zero-shot: no examples at all. prompting) on two sets: the held-out set (more on it later) and a set of 400 real-world regex needs gathered from regex101. They all scored between ~1% and 8%: no obvious winner...

Model Held-out (few-shot) regex101 (few-shot)
Gemma3-1B 1.4% 3.3%
Granite-4.0-350M 2.9% 3.5%
Llama-3.2-1B 4.8% 5.5%
Qwen2.5-Coder-0.5B 5.9% 5.3%
Qwen3-0.6B 2.3% 2.8%
Qwen3.5-0.8B 4.8% 2.8%
Qwen3-1.7B 5.0% 3.5%

Fully correct regular expressions before any training, on the held-out set and on 400 real-world needs (regex101).

After some online research, it seemed like Qwen3-0.6B was one of the most common bases for small-scale post-training: it's strong for its small size, it supports thinking and non-thinking modes, and it fits easily on an average GPU, so it's cheap to iterate on. So I followed the advice and went with Qwen3-0.6B to iterate (~1h per training), and Qwen3-1.7B for the final version (~2h15). I also disabled Qwen3's thinking mode: for a one-line answer, latency matters more than reasoning.

Building the training dataset

Now that we have a model to train, we need data to train it on. It is one of the most important parts of post-training: a good training dataset is what will give you the best results. Let's see what "good" means here:

General guidelines

Quality is worth more than quantity

For a narrow post-training task done with LoRA on a small (~1B parameters) model, hundreds to a few thousands of training examples are sufficient. A few references going in that direction:

Examples must be verified

As we'll explain further, most post-training techniques rely on imitation, and as models imitate mistakes just as faithfully as good answers, a wrong example is worse than one less example. This is why we want to make sure that our "input -> output" pairs are valid. How to validate is really dependent on the problem we're trying to tackle, but in our case of generating regular expressions, this is pretty straightforward:

For each example, we just keep two lists of strings next to the description and the expected regex: should_match and should_not_match

{
  "description": 
    "french phone numbers like 06 12 34 56 78 or 0612345678",
  "regex": "/^0[1-9](?: ?\\d{2}){4}$/",
  "should_match": [
    "06 12 34 56 78", 
    "0612345678", 
    "01 23 45 67 89"
  ],
  "should_not_match": [
    "6 12 34 56 78", 
    "00 12 34 56 78", 
    "06-12-34-56-78", 
    "061234567"
  ]
}

should_match is a list of strings the regex must match and should_not_match is a list of strings the regex must reject. Both are required for the regex to be considered valid: as /.*/ matches every should_match string possibly existing, should_not_match ensures that generated regexes are not too permissive.

Then, we can just validate each example as such (ofc oversimplified code):

// 1. does the regex compile properly?
const re = new RegExp(source, flags);              
const ok =
  // 2. does it match what it should?
  should_match.every(s => re.test(s)) &&
  // 3. does it not match what it shouldn't?
  should_not_match.every(s => !re.test(s));

If one of the three tests fails, the example is not kept. Only examples that pass all the tests are kept in the dataset.

Diversity of examples taxonomy

Dataset examples must look like real-world usage. To drive the generation of examples, we must define a taxonomy of needs that defines the diversity of the cases we want. The end model will be good at the type of examples it saw during training, so the closer our training dataset resembles what the model will get fed in "production", the better it will perform.

This taxonomy must make explicit the following things about what to generate:

Keep one consistent output format

To be able to properly validate the model's answers, we have to make sure that its output follows a format that is constant and unique: just a JS regular expression literal (/…/flags), nothing else. No explanation, no "Here is the regex:", no ```js block around it.

The parsing of the answers is strict on purpose: a chatty answer is considered malformed, because it can't be compiled as is in Node and therefore can't be validated. To achieve that, the system prompt ("write an ES2018 JS regex, only answer with a /…/flags literal") is defined once and shared everywhere: when generating the dataset, when training, when evaluating, and in the playground.

It might seem like a detail, but it's actually the first thing raw models fail at: before any training, Qwen3-1.7B answered in a wrong format 60% of the time (pattern without delimiters, /…/.test(str), text around the regex...). This is also the first thing post-training fixes: after SFT, malformed answers drop to 1.8% on the held-out (7.5% on regex101).

Generating the dataset with a teacher model

So, where does the data come from? There's no public dataset of "description β†’ regex + tests" that matches our needs, so we have to generate it ourselves. That's where a bigger model comes in: a teacherTeacher modelThe big model that writes the training examples. Here DeepSeek V4.1 Flash, through OpenRouter. model writes the exercises, and Node sorts them out.

The guardrails are where most of the work is. A few things that turned out to matter:

Here are the stats of generating the dataset with DeepSeek V4.1 Flash:

Generated needs

27,460

Kept examples (36%)

9,819

Total cost

$6

Only 36% of the generated needs were kept (guardrails really matter!), and this depends a lot on the category: 9% for logs (long lines with nested rules), 61% for identifiers.

Cleaning and splitting

Once we have our examples, we need to split them, and that's an important step: we never evaluate a model on what it has been trained on.

Why bother having benchmarks and a held-out?

Because the held-out comes from the same teacher as the train: same style, same wording, same way of understanding a need. Evaluating only on it would flatter the model, and we'll see later that it actually does, a lot.

Choosing a post-training method

There are a lot of post-training methods, but they can be grouped in three families, depending on how the model learns.

My need was a precise task with a fixed output format, so I went with SFTSFT (Supervised Fine-Tuning)Learning by imitation: the model is shown "input β†’ correct answer" pairs and learns to reproduce the answer.. Strictly speaking, since every answer in the training dataset was written by DeepSeek, it's actually also a form of distillationDistillationTraining a small model (the student) on the answers of a big model (the teacher)..

In practice, SFT almost always comes first when doing post-training, as it puts the model in the right format and on the right domain before any other method can be applied.

Training with SFT and LoRA

How SFT works

SFT is the same loop as pre-training (predict β†’ measure β†’ correct β†’ repeat), but on our data. For each example, the prompt and the beginning of the right answer are given to the model, and it must guess the next token:

Prompt (context, not graded)

"french phone numbers like 06 12 34 56 78 or 0612345678"

Beginning of the right answer

/^0β–―

Model's guesses for β–―

\d 55%

6 20%

[1-9] 12% βœ“

…

Only the answer is graded, the prompt is just context. We don't want to teach the model to rewrite the user's description.

LoRA: train a small patch rather than the whole model

As explained earlier, LoRA freezes the original weights W of the model, and trains two small matrices B and A next to them: the output of a layer becomes W * x + B * A * x.

In our case, with a rankRank (LoRA)The inner size of the two small LoRA matrices: B is (d Γ— r) and A is (r Γ— d), so r sets how many weights are trained. A higher rank lets the adapter learn more, but costs more memory. Common values: 8 to 64. of 16 on the attention and MLP layers of every block (couldn't have guessed it without Opus), only ~1% of the base model weights are trained, and the result is an adapterAdapterThe output of a LoRA training: a small file (a few dozen MB) applied on top of the original model, or merged into it. of a few dozen MB.

From the Mac to the browser

Once the adapter is trained, a few steps remain to get it running in a browser:

For the final model: 2h15 of training, 7.1GB of memory at peak, and 1.43GB once exported. Yes, that's right at the edge of the ~1.5GB limit I gave myself earlier, but I still went with Qwen3-1.7B because it was clearly better in quality, and Qwen3.5-0.8B, which would have fit and was almost as good for almost half the size, was way too slow in-browser (>5s latency, for model architecture reasons).

Measuring the results

How to evaluate

The evaluation follows the same logic as the dataset validation: for each case, the prompt is given to the model, its answer is parsed as a /…/flags literal, and the regex is run in Node on hidden tests.

The main metric is full success: all the tests must pass for a regex to be considered valid. Because in real life, a regex that is 90% right is a wrong regex. We also keep an eye on the rate of passed tests and on malformed answers, to understand how the model fails.

The numbers

On the held-out:

Qwen3-1.7B raw (few-shot)

5.0%

0.6B after SFT

72%

1.7B after SFT

77%

From 5.0% to 77% of fully correct regular expressions, that's huge! But the held-out is the mock exam... On requests the model has never seen, it gets 6 out of 10 simple requests right, and only 1 out of 10 harder ones...

Still, when it fails, ~70% of the tests pass anyway: the regexes are almost right. Which is actually a problem: an almost right regex is a silent error.

Outside the teacher's mold

Here's what happens when we compare the held-out to the benchmarks, by difficulty:

SFT Qwen3-1.7B Held-out Claude benchmark regex101 benchmark
Simple 88% 73% 46%
Medium 77% 38% 14%
Hard 57% 11% 10%

The held-out measures what the model learned from the teacher, not how good it is in the real world. Even on simple requests, we lose ~15% of good results as soon as we leave DeepSeek's distribution...

For reference, Claude Opus 5.5 (reasoning, medium effort) scores 100% on every one of these benchmarks: the tests are not ambiguous, the gap really comes from the model.

Retrospective

What worked

What didn't

Conclusion

Yes, I know, the final numbers are not that good. But to be honest I did not really expect to get 100% success. My goal was mainly to learn the basics of post-training, and to make sure of something: a small specialized model can give decent results on a narrow task. And as shown by the results, the format and specificities of a teacher model can be learned very well from a few thousand examples.

You can try the result by yourself on the playground (Chromium with WebGPU only for now).

Hope you learned something!

Want to reproduce it?