2+Frontier Teachers 0.85Similarity Cutoff 26000+DISTILLED Examples 200%Dataset Expansion A lot of AI tasks do not require, or do not have access to, frontier models. To solve this 'AI at the edge' problem we can distill a dataset to enrich the training of a smaller model, that we can then run locally. Distillation is using a larger model to create 'synthetic data' to be used to train a new, typically smaller model. We use a two-stage pipeline that first, Distills or Enriches a high-quality dataset using multiple frontier models like Anthropic Opus and ChatGPT-5, and then Prunes the result with Sentence Similarity so the final corpus stays diverse rather than repetitive. The core idea: A small model rarely fails because it lacks parameters. It fails because it was trained on narrow, redundant, or low-quality data. Improving the data typically results in improving the model. 01Distill & Enrich Every distillation pipeline either begins with a small, human created, or synthetically created "seed" dataset from a larger model. You have two practical options for sourcing datasets: Manually authored examples, written by domain experts who understand the exact task, tone, and edge cases your model must handle. Curated from existing sources, extracted from custom datasets, internal datasets, or public datasets, then cleaned and reformatted. Depending on the design goals of your model, a few thousand carefully crafted examples is usually enough to start getting results. Accurately Labeled data, and Data Diversity are MUCH more important than quantity. This is because if the data is not diverse enough, 'short cuts' can arise which can unintentionally bias the model. Quality over quantity: For example, assume you're trying to detect prompt injection attacks, and the dataset you're training on has a larger portion of "Do Anything Now" or "DAN" style prompt injections than anything else. The model could get unintentionally biased on "do anything now" or "dan" keywords within benign prompts, causing a large number of false positives AND ignoring prompt injections that don't leverage DAN style attacks. Different frontier models have different strengths, blind spots, and stylistic fingerprints. By blending outputs, you avoid baking the quirks of a single model into your new model. Frontier Model Tends to Excel At Role in the Mix ChatGPT-5 Breadth of coverage, creative variation Creativity Anthropic Opus Nuanced reasoning, careful tone, long-form structure, Code Technical Depth Additional model(s) Domain-specific or stylistic diversity Optional Diversity & Depth A few patterns make expansion far more effective: Cross-model verification, have one model generate an answer and another critique or rewrite it, keeping only outputs both agree are strong. Explicit diversity instructions, ask the model to cover edge cases, rare intents, and adversarial inputs, not just the obvious center of the distribution. Temperature & Length variation, sweep across temperatures to push the models into different regions of the output space and surface fresh phrasings. Multiple Models Matter: If you distill from a single model, your dataset inherits a single model's perspective. Blending more models produces a richer, more balanced target distribution. By the end of this stage you'll have a large, raw corpus. But raw volume hides a problem: frontier models, asked to riff on similar tasks, produce a lot of similar responses. We need to make sure the data is diverse enough to not unintentionally bias our smaller model. 02Prune for Diversity with Sentence Similarity While distillation outputs quantity, pruning restores diversity. A dataset that's 80% paraphrases of the same five ideas will bias your small model to paraphrase a narrow range of things. To solve this we check each example against all the other examples using Sentence Similarity. The workflow is straightforward: Encode every example into a dense vector using a sentence-transformers model (in our case: all-MiniLM-L6-v2). Compute cosine similarity between examples to measure how semantically close they are. When two examples exceed a similarity threshold (e.g. 0.85), treat them as redundant and keep only one. Optionally, cluster the embeddings and sample evenly across clusters to guarantee broad coverage. Stage Example Count Characteristic Initial Set 50,000 High Volume, Low Diversity After pruning 26,168 Reduced Volume, High Diversity Counterintuitively, throwing away half your data often improves the final model. You're removing the redundant data that would have over-weighted common patterns, and preserving the diverse examples that help reduce 'short cuts' from sprouting within the model. Tune the threshold deliberately: Too aggressive (low cutoff) and you delete useful variation; too lenient (high cutoff) and duplicates slip through. Start near 0.85, inspect the borderline pairs by hand, and adjust. 03Train, Evaluate, Iterate Once a pruned and diverse dataset has been obtained, a new model can be created. Typically, it's better to start with a compact base model, split your dataset into 80% training and 20% validation, then train/fine-tune the based model on your training dataset. Once complete, evaluate your model against 20% validation chunk of your original seed set. Treat the whole pipeline as a loop: Find areas where the new model underperforms. Author or source new datasets targeting those weaknesses. Re-run distillation and pruning on the updated datasets. Retrain and Re-benchmark. When done correctly, each cycle helps the small model increase performance on the specific task you care about, without the continued cost of the frontier models. Frontier models are extraordinary data engines. Leveraging models like Opus and GPT-5, combined with pruning on Sentence Similarity, results in creating solid synthetic data sets for a variety of AI tasks.
