Synthetic data generation is the 'just add water' of fine-tuning advice. I tried it at scale for a real classification task. The 30% synthetic sweet spot is real. Here is what goes wrong when you push past it.
The Setup
Task: classify customer support tickets into 12 intent categories for a B2B SaaS product. Starting dataset: 4,200 human-labelled examples from the past 18 months, unevenly distributed (some categories have 800 examples, two have fewer than 80).
Plan: use GPT-4.1 to generate synthetic examples for underrepresented categories, then fine-tune mistral-7b-v0.3 on the augmented dataset. Target: 1,000 examples per category, 12,000 total.
Actual execution: I ended up generating 50,000 synthetic examples across four experiments as I tried to understand where the approach broke.
Hardware: 2x A100 80 GB SXM via Lambda Labs, $3.40/hr. Fine-tuning runs took 40-90 minutes each. Total compute cost: $62. Total LLM cost for generation: $28. Total time: one long weekend.
import openai
import json
from concurrent.futures import
ThreadPoolExecutor
GENERATION_PROMPT = """
Generate {n} diverse customer support ticket
examples that would be classified as intent: {intent}.
Each example should be written as a real user
would write it — informal, sometimes with typos,
varying length (1-4 sentences). Return a JSON
array of strings. No explanations.
Intent definition: {definition}
Existing examples for reference:
{examples}
"""
def generate_batch(intent: str, definition:
str, seed_examples: list[str], n: int = 20) -> list[str]:
prompt = GENERATION_PROMPT.format(
n=n, intent=intent, definition=definition,
examples="\n".join(f"- {e}" for e in
seed_examples[:5])
)
rsp
= openai.chat.completions.create(
model="gpt-4.1-2025-04-14",
messages=[{"role": "user", "content":
prompt}],
temperature=0.9, # high temp
for diversity
response_format={"type": "json_object"},
)
data
= json.loads(rsp.choices[0].message.content)
return data.get("examples", [])
def generate_for_intent(intent: str,
definition: str, seeds: list[str], target: int) -> list[str]:
results = []
batches = target // 20
with
ThreadPoolExecutor(max_workers=8) as pool:
futures = [pool.submit(generate_batch,
intent, definition, seeds) for _ in range(batches)]
for f in futures:
results.extend(f.result())
return results
|
I used temperature=0.9 for diversity — at 0.2 the examples started repeating within 200 generations. ThreadPoolExecutor at 8 workers cut generation time from 6 hours to 50 minutes for the full 50k set.
Experiment 1: Fill the Long Tail to 1,000
First experiment: bring every intent category up to 1,000 examples by filling gaps with synthetic data. The two underrepresented categories (67 and 83 examples originally) got ~900 synthetic examples each.
Results after fine-tuning:
| Split | Accuracy | Macro F1 |
|---|---|---|
| Baseline (4.2k human only) | 83.1% | 0.79 |
| Exp 1: filled to 1k/class | 87.4% | 0.84 |
Good result. 4.3 points of accuracy improvement from synthetic augmentation. The two underrepresented classes went from F1 of 0.61 and 0.58 to 0.81 and 0.79. This is the success story.
Experiment 2: Keep Going to 5,000
Encouraged, I pushed to 5,000 examples per category — all synthetic beyond the 1,000 mark.
| Split | Accuracy | Macro F1 |
|---|---|---|
| Exp 2: 5k/class (~85% synthetic) | 86.1% | 0.82 |
Worse than Experiment 1. Not catastrophically worse, but measurably worse — and I had spent 5x more on LLM calls. The synthetic examples at high volume started introducing a subtle distribution shift: the model generating the data has its own patterns (sentence structure, vocabulary, how it ends questions) and at high concentration these fingerprints dominate the training distribution.
The 30% Rule Held
I ran two more experiments at 20% and 40% synthetic fractions and plotted the curve:
| Synthetic Fraction | Accuracy | Macro F1 |
|---|---|---|
| 0% (baseline) | 83.1% | 0.79 |
| ~18% | 86.8% | 0.83 |
| ~30% | 87.4% | 0.84 |
| ~50% | 86.7% | 0.83 | |
~85% | 86.1% | 0.82 |
The sweet spot is around 25-35% synthetic. This matches what the Latent Space crew discussed in their data-centric AI episode and aligns with findings from the Phi-3 technical report — synthetic data as a complement to real data, not a replacement.
What Broke at Scale
Diversity collapse: Beyond 500 synthetic examples per intent, GPT-4.1 started recycling phrases and sentence structures even at temperature 0.9. "I'm having trouble with" and "I wanted to reach out because" appeared in 31% of generated examples. Real tickets don't have this pattern concentration.
Mislabelled boundary cases: About 2.3% of generated examples were borderline — things I would label differently from the intent category used to generate them. At 1,000 examples that's 23 noisy labels. At 5,000 it's 115. Fine-tuning amplifies label noise.
Temperature trade-off: High temperature fixes diversity collapse but introduces nonsense. At temp 1.2, about 4% of examples were incoherent strings or clearly not support tickets. I ended up with a filter pass using a fast embedding cosine-similarity check against the seed examples, rejecting anything with cosine similarity below 0.45.
Next Steps
• Try self-instruct-style bootstrapping: use fine-tuned model v1 to generate more synthetic data, iterate
• Test with a different generator model — would Gemini 2.5 produce less fingerprinted synthetic data?
• Add diversity scoring to generation pipeline before training (intra-batch cosine spread)
• Code and all experiment configs at github.com/rexcircuit/synthetic-augment-bench
DIAGRAM_HINT: Line chart showing model accuracy and macro F1 vs. synthetic data fraction (0%, 18%, 30%, 50%, 85%), with a vertical marker at the 30% sweet spot.
Figure 4. Line chart showing accuracy and macro F1 vs. synthetic data fraction (0% to 85%), with a vertical marker at the 30% sweet spot.
Comments (0)
Join the conversation!