Fine-Tuning LLMs - Dev Weekends

December 2025 Update: Covers OpenAI fine-tuning, LoRA/QLoRA with Hugging Face, when to fine-tune vs. prompt, and cost analysis.

When to Fine-Tune

Think of fine-tuning like hiring a specialist versus giving better instructions to a generalist. A general-purpose model (GPT-4o) is like a brilliant consultant who can handle anything but needs detailed briefing every time. A fine-tuned model is like an employee who has internalized your company’s style, jargon, and processes — they need less instruction per task but it took investment to train them. The question is always: is the upfront training cost worth the per-request savings? Fine-tuning is not your first option. Consider this decision tree:

┌─────────────────────────────────────────────────────────────┐
│               Do you need fine-tuning?                       │
└─────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │ Try better prompts│
                    │   first?          │
                    └─────────┬─────────┘
                              │ Still not working
                    ┌─────────▼─────────┐
                    │ Try few-shot      │
                    │ examples?         │
                    └─────────┬─────────┘
                              │ Still not working
                    ┌─────────▼─────────┐
                    │ Try RAG for       │
                    │ knowledge?        │
                    └─────────┬─────────┘
                              │ Still not working
                    ┌─────────▼─────────┐
                    │ NOW consider      │
                    │ fine-tuning       │
                    └───────────────────┘

Fine-Tune When

Use Case	Why Fine-Tuning Helps
Specific format	Consistent JSON structure, code style
Domain language	Medical, legal, technical jargon
Brand voice	Consistent tone across all outputs
Latency	Smaller fine-tuned model beats large model
Cost	Reduce tokens by baking knowledge in
Classification	High-accuracy categorization

Don’t Fine-Tune When

Use Case	Better Alternative
Add new knowledge	Use RAG instead
One-off tasks	Better prompts
Rapidly changing info	RAG with fresh data
General improvement	Use a bigger model

OpenAI Fine-Tuning

OpenAI’s hosted fine-tuning is the easiest on-ramp: you upload a JSONL file, click start, and get a custom model endpoint back. No GPUs to manage, no training infrastructure to build. The tradeoff is cost and control — you pay per training token, and you cannot inspect the model weights or run it locally.

Preparing Training Data

import json
from pathlib import Path

def create_training_example(system: str, user: str, assistant: str) -> dict:
    """Create a single training example in OpenAI format.
    
    Each example is a complete conversation: system prompt sets the role,
    user provides the input, assistant provides the ideal response.
    The model learns to mimic the assistant's behavior.
    """
    return {
        "messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": user},
            {"role": "assistant", "content": assistant}
        ]
    }

def prepare_dataset(examples: list[dict], output_path: str):
    """Prepare dataset in JSONL format"""
    with open(output_path, 'w') as f:
        for ex in examples:
            f.write(json.dumps(ex) + '\n')

# Example: Customer support fine-tuning
SYSTEM_PROMPT = "You are a helpful customer support agent for TechCorp."

training_data = [
    create_training_example(
        SYSTEM_PROMPT,
        "How do I reset my password?",
        "I'd be happy to help you reset your password! Here's how:\n\n1. Go to techcorp.com/reset\n2. Enter your email address\n3. Click the link in the email we send you\n4. Create a new password\n\nIf you don't receive the email within 5 minutes, check your spam folder. Need more help?"
    ),
    create_training_example(
        SYSTEM_PROMPT,
        "Your product sucks!",
        "I'm sorry to hear you're frustrated. I'd really like to help make things right. Could you tell me more about what's not working for you? Once I understand the issue, I can either help fix it or connect you with someone who can."
    ),
    # Add 50-100+ examples for good results
]

prepare_dataset(training_data, "training_data.jsonl")

Validating Data

Always validate before uploading. A single malformed line in your JSONL file will cause the entire fine-tuning job to fail, sometimes after you have already waited an hour in the queue. This validator catches the most common mistakes: missing roles, invalid JSON, and examples that exceed the context window.

import json
import tiktoken

def validate_training_file(path: str, model: str = "gpt-4o-mini"):
    """Validate training data before upload.
    
    Practical tip: run this locally before uploading. OpenAI's server-side
    validation is slow and the error messages are sometimes cryptic.
    """
    enc = tiktoken.encoding_for_model("gpt-4")
    
    errors = []
    warnings = []
    total_tokens = 0
    
    with open(path) as f:
        for i, line in enumerate(f, 1):
            try:
                example = json.loads(line)
            except json.JSONDecodeError:
                errors.append(f"Line {i}: Invalid JSON")
                continue
            
            # Check structure
            if "messages" not in example:
                errors.append(f"Line {i}: Missing 'messages' key")
                continue
            
            messages = example["messages"]
            
            # Check roles
            roles = [m.get("role") for m in messages]
            if "assistant" not in roles:
                errors.append(f"Line {i}: No assistant message")
            if "user" not in roles:
                warnings.append(f"Line {i}: No user message")
            
            # Count tokens
            text = " ".join(m.get("content", "") for m in messages)
            tokens = len(enc.encode(text))
            total_tokens += tokens
            
            if tokens > 16000:
                warnings.append(f"Line {i}: {tokens} tokens (may be truncated)")
    
    # Summary
    with open(path) as f:
        num_examples = sum(1 for _ in f)
    
    print(f"Examples: {num_examples}")
    print(f"Total tokens: {total_tokens:,}")
    print(f"Estimated cost: ${total_tokens * 0.008 / 1000:.2f} (training)")
    print(f"Errors: {len(errors)}")
    print(f"Warnings: {len(warnings)}")
    
    for e in errors[:10]:
        print(f"  ❌ {e}")
    for w in warnings[:10]:
        print(f"  ⚠️ {w}")
    
    return len(errors) == 0

# Validate
validate_training_file("training_data.jsonl")

Running Fine-Tuning

The process has four steps: upload your data, create a job, monitor progress, and use the resulting model. The whole pipeline typically takes 30 minutes to a few hours depending on dataset size.

from openai import OpenAI

client = OpenAI()

# Step 1: Upload training file
with open("training_data.jsonl", "rb") as f:
    training_file = client.files.create(
        file=f,
        purpose="fine-tune"
    )
print(f"Uploaded file: {training_file.id}")

# Step 2: Create fine-tuning job
# The suffix appears in your model name: ft:gpt-4o-mini:techcorp-support-v1
# Use it to track which version is deployed
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",  # Base model to fine-tune from
    hyperparameters={
        "n_epochs": 3,  # 2-4 is usually the sweet spot; more risks overfitting
        "batch_size": "auto",         # Let OpenAI optimize this
        "learning_rate_multiplier": "auto"  # Same -- auto is usually best
    },
    suffix="techcorp-support-v1"
)
print(f"Job created: {job.id}")

# Step 3: Monitor progress
import time

while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    print(f"Status: {job.status}")
    
    if job.status in ["succeeded", "failed", "cancelled"]:
        break
    
    time.sleep(60)

# Step 4: Use fine-tuned model
if job.status == "succeeded":
    print(f"Fine-tuned model: {job.fine_tuned_model}")
    
    response = client.chat.completions.create(
        model=job.fine_tuned_model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": "I can't log in"}
        ]
    )
    print(response.choices[0].message.content)

Local Fine-Tuning with LoRA

Why LoRA?

Imagine you need to customize a 7-billion-parameter model. Full fine-tuning means updating all 7 billion weights, which requires 80GB+ of VRAM — that is a $10,000+ GPU. LoRA (Low-Rank Adaptation) is a clever shortcut: instead of updating all weights, it freezes the original model and injects small trainable “adapter” matrices into each layer. You end up training less than 1% of the parameters while getting 90%+ of the quality improvement. It is like adding a specialized lens to a camera instead of rebuilding the entire camera. LoRA lets you fine-tune large models on consumer hardware:

Method	VRAM Needed	Training Time	Quality
Full fine-tune	80GB+	Days	Best
LoRA	16-24GB	Hours	Great
QLoRA	8-12GB	Hours	Good

Setup

pip install transformers datasets peft accelerate bitsandbytes trl

QLoRA Fine-Tuning

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# Model configuration
MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.2"
OUTPUT_DIR = "./fine-tuned-model"

# QLoRA configuration (4-bit quantization)
# This compresses the frozen weights to 4-bit precision, cutting VRAM by ~4x
# while keeping training quality nearly identical to full precision
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,                      # Compress weights to 4 bits
    bnb_4bit_quant_type="nf4",              # NormalFloat4 -- best quality for LLMs
    bnb_4bit_compute_dtype=torch.float16,   # Compute in fp16 for speed
    bnb_4bit_use_double_quant=True,         # Quantize the quantization constants too
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model = prepare_model_for_kbit_training(model)

# LoRA configuration
lora_config = LoraConfig(
    r=16,           # Rank of the adapter matrices. Higher = more capacity but more VRAM.
                    # Start with 8-16; go to 32-64 only if quality plateaus.
    lora_alpha=32,  # Scaling factor. Rule of thumb: set to 2x the rank.
    lora_dropout=0.05,  # Small dropout to prevent overfitting on small datasets
    bias="none",        # Do not train bias terms -- saves memory, rarely helps
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]  # Attention layers only
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Usually <1% of params

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# Load your dataset
dataset = load_dataset("json", data_files="training_data.jsonl")

def format_prompt(example):
    """Format for instruction tuning"""
    messages = example["messages"]
    text = ""
    for msg in messages:
        if msg["role"] == "system":
            text += f"<s>[INST] <<SYS>>\n{msg['content']}\n<</SYS>>\n\n"
        elif msg["role"] == "user":
            text += f"{msg['content']} [/INST] "
        else:
            text += f"{msg['content']}</s>"
    return {"text": text}

dataset = dataset.map(format_prompt)

# Training arguments
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    weight_decay=0.01,
    logging_steps=10,
    save_strategy="epoch",
    fp16=True,
    optim="paged_adamw_8bit",
)

# Train
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    args=training_args,
    max_seq_length=2048,
    dataset_text_field="text",
)

trainer.train()

# Save
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

Inference with LoRA Model

At inference time, you load the base model plus the tiny LoRA adapter (usually 10-50MB). For production, you can merge the adapter into the base weights with merge_and_unload() — this eliminates the adapter overhead and gives you a single model that runs at native speed.

from peft import PeftModel

# Load base model + LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, OUTPUT_DIR)

# Merge for faster inference (optional)
model = model.merge_and_unload()

# Generate
def generate(prompt: str, max_tokens: int = 500) -> str:
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_tokens,
            temperature=0.7,
            do_sample=True,
            top_p=0.9,
        )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Fine-Tuning Best Practices

Data Quality Over Quantity

The single most impactful thing you can do for fine-tuning quality is curate better data. 50 perfect, diverse examples consistently outperform 500 mediocre ones with inconsistent formatting. Think of it like training a new employee: five well-structured onboarding sessions beat a month of watching someone else wing it.

# ❌ Bad: Low-quality, inconsistent examples
bad_examples = [
    {"user": "help", "assistant": "ok"},
    {"user": "????", "assistant": "I don't understand"},
]

# ✅ Good: High-quality, consistent format
good_examples = [
    {
        "user": "How do I export my data?",
        "assistant": "I'd be happy to help you export your data! Here's how:\n\n**For CSV export:**\n1. Go to Settings > Data\n2. Click 'Export'\n3. Select 'CSV' format\n4. Choose the date range\n5. Click 'Download'\n\nThe file will be emailed to you within 5 minutes. Let me know if you need help with anything else!"
    }
]

Evaluation After Fine-Tuning

Never ship a fine-tuned model without comparing it head-to-head against the baseline. Fine-tuning can improve some dimensions while regressing others (e.g., better format compliance but worse factual accuracy). Always measure both.

def evaluate_fine_tuned_model(
    model_id: str,
    eval_dataset: list[dict],
    baseline_model: str = "gpt-4o-mini"
) -> dict:
    """Compare fine-tuned model to baseline.
    
    Practical tip: use a held-out eval set that was NOT part of training.
    If your eval examples overlap with training data, you are measuring
    memorization, not generalization.
    """
    
    results = {"fine_tuned": [], "baseline": []}
    
    for example in eval_dataset:
        # Fine-tuned response
        ft_response = client.chat.completions.create(
            model=model_id,
            messages=example["messages"][:-1]  # Exclude assistant
        ).choices[0].message.content
        
        # Baseline response
        bl_response = client.chat.completions.create(
            model=baseline_model,
            messages=example["messages"][:-1]
        ).choices[0].message.content
        
        # Score both
        expected = example["messages"][-1]["content"]
        
        ft_score = llm_judge(ft_response, expected)
        bl_score = llm_judge(bl_response, expected)
        
        results["fine_tuned"].append(ft_score)
        results["baseline"].append(bl_score)
    
    return {
        "fine_tuned_avg": sum(results["fine_tuned"]) / len(results["fine_tuned"]),
        "baseline_avg": sum(results["baseline"]) / len(results["baseline"]),
        "improvement": (sum(results["fine_tuned"]) - sum(results["baseline"])) / len(results["fine_tuned"])
    }

Cost Comparison

Fine-tuning has an upfront cost (training tokens) that you amortize over inference. The key question: does the reduced prompt size (because you baked knowledge into the model) save enough per-request to offset the training investment? This function does the math. Practical tip: if your break-even is longer than 90 days, the model will likely need retraining before you recoup the investment.

def calculate_fine_tuning_roi(
    training_tokens: int,
    daily_inference_tokens: int,
    days: int = 30
) -> dict:
    """Calculate if fine-tuning is worth it.
    
    A fine-tuned model often needs 40-60% fewer input tokens because
    you can drop few-shot examples and lengthy instructions. This is
    where the real savings come from -- not the per-token rate.
    """
    
    # OpenAI pricing (as of Dec 2025)
    prices = {
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
        "gpt-4o-mini-ft": {"input": 0.30, "output": 1.20, "training": 3.00},
        "gpt-4o": {"input": 2.50, "output": 10.00},
    }
    
    # Training cost (one-time)
    training_cost = (training_tokens / 1_000_000) * prices["gpt-4o-mini-ft"]["training"]
    
    # Inference cost comparison
    # Assume 50% input, 50% output tokens
    input_tokens = daily_inference_tokens * 0.5
    output_tokens = daily_inference_tokens * 0.5
    
    # Scenario 1: Use base model with long prompts
    base_daily = (
        (input_tokens / 1_000_000) * prices["gpt-4o-mini"]["input"] +
        (output_tokens / 1_000_000) * prices["gpt-4o-mini"]["output"]
    )
    
    # Scenario 2: Fine-tuned with shorter prompts (assume 40% fewer tokens)
    ft_input = input_tokens * 0.6
    ft_output = output_tokens  # Output usually similar
    ft_daily = (
        (ft_input / 1_000_000) * prices["gpt-4o-mini-ft"]["input"] +
        (ft_output / 1_000_000) * prices["gpt-4o-mini-ft"]["output"]
    )
    
    total_base = base_daily * days
    total_ft = training_cost + (ft_daily * days)
    
    return {
        "training_cost": f"${training_cost:.2f}",
        "base_model_monthly": f"${total_base:.2f}",
        "fine_tuned_monthly": f"${total_ft:.2f}",
        "savings": f"${total_base - total_ft:.2f}",
        "break_even_days": training_cost / (base_daily - ft_daily) if base_daily > ft_daily else "Never"
    }

# Example
print(calculate_fine_tuning_roi(
    training_tokens=500_000,      # 500K training tokens
    daily_inference_tokens=1_000_000,  # 1M tokens/day
    days=30
))

Key Takeaways

Exhaust Alternatives First

Try prompts, few-shot, and RAG before fine-tuning. It’s often unnecessary.

Quality Over Quantity

50 perfect examples beat 500 mediocre ones. Curate carefully.

LoRA for Local

Use QLoRA to fine-tune 7B+ models on consumer GPUs.

Always Evaluate

Compare fine-tuned model to baseline. Measure the improvement.

What’s Next

Evaluation & Testing

Learn how to properly evaluate your fine-tuned models

Interview Deep-Dive

Your team wants to fine-tune a model for customer support. Walk me through how you decide whether fine-tuning is actually the right approach.

Strong Answer:

Before I even consider fine-tuning, I would exhaust the cheaper alternatives in order. Step one: improve the prompt. Most “the model is not good enough” complaints I have seen in production were actually “the prompt is lazy.” I would spend 2-3 days iterating on the system prompt with clear instructions, formatting examples, and explicit constraints. Step two: add few-shot examples. For customer support specifically, including 3-5 examples of ideal responses in the prompt often gets you 80% of the way there. Step three: if the issue is that the model does not know about our product, that is a knowledge problem and RAG solves it better than fine-tuning. Fine-tuning bakes knowledge into weights, which means it goes stale the moment your product changes.
Fine-tuning becomes the right choice when the problem is behavior, not knowledge. Specific scenarios: I need the model to always respond in a very specific JSON format and it keeps drifting; I need a consistent brand voice that few-shot examples cannot fully capture across hundreds of edge cases; I need to reduce latency by using a smaller model that matches a larger model’s quality on this narrow task; or I need to reduce cost by eliminating long system prompts (fine-tuning bakes instructions into the weights so the prompt can be much shorter).
For the customer support case specifically, I would benchmark the current prompt-engineered approach against a held-out test set of 200 real customer conversations. If accuracy is above 85% and the main issue is consistency of tone, I would try fine-tuning. If accuracy is below 70%, the problem is likely the retrieval or prompt, not the base model capability.
The ROI calculation matters too. Fine-tuning GPT-4o-mini costs roughly $3 per million training tokens. If I am processing 1 million tokens per day in production, and fine-tuning lets me cut my prompt by 40% (because I no longer need lengthy instructions), I save about$ 0.04 per day on input costs with the mini model. That is roughly $1.20 per month in savings against a one-time training cost of maybe$ 5-15. The payback period is about a month, which is reasonable, but only if the quality improvement is real.

Red Flags: Candidate jumps straight to fine-tuning without mentioning prompt engineering or RAG. Another red flag is not knowing the difference between teaching a model new knowledge (use RAG) versus teaching it a new behavior (use fine-tuning).Follow-up: You decide to fine-tune. How many training examples do you need and how do you know when you have enough?For OpenAI fine-tuning, the practical minimum is 50 high-quality examples, but I have seen good results start around 100-200 examples for a focused task like customer support tone. The key word is high-quality. 50 meticulously crafted examples with consistent formatting, complete responses, and covering the main intent categories will outperform 500 sloppy examples copy-pasted from chat logs. I evaluate “enough” by plotting a learning curve: I fine-tune with 50, 100, 200, and 400 examples and measure quality on a held-out test set. If quality plateaus between 200 and 400, adding more data has diminishing returns and my time is better spent improving data quality. The other signal is overfitting: if training loss keeps dropping but eval quality stagnates or degrades, I have either too many epochs or too little data diversity. For customer support, I also make sure the training set covers the long tail of intents proportionally — if 60% of real queries are about returns but 90% of my training data is about returns, the model will underperform on everything else.

Explain the difference between full fine-tuning, LoRA, and QLoRA. When would you pick each?

Strong Answer:

Full fine-tuning updates every parameter in the model. For a 7B parameter model, that means 7 billion weights getting gradient updates, which requires roughly 80GB+ of VRAM and can take days. The result is the highest quality because you have maximum capacity to adapt, but the cost and infrastructure requirements are steep. I would pick full fine-tuning only for mission-critical applications with a large training set (10K+ examples) and access to multi-GPU clusters, or when the task is extremely different from what the base model was trained on.
LoRA (Low-Rank Adaptation) freezes the original model weights and injects small trainable matrices into the attention layers. Instead of updating 7 billion parameters, you update maybe 10-50 million (less than 1% of the model). This drops VRAM requirements to 16-24GB, training time to hours instead of days, and quality is surprisingly close to full fine-tuning for most tasks. The rank parameter r controls the capacity: r=8 is minimal but fast, r=16 is the sweet spot for most tasks, r=64 approaches full fine-tuning capacity. I use LoRA as my default choice for any fine-tuning project because the quality-to-cost ratio is unmatched.
QLoRA adds 4-bit quantization on top of LoRA. The base model weights are quantized to 4-bit precision (using NF4 quantization), and the LoRA adapters train in 16-bit on top of that. This cuts VRAM to 8-12GB, meaning you can fine-tune a 7B model on a single consumer GPU like an RTX 3090. The quality tradeoff is small but measurable — roughly 1-2% below LoRA on most benchmarks. I pick QLoRA when I am prototyping on limited hardware, when I need to iterate quickly on data quality before committing to a full LoRA run, or for hobbyist/research settings.
The practical decision tree: if I have an A100 80GB, I use LoRA. If I have an RTX 4090 24GB, I use LoRA with gradient checkpointing. If I have an RTX 3090 or less, I use QLoRA. If I have a multi-node cluster and a six-figure compute budget, I consider full fine-tuning.

Red Flags: Candidate cannot explain what “low-rank” means in LoRA, confuses fine-tuning with prompt tuning, or does not know approximate VRAM requirements for different approaches.Follow-up: After fine-tuning with LoRA, how do you serve the model in production — do you keep the adapter separate or merge it?It depends on the serving scenario. If I am deploying a single fine-tuned model, I merge the LoRA adapter into the base weights using model.merge_and_unload(). This produces a standard model with zero inference overhead — no adapter routing, no extra memory for the adapter matrices. Inference speed is identical to the base model. If I am serving multiple fine-tuned variants from the same base model (for example, different customer tenants each with their own fine-tune), I keep the adapters separate and load them dynamically. Frameworks like vLLM and LoRAX support multi-adapter serving where the base model weights are shared in GPU memory and adapters are swapped per request with minimal overhead. This is dramatically more memory-efficient than loading N separate full models. At one company we served 12 different LoRA fine-tunes from a single base model on two A100s, where loading 12 full models would have required 24 GPUs.

You fine-tuned a model and it scores well on your eval set but performs poorly on production traffic. What happened?

Strong Answer:

The most likely cause is distribution mismatch between the training data and real-world queries. This is the fine-tuning equivalent of the classic ML problem of training on clean data and deploying into the wild. I would diagnose it in this order.
First, I check for data leakage. If any production-like queries leaked into the training set, the eval set might have been too easy. I verify that the eval set was held out properly and is drawn from a different time period or source than the training data.
Second, I analyze the production failures by category. Are they concentrated in specific intent types? If so, those intents were probably underrepresented in training data. Customer support has a long tail — the top 10 query types might cover 60% of traffic, but the remaining 40% is spread across hundreds of edge cases. If my training set only covered the top 10, the fine-tuned model might have actually gotten worse at handling anything outside that distribution because fine-tuning can cause the model to “forget” some of its general capabilities. This is called catastrophic forgetting.
Third, I check for overfitting. If I trained for too many epochs or my training set was too small and homogeneous, the model memorized patterns instead of learning generalizable behavior. I look at the training loss curve — if it kept dropping well below the validation loss, that is a smoking gun. The fix is fewer epochs (2-3 is usually sufficient for OpenAI fine-tuning), more diverse training data, or a higher LoRA dropout rate.
Fourth, I compare the fine-tuned model against the base model with a good prompt on the failing production queries. If the base model actually handles those queries better, the fine-tuning introduced a regression. This is why I always keep the base model available as a fallback and monitor quality metrics with automatic rollback.

Red Flags: Candidate does not mention distribution mismatch or catastrophic forgetting, assumes more training data always helps, or does not have a rollback strategy.Follow-up: How do you prevent catastrophic forgetting when fine-tuning on a narrow domain?Three strategies. First, mix in general-purpose examples alongside domain-specific ones. I typically allocate 10-20% of the training set to high-quality general conversations so the model retains its broad capabilities. Second, use fewer training epochs — for narrow-domain fine-tuning, 1-2 epochs is often better than 3-4 because each additional epoch pushes the model further from its general knowledge. Third, with LoRA specifically, I use a lower rank (r=8 instead of r=16) for narrow domains because the adapter has less capacity to overwrite the base model’s general behavior. The nuclear option is to keep two models in production: the fine-tuned model for in-domain queries and the base model for everything else, with a classifier routing between them. This is more operational complexity but guarantees you never regress on out-of-domain performance.

How do you calculate whether fine-tuning is worth the investment? Walk me through the cost analysis.

Strong Answer:

The ROI calculation has four components: one-time training cost, ongoing inference cost delta, quality improvement value, and maintenance cost.
Training cost for OpenAI fine-tuning of GPT-4o-mini is roughly $3 per million training tokens. If I have 500 well-crafted examples averaging 500 tokens each, that is 250K tokens, so about$ 0.75 in training cost. Add validation and a couple re-runs for hyperparameter tuning, and total training cost is under $5. This is trivially cheap -- the data preparation labor (20-40 hours of expert time at$ 100+/hour) dwarfs the compute cost by 100x.
Inference cost delta is where the real savings come from. Fine-tuning lets me shrink the prompt because instructions are baked into the weights. If my current system prompt is 2000 tokens and fine-tuning lets me reduce it to 200 tokens, I save 1800 input tokens per call. At GPT-4o-mini rates of $0.15 per million input tokens, and processing 100K requests per day, that saves about$ 27 per day or roughly $810 per month. The fine-tuned model costs 2x per token ($ 0.30 vs $0.15 for input), but the net savings from shorter prompts still comes out positive if the prompt shrinkage is significant.
Quality improvement value is harder to quantify but often the real driver. If fine-tuning reduces customer escalation rate from 15% to 10%, and each escalation costs $5 in human agent time, at 10K queries per day that saves$ 2,500 per day. This dwarfs the compute costs.
Maintenance cost is the hidden one. The fine-tuned model needs re-training when policies change, when new products launch, or when the base model gets deprecated. I budget for quarterly re-training cycles and keep the data preparation pipeline automated so re-training is a one-click operation, not a multi-week project.

Red Flags: Candidate only considers compute costs and ignores data preparation labor, does not mention the 2x inference pricing for fine-tuned models on OpenAI, or forgets about maintenance and retraining costs.Follow-up: When does fine-tuning a smaller model beat just using a larger model with a better prompt?The crossover point is when latency or cost constraints make the larger model impractical. For example, a fine-tuned GPT-4o-mini at

0.30 per million input tokens with a 200-token prompt costs about

0.00006 per request with sub-500ms latency. GPT-4o with a 2000-token prompt costs about $0.005 per request with 1-3 second latency — that is 83x more expensive and 3-6x slower. If I can get the fine-tuned mini model to match GPT-4o quality on my specific narrow task, the economics are overwhelming. The key qualifier is “on my specific narrow task.” Fine-tuned small models excel on focused, well-defined tasks with clear patterns. They fall apart on open-ended reasoning, complex multi-step tasks, or anything requiring broad world knowledge. My rule of thumb: if the task can be described with 50 examples, fine-tune a small model. If it needs 5000 examples to cover the space, the bigger model with a good prompt is cheaper when you include data preparation costs.

​When to Fine-Tune

​Fine-Tune When

​Don’t Fine-Tune When

​OpenAI Fine-Tuning

​Preparing Training Data

​Validating Data

​Running Fine-Tuning

​Local Fine-Tuning with LoRA

​Why LoRA?

​Setup

​QLoRA Fine-Tuning

​Inference with LoRA Model

​Fine-Tuning Best Practices

​Data Quality Over Quantity

​Evaluation After Fine-Tuning

​Cost Comparison

​Key Takeaways

Exhaust Alternatives First

Quality Over Quantity

LoRA for Local

Always Evaluate

​What’s Next

Evaluation & Testing

​Interview Deep-Dive

When to Fine-Tune

Fine-Tune When

Don’t Fine-Tune When

OpenAI Fine-Tuning

Preparing Training Data

Validating Data

Running Fine-Tuning

Local Fine-Tuning with LoRA

Why LoRA?

Setup

QLoRA Fine-Tuning

Inference with LoRA Model

Fine-Tuning Best Practices

Data Quality Over Quantity

Evaluation After Fine-Tuning

Cost Comparison

Key Takeaways

What’s Next

Interview Deep-Dive