Core to Deep Dive
Review fine-tuning strategies, LoRA, PEFT, catastrophic forgetting, data preparation, RLHF, and alignment methods.
Depth
Showing 5 of 5 questions
Short answer
Fine-tuning updates model weights on a domain-specific dataset to improve performance on a targeted task. Use it when prompt engineering and RAG cannot achieve the needed accuracy, latency, or cost profile and you have sufficient high-quality labeled data.
Interview-ready answer
I consider fine-tuning only after prompt engineering and retrieval-based approaches have been pushed to their limits. Fine-tuning can reduce prompt length, lower latency, and improve adherence to domain conventions. But it requires labeled data, risks catastrophic forgetting, introduces model-serving complexity, and may not fix knowledge-gap issues that RAG handles more naturally. I prefer PEFT methods to start, and I evaluate the fine-tuned model against the pre-trained baseline on both target and general capabilities.
Common mistakes
Short answer
LoRA (Low-Rank Adaptation) freezes the original weights and injects trainable low-rank matrices into attention layers. This reduces memory and storage requirements dramatically while maintaining most of the quality of full fine-tuning.
Interview-ready answer
LoRA decomposes the weight update into two low-rank matrices whose product approximates the delta. Training updates only these small adapters, which typically reduces trainable parameters by 10,000x and GPU memory by 2–3x. At inference, the adapters can be merged into the base weights or loaded separately, making it practical to serve many fine-tuned variants from one base model. I choose rank based on task complexity — rank 8 to 16 for most tasks, 32 to 64 for more complex domains.
Common mistakes
Short answer
Catastrophic forgetting occurs when the model loses previously learned capabilities after fine-tuning on a narrow domain. Prevent it with mixed training that includes general data, elastic weight consolidation, replay buffers, LoRA adapters that preserve base weights, and thorough evaluation on general benchmarks.
Interview-ready answer
I mitigate forgetting by reserving 10–20% of each training batch for general-domain data, using LoRA or other PEFT methods that preserve full base weights, and tracking evaluation metrics on both target and general benchmarks throughout training. If general performance drops unacceptably, I reduce learning rate, increase general-data proportion, or switch to a larger base model with more capacity. The acceptance threshold depends on whether the fine-tuned model needs strong general capabilities.
Common mistakes
Short answer
Use prompt engineering first for rapid iteration. Use RAG when the answer depends on external, updateable, or access-controlled knowledge. Use fine-tuning when the task requires a specific behavior, style, or format that cannot be achieved through prompting alone and you have sufficient labeled data.
Interview-ready answer
Each technique addresses a different bottleneck. Prompt engineering changes how the model uses its existing knowledge. RAG supplies new or private information without weight changes. Fine-tuning modifies the model's behavior, tone, or domain fluency. In production these often complement each other: RAG supplies current facts while fine-tuning adapts the model's domain conventions. I start with the cheapest lever — prompt engineering — and add complexity only when metrics show a clear gap.
Common mistakes
Related
Short answer
RLHF (Reinforcement Learning from Human Feedback) aligns model outputs with human preferences through a three-stage process: supervised fine-tuning on demonstrations, training a reward model from human comparisons, and optimizing the policy with PPO or a similar algorithm against the reward model.
Interview-ready answer
RLHF addresses the gap between language-modeling objectives and helpful, harmless behavior. The reward model learns what humans prefer from pairwise comparisons, then the policy is tuned to maximize that reward while staying near the SFT initialization via a KL penalty. This reduces harmful outputs but can also reduce diversity and cause the model to exploit reward-model blind spots. Direct preference optimization (DPO) simplifies the pipeline by treating preference data as the direct training signal. I evaluate alignment interventions on safety, helpfulness, and capability retention.
Common mistakes