• Failure mode: “Your batch job fails overnight and stale predictions are served for 36 hours. How do you detect this, and what is your fallback?” — Tests monitoring and graceful degradation thinking.
• Rollout: “You are adding real-time inference to a system that is currently batch-only. How do you roll this out safely?” — Expect shadow mode, then canary, then gradual traffic shift with rollback triggers.
• Rollback: “After switching to real-time, latency spikes during peak. How do you roll back without losing freshness entirely?” — Strong candidates propose a hybrid fallback: serve cached batch predictions when real-time exceeds latency SLA.
• Measurement: “How do you prove that real-time inference is actually improving the business metric, not just the latency metric?” — A/B test batch-only vs batch+real-time cohorts, measuring downstream engagement or conversion.
• Cost: “Real-time inference is 5x more expensive per prediction. How do you justify the spend to leadership?” — Translate the freshness improvement into revenue impact with a cost-benefit analysis.
• Security/Governance: “Your real-time inference pipeline now processes PII in feature vectors at request time. What changes in your data governance posture?” — Feature-level access controls, encryption in transit, audit logging of feature access at serving time.

• Weak: “Batch is for offline and real-time is for online. You pick the one that fits.” — No trade-off reasoning, no cost awareness, no hybrid thinking.
• Strong: “The decision is a function of user activity ratio, model compute cost, and the business value of freshness. I would start with batch, measure the freshness gap’s impact on the business metric, and add real-time only where the ROI justifies the infrastructure cost.”

• Failure mode: “A feature with skew silently degrades model accuracy for 3 weeks before anyone notices. How do you design your monitoring to catch this sooner?” — Expect automated PSI checks on a daily cadence with alerting, not just dashboards.
• Rollout: “You are migrating 50 models to a new feature store to eliminate skew. How do you sequence the migration?” — Prioritize models by business impact and skew severity; run dual-write (old + new pipeline) with comparison during transition.
• Rollback: “After migrating to the feature store, one model’s accuracy drops. The feature store is computing the feature differently from the legacy pipeline. Which one is ‘correct’?” — The one that matches the training data is correct for the current model. If the feature store is more correct, retrain the model on feature-store-computed features before switching.
• Measurement: “How do you quantify the business impact of fixing training-serving skew?” — A/B test the skew-fixed model vs the skewed model and measure the business metric delta.
• Cost: “The feature store costs $200K/year. How do you justify this to a CFO?” — Calculate the revenue impact of the accuracy improvement from eliminating skew, plus the engineering hours saved from not debugging skew-related incidents.
• Security/Governance: “Feature values are logged for skew detection. Some features contain derived PII. How do you handle this?” — Hash or tokenize PII-derived features in the monitoring logs; restrict access to raw feature logs via role-based access control.

• Weak: “Training-serving skew is when the model gets different data in production. You fix it by testing more carefully.” — No mention of feature stores, no detection strategy, no concrete example.
• Strong: “Skew is the number one silent killer in production ML. I prevent it architecturally with a shared feature definition layer, detect it statistically with PSI monitoring, and catch the edge cases — like timezone skew — through code review of feature computation logic.”

• Quantization (INT8 or FP16): Typically 2x speedup, reducing batch latency from 250ms to roughly 125ms. Now 32/0.125 = 256 req/s per GPU. I need roughly 20 GPUs.
• TensorRT compilation (NVIDIA): Another 1.3-2x on top of quantization for NVIDIA GPUs. Potentially 350+ req/s per GPU, bringing the count to roughly 15 GPUs.

• Failure mode: “One of your 20 GPU instances has a hardware failure and the model fails to load on the replacement. You are now at 19 instances during peak traffic. What happens?” — Expect discussion of capacity headroom, health checks, model loading timeouts, and graceful degradation (serve from remaining instances with slightly higher latency via larger batches).
• Rollout: “You are deploying a new model version that is 30% more accurate but 50% slower. How do you roll it out without violating the latency SLA?” — Canary with latency-based rollback triggers; possibly deploy the new model on more powerful GPUs to compensate for the speed difference.
• Rollback: “The new model is live on all 20 instances and you discover it produces subtly wrong predictions for 5% of inputs. How fast can you roll back, and what is the blast radius?” — Model versioning in Triton allows instant rollback; blast radius depends on how long the bad model was live and whether predictions were cached downstream.
• Measurement: “How do you prove that the INT8 quantized model is not losing meaningful accuracy compared to the FP32 original?” — Run the quantized model on the full evaluation set and compare metrics. Also shadow-serve both models on production traffic and compare prediction distributions.
• Cost: “Leadership asks you to cut GPU costs by 50% without degrading quality. What levers do you pull?” — Migrate to cheaper GPU types (A10G instead of H100), increase batch sizes, add request-level caching for repeated inputs, prune the model, or use model routing to send simple requests to a cheaper model.
• Security/Governance: “Your model serving endpoint is exposed to the internet. What security considerations specific to ML serving are you worried about?” — Model extraction attacks (rate limit and monitor for systematic probing), adversarial inputs designed to cause maximum GPU compute (max-length inputs), and input validation to reject malformed feature vectors.

• Weak: “I would use more GPUs and a load balancer.” — No batching, no quantization, no cost reasoning, no math.
• Strong: “Starting from 5 req/s per GPU, I would layer dynamic batching (128 req/s), INT8 quantization (256 req/s), and TensorRT compilation (350+ req/s) to reach roughly 15-20 GPUs at $150-500K/year depending on GPU choice. I would also set up autoscaling with a warm pool for traffic spikes.”

• Data pipeline issue: Fix the pipeline, monitor for recurrence.
• Concept drift: Retrain with recent data. If fraud patterns have fundamentally changed, may need to add new features or update the model architecture. Consider online learning for faster adaptation.
• New fraud vector: The model was never trained to detect this pattern. Need new features and new labeled examples. In the short term, add rule-based detection for the specific pattern while retraining.
• Label quality issue: Work with the fraud team to catch up on investigations and verify labeling consistency.”

1. Shorten the retraining cycle. Move from monthly to weekly or daily retraining. This requires automating the full retraining pipeline — data extraction, feature computation, training, evaluation, deployment. The evaluation step is critical — automated retraining without automated quality checks is dangerous.
2. Add online learning. Maintain a lightweight model that updates with every new labeled example. This model captures the most recent fraud patterns faster than batch retraining. Use it as an ensemble member alongside the batch-trained model — the batch model provides stable baseline accuracy, and the online model provides responsiveness to new patterns.
3. Feature engineering for adversarial resilience. Add features that are harder for fraudsters to manipulate — behavioral biometrics (typing patterns, mouse movement), device fingerprinting, network graph features (connections between accounts), and sequential patterns (the order of actions, not just the actions themselves).
4. Rule-based augmentation. Work with the fraud investigations team to translate newly discovered patterns into rules that can be deployed in hours (a code change) rather than waiting for model retraining. The rules catch the specific new pattern; the model catches everything else.
5. Anomaly detection as a complement. Instead of only using a supervised model (which requires labeled fraud examples to learn), add an unsupervised anomaly detection model that flags transactions that are statistically unusual, regardless of whether they match known fraud patterns. This catches novel attack vectors that no supervised model has been trained on.”

• Failure mode: “The retraining pipeline produced a new model that accidentally increased false positives by 3x, blocking thousands of legitimate transactions before anyone noticed. How would you prevent this?” — Automated regression testing gates: new model must match or beat the current model on precision and recall on a held-out set before promotion.
• Rollout: “You have a new fraud model that catches 15% more fraud but also has 2% more false positives. How do you decide whether to deploy?” — Frame as a cost analysis: 15% more fraud caught = $X saved vs 2$Y in lost revenue and customer friction. Present the trade-off to stakeholders with the numbers.
• Rollback: “The new model is live and you discover it is blocking all transactions from a specific country due to a bias in the training data. How do you handle this?” — Immediate rollback to the previous model for that country (geographic routing), investigate the training data for geographic bias, add slice-based evaluation per geography to the CI/CD pipeline.
• Measurement: “How do you measure fraud detection performance when labels are delayed by 30-90 days?” — Use early proxy signals (manual review outcomes, customer complaints, transaction reversals) with explicit caveats about label lag. Track a “provisional recall” metric that gets revised as labels arrive.
• Cost: “Each manual fraud review costs $5 and you are routing 10,000 reviews/day to the human team. The ML model could auto-decide 60$30K/day. 2% error = 120 wrong decisions/day. Quantify the cost of each wrong decision (false positive = angry customer + possible churn; false negative = fraud loss) and compare.
• Security/Governance: “Regulators ask you to explain why a specific transaction was flagged as fraud. Your model is XGBoost with 500 features. How do you provide an explanation?” — SHAP values for the specific prediction, identifying the top 5 contributing features. Build an explanation pipeline that auto-generates human-readable justifications for flagged transactions.

• Weak: “The model is probably outdated. I would retrain it with more data.” — No investigation, no root cause analysis, no awareness that retraining without diagnosis might not fix the problem.
• Strong: “I would start with the data pipeline (most common, easiest to fix), then check for concept drift, then label quality, then analyze the missed fraud cases specifically. Remediation depends on root cause — a pipeline fix is hours, concept drift retraining is days, a new fraud vector requires new features and potentially weeks.”

• Classification/scoring tasks where the output is a single token or a small fixed set (yes/no, sentiment, category)
• Embedding tasks where the model produces a vector, not text
• Precomputed results where the LLM generates answers offline and they are served from a cache

• Failure mode: “Your self-hosted Llama model starts generating incoherent outputs after a GPU driver update. How do you detect this and what is your failover plan?” — Golden test set regression catches quality degradation; failover to API model via a routing layer while you investigate the infrastructure issue.
• Rollout: “You have fine-tuned a Llama 3.1 70B model that matches GPT-4 quality on your domain. How do you migrate 10M requests/day from the API to self-hosted without disruption?” — Shadow mode first, then canary (5% traffic), monitor quality with LLM-as-Judge, gradually increase to 100% over 4-6 weeks. Maintain API as fallback.
• Rollback: “Two weeks after full migration to self-hosted, a new GPT-4o release is 20% better on your benchmark. Do you switch back?” — Calculate the cost difference. If $70K/month savings from self-hosting outweighs the quality gap for your task, stay self-hosted. If the quality gap is business-critical, consider a hybrid: self-hosted for 80% of traffic, API for the 20% of hardest queries.
• Measurement: “How do you prove to the CEO that your self-hosted model delivers ‘GPT-4-level quality’ as requested?” — Define a task-specific evaluation rubric, run both models on 500 production-representative prompts, have domain experts blind-rate the outputs. Present win/loss/tie rates.
• Cost: “At 10M requests/day, your self-hosted model costs $50K/month. The CEO asks: 'Can we do it for$20K/month?’” — Explore: smaller model (8B with aggressive fine-tuning), more aggressive quantization (INT4), request caching for repeated queries, model routing (send 80% to the 8B, 20% to the 70B), or speculative decoding.
• Security/Governance: “Legal says customer data processed by the LLM feature must not leave our infrastructure. What does this mean for your architecture?” — Self-hosting becomes mandatory, not just a cost optimization. API models are eliminated unless the provider offers VPC-hosted deployments with contractual data processing agreements.

• Weak: “I would use GPT-4 behind an API and add caching.” — No cost awareness at 10M requests/day, no latency analysis, no consideration of self-hosting.
• Strong: “At 10M requests/day, API costs are $1.9M/month — self-hosting is 40x cheaper. But the 50ms latency target needs reframing because LLM generation is inherently sequential. I would clarify the task type, propose the right architecture for that task, and present the cost-quality-latency trade-off matrix to stakeholders.”

• Parsing: Different document types need different parsers — PDF (PyMuPDF or Unstructured.io for layout-aware parsing), HTML (BeautifulSoup with structure preservation), Confluence pages (API extraction with section hierarchy). The key insight: document structure (headings, sections, tables) must be preserved as metadata, not discarded during parsing.
• Chunking: I would use document-structure-aware chunking. Split by section headings, with a target chunk size of 500-1000 tokens and 100-token overlap. For tables and code blocks, keep them as atomic chunks regardless of size. For FAQ-style content, chunk by question-answer pair. Store the section heading hierarchy as metadata on each chunk (e.g., metadata: {section: 'Employee Benefits > Health Insurance > Dental'}).
• Embedding: Use a strong embedding model — OpenAI text-embedding-3-large (3072 dimensions) or Cohere embed-v3 for better multilingual support. For cost optimization at 50K documents, even the highest-quality embedding model costs less than $10 for initial embedding.
• Storage: pgvector if the team already uses PostgreSQL and 50K documents is the scale. If the knowledge base is expected to grow to millions of chunks or low-latency retrieval is critical, Pinecone or Qdrant.

• Daily document updates trigger re-parsing and re-embedding of changed documents only. Track document versions by hash. When a document changes, delete its old chunks and embed the new version. This avoids re-embedding the entire corpus daily.

• Query expansion: Optionally rewrite the user’s query using an LLM to improve retrieval. ‘How much PTO do I get?’ becomes ‘paid time off vacation days policy annual leave allowance.’ This significantly improves recall for queries that use different terminology than the documents.
• Hybrid search: Combine vector similarity search with BM25 keyword search. Weight them 70/30 (semantic/keyword) for general queries, but use metadata filters for structured queries (‘show me the latest engineering OKRs’ filters on document type and date).
• Reranking: Retrieve top 30 chunks, then rerank with Cohere Rerank or a cross-encoder model to get the top 5. Reranking is the single highest-leverage improvement to RAG quality — it typically improves answer quality by 15-25% over retrieval alone.
• Generation: Pass the top 5 chunks to the LLM with a system prompt that instructs it to answer only from the provided context, cite sources, and say ‘I don’t have enough information to answer this’ when the context does not cover the question.

• Build a golden test set of 100-200 question-answer pairs manually. Run RAGAS evaluations weekly on this test set to track retrieval recall, answer faithfulness, and answer relevance over time.
• Log every query, the retrieved chunks, and the generated answer. Use this for continuous quality improvement — identify queries where the system fails and use them to improve chunking, retrieval, or prompting.
• Monitor embedding model and LLM costs per query. At 50K documents and moderate query volume, costs should be manageable, but track them.

1. Table-aware parsing: Use a parser that preserves table structure (Unstructured.io’s table extraction, or a vision model like GPT-4o to parse table images into structured data). Convert each table to a structured format (JSON or Markdown table).
2. Row-level chunking with header context: For a benefits comparison table, each row becomes a chunk with the column headers prepended. So instead of ‘95% | $500 |$2,000’, the chunk becomes ‘Plan: Gold PPO | Coverage: 95% | Annual Deductible: $500 | Out-of-Pocket Max:$2,000 | Section: Employee Benefits > Health Insurance.’
3. Table summarization: Generate an LLM summary of each table that describes its contents in natural language. Store the summary as an additional chunk. This catches queries phrased in natural language (‘which plan has the lowest deductible?’) that would not match against row-level structured data.
4. Metadata tagging: Tag all chunks from this document with metadata that identifies them as tabular data. This allows the retrieval pipeline to apply table-specific handling (like returning complete tables rather than individual rows when the query is about comparing options).”

• Failure mode: “The ingestion pipeline breaks silently for 2 weeks. The knowledge base is stale but the system keeps answering with confidence. How do you detect this?” — Monitor the timestamp of the most recently indexed document. Alert if no new documents are indexed within a configurable threshold (e.g., 48 hours when daily updates are expected).
• Rollout: “You are replacing the company’s existing keyword search (Elasticsearch) with this RAG system. How do you roll it out without disrupting users?” — Run both systems in parallel. Show RAG results alongside search results. Collect user preference signals (which answer did they click?). Only fully replace when RAG demonstrates higher satisfaction on the metrics.
• Rollback: “A badly chunked document caused the RAG system to give incorrect answers about the company’s refund policy to 500 customers. How do you handle this?” — Immediate: remove the problematic chunks from the vector index. Short-term: identify affected interactions using retrieval logs and notify affected customers. Long-term: add a content validation step to the ingestion pipeline.
• Measurement: “The RAG system answers questions, but how do you know if it is better than having employees just search the wiki?” — A/B test: randomly assign support agents to RAG-assisted vs wiki-search workflows. Measure time-to-resolution, accuracy of answers, and agent satisfaction.
• Cost: “The embedding model and LLM costs are $5K/month for 50K documents. If the company grows to 500K documents, will costs scale 10x?” — Embedding costs scale linearly with document count (one-time per document update). LLM generation costs scale with query volume, not document count. Vector search costs depend on index size but sublinearly with good infrastructure choices.
• Security/Governance: “The knowledge base contains HR documents, financial reports, and engineering docs. Not all employees should see all answers. How do you implement access control in a RAG system?” — Metadata-based filtering at retrieval time: each chunk is tagged with access-control metadata (department, classification level). The query pipeline filters retrieval results based on the requesting user’s permissions before passing to the LLM.

• Weak: “I would chunk the documents, embed them, store in a vector database, and send the results to an LLM.” — No chunking strategy rationale, no reranking, no hybrid search, no evaluation plan.
• Strong: “The highest-impact decisions are chunking strategy (structure-aware, 500-1000 tokens with overlap), reranking (15-25% quality improvement over retrieval alone), hybrid search (catches exact matches semantic search misses), and query expansion (handles vocabulary mismatch). I would evaluate with a golden test set and track retrieval hit rate, faithfulness, and the ‘I don’t know’ rate as ongoing health metrics.”

• Read-only tools (no approval needed): Look up order status, check shipping tracking, retrieve customer account info, search knowledge base for policy information.
• Soft-write tools (automatic with limits): Create a support ticket, add internal notes to an account, send a pre-approved template email.
• Hard-write tools (require approval): Process refund, modify order, apply credit. These go through a confirmation step — either human approval or customer confirmation (‘I will process a refund of $49.99 to your card ending in 1234. Please confirm.’).

• Router: A lightweight classifier that categorizes incoming requests (order inquiry, refund request, technical issue, complaint). Routes to specialized prompt templates with appropriate tool access. A simple inquiry about order status does not need refund tools in scope.
• Agent loop: ReAct pattern with a 10-step limit. The agent reasons about the customer’s intent, calls the appropriate tools, and formulates a response. If the agent cannot resolve the issue in 10 steps, it escalates to a human agent with full context.
• Escalation triggers: Automatic escalation for: customer sentiment is negative (detected by sentiment analysis on the last 3 messages), the agent has called the same tool 3 times without resolving, the request involves legal language (detected by keyword matching), or the refund amount exceeds $500.

• Refund limits: The agent can process refunds up to $100 automatically.$100-500 requires supervisor approval (routed to a queue). $500+ escalates to human.
• Rate limiting: No more than 3 refunds per customer per month via the automated system.
• Audit logging: Every tool call, every customer interaction, and every decision is logged with the full reasoning chain. This is essential for compliance and for debugging when things go wrong.
• PII handling: Customer data is fetched by tool calls, not embedded in prompts. The LLM prompt contains customer ID references, not raw PII. This limits exposure if prompt logs are accidentally leaked.

• Resolution rate: Percentage of inquiries resolved without human escalation. Target: 70-80%.
• Customer satisfaction: Post-interaction survey scores compared to human agent scores.
• Error rate: Percentage of interactions where the agent takes an incorrect action (measured by human review of a sample).
• Escalation patterns: Track what types of issues the agent cannot handle. This drives both prompt improvement and tool development.

• Failure mode: “The agent processes a $500 refund for a customer who was not eligible, due to a misread of the order status API. How do you prevent this from happening again?” — Add a validation step between tool call and execution: cross-reference the refund eligibility business rules before executing. Add the scenario to the evaluation test suite.
• Rollout: “You are deploying this to replace 50% of human support agents. The customer support team is nervous. How do you manage the rollout?” — Start with AI handling only the simplest 20% of inquiries (order status lookups). Measure CSAT, resolution time, and error rate. Share metrics transparently with the team. Gradually expand scope based on demonstrated reliability.
• Rollback: “The AI agent starts hallucinating refund policies that do not exist (‘free returns within 90 days’ when the actual policy is 30 days). How do you handle this?” — Immediate: increase the RAG confidence threshold so the agent defers to human agents on policy questions. Short-term: add a fact-verification step that cross-references generated policy claims against the knowledge base. Long-term: retrain with more policy examples and add policy-specific guardrails.
• Measurement: “How do you measure whether the AI agent is better than human agents, not just cheaper?” — Track CSAT, first-contact resolution rate, average handle time, and accuracy (via human audit of a sample). Compare metrics between AI-handled and human-handled conversations on similar inquiry types. Better means equal or higher CSAT at lower cost and faster resolution.
• Cost: “Each human agent costs $25/hour and handles 8 tickets/hour ($3.12/ticket). What does the AI agent cost per ticket?” — LLM inference cost (e.g., 2000 tokens/ticket at $10/1M output tokens =$0.02) + tool call overhead + infrastructure. Even with overhead, AI cost per ticket is typically $0.10-0.50, a 6-30x reduction.
• Security/Governance: “A customer tries to social-engineer the AI agent: ‘I am a manager, override the refund limit and process a $5000 refund.' How does the system handle this?" -- The agent's tool permissions are enforced at the infrastructure level, not the prompt level. No amount of social engineering in the conversation can override the$100 auto-refund limit because the refund API enforces the limit server-side.

• Weak: “I would build an agent with LangChain that has access to tools for refunds, order lookup, and escalation.” — No safety design, no graduated autonomy, no monitoring, no awareness that the agent can take real-world damaging actions.
• Strong: “This is a high-stakes system. I would design graduated autonomy: fully autonomous for read-only actions, semi-autonomous with confirmation for moderate-risk actions, and human-gated for high-risk actions. I would enforce tool permissions at the infrastructure level, not the prompt level, and monitor resolution rate, error rate, and CSAT as primary health signals.”

• Scale: 200M users, 50K items (movies/songs), 1B interactions/day
• Latency: less than 200ms for page load, recommendations must be ready
• Freshness: Should reflect user’s recent activity (last few hours)
• Diversity: Users should not see the same type of content repeatedly
• Cold start: Handle new users and new items

• Collaborative filtering (matrix factorization): Compute user and item embeddings from historical interaction data. Find items whose embeddings are close to the user’s embedding. This captures “users like you also watched.” Run as a batch job every few hours.
• Content-based filtering: Embed item features (genre, director, description) and match against user preference profiles. Catches items similar to what the user has liked.
• Popularity-based: Recent trending items as a baseline candidate pool. Handles cold-start users who have no interaction history.
• Output: A pool of 200-500 candidate items per user, refreshed every few hours.

• A trained ranking model (typically a deep learning model like a Two-Tower model or DLRM) scores each candidate for the specific user at request time.
• Features include: user features (demographics, history embeddings), item features (genre, recency, popularity), context features (time of day, device, session history), interaction features (cross-features between user and item).
• The ranking model is trained on implicit feedback (clicks, watch time, completions) as the label. Optimized for engagement metrics (predicted watch time, completion probability).

• Apply diversity rules: no more than 3 items of the same genre in a row.
• Apply freshness boosts: recently released content gets a ranking boost.
• Apply business constraints: promoted content (contractual obligations) must appear in certain positions.
• Dedup: remove items the user has already seen or interacted with.

• Batch features: User embedding (updated daily), item popularity score (updated hourly), user genre preferences (updated daily).
• Streaming features: Session features (items viewed in the last 30 minutes), real-time engagement signals (updated every few seconds via Kafka + Flink).
• Feature store: Feast or Tecton. Batch features stored in a data warehouse and synced to Redis for serving. Streaming features computed in Flink and written directly to Redis.

• Candidate generation: Pre-computed and stored in a key-value store (DynamoDB, Redis). Per-user candidate lists are refreshed every 4-6 hours by a batch job.
• Ranking model: Served via Triton Inference Server with dynamic batching. The model is quantized (INT8) for latency. Multiple replicas behind a load balancer.
• Total latency: less than 50ms (candidate fetch from Redis: 5ms, feature fetch: 10ms, model inference: 20ms, re-ranking: 5ms).

• Offline evaluation: NDCG, recall@k on held-out interaction data.
• Online evaluation: A/B test every model change. Primary metrics: engagement (watch time, completion rate). Guardrail metrics: diversity (unique genres per session), coverage (percentage of catalog surfaced).
• Monitoring: Feature drift detection, prediction distribution monitoring, latency and throughput dashboards.

• Batch candidate generation vs real-time: Batch is cheaper and simpler but less responsive to recent behavior. The compromise: batch candidates + real-time ranking with session features.
• Engagement vs diversity: Optimizing purely for engagement leads to filter bubbles. The re-ranking stage enforces diversity constraints.
• Model complexity vs latency: A more complex ranking model may produce better recommendations but exceed the latency budget. Knowledge distillation can compress a complex model into a faster one that serves within budget.

• Failure mode: “The candidate generation batch job fails for 12 hours. Users see stale recommendations. How do you handle this?” — Serve from the last successful candidate list (staleness is better than empty). Alert on batch job failures. Have a lightweight fallback candidate generator (trending + popularity-based) that can run in real-time.
• Rollout: “You are deploying a new ranking model. How do you measure its impact?” — A/B test with the primary metric being engagement (watch time / completion rate) and guardrail metrics being diversity and catalog coverage.
• Rollback: “The new model increases watch time by 5% but reduces catalog diversity by 40%. Users are stuck in filter bubbles. Do you keep it?” — No. Rollback and add diversity constraints to the re-ranking stage, then re-deploy with diversity as a guardrail metric.
• Measurement: “How do you distinguish between ‘users clicked because the recommendation was good’ vs ‘users clicked because it was in position 1’?” — Position-bias correction in the training data. Models like propensity-weighted learning discount clicks in top positions.
• Cost: “The recommendation system uses 50 GPUs for real-time ranking. How do you reduce this?” — Distill the ranking model, reduce the candidate pool (200 instead of 500), cache rankings for users who visit frequently, and use cheaper GPU types (A10G instead of A100).
• Security/Governance: “A regulator asks: ‘Why did you recommend this content to a minor?’ How do you answer?” — SHAP-based explanations on the ranking model, plus audit logs showing the candidate generation and re-ranking steps. Age-gated content filtering in the re-ranking layer.

• Scale: 10,000 transactions/second, 500M transactions/day
• Latency: less than 100ms per transaction (must decide before the transaction is approved)
• Precision: False positives = blocked legitimate transactions = angry customers
• Recall: False negatives = missed fraud = direct financial loss
• Adaptability: Fraud patterns change weekly; the system must adapt quickly

• Hard rules that block obviously fraudulent transactions: card is on a known stolen list, transaction from a sanctioned country, amount exceeds card limit.
• Rules are fast, interpretable, and can be updated in minutes (a code deployment or config change). They catch known fraud patterns.

• A gradient-boosted tree model (XGBoost/LightGBM) for tabular data. Features include: transaction amount, merchant category, time since last transaction, geographic distance from last transaction, device fingerprint, historical fraud rate for the merchant, user’s spending pattern deviation.
• The model outputs a fraud probability. Transactions above a high threshold (e.g., 0.95) are blocked. Transactions between moderate and high thresholds (0.5-0.95) are routed to manual review. Transactions below the moderate threshold are approved.
• Why gradient-boosted trees, not deep learning? For tabular data with well-engineered features, GBTs consistently match or outperform deep learning while being 10-100x faster to train and serve. Speed matters when the latency budget is less than 100ms.

• Build a transaction graph connecting users, devices, merchants, and accounts. Use graph features: is this device connected to known fraud accounts? How many unique cards have been used from this IP in the last hour?
• Graph features are computed in near-real-time (updated every few seconds via streaming) and added to the ML model’s feature set.

• An isolation forest or autoencoder model trained on “normal” transaction patterns. Flags transactions that are statistically unusual, regardless of whether they match known fraud patterns. This catches novel attack vectors that the supervised model has never seen.

• Batch features: User’s historical fraud rate, average transaction amount (30-day), merchant fraud rate, behavioral embeddings.
• Streaming features (critical): Transaction velocity (number of transactions in the last 5/15/60 minutes), geographic velocity (distance traveled implied by last two transactions), amount deviation (how far this amount is from the user’s norm).
• Feature store: Streaming features written to Redis with TTLs matching their window. Batch features synced from the data warehouse to Redis daily.

• Fraud labels are delayed (chargebacks take 30-90 days). During this period, the model operates without ground truth.
• Early signals: manual review outcomes (days), customer complaints (hours), transaction reversals (days).
• The retraining pipeline runs weekly using the latest labeled data. When a new fraud pattern is detected by the investigations team, they can add it as a rule immediately (Layer 1) while waiting for enough labeled examples to retrain the ML model (Layer 2).

• Precision vs recall: The threshold between “block” and “review” determines the trade-off. A lower block threshold catches more fraud but blocks more legitimate transactions. The business decides the acceptable false positive rate.
• Latency vs feature richness: More features (especially graph and streaming features) improve accuracy but add latency. The 100ms budget must be distributed across feature fetch, model inference, and response.
• Adaptability vs stability: Frequent retraining catches new patterns but risks instability. Validate every new model against the current model before deploying.

• Failure mode: “The streaming feature pipeline (transaction velocity, geographic velocity) has a 5-minute outage. The ML model receives null features. What happens?” — If the model is not trained to handle null features gracefully, it may produce wildly inaccurate scores. Design: impute nulls with the user’s historical average (pre-computed batch feature), and add a “feature freshness” indicator feature so the model can learn to discount stale signals.
• Rollout: “You are adding a new graph-based feature to the fraud model. How do you validate that it improves detection without increasing false positives?” — A/B test: route 5% of transactions through the new model with the graph feature, 95% through the existing model. Compare precision and recall. Also shadow-score 100% of transactions with both models and compare.
• Measurement: “How do you calculate the dollar value of improving fraud recall by 2%?” — 2% of total fraud volume * average fraud transaction amount. If you process $1B/year and fraud is 0.5$5M), 2% improvement catches $100K more in fraud annually.
• Cost: “The real-time graph analysis adds 30ms of latency and requires a dedicated graph database cluster ($50K/year). Is it worth it?" -- If the graph features improve recall by 3$5M fraud problem, that is $150K/year in prevented fraud losses. The$50K/year infrastructure cost is justified by 3x ROI.
• Security/Governance: “PCI-DSS requires that cardholder data is encrypted in transit and at rest. How does this affect your ML pipeline?” — Feature computation must work on encrypted data or in a trusted execution environment. Feature values stored in Redis must be encrypted at rest. Model serving logs must not contain raw card numbers.

• Volume: 50,000 support tickets/day, 5,000 concurrent chat sessions
• Resolution: Target 70% automated resolution without human escalation
• Safety: Cannot provide medical, legal, or financial advice. Cannot disclose internal policies or customer data to unauthorized users.
• Integration: Access to order management system, knowledge base, account information

• Knowledge base: Company documentation, product FAQs, return policies, troubleshooting guides. Indexed in a vector database with document-structure-aware chunking.
• Tool access: Order lookup (read-only), account info (read-only), create ticket (write), process refund under $100 (write, with customer confirmation), escalate to human (write).
• Conversation management: Maintain conversation history within the session. For returning customers, retrieve previous ticket summaries from the CRM.

• Input filtering: Detect and reject prompt injection attempts. Detect PII in customer messages and handle appropriately.
• Output filtering: Scan generated responses for: inappropriate content, hallucinated information (claims not grounded in the knowledge base), unauthorized commitments (“I guarantee you will receive a full refund”), and accidental PII disclosure.
• Escalation triggers: Negative sentiment for 3+ consecutive messages, customer explicitly requests human, agent cannot resolve in 5 turns, high-stakes actions (account deletion, legal threats).
• Confidence threshold: If the RAG retrieval confidence is low (no highly relevant documents found), the agent should say “Let me connect you with a specialist” rather than guessing.

• Automated resolution rate: Percentage of conversations resolved without human intervention.
• Customer satisfaction (CSAT): Post-chat survey scores.
• Correctness auditing: Human review of 5% of automated conversations daily to catch errors.
• Latency: Time to first response (target less than 3 seconds), total resolution time.
• Cost per resolution: LLM API costs + infrastructure costs per resolved ticket vs. human agent cost.

• Failure mode: “The LLM generates a response that contradicts the actual refund policy. A customer acts on it and demands the incorrect refund. How do you handle this and prevent it?” — Immediate: honor the commitment made to the customer (the company made the error). Prevention: add a policy-verification guardrail that cross-references generated responses against the policy knowledge base before sending.
• Rollout: “How do you phase the transition from 100% human agents to 70% AI-handled?” — Phase 1: AI drafts responses, human agents approve/edit before sending. Phase 2: AI handles simple lookups autonomously, drafts for moderate complexity. Phase 3: AI handles moderate complexity autonomously with human review on a sample basis. Measure CSAT at each phase.
• Rollback: “CSAT drops 15% after full AI rollout. Leadership panics. What do you do?” — Immediately increase the human escalation rate (lower the confidence threshold for AI handling). Analyze which conversation types are causing the CSAT drop. Route those types back to human agents while improving the AI for the rest.
• Measurement: “How do you measure the true cost savings of the AI system, accounting for the overhead of building and maintaining it?” — Total cost = LLM inference + infrastructure + engineering time (building, maintaining, improving) + human agent cost for escalated and audit conversations. Compare to the counterfactual: human agent cost for all conversations at the same CSAT level.
• Security/Governance: “A customer pastes their full credit card number into the chat. How does the system handle this?” — PII detection on input: detect the card number pattern, redact it before it reaches the LLM, and inform the customer that sensitive information should not be shared in chat. Log the redacted version only.

• Scale: 100M queries/day, 10M items in the index
• Latency: less than 200ms end-to-end (query to results page)
• Relevance: Measured by click-through rate, conversion rate, and query abandonment rate
• Personalization: Results should be influenced by user history and preferences

• Spell correction, query expansion (synonyms, related terms), intent classification (navigational, informational, transactional).
• Example: “iphone cse” -> corrected: “iphone case”, expanded: “iphone case cover protector”, intent: transactional (purchase intent).

• Inverted index (Elasticsearch/Solr): BM25 keyword matching against the item index. Returns top 1000 candidates. Fast, well-understood, handles exact matches.
• Vector retrieval (ANN search): Embed the query and retrieve items with similar embeddings. Captures semantic similarity (query: “comfortable shoes for walking” matches items described as “ergonomic sneakers for all-day wear”).
• Combine via Reciprocal Rank Fusion: Merge the keyword and semantic candidate sets.

• A learned ranking model (LambdaMART, deep cross-network, or transformer-based) scores each candidate based on: query-item relevance features (BM25 score, semantic similarity), item quality features (click-through rate, conversion rate, reviews), user features (purchase history, browsing patterns), context features (time of day, device, location).
• The model is trained on click data with position-bias correction (users are more likely to click higher-ranked results regardless of relevance).

• Diversity: Ensure the top results are not all from the same brand or category.
• Freshness: Boost recently listed items.
• Ads integration: Insert sponsored results at designated positions.
• Personalization: Adjust ranking based on user purchase history (users who frequently buy running gear see running shoes ranked higher).

• Failure mode: “The vector retrieval component returns empty results for 5% of queries. What is happening?” — Out-of-vocabulary terms, queries in a language the embedding model does not handle well, or extremely niche queries with no semantic matches. The BM25 keyword retrieval should cover these cases — this is why hybrid search is essential.
• Rollout: “You are adding semantic search to a system that currently uses only keyword search. How do you validate it is an improvement?” — A/B test: 50% of users get keyword-only results, 50% get hybrid results. Measure click-through rate, conversion rate, and query abandonment rate. Also interleave results from both systems and use click data to compare relevance.
• Measurement: “How do you handle the position bias problem in your training data?” — Use inverse propensity weighting or regression-based debiasing. Items in position 1 get more clicks regardless of relevance. Without correction, the model learns to rank already-top-ranked items higher (a feedback loop).
• Cost: “The ranking model runs on 100 GPUs. How do you reduce this without degrading relevance?” — Distill the ranking model, reduce the candidate pool size (fewer items to rank), cascade with a cheaper first-pass ranker (score 1000 items with a simple model, then score the top 200 with the expensive model), and optimize with TensorRT.
• Security/Governance: “Your search results are being gamed by sellers who stuff keywords into product descriptions. How do you detect and handle this?” — Train a spam/manipulation classifier on known gaming examples. Demote items flagged as manipulative. Monitor for sudden ranking changes that correlate with product description edits.

• Scale: 500M posts/day (text, images, video)
• Latency: less than 500ms for text, less than 2s for images, less than 10s for video (content should not be visible to others before moderation)
• Accuracy: Extremely high precision for auto-removal (do not remove legitimate content). High recall for harmful content (do not miss dangerous content).
• Categories: Hate speech, violence, nudity, spam, misinformation, self-harm

• Compare content against known-bad content databases (PhotoDNA for CSAM, perceptual hashing for known violating images). Exact or near-exact matches are auto-removed immediately.

• Text classifier: Fine-tuned transformer model (e.g., fine-tuned Llama 3.1 8B) trained on labeled moderation data. Multi-label classification (content can be both hateful and violent).
• Image classifier: Vision model (ViT or similar) trained on labeled image datasets. Separate models for different violation types (nudity detection, violence detection, text-in-image detection).
• Video: Sample frames at regular intervals, run image classifiers on key frames. For audio, transcribe and run text classifiers.

• Cases where the classifier confidence is between 40-80% (too uncertain for auto-action) are sent to an LLM for nuanced understanding. The LLM receives the content plus the classifier’s initial assessment and makes a contextual judgment. Example: “This image shows violence” — is it a news article about a conflict (allowed) or a glorification of violence (removed)?

• Cases the LLM cannot confidently classify go to human moderators. Priority queue based on severity (potential CSAM is highest priority).

1. Training-serving skew. The features the model sees during training are different from what it sees during serving. This is the most likely cause and should be investigated first. Check: are the feature distributions in production the same as in the training data? Are there any features computed differently in the batch training pipeline vs the online serving pipeline? Common culprits: timezone handling, null value imputation, encoding differences between Python and Java/Go.
2. Data leakage in offline evaluation. The offline accuracy is inflated because the evaluation data was not properly separated from the training data. Check: was the train/test split done randomly (potential leakage for time-series data) or by time (correct for temporal data)? Are there features that inadvertently encode the label (e.g., using “cancellation date” to predict churn)?
3. Distribution shift between training data and production traffic. The production users are different from the users in the training data. Maybe the training data is from 6 months ago and user behavior has changed. Maybe the training data is from one geographic region and production serves globally. Check: compare the production feature distributions against the training data distributions.
4. Evaluation metric mismatch. The offline metric (accuracy on a balanced test set) does not reflect production reality (class-imbalanced, cost-sensitive). If the training data has 50% positive/negative but production has 95% negative, 95% “accuracy” offline might be meaningless because a model that predicts “negative” for everything would also get 95% on the production distribution.

• Failure mode: “The accuracy gap is not consistent — it is 95% on weekdays but 60% on weekends. What does this tell you?” — Likely a feature that encodes time-of-week behavior differently in training vs serving, or a population shift (weekend users are different from weekday users and underrepresented in training data).
• Rollout: “You have fixed the root cause (training-serving skew). How do you safely redeploy the fixed model?” — Shadow mode first, then canary with the accuracy metric as the rollback trigger. Compare production accuracy against the offline evaluation to verify the gap has closed.
• Rollback: “The fix involves changing how a critical feature is computed in the serving pipeline. But 10 other models also use this feature. How do you manage the blast radius?” — Coordinate with all dependent model owners. Deploy the feature change behind a feature flag. Roll out per-model with each model team validating their accuracy before the flag is enabled for them.
• Measurement: “After fixing the skew, production accuracy improves from 80% to 88% but not to 95%. Where is the remaining 7%?” — The remaining gap is likely distribution shift or overfitting. The offline 95% may have been inflated by an evaluation set that does not represent production traffic. Rebuild the evaluation set from production data to get a more realistic baseline.
• Cost: “The investigation took 2 engineers 3 weeks. How do you justify this to management and prevent it from happening again?” — Quantify the business impact of the 15-point accuracy gap over the months it went undetected. Propose automated monitoring that would catch the gap within days, not months.
• Security/Governance: “The investigation required accessing production feature logs containing user transaction data. Who should have access to these logs?” — Only the ML platform team and the investigating engineers, via time-limited access grants with audit logging. PII-containing features should be masked in diagnostic logs.

• Weak: “Probably overfitting. I would add more regularization and retrain.” — Jumps to a solution without investigation, misses skew and leakage as root causes.
• Strong: “The four most likely causes in order: training-serving skew, data leakage, distribution shift, and evaluation metric mismatch. I would investigate by comparing feature distributions, then label distributions, then slice-based accuracy, and finally check the temporal alignment of the evaluation set.”

​

Follow-up: You confirm it is training-serving skew in a specific feature. The feature is “average transaction amount in the last 30 days.” The training pipeline computes it correctly, but the serving pipeline is returning stale values (up to 12 hours old). How do you fix this?

1. Move the feature to streaming computation. Use Kafka + Flink to maintain a running average that updates with every new transaction. The feature is always current (within seconds). This is the ideal solution but requires streaming infrastructure.
2. Increase batch frequency. Compute the feature every hour instead of every 12 hours. This is the simplest fix — just change the Airflow schedule. Maximum staleness drops from 12 hours to 1 hour. Whether this is sufficient depends on how sensitive the model is to this feature’s freshness.
3. Compute a “freshness correction” at serving time. Store the batch-computed average AND the timestamp it was computed. At serving time, fetch any new transactions since the batch computation and recompute the average incrementally. This is a hybrid approach — the batch gives you the baseline, and the serving-time correction brings it up to date.

​

Follow-up: What if the model has hundreds of features and you suspect several might have skew? How do you systematically identify which ones?

• Prototype phase: Always start with API models. Fastest to iterate. Quality is highest. Cost is irrelevant at prototype scale.
• Production at low-moderate volume: Continue with API models unless privacy/compliance requires self-hosting.
• Production at high volume: Evaluate self-hosting with fine-tuned open-source models. The break-even is typically 10-50M tokens/day for the compute cost alone, but factor in 1-2 engineers’ time for ongoing maintenance.
• Specialized domain: Fine-tune an open-source model. A fine-tuned Llama 3.1 70B on your domain data can match or exceed GPT-4 for your specific task, at a fraction of the cost.

​

Follow-up: Your company is at 50M tokens/day and leadership wants to cut costs. Walk me through the migration from API to self-hosted.

• Current: 50M tokens/day at GPT-4o pricing = roughly $625/day input + roughly$2,500/day output = roughly $3,125/day = roughly$95K/month.
• Self-hosted: 8 H100 GPUs on AWS (p5.xlarge) = roughly $280/day = roughly$8.5K/month. Plus roughly 1 FTE engineer for maintenance (roughly $15K/month fully loaded).
• **Net savings: roughly $70K/month**, or roughly$840K/year.

• Failure mode: “You self-host and the GPU cluster has a 4-hour outage. All LLM features are down. What is your disaster recovery plan?” — Automatic failover to API model via the routing layer. This means maintaining API credentials and integration code even after full migration. The routing layer is the most critical component.
• Rollout: “How do you convince a risk-averse CTO to approve the migration from a reliable API to self-hosted infrastructure?” — Present it as a cost reduction project with a clear rollback plan. Show the $840K/year savings. Emphasize the parallel-running phase and the maintained API fallback. Frame it as “we are adding a cheaper option, not removing a working one.”
• Rollback: “Three months after migration, the open-source model’s quality falls behind a major GPT-5 release. How do you handle this?” — Route the quality-critical 20% of traffic back to the API for the new model. Keep the self-hosted model for the 80% where it is sufficient. Re-evaluate whether fine-tuning on the new base model closes the gap.
• Measurement: “How do you continuously track whether the quality gap between your self-hosted model and the frontier API model is widening or narrowing?” — Monthly automated evaluation using LLM-as-Judge on a standardized test set against the latest API model. Track the win/loss ratio as a time series.
• Cost: “GPU prices drop 40% next year due to new hardware. When do you refresh your fleet?” — When the cost savings from new GPUs exceed the migration cost (re-benchmarking, testing, deployment) plus the remaining depreciation on current hardware. Typically every 18-24 months for GPU-heavy workloads.
• Security/Governance: “Your legal team discovers that the open-source model’s training data may include copyrighted content from a plaintiff in an active lawsuit. What is your risk exposure?” — Consult legal immediately. Assess whether the model’s outputs in your use case could constitute copyright infringement. Consider switching to a model with clearer data provenance or a commercial license that includes indemnification.

• Weak: “Self-hosting is always cheaper so I would self-host.” — No volume analysis, no consideration of engineering cost, no privacy or compliance reasoning.
• Strong: “The crossover point is around 10-50M tokens/day depending on the model. But cost is only one dimension. I would start with API for rapid iteration, migrate to self-hosted when volume justifies it, fine-tune to close the quality gap, and maintain API access as a fallback and quality benchmark.”

• Offline store: Data warehouse (BigQuery, Snowflake, Redshift) or lakehouse (Delta Lake, Iceberg). Handles the heavy lifting of historical feature computation and point-in-time joins.
• Online store: Redis, DynamoDB, or Bigtable. Optimized for key-value lookups with predictable low latency.

​

Follow-up: What is a point-in-time join, and why is it so critical?

• Failure mode: “The online store (Redis) goes down. 200 models cannot fetch features. What happens?” — Every model serving endpoint needs a fallback: serve with default/cached feature values and degrade gracefully. This should be tested regularly with chaos engineering.
• Rollout: “You are rolling out the feature store to 200 models. How do you sequence the migration?” — Start with 5-10 non-critical models. Validate feature parity (feature store produces identical values to the legacy pipeline). Expand to critical models with dual-write verification. Full migration over 6-12 months.
• Rollback: “A feature store bug corrupts the online store. One critical model serves bad predictions for 2 hours. How do you handle this?” — Immediate: flip the model’s feature source to the legacy pipeline (which you kept running during the transition). Investigate the bug. Fix, validate, and re-migrate.
• Measurement: “How do you prove the feature store is worth the investment?” — Track: engineer hours saved per model onboarding (before: 2 weeks of feature pipeline work; after: 2 days of feature store configuration), reduction in training-serving skew incidents, and feature reuse rate (percentage of features consumed by >1 model).
• Cost: “Tecton costs $300K/year. Feast is free but requires 2 engineers to operate. Which do you choose?" -- Depends on whether you have the engineers and whether they have better things to work on. If 2 engineers at$200K/year fully loaded = $400K/year, Tecton is cheaper. If those engineers are already on the platform team and have capacity, Feast saves$300K.
• Security/Governance: “Features derived from PII (e.g., user spending patterns) are stored in the feature store. How do you handle data privacy?” — Feature-level access controls: only models with approved data processing agreements can access PII-derived features. Encrypt PII features at rest and in transit. Maintain an audit log of which models accessed which features.

• Weak: “A feature store is just a database for features. I would use Redis.” — Misses metadata management, point-in-time correctness, and the training-serving consistency guarantee.
• Strong: “A feature store solves three problems: feature reuse, training-serving skew, and point-in-time correctness. The dual offline/online architecture is essential — the offline store for historical training data with asof joins, the online store for low-latency serving. For 200 models, I would buy (Feast or Tecton) and build custom only if we outgrow the tooling.”

• M (number of connections per node): Higher M = higher recall but more memory and slower build. Default 16, range 8-64.
• ef_construction (build-time search width): Higher = better graph quality but slower build. Default 200.
• ef_search (query-time search width): Higher = higher recall but slower queries. This is the primary recall/latency knob at query time. Start at 50, increase until recall meets your target.

• nlist (number of partitions): More partitions = finer granularity. Typically sqrt(N) to 4*sqrt(N) where N is the dataset size.
• nprobe (partitions searched at query time): More probes = higher recall but slower queries. The primary recall/latency knob. Start at 10, increase until recall meets your target.

​

Follow-up: You have 500 million vectors and a 5ms latency budget. Walk me through your index design.

1. Product Quantization: Compress each 1536-dim vector into roughly 192 bytes (split into 192 sub-vectors, quantize each to 1 byte). This reduces storage from 3TB to roughly 96GB — fits in memory.
2. IVF with roughly 22,000 partitions (sqrt(500M) is roughly 22,360). Each partition contains roughly 22,000 vectors on average.
3. nprobe = 32-64: Search 32-64 partitions per query. At 22,000 vectors per partition with PQ distance computation, this is fast enough for the 5ms budget.
4. Expected recall: Roughly 92-95% at nprobe=64 with PQ. If the application requires higher recall, add a re-ranking step: retrieve top 200 candidates with IVF-PQ (approximate), then re-rank using exact distance computation on the original (uncompressed) vectors. This bumps recall to roughly 98-99% with a small latency increase.
5. Infrastructure: FAISS with GPU acceleration (FAISS-GPU supports IVF-PQ natively and provides 5-10x speedup over CPU). Alternatively, Milvus or Zilliz (which use GPU-accelerated FAISS internally) as a managed solution.

• Failure mode: “The HNSW index becomes corrupted after a node crash mid-write. How do you recover?” — Rebuild the index from the stored vectors (HNSW indices are derived data, not primary data). This takes hours for large indices — during recovery, fall back to a replica or a stale snapshot.
• Rollout: “You are migrating from pgvector (10M vectors, 50ms p99) to a dedicated vector database (targeting 5ms p99). How do you execute the migration?” — Dual-write during transition: write to both pgvector and the new database. Compare results for a random sample of queries. Once quality and latency are validated, switch reads to the new database. Decommission pgvector reads.
• Rollback: “After migration, the new vector database has intermittent timeout spikes. How do you handle this while maintaining the service?” — Keep pgvector as a warm fallback. Route traffic back to pgvector when the new database times out. Use a circuit breaker pattern.
• Measurement: “How do you benchmark whether HNSW or IVF is better for your specific workload?” — Run both on your actual data with your actual queries. Measure recall@10, p50/p99 latency, and memory usage. Sweep the tuning knobs (ef_search for HNSW, nprobe for IVF) and plot the recall-latency Pareto frontier for each. The “better” algorithm is the one that achieves your target recall at lower latency.
• Cost: “HNSW needs 800GB of RAM for 100M vectors. Each machine has 256GB. Do you shard across machines or switch to IVF-PQ?” — Options: (1) Shard HNSW across 4 machines with scatter-gather search. (2) Switch to IVF-PQ which compresses to ~200GB. (3) Use a managed vector database that handles sharding for you. The decision depends on operational capability and whether the latency of cross-machine scatter-gather is acceptable.
• Security/Governance: “The vectors encode semantic meaning from customer documents. Can an attacker reconstruct the original documents from the embeddings?” — Current research suggests partial reconstruction is possible for some embedding models. If document confidentiality is critical, consider: encrypting vectors at rest, restricting API access to the vector search to authenticated services, and monitoring for bulk export patterns that could indicate extraction attempts.

• Weak: “HNSW is better because it is faster.” — No understanding of when IVF is appropriate, no mention of memory trade-offs, no tuning knobs.
• Strong: “HNSW is the default choice for <100M vectors in-memory: highest recall, sub-millisecond latency. IVF-PQ is the choice for 100M+ vectors when memory is constrained. The tuning knobs are ef_search (HNSW) and nprobe (IVF) — both trade recall for latency. At 500M vectors, I would use IVF-PQ with product quantization, re-ranking on uncompressed vectors for the top 200 candidates.”

• Pros: Reproducible (same data = same model). Easier to validate (full evaluation on holdout set before deployment). Simpler infrastructure (a scheduled job, not a continuous system). Easier to debug (you can inspect the exact training data and the exact model).
• Cons: The model is always behind — it does not know about anything that happened since the last retraining. For a model retrained daily, the worst-case staleness is 24 hours.
• Best for: Domains that change slowly (product recommendations, content ranking), tasks where stability is critical (healthcare, finance), teams without streaming infrastructure.

• Pros: The model adapts to changes in real-time. Crucial for adversarial domains (fraud detection, ad click prediction) where the targets actively evolve.
• Cons: Harder to reproduce (the model is constantly changing). Vulnerable to poisoning (a burst of bad data can corrupt the model quickly). Requires streaming infrastructure. Harder to validate (you cannot do a full holdout evaluation because the model is changing).
• Risks: Catastrophic forgetting (the model “forgets” old patterns as it absorbs new ones). Feedback loops (the model’s predictions influence the data it trains on, creating a self-reinforcing cycle).
• Best for: Adversarial domains (fraud, ad click prediction), highly dynamic domains (real-time bidding, dynamic pricing), applications where even hourly staleness causes measurable business impact.

• Batch-retrained primary model: Retrained daily or weekly with full validation.
• Online-updated secondary model: A lightweight model that updates continuously and captures recent patterns.
• Ensemble at serving time: Combine the predictions of both models. The batch model provides stability; the online model provides freshness. The ensemble weights can be tuned to favor one or the other.

​

Follow-up: Online learning sounds fragile. How do you prevent bad data from corrupting the model?

1. Data validation gates. Before any data point is used for model update, validate it. Check for: missing features, out-of-range values, impossible feature combinations (e.g., a 3-year-old making a $10,000 purchase), and anomalous label distributions (if the fraud rate suddenly jumps from 1% to 20% in the last hour, hold the data until verified).
2. Learning rate decay and momentum. The online learning rate should be low enough that a single bad data point has minimal impact. Use exponential moving averages or similar momentum techniques so the model’s weights change slowly. A burst of 100 bad data points should shift the model slightly, not dramatically.
3. Model quality monitoring with automatic rollback. Continuously evaluate the online model on a held-out validation set. If performance drops below a threshold (or below the last batch-trained model’s performance), automatically revert to the last known-good model and pause online updates until the data quality issue is resolved. This is the most important safeguard — it limits the blast radius of data quality problems.

• Failure mode: “A data poisoning attack feeds 10,000 fake fraud labels into your online learning system over 2 hours. The model starts flagging legitimate transactions as fraud. How do you detect and recover?” — The automatic rollback via held-out validation set catches the quality drop. Pause online updates, revert to the last known-good checkpoint, investigate the data quality issue, and add anomaly detection on the label stream (sudden label rate changes trigger a hold).
• Rollout: “You currently retrain weekly. Management wants to move to online learning for faster adaptation. How do you phase this in?” — Phase 1: Increase batch frequency to daily (quick win). Phase 2: Add an online-learning model as a secondary scorer alongside the batch model, logging its predictions but not using them for decisions. Phase 3: Add the online model to the ensemble with low weight. Phase 4: Gradually increase the online model’s weight as you build confidence in its stability.
• Rollback: “The online model has drifted significantly from the batch baseline over 2 weeks. How do you ‘reset’ without losing recent learnings?” — Retrain the batch model on the full historical dataset including the last 2 weeks. This captures the recent patterns the online model learned but in a more stable, reproducible framework. Resume online learning from the new batch baseline.
• Measurement: “How do you measure whether online learning is actually providing value over daily batch retraining?” — A/B test: one cohort gets predictions from the batch-only model (retrained daily), the other gets predictions from the hybrid (batch + online). Measure the business metric (e.g., fraud detection recall, conversion rate) for both cohorts. The delta is the value of online learning’s freshness.
• Cost: “Online learning requires streaming infrastructure running 24/7. The batch retraining job costs $500/day. Is online learning worth the infrastructure investment?" -- Quantify the business impact of the freshness improvement (e.g., online learning catches fraud 4 hours faster on average, preventing$X/day in fraud losses). Compare that revenue impact against the streaming infrastructure cost.
• Security/Governance: “An auditor asks: ‘What model was making decisions at 3:47 PM on March 15th?’ With online learning, the model is constantly changing. How do you answer?” — Continuous model checkpointing with timestamps. Every model update is logged with the model state, the data that triggered the update, and the timestamp. You can reconstruct the exact model state at any point in time.

• Weak: “Online learning is better because it is more up-to-date. I would use online learning for everything.” — No awareness of stability risks, poisoning vulnerability, or reproducibility challenges.
• Strong: “The right approach is usually a hybrid: batch-retrained primary model for stability and reproducibility, online-updated secondary model for freshness. The online model needs three layers of defense — data validation gates, low learning rate with momentum, and automatic quality-triggered rollback to the last batch model.”

• For customer support: automated resolution rate, CSAT scores, escalation rate.
• For RAG: answer correctness (human-judged on a sample), retrieval hit rate, “I don’t know” rate.
• For code generation: test pass rate, compilation success rate.
• These measure whether the system is achieving its purpose, not just whether the LLM is producing text.

• Run a judge LLM on a random sample (1-5%) of production outputs, scoring on: relevance (does the answer address the question?), correctness (is the information accurate?), safety (does the output violate any guidelines?), format compliance (is the output in the expected format?).
• Track these scores as time series. An alert fires if any score drops below a threshold or trends downward over a rolling window.

• Response length distribution (sudden changes may indicate prompt/model issues).
• Refusal rate (“I can’t help with that” responses — should be stable).
• Latency distribution (time to first token, total generation time).
• Error rate (API failures, timeout, parsing failures for structured output).
• Token usage (cost monitoring).

• Weekly review of 100-200 production outputs by domain experts on a detailed rubric.
• This catches quality issues that automated metrics miss (subtle incorrectness, tone problems, off-brand responses).

​

Follow-up: You suspect the API model provider updated their model and it is producing lower-quality outputs. How do you verify and what do you do?

1. Run your golden evaluation set (the 200 curated question-answer pairs) against the current model. Compare scores against the last known-good baseline. If quality has dropped significantly, you have confirmation.
2. Check the provider’s changelog or status page. Some providers announce model updates; others do not.
3. Compare outputs side-by-side. Pull 50 production queries from before and after the suspected change date. Have domain experts blind-rate the outputs. If “after” outputs are consistently worse, the model changed.

1. Short term: If the provider offers model version pinning (e.g., gpt-4o-2025-05-13 instead of gpt-4o), pin to the last known-good version. Not all providers support this.
2. Medium term: If your prompts relied on specific model behaviors that changed, update the prompts. Sometimes a model update requires prompt adjustments because the model interprets instructions differently.
3. Long term: This is a strong argument for having a self-hosted model as a fallback. If your API provider degrades, you can route traffic to your self-hosted model while you resolve the API issue. It is also an argument for vendor diversification — test your system against multiple providers so you can switch.

• Failure mode: “Your LLM-as-Judge itself has been silently upgraded by the provider, and it is now scoring outputs differently. Your quality metrics shift, but the production model has not changed. How do you detect this?” — Maintain a “judge calibration set” — 100 outputs with known human quality scores. Run the judge against this set weekly. If judge scores diverge from human scores, the judge has changed, not the production model.
• Rollout: “You are adding LLM-as-Judge monitoring to an existing production system that has no evaluation infrastructure. What is the quickest path to value?” — Week 1: Create 50 golden test cases from production queries with expert-verified answers. Week 2: Run them daily against the live system. Week 3: Add LLM-as-Judge on 2% of production traffic scoring relevance and faithfulness. This gives you basic coverage within 3 weeks.
• Rollback: “Your golden test set scores drop 20% overnight but user feedback has not changed. Which signal do you trust?” — Investigate both. The golden test set may have drifted (the “correct” answers may be outdated). Or user feedback may be lagging. Compare the specific failing test cases against recent production outputs. Trust the signal that aligns with domain expert review of the actual outputs.
• Measurement: “How do you build a business case for investing in LLM evaluation infrastructure? It does not directly generate revenue.” — Frame it as insurance: “Without evaluation, we are flying blind. A silent quality degradation that reduces customer satisfaction by 5% for 3 months costs $X in churn. The evaluation infrastructure costs$Y/year. The ROI is the expected value of quality incidents prevented.”
• Cost: “LLM-as-Judge on 5% of traffic costs $3K/month in API calls. Can you reduce this without losing coverage?” — Sample strategically: evaluate a random sample plus all low-confidence outputs, all outputs that received negative user feedback, and all outputs from recently changed prompt versions. This covers the highest-risk traffic at lower total volume.
• Security/Governance: “Production LLM outputs are being sent to a different LLM provider for judge evaluation. Does this create a data privacy concern?” — Yes. If the production outputs contain PII or confidential information, sending them to a third-party judge LLM may violate data processing agreements. Options: self-host the judge model, use the same provider for both production and judge, or redact PII before sending to the judge.

• Weak: “I would check the outputs manually and see if they look right.” — No scalable evaluation strategy, no automated metrics, no monitoring.
• Strong: “I would build a three-tier evaluation stack: task-specific success metrics (the business outcome), LLM-as-Judge on a continuous sample (automated quality signal), and periodic human evaluation (ground truth calibration). Degradation detection uses golden test set regression, output distribution anomaly detection, and user feedback trend analysis.”

• Web crawling (Common Crawl), curated datasets (Wikipedia, StackOverflow, ArXiv), books, code repositories (GitHub), and licensed data.
• Scale: Llama 3 was trained on roughly 15 trillion tokens. Collecting and managing this volume is a significant data engineering problem.

• Deduplication: Both exact dedup (hash-based) and near-dedup (MinHash/LSH-based). Training on duplicated data causes the model to memorize rather than generalize. Llama 3’s technical report describes extensive deduplication as critical to quality.
• Quality filtering: Remove low-quality text (gibberish, spam, boilerplate, auto-generated content). Common approaches: perplexity-based filtering (use a small language model to score text quality), heuristic rules (minimum length, language detection, formatting checks), and classifier-based filtering (train a quality classifier on curated high/low quality examples).
• Toxicity filtering: Remove hateful, violent, or otherwise harmful content. Use toxicity classifiers (Perspective API, custom models) to score and filter.
• PII removal: Strip personally identifiable information (names, emails, phone numbers, addresses) to prevent the model from memorizing and regurgitating PII. Challenging at scale — rule-based NER is fast but misses context-dependent PII.

• Too much web text, not enough code -> weak at coding
• Too much English, not enough multilingual -> poor multilingual performance
• Too much recent data, not enough older knowledge -> knowledge gaps
• Data mixing is an active research area. Llama 3 uses a carefully tuned mix of web, code, math, science, and multilingual data with different sampling weights per source.

• The dataset must be sharded across the training cluster (each node reads a different shard).
• Shuffling must be thorough — if data is ordered by source (all Wikipedia, then all web crawl), the model learns in biased epochs. Global shuffling across the entire dataset is ideal but expensive at TB scale.

1. Benchmark contamination: If the training data contains the exact questions and answers from evaluation benchmarks (MMLU, HumanEval), the model’s benchmark scores are inflated and do not reflect real capability. Decontamination requires detecting and removing benchmark data from the training set — a non-trivial NLP problem.
2. Copyright issues: Training on copyrighted content without permission is an active legal minefield. Several ongoing lawsuits (New York Times v. OpenAI, Getty Images v. Stability AI) are shaping the legal landscape.
3. Data provenance tracking: At 15 trillion tokens from hundreds of sources, tracking where each piece of data came from is essential for debugging, compliance, and legal purposes — but extremely challenging to implement.
4. Temporal cutoff issues: The training data has a natural cutoff date. Events after that date are unknown to the model. RAG or retrieval integration is needed for up-to-date information.

• Failure mode: “A bug in the deduplication pipeline causes 30% of the training data to be duplicated. The model is trained for 2 weeks before anyone notices. What is the impact and how do you detect this?” — The model over-memorizes the duplicated content, producing lower diversity outputs and potentially regurgitating training data verbatim. Detection: monitor output diversity metrics and memorization tests (prompt the model with training data prefixes and check if it completes them verbatim).
• Rollout: “You are updating the training data for the next model version. How do you validate that the new data mix improves the model without training the full model (which costs $10M)?” — Train smaller proxy models (1B parameter) on different data mixes and evaluate on your benchmark suite. The relative performance of data mixes at small scale is a strong predictor of relative performance at large scale. This is “data mix ablation” and costs 100x less than full-scale training.
• Rollback: “Post-training, you discover that a licensed data source was included despite the license not permitting AI training. What do you do?” — Consult legal immediately. Quantify the proportion of training data from that source. If it is a small fraction, the legal risk may be manageable with a licensing negotiation. If it is significant, you may need to retrain without that data — a multi-million dollar decision.
• Measurement: “How do you measure whether adding a new data source (e.g., medical literature) actually improves the model’s medical knowledge without degrading general performance?” — Add the data source and train a proxy model. Evaluate on medical benchmarks (MedQA, PubMedQA) and general benchmarks (MMLU, HumanEval). The new source should improve medical scores without regressing general scores by more than a threshold.
• Cost: “Web crawling and processing 15T tokens costs $500K in compute. How do you optimize this?” — Incremental crawling (only crawl new/changed pages), aggressive early-stage filtering (discard low-quality content before expensive processing steps), and caching intermediate results so that data mix experiments do not require re-processing raw data.
• Security/Governance: “Your training data includes code from public GitHub repositories. Some repos contain hardcoded API keys and secrets. How do you handle this?” — Add a secret scanning step to the data cleaning pipeline (tools like GitLeaks, TruffleHog). Remove or redact detected secrets before training. Without this, the model may memorize and regurgitate API keys.

• Weak: “Collect data from the internet, clean it, and train.” — No mention of deduplication, quality filtering, data mixing, decontamination, or legal concerns.
• Strong: “The pipeline has five critical stages: collection, cleaning (dedup + quality + toxicity + PII), mixing (the most impactful quality lever), tokenization, and sharding. The common pitfalls are benchmark contamination, copyright issues, and data provenance — all of which have caused real problems for real companies.”