Most machine learning models never make it to production. Many of those that do are quietly abandoned within the first year.
Why? Because training a model on a laptop is fundamentally different from running it in production where it needs to serve 1,000 predictions per second with less than 100ms latency, handle data drift, scale automatically, and cost less than $5,000/month.
After 18 years of building production software — including automation that now handles 85% of insurance claims for Argo Global — we've learned that production ML is 10% data science and 90% software engineering.
Key Insight
This guide is for data scientists who need to deploy their models and software engineers who need to understand ML operations. It's not about training better models—it's about getting those models working reliably in the real world.
1. The Training vs. Production Gap
Training Environment
- ✓Jupyter notebook on laptop
- ✓Clean, labeled dataset (CSV)
- ✓One-time batch processing
- ✓Optimize for accuracy
- ✓No latency requirements
- ✓Experiment freely
- ✓Cost: $0 (local machine)
Production Environment
- ⚠Distributed system (Kubernetes/ECS)
- ⚠Real-time, messy, unlabeled data
- ⚠Continuous predictions (24/7)
- ⚠Balance accuracy vs. latency vs. cost
- ⚠Less than 100ms response time required
- ⚠Handle 10K+ requests/second
- ⚠Cost: $500-$50,000/month
The Reality Check
A 500-line train.py script transforms into a 10,000+ line production ML system with data pipelines, feature stores, model serving, monitoring, A/B testing, and rollback capabilities.
2. Why Most ML Projects Fail in Production
Here are the top 10 reasons ML projects fail in production:
1. No Clear Business Metric
Problem: Trained for 95% accuracy but business needed less than $50K annual cost
Solution: Define business metrics upfront, not just ML metrics
2. Data Not Available in Production
Problem: Model trained on "Last 30 days transaction history" not available for new users
Solution: Only use features available at prediction time
3. Training/Serving Skew
Problem: Different preprocessing in training vs. serving
Solution: Same preprocessing pipeline for both
4. No Monitoring
Problem: Accuracy degraded over 6 months, nobody noticed until customers complained
Solution: Continuous monitoring of predictions and metrics
5. Can't Handle Scale
Problem: Model works fine for 10 predictions/second, production needs 1,000/second
Solution: Load testing, optimization, caching
6. Data Drift
Problem: Model trained on 2023 data, user behavior changed in 2024
Solution: Monitoring and retraining pipelines
7. No Rollback Strategy
Problem: New model performs worse, no way to rollback
Solution: Versioning, gradual rollout, quick rollback
8. Unrealistic Latency
Problem: Deep learning model: 2 seconds, requirement: less than 100ms
Solution: Model optimization, distillation, or simpler model
9. Cost Explosion
Problem: Cloud inference costs $20K/month, business case was $2K/month
Solution: Cost modeling upfront, optimization
10. No Ownership
Problem: Data scientist moved to next project, engineers don't understand ML
Solution: Clear ownership, documentation, cross-training
3. Production ML Architecture Patterns
A complete production ML system includes much more than just a model. Here's the reference architecture we use:
┌─────────────────────────────────────────────────────┐
│ DATA SOURCES │
│ Database | API | Event Stream | File Upload │
└────────────────────┬────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ DATA PIPELINE │
│ • Data validation │
│ • Cleaning & transformation │
│ • Feature engineering │
│ • Apache Airflow / AWS Step Functions │
└────────────────────┬────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ FEATURE STORE │
│ • Precomputed features │
│ • Online (Redis) + Offline (S3/Snowflake) │
│ • Consistent train/serve features │
└────────────────────┬────────────────────────────────┘
│
┌───────────┴───────────┐
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ TRAINING │ │ SERVING │
│ │ │ │
│ • Pull data │ │ • REST API │
│ • Train model│ │ • Load model │
│ • Validate │ │ • Predict │
│ • Register │ │ • <100ms SLA │
└──────┬───────┘ └──────┬───────┘
│ │
▼ │
┌──────────────┐ │
│ MODEL │ │
│ REGISTRY │◄───────────────┘
│ (MLflow) │
└──────┬───────┘
│
▼
┌──────────────────────────────────────────────┐
│ MONITORING & OBSERVABILITY │
│ • Prediction distribution │
│ • Feature drift │
│ • Model performance metrics │
│ • System metrics (latency, errors) │
└──────────────────────────────────────────────┘Offline Components
- • Training pipelines
- • Feature computation
- • Model evaluation
- • Historical data
Online Components
- • Model serving API
- • Feature store (Redis)
- • Real-time predictions
- • Low-latency reads
Monitoring
- • Prediction logging
- • Drift detection
- • Performance metrics
- • Alerting
14. Production ML Checklist
Before Production
After Production
Frequently Asked Questions
How often should I retrain my model?
Monitor performance. Retrain when accuracy drops >5% OR on a schedule (monthly/quarterly). Some models need weekly retraining (e.g., fraud), others are stable for months.
What latency is acceptable?
User-facing: <100ms. Batch: doesn't matter. Backend services: <500ms. Target: As fast as possible without sacrificing too much accuracy.
Should I use deep learning for production?
Only if you need it. XGBoost/LightGBM is often better: faster, smaller, easier to debug. Use DL only if accuracy gain is worth the complexity.
How do I handle missing features in production?
Imputation (fill with median/mean), or train model to handle missing values explicitly. NEVER skip predictions due to missing features.
What if my model predictions are cached and get stale?
Set appropriate TTL (time-to-live). For fraud: 1 hour. For recommendations: 24 hours. For static content: 7 days.
LTK Soft Team
Senior engineers delivering production software since 2008. We specialize in taking models and automation from prototype to production — with the monitoring and engineering to keep them there.
Work With Us