AI Training · Fine-Tuning & LLMOps

Fine-Tuning & LLMOps Training

Your team can kick off a fine-tuning job on a managed platform in an afternoon. That still doesn't tell you whether the result beats the base model or how to roll it back if it doesn't. This hands-on training covers the full lifecycle. Your team decides when adaptation is worth the cost, builds the training and evaluation datasets, runs supervised and parameter-efficient fine-tuning, then sets up deployment, monitoring, and rollback. You leave with a reference pipeline and an evaluation suite for your own task.

modules: 6
hours: 14

Fine-tuning is easy to start and hard to trust. You can kick off a job on a managed platform and wait for it to finish. A few prompts that "sound better" can still tempt you to call it done. Production model adaptation asks more of you. The dataset must teach the target behavior, and the training run needs understandable hyperparameters. An evaluation set must then show whether the result improves before it reaches users.

This is an engineering program, not a strategy briefing. Over two days, your AI and platform team works the full lifecycle on a slice of your own task. You curate and format a training dataset, run supervised and parameter-efficient fine-tuning (LoRA, QLoRA) with explainable hyperparameters, and build a golden evaluation set that catches catastrophic forgetting.

A full module covers what production actually demands after the training run finishes: model registry and versioning, canary rollout against the incumbent model, plus monitoring and rollback. We draw the boundary early. This training doesn't cover corpus and retrieval design for RAG or stand-alone prompt engineering. It also makes no vendor performance guarantees. Those are different problems, and we'll tell you up front if your task is better solved with RAG training instead of fine-tuning.

The syllabus adapts to your task and data within your constraints. The AI training programs catalog covers the adjacent skills. You leave with a reference pipeline and an evaluation suite your team keeps running against your own models after the trainers leave.

Often the wrong first move

Teams reach for fine-tuning before trying a better prompt or a retrieval approach, or avoid it long after it would clearly pay off. The decision needs a framework built around cost and data readiness, plus a clear view of what fine-tuning can and cannot fix.

The training data is the real project

Hyperparameters get the attention, but curation, deduplication, and label quality decide whether a fine-tuned model learns the target behavior. Most failed fine-tuning runs trace back to the dataset. The training configuration is rarely the underlying problem.

No eval harness means no way to know it worked

Teams that compare a handful of outputs by eye cannot tell a genuine improvement from a model that got better on the examples they happened to check and worse everywhere else, including tasks nobody thought to test.

Deployment is a different problem than training

A successful training run is not a production system. Model registry, versioning, canary rollout, and a rollback plan are separate work that most teams only start thinking about after the first bad deploy.

Nobody is watching cost and drift once it's live

A fine-tuned model can quietly degrade as data drifts or the task shifts. A missed retraining window causes the same problem. Teams often notice only after someone downstream complains. Production LLMOps means monitoring for that before it happens.

01
Adaptation landscape: when fine-tuning is the right call
90 minBeginner
Where fine-tuning earns its cost, and where a better prompt or a retrieval approach already does the job. This session sets a shared, hype-free vocabulary. It builds the decision framework before your team commits to a training run.
What fine-tuning changes in a model, and what it cannot fix
Fine-tuning vs. prompting vs. retrieval: when each is the right call
A decision framework for whether adaptation is worth the cost
Full versus parameter-efficient tuning
Anti-use-cases where a better prompt solves it instead
Reading provider fine-tuning docs without the marketing gloss
02
Building the training and evaluation datasets
150 minIntermediate
The unglamorous work decides everything downstream. Your team sources and curates examples that teach the target behavior, formats them correctly and holds out an evaluation set before any training run starts.
Sourcing and curating examples that teach the target behavior
Formatting datasets for supervised fine-tuning across prompt-completion, chat and preference pairs
Deduplication and quality filtering that catches label noise before it trains in
Train, validation, held-out splits
Handling licensing and PII in confidential training data
Synthetic data generation and where it quietly fails
03
Fine-tuning techniques across SFT, PEFT and preference tuning
180 minIntermediate
The core build session covers supervised fine-tuning and parameter-efficient methods like LoRA and QLoRA. Your team works with the hyperparameters that matter, then fine-tunes a model on its own task in a live lab and compares the result against the base model.
Supervised fine-tuning: full-parameter vs. parameter-efficient methods (LoRA, QLoRA, adapters)
Hyperparameters that matter, from learning rate and epochs to catastrophic forgetting
Preference tuning at an awareness level - RLHF and DPO, and when they earn their cost
Running a fine-tuning job on managed platforms (OpenAI, Anthropic, Vertex AI)
Fine-tuning open-weight models with Hugging Face and PEFT
Live lab - fine-tune a model on your own task and compare it against the base model
04
Evaluation before anything ships
150 minAdvanced
Eyeballed output comparisons give way to actual measurement. Your team builds a golden evaluation set from its own task, checks for catastrophic forgetting and gives every new training run a regression gate to pass.
Building a golden evaluation set from your own task examples
Comparing a fine-tuned model against the base model and a prompted baseline
Detecting catastrophic forgetting and regressions on tasks you didn't train for
Task-specific metrics vs. general benchmarks, and when each misleads
Safety and behavior evals before a fine-tuned model reaches production
Regression testing every new training run before it replaces the last one
05
Model registry and versioned deployment
150 minAdvanced
Trusting a fine-tuned model in production starts with tracking what was trained on which data before choosing between managed endpoints and self-hosted serving, then planning a rollout your team can reverse if something goes wrong.
Versioned model registry
Serving options: managed endpoints vs. self-hosted inference (vLLM, TGI)
Canary rollout and A/B testing a fine-tuned model against the incumbent
Quantization and cost-latency tradeoffs at serving time
A rollback plan for when a new version underperforms in production
Model cards and change history for audit and handoff
06
LLMOps for cost, monitoring and governance
120 minIntermediate
The operations discipline starts after launch day. Your team monitors drift and quality regression, tracks cost across training and inference and names who signs off on the next training run.
Monitoring output quality and drift once a fine-tuned model is live
Cost and usage monitoring across training runs and inference
Change control - who approves a new training run or a new deployed version
Incident response when a model regresses
A review cadence for retraining as your task or data shifts
Documentation and audit trail for governance and compliance reviews

What you will learn

Apply a decision framework for fine-tuning versus prompting or retrieval
Build production-ready training and evaluation datasets with disciplined curation plus licensing
Run supervised and parameter-efficient fine-tuning (LoRA, QLoRA) and preference tuning at an awareness level
Build a golden evaluation set to detect catastrophic forgetting before a model ships
A model registry with rollback
Choose managed endpoints or self-hosted serving based on cost and latency
Monitor a fine-tuned model in production for drift, cost and quality regression
Sign-off for the next run

Who should attend

AI/ML engineers fine-tuning or maintaining production models
MLOps and platform engineers who own model deployment and serving
Data scientists building training and evaluation datasets
Applied AI teams shipping a fine-tuned model into production
Engineering leads and CTOs deciding whether fine-tuning is worth the investment
DevOps and infrastructure teams responsible for serving and monitoring models

Fine-tuning work fills two days, usually split into half-day sessions. Onsite and live-online formats cover the same syllabus, built around the task and data slice your team brings to the room.

Format: Onsite or live online
Duration: 2 days (about 14 hours, can be split into half-day sessions)
Group size: Up to 12 participants per group
Language: English or Turkish
Materials: Reference fine-tuning pipeline, evaluation test suite templates, and a takeaway code repository
Certificate: Certificate of completion

Zeo started in 2011 and now works out of San Francisco, Istanbul, Ankara, and Lisbon. We run Copilot Academy and organize Digitalzone, an international digital marketing conference. This program draws on the 10+ years of consulting and training work behind that, applied to corporate AI adoption.