AI Pipeline

AI Pipeline is my reusable MLOps skeleton for computer vision. Instead of starting projects as throwaway notebooks, they now start here: the plumbing already works, and the only thing to change is the data and the model.

The Problem

The early steps of a CV project are the least interesting and the most error-prone — and they're identical every time:

• Data versioning (which image set is latest?)
• A reproducible training run
• Experiment tracking you can actually search
• A served model you can curl

Rewriting these per project is cargo-cult MLOps. A shared scaffold makes every subsequent project faster and safer.

Architecture

Nine CLI stages, each with a clear input and output contract:

dataset → validate → version → train → track → export → serve → /infer

1. Validate — Check data integrity before anything runs.
2. Version — DVC-style tagging so every run is reproducible.
3. Train — YAML-configured runs with seed control.
4. Track — MLflow logging for search and comparison.
5. Export — Torch → ONNX as a first-class step, not an afterthought.
6. Serve — Containerised ONNX endpoint, ready to curl.

One CLI, distinct stages, scriptable and replayable.

Engineering decisions

1. Config as code, not as UI. Training runs are defined by YAML files; a config is a unit you can diff, review, and version.
2. Version data with the code. Data, model, and code get a unified tag so any checkpoint can be traced to the exact data that produced it.
3. Export to ONNX early. The torch → ONNX step is painful if deferred; doing it as a first-class pipeline step means serving is never a scaffolding afterthought.

train:
  model: yolo8s
  data: datasets/shelf  # DVC-tracked
  seed: 7
  batch: 16
  epochs: 100
  device: t4

What it removes from a typical startup

Real-world numbers

The scaffold is credited with cutting cold-start on two subsequent projects (Retail Shelf Intelligence among them) from weeks of plumbing to a couple of days.

Challenges

The choke-point was reproducibility across platforms. MKL, cuDNN, and ONNX runtime versions quietly disagree between machines. Pinning the environment in a lockfile that included GPU libs took the most effort and was worth most of it.

Lessons

• Skeletons should be opinionated. A scaffold with rules prevents chaos better than a scaffold with options.
• The ONNX boundary is worth its weight. It decouples research from serving forever, and it's the thing I'd do again most insistently.

Roadmap

• Airflow DAGs so scheduled retraining is real.
• A dataset health report (duplicates, class imbalance) before training.