DVC: Version Control for ML Data

By Kokil Thapa | Last reviewed: September 2026

DVC: Version Control for ML Data solves a problem every ML team hits early: Git tracks code well, but it chokes on gigabyte CSV files, image folders, and model weights. You need experiment history tied to exact data snapshots, not a shared drive named final_v3_really_final.csv. On production systems I have integrated with LLM APIs and data pipelines, reproducibility is not academic. It is how you debug drift, pass audits, and ship models without guessing which dataset trained yesterday's build. This guide walks through DVC setup, remotes, pipelines, and the Git workflow that keeps ML projects auditable in 2026.

The same discipline applies whether you run on a laptop or on a GPU-backed Kubernetes cluster. DVC sits beside Git. It does not replace it.

What Is DVC and Why Do ML Teams Need Version Control for Data?

Data Version Control (DVC) is an open-source CLI and Python library. It extends Git for machine learning workflows. Small text files stay in Git. Large binaries live in remote storage. DVC stores lightweight pointer files in the repo instead.

Each dataset version gets a hash. Git commits reference that hash. When you check out an old commit, DVC pulls the matching files from remote storage. Your code and data move together.

ML differs from typical app development. Training runs depend on four moving parts: raw data, preprocessing code, hyperparameters, and the resulting model. Change any one and accuracy shifts. Without versioning, teams lose the ability to answer basic questions.

• Which dataset produced the model in production?
• Can we roll back to last month's training set after a bad import?
• Did a colleague retrain on data we never approved?

Git alone cannot answer those. Git LFS helps with large files but lacks pipeline definitions, experiment tracking, and ML-native remotes. Dedicated platforms like MLflow excel at experiment logging. DVC focuses on data and pipeline reproducibility inside your existing Git repo.

For Nepal-based teams building AI features under budget pressure, DVC keeps costs low. You bring your own storage. A data lake on S3 or a VPS with SSH remote works fine. No per-seat SaaS fee is required for core versioning.

How Do You Install and Initialize DVC in an Existing Project?

DVC runs on Linux, macOS, and Windows. Python 3.9 or newer is typical. Install it once per machine or pin it in your project virtualenv.

Install the CLI

python -m pip install "dvc[s3]"
dvc version

The bracket extras pull optional remote backends. Use dvc[gs] for Google Cloud, dvc[azure] for Azure, or dvc[ssh] for SFTP-style remotes. Install only what you need. Keep CI images lean.

Initialize inside a Git repository

git init my-ml-project
cd my-ml-project
dvc init
git add .dvc .dvcignore
git commit -m "Initialize DVC"

dvc init creates a .dvc/ directory and a .dvcignore file. Commit both. The official DVC Get Started guide documents each flag if you need cloud-specific setup.

Track your first dataset

mkdir -p data/raw
cp /path/to/training.csv data/raw/training.csv
dvc add data/raw/training.csv
git add data/raw/training.csv.dvc data/raw/.gitignore
git commit -m "Track training dataset with DVC"

After dvc add, the CSV moves into .dvc/cache/. Git tracks a small training.csv.dvc pointer file instead. Teammates clone the repo, then run dvc pull to fetch the actual CSV from remote storage.

If you already use Docker volumes for persistent data, mount the DVC cache directory as a volume. Re-downloads become rare across container rebuilds.

How Do You Configure DVC Remote Storage for Team Collaboration?

Local cache alone is not enough for teams. Configure a default remote so dvc push and dvc pull sync artifacts across laptops, CI runners, and training servers.

S3 remote example

dvc remote add -d storage s3://my-ml-bucket/dvc-store
dvc remote modify storage region us-east-1
git add .dvc/config
git commit -m "Add default DVC remote"

dvc push
dvc status

The -d flag sets the default remote. Credentials follow standard AWS environment variables or IAM roles on EC2. For Nepal teams using international cloud providers, store keys in CI secrets. Never commit them.

SSH remote for a VPS or on-prem server

dvc remote add -d onprem ssh://user@203.0.113.10/home/user/dvc-storage
dvc remote modify onprem port 22
dvc push

This pattern suits budget setups. A single Ubuntu VPS with rsync-over-SSH can serve a small team. Pair it with nightly backups via your normal Linux server administration routine.

Pick one default remote per repo. Multiple remotes are supported for migration, but one source of truth reduces confusion.

How Do You Define Reproducible ML Pipelines with dvc.yaml?

Tracking static files is step one. Pipelines tie stages together. Each stage declares inputs, outputs, and the command that transforms them. DVC knows what to rerun when upstream data or code changes.

Sample pipeline structure

mkdir -p src data/processed models
cat > src/prepare.py <<'PY'
import pandas as pd
df = pd.read_csv("data/raw/training.csv")
df.to_parquet("data/processed/features.parquet")
PY

cat > src/train.py <<'PY'
import pandas as pd, joblib, sys
df = pd.read_parquet("data/processed/features.parquet")
model = {"weights": [1, 2, 3]}
joblib.dump(model, "models/classifier.pkl")
open("metrics.json", "w").write('{"accuracy": 0.91}')
PY

dvc stage add -n prepare \
  -d data/raw/training.csv -d src/prepare.py \
  -o data/processed/features.parquet \
  python src/prepare.py

dvc stage add -n train \
  -d data/processed/features.parquet -d src/train.py \
  -o models/classifier.pkl -m metrics.json \
  python src/train.py

git add dvc.yaml dvc.lock src/
git commit -m "Add DVC pipeline stages"

DVC writes dvc.yaml for human-readable stage definitions. It writes dvc.lock with content hashes for every dependency. Commit both files. The lock file is your reproducibility contract.

Reproduce the full pipeline

dvc repro
dvc metrics show
dvc metrics diff HEAD~1

dvc repro runs only stale stages. Change raw data and prepare reruns. Train follows automatically. Skip unchanged steps. This saves hours on large datasets.

Pair DVC pipelines with orchestrators when schedules grow complex. Apache Airflow triggers nightly retrains. DVC guarantees the same inputs inside each run. Kubeflow Pipelines fits when you already run ML workloads on Kubernetes.

1. Version raw data with dvc add or pipeline dependencies.
2. Define stages in dvc.yaml with explicit inputs and outputs.
3. Commit lock files alongside application code.
4. Run dvc repro locally and in CI before promoting a model.
5. Push artifacts and metrics after successful runs.

On client projects where I wire LLM features into Laravel backends, we still version evaluation sets with DVC. Prompt changes get Git history. Test corpora get hashed storage. Support tickets become debuggable.

How Does DVC Compare to Git LFS, MLflow, and LakeFS?

Teams often stack tools instead of choosing one winner. Know what each layer owns.

Git LFS is fine for designers storing assets. It does not define ML pipelines or cache-aware reproduction. MLflow shines when scientists compare hundreds of runs. DVC shines when you need deterministic data snapshots per commit.

A practical stack: DVC for data plus pipelines, MLflow for experiment comparison, Git for code review. They overlap on metrics but serve different questions. Use JSON formatting tools to inspect metrics.json before committing if you diff metrics by hand.

For web teams adding ML scoring to an existing product, see how roles split in our AI engineer vs ML engineer vs data scientist guide. DVC usually lands in the ML engineer's toolkit.

What Production Mistakes Break DVC Version Control for ML Data?

DVC is straightforward until process gaps appear. These failures show up on real projects.

Forgetting to push artifacts

A developer runs dvc add, commits the pointer, and pushes Git. They forget dvc push. CI fails with missing files. Add a CI step that runs dvc pull early. Fail fast with a clear message.

Committing cache or data by accident

Keep .dvc/cache/ and tracked data paths in .gitignore. DVC sets local ignores, but custom folders need review. A multi-gigabyte CSV in Git history is painful to purge. Follow the same discipline as test data management for pipelines.

Stale lock files

Editing a stage command without regenerating dvc.lock breaks reproducibility. Always run dvc repro before commit when stage definitions change. Treat lock drift like a failing unit test.

No remote backup policy

DVC remotes are not backups by default. Enable bucket versioning on S3. Snapshot SSH remotes nightly. Ransomware or operator error should not wipe your only model lineage.

Mixing secrets into tracked folders

Never dvc add directories containing API keys or .env files. Scope tracking narrowly. Use environment variables in training scripts instead.

When models reach production, connect versioning to monitoring. Monitor ML models for drift against the dataset hash deployed with each artifact. If live inputs diverge from training data, alerts fire before accuracy collapses.

Regulated domains need extra care. If personal data appears in training sets, align retention with data privacy law in Nepal for web apps. Versioning does not replace anonymisation. It documents which snapshot contained which fields.

Key Takeaways

• DVC: Version Control for ML Data stores large artifacts in remotes while Git tracks lightweight pointer and lock files.
• Run dvc add for datasets, define stages in dvc.yaml, and commit dvc.lock with every pipeline change.
• Configure one default remote early and script dvc push plus dvc pull in CI so teammates never train on missing files.
• Use dvc repro and dvc metrics diff to rerun only stale stages and compare experiment outcomes across commits.
• Combine DVC with MLflow or orchestrators for experiment UI and scheduling, but let DVC own data lineage and pipeline hashes.
• Back up remotes, keep secrets out of tracked paths, and tie production monitoring to the dataset hash shipped with each model.

People Also Ask

Does DVC replace Git?

No. DVC depends on Git for code history and collaboration. It adds data-aware tracking, caching, and pipeline reproduction. You still branch, review, and merge code in Git. DVC handles artifacts Git should never store directly.

Can DVC work without cloud storage?

Yes. A local directory remote or SSH server is valid. Solo researchers often start with a local path remote on an external drive. Teams eventually move to S3 or similar for shared access and CI pulls.

How is DVC different from MLflow?

DVC focuses on versioning datasets, models, and reproducible pipelines inside Git repos. MLflow focuses on experiment logging, parameter search, and model registry workflows. Many teams use both without conflict.

Is DVC suitable for deep learning with huge datasets?

Yes, with realistic expectations. DVC deduplicates by hash, so unchanged files skip re-upload. Very large video or image corpora may need partial tracking or external catalogues. Pipelines still record which snapshot trained each model.

Ship Reproducible ML Data Pipelines with Confidence

DVC: Version Control for ML Data turns experiment chaos into ordinary Git workflow. Pin datasets, automate stages, push to remotes your team controls, and diff metrics between commits. Start with one tracked CSV and a two-stage pipeline. Expand when retraining becomes routine.

If you are adding AI features to a Laravel app, a WooCommerce recommender, or a legal-tech document classifier, reproducible data beats heroic debugging. I have shipped ML-adjacent integrations on platforms like Adventure Third Pole Trek and law-firm portals where audit trails matter.

Need help wiring DVC into CI, choosing remote storage, or connecting models to your web product? See our AI integration and automation services or custom software development offering. For broader pipeline design, read about getting started with data science and enterprise application development.

Validate JSON metrics before they enter Git using our regex tester for log parsing, or browse the full developer tools collection. Explore more write-ups on the blog, review portfolio projects, or learn about our work on the about page and homepage.

When deployment reliability matters as much as model accuracy, pair versioning with testing and optimization and solid API development practices. Read how we migrate sites without data loss if you are moving legacy assets into a modern ML stack.

Contact us to plan DVC adoption for your next ML or AI project. Bring your Git repo, one painful dataset, and we will map a reproducible pipeline you can maintain after launch.