How to add a dataset to the PAD Registry

This page is the canonical procedure for contributing a new dataset to PaperAnalyticalDeviceND/pad_dataset_registry. Once your PR is merged, your dataset appears in the catalog at https://padproject.info/pad_dataset_registry/datasets/<DATASET_NAME>/ and is consumable through the same Croissant-format API as every other entry.

We welcome contributions from any PAD research group. The procedure below is meant to be executable by a human directly or with the help of an LLM-based coding assistant.

How the registry works

The registry is a public GitHub repository with one directory per dataset under datasets/. CI on every push to main runs docs/_scripts/validate_croissant.py and docs/_scripts/generate_catalog.py, then deploys the resulting site to the gh-pages branch. GitHub Pages serves gh-pages at https://padproject.info/pad_dataset_registry/.

There is no PR-time CI — broken croissant.jsonld files surface only after merge. Always run the validator locally before pushing.

What you must ship per dataset

Under datasets/<DATASET_NAME>/:

File Required Purpose
metadata_dev.csv yes Training split
metadata_test.csv yes Test split
metadata_val.csv optional Validation split (recommended if you have one)
labels.csv optional Single line of comma-separated class names (alphabetical convention)
projects.csv optional Single-line parent project lineage tag
class_distribution.csv optional Per-class counts split by #dev / #val / #test / #total
dataset_sizes.md optional Two-row table: total images + unique sample IDs per split
README.md required in practice Rendered as the catalog page on padproject.info. The validator does not reject a missing README, but without one your dataset’s catalog page is an empty directory listing with no description, citation, or context — functionally invisible to anyone browsing the registry
figs/ strongly recommended PNG visualizations referenced from the README. Class distribution and capture-condition heatmaps are what readers scan first when assessing a dataset; their absence makes the catalog page much harder to evaluate. See examples/build_registry_plots.py in this repo for a template that emits four standard plots
croissant.jsonld yes Auto-generated by docs/_scripts/create_croissant.py; commit the result

The metadata CSVs ARE the split

A common source of confusion: the registry materializes splits, not the code that produces them. Once metadata_dev.csv / metadata_val.csv / metadata_test.csv are published, those rows ARE the splits — consumers join on id and use the CSV’s assignment directly. They do not run your seed; they do not re-create your split. If your source pipeline currently generates splits at run time from a random seed (e.g. np.random.seed(42) inside a data-prep script), the build script you adapt in step 1 below should capture the resulting per-image (id, split) assignment as the input to the metadata CSVs and commit those CSVs to the registry. The seed belongs in your source repo’s build script as build-time documentation; it does not need to travel into the registry.

This is why the registry exists — to freeze the split assignment so two consumers reading the same dataset name see byte-identical splits regardless of which build version produced them. Lose the materialized split and you lose the reproducibility guarantee.

The build is CSV-in, CSV-out

A related framing point: the registry submission is fundamentally a CSV-to-CSV transformation. Your build script reads one or more source CSVs (your project’s manifest, your year-grouped exports, whatever you have), reshapes columns into the standard 8 (plus extensions), computes md5 hashes from your cached images, and writes the registry-shaped CSVs. The registry pipeline itself never reads project-internal binaries — no HDF5, no .npz, no pickled DataFrames, no compressed image bundles.

If your source pipeline stores splits in HDF5 (common for deep-learning workflows), your build script’s first step should be to export the (id, split) assignment out of HDF5 into a CSV column, and the rest of the build should work from that CSV. Tying the build script’s main loop to an HDF5 loader makes the registry submission depend on the binary format and your project’s HDF5 schema — neither of which the registry knows about, and neither of which other consumers should need to install to reproduce your submission.

The same applies to images: the registry stores URLs and md5 hashes pointing at an upstream image server (typically pad.crc.nd.edu), not the pixel bytes. Your build script computes hashes from a local PNG cache for integrity, but the catalog page and Croissant download instructions point downstream consumers at the original URLs.

Standard 8-column metadata schema

id, sample_id, sample_name, quantity, camera_type_1, url, hashlib_md5, image_name

Extending the schema

You can append extra columns to the right of the standard 8 (annotation flags, capture conditions, status fields). Strict-schema consumers ignore unknown columns; tools that know them benefit. Document any extensions in your README so consumers understand them.

Transformations: apply, or document for the consumer?

If your source pipeline does any non-trivial transformation on its way to the published metadata — relabelling a class (e.g. mapping a distractor class to a 0% quantity), merging bins, dropping rows, renaming categories — you must pick one of two options and document the choice explicitly in your README:

  1. Apply the transformation in the published metadata. Ship the post-transformation rows. Consumers see exactly the schema you chose; no surprises. Recommended for relabels that consumers would otherwise have to re-derive from a brittle convention.
  2. Ship raw values, document the transformation as a consumer responsibility. Ship the pre-transformation rows and explain in the README how a consumer should reproduce the transform.

Both are legitimate; the harm is leaving it undocumented and letting two consumers derive different post-transformation rows from the same source.

Naming conventions

The dataset directory name follows <Lab>_<ProjectFamily>_<Subset_or_Variant>_v<version> (e.g. Lieberman-Lab_ChemoPADPLStraining2026_Annotated_v1.0, FHI360_FHI2020-FHI2022_MidTrainingSet_Good_v1.0). Hyphens within tokens (Partial-Drug-Set); underscores separate the major segments.

Two non-obvious things worth knowing:

Procedure

  1. Stage the directory in your source repo. Build the metadata CSVs + README + figs in your own repository first so the build is reproducible from your source. The registry clone is just where the files land for the PR.
  2. Clone (or update) the registry locally: git clone https://github.com/PaperAnalyticalDeviceND/pad_dataset_registry.git.
  3. Create a feature branch: git checkout -b add-<DATASET_NAME> (or similar).
  4. Copy your staged directory into datasets/<DATASET_NAME>/.

  5. Generate croissant.jsonld. The generator is interactive; pipe answers via stdin:

    printf 'YOUR DESCRIPTION HERE\nv1.0\nFirstAuthor, SecondAuthor (Lab, Affiliation)\n' \
  | uv run --no-project --with pandas --with jsonschema --with requests \
    python docs/_scripts/create_croissant.py \
      --dataset-dir datasets/<DATASET_NAME>

    The --no-project flag is important because some registry deps install only on Linux with GPU; it lets the generator run in an isolated env on macOS / Windows / WSL.

  6. Validate locally:

    uv run --no-project --with pandas --with jsonschema --with requests \
  python docs/_scripts/validate_croissant.py

    Your new entry should report ✅ <DATASET_NAME>: Basic validation passed along with every other dataset.

  7. Commit on the feature branch: git add datasets/<DATASET_NAME>/ && git commit -m "Add <DATASET_NAME>". Do not commit uv.lock if uv happens to create one in the registry clone — it’s a side-effect, not part of the contribution.
  8. Push the branch (manual step): git push -u origin <BRANCH_NAME>. If you are running this through an LLM coding assistant, the push itself usually needs to come from your shell rather than the assistant — most assistants block cross-organization pushes by default, even with in-conversation authorization. The PR can then be opened from the assistant once the branch is on the remote.
  9. Open a PR against main using the template at the bottom of this page.
  10. Maintainers review and merge. On merge, the Build and Process Dataset Catalog workflow validates → generates catalog → deploys to gh-pages. Your dataset’s page should be live at https://padproject.info/pad_dataset_registry/datasets/<DATASET_NAME>/ within ~2 minutes.

A worked example with code

Two reference scripts live in this repository under examples/ — they’re the working code that produced the Lieberman-Lab_ChemoPADPLStraining2026_Annotated_v1.0 entry:

Both files mark every project-specific knob with a # PROJECT-SPECIFIC: comment. Copy them into your project’s repo, adapt the input paths and the SAMPLE_NAME_MAP, and they should produce a registry-shaped staging directory in one run. See examples/README.md for the full adaptation walkthrough.

AI-assisted contribution

If you use an LLM-based coding assistant (e.g. Claude Code, Cursor, Aider), point it at:

The assistant can produce the metadata CSVs, the figs, and the templated README in one pass given access to your source manifest and a cached image directory. Always have a human review the staging directory and the generated croissant.jsonld before opening the PR.

A note on shell automation: some agentic CLIs (Claude Code in particular) classify pushes to repos outside the active project’s source-control as cross-org actions and block them by default. If you encounter this, run the push from your own shell (git push -u origin <BRANCH_NAME>) — the PR can then be opened from the assistant once the branch is on the remote.

Common gotchas

Issue Fix
Croissant generator fails on uv run with nvidia-cublas-cu11 install error Add --no-project to the uv run invocation.
Generator hangs on first prompt Pipe answers via printf '...\n...\n...\n' \| python …. Three prompts in order: description, version, creator.
Renamed dataset directory but the catalog still shows the old name The croissant embeds the dataset name in @id and contentUrl paths. Regenerate croissant.jsonld after any rename.
figs/ images show as broken links on padproject.info Reference them with ./figs/<name>.png (relative) in the README; generate_catalog.py copies the figs/ directory into the rendered output.
Catalog page didn’t update after merge Check the Actions tab for the Build and Process Dataset Catalog run. A failed validate_croissant.py aborts the deploy.
quantity column has surprises (non-numeric strings, non-zero values for blank / lactose controls, NaNs) Validate the column at build time: it should be entirely numeric and control-class rows should be 0. A common upstream gotcha: HPLC measurements of “blank” can come back as small non-zero values; ship them as 0 in the registry CSVs and document the convention in the README.
Different consumers see different splits for the same dataset name The metadata CSVs metadata_dev.csv / metadata_val.csv / metadata_test.csv ARE the split. Make sure your build script captures the per-image (id, split) assignment from your source pipeline (which may itself use a seed) and freezes it into those CSVs, rather than expecting consumers to re-derive the split from a seed.

PR template

## Summary
- <N> images / <M> unique sample IDs, PAD#-grouped split.
- <One paragraph on what's distinctive about this dataset.>

## Standard schema + extensions
- 8-column schema: id, sample_id, sample_name, quantity, camera_type_1, url, hashlib_md5, image_name.
- Extended columns: <list any extras with one-line descriptions>.

## Local validation
- `docs/_scripts/validate_croissant.py` reports basic validation passed.

## Citation
> <Authors>. *<Dataset Title>, <Version>.* <Lab, Affiliation>. <Source repo URL>

## Test plan
- [ ] Build-and-deploy workflow runs on push to main.
- [ ] Catalog page renders the README + figs.
- [ ] Class distribution + dataset sizes appear correctly.

Questions, help, and conventions