Model Foundry is a reference implementation for how a rigorous modeling repository should be structured. It is not an AutoML product. It will not search for you and it does not promise a better model.
What it does is make the things that quietly ruin applied modeling work — a leaked split, preprocessing fit on the whole dataset, tuning against the test set, a metric that is wrong for the task, a number nobody can reproduce — hard to do by construction rather than discouraged by convention.
install & run
Python 3.10 to 3.12. The mandatory core is numpy, pandas, scikit-learn, scipy and PyYAML. LightGBM, XGBoost, CatBoost, statsmodels, lifelines, scikit-survival, Optuna and SHAP are all optional — a model whose dependency is missing is reported as skipped with a reason, and the run continues.
Not on PyPI. [all] pulls every optional learner; plain -e . gives the light core.
git clone https://github.com/Gariyuuu/model-foundry && cd model-foundry
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]" # or: pip install -e . for the light coreHow many models each task family can reach on this machine, and which are missing.
foundry infoThe dataset validation report is written before a single model is fitted. A structural leak fails here, not in review.
python examples/scripts/make_datasets.py # six synthetic datasets
foundry validate configs/classification_credit_default.yaml
foundry train configs/classification_credit_default.yamlevaluate recomputes metrics from stored predictions rather than re-reading them, and reproduce re-runs the stored config and diffs the result.
foundry evaluate latest # recompute every metric from stored predictions
foundry report latest # re-render the Markdown report
foundry reproduce latest # re-run the stored config and diff the metricsEvery run describes itself well enough to be re-run by someone who was not there.
config.yaml the resolved config, re-runnable as-is
manifest.json environment, package versions, git commit, file index
result.json every metric, the split, the limitations
validation.json the dataset validation report
report.md the human-readable model report
models/<key>.joblib one fitted pipeline per model
predictions/<key>.csv per-row test predictions per model00 — Position
It is
A set of reusable, auditable templates. One orchestration layer, six task implementations, a typed configuration schema, verified splits, and a run artifact that explains itself to someone who was not there.
It is not
An AutoML system, a model zoo, or a leaderboard. There is no search over pipelines, no neural component, no serving layer and no dashboard. Adding an estimator is one registry entry; that is the intended extension point, not a roadmap.
The governing idea
A safeguard that depends on remembering it is not a safeguard. Every rule here is enforced in exactly one place, asserted by a named test, and re-checked on every run — so it cannot hold in one task family and quietly fail in another.
01 — Pipeline
Stages 1 through 4 and 9 through 11 are shared and implemented once. Stages 5 through 8 are delegated, because a survival model is not a classifier with a different metric and a causal estimand is not a prediction at all.
02 — Stages
1.0
Configuration
A run is fully described by one YAML file, parsed into frozen dataclasses before any data is read. Unknown keys are rejected against the schema; enums, task-block matching, metric names and split strategies are all validated up front.
Enforced
A forecasting config cannot express a random split — the schema does not accept one. 43 adversarial malformed configs were rejected; none slipped through.
2.0
Validation
Required columns, class support for stratification, censoring rates, treatment arm sizes, duplicated panel keys, missingness — plus a scan for features that are near-perfect predictors of the target.
Enforced
Errors block the run. Warnings do not — they are copied into the report's limitations, because “we trained anyway despite 45% censoring” is a fact the reader needs.
3.0
Splitting
Partitions must be pairwise disjoint with no duplicate indices, and every cross-validation fold must be a subset of train. Grouped splits are re-checked for entity disjointness; chronological splits for temporal order. Panels are cut at a single global timestamp, so no entity's future informs another's past.
Enforced
The check runs in __post_init__ — on every split ever
constructed. Ten deliberately corrupted splits were built during release review; all ten
raised LeakageError.
4.0
Preprocessing
An imputation median, a scaler's mean, an encoder's category list are all parameters estimated from data. They live inside the estimator's pipeline, so they refit within every fold and cannot be tuned against.
Enforced
A structural detector asserts that all 32 registered models are wrapped
preprocess → model. Poisoning held-out rows with 1e12 leaves every fitted
parameter bit-identical.
5.0
Baselines
Every family fits a baseline first: majority and stratified for classification, mean and median for regression, naive and seasonal naive for forecasting, Kaplan–Meier for survival, z-score and Mahalanobis for anomaly, the unadjusted difference for causal.
Enforced
Baselines are not decoration. In the shipped examples a Mahalanobis distance beats Isolation Forest, and the report says so.
6.0
Tuning
Optuna optimises over the cross-validation folds it is handed, and those folds are guaranteed to live inside train. The whole pipeline is refit inside every fold, so preprocessing statistics are re-estimated per fold.
Enforced
There is no code path by which a trial can be scored on validation or test. For anomaly detection, tuning is refused — a labelled objective would make the labels part of fitting.
7.0
Selection
Every task family's selector reads validation metrics. Test is scored once and never consulted for a decision — not for ranking, not for thresholding, not for early stopping.
Enforced
Proven two ways: the source of all five selectors is parsed and checked, and an adversarial test hands a losing candidate a perfect test score after fitting. The winner does not move.
8.0
Artifact & report
Resolved config, dataset fingerprint, split description and guarantees, seed, package versions, git commit, fitted pipelines, per-row predictions, every metric, and a limitations section written by the runner that produced it.
Enforced
foundry evaluate does not re-read stored numbers — it
recomputes every metric from the prediction files. Tamper with either side and it exits
non-zero.
03 — Families
The orchestration layer is shared. The methodology is not — forcing these into one abstraction would model none of them correctly.
01
Probability-first. Every candidate must expose predict_proba,
because AUC, log loss and calibration all need probabilities. The calibrator is frozen onto
the fitted model and fit on validation — never on train, never on test.
02
Mean and median baselines, because they are optimal constants under different losses — a model that loses to the median on MAE has learned nothing for that loss. Residual bias and spread are reported, not just R².
03
Chronological validation only. Evaluation is a rolling-origin backtest with the model refit at every origin — which is what makes classical and feature-based models comparable at all. MASE is scaled on the training series.
04
Right-censoring is the whole problem. A subject who has not had the event is
not a negative. The target is a structured (event, time) array, the concordance
index is implemented natively, and the proportional-hazards assumption is tested.
05
A score is not a label. Detectors never see labels; the operating point comes from the training score distribution. Without labels the report states plainly that no detector can be shown to be better — the “selected” model is a placeholder.
06
No test-set score, because a counterfactual is never observed. Estimators are ranked by covariate balance, never by effect size — ranking on the effect is how a null result becomes a positive one.
04 — Proof
A table of claims is worth nothing if it drifts from the suite it cites. The document is parsed on every run: a citation that is stale, vacuous or overused fails the build.
| Claim | Proven by |
|---|---|
| Train, validation and test never overlap | test_partitions_are_pairwise_disjoint |
| Hyperparameter search never sees validation or test | test_cv_folds_live_strictly_inside_train |
| A corrupted split raises rather than proceeds | test_a_cv_fold_reaching_into_test_raises |
| A grouped split puts every entity in exactly one partition | test_group_holdout_keeps_every_group_in_one_partition |
| Random splitting does not prevent entity leakage — and says so | test_random_holdout_does_not_prevent_entity_leakage |
| The imputation median is the training median | test_imputer_uses_the_training_median_not_the_full_dataset_median |
| Every model is wrapped in a preprocess→model pipeline | test_every_registered_model_is_wrapped_in_a_preprocessing_pipeline |
| Mutating future rows cannot change an earlier row's lag features | test_lag_features_cannot_see_the_future |
| A rigged, perfect test score cannot change the selected model | test_a_perfect_test_score_cannot_change_the_selected_model |
| Binary and multiclass metrics match scikit-learn exactly | test_binary_metrics_match_sklearn |
| The concordance index matches lifelines, an independent library | test_concordance_index_matches_lifelines |
| Censored rows are not treated as negatives | test_censored_rows_are_not_treated_as_negatives |
| Two runs of one config give bit-identical metrics | test_two_runs_of_one_config_give_identical_metrics |
| The saved model reproduces the saved predictions | test_the_saved_model_reproduces_the_saved_predictions |
foundry evaluate detects a tampered report | test_evaluate_detects_a_tampered_report |
| Every test cited on the guarantee page exists | test_every_cited_test_exists |
05 — Measured
Six worked examples ship with the deterministic datasets that produced them. These are measured values, not illustrations. The ones that flatter nobody are the ones worth keeping.
| Model | Validation RMSE | Test RMSE | Outcome |
|---|---|---|---|
| linear_regression | 41,775 | 43,783 | selected |
| lightgbm | 43,343 | 43,433 | best on test — not selected |
| hist_gradient_boosting | 45,824 | 46,852 | — |
| random_forest | 51,394 | 49,394 | — |
| mean (baseline) | — | 110,810 | R² = 0 by construction |
LightGBM has the best test RMSE. It is not selected, because linear regression had the better validation RMSE. Reporting LightGBM's 43,433 as a held-out estimate would be reporting a number the test set had already been used to choose.
| Estimator | Estimand | Effect | 95% interval | max |SMD| after |
|---|---|---|---|---|
| naive_difference | unadjusted | −4,616 | [−5,519, −3,712] | 0.4409 |
| regression_adjustment | ATE | 1,885 | [1,658, 2,148] | 0.4409 |
| ipw | ATE | 3,416 | [1,605, 5,273] | 0.1176 |
| matching | ATT | 1,786 | [1,252, 2,319] | 0.0736 |
| aipw | ATE | 2,236 | [1,591, 2,881] | 0.1172 |
The unadjusted contrast has the wrong sign: low earners were far likelier to enrol. Every adjusted estimator recovers a positive effect — and the one furthest from the truth, IPW at 3,416, is also the one whose post-adjustment balance breaches the 0.1 convention. The diagnostic flags it before anyone looks at the effect size. The report's own verdict is printed verbatim: “Adjusted estimators disagree materially. Treat the effect as poorly identified rather than picking the most convenient one.”
| Detector | Role | ROC-AUC | PR-AUC |
|---|---|---|---|
| elliptic_envelope | candidate | 0.8320 | 0.3083 |
| mahalanobis | baseline | 0.8269 | 0.2996 |
| local_outlier_factor | candidate | 0.8230 | 0.3062 |
| isolation_forest | candidate | 0.8164 | 0.2170 |
| zscore_max | baseline | 0.7855 | 0.2638 |
Isolation Forest — the default choice in most tutorials — is beaten on PR-AUC by a plain Mahalanobis distance. One of the four injected fault modes breaks the temperature / vibration correlation while leaving both margins individually normal: invisible to a per-feature z-score by construction, visible to a covariance-aware method. That is what the two statistical baselines exist to force.
06 — Defects
An adversarial pass over a repository that already passed its own tests. Each one is recorded because the fix is less interesting than the fact that the working tree did not catch it.
Reproducibility
The dataset fingerprint hashed str(series.dtype). pandas 3.0
renamed the string dtype from object to str, so byte-identical CSVs
fingerprinted differently across that release and foundry reproduce reported a
dataset mismatch that did not exist — a false alarm on the exact check users rely on. String
dtypes now normalise to one token; the value is pinned by a golden test.
Artifact integrity
A causal run persists no fitted model — its output is an effect estimate,
not a predictor. The manifest named the selected estimator anyway, so load_model()
failed on an artifact that advertised it. The field is now None unless the key
actually exists on disk.
CLI
foundry validate --json interleaved progress text with the
payload on stdout, so the output could not be piped without hunting for braces. Human logging
now goes to stderr whenever a machine-readable mode is active.
Splitting
Whole groups move, not rows, so small group counts round badly. Aggressive
split fractions escaped as a raw scikit-learn ValueError about sample counts,
which said nothing about groups or about which config field to change.
Metrics
When censoring left no comparable pairs, the concordance index propagated
a library-specific exception instead of degrading to NaN. An undefined metric must
be undefined, not fatal.
Reporting
Each estimator's assumptions were stored in result.json but
never rendered. A reader of the report alone would not have seen that the matching interval is
a paired-difference approximation, not a valid bootstrap.
Honesty
The README claimed the classification example showed a model selected on validation despite another winning on test. It did not — logistic regression won both. The finding was real but belonged to the regression example. The prose was corrected to the measurement, never the reverse, and a script now re-derives every documented number from the artifacts on each CI run.
07 — Limits
Never refit on train + validation. That gives up training data in exchange for a validation set that stays honest — it is what fits the calibrator and ranks the models. Reported test metrics describe a model trained on the smaller partition.
Test metrics carry no confidence interval, so two close models may not be distinguishable. The causal task is the exception, with caveats of its own.
A full rolling-origin backtest per trial is prohibitively expensive, so tuning scores on the final blocks of the training period. It never touches validation or test, but it is a lower-fidelity proxy for the reported backtest.
It treats matched pairs as independent and understates uncertainty. The standard bootstrap is invalid for nearest-neighbour matching estimators, and no valid analytic variance is computed, so the report labels it a rough guide.
Identical results are verified on one machine with one pinned environment, and re-verified from a pristine clone. Across different hardware, BLAS builds or library versions, small numerical differences are expected — which is what the version manifest and the tolerance flag are for. Reproduction checks detect drift; they cannot eliminate it.