graflag.verify
Check that a finished experiment published a result worth believing.
status.json saying completed means the process exited 0 and wrote a results.json that parses. It does not mean the numbers are the method’s, that they cover the test split, or that the scores are the ones whose AUC the method printed. Every scientific-validity defect recorded in graflag-shared passed completed.
The checks here are the generic half of that – the part that does not need to know what the method computes:
the sample is usable at all: scores and ground truth the same length, both classes present, something left after the evaluator’s filtering
the scores vary (a constant column scores AUC 0.5 and looks like a method)
what the method says it measured is what it published: every *auc* in metadata.summary is compared against eval/evaluation.json, and scored_samples against the actual count
the split is declared, and declared to be the test one
What it cannot check is whether the method’s own number is right – only that the published scores reproduce it. A method that measures the wrong thing consistently passes every check below. Read the run’s log as well.
This is the last of the method-integration skill’s four gates, after
graflag evaluate. It began as the skill’s scripts/verify_run.py, which is
now a shim over main(); graflag verify -e EXP and the MCP server’s
verify_experiment tool reach the same checks through
graflag.core.GraFlag.verify().
- graflag.verify.remote_script(exp_dir)[source]
The probe, as a self-contained program to run on the manager.
- graflag.verify.check(probe)[source]
Turn a probe summary into findings. Pure, so it is testable offline.
Returns a list of (level, message); level is “ERROR”, “WARN” or “OK”.
- graflag.verify.probe(gf, experiment)[source]
Run the probe on the manager through GraFlag’s SSH layer.
Returns the probe summary, or raises RuntimeError with what the manager said when it could not be produced.