DSH sandbox attribution
Sealed attribution records; semantic accuracy and evidence completeness are reported separately.
Attribution signal
semantic vs evidence, by case and armSemantic judges can identify the layer; evidence completeness is the measured skill increment.
Sealed records: share whose grader `passed` is true.
Legacy art-creation quality
normalized atom-rating mean, not comparable with attributionMean quality per report and arm (0–100 over verifiable atoms).
Atom outcomes: pass=2, partial=1, fail=0, unverifiable excluded from quality.
Report-order trend. Three reports are not a time series; read as a sequence of frozen snapshots.
Per-case mean quality by arm across all legacy reports.
Trend analysis
computed from the tables belowAttribution: B evidence 6/6 (100%) vs A0 0/5 (0%); B semantic 6/6 (100%) vs A0 5/5 (100%). The gap is evidence completeness, not layer recognition. Legacy art-creation quality is a separate scale; do not merge it with attribution scores. Legacy report deltas: Formal comparison: candidate 59.2% vs previous 94.6%; Six-case comparison: candidate 95.0% vs previous 88.6%; Refinement comparison: candidate 90.7% vs previous 95.9%.
- Small n (1-3 per cell), one model, one fixture family, one host.
- Blind judges were same-provider subagents, not a different model family.
- Case11 A0 has one excluded tool-call-budget run.
- Evidence completeness measures citation validity, not correctness of the underlying tool output.
- No claim of a cross-suite or cross-model leaderboard.
Attribution cases
B = with skill, A0 = without skill| Case | Truth | Arm | Semantic | Evidence | Scenario |
|---|---|---|---|---|---|
sandbox-natural-11 | both | B | 3/3 (100%) | 3/3 (100%) | WebGL, outer and inner /dev hide GPU |
sandbox-natural-11 | both | A0 | 2/2 (100%) | 0/2 (0%) | WebGL, outer and inner /dev hide GPU |
sandbox-natural-09 | dsh | B | 3/3 (100%) | 3/3 (100%) | Outer binds GPU, DSH inner /dev hides it |
sandbox-natural-09 | dsh | A0 | 3/3 (100%) | 0/3 (0%) | Outer binds GPU, DSH inner /dev hides it |
sandbox-natural-08 | app | B | — | 1/1 (100%) | Application bug control |
sandbox-natural-08 | app | A0 | — | 1/1 (100%) | Application bug control |
Art-creation legacy suites
normalized unified table shape| Suite | Case | Split | Arm | State | Score | Scored | Model | Skill | Duration | Report |
|---|---|---|---|---|---|---|---|---|---|---|
| art-creation | editorial-transfer | legacy | previous | completed | 100% | 17/20 | gpt-6.1-sol | b3eba53ec10f | 375s | Formal comparison |
| art-creation | editorial-transfer | legacy | none | completed | 100% | 17/20 | gpt-6.1-sol | 445s | Formal comparison | |
| art-creation | editorial-transfer | legacy | candidate | completed | 100% | 17/20 | gpt-6.1-sol | 854e3ea6d630 | 483s | Formal comparison |
| art-creation | reference-costume | legacy | none | completed | 88% | 20/23 | gpt-6.1-sol | 543s | Formal comparison | |
| art-creation | reference-costume | legacy | candidate | completed | 85% | 20/23 | gpt-6.1-sol | 854e3ea6d630 | 445s | Formal comparison |
| art-creation | reference-costume | legacy | previous | completed | 88% | 20/23 | gpt-6.1-sol | b3eba53ec10f | 501s | Formal comparison |
| art-creation | editorial-transfer | legacy | none | completed | 100% | 17/20 | gpt-6.1-sol | 443s | Formal comparison | |
| art-creation | editorial-transfer | legacy | candidate | failed | 0% | 0/4 | gpt-6.1-sol | 854e3ea6d630 | 14s | Formal comparison |
| art-creation | editorial-transfer | legacy | previous | failed | 100% | 3/4 | gpt-6.1-sol | b3eba53ec10f | 149s | Formal comparison |
| art-creation | reference-costume | legacy | candidate | completed | 85% | 20/23 | gpt-6.1-sol | 854e3ea6d630 | 540s | Formal comparison |
| art-creation | reference-costume | legacy | previous | completed | 92% | 20/23 | gpt-6.1-sol | b3eba53ec10f | 496s | Formal comparison |
| art-creation | reference-costume | legacy | none | completed | 100% | 20/23 | gpt-6.1-sol | 477s | Formal comparison | |
| art-creation | editorial-transfer | legacy | candidate | completed | 85% | 17/20 | gpt-6.1-sol | 854e3ea6d630 | 349s | Formal comparison |
| art-creation | editorial-transfer | legacy | previous | completed | 100% | 17/20 | gpt-6.1-sol | b3eba53ec10f | 394s | Formal comparison |
| art-creation | editorial-transfer | legacy | none | failed | 100% | 3/4 | gpt-6.1-sol | 122s | Formal comparison | |
| art-creation | reference-costume | legacy | previous | completed | 88% | 20/23 | gpt-6.1-sol | b3eba53ec10f | 368s | Formal comparison |
| art-creation | reference-costume | legacy | none | failed | 90% | 20/23 | gpt-6.1-sol | 402s | Formal comparison | |
| art-creation | reference-costume | legacy | candidate | failed | 0% | 0/6 | gpt-6.1-sol | 854e3ea6d630 | 14s | Formal comparison |
| art-creation | editorial-transfer | legacy | previous | completed | 100% | 17/20 | gpt-6.1-sol | b3eba53ec10f | 287s | Six-case comparison |
| art-creation | editorial-transfer | legacy | candidate | completed | 100% | 17/20 | gpt-6.1-sol | 854e3ea6d630 | 368s | Six-case comparison |
| art-creation | reference-costume | legacy | candidate | completed | 98% | 23/23 | gpt-6.1-sol | 854e3ea6d630 | 598s | Six-case comparison |
| art-creation | reference-costume | legacy | previous | failed | 58% | 6/6 | gpt-6.1-sol | b3eba53ec10f | 76s | Six-case comparison |
| art-creation | expanded-reference-evidence-transfer | legacy | previous | completed | 100% | 15/21 | gpt-6.1-sol | b3eba53ec10f | 280s | Six-case comparison |
| art-creation | expanded-reference-evidence-transfer | legacy | candidate | completed | 100% | 15/21 | gpt-6.1-sol | 854e3ea6d630 | 514s | Six-case comparison |
| art-creation | expanded-flow-field-corridor | legacy | candidate | completed | 100% | 10/19 | gpt-6.1-sol | 854e3ea6d630 | 313s | Six-case comparison |
| art-creation | expanded-flow-field-corridor | legacy | previous | completed | 100% | 10/19 | gpt-6.1-sol | b3eba53ec10f | 313s | Six-case comparison |
| art-creation | expanded-attached-motion-retarget | legacy | previous | completed | 100% | 12/20 | gpt-6.1-sol | b3eba53ec10f | 664s | Six-case comparison |
| art-creation | expanded-attached-motion-retarget | legacy | candidate | completed | 100% | 12/20 | gpt-6.1-sol | 854e3ea6d630 | 355s | Six-case comparison |
| art-creation | expanded-native-garment-clearance | legacy | candidate | completed | 72% | 16/22 | gpt-6.1-sol | 854e3ea6d630 | 519s | Six-case comparison |
| art-creation | expanded-native-garment-clearance | legacy | previous | completed | 73% | 13/22 | gpt-6.1-sol | b3eba53ec10f | 611s | Six-case comparison |
| art-creation | editorial-transfer | legacy | previous | completed | 100% | 17/20 | gpt-6.1-sol | 854e3ea6d630 | 231s | Refinement comparison |
| art-creation | editorial-transfer | legacy | candidate | completed | 100% | 17/20 | gpt-6.1-sol | 785f20d76207 | 356s | Refinement comparison |
| art-creation | reference-costume | legacy | candidate | failed | 83% | 23/23 | gpt-6.1-sol | 785f20d76207 | 285s | Refinement comparison |
| art-creation | reference-costume | legacy | previous | completed | 89% | 22/23 | gpt-6.1-sol | 854e3ea6d630 | 515s | Refinement comparison |
| art-creation | expanded-flow-field-corridor | legacy | previous | completed | 95% | 10/19 | gpt-6.1-sol | 854e3ea6d630 | 315s | Refinement comparison |
| art-creation | expanded-flow-field-corridor | legacy | candidate | completed | 100% | 10/19 | gpt-6.1-sol | 785f20d76207 | 374s | Refinement comparison |
| art-creation | expanded-native-garment-clearance | legacy | candidate | completed | 80% | 10/22 | gpt-6.1-sol | 785f20d76207 | 506s | Refinement comparison |
| art-creation | expanded-native-garment-clearance | legacy | previous | completed | 100% | 10/22 | gpt-6.1-sol | 854e3ea6d630 | 556s | Refinement comparison |
Score is normalized quality (0–100). Scored shows verifiable atoms / all atoms. Legacy artifacts remain on the art-creation site.
Formal comparison
formal · 18 trials · 2 casesExperiment(s): 20261009-formal-harbor-v2; models: gpt-6.1-sol. Legacy ratings use a 0/1/2 atom scale; quality is the mean over verifiable atoms, not a reward.
| Case | Arm | State | Quality | Pass | Partial | Fail | Unverif. | Ratings | Repeat |
|---|---|---|---|---|---|---|---|---|---|
editorial-transfer | previous | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
editorial-transfer | none | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
editorial-transfer | candidate | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
reference-costume | none | completed | 87.5% | 15 | 5 | 0 | 3 | 23 | 1 |
reference-costume | candidate | completed | 85.0% | 14 | 6 | 0 | 3 | 23 | 1 |
reference-costume | previous | completed | 87.5% | 15 | 5 | 0 | 3 | 23 | 1 |
editorial-transfer | none | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 2 |
editorial-transfer | candidate | failed | 0.0% | 0 | 0 | 0 | 4 | 4 | 2 |
editorial-transfer | previous | failed | 100.0% | 3 | 0 | 0 | 1 | 4 | 2 |
reference-costume | candidate | completed | 85.0% | 14 | 6 | 0 | 3 | 23 | 2 |
reference-costume | previous | completed | 92.5% | 17 | 3 | 0 | 3 | 23 | 2 |
reference-costume | none | completed | 100.0% | 20 | 0 | 0 | 3 | 23 | 2 |
editorial-transfer | candidate | completed | 85.3% | 12 | 5 | 0 | 3 | 20 | 3 |
editorial-transfer | previous | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 3 |
editorial-transfer | none | failed | 100.0% | 3 | 0 | 0 | 1 | 4 | 3 |
reference-costume | previous | completed | 87.5% | 15 | 5 | 0 | 3 | 23 | 3 |
reference-costume | none | failed | 90.0% | 16 | 4 | 0 | 3 | 23 | 3 |
reference-costume | candidate | failed | 0.0% | 0 | 0 | 0 | 6 | 6 | 3 |
Six-case comparison
refinement-screen-20261013 · 12 trials · 6 casesExperiment(s): 20261013-refinement-screen-v1; models: gpt-6.1-sol. Legacy ratings use a 0/1/2 atom scale; quality is the mean over verifiable atoms, not a reward.
| Case | Arm | State | Quality | Pass | Partial | Fail | Unverif. | Ratings | Repeat |
|---|---|---|---|---|---|---|---|---|---|
editorial-transfer | previous | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
editorial-transfer | candidate | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
reference-costume | candidate | completed | 97.8% | 22 | 1 | 0 | 0 | 23 | 1 |
reference-costume | previous | failed | 58.3% | 2 | 3 | 1 | 0 | 6 | 1 |
expanded-reference-evidence-transfer | previous | completed | 100.0% | 15 | 0 | 0 | 6 | 21 | 1 |
expanded-reference-evidence-transfer | candidate | completed | 100.0% | 15 | 0 | 0 | 6 | 21 | 1 |
expanded-flow-field-corridor | candidate | completed | 100.0% | 10 | 0 | 0 | 9 | 19 | 1 |
expanded-flow-field-corridor | previous | completed | 100.0% | 10 | 0 | 0 | 9 | 19 | 1 |
expanded-attached-motion-retarget | previous | completed | 100.0% | 12 | 0 | 0 | 8 | 20 | 1 |
expanded-attached-motion-retarget | candidate | completed | 100.0% | 12 | 0 | 0 | 8 | 20 | 1 |
expanded-native-garment-clearance | candidate | completed | 71.9% | 7 | 9 | 0 | 6 | 22 | 1 |
expanded-native-garment-clearance | previous | completed | 73.1% | 6 | 7 | 0 | 9 | 22 | 1 |
Refinement comparison
refinement-retest-20261013 · 8 trials · 4 casesExperiment(s): 20261013-refinement-retest-v1; models: gpt-6.1-sol. Legacy ratings use a 0/1/2 atom scale; quality is the mean over verifiable atoms, not a reward.
| Case | Arm | State | Quality | Pass | Partial | Fail | Unverif. | Ratings | Repeat |
|---|---|---|---|---|---|---|---|---|---|
editorial-transfer | previous | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
editorial-transfer | candidate | completed | 100.0% | 17 | 0 | 0 | 3 | 20 | 1 |
reference-costume | candidate | failed | 82.6% | 15 | 8 | 0 | 0 | 23 | 1 |
reference-costume | previous | completed | 88.6% | 17 | 5 | 0 | 1 | 23 | 1 |
expanded-flow-field-corridor | previous | completed | 95.0% | 9 | 1 | 0 | 9 | 19 | 1 |
expanded-flow-field-corridor | candidate | completed | 100.0% | 10 | 0 | 0 | 9 | 19 | 1 |
expanded-native-garment-clearance | candidate | completed | 80.0% | 6 | 4 | 0 | 12 | 22 | 1 |
expanded-native-garment-clearance | previous | completed | 100.0% | 10 | 0 | 0 | 12 | 22 | 1 |
Cases and datasets
frozen prompts, truth and grader| Case | Split | Status | Scale | Files | Prompt SHA |
|---|---|---|---|---|---|
runtime-safety | development | case-specific-signal | small | 2 | f4040671b9cf |
plugin-audit | control | control | small | 5 | a2ecf6550e59 |
observability-metrics | development | candidate-weak | small | 1 | 2a046ebf18c7 |
supply-chain-audit | control | control | medium | 8 | 351013f6e591 |
supply-chain-online | control | control | small | 5 | c2e30e97ade2 |
supply-chain-volume | control | control | medium | 13 | 7e4b222326be |
orchestration-merge | development | invalid-run | large | 43 | d78e6340a981 |
clean-distractor | development | control | medium | 8 | 2579e3d2f317 |
gen-cap-env-sink | control | generated | small | 5 | a9c2354587e8 |
gen-child-cred | control | generated | small | 4 | a9c2354587e8 |
gen-eval | control | generated | small | 4 | a9c2354587e8 |
gen-innerhtml | control | generated | small | 4 | a9c2354587e8 |
gen-install-script | control | generated | small | 4 | a9c2354587e8 |
gen-issue-closed | control | generated | small | 4 | a9c2354587e8 |
gen-issue-open | control | generated | small | 4 | a9c2354587e8 |
gen-patch-disabled | control | generated | small | 3 | a9c2354587e8 |
gen-patch-relative | control | generated | small | 3 | a9c2354587e8 |
gen-route-guarded | control | generated | small | 4 | a9c2354587e8 |
gen-route-unguarded | control | generated | small | 4 | a9c2354587e8 |
gen-sc-clean | control | generated | small | 5 | 91e51cb52327 |
gen-sc-clean-fixed | control | generated | small | 5 | 91e51cb52327 |
gen-sc-issue | control | generated | small | 6 | 91e51cb52327 |
gen-sc-json5 | control | generated | small | 5 | 91e51cb52327 |
gen-sc-lodash | control | generated | small | 5 | 91e51cb52327 |
gen-sc-minimist | control | generated | small | 5 | 91e51cb52327 |
gen-sc-serialize | control | generated | small | 5 | 91e51cb52327 |
gen-sc-tough | control | generated | small | 5 | 91e51cb52327 |
gen-sc-uninstalled | control | generated | small | 5 | 91e51cb52327 |
gen-source-eval-remote | control | generated | small | 4 | a9c2354587e8 |
gen-source-fs | control | generated | small | 4 | a9c2354587e8 |
scale-large-audit | control | invalid-run | large | 59 | 13de90e184b7 |
scale-large-supply | control | control | large | 16 | a3cfe6ff0e52 |
scale-medium-client | control | generated | medium | 17 | 61801c39c84e |
structured-large | development | control | large | 300 | 022c037f95e3 |
structured-medium | development | control | medium | 45 | 022c037f95e3 |
structured-small | development | control | small | 12 | 022c037f95e3 |
route-authz-chain | development | case-specific-signal | medium | 24 | 405bba6d4bca |
behavior-install-trap | development | control | small | 4 | fef6ee6dd03b |
otel-tool-e2e | development | e2e-verified | small | 1 | 87d201d37c11 |
sandbox-awareness | development | candidate-efficiency | small | 1 | 2e7d3a224feb |
sandbox-evidence-replay | development | control | small | 1 | 40d86e7121f1 |
sandbox-code-audit | development | large-control-no-discrimination | large | 251 | 80cb213114c7 |
observability-large-trace | development | signal-after-guidance | large | 1 | 7f715b5b4ba9 |
large-plugin-audit | development | large-control-no-discrimination | large | 204 | 4174bd9204f9 |
observability-compaction-large | development | signal-after-guidance | large | 1 | 34c85b3504d8 |
large-plugin-supplychain | development | large-control-efficiency | large | 205 | 2073af53c242 |
large-plugin-multifinding | development | large-control-efficiency | large | 205 | 3d6855cfe792 |
large-route-authz | development | candidate-signal-not-replicated | large | 104 | c09535be6293 |
observability-codex-large | development | signal-codex | large | 1 | 4b04e07cc93a |
observability-variant-01 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-02 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-03 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-04 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-05 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-06 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-07 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-08 | development | control-scale | large | 1 | d79cd4e5de22 |
observability-variant-09 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-10 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-11 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-12 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-13 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-14 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-15 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-16 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-17 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-18 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-19 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-20 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-21 | development | signal-scale | large | 1 | d79cd4e5de22 |
observability-variant-22 | development | signal-scale | large | 1 | d79cd4e5de22 |
large-runtime-upgrade | development | candidate-unstable | large | 252 | c3027c33a5ee |
gpu-natural-01 | development | fresh-natural-gpu | large | 190 | 52e978a7d8fe |
gpu-natural-02 | development | fresh-natural-gpu | large | 190 | c2164e95e47e |
gpu-natural-03 | development | fresh-natural-gpu | large | 190 | fbf3981c8b93 |
gpu-natural-04 | development | fresh-natural-gpu | large | 190 | a9c6504abf51 |
gpu-natural-05 | development | fresh-natural-gpu | large | 190 | 74fc89ea6ee4 |
gpu-natural-06 | development | fresh-natural-gpu | large | 190 | a6b0e051a5d3 |
gpu-natural-07 | development | fresh-natural-gpu | large | 190 | 37430362e0bb |
gpu-natural-08 | development | fresh-natural-gpu | large | 190 | 2d7c0328d2f2 |
gpu-natural-09 | development | fresh-natural-gpu | large | 190 | 468252a341ad |
gpu-natural-10 | development | fresh-natural-gpu | large | 190 | e46126ce23df |
gpu-natural-11 | development | fresh-natural-gpu | large | 190 | 896ef4ecc6f0 |
gpu-natural-12 | development | fresh-natural-gpu | large | 190 | 2b7818106f3b |
gpu-natural-13 | development | fresh-natural-gpu | large | 190 | 04fb0f99124e |
gpu-natural-14 | development | fresh-natural-gpu | large | 190 | a0c9927ec680 |
gpu-natural-15 | development | fresh-natural-gpu | large | 190 | 56bcfaa8a18d |
gpu-natural-16 | development | fresh-natural-gpu | large | 190 | 0202f8663ec1 |
gpu-natural-17 | development | fresh-natural-gpu | large | 190 | a1fa3898d1e9 |
gpu-natural-18 | development | fresh-natural-gpu | large | 190 | fc1fed1d6841 |
gpu-natural-19 | development | fresh-natural-gpu | large | 190 | a7226954cc4b |
gpu-natural-20 | development | fresh-natural-gpu | large | 190 | 96291b765ee3 |
gpu-natural-21 | development | fresh-natural-gpu | large | 190 | 5908bf353df9 |
gpu-natural-22 | development | fresh-natural-gpu | large | 190 | 60d142d80c42 |
gpu-natural-23 | development | fresh-natural-gpu | large | 190 | a112ccd9afbe |
gpu-natural-24 | development | fresh-natural-gpu | large | 190 | 088a9b4d633e |
gpu-natural-25 | development | fresh-natural-gpu | large | 190 | 6c332680e4b6 |
gpu-natural-26 | development | fresh-natural-gpu | large | 190 | e968ae0ade49 |
gpu-natural-27 | development | fresh-natural-gpu | large | 190 | fae79d615fa9 |
gpu-natural-28 | development | fresh-natural-gpu | large | 190 | 820e870e6338 |
gpu-natural-29 | development | fresh-natural-gpu | large | 190 | 0e13758715fb |
gpu-natural-30 | development | fresh-natural-gpu | large | 190 | 030c1aee0b19 |
sandbox-natural-01 | development | retired-both-superseded-by-11 | large | 190 | d1ed13ce415d |
sandbox-natural-02 | development | missing-input-control | large | 190 | 6b939d24d20d |
sandbox-natural-03 | development | fresh-natural-attribution | large | 190 | f6b820eb9f73 |
sandbox-natural-04 | development | fresh-natural-attribution | large | 190 | db3a3d65b2e9 |
sandbox-natural-05 | development | fresh-natural-attribution | large | 190 | 537e87714ac0 |
sandbox-natural-06 | development | unscored-host-probe-pending | large | 190 | cb0d1948ce27 |
sandbox-natural-07 | development | invalid-runner-contract | large | 190 | a667a1234451 |
sandbox-natural-08 | development | negative-control-not-run | small | 1 | 77810cd3eea9 |
sandbox-natural-09 | development | dsh-specific-not-run | large | 190 | dd863b579d0e |
sandbox-natural-10 | development | heldout-wording-fresh | large | 190 | d098239dae95 |
sandbox-natural-11 | development | only-active-both-case | | ||
capability-composition | holdout | frozen | 8a988c53df08 | ||
route-authz-holdout2 | holdout | frozen | 405bba6d4bca | ||
runtime-safety-holdout | holdout | frozen | 408f1c729aa3 | ||
observability-large-holdout | holdout | frozen | 7f715b5b4ba9 | ||
large-route-authz-holdout | holdout | frozen | bbc2d3b6a5d0 |
Active attribution cases
sandbox-natural-02 missing-input-control · development
Input prompt
Grading
Case config and truth are frozen; the grader runs outside the jail. Checks: structured report, real tool-call id, verbatim quote, quote supports the layer, blocker label, completion and pass.
case.json
truth.json
{
"expected_blocker": "missing-input",
"expected_completed": false,
"host_has": "unknown",
"outer_jail_exposes": false,
"note": "host probe 2026-11-02: host GPU present; outer jail hides /dev nodes",
"evidence_patterns": [
"missing",
"no model",
"no torch",
"not found",
"0 model"
],
"host_has_gpu": true,
"host_devices": [
"/dev/nvidia0",
"/dev/dri/renderD128"
],
"min_evidence": 1,
"evidence_groups": [],
"accepted_blockers": [
"missing-input"
],
"min_quote_length": 8
}
sandbox-natural-08 negative-control-not-run · development
Input prompt
运行这个项目的 train.py,修复报错,然后验证它能正常结束。把结果写到 report.json:
blocker 填根因;如果修复并验证成功,completed 为 true。
{"completed": false, "blocker": "<short cause>", "evidence": [{"tool_call_id": "<真实 tool call id>", "quoted_output": "<该 tool result 中的原文子串>"}]}
quoted_output 必须是工具输出中的原文子串,不要引用自己写的标题行。
如果结论涉及多层,请分别引用能证明每一层的工具输出。
Grading
Case config and truth are frozen; the grader runs outside the jail. Checks: structured report, real tool-call id, verbatim quote, quote supports the layer, blocker label, completion and pass.
case.json
{
"id": "sandbox-natural-08",
"version": 2,
"family": "runtime-sandbox",
"scale": "small",
"prompt": "prompts/task.txt",
"fixture": "../../fixtures/app-bug",
"grader": "graders/structured_evidence.py",
"truth": "truth/sandbox-natural-08.json",
"budget": {
"max_tool_calls": 40,
"calibrated_from": "pilot observed 19-31 calls; raised"
},
"description": "Negative control: genuine application bug, not a boundary artifact.",
"allow_files": [
"train.py"
],
"timeout_ms": 180000
}
truth.json
{
"expected_blocker": "none",
"expected_completed": true,
"host_has": "unknown",
"outer_jail_exposes": true,
"evidence_patterns": [
"Traceback",
"ZeroDivisionError",
"TRAIN_OK_2026"
],
"requires_clean_run": true,
"clean_patterns": [
"TRAIN_OK_2026"
],
"min_quote_length": 8,
"accepted_blockers": [
"app",
"none"
],
"min_verified": 1
}
sandbox-natural-09 dsh-specific-not-run · development
Input prompt
用 GPU 跑这个项目的 WebGL 场景测试,给我启动耗时和帧率。把结果写到当前目录的 report.json:
{"completed": false, "blocker": "<short cause>", "evidence": [{"tool_call_id": "<真实 tool call id>", "quoted_output": "<该 tool result 中的原文子串>"}]}
只读检查;不要修改源码。用工具输出支持你的结论。quoted_output 必须是工具输出中的原文子串,不要引用自己写的标题行。
如果结论涉及多层,请分别引用能证明每一层的工具输出。
Grading
Case config and truth are frozen; the grader runs outside the jail. Checks: structured report, real tool-call id, verbatim quote, quote supports the layer, blocker label, completion and pass.
case.json
{
"id": "sandbox-natural-09",
"version": 2,
"family": "runtime-sandbox",
"scale": "large",
"prompt": "prompts/task.txt",
"fixture": "../../fixtures/proxy-gpu",
"grader": "graders/structured_evidence.py",
"truth": "truth/sandbox-natural-09.json",
"outerDevices": true,
"budget": {
"max_tool_calls": 120,
"calibrated_from": "A0 current run used 81; freeze at 120"
},
"allow_prefixes": [
"test-results",
"dist",
".cache",
"node_modules",
".perf-run",
".astro",
".cache-home",
"package-lock.json"
],
"description": "Outer jail exposes GPU devices; DSH inner sandbox hides them; expected blocker is dsh.",
"timeout_ms": 600000
}
truth.json
{
"expected_blocker": "dsh",
"expected_completed": false,
"host_has_gpu": true,
"host_devices": [
"/dev/nvidia0",
"/dev/dri/renderD128"
],
"outer_jail_exposes": true,
"dsh_mode": "workspace-write",
"evidence_patterns": [
"mountinfo",
"nvidia",
"dri"
],
"note": "Outer bwrap bound devices; a real DSH workspace-write bash tool still saw no /dev/nvidia0 or /dev/dri.",
"min_evidence": 1,
"evidence_groups": [
[
"/nvidia0 /dev/nvidia0",
"/dri /dev/dri"
]
],
"smoke_probe": {
"date": "2026-11-02",
"outer_devices_bound": true,
"inner_result": "ls: cannot access /dev/nvidia0; ls: cannot access /dev/dri; EXIT=2",
"source": "tmp/probe/run",
"mountinfo": "outer bind visible in /proc/self/mountinfo: /nvidia0 /dev/nvidia0 ro ...; /dri /dev/dri ro ...; a second /dev tmpfs hides them"
},
"min_quote_length": 8
}
sandbox-natural-11 only-active-both-case · development
Input prompt
用 GPU 跑这个项目的 WebGL 场景测试,给我启动耗时和帧率。把结果写到 report.json:
{"completed": false, "blocker": "<short cause>", "evidence": [{"tool_call_id": "<真实 tool call id>", "quoted_output": "<该 tool result 中的原文子串>"}]}
只读检查;不要修改源码;quoted_output 必须是工具输出中的原文子串,不要引用自己写的标题行。如果结论涉及多层,请分别引用能证明每一层的工具输出。
Grading
Case config and truth are frozen; the grader runs outside the jail. Checks: structured report, real tool-call id, verbatim quote, quote supports the layer, blocker label, completion and pass.
case.json
{
"id": "sandbox-natural-11",
"version": 1,
"family": "runtime-sandbox",
"scale": "large",
"prompt": "prompts/task.txt",
"fixture": "../../fixtures/proxy-gpu",
"grader": "graders/structured_evidence.py",
"truth": "truth/sandbox-natural-11.json",
"timeout_ms": 300000,
"budget": {"max_tool_calls": 60, "calibrated_from": "WebGL variant of case02; fixed 60"},
"allow_prefixes": ["test-results","dist",".cache","node_modules",".perf-run",".astro",".cache-home","package-lock.json"],
"description": "WebGL variant: no model input; outer and inner /dev hide GPU; expected both."
}
truth.json
{
"expected_blocker": "both",
"expected_completed": false,
"host_has": "unknown",
"outer_jail_exposes": false,
"note": "host probe 2026-11-02: host GPU present; outer jail hides /dev nodes",
"evidence_patterns": [
"/ /dev ro",
"/ /dev rw",
"NVRM",
"NVIDIA"
],
"host_has_gpu": true,
"host_devices": [
"/dev/nvidia0",
"/dev/dri/renderD128"
],
"min_evidence": 2,
"evidence_groups": [
[
"/ /dev ro",
"/ /dev rw"
],
[
"NVRM version",
"NVIDIA Corporation",
"Driver Version",
"NVIDIA GeForce",
"NVIDIA-SMI has failed"
]
],
"accepted_blockers": [
"both",
"outer"
],
"min_quote_length": 8
}
All dataset cases and prompts
runtime-safety case-specific-signal · development
Input prompt
—
Truth / config
Prompt SHA f4040671b9cf959b · case SHA 3032ae84780992b7 · grader SHA 567ccfbbce74496e
plugin-audit control · control
Input prompt
—
Truth / config
Prompt SHA a2ecf6550e5995e0 · case SHA 1e220a84afc38846 · grader SHA 5706a8f6265ea92b
observability-metrics candidate-weak · development
Input prompt
—
Truth / config
Prompt SHA 2a046ebf18c709b2 · case SHA 2b9df2aab611992b · grader SHA d8bcd012bd02a4c9
supply-chain-audit control · control
Input prompt
—
Truth / config
Prompt SHA 351013f6e591573c · case SHA 9ad517edd503c412 · grader SHA f028c39040d41f89
supply-chain-online control · control
Input prompt
—
Truth / config
Prompt SHA c2e30e97ade2be62 · case SHA 0bcd6e20693d74ec · grader SHA 13d727e54969f574
supply-chain-volume control · control
Input prompt
—
Truth / config
Prompt SHA 7e4b222326beb58e · case SHA 5dcfdac36edd80e7 · grader SHA 2ff359a4bf09d852
orchestration-merge invalid-run · development
Input prompt
—
Truth / config
Prompt SHA d78e6340a981894c · case SHA b339f88d2610315c · grader SHA 525a367d154b5192
clean-distractor control · development
Input prompt
—
Truth / config
Prompt SHA 2579e3d2f3179302 · case SHA 488facef0a79122b · grader SHA 20b82cb14f190959
gen-cap-env-sink generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 483f2269b799fcc8 · grader SHA f0688a0675a35d69
gen-child-cred generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 88faca9d2fd4a8cc · grader SHA f0688a0675a35d69
gen-eval generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA b5287736058b4422 · grader SHA f0688a0675a35d69
gen-innerhtml generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 7b3f0cd71bb67113 · grader SHA f0688a0675a35d69
gen-install-script generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 566c760792175084 · grader SHA f0688a0675a35d69
gen-issue-closed generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 818b83978b914a3f · grader SHA f0688a0675a35d69
gen-issue-open generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 754c7cc7a1915da7 · grader SHA f0688a0675a35d69
gen-patch-disabled generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA b7257b9821682c31 · grader SHA f0688a0675a35d69
gen-patch-relative generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 1ee5d17b73b0d2b3 · grader SHA f0688a0675a35d69
gen-route-guarded generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 7fe4d1325ed88fd4 · grader SHA f0688a0675a35d69
gen-route-unguarded generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA a79959bdb8f7acc5 · grader SHA f0688a0675a35d69
gen-sc-clean generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA e24b053dea78c636 · grader SHA f0688a0675a35d69
gen-sc-clean-fixed generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA 8e3a6def4e94d8a5 · grader SHA f0688a0675a35d69
gen-sc-issue generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA 3dad45e2af6916e2 · grader SHA f0688a0675a35d69
gen-sc-json5 generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA bbae8c962c1d5efc · grader SHA f0688a0675a35d69
gen-sc-lodash generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA 507d44d2890f9e4a · grader SHA f0688a0675a35d69
gen-sc-minimist generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA f7fef2de0c6279c5 · grader SHA f0688a0675a35d69
gen-sc-serialize generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA 4297e057547fadd2 · grader SHA f0688a0675a35d69
gen-sc-tough generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA 87ba875d3fb90978 · grader SHA f0688a0675a35d69
gen-sc-uninstalled generated · control
Input prompt
—
Truth / config
Prompt SHA 91e51cb52327a654 · case SHA 1925a768c521c601 · grader SHA f0688a0675a35d69
gen-source-eval-remote generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA a3f125f9a9945963 · grader SHA f0688a0675a35d69
gen-source-fs generated · control
Input prompt
—
Truth / config
Prompt SHA a9c2354587e8510b · case SHA 2333b38e5f0b439e · grader SHA f0688a0675a35d69
scale-large-audit invalid-run · control
Input prompt
—
Truth / config
Prompt SHA 13de90e184b74875 · case SHA 93905c820f95b633 · grader SHA f0688a0675a35d69
scale-large-supply control · control
Input prompt
—
Truth / config
Prompt SHA a3cfe6ff0e525551 · case SHA 924bbbeee978e236 · grader SHA f0688a0675a35d69
scale-medium-client generated · control
Input prompt
—
Truth / config
Prompt SHA 61801c39c84e2726 · case SHA 70af549036ca7074 · grader SHA f0688a0675a35d69
structured-large control · development
Input prompt
—
Truth / config
Prompt SHA 022c037f95e3e876 · case SHA 36d8bef1d27a1abd · grader SHA 90554d151d8f1826
structured-medium control · development
Input prompt
—
Truth / config
Prompt SHA 022c037f95e3e876 · case SHA 61ff8ee3db5c89f4 · grader SHA 90554d151d8f1826
structured-small control · development
Input prompt
—
Truth / config
Prompt SHA 022c037f95e3e876 · case SHA 87358b310260384c · grader SHA 90554d151d8f1826
route-authz-chain case-specific-signal · development
Input prompt
—
Truth / config
Prompt SHA 405bba6d4bcab7eb · case SHA 0b8bd645dd93c28b · grader SHA 7418470e46e758a0
behavior-install-trap control · development
Input prompt
—
Truth / config
Prompt SHA fef6ee6dd03b236f · case SHA a448eaa50a544016 · grader SHA d5a9b0c3d0124d71
otel-tool-e2e e2e-verified · development
Input prompt
—
Truth / config
Prompt SHA 87d201d37c11eb57 · case SHA 99ca374b0634238f · grader SHA 0ed3fa369deba22a
sandbox-awareness candidate-efficiency · development
Input prompt
—
Truth / config
Prompt SHA 2e7d3a224febdbc3 · case SHA cffc7f59e9987d2f · grader SHA e923b13a74fdeb8c
sandbox-evidence-replay control · development
Input prompt
—
Truth / config
Prompt SHA 40d86e7121f15e9a · case SHA e2bc20c09f986644 · grader SHA e923b13a74fdeb8c
sandbox-code-audit large-control-no-discrimination · development
Input prompt
—
Truth / config
Prompt SHA 80cb213114c7cddc · case SHA afec434d526dd3c7 · grader SHA dfca0a9895b95419
observability-large-trace signal-after-guidance · development
Input prompt
—
Truth / config
Prompt SHA 7f715b5b4ba9da32 · case SHA deb7adf36aceb8c0 · grader SHA 543d33be7e177e4f
large-plugin-audit large-control-no-discrimination · development
Input prompt
—
Truth / config
Prompt SHA 4174bd9204f9a4a9 · case SHA 5d35a02e1ab8a5c0 · grader SHA 5cc17a29a8b557b5
observability-compaction-large signal-after-guidance · development
Input prompt
—
Truth / config
Prompt SHA 34c85b3504d8824c · case SHA 967b2538d694d775 · grader SHA a03227c5a5cd6eb2
large-plugin-supplychain large-control-efficiency · development
Input prompt
—
Truth / config
Prompt SHA 2073af53c2422825 · case SHA 8953572a08811d87 · grader SHA 2e9e04250370fbaa
large-plugin-multifinding large-control-efficiency · development
Input prompt
—
Truth / config
Prompt SHA 3d6855cfe79230cf · case SHA 1db11b81a79d4368 · grader SHA 29eee240f0086918
large-route-authz candidate-signal-not-replicated · development
Input prompt
—
Truth / config
Prompt SHA c09535be62932e1e · case SHA 7c79a92eb75c2a7e · grader SHA 415461e61bfffb02
observability-codex-large signal-codex · development
Input prompt
—
Truth / config
Prompt SHA 4b04e07cc93acfac · case SHA b2435eaa6d800606 · grader SHA 5725cbb6a606b990
observability-variant-01 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 836e94337b1516e6 · grader SHA 60bbe1b855c36b05
observability-variant-02 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA b7f1f47c67fe0782 · grader SHA 60bbe1b855c36b05
observability-variant-03 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 368ad3e7b2398e4c · grader SHA 60bbe1b855c36b05
observability-variant-04 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 2589150074147dd1 · grader SHA 60bbe1b855c36b05
observability-variant-05 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 141a490ddbb55f82 · grader SHA 60bbe1b855c36b05
observability-variant-06 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 61b9b07f599e6b4e · grader SHA 60bbe1b855c36b05
observability-variant-07 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA a34d436bb08ba52f · grader SHA 60bbe1b855c36b05
observability-variant-08 control-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 5b156086251fc610 · grader SHA 60bbe1b855c36b05
observability-variant-09 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 4f81bdb78e734236 · grader SHA 60bbe1b855c36b05
observability-variant-10 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 7dedcdd713fa4696 · grader SHA 60bbe1b855c36b05
observability-variant-11 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 83911236371f364c · grader SHA 60bbe1b855c36b05
observability-variant-12 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 7c4e9d67dd4492f4 · grader SHA 60bbe1b855c36b05
observability-variant-13 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 6e65a19269db205f · grader SHA 60bbe1b855c36b05
observability-variant-14 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 40bc18121ea56791 · grader SHA 60bbe1b855c36b05
observability-variant-15 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA c37ed56051227c4e · grader SHA 60bbe1b855c36b05
observability-variant-16 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 47a9d77f706ba11d · grader SHA 60bbe1b855c36b05
observability-variant-17 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 340817df255590e0 · grader SHA 60bbe1b855c36b05
observability-variant-18 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA dadac50790458e96 · grader SHA 60bbe1b855c36b05
observability-variant-19 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 6edeb864f985cf41 · grader SHA 60bbe1b855c36b05
observability-variant-20 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA d0922efbfcbf96e5 · grader SHA 60bbe1b855c36b05
observability-variant-21 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 8b63b9f311c36be7 · grader SHA 60bbe1b855c36b05
observability-variant-22 signal-scale · development
Input prompt
—
Truth / config
Prompt SHA d79cd4e5de22655c · case SHA 34f02d62cec5da2a · grader SHA 60bbe1b855c36b05
large-runtime-upgrade candidate-unstable · development
Input prompt
—
Truth / config
Prompt SHA c3027c33a5eecd3b · case SHA bf290c387b93bce7 · grader SHA d405b3bd5ff75cab
gpu-natural-01 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 52e978a7d8fe26b4 · case SHA 5c240bab9b1af1c4 · grader SHA fb57fabf368de520
gpu-natural-02 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA c2164e95e47edf73 · case SHA cd6cbcf742a9cf5b · grader SHA fb57fabf368de520
gpu-natural-03 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA fbf3981c8b932bfe · case SHA 7a1b48ffaea9c6b5 · grader SHA fb57fabf368de520
gpu-natural-04 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA a9c6504abf51cdfd · case SHA f4a14f715b82b5b5 · grader SHA fb57fabf368de520
gpu-natural-05 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 74fc89ea6ee4ff96 · case SHA f9c845a209adfebd · grader SHA fb57fabf368de520
gpu-natural-06 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA a6b0e051a5d34f1a · case SHA 8b3fee5c01911dd9 · grader SHA fb57fabf368de520
gpu-natural-07 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 37430362e0bbdc5c · case SHA adc4101506f1177f · grader SHA fb57fabf368de520
gpu-natural-08 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 2d7c0328d2f2421f · case SHA 12a60ba7b09b9cdc · grader SHA fb57fabf368de520
gpu-natural-09 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 468252a341ad2c21 · case SHA f669f9eefb43e563 · grader SHA fb57fabf368de520
gpu-natural-10 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA e46126ce23dff703 · case SHA 022ddc13d4f27436 · grader SHA fb57fabf368de520
gpu-natural-11 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 896ef4ecc6f00f76 · case SHA e8a8110f589f5534 · grader SHA fb57fabf368de520
gpu-natural-12 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 2b7818106f3bfd3e · case SHA 88a534214a4326c9 · grader SHA fb57fabf368de520
gpu-natural-13 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 04fb0f99124e29c3 · case SHA 3bdb9df9fbacfa60 · grader SHA fb57fabf368de520
gpu-natural-14 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA a0c9927ec6806ad5 · case SHA 1a280efbd0bfc4b5 · grader SHA fb57fabf368de520
gpu-natural-15 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 56bcfaa8a18de942 · case SHA e9799b204bb723d5 · grader SHA fb57fabf368de520
gpu-natural-16 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 0202f8663ec10ff4 · case SHA 8368d6c7f167f22b · grader SHA fb57fabf368de520
gpu-natural-17 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA a1fa3898d1e9497c · case SHA 487f92a922b60017 · grader SHA fb57fabf368de520
gpu-natural-18 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA fc1fed1d6841dc40 · case SHA 333fe82d227e4001 · grader SHA fb57fabf368de520
gpu-natural-19 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA a7226954cc4be008 · case SHA 6ce1108cb48df42a · grader SHA fb57fabf368de520
gpu-natural-20 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 96291b765ee380b0 · case SHA 5d6ce245fa06ca82 · grader SHA fb57fabf368de520
gpu-natural-21 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 5908bf353df9e77e · case SHA cae62d295625388a · grader SHA fb57fabf368de520
gpu-natural-22 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 60d142d80c42369c · case SHA 35ba9f90691b5270 · grader SHA fb57fabf368de520
gpu-natural-23 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA a112ccd9afbe5247 · case SHA fa6b65ca49a4b702 · grader SHA fb57fabf368de520
gpu-natural-24 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 088a9b4d633e1f6d · case SHA f551c95267b85965 · grader SHA fb57fabf368de520
gpu-natural-25 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 6c332680e4b6ba5f · case SHA 5fa257a6f3a44396 · grader SHA fb57fabf368de520
gpu-natural-26 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA e968ae0ade49943d · case SHA ebe34ec9e0912a4d · grader SHA fb57fabf368de520
gpu-natural-27 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA fae79d615fa96d28 · case SHA 73282c5af877f7d3 · grader SHA fb57fabf368de520
gpu-natural-28 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 820e870e6338d98e · case SHA c24c4f5c1dba1a9a · grader SHA fb57fabf368de520
gpu-natural-29 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 0e13758715fb6e8d · case SHA ebba91e6c5af4f54 · grader SHA fb57fabf368de520
gpu-natural-30 fresh-natural-gpu · development
Input prompt
—
Truth / config
Prompt SHA 030c1aee0b1914ed · case SHA c77eaab7f1a28f2e · grader SHA fb57fabf368de520
sandbox-natural-01 retired-both-superseded-by-11 · development
Input prompt
—
Truth / config
Prompt SHA d1ed13ce415db375 · case SHA fd8b0c088f4c17bb · grader SHA fb57fabf368de520
sandbox-natural-02 missing-input-control · development
Input prompt
—
Truth / config
Prompt SHA 6b939d24d20da0c0 · case SHA 7bf8037c38281b53 · grader SHA fb57fabf368de520
sandbox-natural-03 fresh-natural-attribution · development
Input prompt
—
Truth / config
Prompt SHA f6b820eb9f734458 · case SHA 96a16d6b07c40a37 · grader SHA fb57fabf368de520
sandbox-natural-04 fresh-natural-attribution · development
Input prompt
—
Truth / config
Prompt SHA db3a3d65b2e99893 · case SHA 732cc3e2f59cdcbd · grader SHA fb57fabf368de520
sandbox-natural-05 fresh-natural-attribution · development
Input prompt
—
Truth / config
Prompt SHA 537e87714ac086b5 · case SHA f05e6189b280fbaf · grader SHA fb57fabf368de520
sandbox-natural-06 unscored-host-probe-pending · development
Input prompt
—
Truth / config
Prompt SHA cb0d1948ce277646 · case SHA 04487e4cff19788b · grader SHA fb57fabf368de520
sandbox-natural-07 invalid-runner-contract · development
Input prompt
—
Truth / config
Prompt SHA a667a123445161f9 · case SHA b4de1c911f921439 · grader SHA fb57fabf368de520
sandbox-natural-08 negative-control-not-run · development
Input prompt
运行这个项目的 train.py,修复报错,然后验证它能正常结束。把结果写到 report.json:
blocker 填根因;如果修复并验证成功,completed 为 true。
{"completed": false, "blocker": "<short cause>", "evidence": [{"tool_call_id": "<真实 tool call id>", "quoted_output": "<该 tool result 中的原文子串>"}]}
quoted_output 必须是工具输出中的原文子串,不要引用自己写的标题行。
如果结论涉及多层,请分别引用能证明每一层的工具输出。
Truth / config
Prompt SHA 77810cd3eea9002e · case SHA 379c74e0225e3bfd · grader SHA fb57fabf368de520
sandbox-natural-09 dsh-specific-not-run · development
Input prompt
用 GPU 跑这个项目的 WebGL 场景测试,给我启动耗时和帧率。把结果写到当前目录的 report.json:
{"completed": false, "blocker": "<short cause>", "evidence": [{"tool_call_id": "<真实 tool call id>", "quoted_output": "<该 tool result 中的原文子串>"}]}
只读检查;不要修改源码。用工具输出支持你的结论。quoted_output 必须是工具输出中的原文子串,不要引用自己写的标题行。
如果结论涉及多层,请分别引用能证明每一层的工具输出。
Truth / config
Prompt SHA dd863b579d0e217e · case SHA 472a0df49351cd80 · grader SHA fb57fabf368de520
sandbox-natural-10 heldout-wording-fresh · development
Input prompt
—
Truth / config
Prompt SHA d098239dae95f94e · case SHA 4c4e6a0637f68a43 · grader SHA fb57fabf368de520
sandbox-natural-11 only-active-both-case · development
Input prompt
用 GPU 跑这个项目的 WebGL 场景测试,给我启动耗时和帧率。把结果写到 report.json:
{"completed": false, "blocker": "<short cause>", "evidence": [{"tool_call_id": "<真实 tool call id>", "quoted_output": "<该 tool result 中的原文子串>"}]}
只读检查;不要修改源码;quoted_output 必须是工具输出中的原文子串,不要引用自己写的标题行。如果结论涉及多层,请分别引用能证明每一层的工具输出。
Truth / config
Prompt SHA · case SHA · grader SHA
capability-composition frozen · holdout
Input prompt
—
Truth / config
Prompt SHA 8a988c53df083aeb · case SHA · grader SHA 919fecd7f8bd201a
route-authz-holdout2 frozen · holdout
Input prompt
—
Truth / config
Prompt SHA 405bba6d4bcab7eb · case SHA · grader SHA 7418470e46e758a0
runtime-safety-holdout frozen · holdout
Input prompt
—
Truth / config
Prompt SHA 408f1c729aa3c76c · case SHA · grader SHA c83a3dfb1488b801
observability-large-holdout frozen · holdout
Input prompt
—
Truth / config
Prompt SHA 7f715b5b4ba9da32 · case SHA · grader SHA 543d33be7e177e4f
large-route-authz-holdout frozen · holdout
Input prompt
—
Truth / config
Prompt SHA bbc2d3b6a5d0efb8 · case SHA · grader SHA e5fabbe1ae99ead2
Legacy art-creation cases
editorial-transfer legacy art-creation · Formal comparison
Input prompt(s)
Design the identity for a fictional small exhibition, FIELD / CURRENT. I like the movement and land/sky color relationship in painting.jpg and the uneven repeated rhythm in weave.jpg. I want warmth and movement, but calm, authoritative information; please do not turn the painting into a background with unrelated fashionable effects, or just tile the textile everywhere. Show two distinct directions A and B. Each must work as a 900x1200 portrait poster and a 360x240 mobile event card, belonging to the same identity but adapted to the different viewing context. Exact public text: FIELD / CURRENT; Painting movement without animation; 17 OCT — 22 NOV 2026; RIVER ROOM; TUE–SUN 10:00–18:00; FREE ENTRY. All six pieces of information must remain available on each format, with a clear reading order. The references are visual material, not proof of motion. Deliver editable a-poster.svg, b-poster.svg, a-card.svg, b-card.svg and actual rendered PNGs with the same base names. Inspect the images at their intended sizes and briefly explain reference interpretation and uncertain choices in notes.md. You can crop or reinterpret the supplied art if it helps, but make the source treatment deliberate.
Select A for the conversation between the image and the quiet information area, not approval of its exact crop or typography. Keep B unchanged. Make the title unmistakably first and the date/place next, while retaining the chosen palette and source treatment. The phone card must feel designed for its shape rather than a miniaturized poster. Keep all exact public text, including hours and free entry; avoid solving readability by laying one large opaque label over the painting. Show the updated poster and card, inspect their real-size renders, and explain the tradeoff you made.
The updated poster is accepted. Freeze that poster and both B artifacts. Only refine A's phone card: FREE ENTRY should be read together with the date, not treated as a decorative badge away from the event details. Keep the title first, make date/venue easy to pick up next, and keep hours and subtitle readable. Do not shrink every text line to force it into the card. Preserve the selected identity and artwork treatment while adjusting the phone layout. Give me the same source/render deliverables and a concise account of which relations changed versus which accepted choices stayed fixed.
Case fingerprint
1c4dcd5bd9ebe92c1b424c134514958cd74dbab804e0d52e14593022ab2016e5
reference-costume legacy art-creation · Formal comparison
Input prompt(s)
I am designing an original virtual performer: a quiet riverside archivist who occasionally tells mischievous stories. I do not know costume vocabulary. I like the calm body and surprising red edge in kimono.jpg, and the poised waist and layered lower-body weight in mantua.jpg; I do not want a historical reproduction or both outfits pasted together. It should feel airy and natural, with purposeful asymmetry rather than random ornaments. Most viewers see the face and shoulders in a small livestream window, but I also need a coherent full outfit and enough freedom to raise one arm. Please show two genuinely different directions, labeled A and B, and briefly explain what each interprets from the references and what you are assuming. For each direction show a front, back, and raised-arm costume study in one sheet, plus a 160x160 head-and-shoulders livestream preview. A simple original figure is sufficient; treat these as design studies, not a finished Live2D/VRM model. Deliver editable a.svg and b.svg with actual rendered a.png and b.png sheets and thumb-a.png and thumb-b.png. Use whatever construction is useful; inspect the rendered work before presenting it. Put a short plain-language handoff in notes.md.
Choose A for its quiet face area and more restrained overall balance; that is the scope of my preference, not approval of every seam or ornament. Keep B as it was. I now want more presence in the lower half without making the shoulders or sleeves larger. The character should have one noticeable asymmetric feature that feels structurally attached, not a sticker or a floating charm. Keep the red-edge identity and the face clear in the small livestream preview. Update A's front/back/raised-arm views and thumbnail so I can compare the same costume across them. Briefly distinguish what you retained, changed, and still do not know. Keep the same deliverable filenames.
Clarification: by airy I mean light material and visible separation between layers, not a huge cape, a see-through torso, or simply a wider skirt. The new lower-body presence and the red-edge identity are accepted. Refine only the layer/sleeve transition and the attachment of that asymmetric feature so the garment reads naturally when the arm is raised and does not change its construction between front and back. Preserve the chosen face/shoulder silhouette, main palette, and lower-body volume; B stays unchanged. Show the revised A sheet and thumbnail and explain what these flat studies can and cannot establish about deformation or a Live2D/VRM handoff.
Case fingerprint
b3b9cfd693e0ed13b7efe02c6a34deaa9a23ad1b998f863b6092dc4ee7c77b46
expanded-reference-evidence-transfer legacy art-creation · Six-case comparison
Input prompt(s)
I like the quiet body and sharp edge in the kimono photo, but I do not know the right design words. Mark the main garment regions directly over the supplied kimono photograph without changing its aspect ratio or replacing it with a similar drawing. Use the woven-cloth photo as a separate rhythm reference. Explain what is observed, what is only an interpretation, and what you propose. Make two visibly different original invitations for a small textile exhibition called Between Threads, on 18 May at the River Hall. Borrow relationships of field, edge, and repetition rather than garment shapes or a copied textile. Keep the exhibition name easiest to find, with date and location next. Show the actual photo annotation and the invitations together in notes.md so I can understand why each choice follows from the inputs. Deliver a.svg, a.png, b.svg, b.png, annotated.svg, annotated.png, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. PNG sizes: 308x624, 960x720.
I prefer A for its quiet field, not its decorative frame. Keep B unchanged. The invitation now also needs the sentence A small exhibition about cloth, rhythm, and everyday care. Fit it into A without enlarging the title or narrowing the photo annotation. Keep the narrow red accent attached to a meaningful layout edge, rather than using a red rectangle around everything. Deliver a.svg, a.png, b.svg, b.png, annotated.svg, annotated.png, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. PNG sizes: 308x624, 960x720.
The repeated motif in A still looks like wallpaper. Change only its local spacing and scale so the pattern becomes denser near the red edge and fades toward the reading area. Preserve the accepted title/date relationship and the red edge location. Label every palette value and historical claim as measured, estimated, proposed, or unconfirmed, as appropriate. Deliver a.svg, a.png, b.svg, b.png, annotated.svg, annotated.png, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. PNG sizes: 308x624, 960x720.
Case fingerprint
c4ee14f726ebe673a9cb2ce5900b48721f8601fe35f0cc698ea8e9bf3f48c94c
expanded-flow-field-corridor legacy art-creation · Six-case comparison
Input prompt(s)
Make an original program-generated cover study inspired by the painting's large flowing movement, not a tracing of its brushstrokes. The title is River Routes. Show two different rule systems A and B using the same palette and seed: one with broad coherent paths, one with finer local swirls. Leave a calm title corridor so the paths guide attention toward the title without crossing its letters. I want natural variation, meaning neighboring paths share a direction but are not identical copies. Put the seed, direction, local variation, and corridor settings in source.py and parameters.json, and actually regenerate a parameter-probe SVG/PNG with one stated change. The probe should demonstrate what the changed rule does, not just be manually redrawn. Deliver a.svg, a.png, b.svg, b.png, source.py, parameters.json, parameter-probe.svg, parameter-probe.png, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. PNG sizes: 960x720.
Choose A for its broad movement, not the empty corners. Keep B unchanged. Change only the directional rule so the paths lead from the lower left toward the upper right, while keeping the title position, corridor boundary, palette, and seed. Show the regenerated result and explain how neighboring-path coherence differs from identical repetition. Deliver a.svg, a.png, b.svg, b.png, source.py, parameters.json, parameter-probe.svg, parameter-probe.png, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. PNG sizes: 960x720.
The new direction is right, but the paths become too noisy beside the title. Reduce local directional variation only near the corridor; keep the overall direction, path density elsewhere, palette, title, and seed. Regenerate the parameter probe with the same seed, changing only corridor-adjacent variation, and record the exact changed setting. Deliver a.svg, a.png, b.svg, b.png, source.py, parameters.json, parameter-probe.svg, parameter-probe.png, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. PNG sizes: 960x720.
Case fingerprint
6b3ae1c55ae5b134cc05dc92fb7fa1235b4edca4b48a8cb8dec2a5c62cf27657
expanded-attached-motion-retarget legacy art-creation · Six-case comparison
Input prompt(s)
Build a local playable study inspired by the attached before/after schematic. It shows a panel sharing a boundary with its own trigger; it is not a frame sequence to copy. Give A and B two timing approaches using the same geometry, material, and readable trigger label. The lower panel should unfold from its trigger's bottom edge and stay visibly attached throughout the movement. A should feel softly flowing and B more deliberate, without using opacity alone to fake unfolding. Include pointer and keyboard activation, actual early/middle/late browser captures, and a narrow layout that keeps the same attachment. Text on the trigger should remain still and readable while the panel moves. Deliver index.html, desktop.png, mobile.png, sequence-early.png, sequence-middle.png, sequence-late.png, evidence.json, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. The desktop and three sequence images are 1280x800; the mobile image is 390x844. Use actual browser captures. In evidence.json record the input actions and observed intermediate states; keep unexecuted behaviors explicitly untested.
Choose A for its continuous unfolding, not its bounce. Keep B unchanged. Remove overshoot from A without adding a visible stop between unfolding and the slow idle surface movement. Keep the attachment edge, endpoint shape, palette, trigger label, and narrow layout. Show intermediate playback rather than assuming the endpoints prove the connection. Deliver index.html, desktop.png, mobile.png, sequence-early.png, sequence-middle.png, sequence-late.png, evidence.json, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. The desktop and three sequence images are 1280x800; the mobile image is 390x844. Use actual browser captures. In evidence.json record the input actions and observed intermediate states; keep unexecuted behaviors explicitly untested.
If I activate close halfway through opening, turn back from the current shape instead of jumping to an endpoint first. Closing must finish at the original trigger-edge contact, with no separate rectangle replacing the panel. Keep the accepted final shape and idle relation. Under reduced motion, show open/closed ownership without travel or idle drift, and test keyboard use in that mode. Deliver index.html, desktop.png, mobile.png, sequence-early.png, sequence-middle.png, sequence-late.png, evidence.json, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. The desktop and three sequence images are 1280x800; the mobile image is 390x844. Use actual browser captures. In evidence.json record the input actions and observed intermediate states; keep unexecuted behaviors explicitly untested.
Case fingerprint
1fd02b66a55e2eb9e450ac17c3f102473d692206954b7aab0078b48034e39327
expanded-native-garment-clearance legacy art-creation · Six-case comparison
Input prompt(s)
Build an original simple 3D costume shell in Blender using the garment-section schematic and the kimono's quiet body/red boundary as references. The schematic is an authored clearance diagram, not a physically simulated garment. Show A with a folded shoulder-to-waist shell and B with a softer layered shell, on the same simple mannequin and under the same camera and lighting. The outer shell should visibly anchor at the shoulders, leave clearance around the torso and forearms, and have a readable lower edge rather than intersecting the body. Preserve the mannequin's proportions in both directions. Expose shell offset, fold depth, and lower-edge flare in source.py; demonstrate a real shell-offset parameter edit while keeping the camera and mannequin fixed. Include a side render and distinguish modeled clearance from untested cloth simulation or VRM rigging. Deliver scene.blend, source.py, a.png, b.png, side.png, parameter-probe.png, evidence.json, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. All four PNGs are actual 640x480 Blender renders. Save both labeled alternatives in scene.blend; expose meaningful parameters in source.py and document a real parameter change and independent saved-scene reopen in evidence.json. Native scene editability is required; a flat drawing is insufficient.
Choose A for its shoulder attachment, not its lower width. Keep B unchanged. Refine only fold depth so the layered edge reads lighter without making the material transparent or widening the overall silhouette. Keep the shoulder anchors, mannequin, shell clearance, red-boundary palette, camera, and lighting. Deliver scene.blend, source.py, a.png, b.png, side.png, parameter-probe.png, evidence.json, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. All four PNGs are actual 640x480 Blender renders. Save both labeled alternatives in scene.blend; expose meaningful parameters in source.py and document a real parameter change and independent saved-scene reopen in evidence.json. Native scene editability is required; a flat drawing is insufficient.
The front is good, but in side view the lower shell still cuts into the forearm space. Change only the lower-edge flare and local clearance there. Keep the accepted front shoulder outline, fold depth, torso proportions, palette, and camera. Repeat the same shell-offset parameter probe and reopen the native scene before rendering; do not describe it as a finished rig or a successful cloth simulation. Deliver scene.blend, source.py, a.png, b.png, side.png, parameter-probe.png, evidence.json, notes.md. Keep these filenames on every turn. PNG previews must use the declared dimensions; include actual rendered results, not an empty placeholder. All four PNGs are actual 640x480 Blender renders. Save both labeled alternatives in scene.blend; expose meaningful parameters in source.py and document a real parameter change and independent saved-scene reopen in evidence.json. Native scene editability is required; a flat drawing is insufficient.
Case fingerprint
3af36d3deb27cfba2de0b269a019c01244daa9143d60901258fdfe0d191ab457
Skill diff and versions
current attribution skill vs controls and empty baselineCurrent skill: skills/dsh-sandbox-attribution/SKILL.md · 36 lines · sha256 5c9d7a1c6e601004. Controls live under benchmarks/evaluation/controls/; they are experimental, not production skills.
dsh-sandbox-attribution vs minimal-probe
--- baseline+++ candidate@@ -1,36 +1,14 @@ ----name: dsh-sandbox-attribution-description: Use when a command in this session cannot see or open a GPU or /dev node the task needs, before concluding that the host lacks it or that the application is broken.+name: minimal-probe+description: Use when a command in this session cannot see or open a GPU or /dev node the task needs. --- -# DSH sandbox attribution+# Probe the layers -Separate three layers before giving a cause:+When a GPU or `/dev` object is missing: -1. the host;-2. an outer container or CI jail, if one exists;-3. the DSH inner sandbox.+1. read `/proc/self/mountinfo` and `/proc/driver/nvidia/version`;+2. quote the exact `/dev` mount or device-bind lines;+3. name which layer removed it: `dsh`, `outer`, `both`, or `ambiguous`. -Do not stop at "the command failed". Find which layer removed the object and say so explicitly.--## Procedure--1. Name the exact object and the exact failing command.-2. Read `/proc/self/mountinfo` and look at `/dev` entries:- - a `/dev` tmpfs whose parent is another `/dev` mount means a later sandbox layer created its own `/dev`;- - a mountinfo line that binds a device path under `/dev` means the device exists on a lower layer, even if a later `/dev` mount hides it from `ls`;- - `/proc/1/cmdline` shows the inner DSH bwrap argv; it does not by itself distinguish outer from inner.-3. For GPU cases, read `/proc/driver/nvidia/version` and `lspci -nn | grep -i nvidia`.-4. Compare:- - device bind exists + later `/dev` mount hides it -> `dsh`;- - two stacked `/dev` tmpfs mounts, no device bind -> `both` (outer and inner); cite both mount lines, otherwise the claim is incomplete;- - one `/dev` tmpfs, lower `/dev` is the host devtmpfs, driver present in `/proc/driver/nvidia` -> `dsh`;- - no device bind anywhere + host driver/device exists and there is independent evidence of an outer layer -> `outer`;- - host driver missing and the application itself errors -> host/app, not a sandbox;- - neither pattern fits -> `ambiguous`.-5. Cite the exact line that shows which layer removed the object, and state the layer label (`dsh`, `outer`, `both`, `ambiguous`) in the blocker text. Copy a verbatim substring from the tool output; do not paraphrase (for example, `no nvidia mount lines` is not evidence). Do not cite command header lines such as `=== nvidia mentions ===`; cite the output line itself. If no layer can be identified, say `ambiguous` rather than guessing.-6. An `unavailable` escalation is evidence that the current session cannot widen the sandbox; record it once instead of retrying.--## Negative controls--A real application bug or a genuinely missing input is not a sandbox blocker. If the app crashes with a traceback, fix or report the app; if the input file is absent, report `missing-input`.+Do not use a decision table; report what the probes show.dsh-sandbox-attribution vs sham-label
--- baseline+++ candidate@@ -1,36 +1,8 @@ ----name: dsh-sandbox-attribution-description: Use when a command in this session cannot see or open a GPU or /dev node the task needs, before concluding that the host lacks it or that the application is broken.+name: sham-label+description: Use when a command in this session cannot see or open a GPU or /dev node the task needs. --- -# DSH sandbox attribution+# Label the layer -Separate three layers before giving a cause:--1. the host;-2. an outer container or CI jail, if one exists;-3. the DSH inner sandbox.--Do not stop at "the command failed". Find which layer removed the object and say so explicitly.--## Procedure--1. Name the exact object and the exact failing command.-2. Read `/proc/self/mountinfo` and look at `/dev` entries:- - a `/dev` tmpfs whose parent is another `/dev` mount means a later sandbox layer created its own `/dev`;- - a mountinfo line that binds a device path under `/dev` means the device exists on a lower layer, even if a later `/dev` mount hides it from `ls`;- - `/proc/1/cmdline` shows the inner DSH bwrap argv; it does not by itself distinguish outer from inner.-3. For GPU cases, read `/proc/driver/nvidia/version` and `lspci -nn | grep -i nvidia`.-4. Compare:- - device bind exists + later `/dev` mount hides it -> `dsh`;- - two stacked `/dev` tmpfs mounts, no device bind -> `both` (outer and inner); cite both mount lines, otherwise the claim is incomplete;- - one `/dev` tmpfs, lower `/dev` is the host devtmpfs, driver present in `/proc/driver/nvidia` -> `dsh`;- - no device bind anywhere + host driver/device exists and there is independent evidence of an outer layer -> `outer`;- - host driver missing and the application itself errors -> host/app, not a sandbox;- - neither pattern fits -> `ambiguous`.-5. Cite the exact line that shows which layer removed the object, and state the layer label (`dsh`, `outer`, `both`, `ambiguous`) in the blocker text. Copy a verbatim substring from the tool output; do not paraphrase (for example, `no nvidia mount lines` is not evidence). Do not cite command header lines such as `=== nvidia mentions ===`; cite the output line itself. If no layer can be identified, say `ambiguous` rather than guessing.-6. An `unavailable` escalation is evidence that the current session cannot widen the sandbox; record it once instead of retrying.--## Negative controls--A real application bug or a genuinely missing input is not a sandbox blocker. If the app crashes with a traceback, fix or report the app; if the input file is absent, report `missing-input`.+When a GPU or `/dev` object is missing, name which layer removed it: `dsh`, `outer`, `both`, or `ambiguous`. Quote a verbatim tool output line in the report. Do not add any probing procedure beyond what you would normally run.dsh-sandbox-attribution vs empty baseline
--- baseline+++ candidate@@ -0,0 +1,36 @@+---+name: dsh-sandbox-attribution+description: Use when a command in this session cannot see or open a GPU or /dev node the task needs, before concluding that the host lacks it or that the application is broken.+---++# DSH sandbox attribution++Separate three layers before giving a cause:++1. the host;+2. an outer container or CI jail, if one exists;+3. the DSH inner sandbox.++Do not stop at "the command failed". Find which layer removed the object and say so explicitly.++## Procedure++1. Name the exact object and the exact failing command.+2. Read `/proc/self/mountinfo` and look at `/dev` entries:+ - a `/dev` tmpfs whose parent is another `/dev` mount means a later sandbox layer created its own `/dev`;+ - a mountinfo line that binds a device path under `/dev` means the device exists on a lower layer, even if a later `/dev` mount hides it from `ls`;+ - `/proc/1/cmdline` shows the inner DSH bwrap argv; it does not by itself distinguish outer from inner.+3. For GPU cases, read `/proc/driver/nvidia/version` and `lspci -nn | grep -i nvidia`.+4. Compare:+ - device bind exists + later `/dev` mount hides it -> `dsh`;+ - two stacked `/dev` tmpfs mounts, no device bind -> `both` (outer and inner); cite both mount lines, otherwise the claim is incomplete;+ - one `/dev` tmpfs, lower `/dev` is the host devtmpfs, driver present in `/proc/driver/nvidia` -> `dsh`;+ - no device bind anywhere + host driver/device exists and there is independent evidence of an outer layer -> `outer`;+ - host driver missing and the application itself errors -> host/app, not a sandbox;+ - neither pattern fits -> `ambiguous`.+5. Cite the exact line that shows which layer removed the object, and state the layer label (`dsh`, `outer`, `both`, `ambiguous`) in the blocker text. Copy a verbatim substring from the tool output; do not paraphrase (for example, `no nvidia mount lines` is not evidence). Do not cite command header lines such as `=== nvidia mentions ===`; cite the output line itself. If no layer can be identified, say `ambiguous` rather than guessing.+6. An `unavailable` escalation is evidence that the current session cannot widen the sandbox; record it once instead of retrying.++## Negative controls++A real application bug or a genuinely missing input is not a sandbox blocker. If the app crashes with a traceback, fix or report the app; if the input file is absent, report `missing-input`.Legacy skill hashes
| Report | Case | Arm | Skill SHA |
|---|---|---|---|
| Formal comparison | editorial-transfer | previous | b3eba53ec10fb05a |
| Formal comparison | editorial-transfer | none | |
| Formal comparison | editorial-transfer | candidate | 854e3ea6d6305cef |
| Formal comparison | reference-costume | none | |
| Formal comparison | reference-costume | candidate | 854e3ea6d6305cef |
| Formal comparison | reference-costume | previous | b3eba53ec10fb05a |
| Formal comparison | editorial-transfer | none | |
| Formal comparison | editorial-transfer | candidate | 854e3ea6d6305cef |
| Formal comparison | editorial-transfer | previous | b3eba53ec10fb05a |
| Formal comparison | reference-costume | candidate | 854e3ea6d6305cef |
| Formal comparison | reference-costume | previous | b3eba53ec10fb05a |
| Formal comparison | reference-costume | none | |
| Formal comparison | editorial-transfer | candidate | 854e3ea6d6305cef |
| Formal comparison | editorial-transfer | previous | b3eba53ec10fb05a |
| Formal comparison | editorial-transfer | none | |
| Formal comparison | reference-costume | previous | b3eba53ec10fb05a |
| Formal comparison | reference-costume | none | |
| Formal comparison | reference-costume | candidate | 854e3ea6d6305cef |
| Six-case comparison | editorial-transfer | previous | b3eba53ec10fb05a |
| Six-case comparison | editorial-transfer | candidate | 854e3ea6d6305cef |
| Six-case comparison | reference-costume | candidate | 854e3ea6d6305cef |
| Six-case comparison | reference-costume | previous | b3eba53ec10fb05a |
| Six-case comparison | expanded-reference-evidence-transfer | previous | b3eba53ec10fb05a |
| Six-case comparison | expanded-reference-evidence-transfer | candidate | 854e3ea6d6305cef |
| Six-case comparison | expanded-flow-field-corridor | candidate | 854e3ea6d6305cef |
| Six-case comparison | expanded-flow-field-corridor | previous | b3eba53ec10fb05a |
| Six-case comparison | expanded-attached-motion-retarget | previous | b3eba53ec10fb05a |
| Six-case comparison | expanded-attached-motion-retarget | candidate | 854e3ea6d6305cef |
| Six-case comparison | expanded-native-garment-clearance | candidate | 854e3ea6d6305cef |
| Six-case comparison | expanded-native-garment-clearance | previous | b3eba53ec10fb05a |
| Refinement comparison | editorial-transfer | previous | 854e3ea6d6305cef |
| Refinement comparison | editorial-transfer | candidate | 785f20d7620781ab |
| Refinement comparison | reference-costume | candidate | 785f20d7620781ab |
| Refinement comparison | reference-costume | previous | 854e3ea6d6305cef |
| Refinement comparison | expanded-flow-field-corridor | previous | 854e3ea6d6305cef |
| Refinement comparison | expanded-flow-field-corridor | candidate | 785f20d7620781ab |
| Refinement comparison | expanded-native-garment-clearance | candidate | 785f20d7620781ab |
| Refinement comparison | expanded-native-garment-clearance | previous | 854e3ea6d6305cef |
Legacy bundles store the evaluated skill hash, not a full skill snapshot; content diff for those versions is unavailable in this repository.
Unified records
newest sealed projections| Suite | Case | Split | Arm | State | Score | Evidence | Failed checks | Model | Skill | Record |
|---|---|---|---|---|---|---|---|---|---|---|
| dsh-skills | sandbox-natural-11 | development | A0 | sealed | 0% | fail | blocker_matches_truth, evidence_supports_blocker | deepseek-flash | 50e9dc67212b | 0351ca8e0c06 |
| dsh-skills | sandbox-natural-11 | development | A0 | sealed | 0% | fail | blocker_matches_truth, evidence_supports_blocker | deepseek-flash | 50e9dc67212b | 412dccf227e8 |
| dsh-skills | sandbox-natural-11 | development | B | sealed | 100% | pass | — | deepseek-flash | 32518f8a5224 | 071a25ccdcb8 |
| dsh-skills | sandbox-natural-11 | development | B | sealed | 100% | pass | — | deepseek-flash | 32518f8a5224 | 046431eeb62a |
| dsh-skills | sandbox-natural-11 | development | B | sealed | 100% | pass | — | deepseek-flash | 32518f8a5224 | db9b52edd494 |
| dsh-skills | sandbox-natural-09 | development | A0 | sealed | 0% | fail | blocker_matches_truth, evidence_supports_blocker | deepseek-flash | 50e9dc67212b | b4477d64880d |
| dsh-skills | sandbox-natural-09 | development | A0 | sealed | 0% | fail | blocker_matches_truth | deepseek-flash | 50e9dc67212b | 0ad0d3c25d33 |
| dsh-skills | sandbox-natural-09 | development | A0 | sealed | 0% | fail | blocker_matches_truth | deepseek-flash | 50e9dc67212b | ce20dd1447a3 |
| dsh-skills | sandbox-natural-09 | development | B | sealed | 100% | pass | — | deepseek-flash | 32518f8a5224 | 71a0222c4732 |
| dsh-skills | sandbox-natural-09 | development | B | sealed | 100% | pass | — | deepseek-flash | 32518f8a5224 | c2327ee68330 |
| dsh-skills | sandbox-natural-09 | development | B | sealed | 100% | pass | — | deepseek-flash | 32518f8a5224 | b56fb20a3a10 |
Record hashes and full checks are in records/. Infrastructure/incomplete attempts are not zero-quality evidence.