seed / reproducibility-first
Thomas
Peng.
Graphic designer turned AI-native builder. I build agentic systems and evaluate them honestly: deterministic scoring, no LLM-judge in the success path, adversarial verification, cost-gated reproducible runs, and honest nulls in the headline.
Reproducibility is the discipline, not the disclaimer.
Every result here ships with the exact command to reproduce it.
Shared kernel
Four artifacts. One shared substrate. Quorum's core/ provides cost-aware model routing, adversarial multi-agent verification, and full tracing. Aegis and FieldAgent vendor it directly. The proof: a substrate built once, proven on multiple problem classes.
Quorum
Task-aware agent orchestrator (flagship)K=3 adversarial verification cut false positives 27.8% to 0.0% (95% CI [11.1, 50.0] to [0, 0]; recall 100% to 77.8%) on a 36-snippet labeled set including prompt-injection traps. Held-out real target: 3/3 genuine bugs found, 0 surviving false positives. Cost-routing claim is operator-gated on an Anthropic key. Presented honestly as "harness committed, live multi-tier number gated."
| Metric | Before | After (K=3) | Notes |
|---|---|---|---|
| False positive rate | 27.8% | 0.0% | 95% CI [0, 0] on 36-snippet set |
| Recall | 100% | 77.8% | Trade-off: fewer FP, some true positives deferred |
| Held-out real bugs | untested | 3/3 found | 0 surviving false positives |
| Concurrency | N/A | cap 8 | Finders per file, K skeptics per finding |
| Run cost | N/A | approx. $0.25 | Multi-tier gated: gated on ANTHROPIC_API_KEY |
| Test suite | N/A | 58 tests | ruff + mypy + CI green |
Aegis
Adaptive red-team gauntletA reasoning model is significantly more robust pre-defense (injection ASR 49.3% vs 68.1%, p=0.0012; canary 10.4% vs 21.5%, p=0.010; overall p=0.0002). But the full defense stack erases the gap entirely (1.7% vs 2.8%, p=0.40, not significant). Defense reduction 29.2% to 4.2%. The input-classifier is the workhorse.
| Metric | Baseline | Result | p-value |
|---|---|---|---|
| Injection ASR, reasoning model | 49.3% | vs 68.1% (standard) | p=0.0012 |
| Canary ASR, reasoning model | 10.4% | vs 21.5% (standard) | p=0.010 |
| With full defense stack | any model | 1.7% vs 2.8% (n.s.) | p=0.40, gap erased |
| Defense reduction | 29.2% ASR | 4.2% ASR (-25%) | significant |
| Adaptation lift | 24.0% | 29.9% | Significant only after scaling (McNemar b=17/c=0). Framed correctly: scaling is the legit power lever. |
| Test suite | N/A | 78 tests | CI + Pages green |
FieldAgent
CUAD contract red-flag finderThe "agentic chunking lift" is model-specific noise, not a real advantage. It looked like +0.45 F1 on DeepSeek only because of a truncation artifact. A fair rerun collapses it to +0.07 (CIs overlap) and it ties on Claude Sonnet. The baseline-independent result: +0.21 F1 over a keyword floor. That is what held. Party names and dollar figures are redacted in the demo.
| Metric | Value | Notes |
|---|---|---|
| Detection F1 | 0.548 | P=0.741 / R=0.435; 95% CI [0.460, 0.637] |
| Keyword floor lift | +0.21 F1 | Baseline-independent, robust |
| Agentic chunking lift (fair rerun) | +0.07 | CIs overlap; was +0.45 due to truncation artifact |
| Eval set | 20 contracts | CUAD held-out gold, span-IoU graded |
| Test suite | 47 tests | CI green |
Skill-Tuning Council
Self-improving skill orchestrator (internal methodology)A 4-proxy council votes on every self-improvement before it ships: taste, pragmatism, intent, anti-drift. Pipeline: adversary, editors, merger, council, escalate-on-disagreement. 576 tests. No public URL. Presented as a methodology piece, not a shipped product.
The real pipeline, as it runs:
| Property | Value |
|---|---|
| Council proxies | 4 (taste, pragmatism, intent, anti-drift) |
| Pipeline stages | adversary, editors, merger, council, escalate-on-disagreement |
| Test suite | 576 tests |
| Public URL | None. Internal infra. |
Eval discipline
How I build and measure
Four principles that run through every artifact. Not aspirational. Demonstrably applied in the results above.
make eval-dry runs without any API key.Contact
Get in touch
Target roles: frontier-lab Applied AI, Forward-Deployed Engineer, Agent Engineer, Design Engineer. If any of the above work looks relevant, email is the right path.