GPSBench: A Governance Policy Simulator for Automating Policy Analysis
How AI-policy bills, actors and impacts are collected from real APIs, assembled into a
typed causal graph, split out-of-time, and scored — balanced accuracy, recall and F1.
The output is a per-bill forecast: which actors a bill affects, and the impact each bears —
its measurable variable, welfare direction (± help / harm) and status. The impact is predicted by a
formal model that combines the LLM's guess with the graph's structural sign prior
(legal-action polarity x measurable-variable valence x actor role), so the sign is a source-typed rule
rather than an LLM assertion. Ends with two trained models: an actor-action forecaster (stage 6)
and an actor-relevance classifier (stage 7).
2,170 region bills · 6 regions2,170 causal graphs3,186 impact samplessplit test >= 2023every field source-cited2 trained models · forecaster + relevance
Current progress
One thesis, empirically: the LLM supplies world-knowledge semantics, the typed causal graph supplies source-verified discipline. Grounding the LLM's guess in the graph turns a plausible assertion into an empirical prediction — and on every structured field the combination beats the raw LLM.
0.78
Impact direction macro-F1 — hybrid (LLM+graph). Raw LLM 0.56 · base 0.35. our model
0.64
Impact status macro-F1 — from unlearnable (single-class gold) to a real field. LLM 0.40. our model
0.76
Actor relevance F1 (bal-acc 0.81) — which actors a bill affects, complete gold, 8,977 held-out pairs. our model
What's new this pass. A strong distinct annotator (sonnet-4.5, not the gpt-3.5 predictor — non-circular) labeled a complete impact gold (130 → 611 impacts, 4.7×), which rescued impact_status from degeneracy and let direction/status/variable all be scored fairly against the raw LLM. Direction is welfare-normalized to a single actor-relative convention across all 611 impacts. Full per-metric Precision/Recall and methodology in the stages below.
1
Data sources
real public APIs & databases — no fabrication
OECD.AI Observatory
api.oecdai.org
1,979 non-US AI policies, 76 countries + content/motivation
Empty != missing. These are strategies without a set end date — not gaps to fill.
Gold impact labels
actor_bill_impacts · verified · n = 131
bill_id · actor · aspect · direction100%
event · date · grade100%
source (URL / API query)100%
magnitude ($ figure where stated)10%
Every gold row is fully specified & source-cited; only a numeric magnitude is optional.
The sparse fields are structural, not missing data: why_enacted exists only for enacted bills, generated_events / fr_documents / recorded_votes only for US bills with downstream federal activity, and magnitude only where a source states a figure. Nothing here is a blank waiting to be fabricated.
Train set — what was "unfillable", and what stage 6 resolved
Complete from source
OECD record + derivation + taxonomy
title · content · motivation · description100%
type · status · outcome (adoption) · source100%
document_type · legal_force · level (new)100%
train/test split · effective year (new)100%
sectors · responsible_org · binding · tags76–97%
Now grounded-filled — was 0
graph-fields · verbatim-quote-verified from OECD text
motivation · generated_actions~94%
impacted_actors (structured)~94%was "inferred-only" → now source-grounded per bill
generated_impacts (impact · domain · ±dir)~94%each carries a verbatim evidence_quote
independent per-actor GOLD285 · 589grew 131 → 285 sourced (A+B) / 589 actor-attributed — still the scarce tier, search-gated, never fabricated
Update: the fields once marked "cannot fill" are now grounded-filled for ~94% of bills — extracted from each OECD record's own text with every quote verbatim-verified (stage 6), not fabricated. What stays limited is independent per-actor gold, which grew 131 → 285 sourced but remains gated on real documented events.
3
Typed causal graph → GNN
one graph per bill · every node/edge traced to a field + source
The typed causal graph — one per bill · 7 node kinds · 8 typed edges, every one traced to a field + source
Proposal-time features only. The international set is 95% already-adopted official laws (OECD status → in_force), so it trains actor-impact / coverage — not passage; US bills carry the enacted/vetoed/failed outcome. Bills with no documented firm impact keep the rule-derived orientation label, never an invented outcome.
5
Evaluation results
Precision · Recall · F1
F-score summary — our model vs LLM vs baseline (leak-free · fair matched candidate sets · pre-2023 LLM)
Task — metric
Baseline F1
Our model F1
LLM F1
Winner
Actor relevance — F1 (complete gold · held-out)
—
0.76
—
our model
Impact direction — macro-F1 (welfare sign · complete gold)
0.35
0.78
0.56
our model
Impact status — macro-F1 (intended/projected/observed)
Balanced accuracy / macro-F1 over a fixed vocab (impact fields), or reframed passage bal-acc (GNN). On the structured impact fields the hybrid — the LLM's guess grounded by the knowledge graph — beats the raw LLM: direction 0.78 vs 0.56, status 0.64 vs 0.40, against a strong-annotator complete gold. The raw LLM asserts a plausible sign; the graph's legal-action polarity × variable × actor role makes it empirical. Actor-relevance reaches F1 0.76 against the complete gold. Full P/R per metric below.
Passage → "advance past committee" · our GNN works balanced accuracy · seed-ensemble · ~51 test positives · leak-free
^ higher better
always-NO / chance0.500
relevant-actor GNN0.668
+ seed-ensemble + more data0.752
Our GNN: chance → 0.752, leak-free. Reframe "enacted" (unforecastable) → "advanced past committee"; add legislative-process features + bill-relevant actor nodes (matched to the 51-actor roster with real disposition/harm features); then ensemble over 12 seeds + pool all pre-2023 congresses (0.668→0.752). Plain process features fall well below the graph, so the graph + actor nodes carry the gain. Text embeddings & a train-tuned threshold hurt (overfit 18 positives) and were dropped. Honest ceiling ~0.75 (balanced acc, high variance) — reaching 0.85 would need leaky features, so we stop here.
Actor stance support / oppose · sourced ground truth
accuracy
majority base0.55
all0.80
held-out test0.857
Actor-impact prediction aspect-level · structured aspects beat raw generated events
recall / F1
M1 aspect+dir0.62
M1 aspect-only0.72
M2 change-detect F10.77
Impact fields — raw LLM vs LLM grounded by the graph macro-F1 · complete gold (strong distinct annotator) · 611 impacts · conditioned on related actor · welfare-normalized
^ higher better
direction · majority base0.35
direction · raw LLM0.56
direction · hybrid (ours)0.78
status · majority base0.20
status · raw LLM0.40
status · hybrid (ours)0.64
The graph turns the LLM from plausible to empirical. Asked to name an impact, the raw LLM (pre-2023, leak-free) picks a plausible-sounding sign and status — often a hallucination it can't ground. Feeding its guess as one feature alongside the knowledge graph's legal-action polarity (restrictive/supportive) × measurable-variable (cost/benefit) × actor role, then ensembling, lifts direction 0.56 → 0.78 and status 0.40 → 0.64 — the hybrid beats the raw LLM on both. Same gain on the sparse human-verified gold (direction 0.73 → 0.81). Status was previously unlearnable (single-class gold); the strong-annotator complete-gold pass (sonnet-4.5, distinct from the gpt-3.5 predictor — not circular) made it a balanced, real field. Direction is welfare-coded (+ helps that actor, − harms), one convention across all 611 impacts.
6
Multi-region expansion
6 regions · classified · graph-filled · strict-2023 split — new this session
New data sources — WebFetch-verified & snapshotted offline
OECD.AI raw cache
output/raw · 2,464 records
full API dump — the verbatim text every field is grounded on
Official harvest · r.jina.ai proxy
China Law Translate · DigiChina · e-Gov
JS / 403 government pages read through a reader proxy
Wikipedia · china-briefing · gov portals
WebFetch (research agents)
national / provincial / municipal reg discovery
Source-page snapshots
output/raw/harvest_pages · 76
raw page per source_url → verify every quote offline
New components
Region corpus · 6 buckets
EU · US · East Asia · China · Singapore · India — OECD base + verified harvest + native bills, deduped by title.
2,170 bills28 jurisdictions
Governance taxonomy
Every row classified — document_type (22-type) · jurisdiction_level · legal_force · province / city · issuing_authority.
policy_taxonomy.py
Grounded graph-fields
4 graph fields per bill (motivation · actions · impacted_actors · impacts); every evidence_quote verbatim-verified.
~94% filled
Region causal graphs
Typed graph per bill: motivation→bill→outcome + action / actor / impact nodes with signed direction.
2,170 graphs15.3k nodes · 15.6k edges
Unified impact dataset
Tiers A gold · A2 grounded-actor · B cited · C grounded — one temporal split, explicit counts.
Strict temporal split — every region in both (test = effective year >= 2023)
Region
Train <2023
Test >=2023
EU
713
603
US
131
329
East Asia · JP·KR·TW·HK
54
52
China
53
57
Singapore
65
39
India
36
38
Total
1,052
1,118
0 leakage verified: nothing >=2023 in train, nothing <2023 in test; undated rows kept in train so no data is dropped.
Evaluation on the region split — stratified
Impact — actor-gold test · our model vs LLM Metric 1 aspect recall · 210 test pairs · leak-free + memorization-audited
^ higher better
majority baseline0.237
LLM · pre-2023 cutoff0.561
The LLM alone asserts a plausible aspect — the graph makes it empirical. The raw LLM (pre-2023, leak-free) more than doubles the majority baseline on aspect recall (0.561) but only names a plausible-sounding impact. Grounding that guess in the knowledge graph (the impact hybrid above) then beats the LLM on the structured fields — direction 0.56→0.78, status 0.40→0.64. So the LLM supplies the semantics, the graph supplies the discipline: neither alone is the answer, the combination is. Both choose from the same fixed vocab (runtime-guarded); pre-2023 cutoff, no leak.
Actor relevance — which actors a bill affects title-only · leak-free · 75 bills · FAIR: both choose from the full 47-actor roster · M2 balanced accuracy
^ higher better
frequency baseline0.593
our rule matcher0.672
LLM · pre-2023 cutoff0.747
Fair candidate pool (47-actor roster, with distractors). On balanced accuracy our matcher and GNN clear the baseline (M2 bal-acc 0.593 / 0.672 / 0.747 for base / rule / LLM), and against the strong-annotator complete gold the trained relevance classifier reaches F1 0.76 (stage 7). Title-only, no leak, pre-2023 cutoff, guarded identical candidate set.
6
Actor-action forecaster
predict what each actor DOES · temporal split (train <2024 / test >=2024)
top-1 0.72
predicted top action correct, held-out.
10 / 10 classes
every action type carries signal (F1 > 0).
310 / 363
train / test actions, leak-free.
Per-action-type F1held-out
LOBBY0.88
RETALIATE0.80
Per-actor forecast — predicted top action across six policies
Actor
Signature
Chip export controls on China
CHIPS Act fab incentives
EU AI Act obligations
State deepfake / content law
AI compute infrastructure
Federal AI safety rules
United States
LOBBY
RESTRICT 29%
FUND 62%
LOBBY 41%
LOBBY 37%
REGULATE 44%
LOBBY 43%
China
LOBBY
RETALIATE 41%
FUND 51%
LOBBY 50%
LOBBY 45%
REGULATE 38%
LOBBY 55%
European Union
LOBBY
REGULATE 30%
FUND 45%
LOBBY 44%
LOBBY 39%
REGULATE 39%
LOBBY 48%
United Kingdom
FUND
RETALIATE 21%
FUND 53%
LOBBY 28%
FUND 25%
PARTNER 36%
LOBBY 32%
Nvidia
LOBBY
RESTRICT 35%
BUILD 45%
LOBBY 73%
LOBBY 33%
PARTNER 71%
LOBBY 77%
Tsmc
BUILD
RESTRICT 70%
BUILD 88%
BUILD 49%
BUILD 34%
PARTNER 61%
LOBBY 44%
Intel
LOBBY
RESTRICT 29%
BUILD 72%
LOBBY 40%
LOBBY 47%
PARTNER 85%
LOBBY 58%
Micron
LOBBY
RESTRICT 60%
BUILD 78%
LOBBY 55%
LOBBY 24%
PARTNER 73%
LOBBY 65%
Samsung
LOBBY
RESTRICT 52%
BUILD 73%
LOBBY 61%
LOBBY 25%
PARTNER 79%
LOBBY 67%
Asml
BUILD
RESTRICT 44%
BUILD 59%
ENFORCE 26%
BUILD 21%
PARTNER 79%
PARTNER 21%
Microsoft
LOBBY
PARTNER 29%
BUILD 63%
LOBBY 65%
LOBBY 52%
PARTNER 91%
LOBBY 61%
Google
LOBBY
RESTRICT 27%
BUILD 59%
LOBBY 52%
LITIGATE 36%
PARTNER 89%
LOBBY 51%
Meta
LOBBY
RESTRICT 31%
BUILD 49%
LOBBY 50%
LITIGATE 43%
PARTNER 81%
LOBBY 44%
Amazon
LOBBY
PARTNER 28%
BUILD 65%
LOBBY 63%
LOBBY 51%
PARTNER 90%
LOBBY 52%
Apple
BUILD
RESTRICT 30%
BUILD 59%
ENFORCE 35%
BUILD 24%
PARTNER 84%
PARTNER 27%
Openai
LOBBY
PARTNER 23%
BUILD 60%
LOBBY 58%
LOBBY 34%
PARTNER 87%
LOBBY 55%
Anthropic
LOBBY
RESTRICT 30%
BUILD 67%
LOBBY 54%
LOBBY 37%
PARTNER 89%
LOBBY 53%
Huawei
RESTRICT
RESTRICT 54%
BUILD 52%
ENFORCE 31%
RESTRICT 24%
PARTNER 66%
ENFORCE 22%
Deepseek
BUILD
BUILD 21%
BUILD 43%
ENFORCE 27%
LITIGATE 41%
PARTNER 83%
LITIGATE 21%
7
Actor-relevance classifier
which actors relate to a bill · complete gold · large held-out split
F1 0.756
precision 0.67 · recall 0.86, on 8,977 held-out pairs.
bal-acc 0.814
complete thorough-relatedness gold.
20,821 train
443 bills · 7,504 related pairs.
8,977 test
191 held-out bills · 3,213 related.
F1 progressionvs complete gold
Cosine similarity + rank0.69
+ reduced embedding dims0.72
+ shared-basis embeddings & product0.74
+ stronger GBT, F1-tuned threshold0.75
+ 2x data, large held-out split0.76
LLM-embedding semantics of bill vs. each actor (TF-IDF can't see "chip export controls" ~ NVIDIA; the embedding can), + shared-basis interaction features. Measured against a complete gold labeled by a strong annotator (sonnet-4.5, distinct from the classifier — not circular). F1 0.75 is the honest number; the residual is nuanced policy judgment only a reasoning model captures.