From public sources to a causal graph, evaluated out-of-time
How AI-policy bills, actors and impacts are collected from real APIs, assembled into a
typed causal graph, split out-of-time, and scored — balanced accuracy, recall and F1 only, never AUC.
Ends with two trained models: an actor-action forecaster (stage 6) and an actor-relevance classifier (stage 7).
2,170 region bills · 6 regions2,170 causal graphs3,186 impact samplessplit test ≥ 2023every field source-cited2 trained models · forecaster + relevance
1
Data sources
real public APIs & databases — no fabrication
OECD.AI Observatory
api.oecdai.org
1,979 non-US AI policies, 76 countries + content/motivation
Empty ≠ missing. These are strategies without a set end date — not gaps to fill.
Gold impact labels
actor_bill_impacts · verified · n = 131
bill_id · actor · aspect · direction100%
event · date · grade100%
source (URL / API query)100%
magnitude ($ figure where stated)10%
Every gold row is fully specified & source-cited; only a numeric magnitude is optional.
The sparse fields are structural, not missing data: why_enacted exists only for enacted bills, generated_events / fr_documents / recorded_votes only for US bills with downstream federal activity, and magnitude only where a source states a figure. Nothing here is a blank waiting to be fabricated.
Train set — what was "unfillable", and what stage 6 resolved
✓ Complete from source
OECD record + derivation + taxonomy
title · content · motivation · description100%
type · status · outcome (adoption) · source100%
document_type · legal_force · level (new)100%
train/test split · effective year (new)100%
sectors · responsible_org · binding · tags76–97%
◐ Now grounded-filled — was 0
graph-fields · verbatim-quote-verified from OECD text
motivation · generated_actions~94%
impacted_actors (structured)~94%was "inferred-only" → now source-grounded per bill
generated_impacts (impact · domain · ±dir)~94%each carries a verbatim evidence_quote
independent per-actor GOLD285 · 589grew 131 → 285 sourced (A+B) / 589 actor-attributed — still the scarce tier, search-gated, never fabricated
Update: the fields once marked "cannot fill" are now grounded-filled for ~94% of bills — extracted from each OECD record's own text with every quote verbatim-verified (stage 6), not fabricated. What stays limited is independent per-actor gold, which grew 131 → 285 sourced but remains gated on real documented events.
3
Typed causal graph → GNN
one graph per bill · every node/edge traced to a field + source
antecedentprior events
motivates
motivationwhy proposed
reason
billprovisions
proposal
outcome ①enacted / vetoed / failed
actorsponsor · firms · govs
supports / opposes
outcome ①mechanism + why
do(enact)
impactworld_impact
impacts
outcome ②per-actor (aspect, ±dir)
structural nodeedge / relationoutcome ① passageoutcome ② impactdata source
A · Source → field → node mind map (which field from which source builds which node)
OECD.AI Observatory api.oecdai.org
englishName / title→bill
description · overview→motivation
type · extentBinding→outcome ① (orientation)
gaiinCountry→actor (country gov)
Congress.gov api.congress.gov · BILLSTATUS
title · crs_summary→bill · motivation
sponsor · cosponsor_party · n_cosponsors→actor
legislative milestones→antecedent_event
status → outcome_type + must_pass / exec_order / strong_standalone→outcome ①
Federal Register federalregister.gov
fr_documents→impact
generated_events→outcome ② world
SEC EDGAR efts.sec.gov · 10-K/10-Q/20-F
filing disclosure (charge · revenue-at-risk)→outcome ② actor (aspect, ±dir)
UN Comtrade comtradeapi.un.org
trade flows (HS 8542 / 8486)→outcome ② actor (trade ±)
USAspending · GSA · DoD · DOE usaspending.gov + releases
contract award $ · ceiling→outcome ② actor (funding +)
Proposal-time features only. The international set is 95% already-adopted official laws (OECD status → in_force), so it trains actor-impact / coverage — not passage; US bills carry the enacted/vetoed/failed outcome. Bills with no documented firm impact keep the rule-derived orientation label, never an invented outcome.
5
Evaluation results
Precision · Recall · F1 — never AUC
F-score summary — our model vs LLM vs baseline (leak-free · fair matched candidate sets · pre-2023 LLM · no AUC)
Task — metric
Baseline F1
Our model F1
LLM F1
Winner
Actor relevance — set-F1 (fair 47-roster)
0.21
0.31
0.22
our model
Impact, actor-gold — set-F1 (9-aspect vocab)
~0.24
0.34
0.43
LLM
Advance-past-committee — F1 (advanced class)
—
0.37
—
our GNN
Passage, enacted — bal-acc (out-of-time)
0.50
~0.50
~0.48
chance ✗
F1 = unbudgeted set Precision·Recall·F1 (aspects / actors) or advanced-class F1 (GNN). Our model owns advance-past-committee; the LLM wins impact (higher recall, over-predicts). Actor-relevance is match-mode-dependent: our rule matcher wins on EXACT F1 (0.31 vs 0.22), the LLM wins on HIERARCHICAL F1 (0.32 vs 0.28, parent-agency / population→firm credit). Passage is unforecastable for everyone once de-leaked. Full P/R per metric below.
Passage — every method, LEAK-FREE out-of-time US-federal balanced accuracy · test 2023–2025 · only 10 positives · outcome leakage removed
↑ higher better
always-NO baseline0.500
our LogReg (tabular)0.500
our causal-graph GNN0.544
LLM · title-only (pre-2023)0.554
LLM · reads content (pre-2023)0.483
Honest negative result. Once outcome leakage is removed, no method beats chance (~0.48–0.55) — LLM, GNN and LogReg alike. Earlier LLM 0.79 / 0.85 read our world_impact field, which literally states the outcome ("Did not take effect / Had it become law…") for 95% of bills — pure leak, discarded. Passage-as-enacted isn't forecastable at 10 positives — but reframing the task fixes it ↓
Passage REFRAMED — "advance past committee" · our GNN works balanced accuracy · seed-ensemble · ~51 test positives · leak-free
↑ higher better
always-NO / chance0.500
process-feature LogReg (ceiling)0.590
relevant-actor GNN0.668
+ seed-ensemble + more data0.752
Our GNN: chance → 0.752, leak-free. Reframe "enacted" (unforecastable) → "advanced past committee"; add legislative-process features + bill-relevant actor nodes (matched to the 51-actor roster with real disposition/harm features); then ensemble over 12 seeds + pool all pre-2023 congresses (0.668→0.752). Plain process features cap at 0.590, so the graph + actor nodes carry the gain. Advanced-class P/R/F1 = 0.32 / 0.45 / 0.37. Text embeddings & a train-tuned threshold hurt (overfit 18 positives) and were dropped. Honest ceiling ~0.75 (balanced acc, high variance) — reaching 0.85 would need leaky features, so we stop here.
Actor stance support / oppose · sourced ground truth
accuracy
majority base0.55
all0.80
held-out test0.857
Actor-impact prediction aspect-level · vs raw generated-events 0.09
outcome (in_force / proposed / not_adopted)→outcome ①
effective year → split→train / test tag
Strict temporal split — every region in both (test = effective year ≥ 2023)
Region
Train <2023
Test ≥2023
EU
713
603
US
131
329
East Asia · JP·KR·TW·HK
54
52
China
53
57
Singapore
65
39
India
36
38
Total
1,052
1,118
0 leakage verified: nothing ≥2023 in train, nothing <2023 in test; undated rows kept in train so no data is dropped.
Evaluation on the region split — stratified, never AUC
Passage — US, the real proposal-stage task balanced accuracy · 7% pass · leak-free · out-of-time (test ≥ 2023)
↑ higher better
majority baseline0.500
trained LogReg0.500
our causal-graph GNN0.544
LLM · reads bill content (pre-2023)0.483
Honest negative result. On leak-free inputs no method beats chance (~0.48–0.55). The earlier 0.79/0.85 came from our world_impact text stating the outcome (95% of bills) — a leak, now removed. Whether an AI bill passes isn't recoverable from its text; it depends on the legislative process.
Impact — actor-gold test · our model vs LLM Metric 1 aspect recall · 210 test pairs · leak-free + memorization-audited
↑ higher better
majority baseline0.237
our trained model (actor-prior)0.243
LLM · pre-2023 cutoff0.561
The LLM is the signal; our trained model isn't. Our actor-prior model barely clears the baseline (0.243 vs 0.237), the LLM (pre-2023-cutoff gpt-3.5-0613, leak-free) more than doubles it. Full P/R/F1 (actor-gold): LLM 0.28/0.90/0.425 · our actor-prior 0.34/0.34/0.344 · baseline ~0.24 — the LLM over-predicts (high recall, low precision); our model is more precise. Fair: both choose from the same fixed 9-aspect vocab (a runtime guard asserts it). Genuine reasoning, not recall (pre-2023 cutoff). (Enrichment domains B: F1 0.28, near baseline.)
Actor relevance — which actors a bill affects title-only · leak-free · 75 bills · FAIR: both choose from the full 47-actor roster · M2 balanced accuracy
↑ higher better
frequency baseline0.593
our rule matcher0.672
LLM · pre-2023 cutoff0.747
Fair candidate pool (47-actor roster, with distractors); winner depends on match mode.EXACT set-F1: baseline 0.21 · our rule matcher 0.31 (wins) · LLM 0.22 — the LLM over-predicts into distractors so its exact F1 falls below our precise matcher. HIERARCHICAL F1 (parent-agency / population→firm credit, shared actor_hierarchy): baseline 0.19 · rule 0.28 · LLM 0.32 (wins) — broad predictions get expanded credit. M2 bal-acc 0.593 / 0.672 / 0.747. Title-only, no leak, pre-2023 cutoff, guarded identical candidate set.
6
Actor-action forecaster
predict what each actor DOES · temporal split (train <2024 / test ≥2024)
macro-F1 0.459
vs baseline 0.08 · gpt-3.5 0.24. 10 action types.
top-1 72%
per-actor F1 0.57.
10 / 10 classes
every action type carries signal (F1 > 0).
310 / 363
train / test actions, leak-free.
Per-action-type F1held-out
LOBBY0.88
RETALIATE0.80
LITIGATE0.50
BUILD0.43
RESTRICT0.40
PARTNER0.36
FUND0.35
REGULATE0.34
ENFORCE0.31
COMPLY0.22
Per-actor forecast — predicted top action across six policies
Actor
Signature
Chip export controls on China
CHIPS Act fab incentives
EU AI Act obligations
State deepfake / content law
AI compute infrastructure
Federal AI safety rules
United States
LOBBY
RESTRICT 29%
FUND 62%
LOBBY 41%
LOBBY 37%
REGULATE 44%
LOBBY 43%
China
LOBBY
RETALIATE 41%
FUND 51%
LOBBY 50%
LOBBY 45%
REGULATE 38%
LOBBY 55%
European Union
LOBBY
REGULATE 30%
FUND 45%
LOBBY 44%
LOBBY 39%
REGULATE 39%
LOBBY 48%
United Kingdom
FUND
RETALIATE 21%
FUND 53%
LOBBY 28%
FUND 25%
PARTNER 36%
LOBBY 32%
Nvidia
LOBBY
RESTRICT 35%
BUILD 45%
LOBBY 73%
LOBBY 33%
PARTNER 71%
LOBBY 77%
Tsmc
BUILD
RESTRICT 70%
BUILD 88%
BUILD 49%
BUILD 34%
PARTNER 61%
LOBBY 44%
Intel
LOBBY
RESTRICT 29%
BUILD 72%
LOBBY 40%
LOBBY 47%
PARTNER 85%
LOBBY 58%
Micron
LOBBY
RESTRICT 60%
BUILD 78%
LOBBY 55%
LOBBY 24%
PARTNER 73%
LOBBY 65%
Samsung
LOBBY
RESTRICT 52%
BUILD 73%
LOBBY 61%
LOBBY 25%
PARTNER 79%
LOBBY 67%
Asml
BUILD
RESTRICT 44%
BUILD 59%
ENFORCE 26%
BUILD 21%
PARTNER 79%
PARTNER 21%
Microsoft
LOBBY
PARTNER 29%
BUILD 63%
LOBBY 65%
LOBBY 52%
PARTNER 91%
LOBBY 61%
Google
LOBBY
RESTRICT 27%
BUILD 59%
LOBBY 52%
LITIGATE 36%
PARTNER 89%
LOBBY 51%
Meta
LOBBY
RESTRICT 31%
BUILD 49%
LOBBY 50%
LITIGATE 43%
PARTNER 81%
LOBBY 44%
Amazon
LOBBY
PARTNER 28%
BUILD 65%
LOBBY 63%
LOBBY 51%
PARTNER 90%
LOBBY 52%
Apple
BUILD
RESTRICT 30%
BUILD 59%
ENFORCE 35%
BUILD 24%
PARTNER 84%
PARTNER 27%
Openai
LOBBY
PARTNER 23%
BUILD 60%
LOBBY 58%
LOBBY 34%
PARTNER 87%
LOBBY 55%
Anthropic
LOBBY
RESTRICT 30%
BUILD 67%
LOBBY 54%
LOBBY 37%
PARTNER 89%
LOBBY 53%
Huawei
RESTRICT
RESTRICT 54%
BUILD 52%
ENFORCE 31%
RESTRICT 24%
PARTNER 66%
ENFORCE 22%
Deepseek
BUILD
BUILD 21%
BUILD 43%
ENFORCE 27%
LITIGATE 41%
PARTNER 83%
LITIGATE 21%
7
Actor-relevance classifier
which actors relate to a bill · complete gold · large held-out split
F1 0.756
precision 0.67 · recall 0.86, on 8,977 held-out pairs.
bal-acc 0.814
complete thorough-relatedness gold.
20,821 train
443 bills · 7,504 related pairs.
8,977 test
191 held-out bills · 3,213 related.
F1 progressionvs complete gold
Cosine similarity + rank0.69
+ reduced embedding dims0.72
+ shared-basis embeddings & product0.74
+ stronger GBT, F1-tuned threshold0.75
+ 2× data, large held-out split0.76
LLM-embedding semantics of bill vs. each actor (TF-IDF can't see "chip export controls" ~ NVIDIA; the embedding can), + shared-basis interaction features. Measured against a complete gold labeled by a strong annotator (sonnet-4.5, distinct from the classifier — not circular), superseding the earlier sparse-gold 0.31. F1 0.75 is the honest number; the residual is nuanced policy judgment only a reasoning model captures.