Finance is where agent mistakes stop being embarrassing and start being regulated events. The design rule that survives contact with auditors: the model gathers and explains; deterministic systems decide and move money. Everything else in this entry is elaboration of that split.
Why the split is non-negotiable
Three properties of financial workflows make free-form model authority unacceptable:
- Regulatory classification. The EU AI Act's Annex III explicitly lists creditworthiness evaluation as high-risk, which triggers documentation, human-oversight, and accuracy obligations. In the US, model-driven decisions inherit existing model-risk-management regimes — the Federal Reserve's SR 11-7 expects inventoried models, validation independent of developers, and documented limitations. An LLM that silently decides is an unvalidated model in production.
- Determinism requirements. Ledger math, limits, sanctions screening, and transaction authorization have exact right answers. A system that is 99% accurate at arithmetic is a defective calculator. Agents add value in the fuzzy layer — document extraction, anomaly narration, evidence assembly — not by re-deriving what a rules engine already computes exactly.
- Audit reconstruction. When a regulator or dispute process asks "why was this transaction approved," the answer must be reconstructable. That means immutable logs of what the agent saw, what it extracted, what the deterministic checks returned, and who approved — the NIST AI Risk Management Framework 's govern/map/measure/manage cycle presumes exactly this traceability.
The architecture that works
A production-shaped finance agent stack has four layers: extraction (the model reads receipts, statements, contracts and emits structured fields with source spans, so every value points back to the pixels or text it came from); validation (deterministic rules recompute totals, check policy thresholds, run sanctions/KYC screens — the model's output is input to these checks, never a substitute); decision (rules auto-approve the clear cases; everything ambiguous, high-value, or policy-flagged routes to a human with the agent's evidence attached); and audit (append-only records tying each decision to model version, prompt, extracted fields, rule results, and approver identity).
PII handling runs through all four: minimize what enters the context window (the model rarely needs full account numbers to classify an expense), apply field-level protection so retention rules can target exactly the sensitive fields, and keep retention jurisdiction-aware.
Failure modes to design against
- Confident extraction of absent data — the model "reads" a total that isn't on the receipt. Source spans plus deterministic recomputation catch this; trusting the number alone doesn't.
- Authority creep — a copilot that starts as "drafts the approval memo" quietly becomes the de-facto approver because humans rubber-stamp. Measure override rates; a reviewer who never rejects is not a control.
- Cross-tenant leakage — retrieval that can see other customers' documents turns one prompt injection into a data breach. Scope retrieval credentials per case, not per service.
Sources: EU AI Act Annex III · Federal Reserve SR 11-7 · NIST AI RMF.
Related: human approval gates, prompt injection defense, agent observability, legal agents, production agents in healthcare.