SourceCheck

client case (person or company)
        |
        v
  Foundry agent (gpt-5-mini)
        |
        |   MCP server, Azure Functions (mcpToolTrigger)
        |--> screen_name    -> matcher.py -> SECO index, 8,604 rows
        |--> check_pep      -> OpenSanctions index, 748,000 PEPs
        |--> lookup_entity  -> GLEIF API (structure, ultimate owner)
        |
        v
  verdict: hit | a_verifier | clean
  + identifiers (SECO:5142, OpenSanctions:Q59321885)
        |
        v
  trace -> SanctionsRecallEvaluator -> CI gate (recall < 100% = red)

The MCP server holds no matching logic of its own: it delegates to matcher.py. It is hosted on Azure Functions through the native mcpToolTrigger binding (Flex Consumption, Sweden Central), guarded by an Azure-generated system key that lives only in the Foundry configuration, never in the repo. Upstream, tools/seco.py turns the consolidated SECO list, a 38 MB XML file that is unreadable as shipped, into a searchable CSV index of 8,604 records; tools/pep.py builds the 748,000-person PEP index from the OpenSanctions bulk data.

• No silent fallback. If an external source goes down, the tool returns an explicit error (gleif_unavailable, pep_index_unavailable), never an empty answer. A tool that says "no match" when its source is down manufactures an undetectable false negative, the worst failure mode in compliance.
• Comparison happens on record identifiers (SECO:5142, OpenSanctions:Q59321885), never on name strings. That is what makes a score checkable by a third party instead of credible on trust.
• Recall first. A false negative is a sanctioned party getting through: a binary failure. A false positive costs analyst time. The gate thresholds encode that asymmetry: 100% recall mandatory, precision above 70%.
• Decomposable scoring. Every match exposes the exact contribution of its five components (token 0.45, phonetic 0.20, coverage 0.20, order 0.10, exact 0.05), whose sum is the score. No black box: in a regulated environment, a score you cannot explain is worthless.
• No LLM and no embeddings in the matcher, Python standard library only, so it stays deterministic and replayable. Pipeline: Unicode normalisation, transliteration (BGN/PCGN Cyrillic, Arabic, Persian, Urdu, Han, Hangul computed from the code point, Greek, Hebrew), tokenisation, two phonetic keys including a tolerant consonant skeleton, token alignment, scoring. Roughly 200 ms to compare one name against 5,000 reference names.

• The bug found by reading a trace. On the first five instrumented cases, precision fell to 0.00 on three of them. The trace said why: the agent was copying all ten candidates returned by screen_name into its hits field, so an ordinary Swede with no connection to any list came back with ten sanctions matches. The answer stayed fluent and cited its sources: a quality evaluator would have passed it without blinking. That is the gap between "the agent answers well" and "the agent is right", in numbers.
• The fix in the tool rather than in the prompt. Fixing this by prompt meant hoping a small model would filter the noise correctly on every call. So the score floor went into screen_name (0.80 by default, tunable via SCREEN_MIN_SCORE), with a suppressed_below_floor field to keep a record of what was cut: the agent never sees sub-threshold candidates again. "Thomas Johansson" went from ten candidates to zero, while a genuine 1.0 match still passes. Defence in depth: the tool cuts the noise, the prompt handles the edge cases.
• The domain disagreement the eval surfaced. Is a PEP a hit or an escalation? The ground truth said hit, the agent said a_verifier, and its argument held up: a PEP calls for enhanced due diligence, not a block. The disagreement stayed documented rather than flattened by a prompt fix. An eval set is not only there to grade the agent, it is also there to reveal that your own definition was arguable.

• Across the 40 cases: recall 1.000, precision 0.838, zero false negatives, 8 false positives. PRD targets met (100% recall, precision above 70%).
• Precision went from 0.912 to 0.838 as more identifiers were listed: a tradeoff taken deliberately in favour of recall. The 8 false positives are mostly a_verifier cases surfacing a legitimate near miss, which the evaluator counts as a fault.
• 77 automated tests green: matcher, MCP tools, evaluator.
• CI green end to end, roughly 17 minutes: OIDC login with no stored secret, replay of all 40 cases against the deployed agent, non-regression gate, scorecard in the GitHub step summary.
• 0.2513 CHF total cost, 80% of it tokens, read from the Cost Management API. Hosting (Functions Flex Consumption) and tracing (Application Insights) at 0.00 thanks to free tiers. About 0.0015 CHF per screened case.
• 15 commits, 4 sprints delivered in 3 days against a window planned across 4 weekends.

Foundry is an AgentOps platform. Its value is not in helping you write an agent, any SDK does that: it is the factory around it, traces, evaluation, non-regression gates, identity governance.

• What it saves you writing: the observability and non-regression harness. Tracing is the most useful service of the set, it is where both real bugs became visible rather than guessed at.
• What it costs: a steep RBAC and identity curve. The "who" of a role assignment is not the user but the service's managed identity, which is counter-intuitive. A 401 announcing "invalid subscription key" when the endpoint actually wants an Entra token sends you down a false trail. And two same-named Foundry resources can coexist, one of them auto-generated, with the configuration landing on the wrong one.
• The OIDC trap, the one worth the detour: the subject GitHub now presents includes immutable numeric identifiers (repo:owner@<id>/repo@<id>:ref:refs/heads/main), not the repo:owner/repo format still shown in Microsoft and GitHub documentation. Hence an AADSTS700213 on the first attempt, solved by recreating the federated credential with the exact subject returned in the error.
• Toolboxes ended up bypassed in favour of an MCP server wired straight onto the Function. The version number is pinned in the URL by default, so updating a toolbox forces you to reconfigure the agent, unless you use the version-less endpoint that serves the default_version. Polished interface, unpolished UX.
• The real constraint was never the bill (0.25 CHF) but the throughput quota: 429s on gpt-5-mini broke a full run, which is why the eval runner carries exponential backoff and a throttle.

This is not a compliance product an analyst could use, and the PRD stated that as an explicit non-goal. 40 cases is a learning test set, not industrial validation. The rest of the caveats, as they stand:

• The application code was written by an AI (Claude Code), deliberately: that was the PRD's delegation rule. I am the author of the decisions, the architecture, the Azure configuration and the tradeoffs, not of the Python.
• The ground truth is not hand-written. It is a pipeline that derives it from real official lists (SECO, OpenSanctions) with an anti-hallucination guard: every identifier is verified as present in the list and cross-checked against the case name, and every invented name verified absent.
• The deliberate regression was demonstrated on the unit gate: date-of-birth checking knocked out on purpose, two tests red, exit code 1, the offending function named in the report. It was not demonstrated on the eval gate: no real run against a deliberately degraded agent was ever executed.
• check_pep does an exact lookup on name and aliases, not fuzzy matching, so it is less tolerant of transliterations than screen_name. An accepted limit, a question of scale across 748,000 entries.
• GLEIF only covers entities that hold an LEI, so it is blind to smaller companies. The OpenSanctions PEP data is CC-BY-NC licensed: usable for a learning project, not for a commercial product.
• Getting the agent to adhere to the output format took four prompt iterations. gpt-5-mini is an accepted limiting factor.
• Out of scope, deliberately: fine-tuning, private networking (BYO VNet), Teams or M365 Copilot publishing, multi-agent orchestration, cost-based model routing.