The exploit that passed review
A domain check that looked fine on careful review let a malformed URL pass as an official source. Running the exploit, not reading the code, is what caught it.
Part of my claim-verification pipeline checks that a source is real: if a model cites a link, the pipeline confirms the link actually comes from who it claims. This is about a bug in that check that a careful review missed and running the code caught.
Specifically: if a model says "TSMC announced X" and gives a link, the source-verification step confirms that link actually comes from an official first-party domain, tsmc.com, and not some unrelated site dressed up to look like one. The way it does that is simple: pull the hostname out of the URL, compare it against an allowlist of domains we trust for that company.
(The allowlist is small and hand-maintained, and the function knows nothing about the companies themselves. For this scale it's fine. At a bigger one it'd be worth revisiting with a proper domain library.)
I started by pulling the hostname with Python's netloc, which is the URL's authority component. It includes the host, but also extras like the port. Then someone can hand you this:
https://evil.example:.tsmc.com/fake-news
Look at it. It obviously shouldn't pass. The real site is evil.example. But netloc doesn't return evil.example, it returns the whole raw string evil.example:.tsmc.com, because the parser treats everything after that first colon as a port and hands the lot back untouched. So the check asks "does evil.example:.tsmc.com end with .tsmc.com?" and the answer is yes. A malicious link passes as an official TSMC source.
In hindsight it's obvious. It didn't surface on its own, though. It came out when I had two models run adversarial reviews of the same function. Gemini tried this class of attack and found it. ChatGPT tested a different class thoroughly, the fake-lookalike-domain kind, and correctly found nothing wrong there, but never tried breaking the parser with a malformed port. Same code, two capable reviews, opposite conclusions.
I didn't take either of their words for it. I ran the exploit against the real function, and it passed when it should have failed. That's the part that matters. Reading the code carefully is good. Running an attack against it and watching what actually happens is better, and here it was the only thing that settled which review was right.
The fix was one line: use hostname instead of netloc. hostname strips the port and returns evil.example, which fails the check like it should. Then I added it to the test suite, so the exact exploit gets run on every change and can never quietly come back.
The aim of this project was verification that actually discriminates: a check only counts if it gives a different answer in the world where the claim is false. In theory this check did that. Whether it did it in practice was a different question, and a much higher standard. A review that reads a check can only go so far. Running it, testing it hard, surfaces the bugs a review was never going to conjure up.