A security researcher sent close to two million AI prompts through the plumbing that sits behind a lot of AI features, with each prompt carrying a unique, trackable secret. Enough of those secrets came back to him, used by strangers within hours in some cases, to establish that data handed to an AI vendor frequently does not stay with that vendor. Ars Technica's write-up of the experiment, a leaky AI supply chain is exposing secret user data at an alarming rate, is worth ten minutes of any owner's time before they sign off on connecting an AI tool to their customer records.
The short version: the thing you think of as "our AI assistant" is usually a chain of companies, and every link in that chain can read what your customers type.
The part of an AI feature you never see
When a product has an AI chat box or an automation that drafts replies, the developer rarely wires it straight to one model company. More often they go through a broker, called an AI router or gateway. OpenRouter and Vercel are two of the named examples in the reporting. The broker takes the request and sends it to whichever model is cheapest or best suited at that moment.
That convenience has a cost that matters to you. As Ars explains, the prompt is encrypted while it travels, but it has to be decrypted at each hop so the broker can read it and decide where to send it. The request may then go to a pool of models running inside different hosting companies, and it can pass from one router to another. Each of those parties is in a position to log the contents and keep them for as long as their own policy allows, which in some cases means indefinitely.
So when your staff paste a customer's contract, an invoice dispute or a medical note into a tool, the number of organisations that can see it is not one. It is however many are in the chain, and your developer may not know the full list either.
What the canaries caught
The researcher, HD Moore of security firm Dragos, built a set of bait credentials: unused AWS access keys tied to empty accounts, passwords, and secret URLs and private repository references he controlled. Each secret went into exactly one prompt sent to one provider. If it ever showed up in use, he knew precisely which link in the chain let it out.
Per the Ars report, three hosting providers appear to have been outright breached, with an attacker copying requests at the front door. Some stolen requests had been rejected before a model ever ran, and some were flagged as Zero Data Retention. The bait keys were then used tens of thousands of times in AWS API calls while someone hunted through storage buckets and secrets managers for more credentials.
Separately, some resellers were logging prompts themselves and passing them on. The researcher identified them by sending the same request to the same model through different sellers and seeing which path leaked.
The finding that should worry a business owner most is the scraping. Ars reports that one Chinese AI lab's systems fetched the secret URLs from more than half the prompts sent to its models over a four month period, including roughly two thirds of requests marked "no data collection". The same scraper appeared when those models were reached through US hosting providers reselling them, so a customer who chose a US provider to keep data onshore did not get what they paid for. Ars notes an update: OpenRouter emailed customers to say some prompts may have been read, pointed at this research, and removed the Zero Data Retention label from the affected endpoints.
A privacy flag is a request, not a guarantee
That last detail is the one to take away. "Zero data retention" and "no data collection" are settings that a vendor promises to honour. They are not technical barriers. If the vendor is compromised, or if the vendor quietly resells your traffic to someone else, the flag does nothing. Moore's line in the piece is blunt: a privacy flag does not help when the attacker is already inside.
That is not an argument against using AI. It is an argument for treating an AI vendor exactly like any other supplier who gets a key to your premises. You would ask a cleaning company who has the key, whether they subcontract, and what happens if a key goes missing. Same questions here.
The pushback is reasonable, and it does not get you off the hook
In the Hacker News discussion of the research, one commenter said the scale seemed too incredible to be true and asked how nobody had noticed keys being reused until now. That is a fair instinct, and the results deserve independent replication. Another commenter in the Ars thread argued the damage is concentrated in the budget end of the market, the equivalent of five dollar web hosting, rather than buying directly from the major labs. Someone else countered that routers also carry the top models, which is the whole point of using one.
Both things can be true. The exposure is worst in the cheap reseller tier, and most small businesses have no idea which tier their tool is actually using, because the tool never tells them. Another commenter pointed to separate academic work that observed canary activations on another major provider, so this is not a single isolated finding.
What we would do on a build
Our position as a studio is simple. For a small or medium business, go direct to one first-tier model provider, under one contract with one privacy policy you can actually read. That is the advice Moore gives for anyone who cannot afford to run models on their own hardware, and it is the right default for almost every project we would be asked to build. Routing through a broker to shave a few cents per request is a false economy when the thing being routed is your customers' correspondence.
Beyond that, the controls are the boring ones that have always worked. Give the AI tool the narrowest possible access to your systems, not an admin token to the whole CRM. Prefer read-only where you can. Strip or mask the fields the model does not need to do its job, like full card details, dates of birth and identity document numbers. Keep a written list of which AI vendors touch which data, so that when one of them has a bad week you know within an hour whether it affects you.
And assume anything typed into a general purpose chatbot by staff is potentially public. That is a policy decision for you, not a technical one.
What to check this week
If you already have an AI feature running, ask whoever built it four questions and get the answers in writing.
- Which company actually receives our prompts, and does it forward them to anyone else?
- What credentials or access tokens did we give that vendor, what can those tokens reach, and can we narrow them?
- Are we relying on a "no retention" setting for anything sensitive, and what is our plan if that setting turns out to be decorative?
- If this vendor announced a breach tomorrow, what exactly of ours would be in it?
If your developer cannot name every company in the chain, that is your finding. Start there.


