Open your feed and it is the same show every week. A new “AI-powered” product, usually a new category, launched with a number that sounds like a solved problem. 97% accuracy. Fully autonomous. Production ready.

If you are actually building something, that show does something specific to you. Not excitement. A quieter feeling: did I miss it? Is this already done, and I am still on the floor doing it the slow way?

I felt that enough times to go read the documents behind the demos. Not blog takes. Court indictments, SEC orders, FTC settlements. Here is what is actually on the record.

Nate: the “AI” checkout was a call center in the Philippines. Nate was a shopping app that promised to “skip the checkout” using AI to complete any purchase in one tap. It raised over $50 million from investors, including a $38 million Series A in 2021. In April 2025 the U.S. Department of Justice indicted founder Albert Saniger for fraud. The DOJ said the app’s actual automation rate was “effectively zero percent.” The purchases were completed by hundreds of human contractors in a call center in the Philippines, and later Romania when a storm knocked the first one offline. Saniger faces two counts, each carrying up to 20 years. Source: DOJ, SDNY.

Presto Automation: the “voice AI” drive-thru needed a human on most orders. Presto sold restaurants an AI that would take drive-thru orders and, in its words, “eliminate human order taking.” In January 2025 the SEC charged the company for misleading AI claims. The order is specific: the product needed human agents, based mostly in the Philippines and India, to step in on roughly 70% of orders. The “95 to 99% automation” it advertised measured orders completed without the restaurant’s staff, not without any human. And they knew internally. Months before the autonomous claims went out, one executive messaged another that with humans in the loop, accuracy was not a major concern and they could get to 95% or more. Source: SEC.

DoNotPay: the “robot lawyer” was never tested against a lawyer. DoNotPay marketed itself as “the world’s first robot lawyer” that could “replace the $200-billion-dollar legal industry with artificial intelligence.” In 2025 the FTC finalized an order making the company stop the deceptive claims, pay $193,000, and notify past subscribers. The FTC found the company never tested whether its AI performed at the level of a human lawyer, and never hired attorneys to check the output. A glowing “review” it quoted, supposedly from the LA Times, was actually a high schooler’s post on the paper’s student blog. Source: FTC.

Amazon Just Walk Out: mostly disputed, worth showing fairly. Amazon’s cashierless “Just Walk Out” stores let you grab items and leave, billed automatically by “computer vision, sensor fusion, and deep learning.” In 2024 it emerged that roughly 700 of every 1,000 transactions were reviewed by a team of over 1,000 people in India. Amazon pushes back hard, saying those workers were mostly labeling video to train the models, not manually ringing you up, and calls the framing “misleading and inaccurate.” I include Amazon’s denial on purpose. But the same month, Amazon began pulling Just Walk Out from its U.S. Fresh stores. Read it however you want. That is the point of showing both sides. Source: reporting.

There is also Builder.ai, the London startup whose “AI” app-builder, Natasha, was often described as secretly powered by 700 engineers in India. That specific claim is contested and probably too clean, so I am not going to repeat it as fact. What is on the record is the money: the company collapsed into bankruptcy in 2025 after a lender seized $37 million, having reportedly told investors it was on track for around $220 million in revenue when the real figure was closer to $50 million. Even the disputed case has a clean version, and the clean version is still a cautionary one. Source: reporting.

Four different regulators. Different products, different countries. The same shape every time: a confident autonomy number in the marketing, and a human doing the actual work underneath it.

Most claims are not Nate-level fraud. The more common move is subtler and legal: measure honestly, on the wrong thing.

A model posts 97% accuracy on a benchmark. That number can be completely real and still mislead, because “97% accurate on average” and “reliable enough to run unsupervised” are different claims. Systems fail on the tail, the small share of cases that actually decide whether a customer trusts you again. Zillow found this the hard way: its algorithmic home-buying arm was shut down in 2021 after the pricing models mispriced homes badly enough to force a writedown of over $500 million and cut about 25% of staff. The average was fine. The tail was ruinous. Source: WSJ.

You see the same distortion in how benchmark progress gets reported. Stanford’s 2026 AI Index notes that one of the most cited AI coding benchmarks appeared to leap from about 60% to nearly 100% in a single year, and that leap got repeated everywhere as “AI can write software now.” The underlying number, the best raw solve rate, actually moved from about 75% to 81% over six months. Real progress. Nothing like the headline. Source: MIT Technology Review.

And when researchers test on realistic work instead of clean benchmarks, the numbers fall through the floor. Carnegie Mellon built a simulated company, real tasks and tools of the kind a knowledge worker does in a day. The best model completed about 30% of the tasks autonomously. Not 97%. Thirty. Source: The Register.

Here is the number that should travel further than any launch tweet. MIT’s NANDA initiative studied 300 real enterprise AI deployments this year, plus 150 executive interviews and a survey of 350 employees. About 95% of generative AI pilots produced no measurable P&L impact. Only about 5% saw real returns, and the difference was not a smarter model. It was tools bought and embedded deep into one specific workflow, versus generic assistants bolted on. Customer support, notably, was one of the functions where the gap between pitch and reality showed up hardest. Source: MIT report via Fortune.

Zoom out and it holds. A National Bureau of Economic Research study this year found that around 90% of firms reported no measurable AI effect on productivity, even while executives kept projecting gains ahead. The confidence and the results are running on separate tracks. Source: NBER.

Meanwhile the label gets slapped on everything. It has a name now, AI-washing, and the tell is comic: Jersey Mike’s, a sandwich chain, name-dropped “AI” 22 times in its IPO filing. Confidence and capital are loud. Neither is evidence. Source: TechCrunch.

I am not writing this to say AI does not work. It is the reason I get to build what I build. I am writing it because I work in the one category, human-quality customer service in Japan, where the gap between “tested well” and “works on a real call, in the right register, with an unhappy customer who will never say they are unhappy” is not a rounding error. It is the entire job.

So the discipline I keep relearning, and the only real takeaway here: ignore the launch, read the eval. Ask what the test actually measured, not what the headline said. Ask what happens on the tail, not the average. And when an autonomy number sounds too clean, assume there is a person in Manila or Bangalore quietly holding it up, because this year, on the public record, more than once there was.

The floor does not care about your benchmark. It cares whether the thing works on the call that is actually happening.

Abhishek

Reply anytime. I read everything.