I Tested AI to Find Errors in My Medical Bills. Here’s What It Found

One in five medical bills contains an error, and a hidden $1,512 overpayment sat undiscovered in one patient's insurance claims until ChatGPT was unleashed on the paperwork. A CNET journalist fed complex health insurance documents into the AI chatbot, which systematically flagged discrepancies that had eluded human scrutiny for months. The test confirms that generative AI can transform medical billing audits into a consumer-empowering tool, but it also reveals the heavy lifting required to make the technology work accurately.

**WHO did WHAT, WHERE, WHEN, WHY it matters now:** CNET reporter Katelyn Andrews tested ChatGPT-4 on her own pile of medical billing statements from 2023, asking the AI to find overcharges, duplicate payments, and denied claims. The result: a clear $1,512 overpayment linked to two billing codes that were incorrectly applied by a hospital network. Her experiment, published in early 2025, lands at a moment when U.S. medical billing errors cost consumers an estimated $68 billion annually, according to a 2023 report by the Medical Billing Advocates of America.

**CONTEXT: Why now?** Medical billing has become one of the most opaque areas of healthcare. A 2022 study in the Journal of the American Medical Association found that 80% of hospital bills contain at least one error, and consumers are left to parse arcane CPT codes, Explanation of Benefits (EOB) statements, and insurance jargon. Until recently, challenging those errors meant endless phone calls and paper appeals. But the rise of large language models (LLMs) like ChatGPT offers a new way to scan hundreds of pages of text in seconds. The problem is that LLMs can hallucinate, misread tables, or miss context—so human oversight remains essential.

**KEY DETAILS:** Andrews uploaded PDFs of seven Explanation of Benefits forms from a hospital stay and separate physician charges totalling about $28,000. She asked ChatGPT: "Identify any duplicate charges, incorrect billing codes, or payments I should not have made." The AI spotted a mismatch between billed CPT codes 99284 (emergency visit, high severity) and 99283 (moderate severity). The hospital had coded the higher level, costing an extra $1,512 after insurance adjustments. It also flagged a $200 pharmacy line item that appeared twice. Andrews verified each finding against her insurer's records and the hospital's own explanation. She says the process took nearly five hours because she had to re-prompt ChatGPT multiple times, correct misinterpretations of tables, and manually confirm every flag.

**ANALYSIS:** The article underscores a promising but still clunky use case for AI in personal finance. "ChatGPT can be a powerful ally for consumers who are willing to put in the work," writes Andrews. "But it's not a magic button." Health policy expert Dr. Elizabeth Rosenthal, author of *An American Sickness*, says AI auditing "has enormous potential to level the playing field between patients and giant health systems, but only if the tools become easier to use." The experiment also highlights the need for better AI training on non-standard medical documents—PDFs with irregular layouts, handwritten notes, and variable abbreviations.

**OUTLOOK:** The next milestone will be consumer-facing startups that integrate AI directly into medical billing apps. Several companies, including Claimmaster and Billshark, are already testing automated dispute services powered by LLMs. Legislation like the No Surprises Act and state-level medical billing transparency laws are creating pressure for digital audit tools. Within two years, a simple voice command like "Siri, check my medical bill for errors" could become a reality—but only if AI models achieve near-perfect accuracy on formatting and code interpretation. For now, Andrews' advice: "Use ChatGPT as a second pair of eyes, but always double-check its work."