Building Clinical Software For A World That Changes Faster Than Code

The FDA still struggles to keep pace with AI-driven clinical software—and that's precisely why developers must bake safety into every line of code. In environments where software directly influences real clinical decisions, rigor is not optional; it is the condition that makes progress safe, as argued in a recent Forbes Tech Council piece. Clinical software developers face a unique challenge: the world of medicine changes faster than code can be updated, yet any error can have life-or-death consequences.

The piece, published on Forbes.com, underscores that while the healthcare industry embraces artificial intelligence and machine learning, regulatory frameworks remain fragmented. The U.S. Food and Drug Administration (FDA) has approved more than 500 AI-based medical devices as of 2023, but many of these tools operate in a gray zone—especially those that continuously learn from new data. The author stresses that voluntary standards and retrospective validation are no longer sufficient; proactive, built-in safety measures are essential.

This is not a new problem. High-profile failures of clinical software—such as the Theranos debacle or flawed electronic health record (EHR) algorithms—have eroded trust. However, the acceleration of generative AI in healthcare has heightened scrutiny. In 2025, the FDA released a draft guidance on predetermined change control plans (PCCPs) for AI/ML-enabled devices, aiming to define when a software update constitutes a new device. But the guidance remains non-binding, leaving developers to navigate uncertainty.

The article highlights key requirements: rigorous testing across diverse populations, transparent documentation of model limitations, and continuous post-market surveillance. It also calls for collaboration between software engineers and clinicians—a gap that has led to many product failures. Companies like EPIC, Cerner (now Oracle Health), and startups such as Babylon Health are under pressure to prove their algorithms are safe before deployment.

Analysis from industry observers suggests that the biggest risk is not malicious hacking, but subtle model drift over time. For example, a sepsis prediction algorithm trained on data from a single hospital may fail when deployed in a rural setting with different patient demographics. The only safeguard is a culture of rigor: version control, audit trails, and a willingness to halt deployment when uncertain.

Looking ahead, the article argues that regulatory bodies will likely mandate real-world evidence for software changes. The FDA's planned Total Product Lifecycle (TPLC) approach for AI devices, expected to be finalized by 2027, could force developers to submit ongoing performance data. Forward-thinking companies are already adopting "software that monitors itself," using second-layer models to detect performance degradation. The message is clear: in clinical software, speed must never come at the cost of safety.