Move fast and not break things
In the late 1990s I was studying clinical medicine. The medical education back then split early pre-clinical years (basic sciences) from the later clinical years (patients). The deductive reasoning of the former sometimes jarred with the pattern recognition of the latter.
I was on a cardiology ward, examining a patient with heart failure. His heart was failing to pump blood properly, which caused fluid to get backed up, collecting in his ankles and lungs. I knew, from my pre-clinical years, there were two reasons for this: his heart did not pump hard enough, and between beats, did not fill up fast enough.
The best practice at the time focused on removing excess fluid (diuretics) and making his heart beat harder (inotropic drugs). I asked the registrar who was teaching us if giving his heart more time to fill up would help. A beta-blocker, perhaps? The answer came quickly: beta-blockers were contraindicated in heart failure.
Every generation questions the best practice of the day. This is an important way the system improves, and I lived through one such improvement in the management of peptic ulcer disease. The established theory at the time was that it was due to stress and bad diet. In a now famous experiment, Australian scientist Barry Marshall drank a culture of Helicobacter pylori to prove his infectious disease hypothesis. His research, published in 1984, was dismissed for nearly a decade before general clinical practice caught up (he went on to win the Nobel prize in 2005).
These improvements in care did not require new basic science or drug discovery, but simply a re-evaluation of the evidence. If this had happened sooner, lives would have been saved.
Unlike maths, science does not prove things. It fails to disprove. Regardless of the field, when a scientific claim is made, the same question is asked: can we disprove this? If we can't, and we've controlled for all the variables, we accept, for now, that it's correct. If we can disprove it, we know it's false.
Medicine is full of variables that are difficult to control. To have enough confidence that the benefits of our treatments outweigh the risks, we need to test larger numbers of different types of people over long periods of time.
Every year, scientists review the current best practice against new evidence. So delays to this process have real-world consequences. People are harmed.
Software is easier to evaluate than medical therapies. The variables are easier to control. When things don't work, you can find out faster. In the early days of Facebook, Mark Zuckerberg famously said "Move fast and break things". He wanted what worked, fast.
So how should we evaluate medical technologies whose impact can match that of a drug? Healthcare systems around the world are trying to work this out. Hospitals are evaluating AI technologies themselves, at the same time as governments are working out how to best regulate these evaluations. Everyone wants technologies that work. Everyone understands the cost of delays.
Companies that claim to improve healthcare must be judged by the same fundamental question: can we disprove it? Our desire to see the system reformed, healthcare improved, and lives saved should not get in the way of this. And if we can disprove it, let's do it quickly.
The speed of our evaluation will depend on the nature of the technology. Many will be evaluated as they always have been, variables controlled, failure modes identified. Some appear to be probabilistic and may be evaluated more like drugs than traditional software. Others adapt and learn from every interaction, presenting a harder problem, a moving target we will need to control. The system should test new ideas quickly, try hard to disprove them, embrace what remains.
Today, beta-blockers are indicated in patients with stable heart failure. The same drugs that were contraindicated in the 1990s have been re-evaluated. The process of evaluating medical treatment is slow. New technologies, affecting clinical systems, need to be evaluated faster. We know the cost of delays. Let's move fast and not break things.