Everyone’s talking about the novel coronavirus, COVID-19. It is genuinely scary. And it’s people’s lives and livelihoods being affected. But with all the numbers flying around, I realised it’s quite a good example of how metrics can be mis-calculated and mislead.
For example, the apparently simple question – what is the mortality rate? is actually really difficult to determine during an epidemic. We need to determine the numerator and the denominator to estimate this. For the numerator, the number of deaths seems the right place to start. The denominator is a little more challenging though. Should it be the total population? Clearly not – so let’s take those who are known to be infected. But we know this will not be accurate: not everyone has been tested, some people have very mild symptoms etc. There is also the challenge of accurate data in such a fast-moving situation. We would need to make sure the data for the numerator and denominator are both as accurate as possible at the same time point.
Once the epidemic has run its course, scientists will be able to determine the actual mortality rate. For example, if scientists are able to develop tests to determine the population exposure (testing for antibodies to COVID-19), then they will be able to make a much better estimate of the mortality rate.
But during the epidemic, there is another challenge with this metric. It actually impacts the numerator. We don’t know whether those who are infected and not yet recovered will die. It can take 2-8 weeks to know the outcome. Some of those infected will sadly die from their infection in the future. And so, the numerator is actually an underestimate.
As we measure processes in clinical trials, we can have similar issues with metrics. If we are trying to use metrics to predict the final drop-out rate from an ongoing trial (patients who discontinue treatment during the trial), dividing the number of drop-outs to-date by the number of patients randomized will be a poor (low) estimate. A patient who has just started treatment will have had little chance to drop out. But a patient who has nearly completed treatment is unlikely to drop out. At the end of the trial, the drop-out rate will be easy to calculate. But during the trial, we need to take account of the amount of time patients have been in treatment. We should weight a patient more if they have completed, or nearly completed treatment. And less if they have just started. We would also want to be sure that the numerator and denominator were accurate at the same time point. If data on drop-outs is delayed then again, our metric will be too low. By considering carefully the way we calculate the metric, we can ensure that we have a leading indicator that helps to predict the final drop-out rate (assuming things stay as is). That might provide an early warning signal so that action can be taken early to reduce a drop-out rate that would otherwise end up invalidating the trial results.
In the mean time, let’s hope the news of this virus starts to improve soon.
Much more detailed analysis of the Case Fatality Rate of COVID-19 is available here.
Text: © 2020 Dorricott MPI Ltd. All rights reserved.