First, I hope you are keeping safe in these disorienting times. This is certainly a time none of us will forget.
There have been lots of really interesting examples during this pandemic of the challenge of measurement. We know that science is key to us getting through this with the minimum impact and measurement is fundamental to science. I described a measurement challenge in my last post. Here’s another one that caught my eye. Deceptively simple and yet…
On 2-Apr-2020, the UK Government announced a target of 100,000 COVID-19 tests a day by the end of April. On 30-Apr-2020, they reported 122,347 tests. So they met the target, right? Well, maybe. To quote the great Donald J. Wheeler’s First Principle for Understanding Data “No data have meaning apart from their context”. So, let’s be sceptical for a moment and see if we can understand what these 122,347 counts actually are. Would it be reasonable to include the following in the total?
-
- Tests that didn’t take place – but where there was the capacity to run those tests
- Tests where a sample was taken but has not yet been reported on as positive or negative
- The number of swabs taken within a test – so a test requiring two swabs which are both analysed counts as two tests
- Multiple tests on the same patient
- Test kits that have been sent out by post on that day but have not yet been returned (and may never be returned)
You might think that including some of these is against the spirit of the target of 100,000 COVID-19 tests a day. Of course, it depends what the question is that the measurement is trying to answer. Is it the number of people who have received test results? Or is it the number of tests supplied (whether results are in or not)? In fact, you could probably list many different questions – each that would give different numbers. Reporting from the Government doesn’t go into all this detail so we’re not sure what they include in their count. And we’re not really sure what question they are asking.
And these differences aren’t just academic. The 122,347 tests include 40,369 test kits that were sent on 30-Apr-2020 but had not been returned (yet). And 73,191 individual patients were tested i.e. a significant number of tests were repeat tests on the same patients.
So, we should perhaps not take this at face value, and we need to ask a more fundamental question – what is the goal we are trying to achieve? Then we can develop measurements that focus on telling us whether the goal has been achieved. If the goal is to have tests performed for everyone that needs them then a simple count of number of tests is not really much use on its own.
As to whether it is wise to set an arbitrary target for a measurement which seems of limited value? To quote Nicola Stonehouse, professor in molecular virology at the University of Leeds, “In terms of 100,000 as a target, I don’t know where that really came from and whether that was a plucked out of thin air target or whether that was based on any logic.” On 6-May-2020, the UK Government announced a target of 200,000 tests a day by the end of May.
Stay safe.
Text: © 2020 Dorricott MPI Ltd. All rights reserved.
Picture – The National Guard