Scientific modelling is valuable – but remember the limitations – only we didn’t.

Owl, on the basis of discussions with trusted and knowledgeable friends, has consistently questioned whether Boris Johnson and his advisers have really been “guided by the science” in a way that could be described as “scientific”. This short article describes the glaring limitations of their approach in a more eloquent way.

The troubling assumptions and use of the modelling from Im­pe­rial Col­lege that un­der­pinned the govern­ment’s be­lief that the na­tion could ride out the epi­demic by let­ting the in­fec­tion sweep through, cre­at­ing “herd im­mu­nity” on the way, is discussed in particular.

Scientific modelling is valuable – but remember the limitations

Ian Sample, Analysis, Guardian 26 March www.pressreader.com 

The lessons to be learned from the coro­n­avirus pan­demic are so nu­mer­ous they will keep schol­ars busy for decades to come. Chief among them is the value of modelling and the fact that an un­crit­i­cal re­liance on their find­ings can lead you badly astray.

A re­cent model from Ox­ford Univer­sity assessed how well dif­fer­ent out­break sce­nar­ios fit­ted the rise in Covid-19 deaths in the UK and Italy. The most ex­treme UK sce­nario as­sumed only a frac­tion of peo­ple were at risk of se­ri­ous ill­ness and es­ti­mated that, as of last week, 68% of the pop­u­la­tion had been ex­posed to the virus. The study, which has not been pub­lished or peer re­viewed, un­leashed a flurry of head­lines declar­ing that coro­n­avirus may have in­fected half the peo­ple in Bri­tain. That’s 34 mil­lion peo­ple.

A re­cent model from Ox­ford Univer­sity assessed how well dif­fer­ent out­break sce­nar­ios fit­ted the rise in Covid-19 deaths in the UK and Italy. The most ex­treme UK sce­nario as­sumed only a frac­tion of peo­ple were at risk of se­ri­ous ill­ness and es­ti­mated that, as of last week, 68% of the pop­u­la­tion had been ex­posed to the virus. The study, which has not been pub­lished or peer re­viewed, un­leashed a flurry of head­lines declar­ing that coro­n­avirus may have in­fected half the peo­ple in Bri­tain. That’s 34 mil­lion peo­ple.

But as infectious dis­ease mod­ellers and pub­lic health ex­perts, in­clud­ing the Ox­ford team them­selves, have pointed out, the model used as­sump­tions be­cause there is no hard data. No one knows what frac­tion of the pub­lic is at risk of se­ri­ous ill­ness. The study merely demon­strates how wildly dif­fer­ent sce­nar­ios can pro­duce the same tragic pat­tern of deaths – and em­pha­sises that we ur­gently need sero­log­i­cal test­ing for an­ti­bod­ies against the virus, to dis­cover which world we are in.

But as infectious dis­ease mod­ellers and pub­lic health ex­perts, in­clud­ing the Ox­ford team them­selves, have pointed out, the model used as­sump­tions be­cause there is no hard data. No one knows what frac­tion of the pub­lic is at risk of se­ri­ous ill­ness. The study merely demon­strates how wildly dif­fer­ent sce­nar­ios can pro­duce the same tragic pat­tern of deaths – and em­pha­sises that we ur­gently need sero­log­i­cal test­ing for an­ti­bod­ies against the virus, to dis­cover which world we are in.

Paul Klen­er­man, one of the Ox­ford re­searchers, called the 68% fig­ure “the most ex­treme” re­sult and ex­plained that “there is an­other ex­treme which is that only a tiny pro­por­tion have been ex­posed”. He added that the true fig­ure – which is un­known – was “likely some­where in be­tween”. In other words, the num­ber of peo­ple in­fected in Bri­tain is ei­ther very large, very small, or mid­dling. This may sound un­help­ful, but it is pre­cisely the point. “We need much more data about who has been ex­posed to in­form pol­icy,” Klen­er­man said.

Paul Klen­er­man, one of the Ox­ford re­searchers, called the 68% fig­ure “the most ex­treme” re­sult and ex­plained that “there is an­other ex­treme which is that only a tiny pro­por­tion have been ex­posed”. He added that the true fig­ure – which is un­known – was “likely some­where in be­tween”. In other words, the num­ber of peo­ple in­fected in Bri­tain is ei­ther very large, very small, or mid­dling. This may sound un­help­ful, but it is pre­cisely the point. “We need much more data about who has been ex­posed to in­form pol­icy,” Klen­er­man said.

The modelling from Im­pe­rial Col­lege that un­der­pinned the govern­ment’s be­lief that the na­tion could ride out the epi­demic by let­ting the in­fec­tion sweep through, cre­at­ing “herd im­mu­nity” on the way, was more trou­bling. The model, based on 13-year-old code for a long-feared in­fluenza pan­demic, as­sumed the demand for in­ten­sive care units would be the same for both in­fec­tions. Data from China soon showed this was dan­ger­ously wrong, but the model was only up­dated when more data poured out of Italy, where ICUs were swiftly over­whelmed and deaths shot up.

The modelling from Im­pe­rial Col­lege that un­der­pinned the govern­ment’s be­lief that the na­tion could ride out the epi­demic by let­ting the in­fec­tion sweep through, cre­at­ing “herd im­mu­nity” on the way, was more trou­bling. The model, based on 13-year-old code for a long-feared in­fluenza pan­demic, as­sumed the demand for in­ten­sive care units would be the same for both in­fec­tions. Data from China soon showed this was dan­ger­ously wrong, but the model was only up­dated when more data poured out of Italy, where ICUs were swiftly over­whelmed and deaths shot up.

It wasn’t the only short­com­ing of the Im­pe­rial model. It did not con­sider the im­pact of wide­spread, rapid test­ing; or contact trac­ing and iso­la­tion, which can be used in the early stages of an epi­demic, or in lock­down con­di­tions, to keep in­fec­tions down to such an ex­tent that when re­stric­tions are lifted the virus should not re­bound.

It wasn’t the only short­com­ing of the Im­pe­rial model. It did not con­sider the im­pact of wide­spread, rapid test­ing; or contact trac­ing and iso­la­tion, which can be used in the early stages of an epi­demic, or in lock­down con­di­tions, to keep in­fec­tions down to such an ex­tent that when re­stric­tions are lifted the virus should not re­bound.

It is not a ques­tion of whether mod­els are flawed, but in what ways are they flawed. That does not make them use­less: mod­els can be enor­mously valu­able if their short­com­ings are ap­pre­ci­ated. But, as with other sources of in­for­ma­tion, they should never be used alone.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.