President Donald Trump speaks during a coronavirus task force briefing at the White House, Friday, April 10, 2020, in Washington.

President Donald Trump speaks during a coronavirus task force briefing at the White House, Friday, April 10, 2020, in Washington. AP Photo/Evan Vucci

SPECIAL REPORT: The Problem With Coronavirus Models Is How We Talk About Them

Despite what political leaders want them to say, COVID-19 models will become less predictive exactly when we need them most.

In the continuing drama of America’s coronavirus response, statistical models play the role of Hamlet’s father: foreboding, influencing the action on the stage but unable to speak for themselves. 

Some health leaders have said that various COVID-19 models’ worst projections — such as the Imperial College London model that predicted that 500,000 Britons would die  — were overly dire. That skepticism about what the models truly say will play a growing role in the discussion about when and how to “re-open” the country. But the idea that the worst of the pandemic is behind us — and that the models prove it — is misleading at best and dangerous at worst, say experts. 

Right now, the world is eager for news that the crisis is abating and that economic and social life can “return to normal.” Largely as a result of massive social distancing efforts, new infections in New York and other hotspots are going down. Some, like Andrew McCarthy at the National Review, feel like they’ve seen enough to toss the modelers out on their rears. “There is no shortage of government spin, regurgitated by media commentators, assuring us that the drastic reductions in the projections over just a few days powerfully illustrate how well social distancing and the substantial shuttering of the economy is working. Nonsense,” McCarthy writes. 

What the models are, in fact, is poorly represented by the policy makers touting them. The vast majority of them aren’t built to project infection rates out beyond a few weeks, so they don’t tell us anything about the resurgences that may come. Some of the most influential models, such as the University of Pennsylvania’s COVID-19 Hospital Impact Model for Epidemics, or CHIME, were designed to help hospital managers stock up for worst-case scenarios. Yet in the Washington, D.C., mayor’s office, CHIME is the model of choice.

CHIME is an SIR model, which stands for "susceptible, infected, recovered." Such models are hardly perfect and they don’t pretend to be, as Michael Z. Levy, an associate professor in the Department of Biostatistics, Epidemiology & Informatics at the University of Pennsylvania, recently said. You can change the outcome drastically with small changes to various parameters. For it to be of the most use, you need to be seeking the answer to a specific question, like “How many beds will this hospital need in May?”

SIR models assume that “everyone is bumping into everyone else like perfect spheres in a vacuum,” Levy said. “We didn’t act like that even before social distancing.” 

Such models are “fine for short-term forecasts, but it becomes problematic when you start to ask different questions,” he said. What will happen if social distancing is implemented in some places but not others? What happens if infections go down but testing is still inadequate? What changes if therapeutic drugs are available only in some places?

The Institute for Health Metrics and Evaluation, or IHME, model used by the White House is a more general public-health model. It predicted peak hospital resource use during the second week in April, along with ICU and hospital overcapacity. But it showed those outcomes could be mitigated through social distancing measures. When most people talk about “flattening the curve,” they’re referring to this model. 

But the model’s creators admit to a big limitation that doesn’t make it into the White House briefings. It was originally designed only to track infection rates and hospital use through June. And while its creators at the University of Washington are constantly using new data to improve it, these updates don’t provide high-confidence predictions about the fall or next year because the number of variables is growing as new measures are implemented in lots of different ways across varied locations.

Another example: a March 25 article in the New York Times  was intended to show the effects of two months of social distancing to “flatten the curve.” But it doesn’t extend past October. Wesley Pegden, an associate professor in the Department of Mathematical Sciences at Carnegie Mellon University, recently took to Medium to point out that two months of lockdown won’t end the pandemic, and people who say otherwise are mistaken or lying. In fact, a two-month lock down that isn’t followed by a uniform strategy to manage new infections without overwhelming the health system leads to a fall resurgence that eclipses the spring “peak.” 

“Two months of mitigations have not improved the outcome of the epidemic in this model, it has just delayed its terrible effects,” Pegden wrote. “In fact, because of the role of weather in the model presented in the [New York Times] article, two months of mitigations actually results in 50% more infections and deaths than two weeks of mitigations, since it pushes the peak of the epidemic to the winter instead of the summer, whose warmer months this model assumes causes lower transmission rates...No model whose purpose is to study the overall benefits of mitigations should end at a time-point before a steady-state is reached.”

In a conversation with Defense One, Pegden said that the messaging around these models has left the public ill-informed. “People have not been given enough sense of the uncertainty. They have not been given a sense of how a lockdown will not fix the problem.” 

When he tells people this, he finds them “surprised and disturbed that a two-month lockdown was not going to fix the problem,” he said.

Pegden’s concerns are much more in line with a model produced by Neil Ferguson of Imperial College London. It predicts resurgent infection spikes until a vaccine becomes widely available in a year or so. 

Even the most dire projections remain a possibility despite the recent slump in new cases, Ferguson said. 

Pegden says that a lockdown from March to May could wind up making the situation much worse later precisely because policymakers overpromised on its effects. A lockdown only buys you time, time to get nationwide testing in order, ventilators to more places, and hospitals ready for a coming surge in patients. In the United States, that hasn’t happened to the extent it needs to. 

“It is not true, a priori, that every transmission you avoid today is saving lives,” says Pegden. “You can save lives by having more transmissions among more people in the short term.” 

He acknowledges that the idea of exposing more people to the virus is counterintuitive, but, he says, “It’s not always the case that the most conservative thing you can do will save the most lives. If transmission levels revert to normal over the next year...then it would turn out that being locked down was a mistake.” 

The Models We Need Now

It’s now up to policymakers to find a coherent way to reopen the country, guided by strategy and not models that will start to fall apart as more and more variables are introduced. A total lockdown until a vaccine arrives is neither politically nor economically viable. The question is how to ease the lockdown without overwhelming the healthcare system. Pegden put out a preprint paper looking at how to do just that. In essence, he says, you want to start lifting restrictions on people who are younger, a population that is less likely to need ICU care. But you don’t want to lift restrictions on all of them since some of them may live with someone in the more vulnerable 60-plus age group. 

“Age-targeting has the potential to greatly reduce total mortalities compared with optimum choices for homogeneous measures. At the same time, we see that the best strategies for age-targeting are sensitive to the fraction of the younger population which can be released. In general, if too few people are released initially, a second wave occurs when transmission levels return to normal,” they write.

A visual representation of the effects of relaxing social distancing rules for some individuals under the age of 50 versus the entire U.S. population, from “Modeling strict age-targeted mitigation strategies for COVID-19”

Policy-makers should also begin to prepare for the illness to start having a much worse effect on people with lower incomes. Reports show that they are more likely to have underlying health conditions and are less likely to be able to work remotely or to maintain social distance, said Levy. That diversity of health outcomes is already occurring to some extent, but it’s likely to get worse. A new visualization tool shows which areas are going to be more affected based on the demographics of the population. 

Levy, in his lecture, pointed out that trying to tweak models to reflect the arrival of anti-COVID drugs won’t much improve them, since there are still too many variables to control for. “To really affect transmission dynamics,” he said, a potential drug “would need to affect these other parameters. You would need an over-the-counter-cold-sniffy-nose-and-COVID drug.”

One thing we probably should begin to model for now is mutation, since that’s going to affect the efficacy of drugs, vaccines, and other measures later on. That's going to be more useful than trying to forecast infection and recovery among a population in an environment where that will become more difficult.

A recent paper by researchers at Carnegie Mellon University offers an interesting and useful take on precisely that very hard problem. 

Predicting when a virus mutates into another virus is more challenging than mapping the spread of a contagion among a group. But looking at how information spreads and mutates online in social networks can provide some insight into how viruses spread and mutate in the real world. 

“There is some similarity from an evolution perspective via the mechanism of natural selection. A virus may go through a number of mutations (mostly a random process). However, among a large number of these, the mutation that leads to highest infectiousness will be ‘naturally selected’ i.e., it will be the one that we will see most commonly. For information it is also similar,” Osman Yağan, an associate research professor at Carnegie Mellon’s Department of Electrical and Computer Engineering, told Defense One in an email. “Several people can attempt to modify a piece of information to make it more attractive. It will be the audience who will then ‘select’ the most appealing one and that particular version of the information will be much more prevalent (say in Twitter) than others.”  

Yağan says that the research could help epidemiologists better prepare in case the virus goes through a rapid mutation process, as well as to understand the effects of protective measures in a new way. “We believe that there is some similarity between ‘a virus mutating and becoming less easy to infect individuals’ and ‘people being careful about social-distancing and thus becoming less easy to get infected.’ The analogy here is strengthened by the fact that those who do not obey social-distancing are more likely to infect each other (a la carrying a second-strain of the virus) than those who are obeying the guidelines.”

The work is partially funded by the U.S. Army. Edward Palazzolo, program manager for the Social and Cognitive Networks Program at the Army Research Office, told Defense One that it's his job to bring people together across different disciplines to find new approaches to problems, which is what they did here. “We’re always supporting basic research” he said, and it's a long-term effort. They began funding Yağan’s work in 2017. 

Speaking in a personal capacity and not as a representative of the U.S. Army, Palazzolo said that he was heartened to see more people learning some of the elementary basics of data science. But he said that contract tracing — a key to other countries’ successful approaches — will be especially difficult in the United States because of the privacy implications of contact-tracing apps. 

Palazzolo warned that leaders should avoid treating models like crystal balls. They “should never take a model and say, ‘This is the truth’,” he said. 

More importantly, they need to work with scientists to understand statistical terms — for example, what a confidence interval is. (It’s the specific range of numbers the modeler is confident that the value falls within.) “Having the right people on the team will make all the difference.” 

Of course, having the right people only matters if you listen to them — lest we rely on models that seem to predict a recovery on the basis of assumptions that are no-longer true, like a tale told by an idiot, full of sound and fury, signifying nothing. 

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.