Masks Work. So What Went Wrong with a Highly Publicized COVID Mask Analysis?

A Harvard professor on the history of science looks at our response to the pandemic, criticizing a report that gave the false impression that masking didn’t help.

The group’s report was published by Cochrane, an organization that collects databases and periodically issues “systematic” reviews of scientific evidence relevant to health care. This year it published a paper addressing the efficacy of physical interventions to slow the spread of respiratory illness such as COVID… The review of studies of masking concluded that the “results were inconclusive…” [and] it was “uncertain whether wearing [surgical] masks or N95/P2 respirators helps to slow the spread of respiratory viruses.” Still, the authors were also uncertain about that uncertainty, stating that their confidence in their conclusion was “low to moderate.” You can see why the average person could be confused… The Cochrane finding was not that masking didn’t work but that scientists lacked sufficient evidence of sufficient quality to conclude that they worked…

Cochrane has made this mistake before. In 2016 a flurry of media reports declared that flossing your teeth was a waste of time… The answer demonstrates a third issue with the Cochrane approach: how it defines evidence. The organization states that its reviews “identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria.” The problem is what those eligibility criteria are. Cochrane Reviews base their findings on randomized controlled trials (RCTs), often called the “gold standard” of scientific evidence. But many questions can’t be answered well with RCTs, and some can’t be answered at all…
In fact, there is strong evidence that masks do work to prevent the spread of respiratory illness. It just doesn’t come from RCTs. It comes from Kansas. In July 2020 the governor of Kansas issued an executive order requiring masks in public places. Just a few weeks earlier, however, the legislature had passed a bill authorizing counties to opt out of any statewide provision. In the months that followed, COVID rates decreased in all 24 counties with mask mandates and continued to increase in 81 other counties that opted out of them… Cochrane ignored this epidemiological evidence because it didn’t meet its rigid standard.

I have called this approach “methodological fetishism,” when scientists fixate on a preferred methodology and dismiss studies that don’t follow it. Sadly, it’s not unique to Cochrane. By dogmatically insisting on a particular definition of rigor, scientists in the past have landed on wrong answers more than once.
Vox also points out that while Cochrane’s review included 78 studies, “only six were actually conducted during the Covid-19 pandemic… Instead, most of them looked at flu transmission in normal conditions, and many of them were about other interventions like hand-washing.

“Only two of the studies are about Covid and masking in particular. Furthermore, neither of those studies looked directly at whether people wear masks, but instead at whether people were encouraged or told to wear masks by researchers.”