No, masks do not cut COVID incidence by 53%

On Thursday last week, The Guardian published an article claiming that a new global study had demonstrated that mask-wearing cuts COVID incidence by 53%. The reddit thread discussing the article was littered with mildly smug comments: one user wrote “This’ll please those anti-maskers”, and promptly received the reply “Bbbbut they can’t breeeeeathe in a mask”. As someone who is emphatically pro-mask, I dislike these sorts of comments for a couple of reasons — firstly, they’re the sort of pointless in-group signalling that I imagine is very unlikely to persuade anyone who is actually a convinced anti-masker (although I think it is possible that this sort of signalling hardens the stance of people who are just leaning pro-mask, which I suppose could be a good thing). But mainly I dislike these comments because they don’t engage with what is an obviously misleading claim from the Guardian, and instead go straight into tribal signalling seemingly without having read either the article or the meta-analysis it was based on.

So, why shouldn’t we take the 53% reduction claim seriously? The main reason is that the studies included in the meta-analysis that the Guardian is reporting on are ripe for confounding - even when introducing controls for obesity, age, gender, and so on, it is immensely likely that the kinds of people who wear masks are likely to be more cautious in ways that make them less likely to get the virus regardless of their mask use. Similarly, the studies that compare countries’ rates of mask-wearing and how the rates are correlated to per-capita COVID deaths are likely to be missing out on how efficacious the strategies of governments were (even when controlling for duration of lockdown, tests per capita, and so on). There’s also a potential problem with the ecological fallacy in the country comparisons specifically - just because a country as a whole had fewer COVID cases than another country doesn’t mean that there were fewer cases among those who wore masks (admittedly, this is a much smaller problem than confounding, but it is still a problem). The authors of the meta-analysis are definitely aware of this - the chart above, from the paper, shows the risk of bias in each study - note that not a single study included is thought the have less than a ‘moderate’ risk of confounding, and the vast majority of studies included have either a high or critical risk of confounding. The risk of bias here is assessed using the ROBINS-I tool for observational data and the RoB 2 tool for studies that involved interventions.

Taking a random study included in the meta-analysis lets us take a peek at exactly how these kinds of studies work - Leffler et al studies the sources of variation in per-capita mortality from COVID. The table above shows the controls included in the study, and there are some obvious flaws here (are the two age buckets really enough to get a good picture of how many COVID deaths we should expect? 61 year olds are much less likely to die of COVID than 90 year olds). But beyond this, the chance that there are omitted variables here is just huge - the authors themselves write that there is significant potential for confounding. The way the conclusion of the study also suggests they aren’t entirely confident that masks do such a wonderful job of preventing infection - they write:

Given the low levels of coronavirus mortality seen in the Asian countries which adopted widespread public mask usage early in the outbreak, it seems highly unlikely that masks are harmful.

It’s true! Masks probably aren’t harmful! But we might want more evidence before declaring that the effect size of mask-wearing on reducing COVID cases is particularly large. The most useful study in the analysis is probably the RCT from Bangladesh, but even that has some potential for confounding with regards to the effects of masks specifically. The design of the study was pretty simple: villages were randomly assigned to either receive the treatment that involved several interventions (free masks, information on the importance of masking and physical distancing, and so on), whereas control villages did not receive any interventions. This is a decent attempt at causal inference, but even then it isn’t specifically about the causal impact of mask wearing, because villages that received the intervention were also told about the importance of social distancing and accordingly 21% more likely to socially distance from one another, which is likely to have had an effect.

The result of the Bangladesh RCT is much more modest than almost all of the other studies - in villages randomly assigned the intervention, the relative reduction in COVID was about 9.3%, adjusting for baseline covariates. This is still a significant reduction, and we should take it seriously, but it is very far off a 53% reduction. The annoying thing about the people sharing stories about how masks are effective is that they’re completely right, they’re just overstating how effective masks are for no good reason. While I’m not confident enough to make the claim that this kind of misleading headline actually makes people less confident in claims about the efficacy of mask-wearing, it certainly seems like a mask-sceptic might be liable to delve into the meta-analysis, note the admission that confounding is a big problem in basically all of the studies, and conclude that the mask-advocates are making their claims on the basis of junk studies.

If you liked this blog, please consider subscribing. If you already subscribe, maybe consider following me on Twitter!