A striking finding invites a challenge: maybe it's the method, not the world. So the authors handed the data every reasonable way to slice it — including their critics' way.
An earlier study found that the American public answers emails from ostensibly Black senders less often than from white senders. As with any influential result, critics raised conceptual and empirical questions about how correspondence audits should be designed and analyzed — and whether the finding would survive a different approach.
A key clarification is about design. Comparing how the same recipient responds across conditions — a within-subjects design — substantially increases statistical power relative to comparing different people. Getting that choice right is central to detecting discrimination cleanly.
The headline of the reply is robustness: any correctly executed analysis — including the alternative approach proposed by critics — yields results substantively and statistically indistinguishable from the original finding. Black senders receive systematically lower email response rates from the public.
Methodological scrutiny is how a field gets things right — and here it ends up reinforcing the conclusion rather than overturning it. When a result holds under your own preferred analysis and your critics' too, the disagreement stops being about the number and becomes about what to do with it.
Block, Ray, Charles Crabtree, John B. Holbein, and J. Quin Monson. “Conceptual and Empirical Issues That Arise When Using Correspondence Audits to Measure Racial Discrimination.” Proceedings of the National Academy of Sciences 119(38): e2210695119, 2022.
read the paper →a methodological paper on correspondence-audit design and analysis · the panels summarize the documented conclusions; statistics live in the paper