Lindsay Rodman, a Marine officer (and, we are told, a Harvard and Duke graduate), attempts to unskew the numbers on sexual assaults in the military in the Wall Street Journal. “The Pentagon’s Bad Math on Sexual Assault” starts by saying that:
In the days since the Defense Department’s May 7 release of its 2012 Annual Report on Sexual Assault in the Military, the media and lawmakers have been abuzz. The report’s estimate that last year 26,000 service members experienced unwanted sexual contact prompted many to conclude, incorrectly, that this reliably estimated the number of victims of sexual assault. The 2012 estimate was also significantly higher than the last estimate, causing some to proclaim a growing “epidemic” of sexual assault in the military. The truth is that the 26,000 figure is such bad math—derived from an unscientific sample set and extrapolated military-wide—that no conclusions can be drawn from it.
The math is bad, Rodman argues, because the survey sample used was unrepresentative, and because the extrapolation led to empirically silly conclusions. Thus, the survey was sent to 108,478 service members and returned by 22,792 of them, about 1 in 5. This is bad data because:
I am one of those who responded to the survey after receiving an email with an online link. None of the males in my office received the email, though nearly every other female did. We have no way of knowing the exact number of male or female respondents to the 2012 WGRA survey because that information wasn’t released.
It’s not entirely clear what Rodman’s point here (the anecdote is not meaningful for statistical purposes) but her implication seems to be that the sample size was skewed towards female members of the military and thus not representative of a military which is about 85% male.
This is not an uncommon problem in surveys. Pollsters can pick a nicely random survey pool to start off, but the vagaries of who responds often creates an unrepresentative group of answers. Like lots of not-uncommon problems, there are well-developed strategies for dealing with it. Pollsters will pick and choose which responses to use so that the collective results are representative of the larger group. To not do so would be statistical malpractice of the worst sort.
Which is why, of course, the survey does, in fact, reweight the responses to ensure a good sample. The result, the study points out, is that “because the results of comparisons are based on a weighted, representative sample, the reader can infer that the results generalize to the active duty force, within the margin of error.”
Rodman is not arguing that the reweighing was incorrect. Instead, she seems to be arguing that it never happened at all, and that’s flatly not the case.
She is also incorrect on her second criticism, and incorrect at such a basic level that I despair of the math skills of the graduates of the Ivy League (and their peers):
These numbers vary widely because incidents involving unwanted sexual contact cannot accurately be extrapolated military-wide using this survey. The number of active-duty personnel is more than one million. The U.S. military as a whole is 14.6% female. Though the 2012 survey does not specify the gender composition of its respondents, the 2010 respondents were 42% female (10,029 women and 14,000 men). Nevertheless, to achieve the 26,000 military-wide estimate in 2012 (and 19,000 in 2010) over half of the victims must have been male. Of course, male victims do exist, but empirically males do not constitute anywhere near the majority of victims of unwanted sexual contact—no less sexual assault.
Urk. This is just appallingly bad. Rodman would have a point if she were talking about the civilian world, where men and women each make up about 50% of the population. Women are sexually assaulted at much higher rates and thus any result that had a majority of male assaultees in it would likely be unrepresentative of the population as a whole.
But, as Rodman herself has just told us, we’re not talking about an evenly divided population. Instead, we’re talking about one which which men substantially outnumber women. Thus, even if women are sexually assaulted at a much higher rate than man, the larger number of men in the group could create a situation where there are more assaults on males.
In fact, that seems to be what happened. The survey says that 6.1% of women reported unwanted sexual contact in 2012, as opposed to 1.2% of the men. Women were more than 5 times likelier than men to experience unwanted sexual contact. In a military of 1.42 million (2012 numbers, p. 243; ~1.2m men, ~200K women), that would work out to 14,000 assaults on men, and 12,000 assaults on women. Men were attacked at a much lower rate, but were nonetheless empirically a majority of the victims.
In essence, then, the mathematical criticisms of the study are simply, and badly, wrong. That’s a shame. The integration of women into all parts of the military is one of the crucial challenges facing the armed forces right now. The discussion on how to do that, and how to deal with the problems that arise, needs useful contributions above all. This is not one.