Ideologically motivated rhetoric aside, it is reasonable that there would be differences in ability between men and women. Our plumbing is certainly different - why would we assume the wiring is the same? Evolutionarily men and women fill different biological niches, and this has given rise to a range of differences in our heights, weights, personalities and, yes, intellectual abilities. On the other hand, as animals go humans are not very “sexually dimorphic” - that is, the non-sexual differences between men and women are relatively small compared to other species. Humans are very good at telling men from women, but even we occassionally have trouble. An alien would probably struggle to tell the difference: a few features, like broad shoulders, are dead give-aways, but not all people have them (we’re assuming here everybody is wearing clothes). It’s not like lions or peacocks.
Summers wasn’t suggesting that women are less intelligent than men on average. He was suggesting that men have larger variation than women, particularly in technical subjects, so they will produce both more geniuses and more idiots. This would explain why most Ivy League professors are men, but it would also explain why most college graduates (at least, that are being produced today) are women. As Summers acknowledged there are large issues of gender bias, both overt and subtle, and possibly differences in innate preferences, but what got the press was his discussion of innate ability.
I wasn’t in the room, and Summer’s speech wasn’t recorded, so I can’t know how hard he sold his arguments, but I do know that they were far from solid. A glaring problem was that the models he was using assumed that male and female aptitudes (in particular, IQ scores) were distributed as a classic bell curve. In this curve, more formally called a “normal distribution”, if you know both the average IQ and how far most people deviate from it, you can perfectly extrapolate how likely people will be to have extremely high or low IQs. But that’s a pathology of normal distributions; the real world doesn’t work that way. Hypothetically male IQ could have an absolute cap at 170, whereas women had a peppering who were up in the 180s - or vice versa. Similarly to how an average doesn’t tells us what’s “normal”, studying the bulk of the population doesn’t tell us about outliers: Summers was really only speculating, possibly irresponsibly. But the bigger problems are 1) even if you grant the assumptions of a bell curve, the measured difference in variation between men and women is not nearly large enough to explain the men-biased faculty ratios seen in universities, and 2) the students being studied in the research Summers was citing were the products of the cultures they had grown up in.
Perhaps I am being too harsh on Summers given his background, and raising my expectation a bit high. He is, after all, an economist...