I looked at the Nature paper Rapid emergence of a maths gender gap in first grade when it came out a year ago, but I got stuck on this chart (Figure 1c) of histograms of percentiles and the lack of shared data. A little context: the study is looking at scores of half a million students per year in France, tested at three time points: T1 is when they start first grade in September; T2 is four months later; and T3 is the start of second grade. (Oddly there’s no end-of-grade test they could use.)
The data indirection, looking at percentile frequencies instead of scores, adds one challenge, but just when you think you understand that, you notice the bars for T1 and T2 aren’t summing to a constant amount. The biggest example is in T2 where the bin for 76-80% has significantly fewer boys and girls as the next bin, 80-84%. I kept thinking I was reading it wrong, but finally realized the bumpiness is due to percentile ties. With 500k students represented in each histogram, you wouldn’t expect a few ties to be a problem, but looking at the provided simulated data, I think I can see why. Here are the distributions of the 21 tests for the 1000 students in the sample. T1 in yellow has 6 tests, T2 in orange has 6 tests, and T3 in red has 9 tests.
The test results are mostly integers and for T1 and T2, most of the tests have very little differentiation. For instance, about 85% of students got a perfect score on the second test. With so little differentiation and fewer tests, T1 and T2 are indeed prone to many tie scores. T3 has more tests and better differentiation, and its histograms follow the expected complementary pattern: when one goes up the other goes down to keep the bin count the same.
As a check, I resamples the simulated data to get the size above 500k students, and remade the histograms.
The details are different because of the randomization, but the amount of spikiness is consistent.
Z-scores over time
The next chart (Figure 1b) from the paper should be easier to understand since it shows relative test scores across time.
However, it also mixes in school type and socioeconomic status, and the line connection is across the socioeconomic status levels, which make it even harder to see the time relationship. The differences across socioeconomic status levels is not news and not the subject of the paper, so I don’t know why that gets the graphical emphasis.
Here’s my remake that makes time the primary factor and highlights the gender gap with shading.
It’s important to note that the z-scores are normalized to an average of 0 and a standard deviation of 1 separately for each time period but over all groups together. So a z-score that goes down over time doesn’t mean a group is getting worse at math, just that their relative score is worse.
The original broke down the regular public school students into four socioeconomic status levels instead of two levels like the other school types, and I collapsed the four down to two since they weren’t adding much.
This view makes a couple things clearer.
- The gap widens more from T2 to T3. The paper does discuss that, but it’s impossible to know how much of the change occurs in the second half of the school year versus during the summer break. However, the headline finding of the paper is “rapid emergence”, which would correspond more to the T1 to T2 period. The T1 to T2 widening is real; I wonder if T3 should be left off of the graph so as not to distract from the main finding.
- The pattern for the Priority and High Priority schools is different. Those schools are typically in disadvantaged areas and receive extra resources. The girls there improved their relative scores in the first term.
Data collection
As mentioned, the paper didn’t provide any raw data: only a simulation of 1000 student scores and some summary tables. I used the simulated data for the histograms above, but for the line chart makeover I extracted the data from the chart using the AI tool graph2table, which I heard about from the Normal Curves podcast. I tried it several times (at the free tier of one use per day) with no success. I was about to give up on it, but after hearing Regina and Kristin rave about it even before it was a sponsor of their podcast, I gave it one more try and this time it extracted the data perfectly.
Graphical oddities
I find it odd that the histogram bars in Figure 1c above don’t start at the origin. I can only guess someone decided to add some padding for aesthetics during post-production. Also, Per Engzell noted on Bluesky that some of the dots are weirdly scaled. I can’t imagine how that happened, and now I notice those lines are not well-centered in their confidence bands.
