Someone once quipped that I only read journal articles by looking at the pictures. I admit the graphs are the first things I look at. Next I check the data availability statement, to see if I can better understand the graphs with a little exploratory analysis. After that, I might also read the text of the article.
The article Countrywide natural experiment links built environment to physical activity uses step-count data from thousands of people to quantify the effect of a city’s walkability on step count activity. Though it’s a “natural” experiment without designed control groups, they cleverly only look at users who appeared to relocate from one city to another. It has a nice variety of graphs and a very promising data availability statement:
Data availability
Data are available at GitHub (https://github.com/behavioral-data/movers-public).
Code availability
Code is available at GitHub (https://github.com/behavioral-data/movers-public).
The code and data to recreate the graphs is indeed provided, but that’s not the same as the raw data that could be used for exploration and verification of the analysis, which is presumably the main reason for sharing data. For instance, I was interested in a deeper dive of the data behind this chart since non-linear curves have many fitting options.
However, the shared data only has the coordinates of the line and shaded area, not the raw data behind them. Unfortunately, that’s not too uncommon.
This excerpt from Figure 1 underscores the main point of the paper.
It shows two moves and their corresponding before-and-after daily step counts in inset graphs. It seems like a clear and dramatic shift before and after the moves. It takes a little effort to mentally align the axes and get the before-and-after order correct, so I wanted to try putting the before-and-after in one chart and maybe look at a few more moves. However, the code for the chart was a bit surprising. Here’s the code for the Seattle inset.
with plt.rc_context({'figure.autolayout': True}):
fig, ax = plt.subplots(figsize=(4, 2));
pre_x = range(-35, -5);
y = np.random.normal(loc=from_df.loc[from_df['from_loc'] == 'Seattle, WA', 'pre_avg'], scale=50., size=(len(pre_x), ));
plt.plot(pre_x, y, lw=5., c='#aa3939');
plt.ylim(5800, 7000);
ax.grid(False);
for item in ([ax.xaxis.label, ax.yaxis.label] + ax.get_xticklabels() + ax.get_yticklabels()):
item.set_fontsize(axis_fontsize);
ax.set(xlabel=r'Days from Move $\left(t - t_{move}\right)$', ylabel='Daily Steps', xticks=range(-35, -4, 10));That wiggle on the line is just random noise! Presumably the mean location is from actual data. The noise is random normal with a standard deviation of 50, which isn’t very representative of the actual data. (Not surprising if you’ve ever looked at step count data.) The text of the paper says the standard deviation for all the step data is 3000 steps. So the line should be much noisier.
One could argue the extra noise would just be distracting and that the important thing is the mean comparison, and so the actual variation has been replaced with an abstraction. Or that the line is an average of hundreds of moves (which would reduce the noise). Whatever the reason, it needs to be explained in the caption or article, which I’m not finding. Otherwise, it’s quite misleading.
There was one figure that did come with raw data, and it’s perhaps the most important, showing the change in step counts versus the change in walkability.
Even though the dots in the image are city-level aggregates, the raw data for all 7500 moves was provided. It was just the walk score difference and the steps differences though, not actual city labels. Happily, I was able to reproduce the both the binned summary line and the regression line from the raw data.
In this view, the raw steps values are severely clipped, but it’s enough to see that the walkability variation on the X axis is far from uniform, meaning the outer bins have far less data (which is reflected in the larger confidence intervals). Here’s the same chart with no axis clipping.
I’m surprised to see such extreme values of Δ daily steps. Supposedly (there’s no daily raw data to check it against), those are averaged over several weeks. And the paper says they discarded step counts over 50,000.
If we restrict the graph to show the bulk of the raw data values, we get:
The large amount variation doesn’t necessarily take away from the significance of the findings. If anything, it shows the power of statistics at quantifying change in the presence of noise.
Zooming in on smaller walkability differences (-25 to 25 on the X axis), which covers the vast majority of moves, and adding a spline smoother, the step count change is less dramatic but still probably real.
