tl;dr. Actually no, the results will probably not surprise you. After analyzing data from 2,074 testers, we found no evidence that Bionic Reading has any positive effect on reading speed. In fact, participants read 2.6 words per minute slower on average with Bionic Reading than without. That said, the difference here is so small (less than 1%), that the real takeaway is Bionic Reading has no impact on reading speed.
In June 2022, we posted an experiment to Hacker News, Reddit, and Twitter to test the claim that Bionic Reading enables you to read faster without any loss of comprehension. Part 1 of this series described in-depth the motivation, the claim, and the experiment design. In this Part 2, we're jumping right into the results.
Preliminaries
Participants were asked to read two 1,000 word essays divided into two halves resulting in articles 1A, 1B, 2A, and 2B. The first font was randomly selected for each tester (either Literata plain, or Literata with Bionic Reading applied) and then alternated from there.
The selected essays were written by the same author (Paul Graham) during the same time period (circa 2010) to minimize variation between articles. The matrix setup depicted above was intended to further control for both inter-article variation (article 1 might be easier/more interesting to read than article 2, or vice versa) and intra-article variation (the beginning of each article might be easier/more interesting to read than the end of each article, or vice versa).
When the cross-matrix reading speeds for each font are averaged, we have a pair of reading speeds per participant that can be fairly compared against one other, isolating the font as the independent variable. We also asked three multiple choice questions at the end of each article half to control for reading comprehension.
Data preparation
As with anything on the internet, completion rates followed a predictable funnel pattern. 3,334 participants completed article 1A and this sample gradually dwindled to 2,074 participants who completed all four.
There are advanced statistical techniques, such as a random effects model, that can take advantage of all the data regardless of whether each participant completed all four halves or not. But for the sake of interpretability, we've decided to disregard any incomplete entries resulting in an initial sample size of n=2074 — still more than enough data to power a robust statistical analysis.
Also as with anything on the internet, there is some junk data which must be removed.
- First, we removed any multiple entries which came from the same IP addresses (37 entries).
- Next, we removed what I call "tirekickers" — curious cats who started the experiment, rapidly scrolled to the bottom, and clicked next to see what happens. We can identify these participants by reading speeds that are humanly impossible, e.g., 2 seconds to get through 500 words. For our purposes here, we used a cutoff of 2,000 words per minute (95 entries).
- Finally, we removed what I call "quitters" — participants who started in earnest but then gave up near the end and rapidly scrolled to the bottom to see their results. For our purposes here, we can identify these participants by reading speeds on 2A or 2B that are 2x greater than their average previous reading speeds on 1A, 1B, and 2A, respectively (20 entries).
This data preparation resulted in a final dataset with n=1916.
We appreciate that outlier removal is a delicate process that risks introducing bias to the dataset, but we feel confident based on our observing of participants and our subject matter expertise that these entries are invalid. Further, we've made publicly available the anonymized raw data (pre-munging) here in the event you wish to apply your own data preparation steps to run your own analyses or to check our work.
Average speeds per font
As mentioned above, we can now average each participant's results for each font to derive a pair of speeds that can be compared apples-to-apples. Let's quickly explore the summary speed statistics for each font.
| Bionic | Non-Bionic | |
|---|---|---|
| Mean | 325.3 | 327.9 |
| Median | 294.0 | 289.6 |
| Standard Deviation | 134.6 | 148.5 |
| Minimum | 60.9 | 100.2 |
| Maximum | 1,118 | 1,293 |
| Count | 1,916 | 1,916 |
As you can see, the average speeds for both fonts (325.3 and 327.9 words per minute, respectively) were virtually identical with non-Bionic Reading actually clocking in approximately 3 words per minute faster. In percentage terms, however, this is a negligible 0.8% difference.
Average speed differences per participant
The beauty of this experiment is that we can perform various "paired" analyses which control not only for reading speed variation across articles and article position, but also reading speed variation across individuals. Observed differences, on average, should be caused by the font rather than any other lurking factor.
The simplest way to look at differences is to subtract each participant's non-Bionic Reading speed from their Bionic Reading speed. If Bionic Reading helps people read faster, we would expect to see a mean difference greater than zero words per minute.
| Delta | |
|---|---|
| Mean | -2.6 |
| Median | 1.8 |
| Standard Deviation | 58.2 |
| Minimum | -439 |
| Maximum | 293 |
| Count | 1,916 |
In plain language, this says the opposite: participants in our sample read on average 2.6 words per minute slower with Bionic Reading than without.
Average speed differences per faster font
Since posting this experiment, I've received a lot of side comments along the lines of, "Well, of course I don't expect Bionic Reading to work for most people, but for [my subpopulation], it really works." If that were the case, we might expect to see disproportionate benefits for those participants who read faster with Bionic Reading than for those who read faster without Bionic Reading. Let's look at how many participants read faster with each font and their average speed gains.
| Count | Percent | Delta (WPM) | |
|---|---|---|---|
| Bionic | 998 | 52% | 35 |
| Non-Bionic | 918 | 48% | 43 |
The number of people who read faster with Bionic Reading was slightly greater (52%) than the number of people who read faster without Bionic Reading (48%). That said, those who read faster with Bionic Reading only picked up 35 words per minute on average. In contrast, those who read faster without Bionic Reading picked up 43 words per minute. It does not appear that when Bionic Reading works, it really works.
Hypothesis test
Now let's run a paired t-test to test the null hypothesis that the average speed with Bionic Reading equals the average speed without.
| Bionic | Non-Bionic | |
|---|---|---|
| Mean | 325.3 | 327.9 |
| Variance | 18,108 | 22,055 |
| Observations | 1,916 | 1,916 |
| Hypothesized Mean Difference | 0 | |
| df | 1915 | |
| t Stat | -1.9196 | |
| P(T<=t) one-tail | 0.0275 | |
| t Critical one-tail | 1.6456 | |
| P(T<=t) two-tail | 0.0551 | |
| t Critical two-tail | 1.9612 |
In statistical terms, the mean speed difference (M = -2.6, SD = 58.2, N = 1,916) was not significantly different than zero (t = -1.92, two-tail p-value = 0.055), meaning that we fail to reject the null hypothesis. A 95% confidence interval about the mean speed difference is [0.1, -5.2], nearly implying that Bionic Reading actually has a negative impact on reading speed.
In plain language, we almost got a statistically significant result suggesting that Bionic Reading is slower. As mentioned in the introduction, however, we're talking about such small magnitudes here (less than 1% difference) that you can probably walk away with the conclusion that Bionic Reading simply has no effect.
Comprehension
Up until this point, we've been testing reading speed without regard for reading comprehension. Let's quickly explore the summary comprehension statistics for each font.
| Bionic | Non-Bionic | |
|---|---|---|
| Mean | 88% | 88% |
| Median | 100% | 100% |
| Standard Deviation | 16% | 15% |
| Minimum | 0% | 17% |
| Maximum | 100% | 100% |
| Count | 1,916 | 1,916 |
Like reading speed, comprehension across fonts was virtually identical. In fact, these comprehension numbers were so close that I feared I messed something up. That said, I triple-checked my calculations and they're correct.
These results correspond with common sense and the existing body of research showing that reading comprehension is inversely correlated with reading speed. The faster you read the less you remember and, conversely, the slower you read the more you remember. Because the reading speeds for both fonts were nearly the same in this experiment, we'd expect to see similar comprehension, which is exactly what happened.
Conclusion
Based on the data collected, it's hard to find any evidence whatsoever that Bionic Reading has any impact on reading speed or reading comprehension. Instead, our results seem to corroborate prior research that there's no universal best font, but idiosyncratic configurations might lead to gains on an individual level.
For more on this, see Accelerating Adult Readers with Typeface: A Study of Individual Preferences and Effectiveness and Towards Individuated Reading Experiences: Different Fonts Increase Reading Speed for Different Individuals (both by researchers at Adobe).
Based on the above, you might be thinking: Well then who cares whether Bionic Reading truly lives up to its claims so long as some people like it? Who cares if it's probably just a placebo? It's just one more font. Give the people what they want!
One thing that might not be obvious is that despite my casual usage of the word "font" herein, Bionic Reading is definitely not just another typeface that can be simply licensed and installed. Instead, it's more of a font style that requires making an API call to the diligent patent/trademark holder (and remunerating said IP holder) followed by the injection of all kinds of tags into your markup.
To give you an idea of what this looks like, here's the first sentence of a Paul Graham article with Bionic Reading applied:
<p><b class="b bionic">Th</b>e <b class="b bionic">bes</b>t <b class="b bionic">wa</b>y <b class="b bionic">t</b>o <b class="b bionic">com</b>e <b class="b bionic">u</b>p <b class="b bionic">wit</b>h <b class="b bionic">start</b>up <b class="b bionic">ide</b>as <b class="b bionic">i</b>s <b class="b bionic">t</b>o <b class="b bionic">as</b>k <b class="b bionic">yourse</b>lf <b class="b bionic">th</b>e <b class="b bionic">questi</b>on: <b class="b bionic">wha</b>t <b class="b bionic">d</b>o <b class="b bionic">yo</b>u <b class="b bionic">wis</b>h <b class="b bionic">someo</b>ne <b class="b bionic">wou</b>ld <b class="b bionic">mak</b>e <b class="b bionic">fo</b>r <b class="b bionic">yo</b>u?</p>
Here it is without:
<p>The best way to come up with startup ideas is to ask yourself the question: what do you wish someone would make for you?</p>
As software developers, is this a dealbreaker? Of course not. But it sure is cumbersome, particularly when you're building an offline app, so we'd prefer a modicum of evidence that the juice is worth the squeeze before introducing all this cost and complexity.
Another thing that might not be obvious is that — despite any skeptical undertones that might come through in these blog posts — we were sincerely hoping that Bionic Reading would be proven effective. We're in the business of reading technology, after all, and we genuinely believe that software has the potential to revolutionize the practice of reading. We're seeking any advantage digital reading might offer over its analog analog to persuade people to make the switch from paper to pixels. This is our mission and the reason we're building reading software in the first place.
For this reason, and because we had so much fun designing and executing this Bionic Reading experiment, we intend to run some more tests in the hopes of discovering a screen reading technique that yields material benefits. We're aware of a couple other technologies that seem interesting including BeeLine Reader, Spritz, and Sans Forgetica.
Are there any we're overlooking? Let us know on Twitter or Hacker News!
Otherwise, stay tuned for our next reading speed experiment 🤓