This post was originally published on the Financial Times blog.
The FT is building a new version of its website.
Conventional wisdom states that web performance matters and the emphasis of new technical standards like H2 and products for publishers like AMP and Instant Article is speed, acceleration, and instantaneousness.
But why does this matter to our business?
We wanted to understand how much the speed of our website affected user engagement, specifically, the quantity of articles read, one of our key measures of success. Using that data we then wanted to quantify the impact on our revenue.
Method
We ran two tests, each about four weeks in duration.
In the first test subscribers were divided into two equal groups. The control group saw the standard website and the other half saw a five-second delay added to each page load.
The first test showed a significant drop in engagement from the slower group so the test was repeated, this time our users were segmented into four groups. The control group, again, saw the website at normal speed, the second group saw a one-second delay added to each page load, the third group a two-second delay, and the last group a three-second delay.
The delay was achieved by inserting a blocking CSS call within each HTML page. The CSS referenced a file that was configured to (artificially) respond in one, two and three seconds, depending on the test variant.
On fast connections, the new FT site generally renders within two seconds, about three seconds quicker than ft.com. The delays we introduced were in addition to this baseline load time.
We were interesting in looking at the effect of a delay on three axes :-
- By device class (and by proxy, network speed) – mobile, tablet, desktop
- By users’ historical level of engagement on ft.com – high, medium, low
- Time – 7 day, 28 day windows
Impact on Session Depth
Session depth is the number of pages a user views during a visit to FT.com. We used this as our conversion rate (from users who viewed at least two pages, up to fifteen) and calculated the difference between users who converted in each of the test variants.
We saw little impact for very short visits of two pages, but for users visiting three pages or more we see a gradual decline across all the test variants – the deeper the journey the greater the drop-off in engagement.
At the slower speeds, five seconds and three seconds, we saw this trend saturate around ten pages (11% worse for five seconds, and 7.5% for the three seconds variant), but at two seconds and one second the difference kept growing.
This trend was apparent from the very first page view and significantly accentuated for subsequent page views on mobile devices where network speed is usually slower.
Tablet users, which typically arrive on faster networks, also produced this result, so network speed is not the only factor we must take into account.
7-Day vs 28-Day Impact
As a publisher, engagement depth metrics are a useful tool but our main interest lies in long-term engagement rather than the one captured in one single visit.
We evaluated article consumption by subscribers over January.
We measured articles read over the course of seven days to gauge immediate impact and also the following twenty-eight days to see if the trend continued or if the user’s reaction to the change abated over time.
Over the first seven days of a user arriving at the test and experiencing a one second delay we saw a 4.9% drop in the number of articles they read compared to the control group. The difference grew to 7.2% for the users in the three seconds delay variant.
After twenty-eight days of being in the test the difference grew for the two- and three-second variants, 4.4% to 5% and 7.2% to 7.9% respectively.
In conclusion, over the testing period users read fewer articles each day whilst experiencing delays loading each web page.
| Page load time | 7 days impact | 28 days |
| 1 second slower | -4.9% | -4.6% |
| 2 second slower | – | -5.0% |
| 3 second slower | -7.2% | -7.9% |
High vs Low Engagement
We surveyed the article consumption of highly engaged readers with less frequent visitors.
Our highly engaged readers show less adverse reaction to the shortest delay we placed on them than the overall sample during both the seven and twenty-eight day periods (Eg, 1.2% at one second over seven days, 3.6% over 28 days), though with the longer delays there was not as clear distinction.
We think highly engaged users are prepared to be patient with a slight degradation in website performance due to their need or habit around the information we publish.
Users with lower engagement rates showed an extreme reaction to even a short delay however this did not carry to twenty-eight days.
This data point did not reach statistical significance. Low engaged users, by definition, use our website less than highly engaged people and therefore we collected less information on how performance affects their behaviour. We need to repeat the test for an extended duration to satisfy this gap in the data.
| Page Load Time | High Engagement users | Low Engagement users | ||
| 7 days | 28 days | 7 days | 28 days | |
| 1 seconds | – | – | -23.9% | – |
| 2 seconds | – | -6% | 19.6% | 31.1% |
| 3 seconds | -7.3% | -8.4% | – | – |
Devices: Mobile vs Tablet vs Desktop
Lastly, we compared the consumption of users arriving with different types of devices.
Over the seven-day window, mobile users seemed unaffected by a one and two-second delay, although we see a slight drop in usage at three seconds. After twenty-eight days, mobile statistics of each variant gravitated towards the average decline in engagement we saw elsewhere.
Our hypothesis is that mobile users are receptive to small delays as they may blame the network connection, but as delay becomes unreasonable and consistent the engagement drops off at higher rates.
Tablet users saw a dramatic decline in engagement (18.8%), even at the introduction of a one-second delay, which lessens as two and three seconds. This trend after twenty-eight days was not statistically conclusive. Do tablet users have a lower patience threshold for lags in render speed? We’d need further research to investigate this.
Desktop also saw a decline, this falls in line with the impacts previously mentioned.
| Page Load Time | 7 days | 28 days | ||||
| Mobile | Tablet | Desktop | Mobile | Tablet | Desktop | |
| 1 secs | – | -18.8% | -6.3% | – | -14.4% | – |
| 2 secs | – | – | – | – | – | – |
| 3 secs | – | – | -6.5% | -9.0% | – | -6.5% |
Revenue Impact
We used the data above to understand the impact on revenues.
Lower engagement impacts two revenue streams at the FT.
Firstly, we have long measured the correlation between engagement and subscription renewal rate. When our user engagement drops, so do our subscriptions.
Secondly, our advertising inventory is measured in part by page views and in part by the time each advert is seen for – the longer a user stays on our website the greater their exposure to our advertisements.
It’s clear from our test that the speed of our website affects both of these revenue streams, over the short term, to the tune of hundreds of thousands of pounds, and in the long-term millions.
In Summary
The speed of the site negatively impacts a user’s session depth, no matter how small the delay.
Slow sites also have a detrimental effect on the number of articles people read.
Largely, the slower the site, the greater the effect.
This trend is borne out over time and typically worsens with continued delays.
The two anomalies to this general trend appear to be people who are highly engaged with FT.com and mobile users, both of whom seem less deterred by a short-term drop in performance.
The data suggests, both in terms of user experience and financial impact, that there are clear and highly valued benefits in making the site even faster. From this research, we’ve chosen to invest even more time in making every aspect of the new FT.com website even faster over the coming months.
Some of our data did not reach 95% statistical significance, therefore, has been omitted from the data tables above, as indicated by a hyphen in the table cells.
Thanks to Thomas Woolrich Jacob Catt, and Tom Parker for their huge help during the development and analysis of this test.
Matt Chadburn is Principle Developer for FT.com.He was previously Developer Manager at Guardian News and Media and Web Development Lead at BBC News.
Nice to see so much research methodology behind your tale but ‘slow sites impede user experience’ is a variation on a ‘pope prays’ and ‘bear seen advancing on woods, toilet paper in hand’ story.