Defining and measuring website performance

Performance & Optimization

Optimization is the attempt to increase or decrease some measureable value to its maximum or minimum. Examples: We can optimize a website for conversion to sales, time spent on site, number of visitors, and so on.

We talk about improving performance as if it is a form of optimization. We consider it to be the attempt to decrease the time it takes for a page to load. However, as discussed below, there is much more to the story.

From the User’s Viewpoint

Ultimately, the users decide a website’s fate. If they condemn its performance en masse, that’s the end of the website. No explanations or pleas for understanding will change this. Welcome to life in the cyberworld!

Users tend to judge a website as acceptable, slow, or too-slow-to-bother-with. If they judge a website as acceptable, they may continue to use it now and at times in the future. If they judge it as too-slow-to-bother-with, they will go elsewhere and will not likely return.

If the users judge the website as slow, but not too-slow-to-bother-with, they may stick around, but they will not be thinking kind thoughts. In some ways, slow is worse than too-slow-to-bother-with because of the incessant repetition of those negative thoughts. Anything repeated often enough will be believed and will likely be shared with others.

A recent study showed that Brits spend an average of two days a year waiting for slow websites. Can you imagine what they’re thinking and saying about the companies that operate those sites?

What Exactly is the User’s Problem?

Since the users’ acceptance or rejection of the website determines its fate and since that acceptance or rejection is partly determined by performance, we are compelled to define performance in terms of the end-user experience. Let me repeat that: We are compelled to define performance in terms of the end-user experience.

So what is it about the user’s experience that leads him to an acceptable, slow, or too-slow-to-bother-with decision? It seems almost obvious, doesn’t it? It’s the time spent waiting.

Every time the user performs an action (e.g., clicks a button, moves the mouse, presses a key on the keyboard), he expects to be able to perform his next action immediately. Having to wait can annoy him, distract him, or merely make him less productive. Since companies value their employees according to their productivity, making these employees wait reduces their value to their employers, and that puts their jobs at risk. This point is not lost on our users.

The user does not always have to wait until the page is fully loaded to get on with the next thing he wants to do. The elements with which he wants to interact may become visible and functional earlier. If the web page is not jumping around because of reflows, the user can continue. He’s done waiting even though the web page is not fully loaded.

We techies tend to focus on what happens from the time a user requests a web page to the time the page is fully available. We tend not to mention the wait times after the page is delivered. Example: If a user rolls the mouse over an element, he may have to wait for a popup to time out and disappear. We don’t talk about that in performance discussions, but we should.

How to Measure Performance

Performance is inversely proportional to the time the user spends waiting, so the user’s wait time, not the application’s wait time, is what should be measured.

If the keyboard and mouse are silent, computer-based measurements may count inactivity as wait time when it is not. If the inactivity is due to a required non-computer activity (e.g., reading, thinking, planning), then it should count as productive time, not wait time.

If the inactivity is due to a user’s decision to go do something else (e.g., do some other job, take a break), then it should not count as wait time or productive time (for this application). However, any of these things should count as wait time if the user does them merely to pass the time while waiting for the computer to do its thing.

Semi-productive times should not be scored as heavily as the thumb-twiddling times. If the user has something productive to do while he is waiting, that’s still a performance hit, but it’s not as bad as waiting and having nothing to do. User multi-tasking, even when it is not within the developer’s control, can affect the perception of performance. And perception is reality.

Let’s consider an example of that last point. If an automated task will take about half an hour, but requires the user to interact in some trivial way (e.g., click a button to move to the next step or make a choice that could have been made up front) every minute or two, the user will likely consider that to be poor performance. However, if he knows that he has a free half-hour because all the questions are answered and all the choices are made up front, he can switch to some other job and be productive elsewhere. Please note both points: He must be free for some amount of time, and he must know that he is free for that amount of time. [Please note: This is not the best option. It is merely a better option. The best option would be to spin that task off into a concurrent background process and let the user continue to work with the application.]

Monitoring and analyzing user experience is the best way to measure a web page’s performance. Performance should be measured as P ÷ (P+W), where W is the total wait time and P is the total productive time. Special attention should be paid to the individual wait times that make up the total – the large ones should be examined closely.

"web performance" || "website performance" || "website optimization" || "webapp performance"

showing it to be one of the most talked-about topics on the World-Wide Web. The following examples show that there is a reason for this much buzz.

In February 2012, Walmart’s research revealed an inverse relationship between page load times and conversion rates. As average load times increased from 0.5 to 2.5 seconds, the conversion to sales rate dropped by 76%. Putting this in non-mathematical terms, about three-quarters of Walmart’s buying customers went somewhere else when the page load time increased by a mere two seconds.

In 2006, when reponding to user requests to increase the number of hits per page, Google experienced a 20% drop in traffic and ad revenue. It turned out that giving the users what they wanted wasn’t such a good thing. They increased the number of hits per page from 10 to 30, which increased the page load time from 0.4 to 0.9 seconds. One-fifth of their ad revenue was lost because of a half-second delay.

Google experimented with a small group of users by increasing their page load times artificially for seven weeks, then returning them to normal for a further five weeks. Compared to a control group, the study group’s use of Google searches dropped immediately and kept dropping for the entire seven weeks. When load time was restored to normal, the study group’s activity level immediately and substantially increased, and continued to increase for the entire five weeks. However, at the end of the five weeks, their usage levels still had not returned to what they were before the study began. [This study also showed that users reacted to pre-header delays more than post-header delays.]

Microsoft conducted similar research. They found that user satisfaction on Bing dropped by almost 1% when a half-second delay was introduced. A 2 second delay reduced revenue by 4.3%.

Google Maps increased their home page’s usage by reducing the number of bytes in the page by 25% and decreasing render time. Usage went up by 10% in the first week and 25% more in the following three weeks.

Amazon.com observed a 1% decrease in sales as a result of a 100 ms increase in page load time. An ongoing one-second delay would result in a $1.6 billion drop in sales.

Supreme School Supply increased its online revenue by 219.9% in 2010 by reducing its page download speed by two seconds.

The Gomez Peak Time Internet Usage Study conducted by Equation Research in February 2010 concluded that a single bad experience causes almost half of our end-users to form a less positive perception of the company behind the site. Worse yet, more than a third told others.

After conducting a 16-month performance improvement program, Shopzilla’s revenue’s increase by 5-12%.

Because website performance is so important to the end-user’s satisfaction, Google announced that page rankings will now be determined in part by page load time. In English, the slower your page, the farther it moves down the list.

And the list goes on…

No matter how you measure it or how you describe it, slow websites cause user dissatisfaction, which can lead to the death of the website.

So, if you’ve been thinking that your website’s performance is a trivial concern, perhaps it’s time to think again. Whether measured or not, the basic truth remains: Slow websites drive people away.

But You Can’t Measure True Performance

That leaves a question dangling, though: How can we accurately measure wait time and productive time from the user’s perspective? No system can know whether the user is waiting or productive because it can’t know what the user is thinking, saying, or doing. True performance, therefore, cannot be measured by automated tools.

Existing measurements provide useful information and some come closer than others to measuring true performance, but none measure true performance perfectly. By all means, developers should use the existing tools to analyze,benchmark, and monitor, but they should also do reality checks. The old-school approach, observing the user in action, is costly, but not as costly as being labelled too-slow-to-bother-with. Remember: The end-user’s wait time is what really matters.