Giganomics Consulting Inc.

The Said Business School at Oxford University, with the support of Cisco Networks announced the findings of its second annual report on Broadband Quality Score (BQS). This has some interesting results but the published findings lack in supporting details on sources and methodology.

Key inputs to each country’s BQS are the download and upload speeds achieved by internet users. The download speed accounts for 55% of the score, while the upload accounts for 23% and latency accounts for the rest.  Scores for 66 countries were compiled based on results from individual users who applied the Speedtest.net online speed test. The Said/Oxford report highlighted the fact that the results were based on 24 million tests between May and July 2009.

While this seems like a lot, it is only a fraction of the more than 440 million broadband subscribers worldwide, according to Point-Topic. [free with registration]  The Said/Oxford report doesn't indicate how many of the 24 million test results were repeat versus unique users, or how they were distributed across countries, time of day, distance to server test site, and other variables that could affect the test.  

The BQS results reported for a country will also differ depending on several aspects of the individuals participating in the Speedtest.net site during the sample period.  The report does not indicate what, if any, steps were taken to normalize the results for these differences.

According to the report, the results of the tests of download speeds among Canadian users indicated an average speed of just under 5 Mbps. When I ran the test (today and earlier in July), the test results indicated download speeds almost double that. The test results would be different for users who have different service levels, chose a different test server site, time of day, and so on.  If the speed test results used in the Said/Oxford analysis were taken mainly from users who were encountering congestion or dissatisfied with their speeds, then the results would tend to be biased downwards.

A more objective check on the results of Said/Oxford report can be found in another publicly available source of information on broadband service download published by Akamai Technologies Inc. Each quarter Akamai’s State of the Internet includes findings on the average download speeds achieved by users in dozens of countries.  The results are based on data that Akamai collects from its globally distributed network of servers with connections to more than 400 million unique IP addresses. 

Both sources put the same three countries – South Korea, Japan, Sweden - at or near the top.  However, the results for many other countries were quite different.  For example, Akamai’s report for the second quarter of 2009 found Hong Kong had an average achieved download speed of almost 7 Mbps but the report from the Said/Oxford report put this country’s speed at just under 5.  The Netherlands did well according to Said/Oxford at about 12 Mbps but according to Akamai the speed was only 5.1 Mbps. Denmark was similarly boosted according to the Said/Oxford results. These examples demonstrate the dramatic differences that can occur when service speeds are based on different methodologies.  In the case of the Said/Oxford report, the results are based on user-selected testing where the selection of participants was not random or adjusted to be representative of the universe.  Compare this to results from Akamai's data which encompassed almost 425 million of the 440 million unique IP addresses worldwide.

As a footnote to the above, Akamai’s report for the second quarter of 2009 provides some positive news for Canada in terms of higher speed broadband service.  The percentage of connections achieving speeds of 5 Mbps or greater has reached 27%, representing a 50% increase over the previous year and moving the country into 10^th place overall, as shown in the following table from that report. 

Akamai, State of the Internet, 2nd Quarter, 2009