Giganomics Consulting Inc.

Studies providing international comparisons of internet services are having a veritable baby boom.  The Berkman Center for Internet & Society at Harvard University released a draft of the report Next Generation Connectivity, A review of broadband Internet transitions and policy from around the world.

There is much to be learned from the growing body of international comparisons, including the Berkman Center Broadband Report.  Unfortunately, for Canada, many will put too much emphasis on a few select rankings from the Report without taking the time to consider how these were derived. 

As noted in the previous post to this website, the report Lagging or Leading, The state of Canada’s broadband infrastructure, covered similar territory and looked at many of the same sources for data.  The Berkman Center’s rankings were based on penetration (service adoption), speeds and prices – all factors considered in Lagging or Leading. And yet the Berkman Center Broadband Report found Canada ranked 22^nd out of 30 – towards the bottom of the pack.

The Berkman Center relied extensively on OECD data for a number of its comparisons.  It used OECD data on broadband penetration per 100 population as a key input.  However, as noted in Lagging or Leading, this measure is biased against countries such as Canada.  The Berkman Center Broadband Report also included penetration per household but relied on older OECD data where Canada ranked 8^th even though more recent international comparisons where Canada ranked as high as 5^th (see Lagging or Leading, Table 4.2.2).

The Berkman Center incorporated into its ranking data on the penetration of 3G mobile services.  Canada was reported to have a very low level of 3G subscribers per 100 population – less than 10%.  There are two concerns with this data point.  First, Canada and the United States do not compare well on measures of mobile penetration per 100 population because subscribers in these countries generally do not have multiple accounts (or SIMs).  The Berkman Center Broadband Report acknowledged this difference yet did not make any adjustments.  Second, Canada’s very low ranking on 3G penetration is not consistent with the fact that the service is available to 91% of the population, as noted in the CRTC Communications Monitoring Report.  Since the Berkman Center’s ranking assigns a 30% weight to this indicator, Canada’s overall ranking on penetration is pulled down.

The Berkman Center next turned to measures of broadband service speeds where OECD data once again played a significant role, accounting for one-half of the country rankings for this indicator.  As noted previously on this website, the OECD rankings on speeds are not reliable.  In the case of Canada, the OECD relied on only 16 observations and, among those, significantly under represented higher service levels and over represented lower service levels.  The OECD relied on many more observations for most other countries, with some countries' lists including duplicate services for these higher speeds.  

The Berkman Center made some effort to address the multiple and duplicate offerings for its analysis of service prices among the 30 countries.  It supplemented the OECD observations with its own analysis of advertised service prices.  Some of these included higher speed services available in Canada, as noted in Figure 4.15 of its Broadband Report.  Yet, none of these observations were included for the speed comparisons.

The other components of the Berkman Center Broadband Report rankings on speed were drawn from results of user-generated tests of achieved throughput speeds and latency conducted using Speedtest.net.  It was acknowledged that this data has “several confounding factors that [require] we interpret the data with caution,” and the results on latency “produced very counterintuitive results”.  Yet, the resulting rankings from this data accounted for the other half of the countries’ rankings on speed.  Concerns respecting the use of data from Speedtest.net were discussed previously on this website.

Prices composed the third aspect of the Berkman Center Broadband Report on international comparisons.  The OECD data on prices figured prominently here as well, accounting for one-half of the overall price ranking.  As noted, the Berkman Center took additional measures to update and expand the OECD pricing data to derive new rankings based on the price across different levels of service.  In each case, Canada’s ranking improved by four to six places. 

While the steps taken to improve the pricing data were helpful, it is curious why the Berkman Center decided to create its overall ranking on price based on both the OECD price observations and a combination of the OECD and its own updated and expanded data.  The latter would, for the most part, encompass the former observations so there is no reason to use both.  In the case of Canada, using both sets of price rankings pulls Canada down in the rankings. 

The other significant factor affecting Canada is the exclusion of very high speed services (download speeds of at least 35 Mbps), notwithstanding the fact that Figure 4.15 of the Berkman Center Broadband Report clearly indicates that it had price data from Canada for such services.  Because this data was ignored, Canada was assigned a nominal ranking of 30 out of 30 for this category.  And because the OECD also failed to include any such services from Canada, this very poor ranking was double-counted. 

The Berkman Center Broadband Report does not do justice to Canada’s broadband performance because of these missteps.  Compounding this is the Report’s heavy reliance on OECD data that have serious methodological problems and biases that weigh against Canada.  Since the Report is only in draft form and is being review by the Federal Communications Commission, perhaps some of these concerns will be addressed.