Brief Recap of Part 1

Understanding Deep Packet Inspection

Understanding Broadband Services

Using DPI to Manage Internet Traffic

Outcome of the Net Neutrality Game — Take 2

Brief Recap of Part 1

In Playing the Net Neutrality Game, Part 1, I presented a discussion of net neutrality that focused on the common carrier aspect of the issue.  That is, proponents of net neutrality argue that the Internet, is an essential component of the nation’s communication system, and as such

Internet access providers should not discriminate with regard to what applications an individual can use, or the content an individual can upload, download, or interacted with over the network. Individuals acquiring services from Internet access providers should be able to use the applications and devices of their choice, and interact with the content of their choice anywhere on the Internet.

Opponents of net neutrality argue claim that since the Internet providers own the infrastructure that supports the Internet, they should be free to manage it as they see fit.

I then indicated that from an economic standpoint, the issue at the heart of the matter is the fact that the Internet “pipes” through which content moves have a limited capacity.  To the extent aggregate flows of content over the Internet (more than) fill the pipes, then how should the capacity be allocated among users? I argued that given the scarcity of capacity, an economically fair way of allocating capacity across users would be to switch from the current per-user based fee system, in which users are free to use as much capacity as they want for a set monthly price, to a fee-based access system, in which users are charged based on the amount of capacity they use.  I then examined the incentives faced by Internet Providers, content providers, and users to determine how Internet usage would differ under a usage-based fee system relative to that under the current user-based fee system.  I concluded that as compared with the current system, which employs per-user based Internet access fees, a system with usage-based Internet access fees would result in

• High intensity Internet users paying higher Internet usage fees and consuming less bandwidth overall
• Low intensity Internet users paying lower Internet usage fees, having more low Intensity users using the Internet, and having low intensity users consuming more bandwidth overall
• Content providers generating lower ad-based revenues
• Internet Providers generating greater revenues from the provision of Internet usage services.

Understanding Deep Packet Inspection

The analysis in Part 1 neglected to include the issue of deep packet inspection (DPI).  To understand the controversy surrounding DPI, one must first understand how information flows over the Internet.  When information (data) leaves its origin to flow to its destination over the Internet, the information is divided into a bunch of different chunks, or packets, which can be more efficiently routed over the Internet. The individual packets may take different routes through the Internet to get to the destination, and when all the packets have arrived, they are re-assembled back into the original file.  The packets can be reassembled because each packet contains all the information about the original files.  The file information is organized into 7 layers, with the most basic information contained in layer 1 (header information), and the actual contents of the file contained in layer 7.  The most primitive (unstructured) form of packet routing, which has been used in the US since the inception of the Internet, utilizes shallow packet inspection, that is, inspection of layers 1 and 2 to get the origin and destination IP addresses, plus a bit more information needed to get the packet to the right place.  Deep packet inspection, on the other hand, utilizes information from layers 3 through 7 to manage traffic flows based on other characteristics of the file, such as its type, application, and whatnot.  Forms of DPI are currently being used by private companies worldwide and by some Internet service providers in other countries to manage Internet traffic flows.

Proponents of net neutrality say that ISPs should only be able to use shallow packet inspection to move traffic across the Internet.  This leads to routing of packets based on a first-come, first-served basis, with no other interference of traffic flows.  They claim that deep packet inspection is a violation of privacy, and they claim that ISPs will use DPI to discriminate for (by giving priority flow status to) or against (by slowing down traffic for) specific files.

Opponents of net neutrality say that deep packet inspection enables ISP’s to manage traffic flows more effectively and efficiently.  From “Deep Packet Inspection Meets 'Net Neutrality, CALEA” by Nate Anderson, in effect, opponents of net neutrality say,

ISPs which "do nothing" to shape traffic on their networks have actually made a choice. In this case, the choice is in favor of chaos and bottlenecks at peak periods. No matter how much bandwidth is currently thrown at the problem, P2P, Usenet, FTP, and streaming video will fill it ... Handling this … data surge responsibly means using traffic shaping, at least during the periods of highest use.

On the other hand, Nate Anderson also notes that proponents of net neutrality argue that:

Recent research has shown that a nondiscriminatory network will in fact require up to twice the peak bandwidth of a tiered and shaped network, but this doesn't necessarily mean that this is the more expensive approach. Pundits like David Isenberg have argued that simple overprovisioning is cheaper in the long run than investing in all the new DPI gear and the manpower to maintain and monitor it.

In response, however, George Ou iterates that overprovisioning will not solve the problem, because Internet traffic will simply increase to fill any available capacity:

Japan is considered one of the most connected broadband nations on the planet with widespread 100 Mbps broadband service.  Many people in this country believe that by simply offering more capacity, there would be no need to manage the network since congestion problems would be gone.  But Japan teaches us that no matter how much capacity you throw at the problem, congestion will always be a problem and the vast majority of it will be caused by P2P traffic.

Understanding Broadband Services

One of the big issues in the net neutrality (NN) debate is whether or not Internet providers have market power.  From cybertelecom.com:

Proponents of NN argue that we currently have a broadband duopoly: DSL from incumbent telephone companies or cable from incumbent cable cos ...

Opponents of NN argue that competition is supplemented by Wifi at coffee houses, municipal broadband, broadband at work, and the promise of the deployment of WIMAX and other services...

If a broadband service has market power, the broadband network has both an incentive and the means to discriminate. A service provider with market power can extract monopoly rent from higher layer applications and content that depend upon that broadband network to access subscribers. In non economic-ese, if there is only one pipe to the home, the owner of the pipe can say that nothing goes over that pipe without the permission of and payment to pipe owner. If is a choice of broadband service providers ... then individuals can switch services when they felt the service was inadequate...

Both sides will argue that their position favors innovation, creativity, and development. Those against regulation argue that the government will thwart innovation. Those in favor of network neutrality regulation argue that the significant market power of monopoly or duopoly local broadband service providers like AT&T will thwart innovation.

To better understand the current US market for broadband, I went to the FCC’s website to find out what constitutes broadband services.   According to the FCC, "the term broadband commonly refers to high-speed Internet access". The FCC defines broadband service as "data transmission speeds exceeding 200 kilobits per second (Kbps), or 200,000 bits per second, in at least one direction…"

The FCC further describes how is broadband is different from dial-up service:

•  Broadband service provides higher speed of data transmission—Allows more content to be carried through the transmission “pipeline.”

•  Broadband provides access to the highest quality Internet services—streaming media, VoIP (Internet phone), gaming, and interactive services. Many of these current and newly developing services require the transfer of large amounts of data which may not be technically feasible with dial-up service. Therefore, broadband service may be increasingly necessary to access the full range of services and opportunities that the Internet can offer.

•  Broadband is always on—Does not block phone lines and no need to reconnect to network after logging off.

•  Less delay in transmission of content when using broadband.

Finally, the FCC provides a description of several high-speed transmission technologies that constitute broadband services:

• Digital Subscriber Line (DSL): DSL is a wireline transmission technology that transmits data faster over traditional copper telephone lines already installed to homes and businesses. DSL-based broadband provides transmission speeds ranging from several hundred Kbps to millions of bits per second (Mbps).
• Cable Modem:  Cable modem service enables cable operators to provide broadband using the same coaxial cables that deliver pictures and sound to your TV set.  Speeds are comparable to DSL.
• Fiber Optic (FiOS): Fiber optic technology converts electrical signals carrying data to light and sends the light through transparent glass fibers about the diameter of a human hair. Fiber transmits data at speeds far exceeding current DSL or cable modem speeds, typically by tens or even hundreds of Mbps.
• Wireless: Wireless broadband connects a home or business to the Internet using a radio link between the customer’s location and the service provider’s facility. Wireless broadband can be mobile or fixed.  Wireless technologies using longer range directional equipment provide broadband service in remote or sparsely populated areas where DSL or cable modem service would be costly to provide. Speeds are generally comparable to DSL and cable modem. An external antenna is usually required.
• Satellite: Satellite broadband is another form of wireless broadband, also useful for serving remote or sparsely populated areas.
• Broadband over Powerlines (BPL): BPL is the delivery of broadband over the existing low and medium voltage electric power distribution network. BPL speeds are comparable to DSL and cable modem speeds.

Next, as a benchmark of comparison for the broadband prices and services offered in the US, I compared US offerings with those in other countries.  If US broadband providers have market power, then prices of broadband services in the US should be higher and Internet transmission speeds should be slower than their counterparts in other countries. 

The following table compares broadband penetration rates, prices, and speeds for many different countries.  The countries in the table are listed in order of increasing broadband penetration rates:

At the bottom of the table, I present the minimum, maximum, mean (average), and median (middle) values for each of the statistics across the different countries.  The mean value can be affected by extreme values (outliers), while the median value is robust to outliers.  Given the large cross-country differences in the various statistics, I think a comparison of US values with the median, rather than the mean, is most appropriate.  Comparing US values with the median value across other countries, the table shows that

• The broadband penetration rate in the US market falls just in the middle of penetration rates for other countries;
• The US population is the highest among all counties included in the analysis;
• The US spans more area than any other country except Canada;
• The US is much less densely populated than other countries;
• The US has a greater per-capita income than most other countries;
• The median monthly subscription price for broadband services in the US falls just in the middle of those for other countries; and
• The median speed of broadband services offered in the US falls just in the middle of those for other countries.

It is more costly to provide communication services in places that (1) cover more area, (2) cover more users, and (3) are less densely populated, because you need to install not only more infrastructure, but also more infrastructure per person, especially that for the “last mile”, for which the provision of infrastructure is particularly expensive.  So given that the US covers more area, holds more people, AND is less densely populated than most other countries, it is significant that (1) broadband penetration rates and speeds are as high as they are and at the same time (2) prices are as low as they are.  This certainly does not prove that broadband service providers in the US do not have market power, but it is suggestive.  And if they don’t have market power then broadband service providers are less likely to abuse this power by using deep packet inspection to discriminate against particular flows of traffic over the Internet.

Since I had the data in hand, I couldn’t resist seeing how the various statistics in the table above correlate with one another.  In other words, I asked myself, of the various statistics in the table above, which are determinants of broadband penetration rates?  To find the answer, I performed a log linear regression of the population ([C]), population density ([E]), per-capita income ([F]), median monthly subscription price ([G]), and median broadband speed ([H]) on broadband penetration rates ([B]).  The analysis indicated that 77% of the variation in broadband penetration rates across the countries in the study can be explained by these five statistics.  Of the five variables, three (per-capita income, median monthly subscription price, and median broadband speed) were statistically significant determinants of broadband penetration rates.  More specifically, a 10% increase in per-capita income leads to a 8.7% increase in broadband penetration, a 10% increase in monthly subscription prices leads to a 2.9% decrease in broadband penetration, and a 10% increase in broadband speed leads to a 1.5% increase in broadband penetration.  This analysis is wholly consistent with the analysis of the Network Neutrality Game I discussed in Part 1, in which I assumed that Internet users care about the price and speed of Internet services.

Using DPI to Manage Internet Traffic

I now understand the nature of broadband, traffic flows, and deep packet inspection.   The next issue to address in the net neutrality game is how broadband services would change from the current per-user pricing scheme (unlimited Internet usage for a fixed monthly fee) to a pricing scheme that utilizes DPI to better manage Internet traffic.

Plusnet, a broadband Internet service provider in England, “voted the Best Value Home Broadband provider in the 2009 Top 10 Broadband awards”, uses a traffic management system to manage its users' traffic.  Plusnet describes its system as follows:

All traffic on our network is prioritised by type, with time-sensitive applications such as gaming and VoIP having the highest priority.

Higher priority traffic gets bandwidth before lower priority traffic. As the total bandwidth available is shared, this mean lower priority traffic can be slower at busy times when there's less bandwidth available.

If this seems confusing, think of it this way: the broadband network is like a motorway. When the traffic is light, all vehicles can move at the national speed-limit. Some lanes of the motorway have been reserved for important traffic, such as buses or emergency vehicles. During rush hour, most vehicles are forced to slow down. However, the traffic on the reserved lanes can continue to travel at high speed.

Without traffic management, P2P and download traffic would flood the network and cause slower speeds for other types of traffic.

Plusnet prioritizes its traffic by Internet service plan (price) and file type as follows (the color coding is mine):

Where

• VoIP (Voice over Internet Protocol): Allows you to make phone calls over your Broadband connection.
• VPN (Virtual Private Network):  A secure, encrypted connection between two points over the Internet. Often used by Home workers to access work systems.
• File Transfer Protocol (FTP):  A protocol used to transfer data from one computer to another via a network, commonly used to upload web pages.
• P2P (Peer to Peer):  The sharing of files by computers that are connected to a network. Although p2p has many legal uses it's most commonly associated with illegal file sharing.

(The monthly usage caps for the Plusnet Value, Extra, and Pro plans don’t make sense to me – I would think the Pro plan would have at least as much usage as the Extra plan.  I checked the numbers a couple of times, and it looks like those reported in the table are accurate.  Perhaps the Extra plan is tailored for intense users of gaming and P2P; alternatively, the usage cap might be a response to a competitor’s offering.)

Now that I see Plusnet’s usage plans, it seems reasonable to expect that if Internet providers in the US switch from per-user based pricing plans to per-usage based plans, they would implement similar types of usage plans, that is, plans in which pricing varies depending on

• Services used (first column of table),
• Priority of traffic, (color coding in table) and
• Usage limits (last four rows of table).

Outcome of the Net Neutrality Game — Take 2

Finally, now that I have some idea about what Internet services pricing will look like under a usage-based plan, I can consider how the outcome of the Net Neutrality Game (see Overview of the Net Neutrality Game for a complete description of the Net Neutrality Game) would change if broadband services providers switched from per-user based pricing to per-usage based pricing.

Lo and behold!  The structure of the Net Neutrality Game and incentives faced by each set of players -- Internet service providers, content providers, and users – for the usage-based pricing system described in Part 1 is basically the same as that which would hold for a system in which Internet usage packages are based on services used, priority of traffic, and usage limits.  See Outcome of the Game: Per-User Internet Fees vs. Per-Usage Internet Fees for a complete description.  To summarize the general outcome,

• Users who were high intensity users under a per-user based fee system will either decrease usage or pay a higher price to maintain previous levels of usage under a usage-based fee system.
• Users who were low intensity users under a per-user based fee system will pay a lower price to maintain previous levels of usage under a usage-based fee system.
• Users who were priced out of the market under a per-user based fee system will now sign up for the lower-priced usage-based plans.  Their usage will thus go from zero under a per-user based fee system to low usage under a per-usage based fee system.
• Because the users who were high intensity users under a per-person based fee system will experience a drop in usage under a fee based system, content providers will end up generating less revenue from ad-based sources.
• The traffic management system used under the usage-based fee system will enable Internet services providers to use their bandwidth more efficiently; that is, they will need less bandwidth to manage a given level of traffic.  The higher returns generated on investments in infrastructure will probably lead Internet service providers to increase their investments in infrastructure.

I also have to consider how changes in the relative priorities of flows of different types of files under a DPI management system will affect supply of and demand for the different types of content.  For simplicity’s sake, I assume that the relative priorities of all traffic flows will conform to those listed in the Plusnet table above.  That is, file priority will change from equal priority of all files without DPI to a hierarchy of file priorities with DPI:

No DPI File Priorities:

Browsing & Email = VoIP = Gaming = Streaming = VPN = FTP = Download = P2P

DPI File Priorities:

VoIP, Gaming

Browsing & Email, Streaming, VPN

FTP, Download

P2P

With priority status given to VoIP and gaming under a DPI management program, the quality of these two types of services will improve.  This should attract more suppliers and users of these two types of services.  At the other extreme, if P2P traffic is given lowest priority status, the resulting decrease in the speed of flows of P2P activity should decrease both its supply and demand.

Furthermore, the preference given to traffic within a particular Internet provider’s system (“external” traffic vs. “internal” traffic) will encourage the collocation of more interdependent sites/servers/activity within the same broadband provider’s realm.  This is just like the healthcare system that has competing networks of providers – you choose the healthcare network that most of your doctors are members of.

Returning to Internet services, given the impact of network effects, if the inter-system differences in speed of traffic flows becomes pronounced, then this will encourage the formation of a couple of larger dominant systems (i.e., a few, dominant broadband service providers), rather than many smaller systems (i.e., many small broadband service providers).  Using the healthcare system analogy, the more difficult or costly it is to see a doctor outside your own network, the greater the incentive you have to stay within your own (preferred) network.  Also, when you have more doctors in one network, that will attract more patients to that network, which in turn, attracts more doctors to that network, and so on.  So then a healthcare system will start out by making it less costly to go outside the system.  This will attract initial users, who can see doctors in the system and outside the system at about the same price.  Once a system becomes large enough, though, the network provider will want to increase the cost to users of going outside the system.  This will lock current users into the system (which will enable the provider to charge users higher prices without fear of losing the patients, because all their doctors are in that network).  Higher profits from current patients in the system will outweigh the loss in profits from patients who are “locked-out”.  Since larger networks (networks with more doctors) are more attractive to patients than smaller networks, the healthcare industry will be led to consolidate into a couple of large, competing systems, rather than to end up with a bunch of smaller systems.  In the same way, Internet service providers will be led to de-prioritize cross-network flows of traffic only by small amounts until enough users have committed to the network.  Once the network reaches critical mass, the Internet service providers will be led to slow cross-network traffic considerably more to encourage network users to stay within the system.

As already discussed in Part 1, without the use of DPI and/or the ability to charge users by their type and intensity of Internet usage, Internet service providers cannot optimize the profits they can generate from the provision of Internet infrastructure.  This means they will not invest as much money in making sure their bandwidth grows with the pace of Internet traffic as they would invest when they are able to use DPI and/or charge users by their type and intensity of Internet usage.

So what we are left with is this.  Without the use of DPI and/or the ability to charge users by their type and intensity of Internet usage, Internet service providers have less incentive to increase capacity (bandwidth), so traffic becomes congested, especially during peak hours; Internet services are more expensive for the average user, and high intensity users and P2P traffic crowd out low intensity users and VoIP and gaming; and content providers generate large amounts of ad based revenues. With the use of DPI and/or the ability to charge users by their type and intensity of Internet usage, Internet service providers have more incentive to increase capacity, so traffic flows smoothly for services with higher priority status and for traffic within each separate provider’s network; Internet services are less expensive for low intensity/priority users, but more expensive for high intensity/priority users; the quantity of P2P traffic decreases, but the quantity and/or quality of VoIP and gaming traffic increases; and content providers generate less ad-based revenues.