IPv6 deployment has come a long way in the last year or two.  Periodically, I make a little informal survey of the state of deployment, and I wanted to share these results.  In blog entries to come I will do the same for Broadband providers and other organizations of interest.These providers were taken from the “Wikipedia Tier-1 Network” (for IPv4) entry, which I am using as a proxy for “big backbone providers.”  I have added “Hurricane Electric” because of their long and deep involvement in IPv6 deployment, and Command Information – not a provider really – where I work and which also specializes in “all things IPv6.”

Figure 1 - Backbone Provider IPv6 Status

What’s interesting here?  A number of things, I think, including:

• Of these 13 providers, all have IPv6 prefixes assigned.
  • A low bar – it is not difficult – nor does it require much investment – to obtain a /32 prefix.
• Of the 13, 12 are announcing their IPv6 prefix.
  • This is a higher bar – it suggests a complement of IPv6-savvy engineering talent at the provider, although the provider could have a “thin” internal IPv6 deployment.
• In terms of “Upstream” peer ASNs, most (all but Savvis) have good diversity (3 or more) (Hurricane has 10).
  • A yet higher bar –providers peering with other providers is another indication of the depth of their IPv6 deployment and a preparatory step for carrying production IPv6 traffic.
• In terms of “Downstream” peer ASNs, some have a significant number (35 or more).
  • These may be paying customers or other cooperative organizations.
  • Note Hurricane Electric with 476 downstreams – this is partly a result of their liberal peering policy for IPv6 and their tunnel-based peering infrastructure.  NTT, for example, or Global Crossing are likely to have a higher proportion of “high quality” (paying) customers downstream within their count.  Hurricane has many paying customers, but also many tunnel-based small peers.
  • NTT posting impressive numbers as well - another provider with a long-standing commitment to IPv6.
• Of the 13 providers, 6 have enabled IPv6 transit on their DNS servers.
  • This is significant in several ways – but mostly I use it as a proxy for the *depth* of IPv6 capability and progress within a provider. This means that IPv6 has moved beyond the core engineering team and into other parts of the organization and other competencies.
• Of the 13, 2 have IPv6-enabled main websites (NTT has IPv6-enabled “www.ntt.net”).
  • No surprise in not being a “Yes” here.  Few large organizations have enabled their “main services” for IPv6.  Google, for example, runs “ipv6.google.com,” rather than having dual-stacked www.google.com (for the masses – Google does have their DNS whitelist server and that is a terrific offering), and no Internet-based user is likely to be running IPv6-only today (especially with no provision for IPv6/IPv4 translation).  Still, my hat is off to providers who have pushed IPv6 so deep into the organization that the main online business site is IPv6-ready.

So, big picture, I think this confirms what we have an innate sense of – big IP backbone providers are actively building out their networks, enabling their support services, and moving IPv6 outside the main engineering groups and deeper into the organization. This is also the case with many broadband ISPs (as we will see next time).  Soon after broadband ISP deployment will come the major content and media providers (Google, Yahoo, YouTube, Bing, etc.) and then enterprises – especially enterprise Internet-facing sites.  Adoption will be uneven – that is leading enterprises will deploy before some lagging providers – but these deployment waves are already shaping up.  Where are you in your deployment?  I would enjoy hearing from you on your experiences with IPv6 deployment in your organization - please comment.  Thanks for reading.

Commercial service providers are all - or mostly all - considering their IPv6 implementation strategies.  Not what kind of advanced peer-to-peer services they can launch, or how they can revolutionize their product and service offerings (that will come later), but the “straightforward” provisioning of IPv6 within their networks and out to the customers.  Simple high-speed Internet, either for organizations or individuals (broadband providers - cable, DSL, and fiber).  Enterprise-focused providers are also looking at their L3 VPN offerings and preparing for IPv6-capable services.  Multicast, and mobility, certainly, offer attractive capabilities for launching next-generation services. 

The providers I have talked to, and worked with, are at different points along the implementation path.  Some are in the design phase, others are conducting pilots on secondary networks, and some have big deployments. 

If there is one overriding realization that all these providers have come to it is that IPv6 implementation is a big project, and a lengthy process - especially the “getting off the ground” phase. 

1) Resolving to launch the project, when there are so many other important projects competing for resources (DSG, IPTV) 

2) Evaluating the hardware and software platforms for IPv6 capability, and putting together a coherent upgrade strategy that does not conflict with upgrades in progress or planned to support other projects (L3 VPN, Voice Services, DOCSIS 3.0, etc.) 

3) Capital budgeting 

4) Breaking down the project into phases, and rationalizing all that into a timeline that provides the right capabilities at the right time 

5) Doing the design work - again within a changing environment - addressing, routing, management, security, peering 

6) Doing the implementation, in a no-downtime environment - again within a changing environment 

7) Provisioning, back-end systems - there are a lot of applications, systems, and process to upgrade as well 

Like any long journey, getting started is the hardest step.  It requires some foresight (likely IPv4 exhaustion - mid-2012), and there are many pressing projects for ISPs - and it is not a great economic climate.  But, the truth is, 2-3 years is not a long time to get as much done as must be done, and what happens in the “late innings” of IPv4 runout is unclear.  Hoping for more time if things get tight is not a strategy. 

So - my advice - start now.  Start slow - but start.  A few people, a modest budget, and a lot of the ground can be prepared for a successful project when it becomes clear a more serious, concerted “push” is required.

I’ve updated my brief on the efforts of commercial service providers (SPs) to launch their efforts to integrate IPv6 into their networks.  Last posting, we looked at a list of 23 large, well-connected SPs, and found that all 23 have obtained IPv6 prefixes.  That is an early step on IPv6 deployment, but requires little real effort on the part of the service provider - it just requires a few forward-looking engineers to go and obtain the prefix. 

Now we start to look at deployment indicators that take more effort.  In this paper, we follow-up on those 23 providers, and see how many are announcing their prefixes onto the IPv6 backbone.  We find that 15 of the 23 providers have taken another step - annoucing their prefix to the world.

One last reminder.  Providers that are announcing their prefix may be doing so from a small lab, with no production users (announce a /32, have a single /64 with one router and a PC on it).  Others may have large deployments, and many production users.  A provider that is not announcing their prefix may have a large, internal deployment - perhaps preparing for production services - and have chosen to “cloak” their efforts by announcing their prefix as a last stage in their deployments.  This is a provider than wants to keep the timeline between when it is apparent they have an advanced IPv6 capability quiet until they are close to customer availability.

Morale:  It is difficult to tell, looking at the “provider shell”, how advanced (or modest) their IPv6 capability is - we can only infer and make guesses.

Read the “Service Provider IPv6 Adoption - Part 2 - Prefix Announcement” brief.

I’ve written a brief on the efforts of commercial service providers (SPs) to launch their efforts to integrate IPv6 into their networks.  Looking at a list of 23 large, well-connected SPs, all 23 have taken the first visible and tangible step towards IPv6 implementation and obtained prefix allocations from their regional registries.  This is admittedly only a small first step, but it does signal intent on the part of all these providers to deploy IPv6 sooner rather than later.  Of these 23, some SPs have well-developed IPv6 service offerings today, and others are much earlier in their deployment.  It is significant, though, that SPs are unanimous in their view that IPv6 deployment is needed soon in order to be competitive in their service offerings.

Read the “Service Provider IPv6 Adoption” brief.

In the last discussion, we talked about service providers using a single IPv4 address shared between subscribers.  Carriers may well do this not as an alternative to IPv6 deployment – which pretty much everyone (finally) agrees is the “real” solution – but as a stopgap measure to allow them more time to complete their IPv6 deployments.  Many providers have done a poor job of preparing their businesses for the “post ready availability of IPv4 routable addresses world”.  Additionally, and as an aside, not all in-home platforms are IPv6 ready and on-by-default (Windows XP, for example, have IPv6 but it is not on by default.  So, there are a number of reasons why providers need to be able to provision IPv4 to existing and new subscribers for another few years.

In the “shared IPv4 address” schemes, rather than have “one IPv4 address per subscriber” there would be a new practice – “one IPv4 address per multiple subscribers”.  There are several implementation schemes under discussion for how the provider would actually do this.  One schemes call for “double NAT”, where the subscriber (with a NAT router at the edge) uses RFC1918 internally, does NAT at the edge, and then the provider does NAT *again*.  Another scheme simply moves the NAT function from the subscriber edge router into the provider cloud, leaving it still “single NAT”, but no longer in the subscriber device.  More about these another time.

The issue left on the table last time was “what do we do about inbound services”?  Suppose a subscriber wants to run a webserver, for example.   Suppose another subscriber sharing that routable IPv4 address also wants to run a webserver?  Clearly, both cannot receive traffic inbound destined for “IPv4_Addr:80” – the standard HTTP TCP service port.  What do we do in this situation?

I think the answer is the simple one – “don’t do that”.

The most likely implementation will be for the “standard” subscriber service offered to home users (“consumer class”) to indeed be a “shared IPv4 address” scheme.   These addresses are probably most often, for almost all carriers, assigned via DHCPv4 anyway, and are dynamic.  Someone using the standard service would not be expected to run services requiring specific inbound ports.  These subscribers would only be expected to run “NAT-friendly” applications – as most subscribers do today.

For subscribers hosting services, or using more advanced applications, however, the provider will offer a “premium service” – a dedicated IPv4 address.  That will be the solution.  A “normal” subscriber gets a shared IPv4 address.  A “premium” subscriber pays a little extra, gets a dedicated IPv4 address (and probably a static IPv4 address, which some carriers offer today as a premium service), and is not subject to NAT within the carrier cloud.  If the subscriber does NAT locally, within their home edge device, that is up to them. 

In some ways, this is a simple solution.  It only appears simple from the subscriber view.  They pay a little extra.  For the provider – not simple.  Remember that solutions only work for providers that are scalable.  Think about the impact on the provisioning and billing systems or the provider.  Think about the network impacts of implementing the “shared IPv4 address” scheme.  Lots of challenges, lots of work to do, and – look around – in a pretty challenging business environment.

I always say it is not easy being a provider.  This is another example.  But what choice do the providers have?  They need to keep signing up new customers, and not all subscribers are ready for IPv6 today (nor will they be in 2009, when the IPv4 address shortage really will begin to constrain their business).

Ahead, we’ll talk about applications that will not work (or as well) in a double-NAT environment, and also about some of the more popular “shared address” schemes.

OK.  Last time we talked about the fact that ISPs are probably going to deploy some stop-gap IPv4-based solution while they implement IPv6 over the next couple years.  This plan also buys a little more time for more Windows95 or other non-IPv6 capable hosts to be retired. 

Important:  the real solution is integration of IPv6 into all networks and devices, and a long-term transition from IPv4 to IPv6 entirely.  In the near-term, though, it is becoming more apparent that ISPs are going to chose to use a few strategies - implement IPv6 but also find a way to get more out of IPv4 in the near-term.  A smooth transition from IPv4 to IPv6 in advance of IPv4 address run-down would have been better - cheaper and more reliable - for the entire Internet community, but looking at the situation today the community has missed that window of opportunity.  So, instead of “elegant integration and transition”, we’re going to be stuck with “stop-gap + elegant integration and transition”.

The initial discussions to extend the life of IPv4 largely involve sharing a single IPv4 address across multiple subscribers.  There are a couple of interesting things for us to keep in mind about this class of solutions.

Quick Application – TCP/UDP – Port Number Review

Applications run today mostly over TCP or UDP, and use port numbers.  There is a source port number and a destination port number.  Most applications we think of as common use TCP.  As an example, a web browser connection is based on TCP, with the destination port number being 80 – one of the well-known port numbers.  These are port numbers between 0 and 1024. The source port number in the web browser example would be a “high port” – in the range 1025 – 65,536 (64K).

Just a moment ago, I connected, using IE7 on Windows Vista, to a handful of websites in quick succession.  Using a sniffer package, I see the following TCP source ports in use during this few second period:  7647, 7659, 7660, 7662, 7663, 7666, 7664, 7667, 7668, 7670, 7671, 7672, 7673, 7674 – and maybe anther handful.  Call it about 20 TCP source ports.  So, for this one application, for 10 seconds, I used 20 of the available TCP source ports on my laptop.

When my computer makes these connections to the Internet, and I am at home behind my NAT firewall router, my laptop’s local TCP source port will be “mapped” to some source port on the outside of my NAT firewall.  Maybe the same port number, maybe a different port number.  Say that, in the example, my laptop address inside my house is 10.3.4.5, and my web browser opens a connection on TCP source port 7647.  This might be mapped to the TCP source port 42355, 6.7.8.9 on the outside of the NAT box – where 6.7.8.9 is the IPv4 address on my NAT firewall’s WAN-facing (Internet-facing) interface, and 42355 was an available port.

It is important to note that the NAT box is able to recycle TCP and UDP port numbers pretty quickly.  But it depends on the implementation just how quickly.  I would think that if a port was unused for a few minutes it would be returned to the “free” list in most cases.

The Way it Works Today

In the case where my house – one subscriber – has their own NAT box and one routable IPv4 address on the WAN-facing interface, all the internal machines have to share those available 64,512 (64K – 1024) TCP port numbers.  For me, that’s a few PCs and a TiVo machine.  I’m not quite sure how many TCP ports my TiVo box uses, but I’ll guess it is not more than, say, 20.

So, then, the high-water mark for my household might be around 100 ports in use at a time.  For today’s applications that is probably more than enough for the average household.  So, sounds like I have 64,512 ports available to me for outbound sessions, and I only need 100.  Lots of headroom.

How it Works if One IPv4 Address Shared Across Multiple Households

In this case, then those multiple households share the available TCP port numbers.  If we say a household needs 100 ports, then we could share 64,512 ports across about 645 households.

So Then This Will Work Fine No Issues – Right?

Well, looks pretty good.  We want it to work all the time though, and we want it to work for awhile – a few years.  A lot can happen in a few years.  One trend of late is applications that open lots of TCP connections – simultaneously using lots of TCP source ports.  My understanding is that Google Maps, for example, opens a lot of TCP sessions and downloads maps in sections – to shorten the amount of time it takes to render the map.  I just tried it, and I was able to fire off 64 TCP sessions in just a couple quick clicks.  Maybe TiVo uses this to improve download speeds – wonder how many ports that might add.  Now I’m less sure about my guess that TiVo probably doesn’t use more than 20 ports.

So, depending on how far this trend towards applications using multiple simultaneous flows goes, and how fast, perhaps we should plan on making 1000 ports available on the shared NAT box per household.  That would mean only 64 households sharing a single IPv4 address.

Any Other Issues?

Yep.  At least two.  Topics for another post.  Just to set the stage, think about:

1. What do we do about inbound sessions?  Suppose a household wants to run a webserver, on TCP port 80, and setup port forwarding to forward packets arriving on the routable address on the WAN side of the NAT box on port 80 to their internal HTTP server.  If multiple households share a single routable address, who gets to use that specific well-known port?
2. What do we do about any application that uses port forwarding?  Bit Torrent, for example, wants to use certain inbound ports (TCP 6881 – 6999).  Who gets to use those?  And, for that matter, how do I setup port forwarding when the NAT is no longer in my closet, but it is a shared resource run by my service provider?

Good questions.  We’ll talk about them next time.

Service Providers (SPs) have figured out they have a new problem.  I think it is hard being an SP anyway – tight margins and intense competition.  Now it is becoming clear to the SPs that the supply of “fresh” routable IPv4 addresses is drawing down.  And that will make new addresses harder to get, or more expensive to get, and that has serious repercussions.  That hits SPs where they live – subscriber growth. 

SPs were not aggressive adopters of IPv6 – with some exceptions.  It seems, however, that SPs are now coming quickly around to their need to deal with this problem.

A few assertions:

1. IPv4 addresses will become scarce in advance of all home users having production-quality IPv6 capability on their devices (read PCs and gateways)
2. SPs are unlikely to accelerate their IPv6 deployments enough to provision production-quality IPv6 services to every subscriber in advance of IPv4 run-out (or, increasing scarcity)
3. IPv6 is still the right solution to IPv4 address exhaustion, but the SP community (at large) needs a 2 or 3 year stop-gap measure to buy themselves time to implement, and ensure they can continue to add subscribers

There are a few ways SPs can get through this period, and many of them were discussed at a recent IETF “interim” meeting in Montreal Canada.  There are a number of possible solutions, and it is too soon to pick winners, but a few things to keep in mind are:

1. Most of the solutions involve sharing routable IPv4 addresses between multiple subscribers.  Note this is a significant change from the current most common deployment, where each subscriber (or household) gets a single routable IPv4 address for the outside of their home gateway (router, firewall, router/wireless access point – whatever)
2. This “sharing an address between multiple households” solution means that the provider will have to deploy a NAT device within their network – often called “Carrier Grade NAT” (CGN)
3. If the subscriber continues to use their existing home gateway, then all their IPv4 traffic will be NAT’ed twice (“double NAT”) – once by the subscriber NAT and once by the provider NAT, which breaks some classes of applications
4. If the subscriber replaces their home gateway (or the provider gives them a new one), then the traffic will not be NAT’ed twice, but it will still be NAT’ed at the provider gateway – which *also* breaks some classes of applications

As I said, there are a number of mechanisms to deal with this problem in the near term.  In the middle and longer term, IPv6 is the solution - all the SPs I talk to realize that and are (finally) making plans in earnest to get it done.  Some of the mechanisms under discussion advance IPv6 deployment within provider networks (in other words providers solve their immediate problem of IPv4 address space run-down and *also* – as a bonus – move their networks towards IPv6 capability).  Other mechanisms are simple stop-gaps – just keep IPv4 running (pretty well, for some classes of applications, for basic usage) just a little longer.

I’ll tell you more about those mechanisms in the weeks ahead.  I’ll be at NANOG, and I’m sure this will be a hot topic.

A phrase you hear a lot in the IPv6 community is “Transition”.  There are transition strategies, transition plans, and transition tools.

In reality, though, *integration* is a better way for us to look to the near future.  Just the word “Transition” is misleading.  I think we’re mostly on the same page in the technical community – we aren’t moving from IPv4 to IPv6 – we’re adding IPv6 to the existing networks to support innovation and to pave the way for the coming wave of – well – everything.  More people.  More devices.  More mobility.  New services.  Higher expectations.  Machines talking to machines.  Exploding bandwidth.  Continued convergence of everything that’s not on IP – onto IP.

But we’re unnerving the guys with the real wood furniture.  “Transition” sounds reckless.  Sounds like we’ve got a new toy and we’re anxious to try it out in the production environment because we can.  Sounds like we’re putting the current IPv4 platform at risk.  “Integrating” a technology upgrade onto the existing network – supporting innovation – building out a more scalable platform – that sounds like the responsible thing for IT to be working on.  That sounds like an obvious thing to do – just part of the natural evolution of the network, just like always.

And it has the benefit of being true.  IPv6 is a platform for innovation in products and services, just like IPv4 was a platform for innovation in the last 15 years or so.  And innovation is the lifeblood of the kind of companies that excel and companies people want to be involved with.  Our ability to innovate on an IPv4 platform is waning, as we become more constrained by its limitations.  Our ability to innovate on IPv6 is expanding, as more IPv6 networks are deployed, more IPv6-ready devices are fielded, and the development community becomes more familiar with the peer-to-peer model IPv6 offers.

So, let’s all try to be clearer about the task at hand.  IPv6 today is not about replacing IPv4.  It is about methodically adding IPv6 to our existing networks, supporting innovation in new service models and applications, and moving the Internet towards becoming an always-on, high-performance, practically everything-connected, fabric-of-the-planet utility.

There is some discussion, and perhaps confusion, about the role DHCPv6 and autoconfiguration play for IPv6 networks.  I’d like to take a second to provide a quick level-set. 

In IPv4 networks, there are two ways to assign addresses – static and DHCPv4.  In IPv6, there are three ways – static, DHCPv6, and autoconfiguration.  Just like for IPv4, some deployments lend themselves to one host provisioning mechanism and some another.  IPv6 simply offers network designers more choices, where autoconfiguration is a valuable addition to the set.  Autoconfiguration is not better or worse than DHCPv6 – just used in different use cases.  (Note - DHCPv6 server and client support has been lacking in most platforms until recently – Windows 2008 Server and Vista have/will have good support) 

For example, the classic deployment model for large enterprises, with sophisticated end-user workstations (read “Windows”) has pretty much always been DHCP for users and static addressing for servers.  For IPv6, I would expect this model to hold – at least for the near future.  DHCPv4 is well-understood by enterprise IT staff, and DHCPv6 is very similar operationally.  Even if we skip a feature-by-feature comparison of the protocols, just given human nature and the fact that a lot of IT policies reflect the use of DHCP to manage address assignments, DHCPv6 is likely to be popular in the enterprise.  That makes sense to me.   

By contrast, the address provisioning model for lightweight, client-only devices like sensors or tracking devices (low power, low memory, low CPU capability, cheap) is likely to be stateless autoconfiguration.  DHCP is a pretty sophisticated protocol to ask of a low-end device – like a temperature sensor dropped by the hundreds from a plane onto a volcanic hillside threatening a remote village.  Using autoconfiguration the sensor would join the just-deployed RF network, autoconfigure an address and set a default route, and start reporting. 

And these are just a couple of today’s big deployment models.  IPv6 will be the IP protocol of choice for 30 years or so (starting now), and there will be lots of new devices and new deployment models invented ahead.  DHCPv6 is a fine provisioning protocol, likely to be popular in the enterprise.  Autoconfiguration will be the right tool for a lot of other types of deployments.  Both will be valuable to this and the next generation of network builders.