Whither (UK) Regional Peering – Pt 2

It’s been a long while since I’ve blogged about this topic

Probably too long, as IXLeeds, something which inspired me to write Pt 1, is now a fully-fledged IX, not just a couple of networks plugged into a switch in a co-lo (all IXPs have to start somewhere!), but has formed a company, with directors, with about 12 active participants connected to its switch. Hurrah!

So, trying to pick up where I left off; in this post, I’m going to talk about shared fate, with respect to Internet Exchanges.

What do I mean by shared fate? Continue reading “Whither (UK) Regional Peering – Pt 2”

Telecity Dominance in Manchester continues?

It’s been a busy week for Manchester-based data centre UK Grid. Firstly, they announce a tie up with AMS-IX and IX Reach to provide a virtual blob of Dutch peering goodness in Manchester’s Science Park, then today, it’s announced that Telecity Group have acquired the UK Grid business, adding three further datacentres to their Manchester operations.

This certainly stays true to Telecity’s current form, which is to buy up competing data centre operators, having acquired Manchester’s IFL and Dublin’s DEG in fairly short order over the last 18 months, significantly increasing Telecity Group’s share, making them something of a dominant player in those markets.

Local (and privately-held) competitor M247 put their own slant on Telecity’s latest move in Manchester by announcing an acquisition of their own: a new building to enable further expansion.

Flash the Cache

Some trends in content distribution have made me think back to my early days on the Internet, both at University, and during my time at a dialup ISP, back in the 1990s.

Bandwidth was a precious commodity, and webcaching, by deploying initially Harvest (remember Harvest?) and latterly it’s offspring Squid, became quite popular, to reduce load on external links. There was also the ability to exchange cache content with other topologically close caches – clustering your cache with those on your neighbour networks.

(I remember that at least one of the UK academic institutions – Imperial College, I think, or maybe it was Janet – had a very popular cache that it openly peered with other caches in the UK, was available via peering across LINX, and as a result was popular and well populated.)

There were attendant problems – staleness of cached content could blight some more active websites, and unless you tried enforced redirection of web traffic (unpopular in some quarters, even today, where transparent cacheing is commonplace), the ISP often had to convince your users to opt-in to using the cache through changing browser settings.

It was no surprise that once bandwidth prices fell, caches started to fall out of favour.

However, that trend has been reversing in recent times… the cache is making a comeback, but in a slightly different guise: Rather than general purpose caches that take a copy of anything passing by, these are very specific caches, targeted and optimised for the content, part of the worldwide growth of the distributed CDN.

Akamai have long been putting their servers out into ISP networks, and into major Internet Exchanges, delivering content locally to ISP subscribers. They famously say they “don’t have a backbone” – they just distribute their content through these local clusters. Akamai are delivering a local copy of “Akamaized” web content to local eyes, and continuing to experience significant growth.

Google is also in the caching game, with the “Google Content Cache” (GCC).

I heard a Google speaker at a recent conference explain how a GCC cluster installation at a broadband ISP provided an 85-90% saving in external bandwith to Google hosted content. To some extent this is clearly driven by YouTube content, but it has other smarts too.

So, what’s helped make caching popular again?

Herd Mentality – Social Networking is King. A link is posted and within minutes, it can have significant numbers of page impressions. For a network with a cache, that content only needs to be fetched once.

Bandwidth Demand – It’s not unheard of for large broadband networks to have huge (nx10G) private peerings with organisations such as Google. At some point, this is going to run into scaling difficulties, and it makes sense to distribute the content closer to the sink.

Fault Tolerance – User expectation is “it should just work”, distributed caches can help prevent localised failures from having widescale effect. (Would it have helped BBC this week?)

Response Speed – Placing the content closer to the user minimises latency, improves the user experience. For instance, GCC apparently takes this one step further, acting as a TCP proxy for more interactive Google services such as Gmail – this helps remove “spongyness” of interactive sessions for those in countries with limited or high-latency external connectivity (some African countries, for instance).

So, great, cacheing is useful and has taken it’s place in the Network Architect’s Swiss Army Knife again. But what’s the tipping point for installing something like the GCC or Akamai cluster on your network? There’s two main things at work: Bandwidth and Power.

Having a CDN cluster on your network doesn’t come for free – even if the CDN owns the hardware, you have to house it and power it and cool it. The normal hardware is a number of general purpose high spec 1U rack-mount PCs.

So the economics seem to be a case of factoring in the cost of bandwidth (whether it’s transit or peering), router interfaces, data centre cross-connects, etc., versus the cost of hosting the gear.

Down at Peckham Market… “Get your addresses here. Laaavley v4 addresses!”

One of the first big deals in the IPv4 address secondary market appears to be happening – Microsoft paying $7.5m for pre-RIR (aka “early registration”) IPv4 address space currently held by Nortel.

There have been deals happening on the secondary market already. But this one is significant for two reasons:

  • The size of the deal – over 600k IPv4 addresses
  • That Nortel’s administrators recognise these unused IPv4 addresses, that Nortel paid either nothing, or only a nominal fee, to recieve, are a real asset which they can realise capital against.

Interesting times… Now, where’s my dodgy yellow van?

The problem with the IETF

There’s been some good efforts to fix the hiatus that’s been perceived to exist between the Internet operator community and the IETF recently. I hope I’m not giving them the kiss of death here… 🙂

A sense of frustration had been bubbling for a while that the IETF had become remote from the people who actually deploy the protocols, that IETF had become the preserve of hardware vendors who lack operational experience, and it’s no wonder they ship deficient protocols.

But, it can’t have always been that way right? Otherwise the Internet wouldn’t work as well as it does?

Well, when the Internet first got going, the people who actually ran the Internet participated in the IETF, because they designed protocols and they hacked at TCP stacks and routing code, as well as running operational networks. Protocols were written with operational considerations to the fore. However, I think people like this are getting fewer and fewer.

As time went by, the Internet moved on, a lot of these same folk stopped running networks day-in-day out, and got jobs with the vendors, but they stayed involved in the IETF, because they were part of that community, they were experienced in developing protocols, and brought operational experience to the working groups that do the development work.

The void in the Network Operations field was filled by the next generation of Network Engineers, and as time has gone by, fewer and fewer of them were interested in deveoping protocols, because they were busy running their rapidly growing networks. Effectively, there had been something of a paradigm shift in the sorts of people who were running networks, which differed from those who had been doing it in the past For the Internet to grow the way it did in such a short time, something had to change, and this was it.

At the same time, the operational engineers were finding more and more issues creeping into increasingly complex protocols. That’s bad for the Internet, right? How did things derail?

The operational experience within the IETF was suffering from two things – 1) it was becoming more and more stale the longer that key IETF participants didn’t have to run networks, and 2) the operator voice present at IETF was getting quieter and quieter, things suggested by operators had been largely rejected as impractical.

Randy Bush had started to refer to it as the IVTF – implying that Vendors had “taken over”.

There have been a few recent attempts to bridge that gap – “outreach” talks and workshops at operations meetings such as RIPE and NANOG sought to get operator input and feedback, however trying to express this without frustration hasn’t always been easy.

However, it looks like we’re getting somewhere…

Rob Shakir has currently got a good Internet Draft out aimed at building a bridge between the ops community who actually deploy the gear and the folks who write the protocol specs and develop the software and hardware.

This has been long overdue and needs to work. It looks good, and is finding support from both the Vendor and Ops communities.

It’s a “meta-problem” here is that one cannot exist without the other, it’s a symbiotic and mutually beneficial relationship that needs to work for a sustainable Internet.

I wonder if it’s actually important for people on the protocol design and vendor side to periodically work on production networks to ensure that they have current operational knowledge, and not relying on that from 10 years ago?