For peering in New York, read New Amsterdam

Dutch East India Company Logo
It’s colonialism all over again. Just not as we know it…

Last week, there was this announcement about the establishment of a new Internet Exchange point in New York by the US arm of the Amsterdam Internet Exchange – “AMS-IX New York” – or should that be “New Amsterdam”… 🙂

This follows on from the vote between AMS-IX members about whether or not the organisation should establish an operation in the US was carried by a fairly narrow majority. I wrote about this a few weeks ago.

This completes the moves by the “big three” European IX operators into the US market, arriving on US shores under the umbrella of the Open-IX initiative to increase market choice and competitiveness of interconnection in the US markets.

LINX have established LINX-NoVA in the Washington DC metro area, and AMS-IX are proceeding with their NY-NJ platform, while DECIX have issued a press statement on their plan to enter the NY market in due course.

One of the key things this does is bring these three IXPs into real direct competition in the same market territory for the first time.

There has always been some level of competition among the larger EU exchanges when attracting new international participants to their exchange, for instance DECIX carved itself a niche for attracting Eastern European and Russian players on account that many carrier services to these regions would hub through Frankfurt anyway.

But each exchange always had it’s indigenous home market to provide a constant base load of members, there wasn’t massive amounts of competition for the local/national peers, even though all three countries have a layer of smaller exchanges active in the home market.

Now, to some extent, they are going head-to-head, not just with US incumbents such as Equinix, TelX and Any2, but potentially with each other as well.

The other thing the AMS-IX move could end up doing is potentially fracture even further the NY peering market, which is already fractured – being served by three, maybe four, sizeable exchanges. Can it sustain a fifth or sixth?

Is it going to be economical for ISPs and Content Providers to connect to a further co-terminous IXP (or two)? Can the NY market support that? Does it make traffic engineering more complex for networks which interconnect in NY? So complex that it’s not worth it? Or does it present an opportunity to be able to more finely slice-and-dice traffic and share the load?

Don’t forget we’re also in a market which has been traditionally biased toward minimising the amount of public switch-based peering in favour of private bi-lateral cross-connects. Sure, the viewpoint is changing, but are we looking for a further swing in a long-term behaviour?

We found out from experience in the 2000s that London can only really sustain two IXPs – LINX and LONAP. There were at least 4 well-known IXPs in London in the 2000s, along with several smaller ones. (Aside… if you Google for LIPEX today, you get a link to a cholesterol-reducing statin drug.)

Going to locations on the East Coast may have made sense when we sailed there in ships and it took us several weeks to do it, but that’s no reason for history to repeat itself in this day and age, is it? So why choose New York now?

Will the EU players become dominant in these markets? Will they manage to help fractured markets such as NY to coalesce? If they do, they will have achieved something that people have been trying to do for years. Or, will it turn out to be an interesting experiment and learning experience?

It will be interesting to see how this plays out over time.

Was the LINX hit by an attack yesterday?

The short answer is “No“.

There has been speculation in the press, such as this Computer Weekly article, but I would say that it’s poorly informed, and even suggests that LINX’s pioneering deployment of Juniper’s PTX MPLS core switch might be a factor (which I think is a red herring).

It looks to have been some sort of storm of flooded traffic (such as unknown unicast, or broadcast) or problem in a network that’s attached to LINX, which managed to either congest the bandwidth of various ISP’s access lines into LINX, or congest the CPU on some of the attached routers, to the extent that they became unable to forward customer traffic, or unable to maintain accurate routing information (i.e. lost control plane integrity).

But, why did it appear to start on one of the two LINX peering platforms (the Extreme-based network) and then cascade to the physically seperate Juniper-based LAN?

I think one of the main reasons is because lots of ISP routers are connected to both LANs, as are the routers operated by the likely “problem” network which originated the flood of traffic in the first place. I’ve written before on this blog about why having a small number of routers connected to a larger number of internet exchanges can be a bad idea.

I’m pressed for time (about to get on a plane), so I’ll quickly sum up with some informed speculation:

I don’t think…

  • The LINX was DDoS-ed (or specifically attacked)
  • The deployment of the Juniper PTX in the preceeding 24 hours had anything to do with it -LINX also seem to think this, as they switched a further PTX into service overnight last night
  • That there was any intentional action which caused this, more likely some sort of failure or bug

I do think…

  • A LINX-attached network had a technical problem which wasn’t isolated and caused a traffic storm
  • It initially affected the Extreme-based platform
  • It affected the CPU of LINX-connected routers belonging to LINX members
  • Some LINX members deliberately disconnected themselves from LINX at the time to protect their own platform
  • The reported loss of peer connectivity on the Juniper platform was “collateral damage” from the initial incident, for reasons I’ve outlined above – busy routers
  • LINX did the right thing continuing their PTX deployment

I’m sure there will be more details forthcoming from LINX in due course. Their staff are trained not to make speculation, nor to talk to the press, during an incident. Even those who handle press enquiries are very careful not to speculate or sensationalise, which I’m sure dissapoints those looking for a story.

The moral of this story is redundancy and diversity are important elements of good network engineering and you shouldn’t be putting all your eggs in one basket.

Disclaimer: I used to work for LINX, and I like to think I’ve got more than half a clue when it comes to how peering and interconnect works.

Successful 1st IXLeeds Open Meeting

I attended by all accounts a very successful first open meeting for the IXLeeds exchange point yesterday – with around 120 attendees, including many faces that are not regulars on the peering circuit making for brilliant networking opportunities and great talks from the likes of the Government super-fast broadband initiative, BDUK, and energy efficient processor giants ARM (behind the technology at the heart of most of the World’s smartphones), as well as more familiar faces such as RIPE NCC and LINX, among others.

Definitely impressed with the frank discussion that followed the talk by the DCMS’ Robert Ling on BDUK funding and framework, but still sceptical that it’s going to be any easier for smaller businesses to successfully get access to the public purse.

Andy Davidson, IXLeeds Director, was able to proudly announce that IXLeeds now provides support for jumbo frames via a seperate vlan overlaid on their switch, which is probably the only IXP in the UK which officially offers and promotes this service – at least for the time being. Of course, they are supporting a 9k frame size

Well done to my friends and colleagues of IXLeeds for making it to this major milestone, and doing it in great style. It seems a long, long way from a discussion over some pizza in 2008.

The only thing I didn’t manage to do while in Leeds is take a look at the progress on the next phase of aql’s Salem Church data centre, but I’m sure I’ll just have to ask nicely and drop by aql at some point in the future. 🙂