What’s next for Open-IX?

I’ve recently returned from the NANOG 61 meeting in Seattle (well, Bellevue, just across the lake), a fantastic meeting with well over 800 attendees. It was good to meet some new folk as well as catch up with some industry contacts and old friends.

One of the topics which came up for discussion was the activities of the Open-IX association. This is a group which exists to promote fairness and open competition between Internet Exchange and Co-location operators in the US, and thus improve the competitiveness of the market for the users of those services, such as ISPs and content providers.

It was originally set-up to address what was something of a market failure and a desire by a number of US network operators to encourage organisations that run Exchange facilities (such as Equinix) to have more transparent dealings with their customer base, such as fair pricing and basic expectations of service level. This is something that is more common in Europe, where a large majority of Internet Exchanges are run as non-profits, owned and steered by their participant communities.

To do this, the Open-IX Association don’t actually plan to own or operate exchanges, but instead act as a certification body, developing a set of basic standards for exchange companies to work to. It’s somewhat succeeded in it’s initial goals of correcting the market failure. New IXP entrants in the shape of the three large European IXPs have entered the North American market, and co-location operators who were previously less active in the interconnection market have become more engaged.

So, one of the questions asked is what next for Open-IX?

(Indeed, my former boss, LINX CEO John Souter even ventured to suggest it’s “served it’s purpose” and could be wound up.)

There has been questions from some smaller IXPs, they can’t meet all the criteria laid down in the OIX-1 standard (and possibly don’t wish to or have means of doing so). Does this some how make them a “less worthy” second-class IXP, despite the fact that they serve their own communities perfectly well?

In particular, both the Seattle Internet Exchange and Toronto Internet Exchange currently can’t comply with OIX-1, but at the same time it’s not important for them to do so. The difference being these are member-driven exchanges, more along the lines of the European model. Their members don’t require them to provide the services which would allow the organisations to confirm to OIX-1.

I don’t think anyone would venture to suggest that the SIX or TorIX are in some way “second class” though, right? They are both well run, have plenty of participants on the exchange fabric, and respected in the IX community.

This is a key difference between these exchanges and commercial operations such as Equinix: The member-driven IXPs such as SIX and TorIX don’t need an Open-IX to set standards for them. Those local communities set their own standards, and it’s worked for them so far.

And maybe that’s where the opportunity lies for Open-IX: To act like this “conscience” for the more commercial operators, in the same way as the members steer the non-profits?

“Ambassador, with these Atlas probes, you’re really spoiling us…”

Okay. So I only expect the Brits to get the title of this. Though if you’re desperate to be in on the “joke”, watch this YouTube video of an old British TV ad for some chocolates.

One of the things I do for the community is act as a “RIPE Atlas Ambassador” – that’s someone who helps distribute RIPE Atlas internet measurement probes into the wider Internet community. The Measurements Community Builders at the RIPE NCC send me a box of Atlas probes, I go to conferences, meetings and other get togethers and I give them out to folk who would like to host a probe, along with answering any questions as best I can.

Recently, Fearghas McKay of the IX Scotland steering group asked me if I had any data from the Atlas project on internet round-trip time for probes located in Scotland, to get to services hosted in Scotland, and if I could talk about it at a meeting of IX Scotland participants.

This is a fairly similar exercise to the one I did for Northern Ireland.

One of the challenges I was faced with was the distinct lack of source data. Firstly, there weren’t that many Atlas probes in Scotland to begin with, and those which are there are mostly located in the “central belt” – around Glasgow and Edinburgh. The furthest North was a single probe in Aberdeen, and Scotland is a big country – it’s around 300 miles from the border at Gretna to Thurso, one of the most northerly towns on the Scottish mainland, as far again as it is from London to Gretna. That’s not even counting the Orkneys, Shetlands or Hebridean Islands, which have their own networking challenges.

The second problem was that of those probes, only three at the time were on an ISP connected directly to IX Scotland, and one of those was down! The majority were on consumer broadband providers such as BT and Virgin Media, which aren’t connected to many regional exchanges.

I saw attending the IX Scotland meeting as a good chance to redress the balance and extend the usefulness of the Atlas platform by distributing probes to networks which could improve the coverage.

This has resulted in what is currently the most Northerly probe in the UK being brought online in Dingwall, not far from Inverness, thanks to the folk at HighNet. They’ve also got a few other probes from me, so expect to see more in that area soon.

Most Northerly Probe in the UK
Most Northerly Probe in the UK

HighNet aren’t connected to IX Scotland yet, but maybe now they’ve got access to this instrumentation it might help them make a business case to follow up on that.

I also issued a number of probes at UKNOF in Manchester last week and I’m looking forward to seeing where they turn up.

I’d really like to get some of the community broadband projects in the UK instrumented, such as B4RN and Gigaclear. These bring some of their own challenges, such as issues with equipment at the customer premises that can actually handle the available bandwidth on the connection! It would also be great to be able to draw comparisons in performance between the community fibre service and the slower ADSL service provided over long copper tails in those areas.

For peering in New York, read New Amsterdam

Dutch East India Company Logo
It’s colonialism all over again. Just not as we know it…

Last week, there was this announcement about the establishment of a new Internet Exchange point in New York by the US arm of the Amsterdam Internet Exchange – “AMS-IX New York” – or should that be “New Amsterdam”… 🙂

This follows on from the vote between AMS-IX members about whether or not the organisation should establish an operation in the US was carried by a fairly narrow majority. I wrote about this a few weeks ago.

This completes the moves by the “big three” European IX operators into the US market, arriving on US shores under the umbrella of the Open-IX initiative to increase market choice and competitiveness of interconnection in the US markets.

LINX have established LINX-NoVA in the Washington DC metro area, and AMS-IX are proceeding with their NY-NJ platform, while DECIX have issued a press statement on their plan to enter the NY market in due course.

One of the key things this does is bring these three IXPs into real direct competition in the same market territory for the first time.

There has always been some level of competition among the larger EU exchanges when attracting new international participants to their exchange, for instance DECIX carved itself a niche for attracting Eastern European and Russian players on account that many carrier services to these regions would hub through Frankfurt anyway.

But each exchange always had it’s indigenous home market to provide a constant base load of members, there wasn’t massive amounts of competition for the local/national peers, even though all three countries have a layer of smaller exchanges active in the home market.

Now, to some extent, they are going head-to-head, not just with US incumbents such as Equinix, TelX and Any2, but potentially with each other as well.

The other thing the AMS-IX move could end up doing is potentially fracture even further the NY peering market, which is already fractured – being served by three, maybe four, sizeable exchanges. Can it sustain a fifth or sixth?

Is it going to be economical for ISPs and Content Providers to connect to a further co-terminous IXP (or two)? Can the NY market support that? Does it make traffic engineering more complex for networks which interconnect in NY? So complex that it’s not worth it? Or does it present an opportunity to be able to more finely slice-and-dice traffic and share the load?

Don’t forget we’re also in a market which has been traditionally biased toward minimising the amount of public switch-based peering in favour of private bi-lateral cross-connects. Sure, the viewpoint is changing, but are we looking for a further swing in a long-term behaviour?

We found out from experience in the 2000s that London can only really sustain two IXPs – LINX and LONAP. There were at least 4 well-known IXPs in London in the 2000s, along with several smaller ones. (Aside… if you Google for LIPEX today, you get a link to a cholesterol-reducing statin drug.)

Going to locations on the East Coast may have made sense when we sailed there in ships and it took us several weeks to do it, but that’s no reason for history to repeat itself in this day and age, is it? So why choose New York now?

Will the EU players become dominant in these markets? Will they manage to help fractured markets such as NY to coalesce? If they do, they will have achieved something that people have been trying to do for years. Or, will it turn out to be an interesting experiment and learning experience?

It will be interesting to see how this plays out over time.

IX Scotland – Why might it work this time?

Yesterday the BBC ran this news item about the launch of a new Internet Exchange in Edinburgh – IX Scotland. This is the latest in an emerging trend of local IXPs developing in the UK, such as IX Leeds and IX Manchester.

There was some belief that this is the first Internet Exchange in Scotland, however those people have short memories. There have been two (or three) depending on how you look at it, attempts at getting a working IXP in Edinburgh in the past 15 years, all of which ultimately failed.

So, why should IX Scotland be any different to it’s predecessors? Continue reading “IX Scotland – Why might it work this time?”

My recent talk at INEX – Video

Or, I never thought of myself as a narcissist but…

Thanks to the folks at HEAnet, here’s a link to the video of the talk “It’s peering, Jim…” that I gave at the recent INEX meeting in Dublin, where I discuss topics such as changes in the US peering community thanks to Open-IX and try to untangle what people mean when they say “Regional Peering”.

The talk lasts around 20-25 minutes and I was really pleased to get around 15 minutes of questions at the end of it.

I also provide some fairly pragmatic advice to those seeking to start an IX in Northern Ireland during the questions. 🙂

mh_inex_video

AMS-IX: Green Light to Incorporate US entity

Members of the Dutch Amsterdam Internet Exchange have given the organisation a green light to incorporate a US entity in order to engage with the Open IX initiative and have the ability to run an exchange in the US while minimising risk to the Dutch association and the Dutch operating company.

This completes the announcements from the big 3 European exchanges (LINX, AMS-IX and DECIX) to operate interconnection services in the US, with the first to make an overt move being LINX, who are in the process of establishing an operation in Northern Virginia. DECIX issued a press release last week that they plan to enter the New York market, and now AMS-IX have a member endorsement to make a move.

There have been concerns amongst the Dutch technical community, who have long held AMS-IX in high regard, that establishing operations in the US will leave the AMS-IX as a whole vulnerable to the sort of systemic monitoring that has been revealed in the press in past weeks. While this is partly the reason for the AMS-IX company suggesting a separate legal entity, in order to hold the US operations at arms length, is it enough for some of the Dutch community? Seems not. In this message the Dutch R&E Network SURFnet seem to think the whole thing was rushed, might not be in the best interests of the community, and voted against the move.

It has been noted that members of the Open IX community, including members of the Open IX Board, were openly calling for AMS-IX members to vote “YES”, and suggesting they also “go out and get 5 other votes”.

What do people think about that? Given that an IX that affiliates to Open IX will have to pay Open IX membership dues, was it right of them to appear to lobby AMS-IX members?

What do people think about the establishment of the separate legal entity? Will this be enough?

Has this done lasting damage to the standing of AMS-IX in the Dutch networking community? Does this matter, or has AMS-IX grown so large that such goodwill doesn’t matter anymore?

On the bigger question, is this sort of thing damaging in the long term to the EU peering community? Does the growth into different countries with different cultures threaten to dilute the member-based ethos that defines a lot of EU exchanges? Or is that just another management challenge for the IX operator to solve?

Might Equinix, who have so far not directly competed with the established EU exchanges, decide they are taking the gloves off and start their own European IX operations in a turf war?

Interesting times.

Was the LINX hit by an attack yesterday?

The short answer is “No“.

There has been speculation in the press, such as this Computer Weekly article, but I would say that it’s poorly informed, and even suggests that LINX’s pioneering deployment of Juniper’s PTX MPLS core switch might be a factor (which I think is a red herring).

It looks to have been some sort of storm of flooded traffic (such as unknown unicast, or broadcast) or problem in a network that’s attached to LINX, which managed to either congest the bandwidth of various ISP’s access lines into LINX, or congest the CPU on some of the attached routers, to the extent that they became unable to forward customer traffic, or unable to maintain accurate routing information (i.e. lost control plane integrity).

But, why did it appear to start on one of the two LINX peering platforms (the Extreme-based network) and then cascade to the physically seperate Juniper-based LAN?

I think one of the main reasons is because lots of ISP routers are connected to both LANs, as are the routers operated by the likely “problem” network which originated the flood of traffic in the first place. I’ve written before on this blog about why having a small number of routers connected to a larger number of internet exchanges can be a bad idea.

I’m pressed for time (about to get on a plane), so I’ll quickly sum up with some informed speculation:

I don’t think…

  • The LINX was DDoS-ed (or specifically attacked)
  • The deployment of the Juniper PTX in the preceeding 24 hours had anything to do with it -LINX also seem to think this, as they switched a further PTX into service overnight last night
  • That there was any intentional action which caused this, more likely some sort of failure or bug

I do think…

  • A LINX-attached network had a technical problem which wasn’t isolated and caused a traffic storm
  • It initially affected the Extreme-based platform
  • It affected the CPU of LINX-connected routers belonging to LINX members
  • Some LINX members deliberately disconnected themselves from LINX at the time to protect their own platform
  • The reported loss of peer connectivity on the Juniper platform was “collateral damage” from the initial incident, for reasons I’ve outlined above – busy routers
  • LINX did the right thing continuing their PTX deployment

I’m sure there will be more details forthcoming from LINX in due course. Their staff are trained not to make speculation, nor to talk to the press, during an incident. Even those who handle press enquiries are very careful not to speculate or sensationalise, which I’m sure dissapoints those looking for a story.

The moral of this story is redundancy and diversity are important elements of good network engineering and you shouldn’t be putting all your eggs in one basket.

Disclaimer: I used to work for LINX, and I like to think I’ve got more than half a clue when it comes to how peering and interconnect works.