It’s colonialism all over again. Just not as we know it…
Last week, there was this announcement about the establishment of a new Internet Exchange point in New York by the US arm of the Amsterdam Internet Exchange – “AMS-IX New York” – or should that be “New Amsterdam”… 🙂
This follows on from the vote between AMS-IX members about whether or not the organisation should establish an operation in the US was carried by a fairly narrow majority. I wrote about this a few weeks ago.
This completes the moves by the “big three” European IX operators into the US market, arriving on US shores under the umbrella of the Open-IX initiative to increase market choice and competitiveness of interconnection in the US markets.
LINX have established LINX-NoVA in the Washington DC metro area, and AMS-IX are proceeding with their NY-NJ platform, while DECIX have issued a press statement on their plan to enter the NY market in due course.
One of the key things this does is bring these three IXPs into real direct competition in the same market territory for the first time.
There has always been some level of competition among the larger EU exchanges when attracting new international participants to their exchange, for instance DECIX carved itself a niche for attracting Eastern European and Russian players on account that many carrier services to these regions would hub through Frankfurt anyway.
But each exchange always had it’s indigenous home market to provide a constant base load of members, there wasn’t massive amounts of competition for the local/national peers, even though all three countries have a layer of smaller exchanges active in the home market.
Now, to some extent, they are going head-to-head, not just with US incumbents such as Equinix, TelX and Any2, but potentially with each other as well.
The other thing the AMS-IX move could end up doing is potentially fracture even further the NY peering market, which is already fractured – being served by three, maybe four, sizeable exchanges. Can it sustain a fifth or sixth?
Is it going to be economical for ISPs and Content Providers to connect to a further co-terminous IXP (or two)? Can the NY market support that? Does it make traffic engineering more complex for networks which interconnect in NY? So complex that it’s not worth it? Or does it present an opportunity to be able to more finely slice-and-dice traffic and share the load?
Don’t forget we’re also in a market which has been traditionally biased toward minimising the amount of public switch-based peering in favour of private bi-lateral cross-connects. Sure, the viewpoint is changing, but are we looking for a further swing in a long-term behaviour?
We found out from experience in the 2000s that London can only really sustain two IXPs – LINX and LONAP. There were at least 4 well-known IXPs in London in the 2000s, along with several smaller ones. (Aside… if you Google for LIPEX today, you get a link to a cholesterol-reducing statin drug.)
Going to locations on the East Coast may have made sense when we sailed there in ships and it took us several weeks to do it, but that’s no reason for history to repeat itself in this day and age, is it? So why choose New York now?
Will the EU players become dominant in these markets? Will they manage to help fractured markets such as NY to coalesce? If they do, they will have achieved something that people have been trying to do for years. Or, will it turn out to be an interesting experiment and learning experience?
It will be interesting to see how this plays out over time.
While doing a bit of technical research (read: running traceroute) for the post on IX Scotland, I came across this blast from the past…
So, not only do we have a bit of a blast from the past in the reverse DNS, but does this really make it look like Scotland is behind a L(3) (ex-BBN/Genuity) Fast Ethernet port?
Actually, it’s more of a testament to how infrequently ISPs check and update their reverse DNS zones, and sometimes how infrequently networks change their transit providers.
The “scotland” referred to here is actually going to be Brightsolid (the former Scotland On-line – changing it’s name more often than changing it’s upstream!), and if you look carefully at the latencies between the highlighted hop 5 and the previous hop 4, you’ll see that hop 5 isn’t actually in Scotland, but will be very close to hop 4 – so actually in Manchester itself. Hop 6 is actually the first hop in Scotland, 14 ms away.
Reverse DNS zones – that’s IP address to hostname lookup, rather than the other way around, which is more common – are notorious for being neglected, containing stale data or sometimes scarcely being populated at all.
Yet, they provide important diagnostic information. For instance, it’s not just the domain and customer info which is out of date, but I suspect the interface information is as well. It’s unlikely to be a Fast Ethernet port in this day and age.
Sadly, the situation seems to be even worse with IPv6. With those awkward long addresses, some just seem to be “not bothering” with reverse entries at all.
If you run a network, reverse entries are really useful tools to both your own netops folk and to your customers and peers, and deserve to be maintained.
Yesterday the BBC ran this news item about the launch of a new Internet Exchange in Edinburgh – IX Scotland. This is the latest in an emerging trend of local IXPs developing in the UK, such as IX Leeds and IX Manchester.
There was some belief that this is the first Internet Exchange in Scotland, however those people have short memories. There have been two (or three) depending on how you look at it, attempts at getting a working IXP in Edinburgh in the past 15 years, all of which ultimately failed.
In some ways, I’m probably a stick in the mud. Years of experience with networks and systems make you wary about the early adopter of “point-zero” software releases, and if you can, wait until they shake the bugs out.
So, despite being a bit of a travel addict, I still managed to not go on an Airbus A380 until last weekend – they didn’t work the routes I’ve been flying and weren’t operated by many airlines I regularly fly.
That changed last week when I flew on British Airways’ new A380 to Los Angeles.
The overall impression I got was how quiet the plane is. It’s seriously quiet. On takeoff, you basically just notice a change in engine note and a bit of acceleration. During the flight, it’s so quiet that you can hear a person snoring a couple of feet away.
While there were one or two rough edges on the service because the aircraft is still fairly new in the BA fleet, such as things taking a little longer than usual as the crew get used to the new equipment, it’s pretty impressive stuff, even though BA chose not to have “bling” elements such as inflight bars, lounges or showers. They chose to try and ensure consistency in the long haul product, keeping a similar look and feel to existing aircraft such as the 777-300, but behind the scenes, the galley equipment is new and different from that flying on the majority of the older fleet.
One thing I noticed was that even on the A380’s upper deck, you don’t have that same exclusiveness of the 747-400 upper deck, often sought-after among frequent fliers. It’s not quite the same on the 380. It’s definitely spacious, but as there’s more of you up there, it doesn’t feel quite as special.
Following on from my recent post regarding a rather poor Environmental Health Report for BA’s “exclusive” Concorde Room, Hillingdon Council, the local authority responsible for Heathrow, have conducted further inspections of BA’s lounge operations at the airport.
This time, the largest lounge in T5, the “Galleries Club South” scored 2 out of 5, like it’s neighbour.
The report for this lounge highlights a number of basic food hygiene failings that seem to indicate a real lack of care.
Or, I never thought of myself as a narcissist but…
Thanks to the folks at HEAnet, here’s a link to the video of the talk “It’s peering, Jim…” that I gave at the recent INEX meeting in Dublin, where I discuss topics such as changes in the US peering community thanks to Open-IX and try to untangle what people mean when they say “Regional Peering”.
The talk lasts around 20-25 minutes and I was really pleased to get around 15 minutes of questions at the end of it.
I also provide some fairly pragmatic advice to those seeking to start an IX in Northern Ireland during the questions. 🙂
Last week, I was over in Dublin having been invited to give a talk by my gracious hosts at the Irish Internet Exchange Point, INEX. I asked what sort of thing they might like me to talk about. We agreed that I’d talk about various trends in global peering, mainly because the INEX meeting audience don’t do massive amounts of peering outside of the island of Ireland.
(If you need to understand the difference between the UK, Great Britain, Northern Ireland, the Republic of Eire and the island of Ireland this video will be a massive help. Thanks CGP Grey.)
One of the discussions we had was what is meant when we say “Regional” when talking about Internet Exchange points? In the UK, we generally mean exchanges which are outside of London, such as IX Leeds. When a “Regional IXP” is discussed in Africa, they actually mean a “super-national” IXP which possibly interconnects several countries across a region.
Why do the communities in these areas want IXPs that span national boundaries?
The main reason:Â latency.
There is a lot of suboptimal routing. Traffic being exchanged between adjacent countries on the same continent can end up making a long “trombone-shaped” trip to Europe and back. This has a negative effect on the user experience and on the local internet economy.
Round-trip times from RIPE Atlas probes in Southern African countries to a destination in South Africa
As you can see above, traffic from the test probes Kenya and Angola, along with the Maldives and the Seychelles is likely being routed to Europe for interconnection, rather than being handled more locally, if the round-trip time is an indication of route taken. The probes in Botswana, Zambia and Tanzania do somewhat better, and are definitely staying on the same continent. The African example is one of the obvious ones. Let’s look at something a bit closer to home…
Regional peering in Northern Ireland and Northern Ireland to the Republic of Ireland
There is already a well established exchange point in Dublin, INEX, with a good number of national and international members. Discussions are taking place between Internet companies in Northern Ireland (which, remember, is part of the UK) about their need for a more local place to exchange traffic, likely in Belfast. The current belief is a large amount of the traffic between sources and sinks in Northern Ireland goes to London or Amsterdam.
Firstly, how does traffic get from the UK (and by inference, most of the rest of Europe) and Northern Ireland? This is what Telegeography say:
Submarine Cables UK to NIRIPE Atlas Probes in Northern Ireland
So, I thought I’d do some RIPE Atlas measurements.
This isn’t meant to be an exhaustive analysis. More just exploring some existing theories and perceptions.
The first trick is to identify probes in Northern Ireland. From the RIPE PoV, these are all indicated as part of the UK (go and watch the video again if you didn’t get it the first time), so I can’t select them by country.
Fortunately, probe owners have to set their probe’s location – there is a certain amount of trust placed in them, there’s nothing stopping me saying my probe is somewhere else, but most probe owners are responsible techy types. The RIPE Atlas people also put the probe locations onto a coverage map.
I also needed some targets. Probes can’t ping each other (well, they can, if you know their IP address, and they’re not behind some NAT or firewall). The Atlas project provides a number of targets, known as “anchors”, as well as nodes in the NLnog ring which can act as targets. There’s an Atlas anchor in Dublin, but that couldn’t take any more measurements, so that wasn’t suitable as a target, but HEAnet (the Irish R&E network) and Amazon (yep, the folks that sell books and whatnot) have NLnog ring nodes in Dublin.
We also needed targets in Northern Ireland that seemed to answer ICMP relatively unmolested, and I chose DNS servers at Tibus and Atlas/Bytel, both of whom are ISPs in the North. The final things to add were “controls”, so I chose a friend’s NLnog ring box which I know is hosted in London, and two other UK-based Atlas probes, the one I have on my network at home, and one on Paul Thornton’s network in Sussex. These effectively provided known UK-Ireland and UK-NI latencies to the targets, and a known NI-London latency for the probes in NI.
So, let’s look at round-trip time from Northern Ireland to the NLnog ring node in London:
ICMP RTT NI Probes to nuqe.net NLnog ring server
So, we can see there are some variations, no doubt based on last mile access technology. In particular, the node shown here with the 54ms RTT (just North of Belfast) consistently scored a high RTT to all test destinations. Anyway, this gives us an idea of NI-London RTT. The fastest being 15ms.
We can therefore make a reasonable assumption that if traffic were to go from Belfast to London and back to Ireland again, a 30ms RTT would be the best one could expect.
(For the interested, the two “control” test probes in the UK had latencies of 5ms and 8ms to the London target.)
Now, take a look at the RTT from Northern Ireland to the node at HEAnet in Dublin:
ICMP RTT all NI probes to HEAnet NLnog ring node, Dublin
Only two of the probes in Northern Ireland have <10ms RTTs to the target in Dublin. All other probes have a greater RTT.
It is not unreasonable to assume, given that some have a >30ms RTT, or have exhibited a >15ms gain in RTT between the RTT to London and the RTT to Dublin, that this traffic is routing via London.
Of the two probes which show a <10ms RTT to HEAnet in Dublin, their upstream networks (AS43599 and AS31641) are directly connected to INEX.
Of the others, some of the host ASNs are connected to INEX, but the RTT suggests an indirect routing, possibly via the UK mainland.
The tests were also run against another target in Dublin, on the Amazon network, and show broadly similar results:
ICMP RTT all NI probes to Amazon NLnog ring node, Dublin
Again, the same two probes show <10ms RTT to Dublin. All others show >30ms. Doesn’t seem to matter if you’re a commercial or an academic network.
Finally, lets look at round trip times within Northern Ireland.
Here’s the test to a nameserver on the Tibus network:
ICMP RTT all NI Probes to Tibus Nameserver
Again, the same two probes report a lower than <10ms latency. I’d surmise that these are either routing via INEX, both host networks are downstream of the same transit provider in Belfast, or are privately interconnected in Belfast. At least two of the other nodes seem to route via the UK mainland.
To check this result, the same tests performed toward a nameserver on the Atlas/Bytel network:
ICMP RTT all NI probes to Atlas/Bytel Nameserver
Obviously, one of our probes is on-net, with a 1ms RTT!
Of the others, we’re definitely looking at “trombone routing” of the traffic, in most cases back to the UK mainland.
This may not be entirely surprising, as I’m told that BT don’t provide a 21CN interconnect node in Northern Ireland, so traffic on BT wholesale access products will “trombone” through the mainland in any case.
So, what’s really needed in Northern Ireland?
We’ve shown that if networks are willing to buy capacity to Dublin, they can happily exchange traffic at INEX and keep the latency down. An obvious concern some may have is the export of traffic from one jurisdiction to another, especially in light of recent revelations about systemic monitoring, if it’s NI to NI traffic.
The utility of IX in Northern Ireland could be hampered due to the lack of BT 21CN interconnect capability, as it may as well, for all intents and purposes be in Glasgow which is the nearest interconnect, for the traffic will still be making two trips across the Irish Sea whatever happens, assuming one end or the other is on the end of a BT wholesale pipe. (At worst, it could be 4 trips if both ends are on a BT pipe!)
If the goal is to foster internet growth (e.g. “production” of bandwidth) in Northern Ireland, where is it going to come from?
Are Northern Irish interests better served by connecting to the mature interconnect community in Dublin?
Is a BT 21CN interconnect in Belfast essential for growth, or can NI operators build around it?
Should INEX put a switch in Belfast? If they do, should it be backhauled to the larger community in Dublin? Or is that somehow overstepping the remit of an exchange point?Â
Members of the Dutch Amsterdam Internet Exchange have given the organisation a green light to incorporate a US entity in order to engage with the Open IX initiative and have the ability to run an exchange in the US while minimising risk to the Dutch association and the Dutch operating company.
This completes the announcements from the big 3 European exchanges (LINX, AMS-IX and DECIX) to operate interconnection services in the US, with the first to make an overt move being LINX, who are in the process of establishing an operation in Northern Virginia. DECIX issued a press release last week that they plan to enter the New York market, and now AMS-IX have a member endorsement to make a move.
There have been concerns amongst the Dutch technical community, who have long held AMS-IX in high regard, that establishing operations in the US will leave the AMS-IX as a whole vulnerable to the sort of systemic monitoring that has been revealed in the press in past weeks. While this is partly the reason for the AMS-IX company suggesting a separate legal entity, in order to hold the US operations at arms length, is it enough for some of the Dutch community? Seems not. In this message the Dutch R&E Network SURFnet seem to think the whole thing was rushed, might not be in the best interests of the community, and voted against the move.
It has been noted that members of the Open IX community, including members of the Open IX Board, were openly calling for AMS-IX members to vote “YES”, and suggesting they also “go out and get 5 other votes”.
What do people think about that? Given that an IX that affiliates to Open IX will have to pay Open IX membership dues, was it right of them to appear to lobby AMS-IX members?
What do people think about the establishment of the separate legal entity? Will this be enough?
Has this done lasting damage to the standing of AMS-IX in the Dutch networking community? Does this matter, or has AMS-IX grown so large that such goodwill doesn’t matter anymore?
On the bigger question, is this sort of thing damaging in the long term to the EU peering community? Does the growth into different countries with different cultures threaten to dilute the member-based ethos that defines a lot of EU exchanges? Or is that just another management challenge for the IX operator to solve?
Might Equinix, who have so far not directly competed with the established EU exchanges, decide they are taking the gloves off and start their own European IX operations in a turf war?
Something of a first world problem admittedly, but it’s recently come to the attention of the various frequent flyer circles that BA’s “flagship” lounges at it’s Heathrow hub, the T5 Concorde Room and First Class Lounge recently only scored 2 (out of 5) on a recent food hygiene inspection.
The low score places this “exclusive” venue (to paraphrase BA), reserved specifically for it’s “top customers”, into the bottom 10% of food service premises in the UK. This is something of a last straw for BA’s loyal frequent flyers who have already been upset by a perceived reduction in the quality and service offered by the lounges since the contract for running the food service operation at all BA’s UK lounges were switched to a new operator earlier this year.
There have been complaints of less choice, simple service failures such as grubby cups, glasses and plates put out for customers to use, and used, dirty pots not being regularly cleared away, food not being cooked through properly, and a previously reasonable hot buffet being replaced with troughs of stodgy “gloop” – unpleasant wet food.
Thanks to a Freedom of Information request, the local authority responsible for the inspection, Hillingdon Council, have made the full contents of the report available, highlighting a catalogue of basic food safety disasters:
Out-of-date food in the kitchens
Multiple food preparation areas being sufficiently dirty to be in need of immediate cleaning
“High-risk” food such as prepared sandwiches and cooked meats being insufficiently chilled
Hot buffet food being kept at a sufficiently low temperature to increase risk
Cross-contamination between raw and cooked food
Kitchen maintenance problems such as holes in the walls and floor
Inadequate documentation of staff training
BA have so far been tight-lipped on the matter, anecdotal reports suggesting that senior BA figures consider this just to be some “noisy people on the Internet” which probably highlights that they don’t get it and have their head firmly in the sand. Does this indicate a level of disrespect within BA for it’s customers?
To their credit BaxterStorey meanwhile have issued a statement which, while conciliatory in tone and recognises the failings to some extent, largely seems to fob the problem off on needing to “refurbish” the kitchen.
This really isn’t a brilliant response. Remember, we’re talking about BA’s flagship lounge at it’s flagship airport.
In terms of apologising, what should BA do?
One of the questions among the frequent flyer community has been over BA’s handling of this. While BA’s sub-contractor has decided to issue a statement, there’s been nothing from BA to the most regular lounge guests, it’s frequent flyers.
It’s my opinion that there’s only one way BA can approach this:
with openness, transparency, responsibility and accountability
I know that’s probably a tough ask of a large multi-national corporation with a slick PR machine which is used to deny accountability for everything from delays to lost luggage.
You may ask why the frequent flyers care so much about getting a response from BA, or why BA should care so much to communicate in an frank and honest way with it’s customers?
The frequent flyers care about getting a spin-free honest reply, because they have made a financial and emotional investment in BA. To earn the magic Silver and Gold cards to get them in the privacy of the Galleries lounges, they have spent a lot of money and time with the airline.
They’ve been good, regular customers, demonstrated loyalty to BA, and so have a built an expectation of being dealt with respectfully and fairly in return. That trust has been betrayed by BA and BaxterStorey.
To feed them spin is likely to just increase the levels of angst and venom. The frequent flyers are actively looking for a reason to forgive.
The more honest and fair BA are with their response the more likely they are to be forgiven by it’s community of regular passengers.
See this as an opportunity to set themselves apart from their competitors. It’s not a disaster that must be avoided. Approach it head on.
You’ve been let down, we failed to meet your expectations. We’ve let our supplier take their eye off the ball. We’re sorry. You deserve better. We’ll do better. Here’s how…
Be honest about the mistakes that led to this, and what’s going to happen to make it better.
Mike's Twitter Overflow - Tech, Travel, other stuff 