Okay. So I only expect the Brits to get the title of this. Though if you’re desperate to be in on the “joke”, watch this YouTube video of an old British TV ad for some chocolates.
One of the things I do for the community is act as a “RIPE AtlasAmbassador” – that’s someone who helps distribute RIPE Atlas internet measurement probes into the wider Internet community. The Measurements Community Builders at the RIPE NCC send me a box of Atlas probes, I go to conferences, meetings and other get togethers and I give them out to folk who would like to host a probe, along with answering any questions as best I can.
Recently, Fearghas McKay of the IX Scotland steering group asked me if I had any data from the Atlas project on internet round-trip time for probes located in Scotland, to get to services hosted in Scotland, and if I could talk about it at a meeting of IX Scotland participants.
One of the challenges I was faced with was the distinct lack of source data. Firstly, there weren’t that many Atlas probes in Scotland to begin with, and those which are there are mostly located in the “central belt” – around Glasgow and Edinburgh. The furthest North was a single probe in Aberdeen, and Scotland is a big country – it’s around 300 miles from the border at Gretna to Thurso, one of the most northerly towns on the Scottish mainland, as far again as it is from London to Gretna. That’s not even counting the Orkneys, Shetlands or Hebridean Islands, which have their own networking challenges.
The second problem was that of those probes, only three at the time were on an ISP connected directly to IX Scotland, and one of those was down! The majority were on consumer broadband providers such as BT and Virgin Media, which aren’t connected to many regional exchanges.
I saw attending the IX Scotland meeting as a good chance to redress the balance and extend the usefulness of the Atlas platform by distributing probes to networks which could improve the coverage.
This has resulted in what is currently the most Northerly probe in the UK being brought online in Dingwall, not far from Inverness, thanks to the folk at HighNet. They’ve also got a few other probes from me, so expect to see more in that area soon.
Most Northerly Probe in the UK
HighNet aren’t connected to IX Scotland yet, but maybe now they’ve got access to this instrumentation it might help them make a business case to follow up on that.
I also issued a number of probes at UKNOF in Manchester last week and I’m looking forward to seeing where they turn up.
I’d really like to get some of the community broadband projects in the UK instrumented, such as B4RN and Gigaclear. These bring some of their own challenges, such as issues with equipment at the customer premises that can actually handle the available bandwidth on the connection! It would also be great to be able to draw comparisons in performance between the community fibre service and the slower ADSL service provided over long copper tails in those areas.
Yesterday the BBC ran this news item about the launch of a new Internet Exchange in Edinburgh – IX Scotland. This is the latest in an emerging trend of local IXPs developing in the UK, such as IX Leeds and IX Manchester.
There was some belief that this is the first Internet Exchange in Scotland, however those people have short memories. There have been two (or three) depending on how you look at it, attempts at getting a working IXP in Edinburgh in the past 15 years, all of which ultimately failed.
Or, I never thought of myself as a narcissist but…
Thanks to the folks at HEAnet, here’s a link to the video of the talk “It’s peering, Jim…” that I gave at the recent INEX meeting in Dublin, where I discuss topics such as changes in the US peering community thanks to Open-IX and try to untangle what people mean when they say “Regional Peering”.
The talk lasts around 20-25 minutes and I was really pleased to get around 15 minutes of questions at the end of it.
I also provide some fairly pragmatic advice to those seeking to start an IX in Northern Ireland during the questions. 🙂
Last week, I was over in Dublin having been invited to give a talk by my gracious hosts at the Irish Internet Exchange Point, INEX. I asked what sort of thing they might like me to talk about. We agreed that I’d talk about various trends in global peering, mainly because the INEX meeting audience don’t do massive amounts of peering outside of the island of Ireland.
(If you need to understand the difference between the UK, Great Britain, Northern Ireland, the Republic of Eire and the island of Ireland this video will be a massive help. Thanks CGP Grey.)
One of the discussions we had was what is meant when we say “Regional” when talking about Internet Exchange points? In the UK, we generally mean exchanges which are outside of London, such as IX Leeds. When a “Regional IXP” is discussed in Africa, they actually mean a “super-national” IXP which possibly interconnects several countries across a region.
Why do the communities in these areas want IXPs that span national boundaries?
The main reason: latency.
There is a lot of suboptimal routing. Traffic being exchanged between adjacent countries on the same continent can end up making a long “trombone-shaped” trip to Europe and back. This has a negative effect on the user experience and on the local internet economy.
Round-trip times from RIPE Atlas probes in Southern African countries to a destination in South Africa
As you can see above, traffic from the test probes Kenya and Angola, along with the Maldives and the Seychelles is likely being routed to Europe for interconnection, rather than being handled more locally, if the round-trip time is an indication of route taken. The probes in Botswana, Zambia and Tanzania do somewhat better, and are definitely staying on the same continent. The African example is one of the obvious ones. Let’s look at something a bit closer to home…
Regional peering in Northern Ireland and Northern Ireland to the Republic of Ireland
There is already a well established exchange point in Dublin, INEX, with a good number of national and international members. Discussions are taking place between Internet companies in Northern Ireland (which, remember, is part of the UK) about their need for a more local place to exchange traffic, likely in Belfast. The current belief is a large amount of the traffic between sources and sinks in Northern Ireland goes to London or Amsterdam.
Firstly, how does traffic get from the UK (and by inference, most of the rest of Europe) and Northern Ireland? This is what Telegeography say:
Submarine Cables UK to NIRIPE Atlas Probes in Northern Ireland
So, I thought I’d do some RIPE Atlas measurements.
This isn’t meant to be an exhaustive analysis. More just exploring some existing theories and perceptions.
The first trick is to identify probes in Northern Ireland. From the RIPE PoV, these are all indicated as part of the UK (go and watch the video again if you didn’t get it the first time), so I can’t select them by country.
Fortunately, probe owners have to set their probe’s location – there is a certain amount of trust placed in them, there’s nothing stopping me saying my probe is somewhere else, but most probe owners are responsible techy types. The RIPE Atlas people also put the probe locations onto a coverage map.
I also needed some targets. Probes can’t ping each other (well, they can, if you know their IP address, and they’re not behind some NAT or firewall). The Atlas project provides a number of targets, known as “anchors”, as well as nodes in the NLnog ring which can act as targets. There’s an Atlas anchor in Dublin, but that couldn’t take any more measurements, so that wasn’t suitable as a target, but HEAnet (the Irish R&E network) and Amazon (yep, the folks that sell books and whatnot) have NLnog ring nodes in Dublin.
We also needed targets in Northern Ireland that seemed to answer ICMP relatively unmolested, and I chose DNS servers at Tibus and Atlas/Bytel, both of whom are ISPs in the North. The final things to add were “controls”, so I chose a friend’s NLnog ring box which I know is hosted in London, and two other UK-based Atlas probes, the one I have on my network at home, and one on Paul Thornton’s network in Sussex. These effectively provided known UK-Ireland and UK-NI latencies to the targets, and a known NI-London latency for the probes in NI.
So, let’s look at round-trip time from Northern Ireland to the NLnog ring node in London:
ICMP RTT NI Probes to nuqe.net NLnog ring server
So, we can see there are some variations, no doubt based on last mile access technology. In particular, the node shown here with the 54ms RTT (just North of Belfast) consistently scored a high RTT to all test destinations. Anyway, this gives us an idea of NI-London RTT. The fastest being 15ms.
We can therefore make a reasonable assumption that if traffic were to go from Belfast to London and back to Ireland again, a 30ms RTT would be the best one could expect.
(For the interested, the two “control” test probes in the UK had latencies of 5ms and 8ms to the London target.)
Now, take a look at the RTT from Northern Ireland to the node at HEAnet in Dublin:
ICMP RTT all NI probes to HEAnet NLnog ring node, Dublin
Only two of the probes in Northern Ireland have <10ms RTTs to the target in Dublin. All other probes have a greater RTT.
It is not unreasonable to assume, given that some have a >30ms RTT, or have exhibited a >15ms gain in RTT between the RTT to London and the RTT to Dublin, that this traffic is routing via London.
Of the two probes which show a <10ms RTT to HEAnet in Dublin, their upstream networks (AS43599 and AS31641) are directly connected to INEX.
Of the others, some of the host ASNs are connected to INEX, but the RTT suggests an indirect routing, possibly via the UK mainland.
The tests were also run against another target in Dublin, on the Amazon network, and show broadly similar results:
ICMP RTT all NI probes to Amazon NLnog ring node, Dublin
Again, the same two probes show <10ms RTT to Dublin. All others show >30ms. Doesn’t seem to matter if you’re a commercial or an academic network.
Finally, lets look at round trip times within Northern Ireland.
Here’s the test to a nameserver on the Tibus network:
ICMP RTT all NI Probes to Tibus Nameserver
Again, the same two probes report a lower than <10ms latency. I’d surmise that these are either routing via INEX, both host networks are downstream of the same transit provider in Belfast, or are privately interconnected in Belfast. At least two of the other nodes seem to route via the UK mainland.
To check this result, the same tests performed toward a nameserver on the Atlas/Bytel network:
ICMP RTT all NI probes to Atlas/Bytel Nameserver
Obviously, one of our probes is on-net, with a 1ms RTT!
Of the others, we’re definitely looking at “trombone routing” of the traffic, in most cases back to the UK mainland.
This may not be entirely surprising, as I’m told that BT don’t provide a 21CN interconnect node in Northern Ireland, so traffic on BT wholesale access products will “trombone” through the mainland in any case.
So, what’s really needed in Northern Ireland?
We’ve shown that if networks are willing to buy capacity to Dublin, they can happily exchange traffic at INEX and keep the latency down. An obvious concern some may have is the export of traffic from one jurisdiction to another, especially in light of recent revelations about systemic monitoring, if it’s NI to NI traffic.
The utility of IX in Northern Ireland could be hampered due to the lack of BT 21CN interconnect capability, as it may as well, for all intents and purposes be in Glasgow which is the nearest interconnect, for the traffic will still be making two trips across the Irish Sea whatever happens, assuming one end or the other is on the end of a BT wholesale pipe. (At worst, it could be 4 trips if both ends are on a BT pipe!)
If the goal is to foster internet growth (e.g. “production” of bandwidth) in Northern Ireland, where is it going to come from?
Are Northern Irish interests better served by connecting to the mature interconnect community in Dublin?
Is a BT 21CN interconnect in Belfast essential for growth, or can NI operators build around it?
Should INEX put a switch in Belfast? If they do, should it be backhauled to the larger community in Dublin? Or is that somehow overstepping the remit of an exchange point?
A few weeks ago, I wondered why a number of posts on my blog which had been quiet for a while saw some renewed interest – the series on regional peering suddenly saw a significant growth in readership – when I received word that there was group forming in Manchester to discuss the subject, instigated by Manchester co-lo operator m247 and involving (my former employer) the largest UK IXP, LINX.
I attended by all accounts a very successful first open meeting for the IXLeeds exchange point yesterday – with around 120 attendees, including many faces that are not regulars on the peering circuit making for brilliant networking opportunities and great talks from the likes of the Government super-fast broadband initiative, BDUK, and energy efficient processor giants ARM (behind the technology at the heart of most of the World’s smartphones), as well as more familiar faces such as RIPE NCC and LINX, among others.
Definitely impressed with the frank discussion that followed the talk by the DCMS’ Robert Ling on BDUK funding and framework, but still sceptical that it’s going to be any easier for smaller businesses to successfully get access to the public purse.
Andy Davidson, IXLeeds Director, was able to proudly announce that IXLeeds now provides support for jumbo frames via a seperate vlan overlaid on their switch, which is probably the only IXP in the UK which officially offers and promotes this service – at least for the time being. Of course, they are supporting a 9k frame size…
Well done to my friends and colleagues of IXLeeds for making it to this major milestone, and doing it in great style. It seems a long, long way from a discussion over some pizza in 2008.
The only thing I didn’t manage to do while in Leeds is take a look at the progress on the next phase of aql’s Salem Church data centre, but I’m sure I’ll just have to ask nicely and drop by aql at some point in the future. 🙂
At the end of the last post, I vaguely threatened that at some point I’d go on to discuss IX Participant Connectivity.
The topic got a “bump” to the front of the queue last week, thanks to a presentation from Kurtis Lindqvist, CEO of Sweden’s Netnod exchange points, given at the RIPE 63 meeting in Vienna.
Netnod have been facing a dilemma. They provide a highly resilient peering service for Sweden, consisting of multiple discrete exchanges in various Swedish cities, with the biggest being in Stockholm – where they operate two physically seperate, redundant exchanges. They currently provide all their participants in Stockholm with the facility to connect to both fabrics, so they can benefit from the redundancy this provides. Sounds great doesn’t it? If one platform in Stockholm goes down, the other is up, traffic keeps flowing. Continue reading “Whither (UK) Regional Peering – Pt 3”
Leeds-based Internet and Telephony Services company aql have announced they are part of a consortium who wants to redevelop part of the site, to include more new co-location space, complementing their nearby redevelopment of the historic Salem Church, another Leeds landmark being saved from dereliction.
This also good news for the rapidly developing Leeds-based IXP – IXLeeds, who’s switch is co-located at the aql Salem Church facility. It opens up further access to co-location for the future, and further promotes technology growth in the region.
Old brewery buildings make good bases for something such as co-location, due to the buildings being engineered for high floor loadings. Part of the old Truman Brewery site on London’s Brick Lane was reborn as a datacentre some years ago, so there’s a sound precedent for this part of the redvelopment.
Adam Beaumont, aql founder, said that he’s “always looking for new ways to combine his interests of technology and beer” :).
This new plan deserves to go ahead for a number of reasons, and not only because it is significantly better than Carlsberg’s original proposal: To build a car park, locally dubbed as “Probably the most unadventurous redevelopment plan in the world“. Hilarious.
It’s been a long while since I’ve blogged about this topic
Probably too long, as IXLeeds, something which inspired me to write Pt 1, is now a fully-fledged IX, not just a couple of networks plugged into a switch in a co-lo (all IXPs have to start somewhere!), but has formed a company, with directors, with about 12 active participants connected to its switch. Hurrah!
So, trying to pick up where I left off; in this post, I’m going to talk about shared fate, with respect to Internet Exchanges.
Just last month, in mid-September, Andy Davidson brought up the switch at IXLeeds, the latest UK regional IXP.
You’ll note I say “the latest”, but how many non-London UK IXPs can you name off the top of your head? Not many, I’ll wager. Fewer that are still operating, too. No, the LINX PoP in Slough doesn’t count in my picture of non-London!
This is the problem: It’s often said that there isn’t the level of regional IP peering going on in the UK that there probably should be for redundancy reasons. The majority of IP peering in the UK happens in London, and when it isn’t happening in London, it’s probably happening in Amsterdam instead.
Let’s face it, on an island that’s ~15ms round-trip top-to-bottom, we’re less likely to peer to reduce latency, especially when the architecture of the incumbent wholesale DSL platform doesn’t encourage networks to do little beside haul all broadband customer traffic to a central point before dispersing it.
Previous attempts at establishing regional IXPs in the UK have had varying levels of success. The most successful to date, in terms of number of participants and achieving critical mass is probably MANAP – which was founded in Manchester in 1997.
Unlike LINX which did survive (well, successfully resisted) a demutualisation attempt, MANAP only sort-of did. It allowed it’s infrastructure to be taken over by a company funded by the local Regional Development Association, and the exchange became a service provided over the infrastructure which was no longer dedicated to IXP operations, but also carried other traffic and provided other services.
The MANAP that exists today is not the same exchange, it has been subsumed into the NWIX platform and operates as Edge-IX, a distributed exchange which is present in both Manchester, elsewhere in the Northwest, and in many other locations, including those in London’s Docklands that it was initially intended to provide redundancy for. It’s has a different flavour, and has lost some elements of it’s “regionality”.
What distinguishes it from a carrier, other than the Edge-IX services being non-profit, while the NWIX ones are?
I’m not suggesting that this is a better or worse model, just different, and probably not regional anymore. If this, i.e. reinventing yourself as an inter-regional IX, is the only way a regional IXP in the UK can survive, then we’ll find it very challenging to reach position of sustainable regional peering in the UK. Could things have been different in Manchester?
You may be questioning what issue I have with a “wide-area” exchange point, distributed over a large geographic area? The main concern is shared fate. A disruption that would otherwise be localised, spreading easily. I can probably write a whole article on that. Maybe I will another day…
So, why would a quick hop over the Pennines to Leeds be any different?
Manchester itself is also at risk of being unattractive as a location for regional IXs – with the recent purchase of IFL by Telecity Group there is very little organisational diversity or competition in the Manchester co-lo market – there’s existing facilities such as Vialtus Serverbank, and recent new entrant Ice Colo. Folks in the Manchester area were very quick on the social networks to state their fears about anticipated price rises and few options as a result of the lack of choice.
The Leeds scene is rather different, with lots of smaller, entrepreneurial companies active in the metro area. This is a double-edged sword, as while it results in competition in the co-lo market which folks like, it also meant that IXLeeds couldn’t be present everywhere the potential IX participants wanted to connect, certainly from day one. There’s a future aspiration to expand within the metro.
One of the early strengths in IXLeeds is that has a good community feel behind it, including the involvement of folks who have experienced peering in Manchester, while the Yorkshire RDA have been involved from the outset in getting folks together, but (so far) haven’t felt the temptation to get in the driving seat, instead choosing to play the role of facilitator.
There’s a will to succeed, so hopefully they will reach the critical mass that is required to sustain the exchange.
A concern I have is the lack of international capacity into the Leeds area, Manchester is in a better position here due to the independant (from London) Transatlantic connectivity arriving in the area.
That said, while international bandwidth a something of a pre-requisite for a national exchange point, is that actually necessary for a successful regional IX?
Then again – what are the success criteria for an IX, especially a regional one? A graph that forever goes up and to the right? They probably are and should be different from a national IX. Is the regional IX not being satisfied with it’s lot, and wanting to be like it’s larger neighbour, what actually destroys it? Maybe that’s another article in itself?
I’d say it depends on how non-London-centric the early IXLeeds ISPs are, how much of their traffic is delivered locally, and how much traffic they have between each other that they might normally route through London.
If my previous experience is anything to go by – such as opening a new PoP for an already successful IX – these things usually “slow start” – so that means patience is required.
I’m going to come back to this topic in the coming weeks, I’ll try and write about some of the side issues I’ve threatened to cover above, and maybe touch on a missed opportunity.
Still think they should have called it the Rhubarb Internet Exchange. Even if it was just to confuse people. 🙂
At the end of the 
Mike's Twitter Overflow - Tech, Travel, other stuff 