Category: Network Operations

The Importance of Transparent Internet Access

Those of you following the UK tech press, or are affected Virgin Media customers, will be aware of an issue that had been affecting some VM users’ access to the Internet.

There was no apparent rhyme or reason to the websites which failed, and in some cases, the site itself may have been working, but made very slow because other collateral hosted on third-party sites (e.g. performance measurement and marketing tools) were unreachable, or very slow.

One of the most memorable articles is the one which contained the comment “The people in the call centre are extremely dumb and it’s like talking to a tree.” (ISP Review).

Much speculation has been directed at some new or changed traffic management, traffic shaping, filtering, or deep-packet inspection (DPI) going awry inside Virgin Media’s network. It’s well known that Virgin Media apply traffic management in their network, such as “clamping” the bandwidth available to super-heavy users who use more than what VM consider a fair share of the bandwidth.

The concern many (especially the various public rights’ groups) have is that the desire some authorities have to increase the amount of monitoring, blocking access to “undesirable sites”, and logging and retaining things such as email conversations, will only serve to increase the amount of unusual, irregular, and hard to trace, service problems such as these.

One thing to bear in mind is that the technology being used in DPI is still an evolving science. This means it has warts and all. I’ve seen DPI devices mangle packets in transit – including packets which shouldn’t have been touched by the DPI, but allowed to pass unhindered – so badly that they were undeliverable to their intended destination.

It seems likely that this is what’s happened here, so it’s not a load of arm-waving about a hollow concern that’s being raised by those who don’t believe in DPI. There’s a real threat here – of unreliability and incorrectly filtered traffic – to legitimate Internet use.

Which brings me on to every cloud having a silver lining, as they say.

In this case, privately owned North West-based provider Zen Internet decided it was time to highlight the Zen approach to Traffic Management – No Throttling, No Squeezing – issuing a news release explaining how they operate a transparent network, with no DPI, and an open, fair and easy to understand pricing policy for internet access, with no complex rules or hidden gotchas.

Good for them.

Disclosure: I am a (happy) Zen Internet customer, they keep my folks’ home online, and do a very good job of it. It just works. I’m also potentially moving to an area where it seems the only high-speed broadband available might be Virgin Media. I spent about half-an-hour trying to work out how their obtuse and opaque pricing structure worked and which was the right “bundle” for me before giving up and hitting the bottle. I’d rather know that what I’m paying for is reliable and unfettered, if slower.

DR still in the doldrums – An Open Letter to Digital Region

A few months ago, I wrote about what I percieved to be going wrong with Digital Region, the local-authority backed superfast broadband wholesale network in South Yorkshire.

It seems that matters have not improved since then: a Sheffield-based hosting company, KDA, has written an Open Letter to Digital Region, which pretty much confirms that everything which was true several months ago is still true today, and goes on to suggest that there’s enough experience and skill in the tech community in South Yorkshire to turn this around, if only those in charge were willing (able?) to change tack and allow the community to steer the organisation.

It’s also alluded that a cut-price disposal of the network assets, which should rightly be the South Yorkshire taxpayer’s, for a cut-price may already be in hand, and that a failure of DR will be associated generally with the South Yorkshire tech industry, tarring it’s (generally good) reputation.

DR shouldn’t be the way it is – DR should be more agile than the large telcos, and find it easier to be more focused on the needs of the local userbase, but it isn’t. It seems to be strangled by inflexibility and bureaucratic behaviour, which needs to change if it’s to survive, and deliver the promise that the local authorities set out to achieve. But, at the moment, I’m doubtful that this will happen. The peppercorn sell-off probably feels like an easy way out, however much it’s short-changing South Yorks residents and business in the process.

You can read the full text of the Open Letter here.

When is it (not) a good time to do maintenance?

With the global nature of the Internet and globalisation of businesses, there’s never really a good time to do maintenance. When it’s 1am in London, it’s 5pm in Silicon Valley and people are trying to wrap up their work day, and it’s first thing in the morning in Hong Kong, neither are going to be happy if they have a maintenance outage to deal with at such important times of the day.

So, you choose your disruptive maintenance windows carefully, to try and cause the smallest impact that you can.

However, if you know the users of the system are local, it’s much easier to choose your maintenance windows: usually when there are the least users on the system.

Try telling that to Transport for London.

This is the front end to TfL’s “Countdown” system. It tells you which buses are due at a given bus stop and an approximate time that they arrive. The countdown database is updated using location equipment on the buses, and drives LED displays at bus stops, and is accessible over the Internet, including a user interface designed for mobiles, and via SMS short code.

It’s especially useful when services on a route are infrequent, such as on a Sunday, where you may be looking at waiting up to 20 minutes for a bus if you managed to just miss the previous one. So, look back at the screen grab above, note the start time for the maintenance window.

Why do TfL think it’s a great idea to take the system down right at the time on a Sunday that people are heading out to visit family, maybe go out for Sunday lunch, or head to sporting events?

Wouldn’t a better time be the middle of the night on a Monday, when things are much quieter, with fewer users?

Paying techs extra to do system maintenance on a Sunday can’t be cheap either?

End of the line for buffers?

This could get confusing…The End of the Line by Alan Walker, licenced for reuse under the Creative Commons License

buffer (noun)

1. something or someone that helps protect from harm,

2. the protective metal parts at the front and back of a train or at the end of a track, that reduce damage if the train hits something,

3. a chemical that keeps a liquid from becoming more or less acidic

…or in this case, none of the above, because I’m referring to network buffers.

For a long time, larger and larger network buffers have been creeping into network equipment, with many equipment vendors telling us big buffers are necessary (and charging us handsomely for them), and various queue management strategies being developed over the years.

Now, with storage networks and clustering moving from specialised platforms such as fibre channel and infiniband to using ethernet and IP, and the transition to distributed clustering (aka “The cloud”), this wisdom is being challenged, not just by researchers, but operators and even equipment manufacturers.

The fact is, in lots of cases, it can often be better to let the higher level protocols in the end stations deal with network congestion rather than introduce variable congestion due to deep buffering and badly configured queueing in the network which attempts to mask the problem and confuses the congestion control behaviours built into the protocols.

So, it was interesting to find two articles in the same day with this as one of the themes.

Firstly, I was at the UKNOF meeting in London, where one of the talks was from a research working on the BufferBloat project, which is approaching the problem from the CPE end – looking at the affordable mass-market home router, or more specifically the software which runs on them and the buffer management therein.

Second thing I came across was a great blog post from technology blogger Greg Ferro’s Ethereal Mind – on his visit to innovative UK ethernet fabric switch startup Gnodal – who are doing some seriously cool stuff which removes buffer bloat from the network (as well as some other original ethernet fabric tech), which is really important for the data centre networking market with it’s latency and jitter sensitivity, which Gnodal are aiming at.

(Yes, I’m as excited as Greg about what the Gnodal guys are up to, as it’s really breaking the mould, and being developed in the UK, of course I’m likely to be a bit biased!)

Is the writing on the wall for super-deep buffers?

Regional Broadband – The Hidden Danger of Uber-projects

It was revealed this week that Digital Region, the centrally funded (to the tune of £90m – mostly public money) superfast broadband initiative in South Yorkshire is facing tough times, in particular a £9.2m loss on a revenue of only £167k (which only just pays the last CEO’s £100k salary – they are currently seeking a new CEO, one assumes to manage a turnaround).

The Yorkshire Post article goes on to explain another £4m of public funds have been ringfenced as a “security”, and that the four participating councils, already under budget pressure from Central Government austerity, may need to as much as £500k per year to secure the operations of Digital Region if the loan can’t be repaid. Is that throwing good money after bad, or is the situation redeemable?

This highlights my belief that these large centrally funded uber-projects contain a more significant risk of failure, and not of delivering the right product. The larger organisations that are able to bid and win such projects can come with higher overheads compared to the smaller community projects such as those serving areas with poor existing broadband service, who have a relatively captive and supportive market, and benefit from a tighter focus – for instance Rutland Telecom’s pioneering FTTC with unbundled sub-loop in Lyddington, which is using the same basic FTTC tech as DR is using, but on a smaller scale and in relative isolation.

The larger scale of the Digital Region deployment obviously needed a much bigger income to support the aggressive build and provisioning costs, along with what looks like a complex structure, and now that revenue hasn’t been realised. As can be seen on the DR website, and highlighted on the ThinkBroadband article, very few ISPs use the DR infrastructure to deliver service and is maybe one of the reasons they aren’t making their targets.

You have to ask yourself why this is? Continue reading “Regional Broadband – The Hidden Danger of Uber-projects”

Please accept new prefixes XYZ behind ASfoo – make it stop!

Those of you who ran networks in the 1990s (possibly even in the early 2000s) will remember the excitement you had joining your first Internet exchange, plugging in that shiny new cable to your router interface, and setting up your first peerings.

Back then, you may also remember that in the rapidly growing Internet of the day, it was common courtesy to let your peers know that you’ve taken on a new customer, or acquired some new address space, so they could update their configs – particularly any filtering they were doing on the routes exchanged with you, which were often quite small and maintained manually, except for the largest providers.

Your message would go something like this:

Continue reading “Please accept new prefixes XYZ behind ASfoo – make it stop!”

SOPA/PIPA Roundup

I’ve sort of wanted to write things about the frankly worrying SOPA bill in the US Senate and PIPA bill going through the US House of Representatives at the moment, but the fact is, others are doing a perfectly good job writing about it elsewhere, and why the hell should I waste even more precious bits repeating the good stuff they have already said.

So, I’ll quickly roll-up what I think are interesting articles:

I’ll add more as I find/read them and think they are worth linking to. There are a lot of articles and opinions out there, as you can imagine, and I’m now just adding to the melee, I suppose.

But, the most worrying thing I find is that what is being proposed is effectively the same type of DNS doctoring and blackholing that other “less liberal” Governments (China, for instance) have been known to use to block access to things like Facebook, Twitter and YouTube.

“Oh, but we’ll only use it for blocking X”, they say. Question is, does the existance of the mechanism to do this constitute an invitation for it to be used for blocking other things in the fullness of time? Are we going to end up with domains being injected into the feed of “bad things” because it hosts something that arbitrarily earned some sort of “dislike” from those who have control?

Paging George Orwell, to a courtesy telephone, please.

Broadband Blindness in North America

B4RN‘s Chris Conder tweeted this interesting ~30 minute video from a producer in Sacramento, CA, on the limitations experienced on broadband in the US, and how the large telcos appear to be failing rural communities, and that deployments of their fastest products tend to be only available in “boutique” (usually high income) areas.

It highlighted how the large telcos found it hard to invest in deploying high speed broadband to sparse communities because of the conflict between affordably delivering a service and paying a shareholder dividend.

The video also spoke to some local community broadband companies in the US who, like B4RN are going their own way, and investing in their own future.

Good quality internet access is starting to become as essential to modern life as a stable electrical supply or safe, drinkable tap water. It’s becoming more of a utility and less of a consumer product.

BGP Convergence with Jumbo Frames

This is something of a follow up to Breaking the IXP MTU Egg is no Chicken’s game…

One of the reasons for adoption that was doing the rounds in the wake of Martin Levy’s internet draft on the topic of enabling jumbo frames across shared media IXPs is that using jumbos will help speed up BGP convergence during startup. The rationale here is that session set up and bulk update exchange will happen more quickly over a jumbo capable transport.

Something about this didn’t sit right in my mind, it seemed like a red herring to me. A tail wagging the dog, so to speak. The primary reasons for wanting jumbos are already documented in the draft and discussed elsewhere. If using jumbos gave a performance boost during convergence, then it was a nice bonus, but that flew in the face of my experience of convergence – that it’s more likely to be bound by the CPU rather than the speed of the data exchange.

I wondered if any research had been done on this, so I had a quick Google to see what was out there.

No formal research on the first page of hits, but some useful (if a few years old) archive material from the Cisco-NSP mailing list, particularly this message…

Spent some time recently trying to tune BGP to get
convergence down as far as possible. Noticed some peculiar
behavior.

I'm running 12.0.28S on GSR12404 PRP-2.

Measuring from when the BGP session first opens, the time to
transmit the full (~128K routes) table from one router to
another, across a jumbo-frame (9000-bytes) GigE link, using
4-port ISE line cards (the routers are about 20 miles apart
over dark fiber).

I noticed that the xmit time decreases from ~ 35 seconds
with a 536-byte MSS to ~ 22 seconds with a 2500-byte MSS.
From there, stays about the same, until I get to 4000,
when it beings increasing dramatically until at 8636 bytes it
takes over 2 minutes.

I had expected that larger frames would decrease the BGP
converence time. Why would the convergence time increase
(and so significantly) as the MSS increases?

Is there some tuning tweak I'm missing here?

Pete.

While not massively scientific, this does seem like a reasonable strawman test of the router architecture of the day (2004), and got this reply from Tony Li:

How are your huge processor buffers set up?

I would not expect a larger MTU/MSS to have much of an
effect, if at all.  BGP is typically not constrained by
throughput.  In fact, what you may be seeing is that
with really large MTUs and without a bigger TCP window,
you're turning TCP into a stop and wait protocol.

Tony

This certainly confirmed my suspicion that BGP convergence performance is not constrained by throughput but by other factors, and primarily by processing power.

Maybe there are some modest gains in convergence time to be had, but there is a danger that loss of a single frame of routing information data (due to some sort of packet loss, maybe a congested link, or a queue drop somewhere in a shallow buffer) could cause retransmits sufficiently damaging to slow reconvergence.

It somewhat indicates that performance gains in BGP convergance are marginal and “nice to have”, rather than a compelling argument to deploy jumbos on your shared fabric. The primary arguments are far more convincing, and my opinion is that we shouldn’t allow a fringe benefit (that may even have it’s downsides) such as this to cloud the main reasoning.

It does seem like some more up-to-date research is necessary to accompany the internet-draft, maybe even considering how other applications (beside BGP) handle packet drops and data retransmits in a jumbo framed environment? Does it reach a point where application performance is being impacted because a big lump of data got retransmitted.

Possibly, there is some expertise to be had from the R&E community which have been using jumbo capable transport for a number of years.?

Breaking the IXP MTU Egg is no Chicken’s game

Networks have a thing called a Maximum Transmission Unit (MTU), and on many networks it’s long been somewhere around 1500 bytes, the default MTU of that pervasive protocol, Ethernet.

Why might you want a larger MTU? For a long time the main reason was if you’re transferring very large amounts of data, you reduced the framing and encapsulation overheads. More recent reasons for wanting a larger MTU include being able to accommodate additional encapsulations (such as MPLS/VPLS) in the network without reducing the end-to-end MTU of the service, to carry protocols which by default have a higher MTU (such as FCoE, which defaults to ~2.5k bytes), and make things like iSCSI more efficient.

There’s often been discussions about whether Ethernet-based Internet Exchange Points – the places where networks meet and interconnect over a shared fabric – have been one of the barriers (there are lots!) to adoption of a higher MTU in the network. Most are based on Ethernet, and most have a standard Ethernet MTU of ~1500 bytes.

Ethernet can carry larger frames, up to around the 9k byte mark in most cases. These are known as “Jumbo Frames“. Here is quite a nice article about the ups and downs of jumbo frame support from the perspective of doing it on your home network.

The inter-provider networks to date where you can usually depend on having a higher end-to-end MTU are the large Research and Educational Networks, such as JANET in the UK. With white-coated (and occasionally bearded) scientists wanting to move huge volumes of experimental data around the world, they’ve long needed and have been getting the benefits of a larger MTU. They deliberately interconnect their networks directly to ensure the MTU isn’t reduced by a third-party enroute.

This has recently resurfaced in the form of this Internet Draft submitted by my learned friend Martin Levy of Hurricane Electric – yes, he of “Just deploy IPv6” fame.

With this draft Martin is trying to break what is at best a chicken-and-egg, and at worst a deadlock:

  • Should an IXP support jumbo frames?
  • What should the maximum frame size be?

He’s trying to support his argument by collecting the various pieces of rationale for supporting inter-provider jumbos in one place, to guide IXP communities in making the right decision, and hopefully documenting the pitfalls and things to watch out for – the worst being that your packets go into the bit-bucket because of a MTU mismatch and PMTUD being broken by accident at best or foolish design at worst.

My own personal, most-recent experience dancing around this particular handbag as with an IXP operators hat on, gave the following results:

  • A miniscule (<5%) proportion of IXP participants wanted to exchange jumbo frames across the IXP
  • Of those who wanted jumbo frames, it was not possible to reach consensus on a supported maximum frame size. Some wanted 9k, others only wanted 4470, some wanted different 9k MTUs (9218, 9126, 9000), likely due to limitations of their own equipment and networks.

It was actually easier for this minority to interconnect bi-laterally over private pieces of wire or fibre, where they could also set the MTU for that link on a bi-lateral basis, rather than across a shared fabric where everyone had to agree.

Martin’s rationale is that folk argue about this because there’s no well-known guidance on the subject, so his draft is being proposed to provide just that and break the previous deadlock.

In terms of IXPs which do support a larger MTU today, there are a few, the most well-known probably being Sweden’s Netnod, which has long had an MTU of 4470, largely due to it’s own ancestry of originating on FDDI, and subsequently using Cisco’s proprietary DPT/SRP technology after the exchange outgrew FDDI (largely because of a local preference for maintaining a higher MTU). When Netnod moved to a Gigabit/10 Gigabit Ethernet based exchange fabric, the 4470 MTU was retained despite the newer ethernet hardware having support for a ~9k MTU, and it’s explicitly required by Netnod that IXP participant interfaces are configured with a 4470 MTU to avoid mismatches. It seems to be working pretty well.

One of the issues which is likely to cause discussion is where 100Mb Ethernet is deployed at an exchange, as this, generally speaking, cannot support jumbo frames. Does this create a “second class” exchange in some way?

Anyway, I applaud Martin for trying to take this slippery subject head-on. Looking forward to seeing where it goes.