Beware the NTP “false-ticker” – or do the time warp again…

For the uninitiated, it’s easy to keep the clocks of computers on the Internet in synch using a protocol called NTP (Network Time Protocol).

Why might you want to do this? It’s very helpful in a large network to know that all your gear is synchronised to the same time, so that things such as transactional and logging information has the correct timestamps. It’s a must have for when you’re debugging and trying to get to the bottom of a problem.

There was an incident earlier this week where the two open NTP servers run by the US Naval Observatory (the “authority” for time within the US) both managed to give out incorrect time – there are reports of computers which synchronised against these (and more importantly, only these, or one or two other systems) had their clocks reset to 2000. The error then corrected, and clocks got put back.

Because the affected systems were chiming either only against the affected master clocks, or a limited number of others, the two incorrect times, but from a high stratum source, were taken as being correct and the affected systems had their local clocks reset.

There’s been discussion about the incident on the NANOG list…

Continue reading “Beware the NTP “false-ticker” – or do the time warp again…”

WCIT – The spinning eye on the middle of the Perfect Storm

With less than two weeks to the WCIT (World Conference on International Telecommunications), and what that means for the internet (see “Is the Internet Facing a Perfect Storm”), the amount of coverage online is increasing rapidly.

Basically in one sentence, some Governments would like to see the ITU (a closed, top-down, Governmental organisation) take the lead in regulating the (open, collaborative, co-operative, bottom-up) Internet.

This video from the folks at accessnow.org explains it pretty well…

Lots of people I know have been working seriously hard in the background to educate those with a vote in the ITU, and keep the Internet based around open standards and collaborative governance.

Let’s face it, if the Internet wasn’t based around open standards, so many of the things which are part of our every day lives could be very different, or simply wouldn’t even exist.

Fortunately, as far as we know in the UK, our Government is still an advocate of the “light touch” and industry self-regulation when it comes to the Internet, but that doesn’t make what’s going to go on behind closed doors in Dubai in a couple of weeks any less of a concern.

Unsensationalist SDN

SDN – Software Defined Networking – has become one of the trendy Silicon Valley buzzwords. Often abused and misused, it seems that all the marketing folks are looking at ways of passing what they already have on their product roadmaps as being “SDN”, while startups are hoping that it will bring in the latest tranche of VC money. Some folk would even have you believe that SDN can cook your toast, make your coffee and iron your shirts.

People are talking about massively centrally orchestrated networks, a move away from the distributed intelligence which on the wider Internet is one of it’s strong points. Yes, you could have a massive network with a single brain, and very centralised policy control, if that’s what you wanted. For some applications, it’s what you need – for instance in Google’s internal networks. But for others you may really need things to be more distributed. The way some people are talking, they make it sound like SDN is an “all or nothing” proposal.

But, is there a place for SDN between distributed and centralised intelligence that people are missing in the hype?

Put all the excitement to one side for a minute, and we’ll go back to basics.

The majority of large networking equipment today usually already has a distributed architecture, such that when a packet of data arrives that the receiving interface doesn’t know what to do with, it’s forwarded from the high-speed forwarding silicon to software running in a CPU. In some devices, there are multiple CPUs, often arranged in a hierarchy with a “master” CPU in charge of the overall system, and local CPUs delegated to run elements of the system, such as logical or physical groups of interfaces, or specific processes in the system.

When the forwarding hardware sends a packet to the CPU, to get a decision on what to do with it, it’s commonly known as a “CPU punt”.

The software running on the CPU examines the packet based on the configuration of the device, compares it against it’s internal tables and any configured policy, and makes a decision about what to do with it.

It sends the packet back to the forwarding engine, along with an instruction of how to program the forwarding hardware so that it knows what to do with other packets with the same properties (i.e. from the same data stream, or with the same destination address, etc.), whether that be to apply policy, forward it, drop it, etc.

This isn’t that different from how OpenFlow works, but the CPU is abstracted from the device in the data path and resides in a thing known as a “Controller”. Effectively, what were CPU punts now become messages to the OpenFlow Controller. The Controller, by nature of it residing on a more generalised computer, is capable of doing things that are less likely to be easily doable on the software running in a router or switch, in terms of making “special” decisions.

Basically, SDN in some respects looks like the next step in something we’ve been doing for years. It’s an evolution, does it need to be a revolution?

So, here’s where I think what’s currently a “missed trick” lies…

The forwarding hardware used in SDN doesn’t have to or need to be totally dumb. Some decisions you might be happy for it to make for itself. It could pick up details of what those are from the policy in the Controller, and these can be written to the local CPU and the local forwarding silicon.

But, other things you know you do want to “punt” to the Controller, get set up (through applied policy) and handled that way.

I can think of occasions in the past where I would have loved to be able to take the stream of CPU punts in a device that can otherwise happily make a lot of it’s own decisions and be able to farm them out for analysing and processing in such a way which wasn’t possible on the device, and convert this back to policy, config, and ultimately CAM programming. But, to be able to do this without basically lobotomising the device?

Does SDN have to be the “all or nothing” approach which seems to be what’s getting proposed by some of the SDN evangelists? Or is a hybrid approach more realistic and aligned with how we actually build networks?

“A Grandmother dies after feeding her daughters pets”: or the importance of punctuation

Just getting food together this evening, and from the kitchen, this was the headline I heard on the 6.30pm BBC London News:

“A Grandmother dies after feeding her daughters pets”

The Mk1 ear picked this up, for my (admittedly random) brain to parse it as “A Grandmother has died, having fed pets to her daughters”.

Actually, what’s happened is a woman has died having been mauled by her daughter’s dogs in South London.

Maybe it’s just Halloween that’s making me think of this sentence as a one-line plot synopsis for an episode of Tales of the Unexpected – a woman covertly feeds her pets to her daughters, and meets some sticky end with a twist in the tail.

Sorry to go all Eats Shoots and Leaves on you… but the apostrophe that gave this statement the proper meaning vanished once it had become the spoken word.

UK 4G LTE Launch and the scramble for spectrum

So, the next path on the road to fast mobile data, 4G LTE finally launches in the UK, after much barracking from competitors, on the “EE” network (the combined Orange and T-Mobile brand).

It’s only available in a handful of markets at the moment, and the BBC’s tech correspondent, Rory Cellan-Jones, did many articles for TV and Radio yesterday, while conducting countless speedtests, which he has extensively blogged about.

Some of the comments have been that it’s no better in terms of speed than a good 3G service in some circumstances, while others complain about the monthly cost of the contracts.

Get locked-in to 4G(EE)?

The initial cost for the early adopters was always going to attract a premium, and those who want it will be prepared to pay for it. It’s also worth noting that there are no “all you can eat” data plans offered on EE’s 4G service. Everything comes with an allowance, and anything above that has to be bought as “extra”.

The most concerning thing as far as the commercial package goes are the minimum contract terms.

12 months appears to be the absolute minimum (SIM only), while 18 months seems to be the offering if you need a device (be it a phone, dongle or MiFi), and 24 month contracts are also being offered.

Pay As You Go is not being offered on EE’s 4G service (as yet), probably because they’ve no incentive to, because there’s no competition.

Are EE trying to make the most of the headstart they have over competitors 3, O2 and Voda and capture those early adopters?

Penetrating matters?

Rory Cellan-Jones referred in his blog about problems with reduced performance when in buildings.

A number of factors affect the propagation of radio waves and how well they penetrate buildings and other obstacles, such as the nature of the building’s construction (for instance a building which exhibits the properties of a Faraday Cage would block radio signals, or attenuate them to the point of being useless), but also the frequency of the radio signal.

Longer wavelengths (lower frequencies) can travel further and are less impacted by having to pass through walls. I’m sure there’s an xkcd on this somewhere, but the best I can find is this….

Electromagnetic Spectrum according to xkcd

The reason EE were able to get a steal on the other mobile phone companies was because OFCOM (the UK regulator, who handle radio spectrum licensing for the Nation) allowed EE to “refarm” (repurpose) some of their existing allocated frequency, previously used for 2G (GSM), and convert it to support 4G. The 2G spectrum available to EE was in the 1800 Mhz range, as that was the 2G spectrum allocated to EE’s constituent companies, Orange and T-Mobile.

Now, 1800 does penetrate buildings, but not as well as the 900 Mhz which are the 2G spectrum allocated to Voda and O2.

Voda are apparently applying to OFCOM for authority to refarm their 900 Mhz spectrum for 4G LTE. Now, this would give a 4G service which had good propagation properties (i.e. travel further from the mast) and better building penetration. Glossing over (non-)availability of devices which talk LTE in the 900 Mhz spectrum, could actually be good for extra-urban/semi-rural areas which are broadband not-spots?

Well, yes, but it might cause problems in dense urban areas where the device density is so high it’s necessary to have a large number of small cells, in order to limit the number of devices associated with a single cell to a manageable amount – each cell can only deal with a finite number of client devices. This is already the case in places suce as city centres, music venues and the like.

Ideally, a single network would have a situation whereby you have a high density of smaller cells (micro- and femto-cells) running on the higher frequency range to intentially limit  (and therefore number of connected devices) it’s reach in very dense urban areas such as city centres, and a lower density of large cells (known as macro-cells) running on lower frequencies to cover less built-up areas and possibly better manage building penetration.

But, that doesn’t fit with our current model of how spectrum is licensed in the UK (and much of the rest of the world).

Could the system of spectrum allocation and use be changed?

One option could be for the mobile operators to all get together and agree to co-operate, effectively exchanging bits of spectrum so that they have the most appropriate bit of spectrum allocated to each base station. But this would involve fierce competitors to get to together and agree, so there would have to be something in it for them, the best incentive being cost savings. This is happening to a limited extent now.

The more drastic approach could be for OFCOM to decouple the operation of base stations (aka cell towers) from the provision of service – effectively moving the radio part of the service to a wholesale model. Right now, providing the consumer service is tightly coupled to building and operating the radio infrastructure, the notable exception being the MVNOs such as Virgin (among others), who don’t own any radio infrastructure, but sell a service provided over one of the main four.

It wouldn’t affect who the man in the street buys his phone service from – it could even increase consumer choice by allowing further new entrants into the market, beyond the MVNO model – but it could result in better use of spectrum which is, after all, a finite resource.

Either model could ensure that regardless of who is providing the consumer with service, the most appropriate bit of radio spectrum is used to service them, depending on where they are and which base stations their device can associate with.

DSL Diary – 23/10/2012

Latest instalment…

Currently away at the NANOG meeting in Dallas. Got an alert from the RIPE Atlas system that my Atlas probe had become unreachable.

Bit of testing from the outside world showed serious packet loss, and nothing on the home network was actually reachable with anything other than very small pings. I’d guessed the line had got into one of it’s seriously errored modes again, but thought I’d try leaving it overnight to see if it cleared itself up. Which it didn’t.

So, how did I get around this, and reset the line, given that by now my tolerant girlfriend would be at work, and couldn’t go into the “internet cupboard” and unplug some wires?

Well, turns out that you get BT to do an invasive test on a line using this tool on bt.com. This has the effect of dropping any calls on the line and resetting.

The line re-negotiated, and came back up with the same speed as before, 3Mb/sec down, 0.45Mb/sec up, no interleave.

Looking at the router log, the VirtualAccess interface state was bouncing up and down during the errored period, so the errors are bad enough to make the PPP session fail and restart (again and again), but the physical layer wasn’t picking this up and renegotiating.

Of course, BT’s test says “No fault found”. In terms of the weather in London, it has been damp and foggy, further fuelling the dry joint theory.

I’ve also had a chat with Mirjam Kuehne from RIPE Labs about seeing if it’s possible to make the Atlas probe’s hardware uptime visible, as well as the “reachability-based” uptime metric. They are looking in to it.

BT and Virgin Media challenge Birmingham’s Broadband deployment

BBC News are reporting that incumbent high speed broadband providers BT and Virgin Media have launched a legal challenge to Birmingham City Council’s proposed independant Superfast Broadband Network.

The city has successfully applied for EU state aid to build network into underserved areas of the city, aligned with the Council’s regeneration plans for those areas. Virgin contest that it is “overbuilding” on their existing network footprint, and as such is unnecessary, effectively using EU subsidy to attack their revenue stream.

Broadband campaigner Chris Conder, one of the people behind the B4RN project, says that this is a case of VM and BT trying to close the stable door after the horse has bolted.

It’s going to be an interesting and important test case.

First Impressions from “new” Lufthansa “Pier A” at Frankfurt

Connecting through Frankfurt, Lufthansa’s major hub, can sometimes be a fraught experience. Given that the Germans generally have a reputation for efficiency and good design, this may come as a surprise to the unseasoned traveller.

It’s something of a lottery. If you’re connecting between two flights arriving on the same gate area, then it’s easy. If not, then it’s game over. Don’t pack your running shoes – keep them handy in your hand baggage. Continue reading “First Impressions from “new” Lufthansa “Pier A” at Frankfurt”

Meerkats vs. Nectar Points

In a change from normal programming…

There’s a lot of competition between comparison sites in the UK, and the favoured weapon in the arsenal seems to be bonus giveaway: Compare the Market give away toys of the meerkat characters from their advertising, while Confused.com are giving away Nectar points.

So, suspending belief that the primary reason we go to these sites is to get a good deal on some insurance, on monetary terms alone which is the better value? The Meerkat or the Nectar points?

Nectar Points have a rough monetary value of 0.5p each – 500 points = £2.50 is the usual redemption value. Confused.com give away 1000 Nectar points if you buy insurance through their website, so that’s worth a fiver.

Cuddly Meerkats currently have a resale value on Ebay for about £30-40 each, and you claim one when you buy car or home insurance.

Looks like the Meerkats win. Simples.

…we now return you to the usual stuff that Sergei would find interesting.

DSL Diary – 18/10/12

You may have read my post regarding a spell of degradation on my home internet access – Interleaves on the Line?

The other recent addition to my home network is a RIPE Atlas probe – this is part of a large scale internet measurement project being run by the RIPE NCC. One of the advantages of hosting a probe is that you get access to the measurements running from your probe, and you can also get the collection platform to email you if your probe becomes unreachable for a long period.

As it turned out, my probe appeared to be down for half an hour last night, but I know I was using the internet connection just fine at that time, so maybe I’ll put that down to interruption between the probe and the collection apparatus?

Well, the current status is that the line has been up for seven days now, at just over 3Mb/sec, Interleaving off.

Still not the fastest connection, but at least it now seems to be more stable.

One thing I’ll keep my eye open for is if the line goes back to Interleaved, as the Atlas probe should show up the difference in latency that you get with Interleaving enabled.