Opinion – Page 3 – Mike's Twitter Overflow

My recent talk at INEX – Video

Or, I never thought of myself as a narcissist but…

Thanks to the folks at HEAnet, here’s a link to the video of the talk “It’s peering, Jim…” that I gave at the recent INEX meeting in Dublin, where I discuss topics such as changes in the US peering community thanks to Open-IX and try to untangle what people mean when they say “Regional Peering”.

The talk lasts around 20-25 minutes and I was really pleased to get around 15 minutes of questions at the end of it.

I also provide some fairly pragmatic advice to those seeking to start an IX in Northern Ireland during the questions. 🙂

AMS-IX: Green Light to Incorporate US entity

Members of the Dutch Amsterdam Internet Exchange have given the organisation a green light to incorporate a US entity in order to engage with the Open IX initiative and have the ability to run an exchange in the US while minimising risk to the Dutch association and the Dutch operating company.

This completes the announcements from the big 3 European exchanges (LINX, AMS-IX and DECIX) to operate interconnection services in the US, with the first to make an overt move being LINX, who are in the process of establishing an operation in Northern Virginia. DECIX issued a press release last week that they plan to enter the New York market, and now AMS-IX have a member endorsement to make a move.

There have been concerns amongst the Dutch technical community, who have long held AMS-IX in high regard, that establishing operations in the US will leave the AMS-IX as a whole vulnerable to the sort of systemic monitoring that has been revealed in the press in past weeks. While this is partly the reason for the AMS-IX company suggesting a separate legal entity, in order to hold the US operations at arms length, is it enough for some of the Dutch community? Seems not. In this message the Dutch R&E Network SURFnet seem to think the whole thing was rushed, might not be in the best interests of the community, and voted against the move.

It has been noted that members of the Open IX community, including members of the Open IX Board, were openly calling for AMS-IX members to vote “YES”, and suggesting they also “go out and get 5 other votes”.

What do people think about that? Given that an IX that affiliates to Open IX will have to pay Open IX membership dues, was it right of them to appear to lobby AMS-IX members?

What do people think about the establishment of the separate legal entity? Will this be enough?

Has this done lasting damage to the standing of AMS-IX in the Dutch networking community? Does this matter, or has AMS-IX grown so large that such goodwill doesn’t matter anymore?

On the bigger question, is this sort of thing damaging in the long term to the EU peering community? Does the growth into different countries with different cultures threaten to dilute the member-based ethos that defines a lot of EU exchanges? Or is that just another management challenge for the IX operator to solve?

Might Equinix, who have so far not directly competed with the established EU exchanges, decide they are taking the gloves off and start their own European IX operations in a turf war?

Interesting times.

BA’s Heathrow Lounge Food: Past it’s sell-by date?

Something of a first world problem admittedly, but it’s recently come to the attention of the various frequent flyer circles that BA’s “flagship” lounges at it’s Heathrow hub, the T5 Concorde Room and First Class Lounge recently only scored 2 (out of 5) on a recent food hygiene inspection.

The low score places this “exclusive” venue (to paraphrase BA), reserved specifically for it’s “top customers”, into the bottom 10% of food service premises in the UK. This is something of a last straw for BA’s loyal frequent flyers who have already been upset by a perceived reduction in the quality and service offered by the lounges since the contract for running the food service operation at all BA’s UK lounges were switched to a new operator earlier this year.

There have been complaints of less choice, simple service failures such as grubby cups, glasses and plates put out for customers to use, and used, dirty pots not being regularly cleared away, food not being cooked through properly, and a previously reasonable hot buffet being replaced with troughs of stodgy “gloop” – unpleasant wet food.

Thanks to a Freedom of Information request, the local authority responsible for the inspection, Hillingdon Council, have made the full contents of the report available, highlighting a catalogue of basic food safety disasters:

Out-of-date food in the kitchens
Multiple food preparation areas being sufficiently dirty to be in need of immediate cleaning
“High-risk” food such as prepared sandwiches and cooked meats being insufficiently chilled
Hot buffet food being kept at a sufficiently low temperature to increase risk
Cross-contamination between raw and cooked food
Kitchen maintenance problems such as holes in the walls and floor
Inadequate documentation of staff training

BA have so far been tight-lipped on the matter, anecdotal reports suggesting that senior BA figures consider this just to be some “noisy people on the Internet” which probably highlights that they don’t get it and have their head firmly in the sand. Does this indicate a level of disrespect within BA for it’s customers?

To their credit BaxterStorey meanwhile have issued a statement which, while conciliatory in tone and recognises the failings to some extent, largely seems to fob the problem off on needing to “refurbish” the kitchen.

This really isn’t a brilliant response. Remember, we’re talking about BA’s flagship lounge at it’s flagship airport.

In terms of apologising, what should BA do?

One of the questions among the frequent flyer community has been over BA’s handling of this. While BA’s sub-contractor has decided to issue a statement, there’s been nothing from BA to the most regular lounge guests, it’s frequent flyers.

It’s my opinion that there’s only one way BA can approach this:

with openness, transparency, responsibility and accountability

I know that’s probably a tough ask of a large multi-national corporation with a slick PR machine which is used to deny accountability for everything from delays to lost luggage.

You may ask why the frequent flyers care so much about getting a response from BA, or why BA should care so much to communicate in an frank and honest way with it’s customers?

The frequent flyers care about getting a spin-free honest reply, because they have made a financial and emotional investment in BA. To earn the magic Silver and Gold cards to get them in the privacy of the Galleries lounges, they have spent a lot of money and time with the airline.

They’ve been good, regular customers, demonstrated loyalty to BA, and so have a built an expectation of being dealt with respectfully and fairly in return. That trust has been betrayed by BA and BaxterStorey.

To feed them spin is likely to just increase the levels of angst and venom. The frequent flyers are actively looking for a reason to forgive.

The more honest and fair BA are with their response the more likely they are to be forgiven by it’s community of regular passengers.

See this as an opportunity to set themselves apart from their competitors. It’s not a disaster that must be avoided. Approach it head on.

You’ve been let down, we failed to meet your expectations. We’ve let our supplier take their eye off the ball. We’re sorry. You deserve better. We’ll do better. Here’s how…

Be honest about the mistakes that led to this, and what’s going to happen to make it better.

Most importantly, mean it then do it.

£75k fine a drop in the ocean for First Group

Train operator First Capital Connect has just been fined £75,000 by a UK judge regarding an incident in which up to 700 passengers were stuck for over 3 hours on a train, partially in a tunnel, with no toilets, no ventilation and minimal lighting.

To a conglomerate such as First, which reported over £200m profits in 2012, this has to be a drop in the ocean, and is an absolutely derisory amount compared to the – just over £100 per stranded passenger.

It also begs the question about who is going to pay for this. First Group shareholders? Unlikely. It feels more likely to come out of our pockets, as fare increases, reduced franchise payments to the Treasury, or increased subsidy from the DfT.

We can’t change the “token” fine imposed by the judge – it should probably have had an extra couple of zeros on the end, really – but what might be reasonable is an assurance from the First Capital Connect MD David Statham or Group CEO Tim O’Toole that this fine will ring-fenced, such that it is paid entirely out of group surplus, and must not be allowed to impact the travelling public at large.

Better still, maybe they could pay it out of their no doubt generous bonuses, given the buck stops with them?

I’m also wondering how much has actually been learned from this incident, given the “analysis paralysis” that seems to affect rail operating incidents at the moment?

Errata in RFC1925: The Twelve Networking Truths

Some things in RFC1925, despite it being one of a series of April Fools’ RFCs (and therefore in the good company of the all time classic RFC1149 and it’s brethren), actually do hold true, for instance:

Fast, Good, Cheap: Pick any two – still tends to hold true.

However, like all good April Fools’ RFCs, it will declare that ‘ERRATA EXIST’ at the top. In this case, there’s definitely a shred of truth to this. Especially when you look at truth number 4:

Some things in life can never be fully appreciated nor
understood unless experienced firsthand. Some things in
networking can never be fully understood by someone who neither
builds commercial networking equipment nor runs an operational
network.

My concern is that this statement no longer holds true for the makers of commercial networking equipment.

If the makers and protocol designers really understood, we wouldn’t be pushing water up hill with things such as IPv6 deployment and encouraging use of other networking best practices, they would have made them easier to deploy in the first place.

Therefore a correction is needed, “Some things in networking can never be fully understood by someone who doesn’t run an operational network“.

The Network Engineering “Skills Gap”

Talking to colleagues in the industry, there’s anecdotal evidence that they are having trouble finding suitable candidates for mid-level Network Engineering roles. They have vacancies which have gone unfilled for some time for want of the right people, or ask where they can go to find good generalists that have a grasp of the whole ecosystem rather than some small corner of it.

Basically, a “skills gap” seems to have opened up in the industry, whereby there are some good all-rounders at a fairly senior level, but trying to find an individual with a few years experience, and a good grounding in IP Networking, system administration (and maybe a bit of coding/scripting), network services (such as DNS) and basic security is very difficult.

Instead, candidates have become siloed, from the basic “network guy/systems guy” split to vendor, technology and service specific skills.

This is even more concerning given the overall trend in the industry toward increasing automation of networking infrastructure deployment and management and a tendency to integrate and coalesce with the service infrastructure such as the data centre and the things in it (such as servers, storage, etc.) – “the data centre as the computer”.

This doesn’t work when there are black and white divisions between the “network guy” and the “server guy” and their specific knowledge.

So, how did we get where we are? Firstly, off down a side-track into some self-indulgence…

I consider myself to be one of the more “all round” guys, although I’ve definitely got more of a lean toward physical networking infrastructure as a result of the roles I’ve had and the direction these took me in.

I come from a generation of engineers who joined the industry during the mid-90’s, when the Internet started to move from the preserve of researchers, academics, and the hardcore geeks, to becoming a more frequently used tool of communication.

Starting out as an Internet user at University (remember NCSA Mosaic and Netscape 0.9?) I got myself a modem and a dialup connection, initially for use when I was back home during the holidays and away from the University’s computing facilities, all thanks to Demon Internet and their “tenner a month” philosophy that meant even poor students like me could afford it. Back then, to get online via dialup, you had to have some grasp of what was going on under the skin when you went online, so you could work out what had gone wrong when things didn’t work. Demonites will have “fond” memories of KA9Q, or the motley collection of things which allowed you to connect using Windows. Back then, TCP/IP stacks were not standard!

So, out I came from University, and fell into a job in the ISP industry.

Back then, you tended to start at the bottom, working in “support”, which in some respects was your apprenticeship in “the Internet’, learning along the way, and touching almost all areas – dialup, hosting, leased lines, ISDN, mail, nntp, Unix sysadmin, etc.

Also, the customers you were talking to were either fellow techies running the IT infrastructure in a business customer, or fellow geeks that were home users. They tended to have the same inquisitiveness that attracted you to the industry, and were on some level a peer.

Those with ambition, skill or natural flair soon found themselves climbing the greasy pole, moving up into more senior roles, handling escalations, or transferring into the systems team that maintained the network and servers. My own natural skill was in networking, and that’s where I ended up. But that didn’t mean I forgot how to work on a Unix command line. Those skills came in useful when building the instrumentation which helped me run the network. I could set up stats collection and monitoring without having to ask someone else to do it for me, which meant I wasn’t beholden to their priorities.

Many of my industry peers date from this period of rapid growth of the Internet.

Where did it start going wrong?

There’s a few sources, like a fire which needs a number of conditions to exist before it will burn, I think a number of things have come together to create the situation that exists today.

My first theory is the growth in outsourcing and offshoring of entry-level roles during the boom years largely cut off this “apprenticeship” route into the industry. There just wasn’t sufficient numbers of jobs for support techs in the countries which now have the demand for the people that most of these support techs might have become.

Coupled with that is the transition of the support level jobs from inquisitive fault-finding and diagnosis to a flowchart-led “reboot/reinstall”, “is it plugged in?” de-skilled operation that seemed to primarily exist for the frustrated to yell at when things didn’t work.

People with half a clue, that had the ability to grow into a good all-round engineer, might not have wanted these jobs, even if they still existed locally and were interested in joining the industry, because they had turned into being verbal punchbags for the rude and technically challenged. (This had already started to some extent in the mid-90s.)

Obviously, the people in these roles by the 2000s weren’t on a fast track to network engineering careers, they were call-centre staff.

My second theory is that vendor specific certification caused a silo mentality to develop. As the all-round apprenticeship of helpdesk work evaporated, did people look to certification to help them get jobs and progress their careers? I suspect this is the case, as there was a growth in the number of various certifications being offered by networking equipment vendors.

This isn’t a criticism of vendor certification per se, it has it’s place when it’s put in the context of a network engineer’s general knowledge. But, when the vendor certification is the majority of that engineer’s knowledge, what this leaves is a person who is good on paper, but can’t cope with being taken off the map, and tends to have difficulty with heterogeneous networking environments.

The other problem sometimes encountered is that people have done enough training to understand the theory, but they haven’t been exposed to enough real-world examples to get their head around the practice. Some have been taught the network equivalent how to fly the equivalent of a Boeing 747 or Airbus A380 on it’s extensive automation without understanding the basics (and fun) of flying stick-and-rudder in a little Cessna.

They haven’t got the experience that being in a “learning on the job” environment brings, and can’t always rationalise why things didn’t work out the way they expected.

The third theory is that there was a divergence of the network from the systems attached to it. During the 2000s, it started to become too much work for the same guys to know everything, and so where there used to be a group of all-rounders, there ended up being “server guys” and “network guys”. The network guys often didn’t know how to write scripts or understand basic system administration.

Finally, it seems we made networking about as glamorous as plumbing. Young folk wanted to go where the cool stuff is, and so fell into Web 2.0 companies and app development, rather than following a career in unblocking virtual drainpipes.

How do we fix it?

There’s no mistaking that this needs to be fixed. The network needs good all-round engineers to be able to deliver what’s going to be asked of it in the coming years.

People wonder why technologies such as IPv6, RPKI and DNSSEC are slow to deploy. I strongly believe that this skills gap is just one reason.

We’ve all heard the term “DevOps”, and whether or not we like it – it can provoke holy-wars, this is an embodiment of the well-rounded skill set that a lot of network operators are now looking for.

Convergence of the network and server environment is growing too. I know Software Defined Networking is often used as a buzzword, but there’s a growing need for people that can understand the interactions, and be able to apply their knowledge to the software-based tools which will be at the heart of such network deployments.

There’s no silver bullet though.

Back in the 2000s, my former employer, LINX, became so concerned about the lack of good network engineering talent, and woeful vendor specific training, that it launched the LINX Accredited Internet Technician programme, working with a training partner to build and deliver a series of platform-agnostic courses which built good all-round Network Engineering skills and how to apply these in the field. These courses are still delivered today through the training partner (SNT), while the syllabus is reviewed and updated to ensure it’s continuing relevance.

IPv6 pioneers HE.net offer a number of online courses in programming languages which are useful to the Network Engineer, in addition to their IPv6 certification programme.

There is also an effort called OpsSchool, which is building a comprehensive syllabus of things Operations Engineers need to know – trying to replicated the solid grounding in technology and techniques that would previously be picked up on the job while working in a helpdesk role, but for the current environment.

We’ve also got attempts to build the inquisitiveness in younger people with projects such as the Raspberry Pi, while venues such as hackspaces and “hacker camps” such as OHM, CCC and EMF exist as venues to exchange knowledge with like-minded folk and maybe learn something new.

We will need to cut our existing network and systems people a bit of slack, and let them embark on their own learning curves to fill the gaps in their knowledge, recognise that their job has changed around them, and make sure they are properly supported.

The fact is that we’re likely to be in this position for a few years yet…

Third Runway, or not Third Runway?

Hot news today is Heathrow Airport’s third runway plans. It seems there’s some realisation that a “Boris Island” won’t be built early enough to satisfy the needs of the South East’s demand for landing slots, and something needs to be done now rather than in 20-odd years.

There is a perception that London lags behind Amsterdam Schiphol, Paris Charles De Gaulle or Frankfurt, in the sense that it’s not an “airline hub” of the same magnitude, and dear old London Town is being left behind.

If anyone has been through any of the above airports recently, I’m not entirely sure that being like them is something we should be aspiring to!

I’ve already made my views known about Frankfurt‘s recent redevelopments, trying to make it less painful than before, and still managing to miss the target.

Anyone who flies to Amsterdam often enough will have experienced the mind-numbingly long taxi to or from their relatively new runway, which far enough away to be built in a completely different town to the airport itself. You would be forgiven for thinking you’re driving to the UK, as the taxi time is often as long as the flight itself, unless you’re lucky enough that the prevailing wind lets you take off and land closer to the terminal.

As for Charles De Gaulle… I’ll just give you a Gallic shrug.

While Heathrow is BA’s “hub”, it’s not really a hub operation in the sense of a US air carrier. Flights don’t arrive and depart in deliberately orchestrated waves, purposely designed to connect, such as Delta’s operations in Atlanta. BA’s hub operation is more by accident, because of the sheer volume of the operation, rather than schedule design. Flights “happen” to connect, rather than do so by design.

Following the effective breakup of the BAA, Heathrow, Gatwick and Stansted are now owned by different operators, and from reading this BBC article each of them seem to be vying for a bit of the cherry, while Boris would like to demolish Heathrow entirely.

What it’s left me wondering is why there is a complete lack of joined up approach?

Danger Will Robinson! Radical thinking…

In terms of land and environmental concerns such as noise, a 2nd runway at Gatwick seems to be an easy win when compared against putting a 3rd runway at Heathrow.

Given that we’re seemingly hell bent on building HS2 (let’s ignore the fact that less than half of the money being spent on HS2 could revolutionise rail in the North of England) , wouldn’t it be eminently sensible to extend it such that it touches Heathrow and extends South to Gatwick? Use the train as a complementary form of transport to the train, rather than as a competitor.

It could then serve a dual-purpose of making it more convenient for those in the Midlands to access Heathrow and Gatwick, while also handling connecting traffic between Gatwick and Heathrow.

What would the Gatwick to Heathrow travel time be on such a train? About 20-25 minutes? I know some airports where it can take just as long to transfer between terminals, or to get from departure lounge to gate!

Might it even be possible to provide trains, or designated sections of trains, for “sterile transit” between the airports, without the need to officially enter the UK?

Yes, this will involve taking on the fearsome NIMBYs of Surrey, but isn’t it all for the “greater good”?

Should we ever decide to build “Boris Island” or devastate Hoo with a big International airport, it’s close enough to HS1 to be hooked up to that. We can offer fast train connections into Central London, and maybe even to France or Brussels from the airport. Just think, it might be preferable to fly in to Boris Island then get the train, if you’re travelling to Lille!

But, as I say, that would require some joined-up thinking. Something we need to get better at.

Anti-spoofing filters, BCP38, IETF SAVVI and your network

I was invited to present at the recent IX Leeds open meeting, as “someone neutral” on the topic of BCP38 – largely in relation to the effects from not deploying it, not just on the wider Internet, but on your IP networking business (if you have one), and on the networks you interconnect with.

I basically broke the topic down:

Introduction: I started by introducing the problem in respect of the attack (“that nearly broke the Internet”) on the CloudFlare hosted Spamhaus website in March 2013.

What and how: Quick overview of address spoofing and how a backscatter amplification attack works.

What you should do: BCP38, uRPF, etc., and what you need to do, and what to ask your suppliers.

Why you should care: Yes, it benefits others, but you have costs in terms of bandwidth and abuse/security response too.

The bleeding edge: IETF SAVI working group.

It wasn’t meant to be a technical how-to, but a non-partisan awareness raiser, as the IX Leeds meeting audiences aren’t full of “usual suspects” but people who are less likely to have been exposed to this.

It’s important to get people doing source address filtering and validation, both themselves, and asking their suppliers for it where it’s appropriate.

Here’s the slide deck (.pdf) if you’re interested.

You’ve now got to be big to do IT for Network Rail

I noticed this article appear on The Register this afternoon. Caught my interests as it’s crosses tech and travel industries.

The main gist of this is that Network Rail, the organisation responsible for rail infrastructure in Great Britain, has changed it’s IT procurement strategy, creating a framework with 5 massive players able to bid for the work in the future.

No doubt dealing with just 5 large organisations is helpful to whoever is managing contracts at Network Rail, who up until now may have had over 250 different IT suppliers.

The questions immediately occurring in my mind are:

Does this risk stifling of innovation? By excluding smaller, agile companies from participating, does it run the risk of NR’s IT becoming dominated by expensive, white elephant, gold-plated mega-systems that try to boil the sea?
Do the cost savings from easier contract management actually weigh up against the threat of an oligopoly developing, which could force up the price for IT services? It’s unlikely that all 5 suppliers in the framework would bid for every tender or work package, maybe two or three would?
How does this line up with one of the alleged benefits of rail privatisation: the dismantling of the BR monolith would allow entrepreneurial organisations to operate in the sector, this is something which has probably only had limited success and then only in specific areas.

At the end of the day, it’s public money that Network Rail is spending here. Hmm…

My local bakery is going stale…

When we moved here, we were really happy to see that the local cluster of shops (useful stuff like Post Office, Chemist, Dry Cleaners, a small super market) that serves our neighbourhood also had one of a dying breed, a traditional baker’s shop, part of a small chain owned by a family business.

Sure, the bread wasn’t made in the shop, they had a more modern bakery in a light industrial unit about 30 minutes drive away which supplied all their shops and wholesale customers, but they sold great tasting loaves with a fantastic light texture and crispy crust.

My stomach really can’t hack cheap supermarket bread, either bulked up with high percentages of soy flour to help improve the consistency of the crumb, or made with more yeast than is necessary to reduce the time needed to prove. Both upset my insides, causing me bloating, discomfort and in some cases, pretty bad indigestion.

So I was delighted when shortly after moving here, the indigestion just stopped dead. The only thing which really changed in my diet was where the bread was coming from (aside from possibly the water coming out of the tap). I even tested this theory by eating regular mass-produced bread, and the gut rot came back within a few days.

Relieved to put a calmer stomach down to the nice crusty bread on my doorstep, it just reinforced all that was good about our new neighbourhood.

Sadly, all good things must come to an end. While the bakery hasn’t closed down, it has recently changed hands, and is now being supplied by the new owners – still a small, local bakery, but it turns out, it isn’t quite the same.

Not to be daunted, we tried a few things from there over the last couple of weeks, only to feel let down.

The breads don’t look the same: uneavenly risen, with a pale and flaccid crust concealing a spongy, yet heavy, dense, interior, with a cotton-wool-like texture. Neither do they smell the same: there’s an overriding smell of yeast about the new owner’s bread.

The old owner’s recipe would go stale by going dry and hard, and would seldom go mouldy. The new owner’s bread goes mouldy, because it seems to retain the moisture for longer.

Sadly, this also extends to their pastries, which leave a feeling like the inside of your your mouth has been coated in a layer of vaseline (I guess they don’t use butter, but some sort of margarine or veg shortening) as well as being so sweet that you get the shakes.

While we’re glad that it’s stayed a bakery, rather than becoming yet another hairdresser, nail bar, beauticians or (our first!) fried chicken shop, we’re gutted that we’ve lost our supply of traditionally baked bread that was on our doorstep.