Category: Uncategorized

#NorthernFail – Making the operational perfect storm

The saga of the May 20th UK railway timetable changes continue. Modified services remain in place on the Northern network, still fewer trains are running than before the May timetable change, though a reinstatement of Rest Day Working for Drivers and a temporary timetable has stabilised things somewhat.

In an amazing show of “can-do”, West Coast Railways – the Carnforth-based people who ran the Hogwarts Express for the Harry Potter films, as well as some splendid rail trips with steam locomotives – worked with Westmorland and Lonsdale MP Tim Farron to manage to return trains to the Lakes Line to Windermere, which hadn’t seen a train for two weeks.

Meanwhile the heads of the involved TOCs and their planning subject matter experts, and colleagues from Network Rail, have been required to give evidence in front of the House of Commons Transport Select Committee.

You can see the full session – almost 2 hours and 30 minutes – at parliamentlive.tv.

What was clear from the evidence is that the entire UK Rail Industry and various Governments had a part to play in letting us down, and worse still, no one person has executive responsibility, even to act on massive red flags.

The existence of the temporary service on the Lakes Line has even focused Northern’s mind it seems, because they’ve now said they expect to return Northern-operated trains to the line earlier than originally anticipated, Northern MD David Brown giving a date of 2nd July to the Committee.

They obviously didn’t much like being shown up by a heritage rail operator.

But back to the Perfect Storm…

A number of conditions came together to cause the chaos wreaked on us since 20th May:

  • Late handover of planned infrastructure improvements: In particular the Preston-Blackpool resignalling and electrification. This work didn’t just put up overhead wires, but replaced all the signalling, changed the positioning of junctions and the entire layout of Blackpool North station. This section of line was effectively a new railway and drivers needed to re-train on the route.
  • Failure to deliver other key infrastructure improvements: The electrification via Bolton to Preston is now delayed until the end of the year. This project has suffered multiple setbacks including the lead contractor withdrawing from the work back in 2015 because they said it would overrun, the replacement contractor (Carillion) going bust, and issues with ground stability (discovery of pockets of sand and abandoned mine shafts). This project will be almost two years late if it is delivered at the December timetable change as currently hoped.
  • Unavailability of promised trains: Northern required a number of extra trains if the 20th May plan was to work properly. These were not new trains, but were being made available through something known in the railway industry as a “cascade“, new trains being delivered elsewhere displace trains to other services. Northern were expecting to get additional trains, compatible with their existing fleets, from Scotland and the South West, however this cascade has failed due to delays in the introduction of new trains. In Scotland the trains are having to be modified because drivers have complained about the visibility through the curved windscreens – likening it to “driving a goldfish bowl”. As a result only a tiny fraction of the expected extra carriages were available.
  • Ongoing labour relations issues in Northern, and this is an area which deserves further explanation in itself.

Why are Northern short of trains now?

This relates to the “cascade” that I eluded to above. Think of the cascade as one of those tile games with one empty square.

The plan was to move trains from Scotland and the Plymouth and Cornwall areas to Northern in time for May 2018. With the odd exception (Class 170 units), these would be stock that are technically identical to that which Northern already have in their operation (Class 150/1, Class 158, Class 319), thus easing requirements for crew training and route-clearance.

However to create the “empty square” that allows the cascades to happen, we had dependencies on four projects:

  • Completion of Paddington to Reading electrification
    • This allows roll out of Class 387 electric trains on Thames Valley suburban services between Reading and Paddington.
    • This permits release of Class 165/166 “Turbo” units to move within GWR from the Thames Valley suburban network to the Bristol area.
    • That allows a cascade of trains within the GWR franchise such that it releases a number of 150/1 “Sprinters”, which already work in the Northern area, to Northern.
  • Edinburgh-Glasgow electrification and roll out of Class 385 Hitachi trains
    • The line has been electrified to plan, but the Class 385s are delayed due to the “goldfish bowl effect” of the curved windscreens.
    • This has delayed release of some Class 158 and Class 170 units from Scotland to Northern.
    • Surplus electric trains  have recently been drafted in to Scotland to cover the gap, and hopefully allow some more trains to move to Northern.
  • Late delivery of upgraded High Speed Train sets, themselves cascaded from GWR by IEP introduction, to Scotrail long distance services
    • This would release further Class 158 and 170 units to Northern.
    • The release of the HSTs from GWR was delayed by slow introduction of the IEP and delays to the Great Western electrification.
  • Non-delivery of Class 769 Bi-Mode “flex” units
    • Converted from Class 319 units by addition of diesel engines so they can operate away from the wires.
    • Made necessary by Government’s indecision on electrification of NW secondary routes such as Oxenholme to Windermere.
    • The prototype has yet to complete testing and be signed off.

Northern was meant to have over 60 additional trains delivered in time for May, including being handed over early enough to allow for crew training on types new to Northern (such as the 170 and 769).

Instead they got less than 30.

Who controls what stock is used and where? To a large extent, despite rail privatisation, there is not really a “free market” for rolling stock.  While stock may be privately owned by banks, trusts and investment companies, much like aircraft leasing, the types of stock and the disposition are largely co-ordinated by the Department for Transport.

Northern is not a unified company

While “Northern by Arriva” might be the brand, and there is one company at the back end, Arriva Rail North Ltd, operationally it is a more complex animal.

Northern is effectively a mash-up of some component parts of previous rail franchises.

Remember First North Western? Remember Northern Spirit (this was Arriva’s previous, and similarly disastrous attempt at running trains in the North)?

15 years ago, the decision was made by Whitehall Mandarins to strip out the long distance routes from Northern Spirit and make a separate “TransPennine Express” franchise, which had aspirations of being more an “Intercity” operator and less of a local operator. Some routes from the old First North Western franchise (such as fast trains from Blackpool or Barrow to Manchester Airport) were also transferred into TPE, in addition.

Once the North Wales operations had also been stripped out of FNW due to Welsh devolution and the creation of a single Wales rail operator, the left overs (the local routes around Manchester, Liverpool, Lancashire & Cumbria) were merged with the left overs from the opposite side of the Pennines.

This created the “zero-growth” Northern franchise of 2004.

However, this wasn’t a unified company, it effectively had two separate halves, an “East Side” and a “West Side”, divided by the Pennines. Each had it’s own trains, and it’s own train crew. More importantly, the terms and conditions of East Side and West Side employees remained as previously – correctly protected by TUPE – but they were never harmonised with the passage of time.

That is a situation that has persisted to this day.

This was complicated further by the 2016 franchise change which took a number of TPE routes and transferred them (and the folk that work them) back into the Northern franchise. This included drivers at depots such as Blackpool and Barrow. These drivers find themselves working once again for Northern, but as their employment was protected by TUPE, their terms and conditions are different again from other Northern “West” drivers.

Northern have ended up with a fragmented workforce, with multiple different contracts, and different terms and conditions.

Furthermore, the railway is still run as two separate halves in terms of traction and train services. Services are either “East side” or “West side”.

This shows in the quality and types of rolling stock deployed in the various areas.

While some trains are obviously captive to their local areas, such as the electric units (Class 323s around Manchester and Class 333 on the Airedale routes), the diesel units are split such that the lion’s share of the more modern units (e.g. air conditioned Class 158 units) are allocated to the East side, while the West side has a greater proportion of older units such as 150/1 and the non-air conditioned Class 156.

This was a significant sea-change on routes such as Blackpool-Manchester which used to enjoy a number of modern air conditioned Class 175 units under First North Western, then Class 185 units when the service passed to Transpennine, but have more recently been served by a mix of the dreaded Pacers, or the older Class 150 or 156 units.

In the cascade of rolling stock, the newer Class 170 units from Scotland being transferred to Northern will also appear to be focused on East Side services.

Conspiracy theorists might suggest that with Northern Rail HQ being in York, this division of resources is somewhat intentional, ensuring the East side operations have the newer and notionally better units!

However this is more likely to be a habit that persists from the British Rail days. Aside from the premier services such as the WCML expresses, the North West has long had to make do with the leftovers and hand-me-downs.

The longer distance and more “premium” services in the North of England (e.g. North & South Trans-Pennine) had historically always been operated by the Eastern Region out of York, with Eastern Region locomotives and rolling stock (something which persisted through to creation of the Transpennine Express franchise). Services run by the Midland Region tended to be shorter distance, potentially slower, more locally focused, and so tended to get the less modern stock.

I’m not saying the local operations in the North West never received stock from new – a good number of Pacers and Class 150/2 “Sprinter” units were allocated here on delivery, but as part of a wider scheme where new trains were rolled out across the BR regional fleet.

So, really, who IS responsible for the mess?

The Rail industry as a whole bears some level of responsibility for this perfect storm, and I extend this to include the relevant bits of Whitehall.

The operators have done the best that they could with a deck that has been stacked against them by vacillating transport strategy coming out of Governments and the DfT.

Franchise re-carves and reshuffles, on-off electrification schemes, no-growth franchise plans, interference in rolling stock distribution.

Pressure from Whitehall to do less with more. This is particularly evident for Network Rail who were told to reduce costs in their train planning functions, which happened, but then led them into attempting one of the most widespread and comprehensive timetable recasts in probably 30 years with an under-resourced planning & timetabling department.

The fact is that successive Governments have manipulated rail investment as a political tool to win votes, rather than having a long-term strategy.

Want to win an election? Promise to electrify. Win the election? “Oh, sorry, that electrification? It’s indefinitely postponed now.” Meanwhile, money is poured into “shiny” schemes.

The lack of a consistent and coherent rail strategy has created such uncertainty for the industry, such that when decisions are made, they seem more like tactical ones, as opposed to long-term strategy.

Furthermore, the fractured nature of the rail industry seems to mean that no-one is prepared to take the tough “executive level” decisions – such as the one which could have allowed operators to roll-forward the pre-May 2018 timetable until the required infrastructure improvements had been turned over, and the rolling stock cascade was assured.

My opinion is that an even bigger storm coming, and no-one is steering the ship.

Metrolink Service Recovery Challenges – Pt 1

In the opening post in this series, I put forward a series of probably common questions as to why it took so long to recover service following the Metrolink disruption on 10th November:

  • Why couldn’t the trams be diverted via 2CC?
  • Why couldn’t the trams be turned around short of the destination?
  • Why couldn’t the trams just run to a different destination?
  • Why was the Bury-Piccadilly service, which doesn’t go anywhere near the incident, stopped?

Let’s take the first of these:

Why couldn’t some trams be diverted via 2CC (the Second City Crossing)?

Once it was ascertained that the incident was only affecting one line, the one which fed trams from St Peter’s Square toward Piccadilly Gardens, then you would think it becomes feasible to divert the approaching City-bound trams from Deansgate via 2CC, taking them “off-route”.

The problem arises with what you do with them once they reach the other side of the City Centre.

It is very difficult for a Piccadilly, Etihad or Ashton-bound tram to regain the booked line of route, as there is no easy way from the far end of 2CC to head back in to town via Market St. There is no direct link between 2CC and this line without reversing direction.

So, the route via 2CC becomes an easy option for the existing Altrincham-Bury services, but that’s largely it without a more complex operation.

It’s also likely that routing all the trams down the 2CC would cause road congestion as well, as Cross Street is a shared right of way between trams and road traffic. So it becomes necessary to “thin out” the service.

Once the trams get to Victoria you have to do something with them, either turning them short of destination, reversing them, or doing something else with them, such as taking them out of traffic and running them to depot.

This leads us nicely onto…

Limited Places to turnback or “recess” trams

While Metrolink has a number of emergency turnback crossings which can be used to maintain service during periods of disruption, these effectively block the line while the driver changes from one end of the tram to the other, then receives a slot to cross over to the other line and return in the opposite direction. A queue of trams could quickly build up if this happened along a busy section of line. Some of these are also “unsignalled moves” so are only useable under very specific circumstances.

Besides the outer ends of the network, there are some specific turnback locations which have extra platforms or sidings to cater for reversing trams:

  • Timperley – has a siding so a Southbound tram can turn around and return toward the City Centre. This is most commonly used when trams can’t operate on the Network Rail managed section between Timperley and Altrincham, but is occasionally used to manage late running.
  • Cornbrook – there is a long centre siding here, this was until recently the usual terminus for the Airport Line, it can hold a good number of trams, and is useful during emergencies, both for recessing trams and reversing trams approaching the City Centre.
  • Deansgate – there is a centre platform which is bi-directionally signalled – that is trams can arrive and depart in either direction and reverse. It is currently regularly used for reversing the Airport service.
  • Piccadilly (Sheffield Street) – outside the back wall of Piccadilly station undercroft this has a centre track for reversing trams that have arrived from the City Centre and sending them back into Piccadilly station and thence toward Piccadilly Gardens. This is normally used for reversing two services, MediaCity-Piccadilly and Bury-Piccadilly. There are a further pair of crossovers in the “tunnel” underneath Piccadilly Station, these are most commonly used for reversing a tram from the Ashton direction and sending it back that way.
  • Etihad Campus – there is a long reversing track between the Etihad and Velopark stations. This is normally used for reversing trams on the Altrincham-Etihad service.
  • Shaw & Crompton – there is one dead-end platform as well as the two through platforms. This is normally used for reversing trams on the Didsbury-Shaw services.

What’s clear to see is that except for Shaw there are no dedicated reversing loops on the Northern side of the City Centre.

That means there’s nowhere convenient to reverse trams without blocking the line to “through traffic”, which causes more delays.

“But what about that third platform at Victoria, the one in the middle?” I hear you ask.

Regular users of the network will be used to this “white elephant”. Installed during the recent Victoria redevelopment, it has rarely been used. I believe it may have been used on one or two occasions during some planned engineering works, but it otherwise sits there, and is never used in normal service. There are even “Tensabarriers” across the centre tracks here, that’s how unused it is.

If this was operational during Friday’s incident then diverting Eccles-Ashton or Altrincham-Etihad trams via 2CC, reversing at Victoria then running via Market St, and regaining their booked route at Piccadilly Gardens might have been possible, but it wasn’t.

There might be very good reasons why it is not used, but these are not shared with us mere mortals. Instead it sits there as an embarrassment.

Trams from the South of the city which are currently shown as terminating at Victoria actually run empty to Queens Road after emptying out – there they have to reverse direction on one of the main lines, either to return South or enter the tram depot at Queens Road. There is no shunting line or reversing siding provided to do this move, and limited space to build one even if there was funding to do it.

There used to be a reversing siding just outside Victoria that would have been useful, but I also believe that to be disconnected/decommissioned and not available for use.

So, in spite of the additional flexibility added by 2CC, there are still some fairly significant constraints facing the controllers, and a big risk of trams backing up.

The redevelopment work for Crumpsall station will provide a new “turnback” facility, which will normally be used as part of the new Trafford Park service, but again provides Metrolink controllers with another useful option on the Northern side of the network when handling disruption.

Anyway, that’s enough for today.

Next time, we’ll find out that despite looking the same not all Manchester trams are equal, and maybe a word or two on power.

Tech Skills: Business Needs to Give Back to Education?

Another day, another skills and tech education rant…

My thoughts here are partly triggered by this tweet about the HE system being effectively broken, from Martin Bryant of Tech North:

The overall takeaway is that the current HE models aren’t working for the tech sector on two counts: a) It moves too fast and b) There is too much to learn. Sounds like a perfect storm. The syllabuses and the people doing the teaching are continually at risk of being out of date.

I agree with the comments in the article that there is an expectation mismatch going on.

Companies can’t realistically expect graduates to just drop into a position and be as useful and productive as a person with several years’ experience, nor can they expect them to know the ins and outs of every protocol or language. As an employer, there needs to be some sort of development plan in place for every person you hire.

I’ve previously written about my own entry into the tech industry, somewhat by accident, in the 1990s: that I left University with a degree in nothing to do with tech, but with a solid understanding of the fundamentals, and basically ended up learning through apprenticeship-type techniques once on the job in an ISP.

Where I believe tech companies can help is by pouring their real-world knowledge and experience back into the teaching, and by this I mean moving beyond just the usual collaboration between industry and education and taking it a stage further: Encouraging their staff to go into Universities and teach, and the Universities to do more to guide and nurture this.

Yes, it takes effort. Yes, for the tech firms releasing staff members one or two days a week to teach, it’s going to be a longer term investment, the payback won’t immediately come for three to five years. But once it does arrive, it seems that the gift will keep giving.

For the Universities, they will need to support people who maybe aren’t experienced in teaching and find new ways of making their curriculum flexible enough to keep up.

I see this potentially having multiple positive paybacks:

  • For the University: they are getting some of the most up-to-date and practical real-world knowledge being taught to their students.
  • For the Students: increased employability as a result of relevant teaching.
  • For the Industry: a reducing skills gap and more inquisitive and employable individuals looking for careers.
  • For the tech employee doing the teaching: a massively rewarding experience that can help increase job satisfaction.

For this, I’d like to put my industry colleague Kevin Epperson on a bit of a pedestal. Kevin is an experienced Internet Engineering professional, and has worked at big industry names such as Microsoft, Level 3 and Netflix, in senior Network Engineering leadership roles. Kevin also teaches at University of Colorado, Boulder and is the instructor of their IP Routing Protocols course.

This is something Kevin has been doing for 15 years, during tenures at three different employers, and recently received an award for his contributions to their ITP courses.

The value of having a real-world industry expert involved in instructing students can’t be underestimated, yet I can think of very few people in my part of the tech industry – Internet Engineering – that actually teach in the formalised way Kevin does. All I can really think of is people doing more informal tutoring at conferences, and helping to organise hackathons and other events, which don’t necessarily reach the HE audience we’re discussing in this article.

The industry needs to be willing to move outside the rigid “full-time” staff model that I still find many companies wedded to, despite claims to the contrary, and be confident about releasing their staff to do these things, in order to give back. It may even improve staff retention and satisfaction for them, as well as produce more productive new hires! Be interested to see if people have real stats, or just good stories, on that.

Yet, at the same time, we’re talking about an industry that won’t give many members of staff a day out of the office to attend a free industry event which will benefit their skills and experience and improve their job satisfaction. Why? Who knows? I have hard data on this from running UKNOF that I’m willing to share if folks are interested.

The answer that I don’t have is how receptive the Universities would be to this? Or do they have their own journey of trust to go on?

Net Eng Skills Gap Redux: Entry Routes into Network Engineering

While I was recently at the LINX meeting in London, I ended up having a side-discussion about entry routes into the Internet Engineering industry, and the relatively small amount of new blood coming into the industry.

With my UKNOF Director’s hat on for a moment, we’re concerned about the lack of new faces showing up to our meetings too.

Let me say one thing here and now:

If you work in any sort of digital business, remember that you are nothing without the network, nothing without the infrastructure. This eventually affects you too.

Yes, I know you can just “shove it in the Cloud”, but this has to be built and operated. It has real costs associated with it, and needs real people to keep it healthily developing and running.

I’ve written about this before here, almost 3 years ago. But it seems we’re still not much better off. I think that’s because we’ve not done enough about it.

One twitter correspondent said, “I didn’t know the entry route, so ended up in sysadmin, then internet research, and not netops.”

This pretty much confirmed some of my previous post, that we’d basically destroyed the previous entry route through commoditisation of first-line support, and that was already happening some time around 1998/1999.

It’s too easy to sit here and bleat, blaming “sexy devops” for robbing Net Eng and Network Infrastructure of keen individuals.

But why are things such as devops and more digital and software oriented industries attracting the new entrants?

One comment is that because a large number of network infra companies are well established, there isn’t the same pioneering spirit, nor the same chance to experiment and build, with infrastructure compared to the environment I joined 20 years ago.

My colleague, Paul Thornton, characterised this pioneering spirit in a recent UKNOF presentation titled “None of us knew what we were doing, we made it up as we went along” – note that it is full of jargon and colloquialism, aimed at a specific techie audience, but if you can excuse that, it really captures in a nutshell the mid-90’s Internet engineering environment the likes of he and I grew up in.

Typing “debug all” on a core router can liven up your afternoon no end… But I didn’t really know what I wanted to do back then, I was green and wet behind the ears.

Many infrastructure providers are dominated by obsessions with high-availability, and as a result resistance to change, because they view a stable and available infrastructure as the utopia. An infrastructure which is being changed and experimented upon, by implication, is not as stable.

do-not-touch-any-of-these-wires
DO NOT TOUCH ANY OF THESE WIRES

Has a desire to learn (from mistakes if necessary!) become mutually exclusive from running infrastructure?

In many organisations, the “labs” – the development and staging environments – are pitiful. They often aren’t running the same equipment as that which exists in production, but are cobbled together from various hand-me-down pieces of gear. This means it’s not always possible to compare apples with
apples, or exactly mimic conditions which will exist in production.

Compare this to the software world, where everything is on fairly generic compute, and the software is largely portable from the development and staging environments, especially so in a world of virtualisation and containerisation. There’s more chances to experiment, test, fail, fix and learn in this environment, than there is in an environment where people are discouraged from touching anything for fear of causing an outage.

This means we Network Engineering types need to spend a lot of time on preparation and nerves of steel before making any changes.

Why are the lab environments often found wanting? Classically it’s because of the high capital cost of network gear, which doesn’t directly earn any revenue. It’s harder to get signoff, unless your company has a clear policy about lab infrastructure.

I’m not saying a blanket “change control is bad”, but a hostile “don’t touch anything” environment may certainly drive away some of the inquisitive folks who are keen to learn through experimentation.

Coupled with the desire of organisations to achieve high availability with the lowest realistically achievable capital spend, it means that when these organisations hire for Network Engineering posts, they often want seasoned and experienced individuals, sometimes with vendor specific certifications. You know how I hold those in high esteem, or not as the case may be, right?

So what do we need to do?

I can’t take all the credit for this, but it’s partly my own opinions, mixed in with what I’ve aggregated from various discussions.

We need to create clear Network and Infra Engineering apprenticeship and potential career paths.

The “Way In” needs to be clearly signposted, and “what’s in it for you” made obvious.

There needs to be an established and recognised industry standard for the teaching in solid basic network engineering principles, that is distinct from vendor-led accreditations.

In some areas of the sector, the “LAIT” (LINX Accredited Internet Technician) programme is recognised and respected for it’s thoroughness in teaching basic Internet engineering skill, but it’s quite a narrow niche. Is there room to expand the recognition this scheme, and possibly others have?

A learning environment needs to exist where we enable people to make mistakes and learn from them, where failure can be tolerated, and priority placed on teaching and information sharing.

This means changing how we approach running the network. Proper labs. Proper tooling. Proper redundant infrastructure. No hostile “change control” environment.

Possibly running more outreach events that are easier for the curious and inquisitive to get into? That’s a whole post in itself. Stay tuned.