Metrolink Service Recovery Challenges – Pt 1

In the opening post in this series, I put forward a series of probably common questions as to why it took so long to recover service following the Metrolink disruption on 10th November:

  • Why couldn’t the trams be diverted via 2CC?
  • Why couldn’t the trams be turned around short of the destination?
  • Why couldn’t the trams just run to a different destination?
  • Why was the Bury-Piccadilly service, which doesn’t go anywhere near the incident, stopped?

Let’s take the first of these:

Why couldn’t some trams be diverted via 2CC (the Second City Crossing)?

Once it was ascertained that the incident was only affecting one line, the one which fed trams from St Peter’s Square toward Piccadilly Gardens, then you would think it becomes feasible to divert the approaching City-bound trams from Deansgate via 2CC, taking them “off-route”.

The problem arises with what you do with them once they reach the other side of the City Centre.

It is very difficult for a Piccadilly, Etihad or Ashton-bound tram to regain the booked line of route, as there is no easy way from the far end of 2CC to head back in to town via Market St. There is no direct link between 2CC and this line without reversing direction.

So, the route via 2CC becomes an easy option for the existing Altrincham-Bury services, but that’s largely it without a more complex operation.

It’s also likely that routing all the trams down the 2CC would cause road congestion as well, as Cross Street is a shared right of way between trams and road traffic. So it becomes necessary to “thin out” the service.

Once the trams get to Victoria you have to do something with them, either turning them short of destination, reversing them, or doing something else with them, such as taking them out of traffic and running them to depot.

This leads us nicely onto…

Limited Places to turnback or “recess” trams

While Metrolink has a number of emergency turnback crossings which can be used to maintain service during periods of disruption, these effectively block the line while the driver changes from one end of the tram to the other, then receives a slot to cross over to the other line and return in the opposite direction. A queue of trams could quickly build up if this happened along a busy section of line. Some of these are also “unsignalled moves” so are only useable under very specific circumstances.

Besides the outer ends of the network, there are some specific turnback locations which have extra platforms or sidings to cater for reversing trams:

  • Timperley – has a siding so a Southbound tram can turn around and return toward the City Centre. This is most commonly used when trams can’t operate on the Network Rail managed section between Timperley and Altrincham, but is occasionally used to manage late running.
  • Cornbrook – there is a long centre siding here, this was until recently the usual terminus for the Airport Line, it can hold a good number of trams, and is useful during emergencies, both for recessing trams and reversing trams approaching the City Centre.
  • Deansgate – there is a centre platform which is bi-directionally signalled – that is trams can arrive and depart in either direction and reverse. It is currently regularly used for reversing the Airport service.
  • Piccadilly (Sheffield Street) – outside the back wall of Piccadilly station undercroft this has a centre track for reversing trams that have arrived from the City Centre and sending them back into Piccadilly station and thence toward Piccadilly Gardens. This is normally used for reversing two services, MediaCity-Piccadilly and Bury-Piccadilly. There are a further pair of crossovers in the “tunnel” underneath Piccadilly Station, these are most commonly used for reversing a tram from the Ashton direction and sending it back that way.
  • Etihad Campus – there is a long reversing track between the Etihad and Velopark stations. This is normally used for reversing trams on the Altrincham-Etihad service.
  • Shaw & Crompton – there is one dead-end platform as well as the two through platforms. This is normally used for reversing trams on the Didsbury-Shaw services.

What’s clear to see is that except for Shaw there are no dedicated reversing loops on the Northern side of the City Centre.

That means there’s nowhere convenient to reverse trams without blocking the line to “through traffic”, which causes more delays.

“But what about that third platform at Victoria, the one in the middle?” I hear you ask.

Regular users of the network will be used to this “white elephant”. Installed during the recent Victoria redevelopment, it has rarely been used. I believe it may have been used on one or two occasions during some planned engineering works, but it otherwise sits there, and is never used in normal service. There are even “Tensabarriers” across the centre tracks here, that’s how unused it is.

If this was operational during Friday’s incident then diverting Eccles-Ashton or Altrincham-Etihad trams via 2CC, reversing at Victoria then running via Market St, and regaining their booked route at Piccadilly Gardens might have been possible, but it wasn’t.

There might be very good reasons why it is not used, but these are not shared with us mere mortals. Instead it sits there as an embarrassment.

Trams from the South of the city which are currently shown as terminating at Victoria actually run empty to Queens Road after emptying out – there they have to reverse direction on one of the main lines, either to return South or enter the tram depot at Queens Road. There is no shunting line or reversing siding provided to do this move, and limited space to build one even if there was funding to do it.

There used to be a reversing siding just outside Victoria that would have been useful, but I also believe that to be disconnected/decommissioned and not available for use.

So, in spite of the additional flexibility added by 2CC, there are still some fairly significant constraints facing the controllers, and a big risk of trams backing up.

The redevelopment work for Crumpsall station will provide a new “turnback” facility, which will normally be used as part of the new Trafford Park service, but again provides Metrolink controllers with another useful option on the Northern side of the network when handling disruption.

Anyway, that’s enough for today.

Next time, we’ll find out that despite looking the same not all Manchester trams are equal, and maybe a word or two on power.

Tech Skills: Business Needs to Give Back to Education?

Another day, another skills and tech education rant…

My thoughts here are partly triggered by this tweet about the HE system being effectively broken, from Martin Bryant of Tech North:

The overall takeaway is that the current HE models aren’t working for the tech sector on two counts: a) It moves too fast and b) There is too much to learn. Sounds like a perfect storm. The syllabuses and the people doing the teaching are continually at risk of being out of date.

I agree with the comments in the article that there is an expectation mismatch going on.

Companies can’t realistically expect graduates to just drop into a position and be as useful and productive as a person with several years’ experience, nor can they expect them to know the ins and outs of every protocol or language. As an employer, there needs to be some sort of development plan in place for every person you hire.

I’ve previously written about my own entry into the tech industry, somewhat by accident, in the 1990s: that I left University with a degree in nothing to do with tech, but with a solid understanding of the fundamentals, and basically ended up learning through apprenticeship-type techniques once on the job in an ISP.

Where I believe tech companies can help is by pouring their real-world knowledge and experience back into the teaching, and by this I mean moving beyond just the usual collaboration between industry and education and taking it a stage further: Encouraging their staff to go into Universities and teach, and the Universities to do more to guide and nurture this.

Yes, it takes effort. Yes, for the tech firms releasing staff members one or two days a week to teach, it’s going to be a longer term investment, the payback won’t immediately come for three to five years. But once it does arrive, it seems that the gift will keep giving.

For the Universities, they will need to support people who maybe aren’t experienced in teaching and find new ways of making their curriculum flexible enough to keep up.

I see this potentially having multiple positive paybacks:

  • For the University: they are getting some of the most up-to-date and practical real-world knowledge being taught to their students.
  • For the Students: increased employability as a result of relevant teaching.
  • For the Industry: a reducing skills gap and more inquisitive and employable individuals looking for careers.
  • For the tech employee doing the teaching: a massively rewarding experience that can help increase job satisfaction.

For this, I’d like to put my industry colleague Kevin Epperson on a bit of a pedestal. Kevin is an experienced Internet Engineering professional, and has worked at big industry names such as Microsoft, Level 3 and Netflix, in senior Network Engineering leadership roles. Kevin also teaches at University of Colorado, Boulder and is the instructor of their IP Routing Protocols course.

This is something Kevin has been doing for 15 years, during tenures at three different employers, and recently received an award for his contributions to their ITP courses.

The value of having a real-world industry expert involved in instructing students can’t be underestimated, yet I can think of very few people in my part of the tech industry – Internet Engineering – that actually teach in the formalised way Kevin does. All I can really think of is people doing more informal tutoring at conferences, and helping to organise hackathons and other events, which don’t necessarily reach the HE audience we’re discussing in this article.

The industry needs to be willing to move outside the rigid “full-time” staff model that I still find many companies wedded to, despite claims to the contrary, and be confident about releasing their staff to do these things, in order to give back. It may even improve staff retention and satisfaction for them, as well as produce more productive new hires! Be interested to see if people have real stats, or just good stories, on that.

Yet, at the same time, we’re talking about an industry that won’t give many members of staff a day out of the office to attend a free industry event which will benefit their skills and experience and improve their job satisfaction. Why? Who knows? I have hard data on this from running UKNOF that I’m willing to share if folks are interested.

The answer that I don’t have is how receptive the Universities would be to this? Or do they have their own journey of trust to go on?

Net Eng Skills Gap Redux: Entry Routes into Network Engineering

While I was recently at the LINX meeting in London, I ended up having a side-discussion about entry routes into the Internet Engineering industry, and the relatively small amount of new blood coming into the industry.

With my UKNOF Director’s hat on for a moment, we’re concerned about the lack of new faces showing up to our meetings too.

Let me say one thing here and now:

If you work in any sort of digital business, remember that you are nothing without the network, nothing without the infrastructure. This eventually affects you too.

Yes, I know you can just “shove it in the Cloud”, but this has to be built and operated. It has real costs associated with it, and needs real people to keep it healthily developing and running.

I’ve written about this before here, almost 3 years ago. But it seems we’re still not much better off. I think that’s because we’ve not done enough about it.

One twitter correspondent said, “I didn’t know the entry route, so ended up in sysadmin, then internet research, and not netops.”

This pretty much confirmed some of my previous post, that we’d basically destroyed the previous entry route through commoditisation of first-line support, and that was already happening some time around 1998/1999.

It’s too easy to sit here and bleat, blaming “sexy devops” for robbing Net Eng and Network Infrastructure of keen individuals.

But why are things such as devops and more digital and software oriented industries attracting the new entrants?

One comment is that because a large number of network infra companies are well established, there isn’t the same pioneering spirit, nor the same chance to experiment and build, with infrastructure compared to the environment I joined 20 years ago.

My colleague, Paul Thornton, characterised this pioneering spirit in a recent UKNOF presentation titled “None of us knew what we were doing, we made it up as we went along” – note that it is full of jargon and colloquialism, aimed at a specific techie audience, but if you can excuse that, it really captures in a nutshell the mid-90’s Internet engineering environment the likes of he and I grew up in.

Typing “debug all” on a core router can liven up your afternoon no end… But I didn’t really know what I wanted to do back then, I was green and wet behind the ears.

Many infrastructure providers are dominated by obsessions with high-availability, and as a result resistance to change, because they view a stable and available infrastructure as the utopia. An infrastructure which is being changed and experimented upon, by implication, is not as stable.

do-not-touch-any-of-these-wires
DO NOT TOUCH ANY OF THESE WIRES

Has a desire to learn (from mistakes if necessary!) become mutually exclusive from running infrastructure?

In many organisations, the “labs” – the development and staging environments – are pitiful. They often aren’t running the same equipment as that which exists in production, but are cobbled together from various hand-me-down pieces of gear. This means it’s not always possible to compare apples with
apples, or exactly mimic conditions which will exist in production.

Compare this to the software world, where everything is on fairly generic compute, and the software is largely portable from the development and staging environments, especially so in a world of virtualisation and containerisation. There’s more chances to experiment, test, fail, fix and learn in this environment, than there is in an environment where people are discouraged from touching anything for fear of causing an outage.

This means we Network Engineering types need to spend a lot of time on preparation and nerves of steel before making any changes.

Why are the lab environments often found wanting? Classically it’s because of the high capital cost of network gear, which doesn’t directly earn any revenue. It’s harder to get signoff, unless your company has a clear policy about lab infrastructure.

I’m not saying a blanket “change control is bad”, but a hostile don’t touch anything” environment may certainly drive away some of the inquisitive folks who are keen to learn through experimentation.

Coupled with the desire of organisations to achieve high availability with the lowest realistically achievable capital spend, it means that when these organisations hire for Network Engineering posts, they often want seasoned and experienced individuals, sometimes with vendor specific certifications. You know how I hold those in high esteem, or not as the case may be, right?

So what do we need to do?

I can’t take all the credit for this, but it’s partly my own opinions, mixed in with what I’ve aggregated from various discussions.

We need to create clear Network and Infra Engineering apprenticeship and potential career paths.

The “Way In” needs to be clearly signposted, and “what’s in it for you” made obvious.

There needs to be an established and recognised industry standard for the teaching in solid basic network engineering principles, that is distinct from vendor-led accreditations.

In some areas of the sector, the “LAIT” (LINX Accredited Internet Technician) programme is recognised and respected for it’s thoroughness in teaching basic Internet engineering skill, but it’s quite a narrow niche. Is there room to expand the recognition this scheme, and possibly others have?

A learning environment needs to exist where we enable people to make mistakes and learn from them, where failure can be tolerated, and priority placed on teaching and information sharing.

This means changing how we approach running the network. Proper labs. Proper tooling. Proper redundant infrastructure. No hostile “change control” environment.

Possibly running more outreach events that are easier for the curious and inquisitive to get into? That’s a whole post in itself. Stay tuned.