Last month, the Elizabeth line suffered two days of problems when the signalling system broke down, and now a clearer timeline of what happened is emerging.
The first inklings that there might be a problem came early at 5am when the Elizabeth line logged intermittent problems with the communications system, but it wasn’t expected to affect passenger services.
26th November 2024
5:15am – the first train departed the Plumstead depot heading towards Paddington, and difficulties were reported communicating with the signalling system, with the service dropping out and restoring repeatedly over the next 25 minutes.
5:44am – Staff reset the system, restoring the communications links, only for it to drop out again 10 minutes later.
6:20am—The decision is taken to close the Elizabeth line’s central core tunnels between Abbey Wood and Paddington. The justification is passenger safety if trains unexpectedly break down in tunnels, especially if there’s overcrowding and a delay in evacuating them. The rest of the network east and west of central London runs under “Contingency COS 0”, which is from Paddington and Liverpool Street mainline stations rather than the tunnel platforms.
7am – Trains were still completing journeys and running empty to depot when they lost connection with the overhead line monitors and the ventilation systems, and then they lost the entire core signalling system.
7:50am – An internal significant incident report said that trains were having problems communicating within the core signalling area (CBTC) while train managers couldn’t see the headcodes on their network maps (LWOD). The train managers were also having to control trains manually due to problems with the Automatic Route Setting (ARS).
8:50am – Drivers also reporting problems with the GSMR radio links, so a full reset of the Elizabeth line takes place.
10:30am – TfL issues statement confirming that they are working on fixing the signalling fault.
10:45am – An internal memo confirms that Siemens had been unable to diagnose the root cause of the signalling problem. A team of specialists hopes to have a resolution by 11:30am. By now, there’s little confidence that they can recover the service by that evening, and plans are being put in place to see if they can run additional services on the Network Rail sides of the line.
Lunchtime The engineers had recovered some of the system but were still having problems getting it fully operational.
1:38pm – Internal memo update on the problems reports that they can restore service, but it drops out again after about 5 minutes. The software engineers were also having problems with computer firewalls regularly locking them out of the network. They now suspect that the firewall itself might be the problem.
Early afternoon – Two more reboots took place, but still couldn’t fully recover the communications network.
3:30pm — A conference call between Siemens and Elizabeth line engineers suggests they have identified the problem. If it works, they might be able to get the line running test trains by 7pm.
5:47pm — An internal memo reports that Siemens is amending some 30 systems to fix the problem. There’s also the possibility that physical equipment at Canary Wharf has broken, which would need to be replaced, delaying the reopening of the line.
7pm – The network is running again, and some test trains were run through the core tunnels to ensure the signalling network was stable. Testing started with one train in each direction along the core tunnels, and was upgraded to a 6 trains per hour service at around 7:30pm.
8:30pm – Conference call to decide if the service can reopen to passengers and if it will open tomorrow as well.
9:45pm – The Elizabeth line reopens to passengers with six trains per hour for the rest of the day.
A bad day, but at least the problems were fixed.
Or so they thought.
27th November 2024
3:30am – The signalling system fails again.
4:45am – Decision taken not to reopen the Elizabeth line that morning.
8am – TfL apologises for the problems and confirms they’re working with Siemens on a fix.
8:30am — The problem is identified and fixed, and they start running some empty test trains through the tunnels to check the service is reliable.
9:12am – Internal memo says that test trains are running reliably and they aim to reopen the line at 11am, but initially the core stations will be exit only until they reach at least 6 trains per hour through the core tunnels.
The decision to open central stations as exit only was taken to manage crowds, as there can be concerns that people might be taken ill (PIOT in the jargon) if there’s overcrowding on the first few trains — which, while obviously bad for the passenger affected, also slows down the service recovery as trains linger longer at stations for first aid to arrive.
11am – The service is open to passengers, but there are fewer trains per hour and delays while trains and drivers are resynced with the timetable.
2pm – Full service restored.
And just as staff started to think they might be able to calm down after a frantic couple of days, Network Rail found a cracked rail crossing between Manor Park and Forest Gate, requiring trains to slow down until the fault was fixed.