Telehouse North to Virtus Enfield link down
Incident Report for EX Networks
Resolved
After a period of stability, we have put this link back into service. This incident has now been resolved and full redundancy has been restored.
Posted Feb 16, 2019 - 11:04 GMT
Monitoring
The fibre has been repaired and connectivity has been restored.

We will be keeping traffic away from the link for the moment to ensure the link is stable and free of errors before bringing it back into full service.

All systems remain stable however the site should be considered "AT RISK" during this time.

EXN have also implemented a change freeze on devices in Enfield and Telehouse East to avoid any potential disruption to the working leg.
Posted Feb 15, 2019 - 21:10 GMT
Update
Our fibre provider has stated:

"Current Update: Crews are onsite splicing at this time. We are starting to see services restore at this time. ETR is 30 minutes."

All systems remain stable however the site should be considered "AT RISK" during this time.

EXN have also implemented a change freeze on devices in Enfield and Telehouse East to avoid any potential disruption to the working leg.
Posted Feb 15, 2019 - 19:11 GMT
Update
Our fibre provider has stated:

"Current Update: Our engineers are continuing to work to reroute services away from the damaged fibre. We are in regular contact with our field engineers and advise we are nearing a resolution. We sincerely apologise for the delay in resolving this incident and hope to bring your service back up soon."

All systems remain stable however the site should be considered "AT RISK" during this time.

EXN have also implemented a change freeze on devices in Enfield and Telehouse East to avoid any potential disruption to the working leg.
Posted Feb 15, 2019 - 17:22 GMT
Update
Our fibre provider has stated:

"Current Update: Technicians have located the break. The location of the fibre break is a construction site in the London Olympic park area. Our 3rd party provider engineers are testing fibres either side of the next available joints."

All systems remain stable however the site should be considered "AT RISK" during this time.

EXN have also implemented a change freeze on devices in Enfield and Telehouse East to avoid any potential disruption to the working leg.
Posted Feb 15, 2019 - 15:50 GMT
Update
Our fibre provider has stated:

"OSP has now confirmed their 2nd Crew of engineers are at the break location investigating. It is believed fiber damage has been caused by contractors/engineering work. OSP is also exploring reroute options. Thank you for your patience. More updates will be provided as they become available"

All systems remain stable however the site should be considered "AT RISK" during this time.

EXN have also implemented a change freeze on devices in Enfield and Telehouse East to avoid any potential disruption to the working leg.
Posted Feb 15, 2019 - 11:18 GMT
Update
Our fibre provider has stated:

"Fiber test crews have provided and ETA of 10:00am GMT. We will provide more updates as they are provided."

All systems remain stable however the site should be considered "AT RISK" during this time.

EXN have also implemented a change freeze on devices in Enfield and Telehouse East to avoid any potential disruption to the working leg.
Posted Feb 15, 2019 - 09:04 GMT
Update
Our fibre provider has stated:

"OSP are having trouble locating the damage. They are still working to narrow the exact joint where the damage is located. We have escalated to their management team to try to get further progress. We have also notified our management team of the delays in locating the break. OSP continue to work towards fault isolation at this time and we will continue to press for updates and progress. More updates to follow as events unfold."

All systems remain stable however the site should be considered "AT RISK" during this time.
Posted Feb 15, 2019 - 05:42 GMT
Identified
Our fibre provider has stated:

"We have located a fibre fault 9 KM from Telehouse. Crews have been dispatched for the repair."

All systems remain stable however the site should be considered "AT RISK" during this time.
Posted Feb 14, 2019 - 23:01 GMT
Update
Our fibre provider has stated:

"Current Update: We have dispatched technicians to test and locate the fibre damage. We have just received word that access has been approved and we are now waiting for technicians to arrive onsite."

All systems remain stable however the site should be considered "AT RISK" during this time.
Posted Feb 14, 2019 - 21:45 GMT
Update
Our fibre provider has stated:

"We have reached out to our technicians and they are working on getting technicians dispatched. We have multiple circuits down in the area and are trying to verify a fibre cut."

All systems remain stable however the site should be considered "AT RISK" during this time.
Posted Feb 14, 2019 - 19:22 GMT
Update
We are continuing to investigate this issue.
Posted Feb 14, 2019 - 19:21 GMT
Update
Our fibre provider is investigating the issue. Next update in approx 1 hour. All systems remain stable however the site should be considered "AT RISK" during this time.
Posted Feb 14, 2019 - 18:19 GMT
Update
Our fibre provider is currently investigating the issue. All systems remain stable however the site should be considered "AT RISK" during this time.
Posted Feb 14, 2019 - 16:29 GMT
Investigating
At 16:01 today EXN lost connectivity between Telehouse North and Virtus Enfield. All services have failed over to link between Telehouse East and Virtus Enfield.

At this stage the Virtus Enfield should be considered "AT RISK".
Posted Feb 14, 2019 - 16:07 GMT
This incident affected: DC Connect, IP Transit Services, Virtual Data Centre (vDC) - Virtus Enfield and Datacentres (Virtus LON1 Enfield).