GS2 Power Issue
Incident Report for EX Networks
Resolved
Global Switch remains on conditioned DRUPS. At 08:17 this morning (26th July 2017) an external power quality issue (Frequency deviation) was experienced on the UK National Grid. This frequency deviation was within the trigger parameters of the H Powerstation and caused the system to switch to emergency mode (load on engines). The engines carried the load for the standard 10 minute period for mains stabilisation and then reverted to normal operation on conditioned power. The DRUPS performed as expected throughout the event with no alarms initiated. Now N+2 resilience has been achieved this Major Incident will now be closed. EXN is requesting a timeline for the full RCA and we will contact affected customers individually.
Posted Jul 26, 2017 - 12:37 BST
Update
Synchronisation works to transfer load have been completed without incident, H Powerstation has now been returned to Normal Operation of Conditioned Power. EXN will continue to work with Global Switch to monitor the Powerstation status overnight and provide an update further updates tomorrow.
Posted Jul 25, 2017 - 19:55 BST
Update
All testing on the H Powerstation has now been completed. DRUPS, in an N+2 configuration, are in the process of being transferred back to the normal running output bar (Bar A) of the H Powerstation. This is being done in preparation for migration back to normal operational mode (Conditioned DRUPS power). Works to transfer load will commence at 19:00 today, 25/07/17. Synchronisation is expected to take up to 15 minutes to complete.
Posted Jul 25, 2017 - 17:11 BST
Update
Confidence testing of DRUPS #3 has progressed well and we are now moving into further testing phases to include a full load run.
Posted Jul 25, 2017 - 10:29 BST
Update
Balancing of DRUPS 3 is continuing and progressing well. However, works are taking longer to complete than previously thought but with no issues to report. Updates will continue throughout the process at regular intervals or for significant information updates.
Posted Jul 24, 2017 - 22:31 BST
Update
Balancing of DRUPS 3 is continuing and progress well. Balancing will continue throughout the day. Balancing is expected to complete later this evening, subject to tests continuing to go well. DRUPS 9 is now currently under performance checks.
Posted Jul 24, 2017 - 16:00 BST
Update
H Powerstation remains on utility mains. Balancing of DRUPS 3 is continuing and progress well. Balancing will continue throughout the day. Balancing is expected to complete later this evening, subject to tests continuing to go well. Updates will be sent if this timeframe changes.
Posted Jul 24, 2017 - 13:12 BST
Update
Initial balancing and confidence testing on DRUPS 3 is now underway and progressing well. Balancing will continue and is expected to take up to Monday evening to complete. However, this may be improved.
Posted Jul 23, 2017 - 11:49 BST
Update
DRUPS 3 has been reinstated. The defective shield cover has been repaired and the unit is being prepared for balancing, these preparations include an off load run to warm up all the moving parts, the vibration will then be re-measured and the initial test weights applied. Balancing will continue and is expected to take up to Monday evening to complete.
Posted Jul 22, 2017 - 22:21 BST
Update
H Powerstation remains on utility mains. DRUPS #2 has been balanced and initial load testing has been completed successfully. Further tests are continuing. DRUPS #3 Works are progressing well as per schedule and the unit should be ready for retesting, rebalancing and recommissioning later on today . Further updates to follow. Updates from UK Power Networks will now be reported by exception with confirmation given by the operator that there are no planned switching actions scheduled on the Network for the next 6 months. Regular checks will continue with the Met Office to confirm if there are any adverse weather issues to report. Updates will continue throughout the process.
Posted Jul 22, 2017 - 14:30 BST
Update
H Powerstation remains on utility mains. The confidence testing on DRUPS #2 is continuing with the unit being connected to a load bank to simulate operational conditions Works have been completed overnight to relocate the rotor shaft for DRUPS #3. It is now being installed and reconnected to the main unit in readiness for retesting, rebalancing and recommissioning. Works will continue throughout the day with further updates to follow. Updates from UK Power Networks will now be reported by exception with confirmation given by the operator that there are no planned switching actions scheduled on the Network for the next 6 months. Regular checks will continue with the Met Office to confirm if there are any adverse weather issues to report.
Posted Jul 21, 2017 - 11:44 BST
Update
H Powerstation remains on utility mains. The confidence testing on DRUPS #2 is continuing with the unit being connected to a load bank to simulate operational conditions. Works have been completed overnight to relocate the rotor shaft for DRUPS #3. It is now being installed and reconnected to the main unit in readiness for retesting, rebalancing and recommissioning. Works will continue throughout the day with further updates to follow. Updates from UK Power Networks will now be reported by exception with confirmation given by the operator that there are no planned switching actions scheduled on the Network for the next 6 months. Regular checks will continue with the Met Office to confirm if there are any adverse weather issues to report.
Posted Jul 21, 2017 - 09:24 BST
Update
H Powerstation remains on utility mains. The re-balancing phase on DRUPS#2 has been completed and the next phase of confidence testing has commenced. This will continue throughout the night, the rotor shaft engineering works on DRUPS#3 have been completed and the unit has just arrived back on site. The unit is being unloaded and preparations are being made for it to be transported to the DRUPS garage for re-installation, re-testing and commissioning. Checks continue to be made with UK Power Networks to confirm there are no planned switching events on the National Grid. The Met Office confirm there are no adverse weather issues to report at this time.
Posted Jul 20, 2017 - 21:25 BST
Update
H Powerstation remains on utility mains. Testing continues to be performed as previously advised. Checks continue to be made with UK Power Networks to confirm there are no planned switching events on the National Grid. The Met Office confirm there are no adverse weather issues to report at this time.
Posted Jul 20, 2017 - 09:30 BST
Update
H Powerstation remains on utility mains. Re-balancing continues to be performed on DRUPS #2. The time taken for these works is taking longer than originally calculated as the process has been refined to help mitigate against the possibility of any issues when the unit is ready to be reinstated. There are no issues to report at this time whilst rebalancing is performed. DRUPS #3 is under test and review at the factory in Birmingham with a condition report to be received. The supplier has committed to continue working around the clock to complete these works and have the unit returned to site as soon as possible. Checks continue to be made with UK Power Networks to confirm there are no planned switching events on the National Grid. The Met Office confirm there are no adverse weather issues to report at this time
Posted Jul 19, 2017 - 21:47 BST
Update
H Powerstation remains on utility mains. Global Switch continues to monitor predicted adverse weather conditions in the area with the Met Office. We can confirm that the lightning storms of last night did not cause any impact to power supplies on the H Powerstation. Work has progressed well over night and is nearing completion in relation to having a serviceable set of DRUPS in an N+1 configuration. A 7th unit is en route to an engineering workshop in Birmingham with the expectation that it will be returned to site on Friday, ready for reinstallation, testing and reinstatement of the Powerstation in an N+2 configuration. Early indications are that we will be in a position to do so at the weekend but this is still to be confirmed. As the situation progresses, further updates will be provided in this regard with advance notice of full reinstatement to be issued.
Posted Jul 19, 2017 - 17:54 BST
Update
H Powerstation remains on utility mains. Global Switch continues to monitor predicted adverse weather conditions in the area with the Met Office. We can confirm that the lightning storms of last night did not cause any impact to power supplies on the H Powerstation Work has progressed well over night and is nearing completion in relation to having a serviceable set of DRUPS in an N+1 configuration A 7th unit is en route to an engineering workshop in Birmingham with the expectation that it will be returned to site on Friday, ready for reinstallation, testing and reinstatement of the Powerstation in an N+2 configuration. Early indications are that we will be in a position to do so at the weekend but this is still to be confirmed. As the situation progresses, further updates will be provided in this regard with advance notice of full reinstatement to be issued.
Posted Jul 19, 2017 - 14:33 BST
Update
H Powerstation remains on utility mains. Global Switch continues to monitor predicted adverse weather conditions in the area with the Met Office with no issues to report currently. Utility supplier UKPN are being contacted every hour to confirm no planned works are intended for the supply network. Testing is nearing completion in an effort to reinstate H Powerstation in a minimum N+1 configuration.
Posted Jul 19, 2017 - 10:46 BST
Update
H Powerstation remains on utility mains. Reliability testing continues with a view to re-instate the H Powerstation in an N+1 configuration. The decision to reinstate will be determined by the successful validation of all 6 units. H2 DRUPS is being re-balanced at the moment with a view to complete by midday tomorrow at the latest, although Global Switch hope to improve on this.
Posted Jul 18, 2017 - 22:11 BST
Update
H Powerstation remains on utility mains. Reliability testing continues with a view to reinstatement of H Powerstation in an N+1 configuration. The decision to reinstate will be determined by the successful validation of all 6 units. We are expecting the next update from Global Switch at 20:00
Posted Jul 18, 2017 - 16:44 BST
Update
H Powerstation remains on utility mains. The 5 No. DRUPS are still in operational mode and reliability testing for the DRUPS is still in progressing. Balancing works continue on DRUPS#2 with a target for return into service within the next 24 hours. The intention is to return to normal “conditioned power” at N+1 at this time.

In parallel to this activity DRUPS #3 is being removed from site to effect repairs at a mechanical engineering workshop in Birmingham, UK. Works will continue round the clock for an estimated return to site of Friday 21st July, evening. At which time the unit will be reintroduced to the support infrastructure with the intent to return to N+2 configuration.
Posted Jul 18, 2017 - 11:29 BST
Update
Remedial repair works have been completed. The 5 DRUPS that were available prior to the incident have been returned to operational mode but remain in bypass. Whilst in bypass, all units are being subjected to reliability testing for prolonged durations, connected to an external load bank. A further DRUPS unit is in the process of being brought into service. Until this is completed and the unit is available, we will continue to run in bypass with the units under test. The units will not be fully returned to service until the 6th unit has been fully tested and alarm free at which time we will provide advance notice before any change of status to the H Powerstation.
Posted Jul 18, 2017 - 09:16 BST
Update
Power remains stable. Initial reports indicate further DRUPS failures and a break to mains. Investigations continue and we have been told all available resources are currently deployed to determine the cause of the incident and to review remedial actions. We await further updates from Global Switch.
Posted Jul 17, 2017 - 18:23 BST
Monitoring
Power has remained stable, we are still waiting for official word from Global Switch on the issue. Further updates to follow.
Posted Jul 17, 2017 - 16:36 BST
Update
Upon investigation, all customer PDUs were successfully powered back on at approximately 15:36. Any colocation customers still experiencing issues in GS2 should raise a ticket with our support team in the usual way.
Posted Jul 17, 2017 - 16:13 BST
Update
The power outage appears to be isolated to the 10th floor suite (7th floor is not affected) and power has returned. Network services are back online at this time and our engineers are bringing up our VMware platform.
Posted Jul 17, 2017 - 16:00 BST
Investigating
We are currently investigating a power outage within Global Switch 2. Further updates to follow.
Posted Jul 17, 2017 - 15:48 BST