Mass service outage - Gamma SIP services
Incident Report for M12 Solutions and Giganet
Resolved
This incident has been resolved.
Posted Feb 14, 2020 - 16:05 GMT
Update
This incident has been resolved, we are waiting on a reason for outage and will update accordingly.
Posted Feb 12, 2020 - 17:30 GMT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 01, 2020 - 06:24 GMT
Update
6:10am - The fibre issue in the London area has now been resolved as of 05:01, our monitoring and testing has confirmed that service is restoring across Gamma platforms and we will continue to monitor network stability throughout this morning.

We apologise for any disruption this incident has caused to your service.

01/02 1am- Work on the stabilisation of re-routed traffic across the temporary network connection remains a challenge and we have engaged our technology partner to work alongside us on this. This will continue to be a priority throughout the evening, but unfortunately we can’t yet provide an outline timescale on when this will complete.

Connection to our Gamma Portal has now been restored and we have tested the number divert capability and this is functioning correctly.

Fibre engineers have identified the break between two fibre joints in the London area and are investigating the options to replace this with a local spare. Once we know the availability of a spare fibre we hope to be able to provide an estimated time to restore.

The next update will be at 08:00, but any material changes to the status will be updated before then if they become available.

01/02 00:00- The network routing change is being implemented and we are starting to move network traffic across a newly established network connection. As we stabilise voice and data traffic, customers will continue to see disruption to service. Our engineering team will continue to focus on this activity as a priority. We can’t yet provide a timeframe to complete but will do so at the earliest opportunity.

Fibre engineers have now isolated the fault in the London area to a smaller patch and are in the process of testing fibre sections in that patch.

Next update will be by 01:00 or sooner if we can.

11pm - The fibre engineers are in the process of isolating the fibre break in the London area and we are expecting a clear view on the fibre restoration before 00:30. We have encountered further technical challenges on the network routing change to restore services in the interim, this was planned for 22:00 but is still work in progress and we will provide a further update on that change before 00:00.

The next update will be provided before 00:00.

We apologise for the ongoing disruption and all of our resources remain committed to restoring service as soon as we can.
Posted Feb 01, 2020 - 06:24 GMT
Update
Fibre engineers are now on site and beginning work on the restoration. Our network team have been continuing the work on the network re-routing of voice and connectivity services and we expect to restore those services by 10pm tonight. We’re confident the re-routing work will fully restore services, however services remain at risk until the fibre restoration work is completed.
Posted Jan 31, 2020 - 21:44 GMT
Update
We now estimate engineers to arrive at the fibre break in the London area at 9pm at which point they can assess the extent of the work required to restore the fibre connectivity. Work will begin at 9pm on fibre break within the Leicester area, however we know the restoration work here is extensive due to an electrical incident in that area. We have taken steps within our network to re-route voice and connectivity services and this action will restore the majority of services, but we expect some ongoing intermittent problems on voice quality and connectivity speed until at one of the two fibre breaks is resolved. The next update will be provided at 21:30.
Posted Jan 31, 2020 - 20:06 GMT
Update
We have isolated the fault to a dual fibre break on the East coast and the London area of our national network. Our technology partner has been engaged and they are deploying engineers to address both breaks. The incident continues to impact our voice and connectivity services and we are taking mitigating action at a network level to try to restore connectivity where we can.

We expect to be able to provide a clearer view on time to resolve by 20:00 hours as deployed engineers establish the extent of the work at each break.
Posted Jan 31, 2020 - 19:43 GMT
Update
We are currently experiencing a Gamma national network fibre break which is impacting Voice and Data connectivity services. Customers will be experiencing voice connectivity and call quality issues due to the congestion as a result of the fibre break.

Gamma engineers are currently working to identify the actual cause of the fibre break at the moment whilst in parallel we are working to review alternative traffic routing measures to restore connectivity and alleviate traffic congestion.

We apologise for any inconvenience caused and will provide a further update within 1 hour.
Posted Jan 31, 2020 - 17:27 GMT
Identified
We are currently aware of a mass service outage (MSO) affecting some Gamma SIP trunk services.

Start Time: 31/01/2020 15:00

SIP Session Border Controllers Affected: N/A

Service Impact: Poor call quality, packet loss, complete loss of service.

Further updates will be provided as we learn more.

We apologise for any inconvenience this outage causes.
Posted Jan 31, 2020 - 15:49 GMT
This incident affected: Wholesale Providers (Gamma SIP Trunks).