NTA - Notice history

RFO 27/06/22

Whilst investigating the root cause of last week's faulty flapping link, the link was accidentally brought into service by our upstream suppliers and caused the issues to re-occur.

This was identified and quickly removed from service again and service restored.

However, as a result of restoring everything to a normal operating situation, a further shutdown was implemented to enable full functionality services to resume.

We are working closely with our suppliers to make sure changes are never implemented, or any attempts to do so happen during an investigation, that may interfere with the normal operation of our services during normal working hours unless specifically instructed.

This will only be in the case where it would be operationally critical.

Any network changes will continue to be scheduled and notifications sent as per our normal procedure

We just resolved the issue!

We implemented a fix and currently monitoring the result.

Whilst investigating the cause of last week's faulty flapping link, the link was accidentally brought into service by our upstream suppliers and caused the issues to re-occur. This was removed from service again and service restored. We are still seeing some issues with calls made during the outage being "Stuck" we are working on this issue.

We are seeing more instances of service affecting issues with inbound and outbound calls, we are continuing to investigate this situation and work to bring service back.

Whilst investigating the cause of last week's faulty flapping link, the link was accidentally brought into service by our upstream suppliers and caused the issues to re-occur. This was removed from service again and service restored. We are still seeing some issues with calls made during the outage being "Stuck" we are working on this issue.

RFO 16/06/22

We became aware of a number of issues affecting us and many other providers too. This had an impact on our service and therefore we took emergency action to mitigate the ongoing issues. We engaged our Disaster Recovery system and rerouted in excess of 8500 inbound calls each with a duration greater than 10 seconds. To alleviate the issues we disabled our hosted platform and BLF notifications temporarily, which enabled outbound and internal calls to be made. Once we were confident the issues were resolved we fully restored our services.

An example of a couple of issues from suppliers:- DWS:- Dear partner, Hi To advise many services were affected today by an issue with a peering company lumin/lvl3. This is a peering partner who deals with hand-off traffic to the wider net for many suppliers. Between 13:15 and 13:45 there was a period of no traffic passing over their network. This was resolved by passing peering traffic to a different peer partner. This is not a DWS specific issue and likely affected multiple suppliers within the UK.

June 16: Problems at BT (British Telecom) BT (British Telecom) is having issues since 09:00 AM GMT. Are you also affected? Leave a message in the comments section! We apologise for any inconvenience caused.

All services have now returned to normal and an RFO will be issued tomorrow after further in-depth investigation.

We have now restored service to the portal and BLFs. We are continuing to monitor and investigate the issues that led to today's outage.

We have re-established inbound calls, some services such as the portal and BLF's are still disabled as we continue to monitor the issue. We will continue to provide updates as these services are re-established.

To advise many services were affected today by an issue with a peering company lumin/lvl3.

This is a peering partner who deals with hand-off traffic to the wider net for many suppliers. We cannot confirm this is the reason for our outage, but we are led to believe it has had an effect.

We are continuing to monitor this issue, outbound and internal calls should be fine. Inbound calls are still being routed via disaster recovery to help reduce load. We have also temporarily disabled BLFs.

We are continuing to slowly restore services and keeping the load on our servers low in order to ensure a stable recovery. You may notice some service restoring slowly, with the exception of the portal.

We are currently investigating this incident. You may experience calls intermittently working as we try to bring our service back.

The Web interface/portal has been temporarily disabled to reduce the load on our servers to enable calls to be re-established. Please note we went into full disaster recovery mode 25 minutes ago. We hope to have this issue resolved shortly.

We are pleased to announce that as of 20:30 on 19.05.22 that the issue that was affecting some of our broadband suppliers Global Switch location has been fully resolved. We again apologise for any inconvenience caused.

At 1pm

STREAM: Troubleshooting

• Isolate G52 Switch 2 from the Network - Completed

• G52 Switch 1 to be configured for core routing - Completed

• Bring up all LAN interfaces on G52 Switch 1 individually to return service - In Progress

Next Status:

• Bring up all the services/connections

• Pending vendor and KCOM investigation on the 2 CAS devices

Current Status: Most servers are now back online but we will keep monitoring until we get a FRO

An update from our supplier.

A major service outage has been detected at Global Switch affecting connectivity, internal systems and portals. Engineers are on route now to investigate

MobeX services are now back up and running and we will continue to look into the calls. We apologise for any inconvenience.

The broadband issue has now stabilised, however we will keep on monitoring the services and will update you with an RFO once we receive one

We have now been informed that the issue has been resolved.