NTA - Notice history

RFO Friday 11th July

10:13 A  router card failure was identified at one of our Datacentres in Enfield. Although the router failure was only for 10 minutes and returned to service at 10:23, it caused an issue with the routing of calls whereby either the call was terminated or it made the call became unanswerable.  
10:50  After identifying the cause of the failure, the affected card was removed from service and the BLFs were disabled, to allow the routing to stabilise.
11.42 The fault was resolved, with services being fully restored, apart from BLFs, due to the load that would suddenly be added to the servers

Weekend of 12.07.25 to 13.07.25 - BLFs were switched back on and all services to the network made fully active

To prevent this incident from reoccurring, the affected router will be replaced on the weekend starting 19.07.25

This incident has been resolved and the BLF lights have been re-enabled. An RFO will be issued later

The outage has been identified as a network issue, which is being fully investigated. All services have been restored to normal, apart from the BLFs, which will be restored later today. An RFO will be issued following the investigation has been completed

BLF's have temporarily been disabled while service is fully restored

Whilst we are aware there are a few extensions with problems still, we will resolve those shortly. Please see below the RFO

In the new implementation of our Cloud Phonebook we decided to use some proxies to improve its availability and also add a few new features. We opted for using HAProxy, a well-known open-source project, that has been in use for many years (it is responsible for handling provisioning, wallboards, MobeX plus many internal projects). We also developed an LDAP Proxy to allow us to add a cache to improve the responses and sorting (something that models like C30 and below cannot do).

The solution was tested internally and worked fine. Unfortunately, while moving it to production, we started to receive reports that the Cloud Phonebook was failing and some devices (A10, B20 plus C30) were rebooting.

After the initial investigation, we found that some of the behaviours were causing the connections to be denied, so we improved it to handle an increase in traffic that we couldn't reproduce during the internal tests. However, the rebooting of the devices was unexpected and the only real hypothesis was that something in the middle between the LDAP servers and the devices was creating packets that the devices couldn't handle.

Initially, we decided to stop using the LDAP Proxy as it could be sending the wrong packets which started to reprovision the devices. This is a process that could take several hours because it would not be good to have all the devices coming to the system at the same time. 

After the C30 devices were reprovisioned, we kept monitoring the systems and took notes of reports. There was also a possibility that the devices that were still rebooting were the ones we could not reach for reprovisioning because they were in use at the time that we sent the requests.

After more internal checks we noticed that this was not the case, so the only other piece of software that could be causing the reboots was the HAProxy itself. The fact that it was affecting only the old Hosted models probably meant that they couldn't handle some packets that were being sent to them.

We started to reprovision all devices to connect directly to the LDAP servers, avoiding any proxies in the middle, after this was done no more reboots were detected. This means that we can't use the HAProxy with the LDAP servers.

However, it doesn't mean that we will not be implementing our LDAP Proxy again in the near future, since the LDAP Proxy was behind the HAProxy, it is possible that the LDAP Proxy was not failing, but just facing the same issues with the HAProxy too.

This implementation will be done in a slow-paced rollout to avoid causing more issues with the cloud phonebook access in the future. The new LDAP Proxy is a really important part of our new cloud phonebook strategy because it will make the Dir requests much quicker and it'll add sorting support for models A, B plus C30.

We really do apologize for this unexpected issue and will keep you updated on our plans in the future

Yealink handsets will be reprovisioned after 18:00 tonight which should resolve the ringtone issues

We implemented a fix and are currently monitoring the result. We are sending the connections directly to the LDAP server therefore bypassing the haproxy now and removing anything obstruction in the chain

We will be reprovisioning the C30 devices, after that it will be A, B, E, F, and then C33 and Ds which will take one hour per group

After this,the number of entries in the Cloud Phonebook will be worked on followed by the ringtones