RFO for Mobex Incident 25th-26th July

After a thorough investigation, it has been discovered that the issue was caused by an extension running MobeX Web that had a connectivity issue and was trying to authenticate repeatedly causing an overload on the web servers, culminating in the partial MobeX failure.

In order to prevent this from happening there will be an amendment made to a rule on MobeX Web to logout any device if the number of failed authentications is too high.

At 10:26 on 03.07.23 it was identified that one unexpected behaviour overloaded our virtual servers. Once the issue was noticed the system moved automatically to disaster recovery. Due to the nature of the fault, the system was not completely offline, but it was kept under Disaster recovery for a few minutes to ensure that the issues were resolved. The system recovered itself by 10:37 and at 10:48 the issue was completely fixed.

We later had an issue with our upstream carrier which caused outbound calls to certain destinations to fail: We were seeing an increase in errors to some transited UK fixed and mobile destinations that resulted in us having to take a major downstream carrier out of route.

We implemented a fix and are currently monitoring the result.

We are experiencing issues with one of our upstream providers, and we are experiencing some issues relating to this morning's outage. We are working to resolve these.