All systems operational

Inbound Calls
Operational
Outbound Calls
Operational
Web Interface
Operational
Busy Lamps
Operational
API
Operational
Secondary Services
Operational
Handset Provisioning
Operational
Wallboards
Operational
UC Integration
Operational
Dialer
Operational
MobeX Mobile
Operational
MobeX Softphone
Operational
NTA Homepage
Operational
Broadband Services
Operational
Lease Lines
Operational
Private DSL
Operational
Wireless Internet
Operational

No Service Impact Expected

Operational

Notice history

May 2024

No notices reported this month

Apr 2024

Mar 2024

Network issue
  • Resolved
    Resolved

    Incident Resolution and Future Plans

    On the morning of 25/03, we encountered issues with our SIP server as multiple extensions were unable to register due to a memory leak. After migrating to our secondary server and rebooting the main server, a deadlock situation arose, preventing either server from providing services. To address this, we performed a quick reboot of both servers, with our Disaster Recovery solution effectively redirecting inbound calls during the process.

    Upon bringing the SIP servers back online, we encountered a recurrence of the issues due to an overwhelming number of BLF requests bombarding the server, causing system instability. This highlighted the need for a more robust solution to handle such scenarios in the future.

    We are currently in the process of updating the SIP server and transitioning to a more advanced version of our redundancy software, with a particular focus on efficiently handling BLF requests. As a result, the new BLF approach and the updated SIP servers will take precedence as critical projects, undergoing extensive testing and meticulous attention.

    We plan to implement these updates on a Saturday night after 10PM to minimize potential minor instability during the transition, and we will provide advance notification before proceeding. We sincerely apologize for the inconvenience caused and assure you that we are leveraging this experience to enhance the redundancy and reliability of our systems, delivering a solution that meets your expectations.

  • Monitoring
    Update

    We now believe the issue to be resolved but will keep monitoring. An RFO will be issued in due course

  • Monitoring
    Update

    We implemented a fix and are currently monitoring the result. There are a few calls that are stuck and are being cleared as soon as possible

  • Monitoring
    Monitoring

    The system is now back and working, however, the BLF function will remain off until this evening so that it can be brought back online when the traffic is lower and the network has fully stabilised

  • Identified
    Update

    Unfortunately the resolution found has not cleared the issue. We are working to resolve the issue and apologise again for any inconvenience

  • Identified
    Update

    We are continuing to work on a fix for this incident. The network is slowly returning to normal and hopefully will be fully functioning shortly

  • Identified
    Identified

    The BLF monitoring will shortly be disabled, the connections will then be momentarily dropped and then the servers restarted to resolve the issue

  • Investigating
    Investigating

    We are aware of an issue affecting our network. We are currently working to resolve this and apologise for any issue this is causing you

RFO and Update for 25.03.24
  • Resolved
    Resolved

    After the recent system outage on 25/03, we encountered a surge of requests from our users, particularly regarding BLF subscriptions. This uptake in BLF usage has been remarkable, showcasing the growing demand for our services. Recognising this shift, we developed a new and robust solution to effectively manage the current and future influx of BLFs within our system.

    Incremental Implementation for Enhanced Performance


    Beginning on 13/04, we initiated the implementation of our new solutions daily, carefully orchestrating this process during off-peak hours to ensure seamless service for our users. While this approach initially posed challenges in testing the solution in real-time scenarios, it allowed us to meticulously fine-tune our processes, guaranteeing a smooth transition without compromising the quality of our services.

    Confidence in Future Network Growth

    Following several days of meticulous development and deployment, we are now extremely confident in our ability to accommodate the anticipated growth of our network. Our new solution, a robust distributed cluster of servers dedicated solely to handling BLF subscriptions, marks a pivotal advancement.
    By segregating BLFs from other SIP packets and enabling on-demand server scaling, we have fortified our infrastructure to swiftly adapt to any unforeseen traffic spikes and effectively reduce any potential "DDOS" attacks.

    We are proud to have established an enhanced system that ensures the seamless management of BLFs and provides a secure and adaptable framework for our network's future growth.

Mar 2024 to May 2024