SingNet fined S$500,000 over last year’s 24-hour fiber outage

Singapore’s Infocomm Media Development Authority (IMDA) has fined Singtel ISP subsidiary SingNet S$500,000 (close to $373,000) for a fiber broadband service disruption in December that turned out to be self-inflicted.

The incident, which occurred on December 3 last year, affected close to 90% of SingNet’s fiber broadband subscribers and knocked out broadband services for close to 24 hours.

An IMDA investigation into the outage found that the the service outage was triggered by planned maintenance by SingNet on its Dynamic Host Control Protocol (DHCP) servers.

The maintenance activity, which was intended to install security patches, overloaded the DHCP servers, which made them unable to process subscriber requests for IP addresses – which in turn made those subscribers unable to access the Internet.

However, the IMDA said the problem wasn’t just the overload incident, but the fact that SingNet should have seen it coming:

Specifically, the utilisation rate of SingNet’s DHCP servers had been steadily increasing and were running at 80-90% levels even before the incident.

SingNet had failed to take prompt action to address the high utilisation loads before the incident. In view of the high utilisation levels, SingNet should have exercised greater due diligence and caution when planning the installation of security patches for its DHCP servers to prevent the overload.

While the S$500,000 penalty is stiff, it could have been worse – in May 2014, Singtel was fined a record S$6 million for a major service outage caused by an exchange fire in October 2013 that affected close to 270,000 subscribers in the western and northern parts of the island.

In calculating the S$500,000 fine, the regulator said it took into account “mitigating factors” such as cooperating with IMDA during the investigation, taking preventive measures to make sure it doesn’t happen again, and the compensation SingNet offered to affected customers.

During the outage, Singtel said it would waive mobile data charges for fiber customers who also use its postpaid mobile service. It also gave fiber customers a 10% discount on their monthly bill the following month.

The IMDA added that SingNet has since upgraded its existing DHCP servers, “and is conducting an end-to-end review of its broadband network architecture, vendor management of critical platforms and its escalation process to improve the resiliency of its network.”