Singtel is investigating the cause of a serious island-wide service outage of its fiber broadband service over the weekend that lasted 24 hours. The one thing the telco is sure of is that the outage wasn’t caused by a DDoS attack similar to one that took many StarHub customers offline last month.
The outage, which impacted Singtel residential customers and some enterprise customers, lasted from 8:46am on Saturday to 8:25am Sunday, according to a statement on Singtel’s Facebook page. The outage only affected fiber broadband customers – Singtel’s IPTV, mobile and fixed line services remained up and running during the outage.
According to a Facebook update Saturday evening, the basic problem appeared to be that Singtel’s servers were unable to assign IP addresses to their customers’ modems to enable broadband connectivity.
Suspicions of a DDoS attack were initially raised, as rival telco StarHub had its internet services disrupted by such an attack on its DNS servers at the end of October.
Singtel said Saturday evening that it had determined that a DDoS attack was not the cause for the service disruption, and was looking at “other plausible causes”.
Those “other plausible causes” could include a problem with Singtel’s DNS servers or a network switch malfunction, Aloysius Cheang, Asia-Pacific managing director of the Cloud Security Alliance, told the Straits Times:
“It’s not uncommon for a telco to take a long time to isolate the cause of the problem,” Mr Cheang said, adding that it is difficult to tell whether it was a human or hardware error.
Singtel – which encouraged its fiber broadband customers who also subscribe to its postpaid mobile service to use the mobile network for internet access – said it would waive local mobile data charges for Saturday and Sunday for affected customers. The telco will also give fiber broadband customers a 10% discount on their monthly bill in January.
However, customer compensation could be the least of its problems.
Singapore’s comms regulator, the Infocomm Media Development Authority (IMDA), is very strict about service uptime. Just last week, the IMDA fined Singtel S$145,000 (around $107,000) after a group of TV channels on its IPTV platform went dark for over one hour.
In May 2014, Singtel was fined a record S$6 million for a major service outage in October 2013 affecting almost 270,000 subscribers in the western and northern parts of the island. The cause of that outage: an exchange fire accidentally caused by unauthorized “hot works” being carried out by Singtel contractors that damaged several fiber-optic cables.
For the record, the exchange owner, CityNet, was fined S$300,000 for not ensuring the works were done correctly, while OpenNet – who built the fiber network (and has since been bought by NetLink, a business trust owned by SingTel) – was fined $S$200,000 for not restoring connectivity fast enough.
According to The New Paper, Zaqy Mohamad, chairman of the Government Parliamentary Committee for Communications and Information, has commended Singtel for handling the outage well in terms of public communication and providing regular updates, but added that many Singaporeans – especially small businesses – who rely heavily on internet connectivity may not be all that forgiving.
A case in point: Daniel Tan, owner of the Angel Supermart, who had to deal with long queues at the Woodlands outlet as a result of the outage, TNP reports:
His systems, which are linked by cloud computing, was down, affecting real-time access to promotions, price changes and membership data.
Fortunately, he had back-up solutions – a Singtel 3G router and a StarHub 3G dongle for the cashiering system.
Mr Tan told TNP that this was the second time he had been affected by a broadband outage.
His Ang Mo Kio outlet was hit when StarHub was affected by a Distributed Denial of Service (DDoS) attack in October, which resulted in two major disruptions.
“It is quite upsetting. I chose fibre Internet because it is supposed to be more stable than 3G,” he said.