UPDATE: Telecom confirms tens of thousands hit by problems this morning | Seven-hour Tuesday morning outage followed by evening blackout | Company adds more detail about cause of foul-ups.
UPDATE 8.35am: Telecom has confirmed that tens of thousands are still suffering problems following Tuesday's twin blackouts, which hit hundreds of thousands.
"Although Telecom has restored service to the majority of its customers, a significant number of customers are receiving intermittent coverage and may find that their connection drops out. It is estimated that the number of customers affected at any one time is in the tens of thousands," the company said in an 8.30am statement.
----------
UPDATE Dec 12, 7am:: Telecom Retail CEO Chris Quin is promising action after customers were hit by two major broadband blackouts yesterday.
"We have a comprehensive review underway to ascertain how these issues can be minimised in the future and apologise unreservedly to customers for any disruption these outages caused in the interim," Mr Quin says.
At 1.15am this morning, Telecom issued a statement saying it had restored service to "the majority" of customers following an outage that began at 9.45pm Tuesday evening.
A number of Auckland and Hamilton customers have told NBR they still have no service this morning.
Telecom says some might have to restart their modem to regain service. The readers say they tried that - waiting the regulation 30 seconds before switching back on - without any luck.
The outage was caused by a server problem.
The company says it was related to the one earlier in the day that saw two thirds of customers and about a quarter of eftpos machines lose service nationwide between 1am and 8.55am after planned maintenance at its giant Mayoral Drive exchange in the Auckland CBD went wrong.
In NBR Comments (see below) and on Twitter, many customers were seething at the second loss of sevice within a day.
But one Auckland man took events in stride, tweeting:
If anyone in #ponsonby is affected by the @telecomnz outage I will come to your house and act out some pornography. Please DM.
— Vaughn Davis (@vaughndavis) December 11, 2012
And comedian Dai Henwood offered more humour from the trenches:
Telecom seem to be running a curfew situation, no broadband between 9pm and 8am
— Dai Henwood (@daihenwood) December 11, 2012
Details fleshed out
"The first outage related to a problem with a software upgrade on an Auckland-based server that is part of a system which handles just over 300,000 customers, representing approximately two-thirds of Telecom’s broadband customer base," Telecom said in a statement this morning.
"The problem meant customer modems were failing to authenticate when trying to communicate with the server. The Auckland server normally runs in parallel with a Christchurch based server and service was initially restored by removing the Auckland server from the system and running solely on the Christchurch server.
"Yesterday afternoon service resumed on both parallel servers, but last evening further problems were experienced with the Christchurch server overloading due to issues synchronising information between the two servers. To address this issue, the Christchurch server was taken off the system late last night, with internet traffic now all going through the Auckland server. Telecom is urgently investigating why the synchronisation issues occurred."
UPDATE / Dec 11, 11.45pm: Telecom Retail CEO Chris Quin has just tweeted:
"Broadband services have just started restoring again, team staying in place through the night to monitor this closely...sorry again." 50,000 customers are back online. All should have service restored within an hour."
UPDATE / Dec 11, 10.30pm: Telecom's broadband service has been hit by a fresh wave of problems.
Spokeswoman Joanne Jalfon told NBR ONLINE an outage hit at 9.45pm.
"The issue is believed to be related to an incident which affected broadband service early this morning," Ms Jalfon said.
The morning woes were blamed on a scheduled maintenance upgrade gone wrong at the company's largest exchange in central Auckland.
"Telecom's customer service team is responding to a high volume of calls and apologise to those who can not currently get through to speak to a staff member," Ms Jalfon said.
The support section of Telecom's website has been slow to acknowledge the problem.
But on social networks, on social media and in Comments below, numerous people around the country have complained their broadband has again blacked out.
Telecom's official Twitter account has been hit by a blizzard of complaints.
Via the social network, the company asked:
Folk on hold on 123, can we please ask you to hang up, as those waiting won't hear recorded messages. Again, really sorry ^RI
— Telecom New Zealand (@TelecomNZ) December 11, 2012
Many customers were exasperated their broadband had gone out for the second time in a day.
"Internet down again tonight. Come on Telecom sort it out," and "So this is how Syria and Iran do it" were typical comments.
Telecom broadband back after seven-hour, nationwide outage
Dec 11, 8am: Telecom's broadband service was restored at 8am following a seven-hour outage. Affected customes should be back online within an hour.
The outage hit around two-thirds of the company's 500,000 or so broadband customers nationwide, spokeswoman Joanne Jalfon told NBR.
About a quarter of the country's eftpos machines were also taken offline.
A planned maintenance upgrade at the company's Mayoral Drive exchange in central Auckland caused the nationwide problem, Ms Jalfon said.
An exchange in Christchurch had taken load off the affected Mayoral Drive system.
Some customer might have to restart their modem to regain service.
A Telecom broadband outage on the morning of November 30 cut off around half the company's customers until 9.45am. That problem was also blamed on an upgrade gone wrong.
