How a one-second change floored web services

 

Gallery

By Stewart Mitchell

Posted on 2 Jul 2012 at 16:46

Web admins running several flavours of Linux had a hectic weekend after a leap second bug caused problems at multiple companies and left air passengers stranded.

The issue surfaced overnight on Saturday, when a second was added to the world’s atomic clocks to take into account minutely lengthening days, with the earth taking slightly longer to rotate than a standard 24 hours.

The issue didn’t affect all Linux tools, but the damage caused by a one second difference in clocks was highlighted when passengers were left unable to check in at several airports because Amadeus Altea, a global booking system used by airlines, had been hit by the bug.

Some Linux systems after the leap second was inserted are experiencing load spikes - usually connected to MySQL, Firefox, Thunderbird, Java, and others

“This incident was caused by the Linux bug triggered by the 'leap second' inserted into clocks worldwide on 30 June,” a spokesperson for the company said in a statement sent to PC Pro.

“We deeply regret this incident and are now investigating how we can enhance our ability to detect and address such bugs in advance.”

Widespread problem

The company said it had dealt with leap second additions successfully in the past, but - like other online services - had not seen this particular bug coming.

The issue was caused by the way servers handled the time after the additional second had been added, causing systems to overload.

“Some Linux systems after the leap second was inserted are experiencing load spikes - usually connected to MySQL, Firefox, Thunderbird, Java, and others,” said John Stultz, a kernel engineer with IBM, who had revealed a patch in advance of the problem, on the LKML forum.

Technicians at companies as diverse as Cisco and Mozilla said they had struggled to handle the transition smoothly, with Cisco users saying they had seen video conferencing services crash in the wake of the update.

“The Cisco VCS [Video Communication Server] seems to be affected by the (generic) Linux leap second bug - we noticed that we had some (not all) VCSs with a load of >100," said Martin Koch, a contributor on the Cisco Support Community forum.

“I would recommend as a quick fix to immediately reboot your VCS. Also check your other Linux systems.”