How a one-second change floored web services
By Stewart Mitchell
Posted on 2 Jul 2012 at 16:46
Web admins running several flavours of Linux had a hectic weekend after a leap second bug caused problems at multiple companies and left air passengers stranded.
The issue surfaced overnight on Saturday, when a second was added to the world’s atomic clocks to take into account minutely lengthening days, with the earth taking slightly longer to rotate than a standard 24 hours.
The issue didn’t affect all Linux tools, but the damage caused by a one second difference in clocks was highlighted when passengers were left unable to check in at several airports because Amadeus Altea, a global booking system used by airlines, had been hit by the bug.
Some Linux systems after the leap second was inserted are experiencing load spikes - usually connected to MySQL, Firefox, Thunderbird, Java, and others
“This incident was caused by the Linux bug triggered by the 'leap second' inserted into clocks worldwide on 30 June,” a spokesperson for the company said in a statement sent to PC Pro.
“We deeply regret this incident and are now investigating how we can enhance our ability to detect and address such bugs in advance.”
The company said it had dealt with leap second additions successfully in the past, but - like other online services - had not seen this particular bug coming.
The issue was caused by the way servers handled the time after the additional second had been added, causing systems to overload.
“Some Linux systems after the leap second was inserted are experiencing load spikes - usually connected to MySQL, Firefox, Thunderbird, Java, and others,” said John Stultz, a kernel engineer with IBM, who had revealed a patch in advance of the problem, on the LKML forum.
Technicians at companies as diverse as Cisco and Mozilla said they had struggled to handle the transition smoothly, with Cisco users saying they had seen video conferencing services crash in the wake of the update.
“The Cisco VCS [Video Communication Server] seems to be affected by the (generic) Linux leap second bug - we noticed that we had some (not all) VCSs with a load of >100," said Martin Koch, a contributor on the Cisco Support Community forum.
“I would recommend as a quick fix to immediately reboot your VCS. Also check your other Linux systems.”
- Flickr redesign: is it enough to tempt photographers back?
- Hands on with the new Google Maps
- Nokia Lumia 925 review: first look
- Why I won't subscribe to Creative Cloud
- GoPro camera strapped to a remote-control helicopter: the ultimate boy's toy
- Acer Iconia A1 review: first look
- Acer Aspire P3 review: first look
- Acer Aspire R7 review: first look
- How we produce the PC Pro podcast
- Google Now draining iPhone battery
- The ICO's shame-faced u-turn on cookies
- Start8 and ModernMix: making Windows 8 work on a desktop
- How to boost your mobile reception
- How to fix Facebook: Social Fixer
- Taking the stress out of WordPress updates
- Where to download free web fonts
- Turn your tablet into a Sky+ remote control
- How to measure the success of a new IT system
- Three years on: the state of the tablet market
- Windows 8: what works and what doesn't