Posted on March 19th, 2010 by Steve Cassidy
What’s that eggy smell in the server room?
Working in close proximity with a team can be a very personal matter. There’s a fine line between acceptable office banter and touchy subjects: You can safely discuss brands of car, football teams, and who should get booted out of The Apprentice, but there are topics that are very firmly off the agenda, even once you allow for the massive variations in office cultures.
I have clients who see nothing curious about coming to work and spending their day in a cape; clients who freely discuss their weekend metrosexual exploits every Monday morning, and clients who don’t even know one another’s first names.
Part of the burden of consulting is to remember the acceptability envelope of the various cultures you’re working with, but this week I think I’ve found a new area of delicacy. One that crosses pretty much everybody’s boundaries.
As the season has changed, for the first time in something like five months, I’ve opened a window in the server room I’ve been working in. The aircon is ferocious but it’s a closed-circuit system – no fresh air comes in from outside, since the main job here is to shift the heat pushed out by the server rack.
The air movement isn’t quite as fierce as it used to be, because the servers have gone through a bit of a downsizing activity; so it was with some surprise that I came in the next day to quite a detectable odour.
Smells in server rooms are pretty much invariably bad news. Something is about to fail, whether it’s a leaking R134 circuit in the aircon or a power supply on the way out, or a 250kg rack sitting on a 15A power cable – there are no upsides. However, this particular pong seemed a lot more biological. It was, pardon my pithiness, a bit farty. I put it down to someone’s substandard eggy soldiers, opened the window and got on with the job, but that same smell was in the room the following morning.
The chances of anyone in a small team suffering the same gastric embarrassment two days on the trot (sorry!) are pretty slim. Something in the server room was building up a pervasive hydrogen sulphide stink overnight while the door was shut. So I started the delicate conversation.
After a lot of hypothesising about aircon, server power supplies, linoleum, dead mice, farting mice(!), gas mains, Victorian sewers, whether H2S is heavier than air or lighter, dropped sandwiches, and tentative enquiries as to the recent health of the team, someone was smart and Googled for “egg smell server room”. This leads to a discussion about APC UPS devices emitting eggy smelling gases which suggests that one of the UPSes is about to blow its capacitors.
Looking down the server rack, there is indeed a UPS with every LED lit up – a whole lot less embarrassing, as a cause of an eggy smell, than the alternatives.
Posted in: Real World Computing
Follow any responses to this entry through the RSS 2.0 feed.
You can skip to the end and leave a response. Pinging is currently not allowed.
30 Responses to “ What’s that eggy smell in the server room? ”
Leave a Reply
Authors
- Barry Collins
- Chris Brennan
- Christine Horton
- Darien Graham-Smith
- Dave Stevenson
- Davey Winder
- David Bayon
- David Fearon
- Ewen Rankin
- Ian Devlin
- Jon Honeyball
- Jonathan Bray
- Kevin Partner
- Mike Jennings
- Nicole Kobie
- Sasha Muller
- Steve Cassidy
- Stewart Mitchell
- Stuart Turton
- Tim Danton
- Tom Arah
Categories
- About the bloggers
- Android App of the Week
- CES 2013
- cloud computing
- From Gmail to Hotmail
- Green
- Hardware
- How To
- iPhone App of the Week
- Just in
- Microsoft Office 2010
- MWC 2013
- Newsdesk
- Online business
- Random
- Rant
- Real World Computing
- Software
- View from the Labs
- Web
- Windows 7
- Windows 8
Archives
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- April 2008
advertisement


March 19th, 2010 at 1:44 pm
That APC page doesn’t contain a single mention of the word “room”. So the google query is effectively “egg smell server”. Which, however hard I try, I can’t get to point me to that page.
March 19th, 2010 at 10:41 pm
The easy way to try, mon vieux, is to see only the result of the search and ask the searcher what he searched for, verbally…!
March 20th, 2010 at 11:11 am
My room smells of eggs and there isn’t a server in sight!
March 20th, 2010 at 9:51 pm
You should be in our server room – some idiot plumbed a sani flow into it – when its left over a hot weekend you have to gag just to get the tape out of the drive!
March 21st, 2010 at 9:45 am
Perhaps an even better indicator of a failing ups would be all the flashing lights?
March 21st, 2010 at 12:04 pm
POOT!
March 21st, 2010 at 12:19 pm
I’m inclined to say: “what, no monitoring?”
My 2c: google for nagios…
March 21st, 2010 at 12:19 pm
Read carefully, Gabby: no blinking lights involved. They were all on, which everyone took as Situation Normal for a highly loaded rack. Now, however, it seems this set of lights actually indicates “imminently dead”.
March 21st, 2010 at 1:59 pm
Some of our group of geeks and gear-heads had driven thousands of miles to get to the desert for an off-roading excursion. Second day in, miles into the trail, we’re smelling something like rotten eggs, and can’t source it – after all the exhaust of a 1960’s Land Rover doesn’t have catalytic converters to go bad. So after pondering the situation no-one puts 2+2 together, we shrug our shoulders, the convoy starts to creep along and we carry on. Next morning the stinky Series IIa Rover isn’t starting on the trail. It had been bump started a few times but this is not good. More determined investigation indicates the electrics have run out of juice, and literally the battery is nearly dry. A collective d’OH! arises and water bottles are emptied into the unintentionally dry cells. — BTW, on the left side of the pond a capacitor seems to be defined as having an insulator between plates, which in the case of UPCs are large cans with plates and insulators wrapped in a spiral where the insulator is usually some sort of petroleum based/doped product, and in my experience when those smoke it is quite different smell to the liquid boiling sulphuric acid out of a lead-acid battery leading to the release of nasty gasses both poisonous and very explosive.
March 21st, 2010 at 2:55 pm
From a security point of view, I would not want a server room that had windows
March 21st, 2010 at 3:22 pm
Try this;
http://www.dailycampus.com/2.7440/physics-building-evacuates-1.1057279
I used “egg smell server room” in google…
March 21st, 2010 at 4:39 pm
Ahh, but what did the web management card say — did it report a problem? All of my UPS’s are set to send SNMP alerts when they detect a problem (and I get alerted if it drops off the network), so I’m curious if the card detected anything, or at least failed itself.
March 21st, 2010 at 7:31 pm
I’m a field service engineer with APC, and I can tell you that an “eggy smell” is usually a battery that has blown one of its vent caps. An overcharging battery builds up hydrogen until either the pressure pops a vent cap or a spark ignites the gas, popping the cap open. Excess hydrogen sulfate gas then vents out, smelling like rotten eggs.
March 21st, 2010 at 7:57 pm
Thanks Scott; I think we are moving towards “Horse’s Mouth of the week” award, and you’re in the frame…
March 22nd, 2010 at 2:09 am
Hmmm.. given the slightly patronzing way of responding (Steve Cassidy post #8), it must be said that Steve forgot that monitoring is not someone sitting in a server room checking lights every few minutes.. As John said in post #12, it’s about using SNMP.
Steve didn’t then say “ah yes, SNMP. Comment #8 was a bit reactionary and I didn’t think of SNMP”
!!
Quoting Steve:
“Read carefully, Gabby: no blinking lights involved. They were all on, which everyone took as Situation Normal for a highly loaded rack. Now, however, it seems this set of lights actually idicates “imminently dead”.”
March 22nd, 2010 at 2:10 am
Read carefully, Steve: no SNMP was considered by you.
March 22nd, 2010 at 2:54 am
Even worse, the burnt plastic smell in a server room which pretty much means disaster in 15 minutes or less.
March 22nd, 2010 at 4:31 am
All of the electronic sensors in the world cannot tell you the pre-failure status of components that are aging/degrading. Some components (capacitors, transformers, battery cells) do age, and their materials do break down over time. This can be detected by visual inspection (capacitors bulging, oily leakage around transformers, smelly sulfur from blown vent caps on batteries) or audible inspection (fans starting to whine) but may still be fully functional until their actual performance starts to fall out of nominal operating range.
The human inspection simply cannot be understated in terms of the value in detecting pre-failure conditions. For whatever that’s worth.
March 22nd, 2010 at 8:03 am
Years since I was near a server room, but back in the day I recall a christmas party at which someone left the garbage bags full of seafood shells in the corner. For a week.
March 22nd, 2010 at 8:49 am
@Bill: Surely you mean “no-one puts 4×4 together …”
March 22nd, 2010 at 9:54 am
Spencer: you fearless seeker after the truth, you! Fact is I don’t know if these guys had SNMP set up: one can wish for many things at a client site (including polite responses…) but they’re not always perfect, or ideal. Heaven forbid.
March 22nd, 2010 at 10:07 am
Margit: Very fair point. I wonder how many of those remote-monitoring systems for hosted server rooms would pick up on this type of indication? Is there even an electronic Hydrogen Sulfate (important distinction from Sulphide) detector?
March 22nd, 2010 at 2:15 pm
Could be the batteries of your UPS, had this happen once and it smelled like rotten eggs.
March 22nd, 2010 at 3:39 pm
Our huge APC UPS kicked out more than an eggy smell. We have to evacuate the building. They’d been pumping H2S out constantly all night at full charge-whack.
We had to go in with Hazmat and specced breathing apparatus.
March 23rd, 2010 at 1:54 am
I had an APC 2200VA Smart UPS boil over and it stunk the entire office up. It was like burnt hair and eggs. The batteries expended through the inner framework and had to be surgically removed.
March 23rd, 2010 at 11:57 am
Have to say, I’m more concerned with opening windows (or windows at all) in a server room than any noxious smells that may be in there…
In my experience these should be mutually exclusive… Is this something you encounter often Steve?
March 23rd, 2010 at 12:08 pm
which, smells or windows? (or do you mean Windows, in the Linux-evangelist sense) I guess I come across windows about 2/3rds of the time, and a smallish number that open. They tend to survive death of the aircon better than the other kinds (except for the room with 13 aisles of racks – that’s surrounded by windows but if it ever lost air-handling, life would be … stressful)
March 31st, 2010 at 3:14 pm
“clients who see nothing curious about spending their day in a cape”
Really? REALLY?
“clients who don’t even know one another’s first names” How can that work?
March 31st, 2010 at 4:20 pm
Yes to the first, and using last names, to the second, Daniel. I won’t give clues on the organisation of the first client, but you may jump to understanding the second lot if I tell you that it’s “herr” this and “frau” that, at that company.
April 7th, 2010 at 3:32 pm
Hi Scott (comment #13), careful with your chemistry there old chap. Hydrogen sulfate (or sulphate east side of the Atlantic) is sulphuric acid, which is a syrupy, clear, colourless liquid at room temperature and pressure (H2SO4), which when diluted with water forms sulphuric acid. Hydrogen sulphide (H2S) is the “rotten eggs” smelly gas and Steve, for your pub quiz fact file, it is slightly heavier than air (as are all gasses with a molecular weight above 29 g/mol). As other posters have pointed out, the most likely source of the sulphur to form H2S is the sulphuric acid within the batteries of the UPS, assuming a lead/acid battery chemistry. I must point out though that I’m not a battery specialist. Hope all this helps.