Skip to navigation

PCPro-Computing in the Real World Printed from www.pcpro.co.uk

Register to receive our regular email newsletter at http://www.pcpro.co.uk/registration.

The newsletter contains links to our latest PC news, product reviews, features and how-to guides, plus special offers and competitions.

// Home / Blogs

Posted on March 19th, 2010 by Steve Cassidy

What’s that eggy smell in the server room?

Man screamingWorking in close proximity with a team can be a very personal matter. There’s a fine line between acceptable office banter and touchy subjects: You can safely discuss brands of car, football teams, and who should get booted out of The Apprentice, but there are topics that are very firmly off the agenda, even once you allow for the massive variations in office cultures.

I have clients who see nothing curious about coming to work and spending their day in a cape; clients who freely discuss their weekend metrosexual exploits every Monday morning, and clients who don’t even know one another’s first names.

Part of the burden of consulting is to remember the acceptability envelope of the various cultures you’re working with, but this week I think I’ve found a new area of delicacy. One that crosses pretty much everybody’s boundaries.

As the season has changed, for the first time in something like five months, I’ve opened a window in the server room I’ve been working in. The aircon is ferocious but it’s a closed-circuit system – no fresh air comes in from outside, since the main job here is to shift the heat pushed out by the server rack.

The air movement isn’t quite as fierce as it used to be, because the servers have gone through a bit of a downsizing activity; so it was with some surprise that I came in the next day to quite a detectable odour.

Smells in server rooms are pretty much invariably bad news. Something is about to fail, whether it’s a leaking R134 circuit in the aircon or a power supply on the way out, or a 250kg rack sitting on a 15A power cable – there are no upsides. However, this particular pong seemed a lot more biological. It was, pardon my pithiness, a bit farty. I put it down to someone’s substandard eggy soldiers, opened the window and got on with the job, but that same smell was in the room the following morning.

The chances of anyone in a small team suffering the same gastric embarrassment two days on the trot (sorry!) are pretty slim. Something in the server room was building up a pervasive hydrogen sulphide stink overnight while the door was shut. So I started the delicate conversation.

After a lot of hypothesising about aircon, server power supplies, linoleum, dead mice, farting mice(!), gas mains, Victorian sewers, whether H2S is heavier than air or lighter, dropped sandwiches, and tentative enquiries as to the recent health of the team, someone was smart and Googled for “egg smell server room”. This leads to a discussion about APC UPS devices emitting eggy smelling gases which suggests that one of the UPSes is about to blow its capacitors.

Looking down the server rack, there is indeed a UPS with every LED lit up – a whole lot less embarrassing, as a cause of an eggy smell, than the alternatives.

Posted in: Real World Computing

Permalink

Follow any responses to this entry through the RSS 2.0 feed.

You can skip to the end and leave a response. Pinging is currently not allowed.

31 Responses to “ What’s that eggy smell in the server room? ”

  1. Rob Schifreen Says:
    March 19th, 2010 at 1:44 pm

    That APC page doesn’t contain a single mention of the word “room”. So the google query is effectively “egg smell server”. Which, however hard I try, I can’t get to point me to that page.

     
  2. Steve Cassidy Says:
    March 19th, 2010 at 10:41 pm

    The easy way to try, mon vieux, is to see only the result of the search and ask the searcher what he searched for, verbally…!

     
  3. Miles Says:
    March 20th, 2010 at 11:11 am

    My room smells of eggs and there isn’t a server in sight!

     
  4. Mike Says:
    March 20th, 2010 at 9:51 pm

    You should be in our server room – some idiot plumbed a sani flow into it – when its left over a hot weekend you have to gag just to get the tape out of the drive!

     
  5. gabby Says:
    March 21st, 2010 at 9:45 am

    Perhaps an even better indicator of a failing ups would be all the flashing lights?

     
  6. Scooter Com-Pooter Says:
    March 21st, 2010 at 12:04 pm

    POOT!

     
  7. Eddy Says:
    March 21st, 2010 at 12:19 pm

    I’m inclined to say: “what, no monitoring?”

    My 2c: google for nagios… ;)

     
  8. Steve Cassidy Says:
    March 21st, 2010 at 12:19 pm

    Read carefully, Gabby: no blinking lights involved. They were all on, which everyone took as Situation Normal for a highly loaded rack. Now, however, it seems this set of lights actually indicates “imminently dead”.

     
  9. Bill Says:
    March 21st, 2010 at 1:59 pm

    Some of our group of geeks and gear-heads had driven thousands of miles to get to the desert for an off-roading excursion. Second day in, miles into the trail, we’re smelling something like rotten eggs, and can’t source it – after all the exhaust of a 1960’s Land Rover doesn’t have catalytic converters to go bad. So after pondering the situation no-one puts 2+2 together, we shrug our shoulders, the convoy starts to creep along and we carry on. Next morning the stinky Series IIa Rover isn’t starting on the trail. It had been bump started a few times but this is not good. More determined investigation indicates the electrics have run out of juice, and literally the battery is nearly dry. A collective d’OH! arises and water bottles are emptied into the unintentionally dry cells. — BTW, on the left side of the pond a capacitor seems to be defined as having an insulator between plates, which in the case of UPCs are large cans with plates and insulators wrapped in a spiral where the insulator is usually some sort of petroleum based/doped product, and in my experience when those smoke it is quite different smell to the liquid boiling sulphuric acid out of a lead-acid battery leading to the release of nasty gasses both poisonous and very explosive.

     
  10. erica Says:
    March 21st, 2010 at 2:55 pm

    From a security point of view, I would not want a server room that had windows

     
  11. Michael Says:
    March 21st, 2010 at 3:22 pm

    Try this;
    http://www.dailycampus.com/2.7440/physics-building-evacuates-1.1057279

    I used “egg smell server room” in google…

     
  12. John Says:
    March 21st, 2010 at 4:39 pm

    Ahh, but what did the web management card say — did it report a problem? All of my UPS’s are set to send SNMP alerts when they detect a problem (and I get alerted if it drops off the network), so I’m curious if the card detected anything, or at least failed itself.

     
  13. Scott Says:
    March 21st, 2010 at 7:31 pm

    I’m a field service engineer with APC, and I can tell you that an “eggy smell” is usually a battery that has blown one of its vent caps. An overcharging battery builds up hydrogen until either the pressure pops a vent cap or a spark ignites the gas, popping the cap open. Excess hydrogen sulfate gas then vents out, smelling like rotten eggs.

     
  14. Steve Cassidy Says:
    March 21st, 2010 at 7:57 pm

    Thanks Scott; I think we are moving towards “Horse’s Mouth of the week” award, and you’re in the frame…

     
  15. Spencer S Says:
    March 22nd, 2010 at 2:09 am

    Hmmm.. given the slightly patronzing way of responding (Steve Cassidy post #8), it must be said that Steve forgot that monitoring is not someone sitting in a server room checking lights every few minutes.. As John said in post #12, it’s about using SNMP.

    Steve didn’t then say “ah yes, SNMP. Comment #8 was a bit reactionary and I didn’t think of SNMP”

    !!

    Quoting Steve:

    “Read carefully, Gabby: no blinking lights involved. They were all on, which everyone took as Situation Normal for a highly loaded rack. Now, however, it seems this set of lights actually idicates “imminently dead”.”

     
  16. Spencer S Says:
    March 22nd, 2010 at 2:10 am

    Read carefully, Steve: no SNMP was considered by you.

     
  17. Rick Says:
    March 22nd, 2010 at 2:54 am

    Even worse, the burnt plastic smell in a server room which pretty much means disaster in 15 minutes or less.

     
  18. Margit Says:
    March 22nd, 2010 at 4:31 am

    All of the electronic sensors in the world cannot tell you the pre-failure status of components that are aging/degrading. Some components (capacitors, transformers, battery cells) do age, and their materials do break down over time. This can be detected by visual inspection (capacitors bulging, oily leakage around transformers, smelly sulfur from blown vent caps on batteries) or audible inspection (fans starting to whine) but may still be fully functional until their actual performance starts to fall out of nominal operating range.

    The human inspection simply cannot be understated in terms of the value in detecting pre-failure conditions. For whatever that’s worth.

     
  19. spotrick Says:
    March 22nd, 2010 at 8:03 am

    Years since I was near a server room, but back in the day I recall a christmas party at which someone left the garbage bags full of seafood shells in the corner. For a week.

     
  20. Mike Says:
    March 22nd, 2010 at 8:49 am

    @Bill: Surely you mean “no-one puts 4×4 together …” :-D

     
  21. Steve Cassidy Says:
    March 22nd, 2010 at 9:54 am

    Spencer: you fearless seeker after the truth, you! Fact is I don’t know if these guys had SNMP set up: one can wish for many things at a client site (including polite responses…) but they’re not always perfect, or ideal. Heaven forbid.

     
  22. Steve Cassidy Says:
    March 22nd, 2010 at 10:07 am

    Margit: Very fair point. I wonder how many of those remote-monitoring systems for hosted server rooms would pick up on this type of indication? Is there even an electronic Hydrogen Sulfate (important distinction from Sulphide) detector?

     
  23. Tom Ace Says:
    March 22nd, 2010 at 2:15 pm

    Could be the batteries of your UPS, had this happen once and it smelled like rotten eggs.

     
  24. John Marks Says:
    March 22nd, 2010 at 3:39 pm

    Our huge APC UPS kicked out more than an eggy smell. We have to evacuate the building. They’d been pumping H2S out constantly all night at full charge-whack.

    We had to go in with Hazmat and specced breathing apparatus.

     
  25. jpayne Says:
    March 23rd, 2010 at 1:54 am

    I had an APC 2200VA Smart UPS boil over and it stunk the entire office up. It was like burnt hair and eggs. The batteries expended through the inner framework and had to be surgically removed.

     
  26. Bluespider Says:
    March 23rd, 2010 at 11:57 am

    Have to say, I’m more concerned with opening windows (or windows at all) in a server room than any noxious smells that may be in there…

    In my experience these should be mutually exclusive… Is this something you encounter often Steve?

     
  27. Steve Cassidy Says:
    March 23rd, 2010 at 12:08 pm

    which, smells or windows? (or do you mean Windows, in the Linux-evangelist sense) I guess I come across windows about 2/3rds of the time, and a smallish number that open. They tend to survive death of the aircon better than the other kinds (except for the room with 13 aisles of racks – that’s surrounded by windows but if it ever lost air-handling, life would be … stressful)

     
  28. Daniel Titley Says:
    March 31st, 2010 at 3:14 pm

    “clients who see nothing curious about spending their day in a cape”

    Really? REALLY?

    “clients who don’t even know one another’s first names” How can that work?

     
  29. Steve Cassidy Says:
    March 31st, 2010 at 4:20 pm

    Yes to the first, and using last names, to the second, Daniel. I won’t give clues on the organisation of the first client, but you may jump to understanding the second lot if I tell you that it’s “herr” this and “frau” that, at that company.

     
  30. Bass-Playing Chemist Says:
    April 7th, 2010 at 3:32 pm

    Hi Scott (comment #13), careful with your chemistry there old chap. Hydrogen sulfate (or sulphate east side of the Atlantic) is sulphuric acid, which is a syrupy, clear, colourless liquid at room temperature and pressure (H2SO4), which when diluted with water forms sulphuric acid. Hydrogen sulphide (H2S) is the “rotten eggs” smelly gas and Steve, for your pub quiz fact file, it is slightly heavier than air (as are all gasses with a molecular weight above 29 g/mol). As other posters have pointed out, the most likely source of the sulphur to form H2S is the sulphuric acid within the batteries of the UPS, assuming a lead/acid battery chemistry. I must point out though that I’m not a battery specialist. Hope all this helps.

     
  31. mayak Says:
    November 9th, 2013 at 12:04 pm

    i have a upc 5000 doing the same thing right now — the stink is simply incredible — 400 sq meters is smelling.

    thanks!

     

Leave a Reply

Spam Protection by WP-SpamFree

* required fields

* Will not be published

Authors

Categories

Archives

advertisement

SEARCH
SIGN UP

Your email:

Your password:

remember me

advertisement


Hitwise Top 10 Website 2010