The Data Center Walk-through...

One thing I'm making sure I do at one customer in particular, is a walk-through of the data center.  Once a week, one member of my team walks through the customer data center.  Its important.  There is a lot of gear in this data center.  Probably around 40 racks or so.
What are we looking for?  Lights.  Or, missing lights.  Amber lights.  Anything that doesnt look right.  I know that in the top of each rack, I expect to see 6 power supply lights.  And in some racks, 8 power supply lights.  I know that for each Aggregation and Core switches (12 total), that I should see 4 power supply lights, 2 supervisor lights, 6 lights across the top front of the Nexus gear, and I should never see an amber light, or one that is out (or blinking).
Every rack has a certain amount of "green lights" on it.  Even the sound can be help you determine if something isn't right.  And if you look at these often enough, you start to notice when something doesn't "look right" or "sound right".  It actually gets easy to see when something isn't functioning correctly.  For example, I can tell you within a few seconds if a power supply has gone bad, just by glancing at the rack.  I've trained my eyes on what to look for, and its a huge benefit to the operation of the data center.  Could I automate this?  Probably.  But I would rather "know" the data center myself instead of depending on even more electronics and software to tell me when it thinks something is wrong.
Get to know your data center (or closets) more intimately.  You will be glad you did.