Friday, September 11, 2009

Give me a lever and a place to stand

A few months ago I joined a new lab at CCNY.  Shortly before I arrived, the lab purchased a high-performance computing (HPC) cluster:
For a while it was running in our lab.  Unfortunately, the lab is not a data center: It is not climate controlled, not dust controlled, etc.  But we needed it there to set up some stuff, to heat our lab and to provide deafening ambient noise.  Fortunately, CCNY does have a data center, though it's in a different building (NAC), about 1000 feet from the building the lab is in (Steinman Hall).

View Moving an HPC in a larger map

This past Wednesday was the epic move to CCNY's data center.

WARNING: If you are a professional IT person or server administrator, or are pregnant, have a heart condition, or are overly sensitive to computer abuse, please, stop reading at this point, and go find pictures of butterflies or daisies.  Continue reading at your own risk.



Firstly, although there are small wheels on the bottom of the rack, they're designed for sliding the thing around short distances on smooth surfaces, not for wheeling down the block.  So we arranged with a facilities person, who said he's moved a few HPCs in his time, to move the cluster. He told us to wheel the thing out of the lab into the hallway and he'd bring his equipment:
That's right, his equipment for moving a very expensive, fragile computing cluster was a pallet truck.  After he looked over the cluster for a few seconds he announced, "I'll be right back.  I need a...[dramatic pause]...Johnson Bar."
Turns out, a Johnson Bar is a giant, wheeled lever which he used to get help the HPC onto the truck.  I say 'help' because it just got one side precariously onto the forks.  He then went to the other side, levered and lifted and we pushed the rack (trust me, you do not want to hear the scraping sound this makes) fully on to the tines .
Well, almost fully on to the tines.  After a brief ride in the freight "only" elevator,
we got stuck getting out of the building:

To get it out, we propped it up on bricks to help get it off the truck,
[stole its tires,] and wheeled it out of the building.

Before I continue this epic tale, I would like to draw your attention to the following detail:
Remember what I said about little wheels?  To get the cluster over that bump we needed to get a running start and develop enough momentum to hit that threshold and keep going.  Just imagine the jolt and vibration those poor servers were getting!

In any case, we got it out, back onto the truck, and started wheeling it down the sidewalk, and then onto the street:
Remember earlier when we barely got it onto the tines? This time was no different.  So why, you might be thinking, is the rack all the way up against the far edge of the hand truck?  Well, when we went down the wheelchair ramp thing from the sidewalk to the street, the cart got stuck, and the server slid forward and slammed into that lip.

Anyway, we wheeled the thing down the street
narrowly missing being smashed by some speeding buses and nearly running over a few pedestrians. Keep in mind that this pallet truck didn't really have any suspension to speak of, and this was a NYC street, so there was quite a bit of vibration during this leg of the trip, too. After a few near misses with some giant, man-potholes, we got stuck again, this time outside NAC:

Well, after another brick trick, and another rough trip over another threshold, we were inside! Only to discover that one of the servers had been shaken free from its mount:


After re-securing that server (and checking the others), we continued on our way, bouncing over the textured, uneven tiles of the NAC lobby then down a hall to the data center!
We found our spot, in a cozy corner, plugged it in
and, miraculously,
IT TURNED ON!!! A few connections were loose in the back, but otherwise all 11 computers, somehow, survived the trip!
Now our lab is quiet, the cluster is happy, I learned what a Johnson bar is, I beat Aryeh's record for number of pictures in a single blog post, and I now know how to move a high performance computing cluster.

4 comments:

  1. Wow, I really expected the cluster to sustain some sort of injury from that journey. Could it be that HPCs are nowhere near as fragile as one might think, or did you guys just get incredibly lucky??

    ReplyDelete
  2. s/and I now know how to move a high performance computing cluster./and I now know how not to move a high performance computing cluster./g

    ReplyDelete
  3. you could have at least put a cardboard box on the tines to give it some corrugated padding.

    ReplyDelete
  4. notElon - I think we got lucky.
    Michal - Yeah. But I'm kinda hoping to top the Google results for that query...
    Yoni - When we slid the rack across the tines any cardboard would have been torn to shreds.

    ReplyDelete