Sunday 22 May 2016

Dell 2950 Server - Fan Modification - Part 3

This is the software side of modifying my Dell Poweredge 2950 (Model II) servers to be much quieter.

It WORKS!  But please beware, THERE ARE POTENTIAL RISKS IN ALTERING YOUR BMS FIRMWARE, IF YOU ARE READING THIS, IT IS ADVICE ONLY, ALL AND ANY ACTIONS YOU PERFORM ARE AT YOUR OWN RISK!  Just because it worked for me, doesn't mean you will be successful.

Part 1, shows how I added 47ohm 0.5watt resistors to the power (red cables) on each of the four internal fans.  I'm yet to modify the power supply fans, but I will.

Part 2, is my success video recorded at 3am last night when I was very tired, and had everything working.

And this, Part 3 is to cover my complete solution in how to do the software (BMC) firmware flashing.


The Centos Linux OS was used for the change, specifically the older version 6.4, downloaded from Oxford University here http://mirror.ox.ac.uk/sites/mirror.centos.org/6.7/isos/i386/CentOS-6.7-i386-LiveCD.iso.

This was burned to a DVD ROM, and as you can see in the image, I had to re-fit a DVD into the server as I have three servers, but only one DVD drive... DOH!

Next, I booted into Text Mode, so when the boot menu appeared from the DVD, I pressed Escape before the auto boot timed out, and selected to boot in Text mode.  Once booted, the login is "root", with no password.

My first command is to ensure the ipmi tools are installed, python, wget and rpm are on that image, but for completeness to the source material I include them in the installation:

sudo yum install freeipmi freeipmi-tools openipmi ipmitools python wget rpm

Once complete you can check the ipmi sensors are working with:

ipmi-sensors

You will get a huge list of all the sensors on the board.

Next you need to download the BMC firmware, to anywhere on the machine, you don't need any storage, the Live CD boot you're using will work fine just with RAM in the server!.. So no worry about overwriting any disks or data!

wget http://downloads.dell.com/esm/BMC_FRMW_LX_R202152.BIN

Once this has downloaded, you need to make it executable

sudo chmod +x ./BMC_FRMW_LX_R202152.BIN

Now make a directory as root:

sudo mkdir bmc

And extract this firmware you've downloaded into that path:

sudo ./BMC_FRMW_LX_R202152.BIN --extract bmc

And change into that directory:

cd bmc

Now we need the path script, but as we're in a directory created with sudo, we need to also run wget with sudo:

sudo wget http://projects.nuschkys.net/downloads/dell-adjust-fan-thresholds.py

And make this executable:

chmod +x dell-adjust-fan-threaholds.py

To then extract the values from the firmware and edit them, we run the script against the payload dat file:

./dell-adjust-fan-thresholds.py payload/bmcflsh.dat

This will extract a load of information, then ask you how many fans, or what model to edit, for the 2950, with four system fans and two power supply fans, I selected option 3.  To give me six fans to modify.

It lists each fan and their threshold value, so now select the fan by entering it's number, so try 1 first.  Then enter a number, now if I recall the default 2250 threshold is a value of 25.  I tried various values, but the best I found to modify for the resistors I'm using was 22, this gives a new minimum RPM of 1650 for the fans.  And I never saw any fan drop below 1800 RPM.

23 is 1750 RPM as a threshold, but I thought that too close to the lowest RPM I'd observed before starting this software mod.

So, 1 through 6 all selected and 22 entered for each, you then write this file back with the 'w' command.  This will repack the payload/bmcflsh.dat file with your new values.

Armed with this modified dat file you therefore need to flash the firmware, THIS IS THE STEP WHICH CAN BRICK YOUR MACHINE, PROCEED AT YOUR OWN RISK!!!

We're still in the bmc folder created before, so change nothing after altering the payload with the script, and you type:

sudo LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l
 -i=payload/bmcflsh.dat -f

And as you can see in my full video coverage, this worked for me.

I hope this helps, below are the references I went to, but I must have read about 15 different posts, dozens (if not hundreds) of the comments to piece this together.  The biggest problems I had from other posts was understanding the problems using Ubuntu and other distros, they don't work.  There are problems with the libstdc++ libraries, and then the i686 installation and all manner of hoops and tricks I tried.

Centos 6.4 i386, is the only live DVD image I've been able to use... and this all work easily.

The main instructions I followed were from Martin Denizet, however, even his post has issues and it not clear how to get the right distribution.

I also tried to follow RatzBlog, but was thwarted at every step by the OS's I was using, even trying to use Centos 6 didn't work with his instructions.

11 comments:

  1. First off, thanks for the great instructions! A+! That being said, I'm getting a error. "sudo: unable to execute ./bmcfl32l: No such file or directory" but when i "ls" the bmc dir its there and had x permission. Am I missing something?

    ReplyDelete
    Replies
    1. The instructions are correct here, you have to be in the bmc folder, and you have to use "./" at the front to tell linux that you're running from "this" folder etc... You say when you "ls" you can see the bmc folder, you should be in it and you should see the bmc flash program... and it should be executable :) Hope that helps.

      Delete
    2. Still someone using this tricks, and i succeeded using centos 7, and to fix this particular issue i had to chmod +x ./bmcfl32l, and i also had a typo the first time.

      Delete
  2. I did the 47ohm 0.5watt resistors mod to all the fans and the PSU went full blast so I had to do the PSU as well, now everything is quiet BUT I have a problem... after 2 hs or so the server restart for no reason by itself. If I put the original fans and the other PSU that I didnt modded it nevers get to restart by itself...any help would be appreciate. BTW I haven't mess with the BMS Im scared to do it.

    ReplyDelete
    Replies
    1. Sounds like maybe you over did it and after 2 hours of being "quiet" your machine is actually also "roasting hot" and reboots.

      Delete
  3. Yeah, but I did what you did with the exception of doing the PSU as well, the problem is that if I don't do the PSU it sounds extremely loud so I had no choice, modding the fans and not the PSU make no sense in my case. You think the difference is you messed with the Firmware? Yours is "Quiet" with out modifying the PSU?? I tried installing 3 120mm fans on top of the 32 Gb ram (which get extremily hot)and over the chips next to it and the cover is open so it wont heat up so much, any suggestion how to cool off better the server?

    ReplyDelete
    Replies
    1. Mine is comparatively quite, the PSU was not loud from the get go. But is the loudest thing after the modification.... But you say the cover is open.... are you holding down the top panel access switch? as if not all the fans (including the psu) fans will be running at max speed and so very loud.

      Delete
  4. A+, I just modify the fan with 47 ohm 1/2 watts resistors and then flash the BMC. Oh boy now I am running a jet plane to a bicycle, what an amazing tutorial. Big Thanks to everyone who contributed to this step by step tutorial. NOTE: Please Please Please use centos 6 32 bit to flash the BMC, otherwise you will bang your head every where. Thanks again

    ReplyDelete
    Replies
    1. Yup, you should use "Centos 6.4 i386"... as I say in the article... It was the only live DVD image I've been able to use... and this all work easily."

      Delete
  5. Gopod instructions, but when trying to run "./dell-adjust-fan-thresholds.py payload/bmcflsh.dat" using CentOS 7 Live I still needed to prefix it with 'python' ie; "python ./dell-adjust-fan-thresholds.py payload/bmcflsh.dat" or else I would receive "No such file or directory" error

    ReplyDelete
    Replies
    1. Sounds like on your installation python wasn't the associated executable for ".py" extension files is all, but sure, if you run the script you run it with python :) Good call.

      Delete