HP-UX: Useful Fiber Channel and Multipath Commands

So this was an interesting exercise I had the other day on one of our HP-UX servers that took me quite a bit of time to research. Which of course makes it possible good blogging material because along the way there are useful commands for fiber channel discovery on HP-UX.

Which incidentally makes it possibly not good blogging material..considering it’s on HP-UX. Meaning … this is probably a really small target audience here.

(Anyone actually still using HP-UX out there?

Seriously, I feel like that scene in “Sixteen Candles” when “The Donger” is trying to tell Jake that Samantha is actually at the church getting married.

Jake: “Don’t Jerk me around man, what unix are you using?”
Donger: “HP-UX”
Jake: “What?”
Donger: “We’re using HP-UX. We have HP PA-RISC servers running HP-UX”
Jake: “HP-UX?”
Donger:”HP-UX.”
Jake: “HP-UX?”
Donger: “Yeah! HP-UX.”
**Door Slams**
Jake: “HP-UX?”
Donger (behind closed door): “Yeah! HP-UX! Geeeez”

Sixteen Candles Long Duck Dong "What's happening hot stuff?"

The HP-UX Problem I Am Having With Our Fiber Storage

Short version: The HP-UX server was attached to the external fiber storage via a switch. There are multiple (2) fiber connections which means there are multiple paths to the storage (think: multiple ways to get home from work).

External Fiber storage, in an effort to load balance, chooses “preferred paths” for each logical disk it presents to the storage.

So as an example, say logical disks 1 through 10 the storage manager will say “You use path A”. And then logical disks 11 through 20 the storage might say “You use path B”.

The problem I am having: the logical disk does not stay on the path that the storage wants it to. It keeps bouncing between the preferred path and non-preferred path.

Possible Solutions

Off the top of my head I wasn’t sure but the things I figured I would need to look at:

1) There is an intermittent hardware/connectivity issue on the preferred path which is why it kept flapping
2) The preferred path is set on HP-UX is opposite of the preferred path designated by the storage
3) The load balancing is not set correctly for preferred path (perhaps it is set to round robin)


Sponsored Links

Restrictions

Could not get access to the fiber switch. I could get access to the HP-UX server itself and could also see the storage GUI (incidentally it was IBM Storage Manager Client 10 managing an external DS4000 series storage, and it was the Storage Manager Recovery Guru that would complain on-and-off as the LUN mapping kept changing from preferred path to non-preferred path)

Typographical Conventions

Bold: Command line prompts
Bold Blue: Things you type
Bold Red: Output on the screen to look for

Let’s GO!

I like to do a bit of fact finding first, to get my head wrapped around the environment.

Step 1: Use the “ioscan” command to get some fiber card info

# ioscan -fnC fc
Class I H/W Path Driver S/W State H/W Type Description
================================================================
fc 2 0/3/1/0 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
/dev/fcd2
fc 3 0/3/1/1 fcd CLAIMED INTERFACE HP AB379-60101 4Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 2)
/dev/fcd3
#

This gives us the two HBA devices and their hardware paths:

/dev/fcd2: 0/3/1/0
/dev/fcd3: 0/3/1/1

Step 2: Use the “fcmsutil” command to get some WWN info

# fcmsutil /dev/fcd2

Vendor ID is = 0x1077
Device ID is = 0x2422
PCI Sub-system Vendor ID is = 0x103C
PCI Sub-system ID is = 0x12D7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.4.4
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x010500
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b000069b1d9
N_Port Port World Wide Name = 0x50060b000069b1d8
Switch Port World Wide Name = 0x200500051e05b304
Switch Node World Wide Name = 0x100000051e05b304
N_Port Symbolic Port Name = odevux34_fcd2
N_Port Symbolic Node Name = odevux34_HP-UX_B.11.31
Driver state = ONLINE
Hardware Path is = 0/3/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) fcd B.11.31.0909 Jun 8 2009

# fcmsutil /dev/fcd3

Vendor ID is = 0x1077
Device ID is = 0x2422
PCI Sub-system Vendor ID is = 0x103C
PCI Sub-system ID is = 0x12D7
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.4.4
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x010500
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b000069b1db
N_Port Port World Wide Name = 0x50060b000069b1da
Switch Port World Wide Name = 0x200500051e05940a
Switch Node World Wide Name = 0x100000051e05940a
N_Port Symbolic Port Name = odevux34_fcd3
N_Port Symbolic Node Name = odevux34_HP-UX_B.11.31
Driver state = ONLINE
Hardware Path is = 0/3/1/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) fcd B.11.31.0909 Jun 8 2009

#

We can see the WWN’s that will appear on the switches (and also in the host group on the storage manager) as well as the status (“ONLINE”) and the hardware path.

Step 3: Use “ioscan” to find the external storage disks in question

# ioscan -f -n -C disk
Class I H/W Path Driver S/W State H/W Type Description
=======================================================================
disk 2 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-C
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 0 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373455LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 1 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373455LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 6 0/3/1/0.1.7.0.0.0.0 sdisk CLAIMED DEVICE IBM 1722-600
/dev/dsk/c9t0d0 /dev/rdsk/c9t0d0

disk 47 0/3/1/0.1.7.0.0.3.7 sdisk CLAIMED DEVICE IBM Universal Xport
/dev/dsk/c9t3d7 /dev/rdsk/c9t3d7
disk 30 0/3/1/1.1.7.0.0.0.0 sdisk CLAIMED DEVICE IBM 1722-600
/dev/dsk/c8t0d0 /dev/rdsk/c8t0d0

disk 48 0/3/1/1.1.7.0.0.3.7 sdisk CLAIMED DEVICE IBM Universal Xport
/dev/dsk/c8t3d7 /dev/rdsk/c8t3d7
#

Above you can see there are two IBM paths. This represents two single direct paths to the same storage.

/dev/dsk/c9t0d0 with hardware path 0/3/1/0.1.7.0.0.0.0
/dev/dsk/c8t0d0 with hardware path 0/3/1/1.1.7.0.0.0.0

And if you remember from the above “ioscan -fNc fc” output

/dev/fcd2 (device driver) = 0/3/1/0 (physical hardware path)
/dev/fcd3 (device driver) = 0/3/1/1 (physical hardware path)

Which means:

/dev/dsk/c9t0d0 is talking on the fiber port designated as /dev/fcd2
/dev/dsk/c8t0d0 is talking on the fiber port designated as /dev/fcd3

This gives us some good possible debugging info, especially if we later have access to the switch and can see error logs on the switch, since on the switch and in the external storage manager GUI:

/dev/fcd2 will have the port name 0x50060b000069b1d8
/dev/fcd3 will have the port name 0x50060b000069b1da

Next command:

# ioscan -f -n -N -C disk
Class I H/W Path Driver S/W State H/W Type Description
===================================================================
disk 3 64000/0xfa00/0x0 esdisk CLAIMED DEVICE HP 73.4GST373455LC
/dev/disk/disk3 /dev/rdisk/disk3
disk 4 64000/0xfa00/0x1 esdisk CLAIMED DEVICE HP 73.4GST373455LC
/dev/disk/disk4 /dev/rdisk/disk4
disk 5 64000/0xfa00/0x2 esdisk CLAIMED DEVICE TEAC DV-28E-C
/dev/disk/disk5 /dev/rdisk/disk5
disk 46 64000/0xfa00/0xe esdisk CLAIMED DEVICE IBM 1722-600
/dev/disk/disk46 /dev/rdisk/disk46

disk 50 64000/0xfa00/0xf esdisk CLAIMED DEVICE IBM Universal Xport
/dev/disk/disk50 /dev/rdisk/disk50
disk 51 64000/0xfa00/0x10 esdisk CLAIMED DEVICE IBM Universal Xport
/dev/disk/disk51 /dev/rdisk/disk51
#

Here you can see /dev/disk/disk46 is the multipath disk with hardware path 64000/0xfa00/0xe.

Meaning when you refer to the disk through the device /dev/disk/disk46, that device will write to the disk using one of the two direct paths of /dev/dsk/c9t0d0 and /dev/dsk/c8t0d0 depending on the settings for load balancing, preferred path, etc.

So far: mostly what we have done is determined that yes there is a multipath, and yes it seems to be working.

Step 4: Use the “scsimgr” command to get load balancing

So let’s check out that multipath

Using the hardware path for the multipath disk, let’s see what the load balancing policy is.

# scsimgr get_attr -H 64000/0xfa00/0xe -a load_bal_policy

SCSI ATTRIBUTES FOR LUN : 64000/0xfa00/0xe

name = load_bal_policy
current = round_robin
default = round_robin
saved =

#

AY! THERE’S THE RUB!

And there we are. The HP system is using a round-robin load balancing policy so it will keep switching the path used, which is why our storage manager is going crazy.

Step 5: Use the “scsimgr” command to set the load balancing to “preferred path”

# scsimgr save_attr -H 64000/0xfa00/0xe -a load_bal_policy="preferred_path"

And let’s make sure it took:

# scsimgr get_attr -H 64000/0xfa00/0xe -a load_bal_policy

SCSI ATTRIBUTES FOR LUN : 64000/0xfa00/0xe

name = load_bal_policy
current = preferred_path
default = round_robin
saved = preferred_path

#

Step 6: Telling HP-UX Which Path To Use As The Preferred Path

And now we have to tell the HP-UX system WHICH path to use as the preferred path.

Unfortunately, this will be a 50/50 guess. Because I do not have access to the switch. I can see on the Storage Manager GUI it wants the preferred path to be on Controller B. However, on the storage manager GUI you only define host groups which means you throw in Storage Controller A and Storage Controller B into the same host group as the two fiber ports on the HP-UX server.

If I had access to the switch, I might have an easier time seeing which one of the two HP-UX fiber ports is talking to Controller B.

Since I can’t see that, the only thing to do is set the path to one, see if the Storage Manager complains, and if there are no complaints, we guessed right! If it complains, than we set it to the other path.

First, we can verify that no preferred path is set:

# scsimgr get_attr -D /dev/rdisk/disk46 -a preferred_path

SCSI ATTRIBUTES FOR LUN : /dev/rdisk/disk46

name = preferred_path
current =
default =
saved =

#

And let’s take a quick look at the hardware paths again

# scsimgr lun_map -D /dev/rdisk/disk46

LUN PATH INFORMATION FOR LUN : /dev/rdisk/disk46

Total number of LUN paths = 2
World Wide Identifier(WWID) = 0x600a0b8000225d7d00005b3150fd79bf

LUN path : lunpath36
Class = lunpath
Instance = 36
Hardware path = 0/3/1/0.0x200700a0b81984d6.0x4000000000000000
SCSI transport protocol = fibre_channel
State = ACTIVE
Last Open or Close state = ACTIVE

LUN path : lunpath35
Class = lunpath
Instance = 35
Hardware path = 0/3/1/1.0x200600a0b81984d5.0x4000000000000000
SCSI transport protocol = fibre_channel
State = ACTIVE
Last Open or Close state = ACTIVE

#

And now we pick one. And set it

# scsimgr save_attr -D /dev/rdisk/disk46 -a preferred_path=0/3/1/0.0x200700a0b81984d6.0x4000000000000000
Value of attribute preferred_path saved successfully
#

And now check it

# scsimgr get_attr -D /dev/rdisk/disk46 -a preferred_path

SCSI ATTRIBUTES FOR LUN : /dev/rdisk/disk46

name = preferred_path
current = 0/3/1/0.0x200700a0b81984d6.0x4000000000000000(lunpath36)
default =
saved = 0/3/1/0.0x200700a0b81984d6.0x4000000000000000(lunpath36)

#

If it complains, then try the other hardware path. If it doesn’t, then you’re good!

For the sake of this blog entry, I’m going to say..woohooo! My first guess worked!

And…DONE!


Sponsored Links

P.S

Looking at this blog post. .wow I’m going to have to find a better theme and code output to make this thing readable …. yeesh apologies!

P.P.S

, Unix

6 thoughts on “HP-UX: Useful Fiber Channel and Multipath Commands

  1. woohooo! You have a very nice way of explaining; even a dummy can understand your explanation. I hope to interact with you sometimes once I replace my faulty fiber channel adapters on rp4440.
    You can help – there are two adapter, 8 external disk & I was told by Admin guys that my system had crashed. On check found both adapter dead..! Surely one must have gone down long ago maybe around 2010 – 2011. Can I find out failure date of each adapter?
    I am sure you have the script… do drop me your steps… at jamini.padhi@gmail.com
    Thanks in advance… have fun.

    • Hey JB: That’s a tough one. You’d need to have your messages from 2010/2011 timeframe. It would be logged in some of your messages file, possibly that one of your paths went down. I don’t think I can pinpoint otherwise when an adapter went down. Sorry, can’t help :/

  2. You would be amazed how many people still use HP-UX – we do, so so do some of our customers. I look after 12 HP-UX boxes globally.

    Thanks for the guide – I successfully added one of our boxes to our 3PAR.

    • Hey Andy – it may sound strange but I would be both amazed and not surprised. A lot of the old Sun and HP servers are like tanks – pretty reliable (except for those Sun servers in the early 2000′s, the E4500 or E450 if I recall correctly?). And the p-series by IBM … tanks. :) Thanks for your comment and glad I could help!

    • I’ll tell you what, it kinda warms my heart to hear comments like this. My first job was at a CAD outfit (big one) and all servers and desktops were sun sparc and hp pa-risc and a small number of sgi too. So even though I was very new, the fact I got to touch those machines way back when AND they’re still around is just really cool. Reminiscing fondly I guess :)

      I even remember at the time big storage was just starting out and we had this big big big huge Auspex and this new company was also coming out with storage called NAC (at least that’s what the field engineer called it at the time ..a.k.a. Network Appliance Company a.k.a. Netapp). And the Auspex was huge and a beast but had so many problems and continued to have so many problems and then we just kept buying more and more and more NACs. I remember each time our Auspex guy came in to service the Auspex (which was OFTEN) he’d be visibly dismayed seeing that since his last visit we’d have added another NAC.

      Fun times :)

Leave a Comment

Your email address will not be published. Required fields are marked *

Spam protection by WP Captcha-Free