Dynamically Resize Multipath SCSI devices

I've seen quite a few "ways" on the web to resize the multipath SCSI devices that are created via the Gnu/Linux multipathd service, none of which actually did what they claimed or would ultimately result in downtime or unavailability of the multipathed physical storage. Here is what I did, with some addition comments, to get a NetApp filer LUN resized and then the larger storage available to the hosts that direct-connect (via LC/LC optical cables and a Qlogic HBA) to linux hosts running RHEL5u5.

SETUP

• 300GB /vol/vol_aiihdf5/base-lun on NetApp 3160A filer
• RHEL5u5 host, dual-single port 8GB QLogic HBA

RESIZING THE STORAGE

1. log into the primary controller for the physical storage location
   
   
2. assuming available space remaining on the aggregate on which the volume exists:
   
   # vol size awipsiihdf5 +170g
   
   The above increases the volume named awipsiihdf5 by 170GB. Note this is a delta increase, the exisiting size is 400GB, so the increase will set the size to 570GB.
   
   
3. Resize the pre-exisiting LUN created under the volume:
   
   # lun resize /vol/vol_awipsiihdf5/base-lun 570g
   
   
4. You can check that the resize was successful by issuing a lun show or other commands in the CLI on the filer, or you can use the NetApp Network Manager or Filer View interface instead. Regardless, resizing the physical storage on the filer is the first step in the process.
   
   
   

RESIZING THE MULTIPATH SCSI DEVICES

1. While you aren't really "resizing" the SCSI devices, you are re-registering them to "see" the new underlying physical storage increase on the linux hosts. First, check that the HBA correctly "sees" the newly resize physical storage on the filer. Assuming that you have some CLI interface on the linux host such as santools by NetApp or scli from QLogic.
   
   # sanlun lun show -p dx1-das:/vol/vol/awipsiihdf5/base-lun
   
   

   dx1-das:/vol/awipsiihdf5/base-lun (LUN 3) Lun state: GOOD
   Lun Size: 570.0g (609934639104) Controller_CF_State: Cluster Enabled
   Protocol: FCP Controller Partner: dx2-das
   DM-MP DevName: mpath4 (360a98000503361384e6f624e41594c59) dm-14
   Multipath-provider: NATIVE
   --------- ---------- ------- ------------ --------------------------------------------- ---------------
   sanlun Controller Primary Partner
   path Path /dev/ Host Controller Controller
   state type node HBA port port
   --------- ---------- ------- ------------ --------------------------------------------- ---------------
   GOOD primary sdd host2 0a --
   GOOD secondary sdh host3 -- 0a

   What we are are looking for is acknowledgement on the linux host that the HBA recognizes the new size of the LUN, indeed above the output shows 570.0g, which is what we resized to in previous steps.
   
   Should the new sizes not be seen, you may want to rescan the host bus, for example:
   
   # echo 1 > /sys/class/fc_host/host2/issue_lip
   # echo 1 > /sys/class/fc_host/host3/issue_lip
   
   This will cause the kernel to rescan the connections on the fibre channel host BUS adapters connected to PCI2 and PCI3 and hopefully relearn the sizes of the physical storage.
   
   
2. Check the multipath, it should show the original, unresized, value for the path. Note the mpath identifier is mpath4 from the above output.
   
   # multipath -ll mpath4
   
   

   mpath4 (360a98000503361384e6f624e41594c59) dm-14 NETAPP,LUN
   [size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
   \_ round-robin 0 [prio=4][active]
   \_ 2:0:0:3 sdd 8:48 [active][ready]
   \_ round-robin 0 [prio=1][enabled]
   \_ 3:0:0:3 sdh 8:112 [active][ready

   
   Indeed, multipath still shows the size of 400G. For future steps, note the two SCSI device IDs assigned by multipath to the fibre connections to the filer, above it is sdd (SCSI ID 2:0:0:3) with Major:Minor number 8:48 and sdh (SCSI ID 3:0:0:3) with Major:Minor 8:112.
   
   
3. This is where I see a myriad of suggestions from editing the device-mapper table manually, to removing the SCSI devices from the system and readding them manually. These are rather drastic steps considering multipath provides a nice CLI into the daemon, multipathd -k.
   
   Before continuing however, save off the exisiting device-mapper table in case we muck things up in the process, we can restore the table via dmsetup [suspend|reload|resume].
   
   # dmsetup table > /root/dmsetup_table.orig
   
   Should we muck things up, and need to reload the table, you would issue the below, but do not do this now, this is only an FYI:
   
   # dmsetup suspend; dmsetup reload /root/dmsetup_table.orig; dmsetup resume
   
   OK, back to resizing, once saving off the device-mapper table, check the current block device sizes of the SCSI devices assigned by multipath, remember above we noted sdd and sdh from the output of multipath -ll mpath4:
   
   # blockdev --getsize /dev/sdd
   836849664
   
   Blockdev allows you to call device ioctls from the command line, the --getsize options outputs the block size in 512-byte increments. One byte is 9.31322575 X 10^-10 GB in case you want to verify, indeed the current size of the block device assigned by multipath to the physical storage is 400GB (note: there may be some discrepancies by about 1 GB due to conversions and overhead).
   
   Reread the device from the partition table:
   
   # blockdev --rereadpt /dev/sdd
   
   And issue a getsize again:
   
   # blockdev -- getsize /dev/sdd
   1191278592
   
   Indeed now, the device is "seen" as 570GB. However, only one of the two paths has been reread, remember there are at least two paths (or SCSI devices assigned) when using multipath (hence the name). There could be more than two paths, but at minimum, and in this case, we have two. Reread the second (and subsequent) SCSI devices:
   
   # blockdev --getsize /dev/sdh
   836849664
   
   # blockdev --rereadpt /dev/sdh
   
   # blockdev -- getsize /dev/sdh
   1191278592
   
   OK, so now both SCSI devices in the multipath are changed, in the kernel layer. Multipath however still has not been updated to the new sizes. We now need to re-register multipath using the CLI.
   
   # multipathd -k
   
   This will get you into the multipath interface, simply resize the multipath ID assigned to the LUN, in our case it was mpath4:
   
   multipathd> resize multipath mpath4
   OK
   
   That's it, now reissue a scan on the multipath and you should now see the larger size:
   
   # multipath -ll mpath4
   
   

   
   mpath4 (360a98000503361384e6f624e41594c59) dm-14 NETAPP,LUN
   [size=568G][features=1 queue_if_no_path][hwhandler=0][rw]
   \_ round-robin 0 [prio=4][enabled]
   \_ 2:0:0:3 sdd 8:48 [active][ready]
   \_ round-robin 0 [prio=1][enabled]
   \_ 3:0:0:3 sdh 8:112 [active][ready]

   
   That's all fine-and-dandy, but the filesystem we created on this storage deivce path won't be resized until we then resize it. We created a physical volume out of the device multipath (/dev/dm-14) and use LVM. Note /dev/dm-14 is sort of an all-incompasing device map to the storage location. You can theoretically use the single SCSI block devices, e.g /dev/sdd or /dev/sdh, but if you only use one, the should that path fail, you will lose connectivity to your storage. Always use the device-multipath device name.
   
   Here is a quick, "what to do" should you have to resize a LVM setup on the physical storage, after this, you will have a 170GB larger filesystem to use and abuse until management tells you to increase it again instead of fixing the problems related to the software not purging the directory ;-)
   
   # pvscan | grep hdf5
   PV /dev/dm-14 VG vg_aiihdf5 lvm2 [399.04 GB / 40.00 MB free]
   
   # pvresize /dev/dm-14
   Physical volume "/dev/dm-14" changed
   1 physical volume(s) resized / 0 physical volume(s) not resized
   
   # pvscan | grep hdf5
   PV /dev/dm-14 VG vg_aiihdf5 lvm2 [568.04 GB / 169.04 GB free]
   
   
   Ensure volume group was resized during pvresize
   
   # vgdisplay /dev/vg_aiihdf5 | grep "Size"
   VG Size 568.04 GB
   PE Size 4.00 MB
   Alloc PE / Size 102144 / 399.00 GB
   Free PE / Size 43275 / 169.04 GB
   
   
   Resize the logical volume, while online (without having to umount or lvchange). Note this will take a few minutes
   
   # resize2fs -fp /dev/vg_aiihdf5/awipsiihdf5
   resize2fs 1.39 (29-May-2006)
   Filesystem at /dev/vg_aiihdf5/awipsiihdf5 is mounted on /awips2/edex/data/hdf5; on-line resizing required
   Performing an on-line resize of /dev/vg_aiihdf5/awipsiihdf5 to 148111360 (4k) blocks.
   The filesystem on /dev/vg_aiihdf5/awipsiihdf5 is now 148111360 blocks long.
   
   
   Verify the hdf5 partition has been resized:
   
   # df | grep hdf5
   /dev/mapper/vg_aiihdf5-awipsiihdf5
   583149656 142582728 410946784 26% /awips2/edex/data/hdf5