Veritas DMP statistics notes

Original source:http://unix-essentials.blogspot.com/search/label/SAN

VxVM - DMP Load Balancing

How to configure load balancing across multiple primary paths in an Active/Passive DMP environment using VERITAS Volume Manager (tm) 4.x

Details:

In an Active/Passive Dynamic Multipathing (DMP) environment, it is possible that there may be multiple primary paths to a device due to a mesh of SAN switches. Volume Manager 4.0 can be configured to allow load balancing across multiple primary paths. This TechNote explains how to do this.

Note: This feature was not available in releases prior to 4.0

In the following example, device c3t2d15 belongs to a disk group named tonydg and contains a simple concatenated volume named tony1. The device has eight primary paths:

# vxdisk list c3t2d15s2
Device: c3t2d15s2
<..snip..>
numpaths: 8
c2t0d15s2 state=enabled type=primary
c2t1d15s2 state=enabled type=primary
c3t1d15s2 state=enabled type=primary
c3t2d15s2 state=enabled type=primary
c4t3d15s2 state=enabled type=primary
c4t4d15s2 state=enabled type=primary
c5t3d15s2 state=enabled type=primary
c5t5d15s2 state=enabled type=primary



DMP statistics are enabled and a read based workload is applied to the volume:

# vxdmpadm iostat start

# dd if=/dev/vx/rdsk/tonydg/tony1 of=/dev/null bs=512k count=6144



By displaying the DMP statistics for device c3t2d15, it can be seen that all I/Os are being directed to one subpath, c5t5d15:

# vxdmpadm iostat show dmpnodename=c3t2d15s2
cpu usage = 8023us per cpu memory = 32768b
OPERATIONS MBYTES AVG TIME(ms)
PATHNAME READS WRITES READS WRITES READS WRITES
c2t0d15s2 0 0 0 0 0.000000 0.000000
c2t1d15s2 0 0 0 0 0.000000 0.000000
c3t1d15s2 0 0 0 0 0.000000 0.000000
c3t2d15s2 0 0 0 0 0.000000 0.000000
c4t3d15s2 0 0 0 0 0.000000 0.000000
c4t4d15s2 0 0 0 0 0.000000 0.000000
c5t3d15s2 0 0 0 0 0.000000 0.000000
c5t5d15s2 6144 0 3145728 0 0.011081 0.000000



Using vxdmpadm, it is clear that the default iopolicy for this enclosure is Single Active (which explains the above behavior):

# vxdmpadm getattr enclosure HDS92000 iopolicy
ENCLR_NAME DEFAULT CURRENT
============================================
HDS92000 Single-Active Single-Active



To load balance across multiple primaries, the iopolicy should be set to round-robin:

# vxdmpadm setattr enclosure HDS92000 iopolicy=round-robin

# vxdmpadm getattr enclosure HDS92000 iopolicy
ENCLR_NAME DEFAULT CURRENT
============================================
HDS92000 Single-Active Round-Robin



The DMP statistics are reset and the same workload is applied to the volume, load balancing across the primary paths can be seen.

# vxdmpadm iostat reset

# dd if=/dev/vx/rdsk/tonydg/tony1 of=/dev/null bs=512k count=6144

# vxdmpadm iostat show dmpnodename=c3t2d15s2
cpu usage = 7223us per cpu memory = 32768b
OPERATIONS MBYTES AVG TIME(ms)
PATHNAME READS WRITES READS WRITES READS WRITES
c2t0d15s2 786 0 402432 0 0.011013 0.000000
c2t1d15s2 778 0 398336 0 0.011267 0.000000
c3t1d15s2 772 0 395264 0 0.011205 0.000000
c3t2d15s2 744 0 380928 0 0.011094 0.000000
c4t3d15s2 750 0 384000 0 0.011029 0.000000
c4t4d15s2 768 0 393216 0 0.011169 0.000000
c5t3d15s2 733 0 375296 0 0.010943 0.000000
c5t5d15s2 813 0 416256 0 0.011068 0.000000



The enclosure can be returned to its default policy of single active with the following:

# vxdmpadm setattr enclosure HDS92000 iopolicy=singleactive



Note: This setting is not persistent across reboots, I/O policies will revert to their default. Persistence can be configured by adding the relevant vxdmpadm command to a boot script.

Some veritas usefull troubleshooting issues

original source: http://www.knodel.net/jeff/blog

Filesystems in shared disk group will not mount shared:

Tuesday, July 25th, 2006

Problem: You have imported your CVM disk group as shared, and its imported on all nodes of your cluster, but when you try to mount a volume, you get:

# mount -F vxfs -o cluster /dev/vx/dsk/oratestdg/oratest /oratest
UX:vxfs mount: ERROR: V-3-20003: Cannot open /dev/vx/dsk/oratestdg/oratest: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
Solution:  The problem is that you do not have the disk group activation mode set properly.  If you run a vxdg list [dgname] , you can see the activation mode for each node.

Examine the contents of your /etc/default/vxdg.  You will probably see a line like:

default_activation_mode=off

Change it to default_activation_mode=sw, and restart veritas. 

 If you want to do it interactively by disk group, try vxdg -g [dgname] set activation=sw

 

Posted in Veritas, Veritas Cluster | No Comments »

Cannot add device to a CDS Disk Group

Monday, July 3rd, 2006

Problem: You are trying to initialize a disk group, and you are getting the following error:

# vxdg init vxfencoorddg c2t0d64
VxVM vxdg ERROR V-5-1-6478 Device c2t0d64s2 cannot be added to a CDS disk group
Solution: Set up the disk to be a CDS disk before running the init.

# vxdisksetup -i c2t0d64  format=cdsdisk

# vxdg init vxfencoorddg c2t0d64 

Posted in Veritas, Veritas Cluster | No Comments »

VERITAS vxfentsthdw will not process i/o fencing quorum disks

Monday, July 3rd, 2006

Problem: You are trying to run veritas’s hardware test, and despite meeting the hardware requirements, the tests fail with:

VCS FEN vxfenadm ERROR V-11-2-1204 Command 3 FAILED for: /dev/vx/rdmp/c2t0d215s2
VCS FEN vxfenadm ERROR V-11-2-1210 CLEAR failed for: /dev/vx/rdmp/c2t0d215s2
VCS FEN vxfenadm ERROR V-11-2-1133 Error returned
Solution: Veritas’ tool cannot use the Veritas Dynamic Multipathing device entry; you cannot use the /dev/rdmp path to the disks.  Try using the first /dev/rdsk/CTDS path to the device available.

While I am at it — if you are using the raw disk path (ie /dev/rdsk…) and it says that the device does not exist, and an ls -l (path) shows that it does — you need to write a solaris label onto the disk before the veritas processes can use the disk.

Sample Error: “Disk /dev/rdsk/c3t0d219s2 is not existent on host hostname”
#  format c#t#d#; vxdctl enable

Posted in Veritas, Veritas Cluster | No Comments »

Reattach Failed or Failing disks in Veritas

Monday, June 26th, 2006

Problem: A temporary problem caused veritas to label your disks as “Failed” or “Failing”:

Solution for Failed: vxreattach

Solution for Failing: vxedit -g  set failing=false

Posted in Veritas | No Comments »

Veritas Plexes show “DISABLED RECOVER” in status

Monday, June 26th, 2006

Problem: The vxprint output shows that the plexes for the volume are in “DISABLED RECOVER” state.

Solution:

The following commands must be run on a plex to change the state of the plex to “CLEAN”. The volume can then be started, but a fsck may be required before mounting the file system.# vxmend -o force off

# vxmend on

# vxmend fix clean

# vxvol start

# fsck -F vxfs /dev/vx/rdsk//

# mount -F vxfs /dev/vx/dsk// /mountpoint

Filesystems in shared disk group will not mount shared

Problem: You have imported your CVM disk group as shared, and its imported on all nodes of your cluster, but when you try to mount a volume, you get:

# mount -F vxfs -o cluster /dev/vx/dsk/oratestdg/oratest /oratest
UX:vxfs mount: ERROR: V-3-20003: Cannot open /dev/vx/dsk/oratestdg/oratest: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
Solution:  The problem is that you do not have the disk group activation mode set properly.  If you run a vxdg list [dgname] , you can see the activation mode for each node.

Examine the contents of your /etc/default/vxdg.  You will probably see a line like:

default_activation_mode=off

Change it to default_activation_mode=sw, and restart veritas. 

 If you want to do it interactively by disk group, try vxdg -g [dgname] set activation=sw

Show list of emc disks that do not have SCSI-3 Persistant Reservation set

Problem: How do you see which emc disks do not have SCSI-3 Persistant Reservation set:

for disk in `symdev list |grep -v “Not” |awk ‘{print $1}’` |sort -u do

print -n $disk

symdev show $disk |grep SCSI

done |grep -v Enabled

Apache Missing from Solaris 10

If you find you are missing your ‘built in’ apache2 installation, go back to your install media (or jumpstart image) and get the following packages:

Apache: SUNWapchS, SUNWapchd, SUNWapchr, SUNWapchu

Apache2:SUNWapch2S, SUNWapch2d, SUNWapch2r, SUNWapch2u

Install each using pkgadd.  Once you have done that, perform a kill -HUP on your /lib/svc/bin/svc.configd process (ps -ef |grep svc to get the processs id). or you will end up with a “svcadm: Pattern ‘apache2′ doesn’t match any instances” error when you run the svcadm startup command. 

Which happens to be: svcadm enable apache2
You’ll probably want to configure the /etc/apache2/httpd.conf as well.  (just copy over the /etc/apache2/httpd.conf.example and edit the ServerName and ServerAdmin as a start.)

Jumpstart Pain

So, here’s the jumpstart errors that I’ve run into recently, and what the problem was:

Command:

 boot net:speed=100,duplex=full - install

Hangs on:

 Timeout waiting for ARP/RARP packet

This error basically means that no one is answering your servers ‘who am i’ requests — in other words, your boot server is not answering/seeing its ARP requests.

 In this case, for me, it was that my ‘net’ device was set to a device that was connected to a cluster heartbeat, and not a real network.  Could also be because you have not set up your boot server properly.  Try a net-watch-all from the eeprom and see what networks have traffic.

Command:

 boot /pci@1f,700000/network@1:speed=100,duplex=full - install -v

Hangs on: Requesting Internet address for MAC

You probably forgot to start/restart rpcbootparamd

/usr/sbin/rpc.bootparamd start

Hangs on:

boot: cannot open kernel/sparcv9/unix
Enter filename [kernel/sparcv9/unix]:

Snoop Says: NFS R LOOKUP3 Permission denied

Check that your boot server is actually properly sharing the Tools/Boot Directory — I actually had to do an nfs.server stop and nfs.server start.

Says: Select A Language…

You’re not finding your sysidcfg file — check bootparams on the bootserver, make sure that sysidcfg is shared properly from the install server.

Searching for JumpStart directory…
not found
Check for matching shortname entry in /etc/hosts on install server.  Also make sure IP/Netmask not specified in sysidcfg.

Mounting an ISO under solaris

Mounting an ISO under solaris
If you have an iso image you want to view as a filesystem, rather than burn to something:

mount -F hsfs -o ro `lofiadm -a /path/to/source.iso` /mountpoint