Some veritas usefull troubleshooting issues

original source: http://www.knodel.net/jeff/blog

Filesystems in shared disk group will not mount shared:

Tuesday, July 25th, 2006

Problem: You have imported your CVM disk group as shared, and its imported on all nodes of your cluster, but when you try to mount a volume, you get:

# mount -F vxfs -o cluster /dev/vx/dsk/oratestdg/oratest /oratest
UX:vxfs mount: ERROR: V-3-20003: Cannot open /dev/vx/dsk/oratestdg/oratest: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
Solution:  The problem is that you do not have the disk group activation mode set properly.  If you run a vxdg list [dgname] , you can see the activation mode for each node.

Examine the contents of your /etc/default/vxdg.  You will probably see a line like:

default_activation_mode=off

Change it to default_activation_mode=sw, and restart veritas. 

 If you want to do it interactively by disk group, try vxdg -g [dgname] set activation=sw

 

Posted in Veritas, Veritas Cluster | No Comments »

Cannot add device to a CDS Disk Group

Monday, July 3rd, 2006

Problem: You are trying to initialize a disk group, and you are getting the following error:

# vxdg init vxfencoorddg c2t0d64
VxVM vxdg ERROR V-5-1-6478 Device c2t0d64s2 cannot be added to a CDS disk group
Solution: Set up the disk to be a CDS disk before running the init.

# vxdisksetup -i c2t0d64  format=cdsdisk

# vxdg init vxfencoorddg c2t0d64 

Posted in Veritas, Veritas Cluster | No Comments »

VERITAS vxfentsthdw will not process i/o fencing quorum disks

Monday, July 3rd, 2006

Problem: You are trying to run veritas’s hardware test, and despite meeting the hardware requirements, the tests fail with:

VCS FEN vxfenadm ERROR V-11-2-1204 Command 3 FAILED for: /dev/vx/rdmp/c2t0d215s2
VCS FEN vxfenadm ERROR V-11-2-1210 CLEAR failed for: /dev/vx/rdmp/c2t0d215s2
VCS FEN vxfenadm ERROR V-11-2-1133 Error returned
Solution: Veritas’ tool cannot use the Veritas Dynamic Multipathing device entry; you cannot use the /dev/rdmp path to the disks.  Try using the first /dev/rdsk/CTDS path to the device available.

While I am at it — if you are using the raw disk path (ie /dev/rdsk…) and it says that the device does not exist, and an ls -l (path) shows that it does — you need to write a solaris label onto the disk before the veritas processes can use the disk.

Sample Error: “Disk /dev/rdsk/c3t0d219s2 is not existent on host hostname”
#  format c#t#d#; vxdctl enable

Posted in Veritas, Veritas Cluster | No Comments »

Reattach Failed or Failing disks in Veritas

Monday, June 26th, 2006

Problem: A temporary problem caused veritas to label your disks as “Failed” or “Failing”:

Solution for Failed: vxreattach

Solution for Failing: vxedit -g  set failing=false

Posted in Veritas | No Comments »

Veritas Plexes show “DISABLED RECOVER” in status

Monday, June 26th, 2006

Problem: The vxprint output shows that the plexes for the volume are in “DISABLED RECOVER” state.

Solution:

The following commands must be run on a plex to change the state of the plex to “CLEAN”. The volume can then be started, but a fsck may be required before mounting the file system.# vxmend -o force off

# vxmend on

# vxmend fix clean

# vxvol start

# fsck -F vxfs /dev/vx/rdsk//

# mount -F vxfs /dev/vx/dsk// /mountpoint

Filesystems in shared disk group will not mount shared

Problem: You have imported your CVM disk group as shared, and its imported on all nodes of your cluster, but when you try to mount a volume, you get:

# mount -F vxfs -o cluster /dev/vx/dsk/oratestdg/oratest /oratest
UX:vxfs mount: ERROR: V-3-20003: Cannot open /dev/vx/dsk/oratestdg/oratest: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
Solution:  The problem is that you do not have the disk group activation mode set properly.  If you run a vxdg list [dgname] , you can see the activation mode for each node.

Examine the contents of your /etc/default/vxdg.  You will probably see a line like:

default_activation_mode=off

Change it to default_activation_mode=sw, and restart veritas. 

 If you want to do it interactively by disk group, try vxdg -g [dgname] set activation=sw

Show list of emc disks that do not have SCSI-3 Persistant Reservation set

Problem: How do you see which emc disks do not have SCSI-3 Persistant Reservation set:

for disk in `symdev list |grep -v “Not” |awk ‘{print $1}’` |sort -u do

print -n $disk

symdev show $disk |grep SCSI

done |grep -v Enabled

Apache Missing from Solaris 10

If you find you are missing your ‘built in’ apache2 installation, go back to your install media (or jumpstart image) and get the following packages:

Apache: SUNWapchS, SUNWapchd, SUNWapchr, SUNWapchu

Apache2:SUNWapch2S, SUNWapch2d, SUNWapch2r, SUNWapch2u

Install each using pkgadd.  Once you have done that, perform a kill -HUP on your /lib/svc/bin/svc.configd process (ps -ef |grep svc to get the processs id). or you will end up with a “svcadm: Pattern ‘apache2′ doesn’t match any instances” error when you run the svcadm startup command. 

Which happens to be: svcadm enable apache2
You’ll probably want to configure the /etc/apache2/httpd.conf as well.  (just copy over the /etc/apache2/httpd.conf.example and edit the ServerName and ServerAdmin as a start.)

Jumpstart Pain

So, here’s the jumpstart errors that I’ve run into recently, and what the problem was:

Command:

 boot net:speed=100,duplex=full - install

Hangs on:

 Timeout waiting for ARP/RARP packet

This error basically means that no one is answering your servers ‘who am i’ requests — in other words, your boot server is not answering/seeing its ARP requests.

 In this case, for me, it was that my ‘net’ device was set to a device that was connected to a cluster heartbeat, and not a real network.  Could also be because you have not set up your boot server properly.  Try a net-watch-all from the eeprom and see what networks have traffic.

Command:

 boot /pci@1f,700000/network@1:speed=100,duplex=full - install -v

Hangs on: Requesting Internet address for MAC

You probably forgot to start/restart rpcbootparamd

/usr/sbin/rpc.bootparamd start

Hangs on:

boot: cannot open kernel/sparcv9/unix
Enter filename [kernel/sparcv9/unix]:

Snoop Says: NFS R LOOKUP3 Permission denied

Check that your boot server is actually properly sharing the Tools/Boot Directory — I actually had to do an nfs.server stop and nfs.server start.

Says: Select A Language…

You’re not finding your sysidcfg file — check bootparams on the bootserver, make sure that sysidcfg is shared properly from the install server.

Searching for JumpStart directory…
not found
Check for matching shortname entry in /etc/hosts on install server.  Also make sure IP/Netmask not specified in sysidcfg.

Mounting an ISO under solaris

Mounting an ISO under solaris
If you have an iso image you want to view as a filesystem, rather than burn to something:

mount -F hsfs -o ro `lofiadm -a /path/to/source.iso` /mountpoint