Solaris Volume Manager recovery of metdb on Solaris 9

We had an sad issue, when someone was patching v240 with Solaris 9 with some patch cluster, and accidently everything went nuts. Maybe not making reconfigure reboot, but just ordinary one cause this, I don`t know - we were seeing this horror after reboot:

Rebooting with command: boot
Boot device: disk  File and args:                                              
SunOS Release 5.9 Version Generic_122300-42 64-bit                             
Copyright 1983-2003 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.                                               
sorry, variable 'ssfcp_enable_auto_configuration' is not defined in the 'fcp' mo
dule                                                                           
Cannot mount root on /pseudo/md@0:0,0,blk fstype ufs                           

panic[cpu1]/thread=140a000: vfs_mountroot: cannot mount root

0000000001409970 genunix:vfs_mountroot+70 (0, 0, 0, 200, 1472178, 0)           
%l0-3: 0000000001465800 0000000001465800 0000000000002000 00000000014af400   
%l4-7: 00000000014b5000 0000000001411e88 0000000001466000 0000000001469400   
0000000001409a20 genunix:main+90 (1409ba0, f005bf10, 1409ec0, 395dac, 2000, 500)
%l0-3: 0000000000000001 000000000140a000 0000000001413048 0000000000000000   
%l4-7: 0000000078002000 000000000039e000 00000000014bdc10 0000000001067f00   

skipping system dump - no dump device configured                               
rebooting...

Googling on this issue led to conclusion, that metadb went nuts to some reason, and now system cannot correctly startup. To resolve this issue we need to boot system off net or dvd, mount HDD with root on it, tweak some files responsible for metadb and SVM in general, recreate metadevices & metadb, reattach them and restart.

So, let`s go:

1. Boot system from net (in my case I have JET server in same subnet)

{l} ok boot net -s

2. Mount root slice of one of the HDD`s that were used in / partition

# mount /dev/dsk/c1t0d0s0 /mnt

3. Modify /mnt/etc/system file to delete (or comment with * symbol) everything between

Begin MDD root info (Do not edit this line.)

End MDD root info (Do not edit this line.)

4. Modify the /mnt/etc/vfstab file to replace or comment all the /dev/md/dsk entries with /dev/dsk, i.e like this one:

#/dev/md/dsk/d0   /dev/md/rdsk/d0   /   ufs   1   no   logging

/dev/dsk/c1t0d0s0   /dev/rdsk/c1t0d0s0   /   ufs   1   no   logging

You could consult the format command for partition / slice info for particular HDD.

5. Solaris Volume Manager is using three files for SVM configuration: /kernel/drv/md.conf, /etc/lvm/mddb.cf, and /etc/lvm/md.cf. Overwrite two last of them files from a system that does not use Solaris Volume Manager, and in the first one delete everything except for the first uncommented line, i.e. leave only this uncommented line:

name="md" parent="pseudo" nmd=128 md_nmsets=4;

6. Be sure, that you have both / and swap partitions enabled to load from HDD in vfstab, and restart server & boot from this HDD

init 0

{1} ok boot disk0

7. So, you`re booted into server, good! Now, we`ll need to recreate all metadevices, since I had RAID1 (mirror), I need to create metadb, mirrors, make root mirrored and bootable and put new devices in /etc/vfstab.

8. Creating metadb

metadb –a –c 3 -f c1t0d0s7 c1t1d0s7

9. Create metadevices - submirrors for root partition

metainit -f d10 1 1 c1t0d0s0 && metainit -f d20 1 1 c1t1d0s0

10. Create metadevice - main mirror for root partition (we`ll add only one submirror, and later after reboot we`ll attach another for resync)

metainit d0 -m d10

11. Make root mirror bootable (only needed for root mirror!)

metaroot d0

12. Repeat steps 9-10 for any mirror that needs to be created alongside with their submirror

13. Revert /etc/vfstab for it to be again using md pathes to metadevices, i.e. like this:

#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/md/dsk/d1  -       -       swap    -       no      -
#/dev/dsk/c1t0d0s1      /dev/rdsk/c1t0d0s1      -       swap    -       no      -
/dev/md/dsk/d0  /dev/md/rdsk/d0 /       ufs     1       no      logging
/dev/md/dsk/d3  /dev/md/rdsk/d3 /var    ufs     1       no      logging
#/dev/dsk/c1t0d0s3      /dev/rdsk/c1t0d0s3      /var    ufs     1       no      logging
/dev/md/dsk/d5  /dev/md/rdsk/d5 /apps   ufs     1       no      logging
#/dev/dsk/c1t0d0s5      /dev/rdsk/c1t0d0s5      /apps   ufs     1       no      logging
swap    -       /tmp    tmpfs   -       yes     -

14. Reboot (you could issue reconfiguration reboot, just to be sure, and you should be again in your system with svm mirrors attached.

15. Attach submirror to every mirror, to launch resync procedure:

metattatch d0 d20

16. Repeat step 15 for every mirror

17. Check the metastat output, you should see every mirror has submirrors, which are resyncing

18. Grab a beer - you deserve it!