Shrinking Root with Software Array, then Replacing Root with another Root Image

0. Start console.  This is a hardware thing, whereby you can monitor the system console even during a reboot.  For example:

ssh console@slcradar03-sp.con.slc.ebay.com

start /SP/console

escape + ( gets out of the console.  then, you can do a “reset/SYS” to reboot

1. Take down the raid superblock:
   1. turn off swap
      1. swapoff -a
   2. Check swaps to confirm
      1. cat /proc/swaps
   3. Check state of software raid
      1. cat /proc/mdstat
   4. Stop the md1
      1. mdadm –stop /dev/md1 (will be restarted if you reboot)
   5. Check/confirm previous command
      1. cat /proc/mdstat
   6. Examin /dev/sdb5
      1. mdadm -E /dev/sdb5
   7. makes it so that the device doesn’t return on reboot.
      1. mdadm –zero-superblock /dev/sdb5
      2. mdadm –zero-superblock /dev/sda5
   8. Nuke sda2 and sda5
      1. fdisk /dev/sda
      2. p (to print)
      3. d 5 (delete sda5)
      4. d 2 (delete sda2)
      5. w (confirm)
      6. q (quit)
      7. partprobe (from commandline.  Deals with kernel still having locks on the drive.  Saves you from having to reboot. Partprobe will warn you if you still need to reboot)
      8. Repeat for /dev/sdb
   9. update /etc/fstab
      1. remove the swap line
      2. comment out the line with UUID on the root partition
      3. replace line with /dev/md0 ext4 errors=remount-ro 0 1 (tell it to boot with /dev/md0 instead of uuid)
   10. Make grub show us console messages on boot
       1. vi /etc/default/grub
       2. find line GRUB_CMDLINE_LINUX, add “console=ttyS0,9600n8” <–S0 is serial port 0.  9600 is speed
       3. update-grub
       4. verify in /boot/grub/grub.cfg
       5. remove everything between /30_cs-prober begin and end
       6. reboot
   11. In grub, do e.  Nuke the word quiet if you want to get detailed information on boot.  Control X to boot.
   12. Remove sdb1 from software raid
       1. mdmadm —remove /dev/md0 /dev/sdb1  (this will fail, becaues it’s healthy, so…)
       2. mdmadm –fail /dev/md0 /dev/sdb1
       3. try the remove command again
       4. If you cat /proc/mdstat, you will no longer see /dev/sdb1
       5. mdadm –zero-superblock /dev/sdb1 (so that sdb1 no longer thinks it belongs to md0)
       6. verify with: mdadm -E /dev/sdb1
   13. mount sdb1, and backup its content
       1. mkdir /mnt/tmp
       2. mount /dev/sdb1 /mnt/tmp
       3. cd /mnt/tmp
       4. tar cvzf /mnt/root.tgz .
       5. umount /mnt/tmp
   14. kill the drive
       1. fdisk /dev/sdb
       2. d
       3. n (new)
       4. p
       5. 1
       6. 2048
       7. +5G
       8. fd
       9. p
       10. Create a second partition with the remaining space
       11. n
       12. p
       13. get the first sector from the last sector printed out above
       14. leave default on end sector
       15. 8e (LVM type)
       16. w (commit)
   15. mkfs.ext4 /dev/sdb1
   16. mount it again, and restore data
       1. mount /dev/sdb1 /mnt/tmp
       2. cd /mnt/tmp
       3. tar xvzf /mnt/root.tgz
   17. Tell machine to boot with sdb1
       1. cd /mnt/tmp
       2. vi /etc/fstab
       3. change /dev/md0 to /dev/sdb1
       4. reboot (make sure console is open)
   18. At grub, use e.  Remove the search line.  find the line with “linux /boot”, remove UUID and replace with root=/dev/sdb1
   19. mount /dev/md0 /mnt/temp
   20. copy root.tgz
       1. cp /mnt/temp/root.tgz /mnt
   21. unmount temp
   22. Delete whole concept of md0
       1. mdadm –stop /dev/md0
       2. to verify, cat /proc/mdstat
       3. mdadm –zero-superblock /dev/sda1
       4. If you now scan superblock with mdadm -Es, you shouldn’t see anything
       5. fdisk
       6. u
   23. We want ot make sda look exactly like sub.
       1. dd if=/dev/sdb of=/dev/sda bs=512 count=1 (BE CAREFUL!)
       2. partprobe
       3. verify with fdisk that /dev/sda and /dev/sdb are the same
   24. Create a raid array
       1. mdadm –create /dev/md0 –level 1 -n 2 /dev/sda1 missing
       2. verify using cat /proc/mdstat
       3. mkfs.ext4 /dev/md0
       4. mount /dev/md0 temp
       5. cd /mnt/temp
       6. tar xvzf ../root.tgz
       7. mdadm -Es > etc/mdadm/mdadm.conf (the mdadm.conf inside md0)
       8. chroot /mnt/temp
       9. update-initramfs -u
       10. cd /boot/grub
       11. vi grub.cfg
       12. nuke the search line
       13. find the line with the UUID, replace with root=/dev/md0
           sync
       14. umount /mnt/temp
       15. reboot
   25. Add sdb1 to md0 and wait for stuff to sync
       1. mdadm –manage /dev/md0 –add /dev/sdb1 (now, wait for sync…..)
       2. mdadm –create /dev/md1 –level 1 -n 2 /dev/sda2 /dev/sdb2 (wait for stuff to sync….)
   26. Create LVM
       1. Install lvm2 and push dependencies and install the debs according to the instructions at Virtual Machines Administration#AddingAdditionalPackages
       2. Or you can just log into radaradm@tnsar-imagehost, and copy the 3 deb files that will enable lvm on ops image in ops-image-lvm2-debs
       3. pvcreate /dev/md1 9create physical volume)
       4. vgcreate volmd1 /dev/md1 (create volume group)
       5. lvcreate -n swap -L 32G volmd1 (create swap logical volume)
       6. lvcreate -n ebay -L 200G volmd1 (create logical volume ebay)
       7. To verify, run pvscan, vgscan, lvscan
       8. mkswap /dev/mapper/volmd1-swap
       9. mkfs.ext4 L “/ebay” /dev/mapper/volmd1ebay
       10. mdadm -Es > /etc/mdadm/mdadm.conf  (so reboot will sitll see md1)
       11. edit /etc/fstab to auto-mount if required
   27. Add radaradm key to new machine’s /root/.ssh/auth_keys (on /dev/md0 aka opsroot)
   28. Deploy our own vdi file
       1. deploy_host host.vdi <machine name>
   29. update-grub (if you don’t see your os, check /etc/grub.d/ to see if 30_os-prober is executable
       1. verify.  Look at /boot/grub/grub.cfg
       2. edit /boot/grub/grub.cfg, set default=”4″ (or whatever number you want the default to be