here be monsters

my thread on the OpenWrt forums explains the problems of late. CF issues have cropped up again, but only after i ran nvram set lan_ifname=eth0. i can boot off the internal flash and e2fsck the card and it checks out clean. and i see that when it boots, my red led lights showing that the card reader is seen and even the activity light flashes that something is going on, but all that i get from tcpdump is:

# tcpdump -vv -i eth1
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
22:24:39.486434 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.0.4919: UDP, length 1001
22:24:39.504939 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.255.4919: UDP, length 1001
22:25:05.436622 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.0.4919: UDP, length 1001
22:25:05.455131 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.255.4919: UDP, length 1001

i.e. diddly for network. the first pair is when it boots off the internal flash, the second pair - i think it happens after it pivot_root's, but i'm not 100% sure yet.

got grub?

found this little gem in my inbox this morning:

This is an automatically generated mail message from mdadm
running on obiwan

A Fail event had been detected on md device /dev/md0.

Faithfully yours, etc.

oh yay! failed drives! luckily obiwan is still the "sandbox system" for now - it was supposed to be turned into my main externally-facing server once i was done with openwrt/dmz setup/etc. so much for good intentions - i'll never get this shit done.

so, i at least had the forethought to mirror the drives - it's dual 60GB ATA100 drives - good ol' hda and hdb. on each drive, i created two partitions - the first partition is /boot and the other is half of md0 - a raid1 device. i then built on md0 some logical volumes with LVM2, i usually name them /dev/linux/root, /dev/centos/usr, /dev/obiwan/home, or something like that. as far as the other partition, i thought i was doing the right thing by performing:

rsync -av --delete /boot /boot2

... to sync the kernel/initrd after a yum update included a kernel update, but that's only 1/2 of it. in today's failed case, it was hda that failed, which brings us to the crux of the problem - where's your bootloader now, eh? basically, nowhere. i'm screwed. so, i broke out the knoppix dvd and get to installing a bootloader on the second drive so i could bring the system up. how could i have prevented this from happening?

well, i think i have it worked out:

1. edit /boot/grub/device.map. make sure there's an entry for the second device there. in my case, it would be:

   (hd1) /dev/hdb

2. since grub-install likes to install in /boot of the grub root (very different from the system root - "/"), i gave it a little symlink hack:

   cd /boot; ln -s . boot

3. clean-up! get rid of all those old kernels that were installed with yum update:

   rpm -e kernel-old-version-blah

4. re-sync everything:

   rsync -av --delete /boot /boot2

5. now install grub on the second drive:

   grub-install --root-directory=/boot2 /dev/hdb

i think that should do it. i'm going to see if there's a way i can test this - maybe i'll pull some of the really 2GB drives out of the closet and get them in the test system to simulate failure.

Update: so much for that... i just got:

This is an automatically generated mail message from mdadm
running on obiwan

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

ding-dong, the system's dead. if i'm gonna be using knoppix so much, maybe i should re-download the latest dvd. sigh

interface layout & nvram cleanup

i'm trying to get the interface information for the WRTSL54GS straightened out so I can start setting up the DMZ. i found network config info in the wiki, including a diagram for my old WRT54Gv2.2, but not one for the new router. i'm in the middle of modifying the diagram to match the new router, but there's a lot of info - none of it too clear. i've posted on the openwrt forums asking for clarification. actually, i'm looking at making 2 diagrams - the "default" as shipped and my config - which will be w/o the bridge interface, with a dmz interface and a openvpn tunnel inteface setup.

in an effort to clarify things, i decided to tidy up my own setup by cleaning up the NVRAM variables (the safe way). so far, so good - after a reboot it's still there. :-)

root@OpenWrt:~# cd /tmp
root@OpenWrt:~# wget http://downloads.openwrt.org/people/kaloz/nvram-clean.sh
Connecting to downloads.openwrt.org[195.56.146.238]:80
nvram-clean.sh  100% |*************************************|  4702  00:00 ETA
root@OpenWrt:~# chmod a+x /tmp/nvram-clean.sh
root@OpenWrt:~# /tmp/nvram-clean.sh
Before: size: 11055 bytes (21713 left)
After: size: 3541 bytes (29227 left)
root@OpenWrt:~# nvram commit

rtg cgis

I posted my RTG CGIs to the rtg mailing list today. It's more of a work-thing as opposed to a home-project-thing, but since they're released under the GPL and it's a giving-back-to-the-community-thing, it figured it was worth mentioning. I'm still a little annoyed that the RTG database desperately needs normalization, but I understand the performance considerations and realize it's a design decision.

renewing certs

note to self: don't misplace the post-it with the passphrases for your CA. i ripped my whole apartment apart looking for it this morning. i need to update my openssl docs on how to renew a cert. back in 2001, i had no idea how to renew a cert. it's really as simple as just re-generating it with the same csr, and letting the serial number be incremented. however, without your CA passphrase, you'd be screwed. luckily, i found it and so i'm back in business. hopefully, the rest of the family using the site didn't notice.