T-P-L.com

My server got hacked, so this probably won’t get posted for a while. I’m typing this as I rebuild the box.

My server got hacked, so this probably won’t get posted for a while. I’m typing this as I rebuild the box.

Tue Nov 22

There were a few weird things going on with the box for a while, I found it odd but didn’t give it much thought. The main thing that was weird was the fact that I couldn’t send a SIGINT or SIGTERM (ie. ctrl-c and soft kill) to any process, well I could, but it would be ignored. By just popping in a fresh Gentoo cd and pressing ctrl-c I was able to see that it isnt anything hardware related. Now this could have been an obscure thing I turned on in the kernel, I’ve no idea.

Before I could even get to backing it all up, of course the box had to play hard to boot. The bios doesn’t have an option to boot from a usb cdrom .. so I had to search around the house for a cdrom I can put in the box, search for some udma cables (I hate the 80 strand ribbon cable, I wish cdroms were made with SATA)

Anyways I’m in the middle of backing up the whole setup before I begin, for both data recovery and forensic reasons. Just for reference, the copy command I used to do this backup was:

livecd /mnt/dump # cp -avfg --reply=yes /mnt/gentoo/. .

The copy was started at ~00:30 utc dec 18 (the server’s time is way off, but I’ll update with the final time after the copy to see how long it took)

the copy .. ended at ~02:45 utc .. so that was ~2h 15 min .. to copy .. I realized just the / and /boot partitions (~15G) ..

I resumed the copy of the /mnt/raid partition at around 03:10 utc .. and this one had ~45G of data on it .. so it won’t be done anytime soon

To initiate the copy I used the below, basically I added -u which skips over files that are already existing.

livecd /mnt/dump # cp -avfg --reply=yes /mnt/gentoo/. .

15g / 2.25h = ~6.6g/h
45G / 6.6g/h = ~6h

so I’m looking at this finishing at around 5am .. I guess I’ll be able to check in on it in the morning .. but I have to head to work .. so I won’t get anything done until I come back

Morning Wed Nov 23

I checked on the server, it actually took just over an hour to copy the other drive, I guess it goes to show that tiny files are evil =)

anyways .. off to work …

later that evening …

so I got back, the copy was over, so all I had to do was start a fresh install, but first I had to do some cleanup. I broke out the zap straps and strapped just about every loose cable in the box, replaced a horrible round ide cable with a simple flat one, oddly enough those flat ones mold way better. I tossed out the original 30G drive from the box. I even tossed the power adaptors for sata, since the drive trays I have my 2 sata drives have standard power connectors. I even replaced the two extra long sata cables that came with the raid card with 2 shorter (just about the right size really) sata cables. All this in an effort to clean up the amount of cables in the case. In the end I was pretty happy, its much more roomy in the case now.

Next came whipping the raid into proper shape, by robin’s advice I went with software raid over single partitions. Here is the partition scheme I used:

120G space

128M /boot
2G   swap
5G   /
40G  /usr
2G   /tmp
20G  /var
~50  /home

Then everything other than swap is mirrored across the drives. I decided to go with RaiserFS for the partition type for most the drives (/boot is ext3)

Well I decided to make /boot ext3 after I already formatted it raiser and started extracting the gentoo stage3 on it. Lets’s hope it doesn’t put much on /boot so I can format it ext3 instead.

.. since the initial portage sync took super long .. (ie .. its 11:30 .. and its not done yet) .. I’m not going to able to get this box to boot on its own (or past this step really) .,. so the saga moves on to another day

Evening Thu Nov 24

After chatting it up with a helpfull guy on #gentoo on freenode I became convinced that the way to go for me was that of linux software raid and LVM. The problem was that at the end of the day I had a setup working (ie booting) without LVM but I wanted LVM .. so the solution was to tar is all up to another removable disk and untar it to a new storage config. The biggest thing I learned was that for linux software raid you need to set the partitions in question to a the linux raid autodetect partition types then everything starts working magically.

Evening Fri Nov 25 - Sun Nov 27

This was the day of the biggest breakthroughs. I configured software raid again with LVM this time. Moved all the data (ie .. essentially a stage4 tarball) to the new storage solution and got the box to boot with that. All was good .. until I started to want to copy the old data over.

I came to realize that I backed up the old data to a bad disk. It was essentially all gone. I did originally merge the data from both a raid mirror and a single disk, so I did still have the disk un-wiped. So I was able to recover some stuff, but not nearly all. I lost everything in /home. So over the weekend I got it all copied what was left and started slowly brining up services (well not really not yet) the data loss got me kinda down so I slowed down .. and pretty much left the box alone in a bootable state.

Mon Nov 28

I got qmail back up, I flushed the mail queue on Robin’s box and instantly got 500 new emails to my server .. yay? .. well only way to access the mail for now was mutt. But the box was receiving mail again.

Tue Nov 29

I got courier-imapd up and running, so I could actually check my mail remotely.

Wed Nov 30

The project for the day was to get web back up. I decided to play with lighttpd. First thing to come back up was svn (I needed to know what I have in there). Found an old copy of my website there, not too bad, I can use that to eventually recover. I started doing that .. realizing it was old, but having a new sql dump .. I dropped just a clean new install of wordpress and dropped the database in. It worked more or less, no magic url’s because of lighttpd (no .htaccess support), but its not too bad. I have a blog.