For the first time in history, I performed a remote kernel upgrade on the server that was previously cracked (goodbye local root exploit) and it actually worked when I rebooted the box in the middle of the night.
As a devoted Debian user, here's what I did...
Get New Kernel Source, Unpack It, Make Symlink
# apt-get install kernel-source-2.4.27 # cd /usr/src # bunzip -c kernel-source-2.4.27.tar.bz2 | tar -xvf - # rm linux # ln -s kernel-source-2.4.27 linux
Copy Old Kernel Config, Update It
# cd linux # cp ../kernel-source-2.4.23/.config . # make oldconfig (answer No to everything)
Compile Kernel, Build Debian Package, Install
This particular machine uses a non-initrd kernel, so I do this:
# make-kpkg clean # make-kpkg kernel_image
Otherwise I'd do this to get an initrd-based kernel package:
# make-kpkg clean # make-kpkg --initrd kernel-image
In either case, installation of the new kernel is trivial, thanks to Debian's kernel-package:
# cd .. # dpkg -i kernel-image-VERSION_i386.deb
That populates everything needed in the /boot directory and moves the necessary symlink to make the current kernel image the backup (old) kernel image. It then will offer to make a boot floppy and run lilo.
Cross Fingers, Reboot
After I double checked that everything looked right in the / and /boot directories and that /etc/lilo.conf was reasonable, I did asked for a reboot and crossed my fingers.
# reboot
And left a command running in another window:
$ ping family2.zawodny.com
When it started returning pings again, I knew things were in decent shape. Twenty seconds later I was able to ssh in and verify that things were all happy:
jzawodn@family2:~$ uname -a Linux family2 2.4.27 #1 SMP Sat Feb 12 01:03:32 PST 2005 i686 GNU/Linux
Yeay!
Now I can start moving everything back off the 3.5 year old backup machine onto this much faster box.
Posted by jzawodn at February 12, 2005 12:32 PM
if you use GRUB, it has some functionality that can make it an even safer procedure. you can specify a fallback kernel; if the first one fails for some reason, it will try the second one. the technique is outlined here: http://gentoo-wiki.com/HOWTO_Remote_Kernel_Upgrade (it's in the gentoo wiki but isn't very gentoo specific).
it's not foolproof, but it makes things a *little* less nerve-racking.
You're gutsy, Jeremy. I wouldn't even *think* of running a remote kernel upgrade unless I was at the console itself.... not on an enterprise box anyway.
I've been there, and it always drives me crazy. That is, of course, until I did something even crazier.
I have a colocated box at a big data center in Texas. I'm in Ohio. The box came with Redhat 7.1, and was showing it's age. Even if I were in texas, I would have zero physical access to the box. The colo service would, for $49, remove my hard drive and replace it with one with a Redhat 9 image, and I could go about rebuilding all my data myself.
So, I actually changed distributions on a remote box, with no access. I created a filesystem in the 2gig swap partition, laid down a stage3 gentoo install, configured it to boot. I then proceded to check and recheck my config a half million times.
When I finally typed 'shutdown -r now' on the old distribution, it was the most harrowing 90 seconds of my life as I waited for it to ping. It did.
And now I've got gentoo, which is upgraded incrementally and should, hopefully, never require that again.
lilo -R linuxtest panic=20
can be used to one-off test a new kernel build. Which is nice.
I've never used linux much, but I'm about to try a gentoo conversion on a kuro, so I have to ask...what's the big deal with remote upgrades? I do it all the time on my BSD boxes. I won't have much console access to the kuro once it's set up so is there something I should know? Is this a bad idea?
Brandon,
Dependencies, dependencies, dependencies. If one thing isn't right you run the risk of not being able to boot into the box... And unless you're at the console, that means trouble. You can't throw an emergency CD or boot into single user mode sitting in your home 700 miles away from the DC...
i know exactly where Jeremy is coming from - trying out something like a remote kernel upgrade is always fraught with trepidation, but then when it JUST WORKS , it worthy of a blogpost.
Just goes to show that no matter how much you think you know about Linux, there's always something you havent tried out yet
Glad to see you use Debian Jeremy btw
This may sound a bit stupid but... why upgrading to 2.4.27 instead of 2.6.x?
OK, I know Linus is introducing some instabilities in the 2.6.x branch, but you can always use a somewhat "old" 2.6 kernel, like say, 2.6.7. That's what I do and never failed.
No need to go to 2.6. 2.4 has been serving me quite well for a long time now. Next time I do a major overhaul, I'll move to 2.6.
Well, 2.4.27 is the latest in the debian sources. Besides, all the critical patches have been incorporated anyway.
