For the first time in history, I performed a remote kernel upgrade on the server that was previously cracked (goodbye local root exploit) and it actually worked when I rebooted the box in the middle of the night.

As a devoted Debian user, here's what I did...

Get New Kernel Source, Unpack It, Make Symlink

# apt-get install kernel-source-2.4.27
# cd /usr/src
# bunzip -c kernel-source-2.4.27.tar.bz2 | tar -xvf -
# rm linux
# ln -s kernel-source-2.4.27 linux

Copy Old Kernel Config, Update It

# cd linux
# cp ../kernel-source-2.4.23/.config .
# make oldconfig
(answer No to everything)

Compile Kernel, Build Debian Package, Install

This particular machine uses a non-initrd kernel, so I do this:

# make-kpkg clean
# make-kpkg kernel_image

Otherwise I'd do this to get an initrd-based kernel package:

# make-kpkg clean
# make-kpkg --initrd kernel-image

In either case, installation of the new kernel is trivial, thanks to Debian's kernel-package:

# cd ..
# dpkg -i kernel-image-VERSION_i386.deb

That populates everything needed in the /boot directory and moves the necessary symlink to make the current kernel image the backup (old) kernel image. It then will offer to make a boot floppy and run lilo.

Cross Fingers, Reboot

After I double checked that everything looked right in the / and /boot directories and that /etc/lilo.conf was reasonable, I did asked for a reboot and crossed my fingers.

# reboot

And left a command running in another window:

$ ping family2.zawodny.com

When it started returning pings again, I knew things were in decent shape. Twenty seconds later I was able to ssh in and verify that things were all happy:

jzawodn@family2:~$ uname -a
Linux family2 2.4.27 #1 SMP Sat Feb 12 01:03:32 PST 2005 i686 GNU/Linux

Yeay!

Now I can start moving everything back off the 3.5 year old backup machine onto this much faster box.

Posted by jzawodn at February 12, 2005 12:32 PM

Reader Comments
# anders said:

if you use GRUB, it has some functionality that can make it an even safer procedure. you can specify a fallback kernel; if the first one fails for some reason, it will try the second one. the technique is outlined here: http://gentoo-wiki.com/HOWTO_Remote_Kernel_Upgrade (it's in the gentoo wiki but isn't very gentoo specific).

it's not foolproof, but it makes things a *little* less nerve-racking.

on February 12, 2005 01:11 PM
# Aaron said:

You're gutsy, Jeremy. I wouldn't even *think* of running a remote kernel upgrade unless I was at the console itself.... not on an enterprise box anyway.

on February 12, 2005 01:16 PM
# Chris Thompson said:

I've been there, and it always drives me crazy. That is, of course, until I did something even crazier.

I have a colocated box at a big data center in Texas. I'm in Ohio. The box came with Redhat 7.1, and was showing it's age. Even if I were in texas, I would have zero physical access to the box. The colo service would, for $49, remove my hard drive and replace it with one with a Redhat 9 image, and I could go about rebuilding all my data myself.

So, I actually changed distributions on a remote box, with no access. I created a filesystem in the 2gig swap partition, laid down a stage3 gentoo install, configured it to boot. I then proceded to check and recheck my config a half million times.

When I finally typed 'shutdown -r now' on the old distribution, it was the most harrowing 90 seconds of my life as I waited for it to ping. It did.

And now I've got gentoo, which is upgraded incrementally and should, hopefully, never require that again.

on February 12, 2005 01:22 PM
# Tom Allender said:

lilo -R linuxtest panic=20

can be used to one-off test a new kernel build. Which is nice.

on February 12, 2005 02:22 PM
# Brandon said:

I've never used linux much, but I'm about to try a gentoo conversion on a kuro, so I have to ask...what's the big deal with remote upgrades? I do it all the time on my BSD boxes. I won't have much console access to the kuro once it's set up so is there something I should know? Is this a bad idea?

on February 12, 2005 03:33 PM
# Aaron said:

Brandon,

Dependencies, dependencies, dependencies. If one thing isn't right you run the risk of not being able to boot into the box... And unless you're at the console, that means trouble. You can't throw an emergency CD or boot into single user mode sitting in your home 700 miles away from the DC...

on February 12, 2005 03:41 PM
# Justin said:

i know exactly where Jeremy is coming from - trying out something like a remote kernel upgrade is always fraught with trepidation, but then when it JUST WORKS , it worthy of a blogpost.

Just goes to show that no matter how much you think you know about Linux, there's always something you havent tried out yet

Glad to see you use Debian Jeremy btw


on February 12, 2005 05:07 PM
# d said:

Welcome to the family Jeremy. :)

on February 12, 2005 11:03 PM
# Pau Garcia i Quiles said:

This may sound a bit stupid but... why upgrading to 2.4.27 instead of 2.6.x?

OK, I know Linus is introducing some instabilities in the 2.6.x branch, but you can always use a somewhat "old" 2.6 kernel, like say, 2.6.7. That's what I do and never failed.

on February 13, 2005 08:00 AM
# Jeremy Zawodny said:

No need to go to 2.6. 2.4 has been serving me quite well for a long time now. Next time I do a major overhaul, I'll move to 2.6.

on February 13, 2005 09:02 AM
# Chris said:

Why upgrade to 2.4.27 when 2.4.29 is the latest?

on February 14, 2005 09:34 AM
# Jeremy Zawodny said:

Well, 2.4.27 is the latest in the debian sources. Besides, all the critical patches have been incorporated anyway.

on February 14, 2005 11:09 AM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.

 

Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.