It happens to all of us, today it happened to me. Conveniently on a production system. Instead of rm -Rf ./*
I typed rm -Rf /*
, and lots of stuff was gone.
Note that this will only help you if you’ve noticed that something is wrong before the command wiped your entire system and canceled it. I’m not sure where exactly the point of no return is, but I expect it to be at /lib
.
Also, for this to work, you’ll need a backup of your system, preferably in a place you can make accessible from the web.
In my case, I had deleted everything up to and including /etc/
, which of course caused my SSHd do not accept connections anymore, so do not close your SSH session. What I did to restore the system was based on the fact that I still hat /usr/bin
, where lots of nice tools reside.
Namely, I used the following tools:
-
which
to check if you’re screwed or not -
echo
which is part of bash -
/usr/bin/base64
-
/usr/bin/wget
-
/usr/bin/perl
-
tar
which I had to restore too
Step 1 - Check if you’re screwed
Try executing echo
, base64
, and wget
. If any of those commands doesn’t work, you’ll have to find another way to save your system.
Step 2 - Recover tar
First of all, we need to create the necessary directories, but we don’t have mkdir
, so we’ll need to use perl
(or something similar):
$ echo "mkdir '/bin', 0777;" | perl
This will create your /bin
directory. The unrestrictive permissions are probably unnecessary, but you can change them later when you have full control again.
Now you need to get to your backup system and find the tar
executable, and execute (assuming that you are inside the directory where the executable is located):
$ cat ./tar | base64
yv66vgAAAAIBAAAHgAAAAwAAEAAAAE5QAAAADAAAAAcAAAADAABgAAAATfAAAAAMAAAAAA...
Copy the output from that command to the shell of your broken system, like this:
$ echo -n "<output from previous command>" | base64 -d > /bin/tar
This will leave you with your fancy tar
file, but it is not executable yet, so we do:
$ echo "chmod 777, '/bin/tar';" | perl
Now you should be eable to execute tar
from within your broken system’s shell:
$ tar
tar: Must specify one of -c, -r, -t, -u, -x
Step 3 - Recover /bin and /etc
From here on its a piece of cake, mostly. You’ll need to pack your /bin
folder on your backup system, like this (assuming you are in /path/to/backup/bin
):
$ tar -cf bin.tar ./*
Now you’ll have to move the archive to some place accessible from the broken system, so you can execute the following commands on it:
$ cd /bin
$ wget http://somesite.org/foo/bin.tar
$ tar -xf ./bin.tar .
This should leave you with a fully restored /bin
folder, which is something you can build upon. Now you can restore your /etc
folder using the exact same steps (remember to delete your web-accessible archives once you’ve downloaded them, because they probably contain sensitive information).
Step 4 - Restoring /dev
I haven’t found a way yet to fully restore /dev “automagically”. However, I was able to restore some of the really essential items, so that at least SSH based applications such as rsync and SCP were working properly again:
$ mknod -m 0644 /dev/random c 1 8
$ mknod -m 0644 /dev/urandom c 1 9
$ mknod -m 666 /dev/null c 1 3
$ mknod /dev/ptmx c 5 2
These were all necessary for me so that I could directly transfer the remaining backups via rsync
and especially to create new SSH connections, which wasn’t possible until I restored /dev/ptmx
.