How To Destroy Your Web Server and Ruin Your Night

You could say I have a fairly healthy interest for the systems admin side of tech. I wouldn’t say I’m an expert, but I think I could get by as an entry to mid level admin. Though that certainly doesn’t stop me from doing some colossally stupid things with my personal server(s).

...

As of right now, I have two hosted VPSs, including the one that this blog runs on. This server is supposed to be my “production” server, so it’s as locked down as I can make it, with only the necessary services running. Beyond that, the next cardinal rule of managing a production environment is testing software updates somewhere else before you run them on the production machine(s). Even though I’m fully aware of this rule, and of the possible ramifications of not following it, I’m usually – read: always – too lazy to test updates on a mirrored environment first. Cue ominous music.

That feels about right

That feels about right

The Incident

Yesterday before I left for the gym, I decided to log into my server and run, just like I do every couple of weeks, probably the most dangerous command I have in my bash aliases.

alias update='sudo apt-get check && sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade && sudo apt-get autoclean'

A command that I’ve run hundreds of times without incident, until now. After running the command and clearing my terminal, I went through my typical cursory check of this site before heading out to the gym.

Pictured: Brief confusion, before panic sets in.

Pictured: Brief confusion, before panic sets in.

No. F$@KING. Way!

Debugging – Panic Style

I was so confused, I failed to realize I was getting the default apache page, when…you know…this is supposed to be an nginx only box… I checked the site config files and “restarted” nginx and php-fpm a couple of times before I realized apache was somehow installed and running. I shut down the apache process, thinking it was just conflicting with the nginx process, but what I didn’t realize was that nginx was somehow uninstalled from the system!

Crap! Crap! Crap! Crap! Crap!

More investigation revealed that not only had nginx been uninstalled, but so had php-fpm! No wonder restarting though the /etc/init.d/ executables did nothing. What exactly did I miss?!

mistake

That would explain it…

This is what happens when you do this crap in a hurry! But no big deal, the configs all seemed to be intact, just install everything again and we are back in business. Right?

~ $ install nginx-full
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
nginx-full : Depends: libgeoip1 (>= 1.4.8+dfsg) but 1.4.7~beta6+dfsg-1 is to be installed
Depends: libpcre3 (>= 8.10) but 8.02-1.1 is to be installed
Depends: libssl1.0.0 (>= 1.0.1) but it is not installable
E: Broken packages

Wait…what? Why the hell are the nginx/php packages dependant on library versions that don’t exist or are uninstallable?!

Dreadful Realization

At that moment, I remembered something a friend told me a few days prior: Debian 7 was just released. Somehow I was trying to install packages in Debian 6, that had dependencies on libraries in Debian 7! But how?!

Now we add Dotdeb, a great resource for debian installs that provides bleeding edge packages for a stable and up to date LA(/E)MP stack. Update your /etc/apt/sources.list to include the repo:

deb http://packages.dotdeb.org stable all
deb-src http://packages.dotdeb.org stable all

Then add the GnuPG key:

wget http://www.dotdeb.org/dotdeb.gpg
apt-key add dotdeb.gpg

- Adventures With Nginx and PHP

Ohhhhhhh

Using Dotdeb is very simple:

1. Add the main repository to your sources.list

Depending on your distribution (Wheezy or Squeeze), add these two lines to your /etc/apt/sources.list file (you can also choose a mirror near you) :

deb http://packages.dotdeb.org wheezy all
deb-src http://packages.dotdeb.org wheezy all

or

deb http://packages.dotdeb.org squeeze all
deb-src http://packages.dotdeb.org squeeze all

2. Options

If you want to install :

  • PHP 5.4 on Debian 6.0 “Squeeze”, add these two lines too :
    deb http://packages.dotdeb.org squeeze-php54 all
    deb-src http://packages.dotdeb.org squeeze-php54 all

- Dotdeb Instructions

OHHHHHHHH

HOLY CRAP I NEVER SPECIFIED THE VERSION OF DEBIAN!

 Cleaning Up My Mess

It’s not such a big deal to screw up, as long as you can fix it, right? Well that’s what I’m going with here.

I followed Dotdeb’s (updated) instructions to specify the version of Debian I was using, and tried to install stuff again.

The following packages have unmet dependencies:
nginx-full : Depends: nginx-common (= 1.4.1-1~dotdeb.0) but 1.4.1-1~dotdeb.1 is to be installed

What a nightmare! Thankfully one that isn’t ridiculous to work through. After running

sudo apt-get autoremove

I was finally able to successfully reinstall all of the packages I accidentally removed

Woo hoo!

Woo hoo!

Word of Warning

While I was writing this post, and further investigating what happened, I discovered that there is something…strange with the Dotdeb PHP 5.4 packages. I would not recommend using them with Debian 6 (squeeze) just yet, especially if you want to install PHP extensions like mcrypt or suhosin.

If you end up with issues installing PHP extensions you’ll want to find out what PHP components you have installed that are 5.4, and remove them with apt. In a similar setup as mine, you’ll want to remove php5-fpm and php5-common to clean up all of the problematic components.

dpkg

After that, you should just be able to install PHP 5.3 along with whatever extensions you need!

Performance Optimization WordPress Plugins by W3 EDGE