You could say I have a fairly healthy interest for the systems admin side of tech. I wouldn’t say I’m an expert, but I think I could get by as an entry to mid level admin. Though that certainly doesn’t stop me from doing some colossally stupid things with my personal server(s).
…
As of right now, I have two hosted VPSs, including the one that this blog runs on. This server is supposed to be my “production” server, so it’s as locked down as I can make it, with only the necessary services running. Beyond that, the next cardinal rule of managing a production environment is testing software updates somewhere else before you run them on the production machine(s). Even though I’m fully aware of this rule, and of the possible ramifications of not following it, I’m usually – read: always – too lazy to test updates on a mirrored environment first. Cue ominous music.
That feels about right
The Incident
Yesterday before I left for the gym, I decided to log into my server and run, just like I do every couple of weeks, probably the most dangerous command I have in my bash aliases.
alias update='sudo apt-get check && sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade && sudo apt-get autoclean'
A command that I’ve run hundreds of times without incident, until now. After running the command and clearing my terminal, I went through my typical cursory check of this site before heading out to the gym.
Pictured: Brief confusion, before panic sets in.
No. F$@KING. Way!
Debugging – Panic Style
I was so confused, I failed to realize I was getting the default apache page, when…you know…this is supposed to be an nginx only box… I checked the site config files and “restarted” nginx and php-fpm a couple of times before I realized apache was somehow installed and running. I shut down the apache process, thinking it was just conflicting with the nginx process, but what I didn’t realize was that nginx was somehow uninstalled from the system!
Crap! Crap! Crap! Crap! Crap!
More investigation revealed that not only had nginx been uninstalled, but so had php-fpm! No wonder restarting though the /etc/init.d/ executables did nothing. What exactly did I miss?!
That would explain it…
This is what happens when you do this crap in a hurry! But no big deal, the configs all seemed to be intact, just install everything again and we are back in business. Right?
~ $ install nginx-full Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: nginx-full : Depends: libgeoip1 (>= 1.4.8+dfsg) but 1.4.7~beta6+dfsg-1 is to be installed Depends: libpcre3 (>= 8.10) but 8.02-1.1 is to be installed Depends: libssl1.0.0 (>= 1.0.1) but it is not installable E: Broken packages
Wait…what? Why the hell are the nginx/php packages dependant on library versions that don’t exist or are uninstallable?!
Dreadful Realization
At that moment, I remembered something a friend told me a few days prior: Debian 7 was just released. Somehow I was trying to install packages in Debian 6, that had dependencies on libraries in Debian 7! But how?!
Now we add Dotdeb, a great resource for debian installs that provides bleeding edge packages for a stable and up to date LA(/E)MP stack. Update your /etc/apt/sources.list to include the repo:
deb http://packages.dotdeb.org stable all deb-src http://packages.dotdeb.org stable allThen add the GnuPG key:
wget http://www.dotdeb.org/dotdeb.gpg apt-key add dotdeb.gpg
Ohhhhhhh
Using Dotdeb is very simple:
1. Add the main repository to your sources.list
Depending on your distribution (Wheezy or Squeeze), add these two lines to your /etc/apt/sources.list file (you can also choose a mirror near you) :
deb http://packages.dotdeb.org wheezy all deb-src http://packages.dotdeb.org wheezy allor
deb http://packages.dotdeb.org squeeze all deb-src http://packages.dotdeb.org squeeze all2. Options
If you want to install :
- PHP 5.4 on Debian 6.0 “Squeeze”, add these two lines too :
deb http://packages.dotdeb.org squeeze-php54 all deb-src http://packages.dotdeb.org squeeze-php54 all
OHHHHHHHH
HOLY CRAP I NEVER SPECIFIED THE VERSION OF DEBIAN!
Cleaning Up My Mess
It’s not such a big deal to screw up, as long as you can fix it, right? Well that’s what I’m going with here.
I followed Dotdeb’s (updated) instructions to specify the version of Debian I was using, and tried to install stuff again.
The following packages have unmet dependencies:
nginx-full : Depends: nginx-common (= 1.4.1-1~dotdeb.0) but 1.4.1-1~dotdeb.1 is to be installed
What a nightmare! Thankfully one that isn’t ridiculous to work through. After running
sudo apt-get autoremove
I was finally able to successfully reinstall all of the packages I accidentally removed
Woo hoo!
Word of Warning
While I was writing this post, and further investigating what happened, I discovered that there is something…strange with the Dotdeb PHP 5.4 packages. I would not recommend using them with Debian 6 (squeeze) just yet, especially if you want to install PHP extensions like mcrypt or suhosin.
If you end up with issues installing PHP extensions you’ll want to find out what PHP components you have installed that are 5.4, and remove them with apt. In a similar setup as mine, you’ll want to remove php5-fpm and php5-common to clean up all of the problematic components.
After that, you should just be able to install PHP 5.3 along with whatever extensions you need!
