After filing a bug report and working through it with the WordPress team, a solution was eventually found (see the final update to this post). If you're not inclined to skip to the end then here is a spoiler: turn off open_basedir or make the declaration less restrictive.

If you're not experiencing this problem (and Google says you're landing here if you're looking for help setting up SFTP/SSH updating for WordPress), I don't think this write-up will be of much help. But feel free to stick around; I've got snacks!

After upgrading to WordPress 2.8 I discovered that this update has broken automatic core and plugin updates for me. I use SSH2/SFTP as I don't trust, like, need, or support FTP, and the SSH log only shows that the PECL module is opening a connection and then closing it, with WordPress returning only the following to to the browser window.

Unable to locate WordPress Content directory (wp-content).

I dug through how WordPress handles upgrades, and tracked this all down to the /wp-admin/includes/class-wp-filesystem-ssh2.php file.

UPDATE 6/13/2009 @4:40PM EST

From what I found, the rewrite of the SSH/SFTP2 update function happened because the ssh2.sftp wrapper was introduced. This replaced the older version of the function which copied contents into temp files and pushed the data around. It was a drastic speed increase, but it doesn't bloody work on my system for one reason or another.

UPDATE 6/13/2009 @7:16PM EST

I've officially cried "uncle!" and turned to the most wretched hive of scum and villainy in blogging, the WordPress forums (as well as their bug reporting and ticketing system). I also identified the patch which introduced the code which caused SSH2 support to go flying off the rails for me.

UPDATE 6/18/2009 @4:17PM EST

The WordPress team has marked this bug as belonging to milestone 2.9. There is a slim chance that maybe I'll see some relief in 2.8.1, but more than likely they're going to yet again refactor how SSH2 works for automatic upgrades. If this changes, I will update this post accordingly. For now, I believe I may have to simply let it be broken.

UPDATE 12/19/2009 @10:30PM EST

Months later, I worked out what the problem was. Ultimately a conflict caused by use of PHP's open_basedir and safe_mode kept the SSH2 module from accessing resources it needed. The way the ssh2.sftp wrapper works would bump into overly restrictive basedir declarations. If you are experiencing this problem and you check your error logs after turning them up to "debug", you'll probably find out what directory or file is causing this conflict. It's ultimately moot, since things like safe_mode are deprecated in PHP 5.3 and being outright removed in PHP 6.

Oddly enough, updating libssh2 to something more up-to-date than the outdated RPM provided by Dag for RedHat Enterprise Linux/CentOS made a HUGE speed difference. Definitely something to look into if SSH updating is slow for you.

Other than that, this round of upgrades has been mostly painless, and the hyped speed increases are not just hyperbole. WordPress 2.8 feels only slight under-baked beyond SSH2 hiccups. I confess that I haven't had a chance to play with the redesigned sidebar widget administration yet though, as I haven't even figured out how I want to implement support for sidebar widgets into this theme.

The bug that deletes some custom fields if you update posts that have them is still there. No idea how to begin tracking that one down, but I can state authoritatively that it's theme and plugin independent. Why am I linking to some dudes blog, and not the official WordPress bug tracker? because I don't even know where WordPress keeps its bug tracker.

I also had to reset WordPress Blog stats because WordPress was tracking the wrong subdomain for all of my stats. The numbers tallied up with Google Analytics and the AWFFull log grinder, but the URLs were all effed up. I fixed the problem on the WordPress.com backend, and in the process lost months of aggregated history.

The upshot to this loss is that I hope to have a new round of search hit inspired commentary ready to go in a week or two.

I sat down, installed nginx (nginx 0.6.33) from the EPEL (Extra Packages for Enterprise Linux) repositories and lighttpd (lighttpd 1.4.20) from the RPMforge repositories (both third party repositories worth writing about in detail at a later date). Apache was already installed (httpd 2.2.3), so a virtualhost was configured to serve data from the same docroot using all three servers, and PHP through fastcgi was configured for lighttpd and nginx¹.

I then set up a stock WordPress 2.7 blog, some large images, some thumbnails, and created what amount to approximately a 20k index page with nothing cached. Using ApacheBench to fetch 1000 requests, 100 requests at a time, I came up with some simple numbers showing how long it took to return the same page for each web server.

While I openly admit that the benchmarks produced here are arbitrary and superficial, they do give some rough insight (10,000ft overview level insight) as to how much faster nginx and fastcgi are over apache 2.2 and mod_php. lighttpd also performed admirably, and with less hassle involved in set up than nginx, but I suspect that may just be previous familiarity with the configuration.

A table of numbers (soon to be a graph, when I feel like making it) and some thoughts as to what they mean and why we (the web hosting community) still use Apache religiously follows after the jump.

So what does all this mean? Well, the gist of it is that nginx is really, really god damned fast. Embarrassingly fast. Note that I didn't say that Apache is slow. It's just not nearly as fast as nginx. However, that being said, nginx is also not nearly as extensible as Apache is. The lack of support for things that are taken for granted nowadays (.htaccess files come to mind) can be a sticking point for use in a shared hosting environment. lighttpd doesn't support most of those "expected" bells and whistles either, and an argument could be made for such things being a waste of cycles… But the bottom line is that such bells and whistles are expected to work in a shared hosting environment.

As for why the web hosting community at large drops Apache into place whenever we need an http server, I think it boils down to comfort and laziness. It's bundled with most linux distros by default, we know it because it's been around forever, and its behavior is predictable and familiar. It's become the Microsoft Office of http serving, for better or for worse, and it's not going anywhere any time soon. Even the 800lb gorilla behemoth that is Microsoft IIS has failed to dethrone Apache for sheer ubiquity. I think the performance or security conscious are always going to migrate towards something with better speed and less attack vectors, but the typical admin probably just won't care.

And me? For now I'm going to continue using Apache, while I work out if using nginx for any of the sites I host for people would be a problem. If I can get the rewrite rules taken care of, and work out simple site provisioning, I might well make the switch. nginx seems like a no-brainer on single site virtual servers, where resources are scarce, and Apache isn't getting any more svelte these days.

--

• The control panel to write human-readable, reasonable configuration files: This is where Webmin falls down on the job. The config files are valid, but god are they awful to read.

• No mention or hint of reseller anything: I just want administrative accounts running the show and hosted accounts with domains linked to them. Maybe some way for the hosted to monkey with the non-essentials of their own accounts. So on the grounds of overkill, pretty much every commercial panel fails my simple needs.

• Email support for something that's not antiquated: some panels use qmail (or some version therein, which hasn't had a proper update in years), some use sendmail (which is older than dirt and half as secure), and some use whatever the flavor of the month is (read: whatever ships on the supported OS distro). Me? I'm a postfix guy. I'm not wedded to it per se, but it's updated, feature packed, works well and its configuration is only modestly arcane (qmail, by the way, fails due to having an arcane configuration file structure that doesn't plug into antivirus or spam very easily).

  Regarding IMAP and POP support, I find that these panels often use their own "questionable" clients, or rely on courier. There isn't really anything wrong with that, I just like dovecot more (to be fair, DirectAdmin supports dovecot last I checked).

  I don't want to even get started with authorized SMTP support from these packages. God, the horror.

• No byzantine dependencies: Don't tell me that I need some god awful fossilized encryption library or ldap to make this system work.

  "Listen here, Mr. Control Panel, I don't use LDAP, and unless that is what you're using as a backing store for your hosting information, neither should you."

To that end, I've begun playing with Ruby and Thor to whip up some simple add/delete scripts (with initial success!), and the biggest stumbling block I've encountered so far is just parsing out the accounts that exist.

The Apache configuration file format is… archaic in its own right, being neither XML nor attribute: value pairs. It's readable, but there is a reason that an entire perl package (HTTPD::Config, which may or may not exist anymore) was produced just to scrape httpd.conf files.

I fear that without stepping back a little and engineering a proper solution I'm just going to wind up reimplementing what I encountered when I did overnight support for Reality Check Networks (no link provided, because, well, you know. Those dudes…).