Linux

Catching Apache segfaults due to eAccelerator

Last night we migrated our Linode for Gamers With Jobs to a new Xen VPS and we've noticed a significant performance boost. We did, however, start encountering a random issue with segmentation faults in Apache. If you haven't seen this happen before, it tends to begin innocently with one Apache process dying, and therefore giving errors (usually WSOD), but quickly balloons into dozens of dead processes. It essentially hoses Apache.

Apparently the issue is due to eAccelerator, so I reinstalled it and cleared its caches in the hope that it might limit its occurrence. Just in case, though, 2bits has a great fix for it, using the logwatcher script by Firebright, Inc. I was able to quickly get it going, and the only difference is that I used the Debian init.d script provided by Derek Laventure to run it.

Just got rolling with a VPS on Linode (Part 2)

Now that I had a demonstrably working and functional web server going on my Linode (see Just got rolling with a VPS on Linode (Part 1)), it was time to get the rest of my toolkit on the box, setup users and secure the server a bit.

Installing Subversion and migrating repositories

Well, installing Subversion couldn't be any simpler:

apt-get install subversion

Login to old server and dump current repositories:

svnadmin dump /path/to/repository > repository.dump

Copy dump file to new server, and on new server:

svnadmin create /path/to/repository
svnadmin load /path/to/repository < /path/to/dump/repository.dump

Adding users and groups

I decided I didn't want to be logged in as root all the time, especially since I'll most likely be bringing some other folks in to work on the server in the future. So, I setup the admin group, created myself a new user and put myself in both the admin and staff groups.

addgroup admin
adduser jrbeeman
usermod -G staff,admin jrbeeman

Next, I wanted to make sure admins could sudo to root, so that they could install programs and do other root-y things. The sudoers file, as far as I can tell, can only be edited with the command visudo:

visudo

...and added the line:

%admin  ALL=(ALL) ALL

Setting up the firewall

This was probably the least-traveled territory in the whole VPS setup for me. Thankfully, there is an awesome resource in the website IP Tables Rocks, with a full rundown of how to lock down unneeded ports. It emphasizes locking down everything, and then only opening up those services you want open. I essentially followed the tutorial, but proceeded to lock down every port except those that I knew I would need for web services and working with the server (22, 80, 443, etc.)

Performance

By this point, I've started working on getting the Gamers With Jobs development site migrated over, and I'm working on nailing down any performance issues. As I said in part 1, the main reason for going to a VPS was the sheer size and load on the GWJ site and how shared hosting was really hosing the speed. Most of the tweaks from here on out are related to the GWJ site.

Tweak MySQL settings

Since the Gamers With Jobs site is very database intensive, getting MySQL to perform optimally given the site's load is important. I'm still tweaking these settings here and there, but here's what I'm at so far. I'm attempting to go for large enough buffers and caches to keep things snappy, but without bloating out the caches to the point that things slow down.

#
# * Fine Tuning
#
key_buffer              = 256M
max_allowed_packet      = 16M
thread_stack            = 128K
thread_cache_size       = 8
#max_connections        = 100
table_cache             = 256
thread_concurrency      = 4
sort_buffer_size        = 1M
read_buffer_size        = 1M
read_rnd_buffer_size    = 4M
myisam_sort_buffer_size = 64M
#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M
#
# Turn on slow query logging to help track down performance killers
#
log_slow_queries        = /var/log/mysql/mysql-slow.log
long_query_time = 5
#
# Some further table-type tweaks
#
[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

Bringing over the GWJ site required quite a bit of scripting of INSERT and DELETE statements that fudged with table lengths, so I also optimized all the tables with free data space:

-- Get the table names...
SHOW TABLE STATUS WHERE Data_free > 0;
-- ...and run the following for each
OPTIMIZE TABLE table_name;

Tweak Apache settings

The YSlow utility from Yahoo is a great way to track down potential end-user performance issues, so I ran it against the GWJ dev site and tweaked quite a few things to improve the rating and speed reported there.

First, I needed to enable a few Apache modules:

a2enmod deflate
a2enmod expires
a2enmod cache

Then, in /etc/apache2/httpd.conf, I added the following lines to the stanza of the GWJ virtual host definition:

# Gzip html, css, js, etc.
AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/json
# Set expires headers on html, css, js, etc.
<IfModule mod_expires.c>
  ExpiresActive On
  ExpiresByType text/html "access plus 1 seconds"
  ExpiresByType image/gif "access plus 1 month"
  ExpiresByType image/jpeg "access plus 1 month"
  ExpiresByType image/png "access plus 1 month"
  ExpiresByType text/css "access plus 1 week"
  ExpiresByType text/javascript "access plus 1 month"
  ExpiresByType application/x-javascript "access plus 1 month"
</IfModule>
# Set ETags
FileETag MTime Size

Install memcached

In order to squeeze a bit more performance out of the server, I decided to install memcached and the related Drupal module, which allows you to configure Drupal to store certain cache data in memory. I essentially followed the instructions in Robert Douglass's article on Lullabot, but with a couple of modifications.

First, libevent1-1.3b and memcached-1.2.1-1 can be installed via apt-get on Ubuntu gutsy, all with:

apt-get install memcached

Then, I enabled the Apache module:

a2enmod mem_cache

Install eaccelerator

Not much to write here, aside from noting that I followed the great article on 2Bits to get going.

Done... sorta

Seeing how I started writing this article a couple of weeks ago and am just getting around to publishing it, I think I'll call it "finished," for now. I hope that someone out there finds this useful!

Just got rolling with a VPS on Linode (Part 1)

Note: A large part of this is taken from Victor Kane's article on Awebfactory about setting up Drupal on a fresh Linode, but I've documented some other things here and did some things a little differently than he did, so I figured it'd be worth writing up a post on the process. I've kept the details thin here in places where Victor's notes are more than satisfactory, but I've made sure to note where that happens.

Update: Be sure to check out part 2 of this article, as well.

I've spent the last several months of my off-work hours plugging away at helping the folks over at Gamers With Jobs get rolling with an upgraded version of Drupal, and in the process we decided to move from a shared hosting environment to a place where we've got a lot more control over performance and site configuration. In the meantime, Victor Kane's article on getting Drupal up and running on a Linode came across my RSS reader and provided the kick in the pants I needed to really investigate it. I looked at several VPS options, but in the end Linode seemed to be the best. They offered a seven day money back guarantee, which honestly isn't much, but it was long enough for me to feel comfortable giving it a shot without being out sixty bucks, so I decided to try it out.read more...

Syndicate content