The subject is: performance measurement of a Linux server.

Measuring bandwidth

First, create a file of, say, 100 megabytes as follows:

 $ dd if=/dev/urandom of=a.log bs=1M count=100

(if you're on something BSDish, or OS X, use "1m" as the blocksize)

Then simply copy the file with scp, which will print the mean transfer speed. If it's a networked drive, use time:

 $ time cp test10mb.bin /Volumes/dav/
 real	0m43.624s 
 user	0m0.002s
 sys	0m11.364s
 $ 

vmstat

Do vmstat 5 3.

• If cpu wa (wait) is high, multiple processes are blocking and io bi/bo is high too, the server its disks are the bottleneck
• If procs r is high, the CPU is the bottleneck

harddisk performance

To test harddisk performance, there are many tools available. A quick script for running hdparm a few times and printing out the mean:

 #!/bin/sh
 n=5
 total=0
 for i in `seq 1 $n`; do
   tmp=`sudo hdparm -t /dev/sda | grep "^ Timi" | awk '{print($11)}'`
   tmp=`echo $tmp | sed -e "s/\..*//"`
   echo "Test $i: $tmp"
   (( total+=$tmp ))
 done
 (( mean=$total/$n ))
 echo "Mean: $mean"

Memory usage

To find out the 10 biggest memory hogging process on a server, type:

  # ps -e -o rss,vsz,cmd --sort=rss | sort -n -r | head

Scaling down MySQL

On a default Redhat or CentOS installation, MySQL memory consumption looks roughly as follows:

 # ps -e -o rss,cmd | grep -i mysql
 38396 /usr/libexec/mysqld

Put the following entries in the /etc/my.cnf file, under the [mysqld] section:

 innodb_buffer_pool_size = 128k
 key_buffer_size = 128k      
 myisam_sort_buffer_size = 128k
 query_cache_size = 1M 

The result:

 # ps -e -o rss,cmd | grep -i mysql
 15720 /usr/libexec/mysqld

Nice, memory is more than halved. Of course, your MySQL usage should allow for this -- but it's fine for a website lightly using the database.

Apache memory usage

For light sites, the number of Apache processes can be scaled down as well. Our baseline this time isn't very high for Apache to start with, but there's room for improvements:

 # ps -e -o rss,fname  | grep -e httpd  | awk '{sum=sum+$1;} END {print(sum)}'
 55076

That's 55 megabytes in memory. Now edit the file /etc/httpd/conf/httpd.conf and find the prefork section. Then reset the following values:

 StartServers       1
 MinSpareServers    1
 MaxSpareServers    3

It's easily enough for a lightly used website. The results after a restart:

 # ps -e -o rss,fname  | grep -e httpd  | awk '{sum=sum+$1;} END {print(sum)}'
 15656

Using lighttpd on Centos

Lighttpd (also nicknamed 'lighty') is a small-footprint webserver. It's easily installable on RedHat and CentOS.

First install the RPMForge repository as described here. Then install lighttpd with:

  # yum install lighttpd lighttpd-fastcgi

Edit the configuration file /etc/lighttpd/lighttpd.conf and:

• Uncomment the line with 'mod_fastcgi' in it.
• Search for the line with server.documentroot and set it to whatever you're using. On a default install, this would be /var/www/html
• Search for the block that starts with fastcgi.server and uncomment it. Then add a max-procs directive and adjust it appropriately. The default is 4. Now this piece of configuration file looks as follows:

 fastcgi.server             = ( ".php" =>           
                                ( "localhost" =>    
                                  (                 
                                   "socket" =>  "/var/run/lighttpd/php-fastcgi.socket",
                                    "bin-path" => "/usr/bin/php-cgi",
                                    "max-procs" => 2,                                      
                                  )                 
                                )                   
                             ) 

• If you want user directories (like http://example.com/~username/) then search for mod_userdir and uncomment that line. Also add the line userdir.path = "public_html"

Now set the permissions correctly for the PID files of PHP:

  # sudo mkdir /var/run/lighttpd
  # sudo chown lighttpd:lighttpd /var/run/lighttpd

Also, PHP needs to be able to write session files, so set ownership for that directory correctly:

  # sudo chgrp lighttpd /var/lib/php/session

Stop Apache if that's what you were running previously, and don't start it at next boot:

  # service httpd stop
  # chkconfig httpd off

Start your fresh lighttpd server with the service stanza, and set to start on the next reboot:

  # service lighttpd start
  # chkconfig lighttpd on

Now test your scripts and be happy.

Some points:

• Some webbased software likes to keep a cache directory, and if you've previously ran Apache, those directories might have write rights for the Apache user. These rights will need to be adjusted so the lighttpd user can write to those directories
• If you want aliases, search for the line with mod_alias and uncomment it. Then add lines in the form of

    alias.url += ( "/wiki/" => "/some/other/root" )

• To support SSL (e.g. you're accessible through https://....) you need to run a separate instance, see Lighttpd for both SSL and non SSL
• Some pointers: an O'Reilly article

Let's see what we have saved on memory. We will look at the number resident set size (rss) to see what's actually occupied in memory and not swapped to disk:

 # ps -e -o rss,fname  | grep httpd
 12844 httpd
 10276 httpd
  9936 httpd
  7004 httpd
  6284 httpd

To add this up:

 # ps -e -o rss,fname  | grep httpd | awk '{sum=sum+$1;} END {print(sum)}'
 47840

This Apache 2.0 is configured as a preforked server with StartServers set to 3. Now show what Lighty uses for memory:

 # ps -e -o rss,fname  | grep -e lighttpd -e php-cgi | awk '{sum=sum+$1;} END {print(sum)}'
 21360

That's quite a win, I'd say.

Links

Flavio's TechnoTalk