Before we begin, let us explain what swap is…
Swap, in server terms, is hard disk space reserved by the system to be used as a backup memory in case the RAM is fully used. Once a server starts using swap memory, then you will notice some degradation in the performance because the RAM is always much, much faster than a hard disk. SSD hard disks fare much better when the system starts using the swap.
A very large racing website (built on Joomla) experienced extremely heavy load in the past weekend. The website eventually crashed. Granted – the website was getting a record number of visitors because of a huge racing event, but the website was highly optimized and it was a bit weird that it crashed. The site’s administrator called us for help, and, of course, we obliged…
When a large website crashes, we take the following steps:
- We ssh to the site’s server as root.
-
We stop MySQL and then Apache by executing the following commands in the shell:
/etc/init.d/mysql stop
/etc/init.d/apache2 stop -
We add the following to the very beginning of the .htaccess file (which is located under the root directory of the website):
deny from all
allow from <Our IP Address> -
We restart Apache and MySQL using the following commands:
/etc/init.d/apache2 start
/etc/init.d/mysql start
Doing the above will ensure 2 things: 1) We have exclusive access to the website, and 2) all the load associated with the website is no more.
So, after doing that, we examined the slow query log and we discovered that there was nothing of importance there. The slow query log was very slim (it was a few hundred kilobytes and it was gathering slow queries for weeks…).
We then checked the disk space available on the hard disk (maybe we ran out of disk space) using the following shell command:
df -m
To our disappointment, the server had more than ample disk space in all its partitions! So that wasn’t a disk space problem.
We then checked the memory allocated to the server to see if there’s something wrong there, using the following command:
free -g
Here’s what we got:
total used free Mem: 16 12 4 Swap: 4 4 0
Aha! The system ran out of swap! Not only that, for some reason, the swap allocated to the system was sort of a locked! Which meant that the Joomla website wasn’t able to use it when the load gets serious! So what we did was that we released the swap by turning it off, and then we turned it on again by issuing the following 2 shell commands:
swapoff -a
swapon -a
We then removed the deny from all line added to the .htaccess to allow everyone to access the website. Once we did that, we monitored the load and it was OK, but we knew that it’s only a matter of time before the same problem happens again… Why?
Because of 2 things:
- If the system runs out of swap, it means that it ran out first of memory, which means that the memory is no longer enough for that large website (the website gets about 2 million visits every month and it is in the top 10k most visited websites in America).
-
Even if we accept the fact that the system has to touch the swap then and now, the swap allocated is too little. In fact, we think that the amount of swap on a server should be the same as the RAM. So, if you have 16 GB of RAM on your server, you should have 16 GB of swap.
So, how did we solve the problem for the long term?
We solved the problem pretty quickly and easily: 1) We asked the host to upgrade the RAM on the server to 32 GB, and 2) We asked the host to increase the amount of swap to 32 GB. Doing that ensured long term stability of the website.
If your website crashes often immediately after a huge load, then check the swap on your server. If you see that your swap is full, then most likely your server’s memory is no longer enough and you need to increase it (and also increase the swap). If you think this whole thing is not exactly in your court, then throw it in ours, and let us handle the problem. Just contact us and let us prove to you how fast, efficient, professional, and affordable our work is!