How to repair a damaged site
[This article is frequently updated and expanded. It is gradually being broken apart into separate articles because the Google language translator doesn't translate all the text of large pages.]
[Contrary to one of the tags applied to this article at StumbleUpon, this site has never been hacked in more than 1,000,000 attempts. Everything reported here is based on experience gained from helping others with compromised sites and from continual study about improved methods of protection.]
Step-by-step site repair
- Hopefullly, this detailed step-by-step procedure will help focus on the tasks and avoid panic.
- The concepts apply to any server even though it is Linux, Apache, and cPanel methods that are described in detail.
- The steps are in order of priority if the evidence you've found so far hasn't already given you a clear idea what things to focus on first.
If you just start at step 1, focus, and dive in, what you learn now will help you manage your site with a lot more confidence in the future. These are all useful things to know how to do. You might even wind up feeling like an expert.
What not to doDon't just repair the damaged files and hope this experience doesn't happen again. That is not enough.
Nobody is ever supposed to be able to add, delete, or change files in your website without your permission. It should never happen, and it usually doesn't. Most websites don't get hacked. If yours did, there is something wrong with it, or with the server, or with the webhost, or with the security on your PC. You have to figure out how this happened so you can prevent it from happening again.
Ok, let's get started... The checkboxes don't do anything. You can check them to help keep your place as you go.
1) Log into cPanelMost webhosts provide some kind of control panel such as cPanel or Plesk where you can manage your website's configuration and files. One reason for logging in now is to check for unauthorized logins as described below. The more important reason is to make sure you know how to do it, because several of the later steps are done in control panel.
If you've never logged into your control panel before now, go to the home page of your webhost's website and look for a customer login box. If there isn't one, look for a FAQ page where they might describe how to access your control panel. If you still find nothing, file a support ticket and ask them.
In cPanel (and possibly in Plesk), the line that says "Last login from:" should always be your IP address from the last time you logged in. If it isn't, write it down.
If you don't know your IP address, it appears to be 126.96.36.199, but that could be incorrect if you are viewing an old copy of this page from your browser cache or a search engine cache. You can find your IP address in Windows XP by either of these two methods (you must be connected to the internet at the time you do this):
- Click on the internet connection icon in your system tray (lower right of screen) . In the dialog box that opens, click the Details tab, and then read the line that says Client IP address.
- Open a Command Prompt and run the ipconfig program:
start > Run > cmd
Read the line that says IP Address
If someone was able to log in to your control panel (like you do), they have your userID, password, and all the same access to your site that you have. They can probably also get FTP access, which is what they are more likely to use than cPanel. However, before you assume the worst, an unfamiliar IP could be legitimate if your site is at a webhosting company and you recently submitted a support ticket. A technician might have logged into your account while investigating.
The three pieces of information you should keep from this step are:
- How to log in to your control panel.
- Your legitimate IP address, so you can recognize IP addresses that are not yours in places where only yours should be.
- Suspicious IP addresses you find reported in cPanel.
- Go to cPanel > Raw Log Manager (the name varies in different cPanel versions).
- Check the "Archive Logs..." box.
- Uncheck the "Remove the previous month's archived logs..." box.
- Click Save
3) Take your website offlineIf your pages have become infected with viruses that will attack your site visitors, which is usually the case, you should protect your visitors, and your reputation, by taking your site offline, which involves adding a few lines to your .htaccess and optionally uploading a file. If you do this right away, you might avoid getting the "This site may harm your computer" warning in Google search results and a similar warning at Yahoo.
Are you hesitant to take your site offline? Consider this: a visitor who finds your site down will hardly notice the incident and will (or at least might) come back later. A visitor who gets attacked by a virus from your site will develop a strong memory of the incident and probably not come back, ever.
In addition, it is possible that a script with a security hole was the reason the site got hacked. As long as that script is publicly accessible, the site remains vulnerable, which means it could get hacked again even while you're trying to repair it.
Lastly, it is possible the attacker installed a backdoor script to let themselves back into the site. Closing the site at least has a chance of locking them out and making it impossible for them to use the backdoor, giving you time to find and delete it.
4) Notify your web hosting companyFile a support ticket.
- Tell them what has happened. Give them as much detail as you can about the evidence that the site is compromised.
- If you have some idea when it happened, or when you first noticed it, tell them.
- If you found an unknown IP address in cPanel, report it.
- Give them a secondary email address that is not at your website so your host can still contact you if your site goes down or if the hacker is reading or deleting your website email.
- Some webhosts will be willing to help you investigate and clean the site. Others won't, but it doesn't hurt to ask if they can help or give advice.
- If you're on shared hosting, it is possible that the host is aware of other sites on your server that are affected. They probably won't publicize it and might not even tell you, but your report will help them, even if they don't admit it.
- Also, only your webhost can clean up files outside your webspace that might have been affected.
Make sure everyone who has password access to the website does at least one, and preferably two, antivirus and antispyware scans on their local computers, using two different scanners they don't normally use, to find threats that got past the AV scanner they were using. Some free scanners are at: Trend Micro Housecall, Kaspersky, Malwarebytes, Symantec (Norton), BitDefender, Windows Live OneCare, Computer Associates, McAfee, F-Secure.
As long as the webmaster's PC is infected, changing the password is no use. The new one gets stolen, too.
6) Change all passwords: cPanel, FTP, databases, emailAfter the administrator PCs are free of viruses and spyware, change all the website passwords that you use for control panel, FTP, database connections, email, everything. Use strong passwords. If you have been using a single password for more than one purpose, take this opportunity to make every password different. The linked article explains why this is important.
a) If the FrontPage Extensions are installed on the site, change your FrontPage password first:
- Open your local copy of your site in FrontPage
- Click the Remote Web Site tab and log in
- Click Open your Remote Web site in FrontPage (this will open a new copy of FrontPage with your remote site in it)
- Click Tools > Server > Change Password and follow the instructions. Whenever you get a password prompt during this procedure, it wants the old one. It doesn't want the new one until it asks for it.
b) Log in to your webhosting account and change your cPanel / FTP passwords thereIn cPanel, look for a "Change Password" icon or link. If you find none, your webhost might provide a separate login location for making password changes, so search their FAQ, forum, or ask customer support.
If you have scripts that use your cPanel userID/password to open database connections, the password change will cause those scripts to stop working, and you will get connection failure or "Could not connect" errors:
- If the connection data is hard-coded into the scipts, go through the scripts and change the password in all of them.
- If your scripts read the connection data from an include (or other) file, change it in that file.
- Since you're editing the files anyway, a better and more permanent solution is to stop using your cPanel userID/password, create a different user/password just for database connections, put the connection data in one protected include file, and have all your scripts read the data from that file.
- Go to cPanel > MySQL® Databases > Current Users.
- In the list, find the user you want to modify. In shared hosting (and maybe in other environments, too), the username is prefixed with YourUserID_.
- In Username: enter the name of the user, but do not enter the prefix or the underscore. Enter only the part after the underscore. If the user is userID_example, then you enter example.
- In Password: enter the new password.
- Click Create User.
- The confirmation screen will tell you that the user was created with the new password.
- When you return to the MySQL Account Maintenance screen, you'll see that you have not really added a user, but only replaced the old one's password, and that this user still has the same privileges in the same databases that it had previously. You will also see that cPanel has automatically added the userID_ prefix to the username.
- Now change all your scripts to use the new passwords. See the bullet points in section b) above.
d) Change the passwords for all your email accounts
- Go to: cPanel > Mail > Add/Remove/Manage Accounts.
- Set a new password for each account.
- If you access your email with a POP or IMAP email client such as Microsoft Outlook, change its configuration settings so it knows the correct new password for each account.
7) Upgrade all third party scripts to latest versionsMake a list of all the scripts you use. For each, if you are not using the latest version, upgrade now.
Follow links in the table below to find latest version information for some common scripts, and to view the latest security advisories at Secunia.com. The Secunia page often lists vulnerabilities found in plug-ins or add-ons. Check those, too. If there is a recent security advisory for a script you use that is outdated, there is a good chance you've found the reason your site was hacked.
|Link to latest version information||Security advisories at Secunia.com|
|CKEditor / FCKeditor||Security Advisories|
|Coppermine Photo Gallery||Security Advisories|
|Joomla (all versions)||Security Advisories - Joomla itself |
Wider search to also find components
Joomla Vulnerable Extensions List (VEL)
|Noah's Classifieds||Security Advisories|
|Nucleus CMS||Security Advisories|
|osCommerce||Security Advisories Just keeping up to date has not always been enough to keep osCommerce secure. The osCommerce forum has additional security advice for Version 2.x and Version 3.x|
|SMF (Simple Machines Forum)||Security Advisories|
|WordPress|| Security Advisories "WordPress" |
Security Advisories for WordPress 2.x
Security Advisories for WordPress MU 1.x
Or find WordPress on this page
|Zen Cart||Security Advisories|
8) Examine your own PHP or ASP.NET code for security holesThe "What is a website hack?" article (top of this page) has more information about the following three most common exploits of custom code, and some others:
Remote File Inclusion (RFI), Local File Inclusion (LFI)The following PHP functions:
can be tricked into fetching a malicious script from a remote server and running it as part of the currently executing script if the value of $variable came from an HTTP query string or other user-supplied input and if the value supplied is a URL (web address) rather than the value that the programmer expected.
They can be tricked into divulging the contents of password or other sensitive files if the supplied value of $variable is a local file path on the server.
SQL InjectionWhen an HTTP query string, or any other data from the outside such as input to a search box, is used in the building of an SQL database command string, maliciously crafted input can corrupt the SQL command, causing it to inject content into database tables or list the contents of the database (such as user names and passwords) on the output page. A widespread attack that used SQL injection was called ASPROX.
If you suspect that a script you wrote yourself might be the security weakness, it is safest to stop using that script until you can examine it carefully. After making a local copy for yourself, delete the script from the server. Removing the links to it isn't enough. As long as the script is on the server, anyone who already knows its name can still access and exploit it. If you leave it on the server, at least rename it.
9) Find and repair all the malicious changes that were madeNow that you have discovered where the security weakness was, and fixed it, it is now safe to repair your website's content, because the attackers won't be able to damage it again.
As described in the "What is a website hack?" article (top of this page), after someone has gained access to your site, they can change anything they want and can do an extraordinary amount of damage. In order of most to least common:
- Modify database tables, usually to inject the same types of content listed above, so it will appear on your pages.
- Add new files.
- Add executable programs to let the attackers "manage" your website files remotely, grant them access even after you clean up (back doors), send spam, connect to IRC servers for botnet communications, mass-attack other websites, etc.
- Subvert the operating system, putting the entire server under the control of a remote operator.
Two "clean sweep" shortcuts: replace entire website from known-good backups
Steps 9a) to 9d) describe ways to locate and repair files that have been maliciously altered, which can be a time-consuming and painstaking chore, especially if you're not comfortable working with HTML code.
In some cases, it can save time to simply replace everything that might have been damaged with fresh copies that you know are clean. However, doing this destroys the evidence you might need for determining how the attack occurred and how to prevent it happening again. Therefore, before doing this, you should already have a clear idea why the attack succeeded, or should make a copy of the hacked site so you can study it later:
- Less drastic - replace contents of public_html: If you are thoroughly familiar with what is in your public_html folder and you are certain this method won't destroy irreplaceable files, you can use cPanel > File Manager or FTP to delete all the files and folders inside /public_html (but don't delete the public_html folder itself) and republish the entire site from a known-good backup.
It will still be a good idea to look for damaged files or malicious new ones in your root directory (/) and its other subdirectories other than public_html.
- More drastic - reprovision: To really start fresh at a shared host, you can ask the host to "reprovision" your account, to recreate it as though it is brand new. You lose your historical logs and stats and must build the site up from nothing. I recommend against this unless all other options have failed.
9a) Get a complete listing of all the files in your websiteThese sections (9abc) describe three ways to view a list of all the files in your website: shell command (cron), FTP, and cPanel File Manager.
Linux "cron" allows you to run a shell command that emails to you a complete listing of all the files in your site, showing for each the name, timestamp, size, owner, and all the permissions settings. This is by far the best method..
How to use the directory listing:
It is ideal if you have a similar list that you made previously when the site was clean. You can compare the two to find files that have changed size, files whose timestamps or permissions are not what they should be, and new files that shouldn't be there.
If you don't have a known-good list to compare against, you can still review the new list for files that seem out of place or have wrong ownership or permissions. This will be discussed below.
9b) Examine your site's files in cPanel > File ManagerFileManager allows you to easily review filenames and permissions, but it doesn't show any other information about the files, and navigating up and down the directory tree is a tedious process. File and folder permissions are shown numerically. The article linked above at "Get a complete listing" describes how to translate between numeric "755" and "rwx" notation.
9c) Examine your site's files using FTPIn an FTP view of your website, the folders and files look like what you are used to in Windows Explorer, with a navigational directory tree pane on the left and a folder contents pane on the right. FTP view is easy to navigate, and it allows sorting on the Date Modified column to easily spot recently changed files.
9d) What to look for in the list of files
- New files with obviously suspicious names. Some hacks install files with names like hacked.html or vulnerable.php, etc. Others might have nonsensical names or names consisting of random character strings. Some might be in locations that make them suspicious, like a .php file in your /images folder. If you find a file that was definitely installed by the attack, search for other files that have almost the exact same timestamp.
- Files you don't recognize. Determine whether each one is malicious or not. You can examine plain text PHP (.php) or Perl (.pl) scripts in a text editor.
Unfortunately, you cannot simply delete all the files that aren't yours. Some are required system files that you just never noticed before. When in doubt, do a web search on the filename or post a question in a forum. Research the names of unfamiliar CGI programs, since they cannot be examined visually.
If an exploit modified files on your server but didn't affect your displayable pages, it suggests that your site visitors weren't the target of the attack. Instead, it might have been trying to turn your site into a spam emailer or into a robot crawler to attack other sites, or to install on your site a library of malicious scripts or other content to be called by injected iframes or RFI attacks on other websites.
- Check your root directory ("/") and its subdirectories for malicious or altered files. Even if you delete the contents of your public_html and republish the site from scratch, that doesn't overwrite your folders above public_html, so you must check those manually.
9e) Search your website files for suspicious changesThere are many PHP script that can help search your website for suspicious filenames, for suspicious code, and for other suspicious text.
10) Check that your file and folder permissions are secureUsing the complete file list you made, make sure file and folder permissions are what they should be. Although your complete file list is a text file, the search isn't too difficult. You can search for suspicious "world-writable" 777 folder permissions by searching for the equivalent "rwxrwxrwx" in the text. World-writable 666 file permissions appear in the text as "rw-rw-rw-".
Common correct permissions for world-readable (but not world-writable) folders are 755 (rwxr-xr-x), and common permissions for world-readable files are 644 (rw-r--r--). Those are what you should mostly expect to see.
There are only two situations where world needs write access (777 / 666), and both only apply if your server is configured with PHP as an Apache module:
- A file needs 666 permissions if PHP needs to a) open the file and write data into it, or b) copy another file to the directory entry currently occupied by this file.
- A folder needs 777 permissions if PHP needs to a) dynamically create new files in it, or b) delete existing files from it. However, if PHP only needs to open and modify the contents of an existing file or even copy another file to the directory slot occupied by an existing file, the folder does not need 777 permissions. It is only necessary that the destination file have 666 permissions. That is counterintuitive because you would think that copying a file involves deleting the existing file and putting the new file where it was, but that is not how Linux views it. It only considers it a change in the file's content, not a change to the directory, so the directory can remain read-only. This is important because there may be some files that PHP only needs to create once, during a program's initial installation when it's setting up its data files. After that, it's possible PHP can do everything it needs with the file set to 666 but the directory locked back down to read-only 755. That is much better because although that one file remains potentially vulnerable to modification, a hacker cannot put new malicious files in a 755 directory.
- Check the contents of 777 folders to ensure they don't contain malicious new files.
- Check the contents of 666 files to ensure they don't contain new malicious code.
- If you can't think of a good reason why the loose permissions are necessary (does PHP really need to make the changes those permissions allow?), try tightening them to 755 / 644.
- Even if you do know why the loose permissions are necessary, try to think of a way to make those permissions unnecessary.
11) Change all your passwords againIn case someone was "watching" inside your site while you did it the first time, do it again now that you know the site is clean.
12) Try to identify the IP address that attacked youThis is not to hunt down the attacker, which is usually pointless (most are robots, and there are millions of them). Rather, the IP address helps find other important information about the attack.
If you can identify their IP address, you will be able to search all your logs for all the places where that IP address appears. That will help identify what weak part of your site was attacked, how it was attacked, and what malicious actions were performed.
Stats programs like Analog, Webalizer, or AWStats won't be much help because they generate aggregated summary statistics. You need the details about individual page requests.
cPanel > Web/FTP Stats > Latest Visitors is useful and easy. It's a good place to go when you first discover the problem, but it's only a start. The raw log text files are a better source of information.
7-Zip. It is a command line utility that you run from a "Command Prompt" (aka "DOS box").
b) Get your logs from cPanel > Raw Log ManagerThe log file location in Plesk has a similar name. If you don't have cPanel, Plesk, or a comparable control panel, you can usually get the logs by FTP, usually from a folder outside public_html, with "logs" or "access logs" in its name. Some shared webhosts don't provide access logs, or they charge an extra fee for them.
- Go to cPanel > Raw Log Manager. If you don't see a log file there, try cPanel > Raw Access Logs. That is a holding file where your data is stored until the server does its daily statistics processing, after which the data file is transferred to Raw Log Manager.
- Click the name of the file you want to download.
- At the Open or Save prompt, click Save. Use a descriptive filename. Save the file to a folder that will be easy to navigate to in a Command Prompt. C:\TEMP works well.
- Open a Command Prompt:
Start > All Programs > Accessories > Command Prompt, or
Start > Run > cmd.exe
- Go to the folder where you saved the .gz file: cd \TEMP
- Type the command line to extract the .gz file: 7za.exe x filename.gz
- You should get a report that says "Everything is Ok".
- I usually delete the .gz file and rename the output file to .log.
- The unzipped log files can be extremely large. In Windows, WordPad can handle up to about 12MB. For easier viewing, set the font to a monospaced font like Courier New, with word wrap Off. Notepad++ can handle files of 100MB or more. In Linux, the gedit editor capacity seems almost unlimited.
- If you are comfortable using Microsoft Access, the Webstats.mdb database has tables into which you can import your log files.
- The HTTP log will also import into Excel, but you will need to tweak the text import wizard settings to get the fields into their columns properly.
Your HTTP log shows the visits to your site by HTTP, the request method normally used by ordinary visitors (using their browsers), robots, and hackers.
It's not always easy to determine which lines in an HTTP log are suspicious and which ones aren't. It classifies the attempts by type so you can see what ways your site is being attacked, and it explains how the different types of attack work.
If you find suspicious changes made to your site (such as file timestamps that are not from when you changed the files yourself), you can try to correlate those changes with the suspicious entries in your log.
For example, a hacked file's timestamp will often show when the hack occurred (unless the hacker made a special effort not to change the timestamp). If your HTTP log shows a malicious request at the moment of the changed file's timestamp, that is very suspicious.
It could indicate that the file requested by the hack attempt had a security vulnerability that the hacker was able to exploit with their request. The exploitable file does not have to be the same file that was modified. The exploitable file is just the doorway to get at the other files. In this case, you would examine the requested file (not the modified file) for possible security vulnerabilities. This is how your logs can help identify how a hacker got in.
As another example, if you use a database, and if SQL injection attacks are the only type of hack attempt your site ever receives, SQL Injection becomes your primary suspect.
Your FTP log shows FTP accesses to your site. FTP stands for File Transfer Protocol. In contrast to HTTP, which is most often used to request files for viewing, FTP is a method of transferring files both to and from your server. It's normally used only by you, the site administrator, but if malicious people or robots manage to log into your FTP as you, they can download your pages, modify them, and upload them back to your website. The only IP addresses in the FTP log should be yours and other authorized FTP users. Make sure the timestamps match times you were logged in and doing transfers.
There is reference information about FTP log file format at Apple Developer Connection.
I've seen reports of numerous instances where a webhost spotted in an FTP log a transfer from an IP address other than that of the site owner and immediately informed the owner that their password had been stolen. In too many of these instances, the surrounding circumstances make the webhost's claim unbelievable. Here is an alternative explanation:
PHP scripts called by RFI attacks sometimes use PHP's FTP file transfer functions to download additional malicious scripts and related files from a remote server so it can run or install them. The initial RFI includes the remote script into a legitimate script on the victim server, at which point it becomes a part of that script. The script then initiates an FTP transfer, which is recorded in the FTP log. The server does not show its own IP address in the FTP log, but rather that of the second party to the transfer, the remote website. The log of the session makes it appear as though someone logged in (which would have required the password) and initiated an FTP transfer, but in fact there never was a login. There didn't have to be one, because the session was initiated on the server, from the inside.
Remember this as a possibility if you find IP addresses other than yours in your FTP log or if your webhost tries to convince you too quickly, without considering other evidence, that your password "must have been" cracked. The danger of believing this easy story line (if it is not true) is that it can lead you to believe that all you have to do is change your password. However, if the real initiator of the FTP transfer was an RFI attack, changing your password won't help at all.
c) Use .htaccess or cPanel > Deny IP to block the hacker's HTTP access to your siteIf you identified the hacker's IP address, one site where you can look it up to get more information about it is http://whois.domaintools.com/.
You can ban the IP address from your site using your public_html/.htaccess file. Apache documentation for this is at: http://httpd.apache.org/docs/1.3/mod/mod_access.html.
Review the instructions in a prior article for how to open .htaccess for editing. As described there, insert the following line in a part of the file that is not enclosed in HTML-like tags.
deny from nnn.nnn.nnn.nnn
The nnn's are the IP address to block.
If the hacker returns with a different IP that is in the same IP range (i.e. using the same ISP), you can block the whole range for a while, although that carries the risk of banning legitimate visitors, too.
The Apache documentation has instructions for banning a range. Some IP ranges are easily specified using a simple wildcard notation. Others ranges can only be successfully defined using "CIDR/netmask" notation. Although it looks intimidating, it's easy after the first time you do it.
d) If the hacker has obtained access to your cPanel or FTP, banning their IP address in .htaccess will NOT keep them out of cPanel and FTP.
If they have scripts that they call by HTTP, it will prevent them from doing that, but only until they log into cPanel and un-ban themselves in .htaccess.
13) Report or go after the hacker legally?Hacking is a violation of the terms of service for any legitimate webhost or ISP. If you can prove conclusively that someone is using a particular IP address for hacking (or spamming, too), you could report the incident to the webhost or ISP in hopes that they might shut the perpetrator down. The contact email is often abuse@ the company.
However, your chances of getting anywhere with this aren't very great. Even if you succeed, it's a drop in the bucket. Although you might feel as though you are in a battle of wits with a wily adversary, it is thousands of times more likely that you were hit by an automated drive-by attack that is playing a percentage game, with malicious requests being launched against millions of websites, from hundreds of malicious servers. If one is shut down, it's just a cost of doing business for them.
It is a more worthwhile use of your time to do everything you can to protect your site from all hackers, regardless of who they are, and understand that there will be a constant flood of attacks against your site.