Forbid Access to Bad Webbots (and others?)

### Forbid access from certain known-malicious browsers/bots
RewriteCond %{HTTP_USER_AGENT} nhnbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} naver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NHN.Corp [NC,OR]
RewriteCond %{HTTP_USER_AGENT} naverbot [NC,OR]
# Korean addresses that Naverbot might use
RewriteCond %{REMOTE_ADDR} ^61\.(7[89]|8[0-5])\. [OR]
# Korean addresses that Naverbot might use
RewriteCond %{REMOTE_ADDR} ^218\.(14[4-9]|15[0-9])\. [OR]
RewriteCond %{HTTP_USER_AGENT} Sleipnir [NC]
# Allow access to robots.txt and forbidden message
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{REQUEST_URI} !^/403\.shtml$
RewriteRule ^.* - [F,L]

• Note the chaining of rewrite conditions. These condition lines implicitly have "ands" between each two line unless an OR is used. They also apply "anded" to the actual rewriting rule.
  
  
• Note that this primarily applies to robots, so it would not have been a bad idea to also check to make sure HTTP_REFERER was empty. Most robots enter the site without any referrer. If you have HTTP_USER_AGENT checks that may accidentally catch real users, a second check making sure the referrer is empty wouldn't be a bad idea.
  
  
• Note the explicit check for robots.txt and 403.shtml. Without this check, the robots will be forbidden from seeing your custom built 403 message and your robots.txt which tells the robot where it should and should not be.
  
  
• Note the use of the F option on the rewrite rule. This instructs the web browser to send a 403 Forbidden.
  
  
• Note the use of regular expressions to pick out IP address ranges. A strong grasp of regular expressions will be very helpful when writing these rules and conditions.

Forbid Access to Only Certain Types of Files from Certain Agents

### Forbid Yahoo's MMCrawler from accesing multimedia (anything non-text)
RewriteCond %{HTTP_USER_AGENT} MMCrawler [NC]
RewriteCond %{REQUEST_URI} !^/.*\.(txt|tex|ps|pdf|php|htm|html|shtm|shtml)$ [NC]
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{REQUEST_URI} !^/403\.shtml$
RewriteRule ^.* - [F,L]

• This rule keeps MMCrawler from grabbing anything but true CONTENT.
  
  
• Again, notice the explicit exclusion of robots.txt and 403.shtml from this rule. In this rule, this is NOT NECESSARY since these are already excluded by the rest of the rule.
  
  
• Note the rule expression could be more compact, but it is easier to read this way.

Forbid Access to Certain Documents

### Forbid access to sensitive documents
RewriteCond %{REQUEST_URI} (^|/)\.htaccess$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/guestbook\.csv$ [OR]
RewriteCond %{REQUEST_URI} ^/post_template\.php$ [OR]
RewriteCond %{REQUEST_URI} ^/page_template\.php$ [OR]
RewriteCond %{REQUEST_URI} ^/hitlist\.db$ [OR]
RewriteCond %{REQUEST_URI} ^/includes($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/ban($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/removed($|/.*$) 
RewriteRule ^.* - [F,L]

• This is a simple rule. It shows more regular expressions and how to block access to important non-content files that just happen to live in the same directory (or close) as web content.
  
  
• NOTE that the last three of these rules block entire directories AND ALL OF THEIR CHILDREN.

Prevent Good Spiders from Entering Traps for Bad Spiders

### Forbid access from known good spiders to spam traps and other nasty spots
### (this PROTECTS the good guys!!)
RewriteCond %{REMOTE_ADDR} ^61\.(7[89]|8[0-5])\. [OR]
# Googlebot
RewriteCond %{REMOTE_ADDR} ^64\.68\.82\. [OR]
RewriteCond %{REMOTE_ADDR} ^216\.239\.39\.5$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.249\.(6[4-9]|[78][0-9]|9[0-5])\. [OR]
# Yahoo Slurp
RewriteCond %{REMOTE_ADDR} ^66\.196\.(6[4-9]|(7|8|9|10|11)[0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^68\.142\.(19[2-9]|2[1-4][0-9]|25[0-5])\. [OR]
# msnbot
RewriteCond %{REMOTE_ADDR} ^207\.46\. [OR]
# psbot
RewriteCond %{REMOTE_ADDR} ^62\.119\.133\.([0-5][0-9]|6[0-3])$ [OR]
# Cyveillance
RewriteCond %{REMOTE_ADDR} ^63.148.99.2(2[4-9]|[34][0-9]|5[0-5])$
# Bots don't come from referrers
RewriteCond %{HTTP_REFERER} ^$ 
# Prohibit suckerdir and trapme access
RewriteCond %{REQUEST_URI} ^/(suckerdir|trapme)(/|$)
RewriteRule ^.* - [F,L]

• There are a number of methods to trick spambots into areas that record their presence, submit their IP to authorities, and block them from further access to the site.
  
  
• These methods often "poison" the spambot as well by providing ficticious e-mail addresses and (perhaps not obviously) recursive links. Clearly, it would be bad if a GOOD bot ever found its way into such traps. This would waste the resources of the good bot. This would also possibly submit bad content onto a search engine. It additionally might ban a legitimate bot from your site (and others).
  
  
• This rule tries to prevent good bots from wandering into bad traps.
  
  
• Notice the restriction is on a class of requests that start with a particular string.

Prevent Real People from Entering Traps for Bad Spiders

### Forbid access from unknowing web browsers who happened upon the traps
### (this PROTECTS the little people!!)
# Real people often do come from referrers. Protect them. 
RewriteCond %{HTTP_REFERER} !^$ 
# Prohibit suckerdir and trapme access
RewriteCond %{REQUEST_URI} ^/(suckerdir|trapme)(/|$)
RewriteRule ^.* - [F,L]

• It would also be bad if real people came upon these requests. Most likely, if they hear of these requests, it'll be from a link that some jerk has put on a page somewhere.
  
  
• Again, remember that bots usually carry no HTTP_REFERER.
  
  
• Since these traps are designed for bots, forbid access from links. Make sure the HTTP_REFERER is empty.

Setup an Environment for Bad Spider Traps

### Allow access into the suckerdir and trapme traps for all others
## Setup the sand traps, suckerdir and trapme
# This RedirectMatch makes sure there's a trailing / on "directories"
RedirectMatch /(suckerdir|trapme)$ http://www.tedpavlic.com/$1/
# This RewriteRule makes sure there's a trailing / on "directories"
RewriteCond %{REQUEST_URI} (suckerdir|trapme)/(.+)$
RewriteCond %{REQUEST_URI} !(suckerdir|trapme)/(.+)(\.(html?|php)|/)$
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R,L]
# This RewriteRule makes index.html the "DirectoryIndex"
RewriteRule ^(suckerdir|trapme)(/|/(.*)/)$ $1$2/index.html
# This RewriteRule actually generates the content for each query
RewriteRule ^(suckerdir|trapme)/.+$ $1/$1.php

• This set of rules helps to "create" the traps mentioned above. The actual traps also involve some scripts that generate the bad content, but these rules make those scripts more believable. As you can see from the last line, the scripts that make all the magic of this trap are suckerdir.php and trapme.php. Note that the index.html file does not exist and the second-to-last line really isn't needed. It's just there for my own amusement in the wonder and power of mod_rewrite.
  
  
• Notice the use of HTTP_HOST. If and when the site name changes in the future, this makes it easy to transport this rule to the new site name. REMEMBER that one of the FIRST RULES redirected to the desired site name, so HTTP_HOST at this point is a known quantity.
  
  
• Remember that you can only use %{HTTP_HOST} with the mod_rewrite directives. This DOES NOT EXIST with the mod_alias directives.
  
  
• The end effect of this rule is that any request to a DIRECTORY, html FILE, or php SCRIPT that BEGINS with suckerdir or trapme ACTUALLY gets EXECUTED by the suckerdir.php and trapme.php scripts WITHOUT THE AGENT EVER KNOWING.
  
  
• These rules also make sure that requests that look like requests to directories without the trailing slash get redirected to the version that does have the trailing slash before actually getting processed. This helps convince a bot that it's looking at real content.
  
  
• This shows a way to make dynamic content LOOK STATIC.
  
  
• It also shows how one script can operate AN ENTIRE SITE and the user will PERCEIVE that the site is MANY pages with an entire DIRECTORY STRUCTURE.

Strip the Query Strings from Requests from Bots

### If we detect a bot at all, set an environment variable
# NOTE: It is okay to match bad bots here too. We just don't want to match
#       real human people.
# To match most bots, check out User-Agent and look for empty referrer
RewriteCond %{HTTP_USER_AGENT} (google|other_bots|wisenutbot) [NC]
RewriteCond %{HTTP_REFERER} ^$
RewriteRule ^.* - [E=HTTP_CLIENT_IS_BOT:1]
# Certain bots actually do have referrers. Catch them too.
RewriteCond %{HTTP_USER_AGENT} (becomebot) [NC]
RewriteRule ^.* - [E=HTTP_CLIENT_IS_BOT:1]

### If we match a bot, strip query strings from requests
# Match a bot
RewriteCond %{ENV:HTTP_CLIENT_IS_BOT} ^1$
# Look for non-empty query string
RewriteCond %{QUERY_STRING} !^$
# Force it empty and tell the bot that it's a permanent change
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1? [R=permanent,L]

• I've done both setting and checking environment variables here. This isn't necessary. Those lines could be combined, but I thought this was a good place to show an example of using environment variables in this way.
  
  
• One advantage of using environment variables here is that it passes useful information back to my web scripts. In this case, checking the HTTP_CLIENT_IS_BOT environment variable lets me know that it has met my "bot criteria" setup here in the .htaccess file. I can then tailor my content for bots.
  
  
• The first half of these rules identify probable web bots. Since nearly all bots always have empty referrers, it's easy to reduce false positives by checking for an empty HTTP_REFERER.
  
  
• The second half strips the query string from all queries identified as being from bots by the first half.
  
  
• This is useful to me since many of my pages use query strings to change the display format, but the content stays the same. These rules prevent redundant indexing of content.
  
  
• Notice the use of the ? at the end of the rewriting rule. A single ? at the end of the rule REMOVES THE QUERY STRING. This is one of those documented features that is OFTEN OVERLOOKED.
  
  
• NOTICE that this REDIRECTION does not occur if the QUERY_STRING is EMPTY. This prevents REDIRECT LOOPS!!

Some Cheap and Simple Redirects

# Redirect requests that should be going to subdomains directly
RewriteRule ^osufirst($|/.*$) http://osufirst.tedpavlic.com$1 [R=permanent,L]
RewriteRule ^schedule($|/.*$) http://schedule.tedpavlic.com$1 [R=permanent,L]
RewriteRule ^europa/schedule($|/.*$) http://schedule.tedpavlic.com$1 [R=permanent,L]
RewriteRule ^blocko($|/.*$) http://blocko.tedpavlic.com$1 [R=permanent,L]
RewriteRule ^europa/blocko($|/.*$) http://blocko.tedpavlic.com$1 [R=permanent,L]

# Redirect some renames that may still be linked elsewhere under old names
RewriteRule ^servo.php$ http://www.tedpavlic.com/post_servo.php [R=permanent,L]
RewriteRule ^riley\.(jpg|gif|png)$ http://links.tedpavlic.com/riley.$1 [R=permanent,L]

• Notice the permanent keyword. Using this keyword is identical to using R=301. If no keyword is given, R=302 is implied, which performs a temporary redirect.
  
  
• Other status codes may be used as long as they correspond to REDIRECT operations. In other words, other status codes may be used as long as they are greater than 299 and less than 400.
  
  
• To indicate that a page is gone or forbidden, use the G or F flags, respectively and replace the target URL with a - (hyphen). See the next section for an example.

Redirects with Other Status Messages

# Now pages that have just been removed and replaced
RewriteRule ^opinions.php$ http://www.tedpavlic.com/general_posts.php [R=seeother,L]

# Now pages that have just been removed entirely
RewriteRule ^analog_ee.php$ - [G,L]
RewriteRule ^teaching.php$ - [G,L]
RewriteRule ^wav($|/.*$) - [G,L]
RewriteRule ^toys_.*.s?html?$ - [G,L]

# Now forbid some pages that I want to keep around but don't want people to see
Redirect 403 /phpinfo.php

• seeother provides a good alternative to a permanent redirect. It isn't implying that the target page is a replacement, but it is stating that the desired page is gone, but a similar page is available.
  
  
• gone does not take a target parameter with mod_alias directives and takes a simple hyphen (-) as a parameter with mod_rewrite directives. Use this status when a page has been permanently removed. Note that using G instead of R=gone is NOT possible using mod_rewrite's RewriteRule since - is given for redirection.
  
  
• The final mod_alias rule uses 403 to indicate that all requests to /phpinfo.php should be given the 403 Forbbiden status. There is no mod_alias keyword for forbidding pages; however, mod_rewrite provides the F parameter.
  
  
• NOTE that the final rule uses a mod_alias keyword, so it applies to all subdirectories. This includes subdomains that happen to be hosted beneath this directory. So this single rule prohibits phpinfo.php access on ANY of my subdomains.
  
  
• Being smart about these very technical issues helps to make sure search engines (and users in general) stay up to date with the layout of your site.

Case Insensitive robots.txt

# Any request for robots.txt, regardless of case, should go through
RewriteRule ^robots.txt$ robots.txt [NC]

• This rule allows robots.txt to be fetched with any name. It makes the robots.txt file completely case insensitive.
  
  
• With the addition of this rule, it is a good idea to make every other robots.txt condition case insensitive with the NC parameter.