93/247 Ways To Get Traffic
Restored from the Empower Network archive (2011–2017), lightly edited to meet our current advertising standards. Views are the original author’s.
Happy Sunday!!! Hopefully you had a great weekend, like I did. Quality time with family, and all my chores caught up (with the exception of my lawn, but until I get the okay from my physiotherapist, I have to leave it alone ) Apart from that, it was regular Fraser Valley weather…. rain and grey But over this last 48 hours, I have come to a couple of decisions. I need to do a week or so of clean up on my site. I want to make sure all of my links are working, from each separate page and from the 247 Ways to get Traffic main page. I also have a new site, WWW.CasinoCrushing.COM that I must get up and running, and as you may know, I have at least 93 Ways to get Traffic to set up on it. That being said, it is about time to get into things, after this week I will start my clean up.
So if you are looking for the full 247 Ways to get Traffic, then here you go!!! If you are looking for the reason behind all of this, here is a great place to start. And if you are looking for tipi number 93 in our Ways to get Traffic list, then just keep right on reading!!!
93. Add a robots.txt file to your site to ensure it’s crawled properly by search engines.
/robots.txt
Web site owners use the /robots.txt file to give web robots instructions about their site; this is called The Robots Exclusion Protocol.
If a robot wants to vist a Web site URL, say http://www.example.com/welcome.html. Before it does so, it first checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.
There are two main considerations when using /robots.txt:
robots can ignore your /robots.txt. Specifically malware robots that scan for security vulnerabilities, and email address harvesters that also will pay no attention.
The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.
So don’t try to use /robots.txt to hide information.
The /robots.txt is a standard, and is not owned by any standards body. There are two historical descriptions:
the original 1994 A Standard for Robot Exclusion document.
a 1997 Internet Draft specification A Method for Web Robots Control
In addition there are external resources:
HTML 4.01 specification, Appendix B.4.1
Wikipedia – Robots Exclusion Standard
The /robots.txt standard is not an actively developed item. But you can see further development of /robots.txt? for more discussion.
Now where should you put it?
The short answer is, in the top-level directory of your web server.
The not so short:
When robots look for the “/robots.txt” file for URL, it strips the path component from the URL (everything from the first single slash), and puts “/robots.txt” in its place.
For example, for “http://www.example.com/shop/index.html, it will remove the “/shop/index.html”, and replace it with “/robots.txt”, and will end up with “http://www.example.com/robots.txt”.
So, as a web site owner you need to put it in the right place on your web server for that URL to work. Usually that is the same place where you put your web site’s main “index.html” welcome page. Where exactly that is, and how to put the file there, depends on your web server software.
Remember to use all lower case for the filename: “robots.txt”, not “Robots.TXT.
What to put in it
The “/robots.txt” file is a text file, with one or more records. Usually contains a single record looking like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
In this example, three directories are excluded.
You need a separate “Disallow” line for every URL prefix you want to exclude — you cannot say “Disallow: /cgi-bin/ /tmp/” on a single line. Also, you may not have blank lines in a record, they are used to delimit multiple records. SO remember, one line for each page of disallow.
Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The ‘*’ in the User-agent field is a special value meaning “any robot”. Specifically, you cannot have lines like “User-agent: *bot*”, “Disallow: /tmp/*” or “Disallow: *.gif”.
What you want to exclude depends on your server. Everything not explicitly disallowed is considered fair game to retrieve. Here follow some examples:
To exclude all robots from the entire server
User-agent: *
Disallow: /
To allow all robots complete access
User-agent: *
Disallow:
(or just create an empty “/robots.txt” file, or don’t use one at all)
To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
To exclude a single robot
User-agent: BadBot
Disallow: /
To allow a single robot
User-agent: Google
Disallow:
User-agent: *
Disallow: /
To exclude all files except one
This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “stuff”, and leave the one file in the level above this directory:
User-agent: *
Disallow: /~joe/stuff/
Alternatively you can explicitly disallow all disallowed pages:
User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html
Now I know this is a little more technical than most of the points we have covered here so far, but it is a great idea to keep on top of these. If you ever want to see a great reason why, search “Google Dorks” and see all of the different ways you can get to pages and downloadable items you shouldn’t be “allowed” to get too. I have seen hundreds of web pages with their downloadable product available because of the lack of a disallow, and all of their paid product open to the public. If you want to know more, it really is worth looking into, and maybe would make a great set of articles in the future. For now though, have a great night, and I will see you tomorrow!!! DL
Traffic, traffic, traffic!!! DL
Dan Loewen