Search Engine Optimization History

By Christoph Puetz

Webmasters today spend quite some time optimizing their websites for search engines. Books have been written about search engine optimization and some sort of industry has developed to offer search engine optimization services to potential clients. But where did this all start? How did we end up with the SEO world we live in today (from a webmaster standpoint seen)?

A guy named Alan Emtage, a student at the University of McGill, developed the first search engine for the Internet in 1990. This search engine was called "Archie" and was designed to archive documents available on the Internet at that time. About a year later, Gopher, an alternative search engine to Archie, was developed at the University of Minnesota. These two kinda search engines triggered the birth of what we use as search engines today.

In 1993, Matthew Gray developed very first search engine robot - the World Wide Web Wanderer. However, it took until 1994 that search engines as we know them today were born. Lycos, Yahoo! And Galaxy were started and as you probably - two of those are still around today (2005).

In 1994 some companies started experimenting with the concept of search engine optimization. The emphasis was put solely on the submission process at that time. Within 12 months, the first automated submission software packages were released. Of course it did not take long until the concept of spamming search engines was 'invented'. Some webmasters quickly realized that they could swamp and manipulate search results pages by over-submission of their sites. However - the search engines soon fought back and changed things to prevent this from happen.

Soon, search engine optimizers and the search engines started playing some sort of a "cat and mouse" game. Once a way to manipulate a search engine was discovered by the SE-optimizers they took advantage of this. The search engines subsequently revised and enhanced their ranking algorithms to respond to these strategies. It was clear very soon that mainly a small group of webmasters was abusing the search engine algorithms to gain advantage over the competition. Black Hat search engine optimization was born. The unethical way of manipulating search engine resulted in faster responses from search engines. Search engines are trying to keep the search results clean of SPAM to provide the best service to customers.

The search engine industry quickly realized that SEO (Search Engine Optimization) as an industry would not go away, and in order to maintain useful indexes, they would need to at least accept the industry. Search engines now partially work with the SEO industry but are still very eager to sort out SPAMMERS that are trying to manipulate the results.

When Google.com started to be the search engine of choice for more than 50 f the Internet users it was highly visible to anyone in the industry that search engine spamming had reached a new dimension. Google.com was so much more important to the success of a website that many webmasters solely concentrated on optimizing their sites for Google only as the payoff was worth the efforts. Again - Black Hat SEO took place, pushing down the honest webmaster and their sites in search results delivered. Google started fighting back. Several major updates to Google's algorithms forced all webmaster to adapt to new strategies. Black Hat SE-optimizers but suddenly saw something different happening. Instead of just being pushed down in the search results their websites were suddenly completely removed from the search index.

And then there was something called the "Google Sandbox" to show up in discussions. Websites either disappeared into the sandbox or new websites never made it into the index and were considered in the Google Sandbox. The sandbox seemed to be the place where Google would 'park' websites either considered SPAMMY or not to be conform with Google's policies (duplicate websites under different domain names, etc.). The Google Sandbox so far has not been confirmed or denied by Google and many webmasters consider it to be myth.

In late 2004 Google announced to have 8 billion pages/sites in the search index. The gap between Google and the next two competitors (MSN and Yahoo!) seemed to grow. However - in 2005 MSN as well as Yahoo! Started fighting back putting life back into the search engine war. MSN and Yahoo seemed to gain ground in delivering better and cleaner results compared to Google. In July of 2005 Yahoo! Announced to have over 20 billion pages/sites in the search index - leaving Google far behind. No one search engine has won the war yet. The three major search engines however are eagerly fighting for market share and one mistake could change the fortune of a search engine. It will be a rocky ride - but worth watching from the sidelines.