Search Optimization Dynamic Web Pages Tested  |
| Dynamic Web Pages Tested |
| Wednesday, 22 September 2004 | ||||||||||||||||||||||||||||||||||
Topic: Dynamic Web Pages — How our test site had a total of 70,000+ dynamic web pages indexed by four crawling search enginesWe recently released the recording of our Dynamic Web Pages clinic. You can listen to a recording of this clinic here: Windows Media Audio: RealMedia: This research brief will answer the following questions:
1. What are dynamic web pages?Dynamic (or database-driven) web pages are created "on the fly" for visitors as they browse a website. This is accomplished with server-side technology such as ASP, ColdFusion, Perl, etc. Dynamic pages are created by the web server when a visitor defines a number of variables. The variables can include such things as product ID, product specs, session ID, language, geographic location, search terms, and so on. Often, these variables are selected by the visitor simply clicking links on the site. Dynamic pages don't actually exist until they are requested by the user. This differs from static HTML pages which exist as individual files on the server. As we will see below, dynamic web pages, while pragmatic for many reasons, do present some difficulty for certain types of search engines. 2. What are the advantages and disadvantages of using dynamic web pages?There are a number of advantages that dynamic web pages provide. They include:
However, these features do come at a price: fewer/lower natural search rankings. Dynamic websites are often not indexed well by search engines that "crawl" or "spider" websites (as apposed to those based only on manual URL submissions). It is difficult to get dynamic websites properly indexed without the right kind of optimization. (We will discuss this in detail below.) Many spiders do not read past the "?" in a URL. (The part of the URL that follows the "?" is called the "query string" and is an indicator that the page is generated by a database.) Spiders don't like these characters because dynamic pages can create an "endless loop" or cause them to get "lost" in the database. One of the biggest problems spiders encounter with dynamic pages is session IDs. If a spider looks at the same page twice, it will see different session IDs and assume it is looking at two different pages. This, more than any other factor, can create an "endless loop" situation. Recently, some search engines (notably Google) have improved their spiders to be able to index SOME dynamic pages. However, dynamic sites still risk not being indexed to their greatest potential without the right kind of optimization. KEY POINT: The more pages a site has, the greater chance it has of appearing in results for relevant search terms. But thousands or even millions of pages won't do you any good if the spiders aren't indexing them. 3. Can you increase the likelihood that dynamic web pages will be indexed by search engines?In order to answer this question, we compared Test Site A, which had undergone a number of key optimization techniques, to Test Site B, which had little or no optimization to its dynamic pages. Here are the results of that comparison:
Why did Test Site A perform so much better in the indexing search engines than Test Site B? Test Site A used a number of optimization techniques for its dynamic pages. Most notably it implemented the rewriting of characters such as "?" as more spider-friendly characters. We will cover this in detail in the section below. (Note that Google indexed Test Site B much better than the other three search engines did. This is due to recent improvements in their spider that allows it to index some dynamic pages.) The data above was generated with this tool, which you may find useful in your own page-optimization efforts. KEY POINT: If you use dynamic web pages but do not optimize for the "crawling" search engines, you will be limited to search traffic from paid-inclusion (PPC and other) programs and the search engines that accept direct submissions. You will lose out on the "deep" indexing that a crawling engine can offer. 4. What are most effective techniques you should keep in mind when optimizing dynamic web pages? (10 techniques)
Using these techniques, you can make a significant impact on the volume of your dynamic web pages that are indexed by crawling search engines. This will result in greater exposure for your site and more traffic from these "free" search engines. For more on the topic of "free" search engines, see our recent report on Natural Search: RELATED MEC REPORTS:Natural Search Engines Tested: http://meclabs.com/cgi-bin/pl/pl.cgi?mns Configurator Tested: http://meclabs.com/cgi-bin/pl/pl.cgi?mc Data Feeds Tested: http://meclabs.com/cgi-bin/pl/pl.cgi?mdf Google PageRank Tested: http://meclabs.com/cgi-bin/pl/pl.cgi?mgp Landing Pages Tested: http://meclabs.com/cgi-bin/pl/pl.cgi?mlp Site Design Tested, Part 1: http://meclabs.com/cgi-bin/pl/pl.cgi?ms1 Site Design Tested, Part 2: http://meclabs.com/cgi-bin/pl/pl.cgi?ms2 Transparent Marketing: 
As part of our research on this topic, we have prepared a review of the best Internet resources on this topic. Rating SystemThese sites were rated for usefulness and clarity, but alas, the rating is purely subjective. * = Decent ** = Good *** = Excellent **** = Indispensable Search Engine Watch Forums – Dynamic Web Site Issues **** Search Engine Saturation Tool **** Dynamic Site SEO Tips and Hints *** Dynamic Pages and Search Engines *** Dynamic Web Page Optimization and Search Engine Inclusion *** Building Dynamic Pages with Search Engines in Mind *** Search Engine Friendly E-Commerce Catalogs *** Masquerading Your CGI/PHP Scripts as Static HTML Pages *** Make Way for the Deep Crawl *** Registering Dynamic Sites *** Solutions for Dynamic Page Registration *** Making Dynamic and E-Commerce Sites Search Engine Friendly *** Optimizing Dynamic Pages – Part I *** Optimizing Dynamic Pages – Part II *** How to Optimize Dynamic Web Sites *** Deep Submit Your Dynamic Pages *** How to Convert Dynamic Pages into Static Pages *** Dynamic Site SEO Tips and Hints *** Getting Millions of Dynamic Pages Indexed *** Making Your Site Search Engine Friendly ** Invite Search Engine Spiders Into Your Dynamic Web Site ** Listing Dynamic Web Pages in Search Engines ** Is Google Indexing Dynamic Pages Now? ** Dynamic Web Pages – How Do They Work? ** 
Credits:
|
||||||||||||||||||||||||||||||||||
What You Need To UNDERSTAND: Test Site A has less than 100,000 pages and Test Site B has well over 1,000,000. Both sites use predominantly dynamic pages, but Test Site A, despite being a much smaller site, still had 336% more total pages indexed by the crawling search engines.
