Case Study – Improper Indexing of Product Pages

Background

A local home décor business with an ecommerce website built with Magento signed with Boomtown hoping to improve their presence in Google’s search results. They had previously been with another SEO agency, but after several consecutive months of declining traffic decided to move on.

Organic Search Traffic Steadily Declining Before Working with Boomtown
Organic Search Traffic Steadily Declining Before Working with Boomtown

Within Boomtown’s first month (April 2015) of working with the client, we discovered a major technical issue with the website and how search engines were indexing it.

Problem

The previous SEO agency had submitted a Robots.txt file requesting search engines not to index any URL containing /index.php/

Many sites require there to be /index.php/ URLs in addition to the clean URLs and it is common to block search engines with Robots.txt

However, in addition to the Robots.txt file, the previous SEO agency had also added canonical tags on every clean URL specifying that the URL with /index.php/ was the preferred version.

Canonical tags are used when a website has 2 identical, or very similar, webpages, which is common with ecommerce sites. The tag tells the search engine which of the 2 pages is the preferred page to show up in search results.

Essentially, the website was telling search engines that they preferred the /index.php/ URLs instead of the clean URLs, but at the same time were telling search engines NOT to index those /index.php/ URLs.

Consequences of the Problem

Most of the product and product category pages were not being indexed by the search engines. In fact, Google had only 14 pages indexed for the entire site even though the site actually had over 1,200 pages. This resulted in the vast majority of their webpages being left out of search results, meaning that they received zero visits from Google searches. So, if you were to conduct a Google search for a very specific product, while you were located in the business’s hometown, their product page would not be listed in the results. The only page that was ever included in search results was the homepage.

Solution

Boomtown changed the canonical tags so that the clean URL was the preffered version, rather than the /index.php/ version. For example, we set http://www.example.com/products/ as the preferred version instead of http://www.example.com/index.php/products which was not being indexed due to the Robots.txt file.

Results

The next time search engines crawled the site they indexed all of the product and product category pages. Soon, these pages began to show up in search results and receive traffic.

Organic traffic increased 80.11% in just the first 3 months, from 362 visits in March up to 652 visits in June.
Organic traffic increased 80.11% in just the first 3 months, from 362 visits in March up to 652 visits in June.

If you’re finding that your website is not showing up in search results, please contact us and we’ll see if we can fix the problem and get your business back on track.