Duplicate content: how to spot and avoid it

One of the most important prin­ciples of search engine op­tim­isa­tion is unique content. This ensures a better ranking of a website among search results and makes for a more positive user ex­per­i­ence. It’s the basis for a suc­cess­ful content strategy.

The opposite of unique content is known as duplicate content. The term describes text blocks or entire web pages that are du­plic­ated across multiple URLs. Avoiding this type of content in favour of unique content is important for suc­cess­ful search engine op­tim­isa­tion. Du­plic­ated content neg­at­ively affects website ranking and usability.

The term duplicate content refers to web pages or text passages that are du­plic­ated across more than one URL.

A dis­tinc­tion is generally made between two types:

• Internal duplicate content refers to content du­plic­ated across a single domain.
• External duplicate content is found across domains.

Both terms refer to pages or text blocks that are shared without modi­fic­a­tion. Where content isn’t 100 percent identical but almost identical, it’s referred to as near duplicate content.

Most website owners are aware of the negative effects of duplicate content and therefore avoid it. However, internal duplicate content is a lot more common. Often this has technical causes. You can find more in­form­a­tion on this in the section ‘Technical causes for duplicate content’.

Often, the same content can be found several times across multiple sub-pages of a domain. Online shops in par­tic­u­lar struggle with this issue. When a product is assigned to several cat­egor­ies or is available in different vari­ations, the product de­scrip­tion is often largely the same across multiple pages. This is counted as duplicate content. PDFs con­tain­ing product in­form­a­tion are commonly un­der­es­tim­ated. If their content matches that of a product landing page, they are counted as duplicate content. Another example is a company’s philo­sophy statement that may appear across several sub-pages.

Online shops are also often affected by instances of external duplicate content. When you purchase your products from a whole­saler, other retailers may be using the same product de­scrip­tions on their websites. In this case, identical content not only neg­at­ively affects the search engine ranking, but also the pur­chas­ing decision of potential customers. If a product present­a­tion doesn’t differ between dealers, it’s the price that decides. It’s best to use unique product de­scrip­tions and regularly check whether other retailers have copied them.

External duplicate content is also created when you copy third-party content even as part of a co­oper­a­tion and with the consent of the author. Different country and language versions of your website across different domain names also pose a risk of external duplicate content.

Search engines like Google use an algorithm to evaluate all po­ten­tially relevant websites and use certain criteria to determine the order in which the search results are displayed. The aim of this eval­u­ation is to present the user with the most relevant content at the top of the ranking.

If the same content appears across several websites, search engines fail to make a proper eval­u­ation. This com­plic­ates the as­sign­ment of trust, relevance, and authority and, as a result, also the creation of a ranking. Search engines therefore generally avoid indexing the same content multiple times and dis­play­ing it in the search results, as this does not offer any added value for users. Therefore, the ranking of pages that contain duplicate content will decrease.

Due to the negative effects of duplicate content, so-called web scrapers are feared. This type of software copies websites one-to-one. A search engine like Google can now recognise scraper sites based on various para­met­ers and dis­tin­guishes them from ‘true’ websites.

Around 25 to 35 percent of the content of all websites is duplicate content. Duplicate content isn’t ne­ces­sar­ily always a bad thing. This video explains how Google deals with duplicate content and what to avoid in any case:

It’s advisable to check your website regularly for duplicate content. In many cases, it happens without the website operator’s knowledge, for example when new pages are created or when internal links are in­con­sist­ent. Below, we’ll show you the most common methods you can use to track down duplicate content.

If your website consists of a limited number of sub-pages, it’s a good idea to check them manually. Pay par­tic­u­lar attention to text sections that you use several times across your website. Typically, these are company state­ments/present­a­tions or a call-to-action.

Suspect a text module to occur more than once? In this case, a Google search can help. Enter the text in quotation marks in the search box and see if different URLs for your website appear in the search results.

In the case of large numbers of sub-pages or in eCommerce, manual searches are time-intensive. Because many website operators face this problem, various tools are available that auto­mat­ic­ally search for duplicate content.

The Google Search Console is a free and useful analysis tool that supports search engine op­tim­isa­tion and the search for internal duplicate content. It finds the dominant keywords under which your website is found and viewed. Then you can filter for exact inquiries. If the tool lists multiple subpages, you check them for duplicate content. In the ‘Index Coverage’ report, under ‘Excluded Pages’, you can view the subpages that have been iden­ti­fied as du­plic­ates and con­sequently excluded.

Now you know how to spot duplicate content. But it’s best to avoid it in the first place. These tips should help you:

• Create sub-pages that are clearly dis­tin­guish­able them­at­ic­ally and use different main keywords across each page. The best way to keep track of things is to work out a keyword strategy be­fore­hand.
• Avoid placing gen­er­al­ised para­graphs across multiple subpages.
• Avoid copying pre-written texts (unless it’s a des­ig­nated quotation or legal quote).
• Pay attention to the con­sist­ency of internal links and avoid different entries for the same URL. These are typically created by adding index.htm to the homepage URL or through variants with or without the trailing slash (/).
• Use top level domains across multiple language and country versions of your website such as `https://www.example.com´ as opposed to sub-domains like `https://www.example.org/en´.

Generally, you cannot influence the creation of external duplicate content if other web­mas­ters copy your content. Therefore, it is advisable to initiate crawling manually after creating a new page. If your website is indexed with the re­spect­ive content as the first page, it gets clas­si­fied as the original.

Technical causes often lead to internal duplicate content being created without the knowledge of the website operator. It is advisable to check your online presence for the following points:

When you switch to encrypted HTTPS (Hypertext Transfer Protocol Secure), it’s important to set up for­ward­ing from your old web address. If your old website remains ac­cess­ible via http://, 100 percent con­gru­ently duplicate content is created!

Make sure you check whether your website can be accessed using different spellings. Typical examples include:

• Your homepage is ac­cess­ible via index.php as well as with and without trailing slash.
• Your website is ac­cess­ible with and without www as part of the URL.
• Your URL is case-sensitive.

If you spot two or more URLs that lead to the same subpage, set up a 301 redirect to the main page for all variants.

• If your website has different language or country versions, mark them with hreflang to avoid duplicate content.
• Check your URL para­met­ers. This often creates many unique URLs that lead to the same content. This is a common cause of duplicate content, es­pe­cially with filter functions in online shops.
• Pay attention to session IDs that form part of a URL. As a result, crawlers may receive a new ID each time a subpage is accessed and thus reach a new URL.
• Exclude printer-friendly versions of websites from indexing.

Ma­nip­u­lat­ive and in­ten­tion­al copying of third-party content is pro­hib­ited for copyright reasons and according to Google guidelines. However, the oc­cur­rence of duplicate content online is normal and is not directly punished. Nev­er­the­less, it’s advisable to avoid duplicate content whenever possible.

Where a website contains two or more sub-pages with similar or identical content, you can merge the content to a single page or expand the re­spect­ive sub-pages by adding unique content and in­di­vidu­al keywords to optimise their SEO content. The option that’s best for you depends on the relevance of the pages and whether there is an op­por­tun­ity to strengthen keywords.

Choose stronger keywords for headings, your meta de­scrip­tion, and the meta title. This avoids them being clas­si­fied as duplicate content and improves your ranking across search engines. To modify existing texts and dif­fer­en­ti­ate them, use bullet points, lists, and tables, or integrate media such as images and videos.

If you want to share existing content, for example, a blog article or a press release on other sites, use this canonical tag rel="canonical" in the header. By doing so, you are declaring the selected URL as the standard resource or original URL. The tag is invisible to users and still makes it clear to search engines how the pages are linked to one another.