How search engines see your site

ProcessCrawl, render, index, rank
  1. 1

    Discover

    Search engines find URLs through links and sitemaps.

  2. 2

    Crawl

    A bot fetches the page, if robots.txt allows.

  3. 3

    Render

    The page is processed, including JavaScript.

  4. 4

    Index

    Eligible, useful pages are stored.

  5. 5

    Rank

    Indexed pages are ordered for each query.

A problem at any step stops the pages that follow. That is why the checklist begins at the start of the chain.

1. Crawling and indexing

ChecklistCrawl and index checks
  • robots.txt exists and does not block important pages or resources
  • An XML sitemap lists all indexable URLs and is submitted in Search Console
  • Important pages are not set to noindex by mistake (check templates and staging leftovers)
  • Each page has a self-referencing canonical tag pointing to its preferred URL
  • Duplicate versions (http/https, www/non-www, trailing slashes) redirect to one
  • No important pages are orphaned — each is linked from somewhere
  • Search Console shows no large “Excluded” or error buckets you did not expect
  • Logical hierarchy: home → category/service → detail pages, ideally within three clicks of the homepage.
  • Clean, descriptive URLs: short, lowercase, hyphenated, no unnecessary parameters.
  • Internal links with descriptive anchor text: link related pages together; it spreads authority and helps users.
  • Breadcrumbs: help users and give search engines the hierarchy.
  • No broken links or long redirect chains: fix 404s and collapse multi-step redirects.

3. Speed and mobile

Google primarily uses the mobile version of your site for indexing, so the mobile experience is the experience. Check that content and links on mobile match desktop, tap targets are usable and pages load quickly. Our guide to Core Web Vitals covers the loading, responsiveness and stability metrics and their fixes in detail.

ChecklistMobile and speed checks
  • Responsive layout with the viewport meta tag set
  • Text readable without zooming; tap targets not crammed together
  • Images compressed, correctly sized and in modern formats
  • Largest content element loads quickly (LCP)
  • No intrusive pop-ups blocking content
  • Third-party scripts audited and deferred where possible

4. On-page technical elements

Elements every indexable page should have
ElementGood practice
Title tagUnique, descriptive, roughly 50–60 characters, key topic first
Meta descriptionUnique summary, roughly 150–160 characters, written for clicks
One H1Clear page topic; sensible H2/H3 hierarchy below
Image alt textDescribes the image; useful for accessibility and image search
Canonical tagPoints to the preferred URL
Open Graph tagsControls how links look when shared on social media

5. Structured data

Structured data (schema.org markup, usually JSON-LD) tells search engines what a page is about in a machine-readable way and can make you eligible for rich results. Useful types for businesses include Organization, LocalBusiness, Article/BlogPosting, FAQPage, BreadcrumbList and Product. Mark up only what is actually visible on the page, and validate it with Google’s Rich Results Test.

6. Security and hygiene

  • HTTPS everywhere, with valid certificates and no mixed content.
  • Redirect old URLs when you restructure — preserve rankings with 301 redirects.
  • Return real 404 status codes for missing pages (not “soft 404s” with a 200 status).
  • Keep software and plugins updated; hacked sites lose rankings fast.
  • Handle pagination and filters carefully to avoid thousands of near-duplicate URLs.

7. JavaScript sites and rendering

Modern sites built with JavaScript frameworks can be indexed, but it adds risk: content that only appears after scripts run may be crawled later or missed by other bots and social previews. For important pages, make sure the key content, links and metadata are present in the initial HTML — through server-side rendering or pre-rendering — and test with the URL Inspection tool.

How to run the audit

ProcessA simple audit routine
  1. 1

    Search Console

    Review indexing, sitemaps, Core Web Vitals and manual actions.

  2. 2

    Crawl the site

    Use a crawler to find broken links, redirects, missing tags and duplicates.

  3. 3

    Test key templates

    Check home, service, article and product pages on mobile.

  4. 4

    Prioritise fixes

    Indexing blockers first, then speed, then polish.

  5. 5

    Re-check monthly

    Technical SEO is maintenance, not a one-off project.

Technical SEO is far cheaper to build in than to retrofit — it is one reason we scope it into every project, as described in our website cost guide. If you are hiring help, our checklist for choosing a development partner applies here too.