How search engines find, understand
and choose your pages.
An introduction for people who already know SEO.
How search engines find, understand
and choose your pages.
An introduction for people who already know SEO.
Chris Spann · chris-spann.co.uk
Discover → Crawl → Render → Index → Serve
| Question | What we are checking |
|---|---|
| Can Google find it? | Discovery and internal links |
| Can Google fetch and read it? | Crawling and rendering |
| Will Google keep this version? | Indexing and canonical selection |
| Will it show for this search? | Relevance, ranking and search appearance |
A live page is only the start.
Google may know a URL, fetch it, and still decide not to include it in the index.
Indexed pages are not guaranteed to appear for a particular query.
<a href="/guides/choosing-a-tent/">Choosing a tent</a>A URL identifies a resource. Google requests that address and checks what comes back.
GET /guides/choosing-a-tent/ → 200 OK + headers + HTML
| Different parameters mean different URLs |
|---|
| example.com/guides/choosing-a-tent/?utm_source=email |
| example.com/guides/choosing-a-tent/?utm_source=social |
Google can discover and request each URL separately, even if both show the same guide.
“What do I get back when I request this URL?” The response tells Google what is there. Canonicalisation can later group duplicate URLs.
| Code | Meaning | SEO question |
|---|---|---|
| 200 | Request succeeded | Is the expected content actually here? |
| 301 / 308 | Permanent redirect | Is the destination the right replacement? |
| 302 / 307 | Temporary redirect | Is the move really temporary? |
| 404 / 410 | Not found / gone | Should this URL still exist? |
| 403 / 429 | Forbidden / too many requests | Are legitimate crawlers being restricted? |
| 5xx | Server problem | How widespread and persistent is it? |
| Initial HTML | Rendered page |
|---|---|
| What the server sends. | What exists after scripts run. |
| May contain the full article. | May add the article, navigation or product details. |
| View Source shows this. | Developer tools show the current DOM. |
Compare both when important content depends on JavaScript.
The live test does not tell you which canonical Google will choose.
A sitemap helps discovery. It does not force indexing.
Can Google keep up with the pages that matter?
robots.txt controls crawling.
noindex controls indexing.
Neither is a way to secure private information.
User-agent: *
Disallow: /internal-search/
Sitemap: https://example.com/sitemap.xml<meta name="robots" content="noindex">Or in the HTTP response headers:
X-Robots-Tag: noindexGoogle must be allowed to fetch the URL to see the instruction.
| What you want | Starting point |
|---|---|
| Keep private files private | Authentication and access control |
| Keep a public page out of search | noindex, while allowing crawling |
| Keep duplicate pages accessible | Canonical to an equivalent preferred page |
| Move visitors to a replacement URL | Redirect |
| Stop crawling an unwanted URL space | Consider robots.txt; assess indexing separately |
Several URLs.
One preferred version.
Canonicalisation groups duplicate or very similar pages and selects a representative URL.
https://example.com/tents/alpine-2/
https://example.com/tents/alpine-2/?utm_source=emailBoth pages declare:
<link rel="canonical"
href="https://example.com/tents/alpine-2/">The alternate URL remains accessible.
Google can choose a different canonical. Check what it actually selected.
Check the page template, not just one URL.
The browser goes somewhere else.
/old-tent-guide/ → /guides/choosing-a-tent/Use the closest relevant replacement when a page moves or is merged.
| Move | Common codes | Example |
|---|---|---|
| Permanent | 301 / 308 | A guide has a new lasting URL. |
| Temporary | 302 / 307 | A page temporarily sends visitors elsewhere. |
For Google, permanent redirects signal that the destination should become canonical.
Chain: A → B → C
Better: A → C
Loop: A → B → A → …| Situation | Reasonable response |
|---|---|
| A useful equivalent page exists | Redirect to that page. |
| The content is gone with no equivalent | Return 404 or 410. |
| The page is still useful | Keep it live and keep the content honest. |
Redirecting everything to the homepage does not solve the problem.
| Situation | Your decision |
|---|---|
| Same product, email tracking parameter | Canonical, redirect or noindex? |
| Old guide replaced by an updated guide | Canonical, redirect or noindex? |
| Public campaign thank-you page | Canonical, redirect or noindex? |
| Removed service, no equivalent replacement | Redirect, 404 or 410? |
Say what you would check before making the change.
| Element | Job |
|---|---|
| Title element | Describe the page; can inform the search result title. |
| Main heading | Tell readers what this page covers. |
| Meta description | Suggest a summary for the search snippet. |
Check what the template outputs, not just what the CMS field says.
How long until the largest visible piece of content appears?
Example: the header appears, but the main product image takes four seconds to show.
How quickly does the page show a response to a click, tap or key press?
Example: you click a filter, but the page cannot even show that it registered your click.
How much does visible content move unexpectedly?
Example: you are reading a paragraph, then a banner loads above it and moves the text down.
| Metric | What it measures | Good target |
|---|---|---|
| LCP | When the largest visible content element appears | ≤ 2.5 seconds |
| INP | Responsiveness to interaction | ≤ 200 milliseconds |
| CLS | Unexpected layout movement | ≤ 0.1 |
Assess the 75th percentile of real visits, separately for mobile and desktop.
| Source | What it measures | Use it for |
|---|---|---|
| PageSpeed Insights: Lighthouse | A fresh simulated test run on Google’s infrastructure, with controlled device/network settings. | Diagnosing the page now; comparing repeat tests under similar conditions. |
| DevTools: Lighthouse | A lab audit run from Chrome on your computer. Hardware, settings, browser state and extensions can affect it. | Debugging, including local or staging pages, and checking changes before release. |
| CrUX: real-user data | Aggregated measurements from eligible Chrome visits. PageSpeed Insights shows a rolling 28-day window. | Checking the experience Google has recorded for real users. |
PageSpeed Insights contains both CrUX field data and a separate Lighthouse lab test.
CrUX is Google’s real-user performance dataset. A Lighthouse score is a diagnostic test.
Use Lighthouse to investigate and test fixes. Use CrUX to check whether recorded user experience improves.
Machine-readable information about the content on a page.
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Choosing a tent"
}Illustrative fragment, not complete production markup.
| Type | Where it belongs | Possible search appearance |
|---|---|---|
| Product + Offer | Individual product pages; not category grids | Price, availability and eligible product ratings |
| BreadcrumbList | Pages with a breadcrumb trail | Site hierarchy in desktop results |
| Article / NewsArticle / BlogPosting | News stories and individual blog articles | Article titles, images and dates |
| LocalBusiness | Business or individual location pages | Business details such as opening hours |
| VideoObject | Watch pages where video is the main content | Video results and eligible key moments |
| Recipe | Individual recipe pages | Ingredients, cooking time, images and ratings |
| Event | A dedicated page for one public event | Event dates, location and ticket details |
| JobPosting | An individual, current job vacancy | Eligibility for Google’s job search experience |
Markup must match visible content. Each feature has its own eligibility rules.
Valid markup does not guarantee a rich result or higher rankings.
Average position can change because the query mix changed.
Invented case: a new guide has no search traffic.
Status: 200
robots.txt: allowed
Meta robots: noindex
Canonical: points to itself
Sitemap: includes the URLWhat is the strongest lead? What would you check next?
Removing the block restores eligibility. It does not guarantee traffic.
| Include | Example |
|---|---|
| Affected scope | Guide template; 42 URLs in our crawl |
| Observed behaviour | HTML contains noindex |
| Expected behaviour | Published guides should be eligible for indexing |
| Evidence | Example URL, response and timestamp |
| Acceptance check | No noindex in HTML or headers on sampled live guides |
| Ticket section | What the ticket says | Why it helps |
|---|---|---|
| Title | Remove unintended noindex from published buying guides | Names the fault, the affected pages and the change needed. |
| Scope and impact | 42 published /guides/ URLs use the affected template. These pages are intended for organic search. Priority: high, subject to triage. | Gives a bounded scope and a reason to prioritise. Makes no claim about traffic recovered. |
| Evidence | Attach affected-urls.csv and raw-response.txt. Example: https://example.com/guides/choosing-a-tent/. Captured 3 Nov 2026, 10:00 UTC: HTTP 200; HTML robots tag = noindex. | Provides a URL, timestamp and inspectable evidence. Separates an observed block from a theory about its cause. |
| Ticket section | What the ticket says | Why it helps |
|---|---|---|
| Steps to reproduce | 1. Request the example URL without signing in. 2. Inspect the response HTML for meta name="robots". 3. Check X-Robots-Tag in response headers. Repeat with the attached URL list. | A developer can reproduce the issue independently. Checks both places an indexing directive can be sent. |
| Actual behaviour | Published guides return noindex in the initial HTML. The response capture shows the problem before JavaScript runs. | States exactly what happened and where it was observed. |
| Expected behaviour and scope | Published public guides should allow indexing. Remove the unintended directive from this template. Keep drafts, private pages and staging excluded. Investigate where the directive is set. | Defines the required behaviour and its boundaries without guessing which line of code caused it. |
| Ticket section | What the ticket says | Why it helps |
|---|---|---|
| Acceptance criteria | All 42 affected URLs return 200 with no noindex in initial HTML, rendered HTML or X-Robots-Tag. Their existing self-canonicals remain correct. Attach a fresh crawl showing the result. | Gives QA concrete pass/fail checks. The developer can complete the ticket without waiting for Google. |
| Regression checks | A newly published guide also passes. Drafts, private pages and staging still prevent indexing; private content still requires authentication. | Checks the template fix lasts and protects pages that should stay excluded. Noindex is not access control. |
| Release and follow-up | Developer: verify staging, then deploy and purge affected caches. SEO: check production and monitor recrawling/indexing. Revert the change if regression checks fail. | Assigns responsibilities, covers cached responses and separates release verification from later search outcomes. |
Can it be found?
Can it be read?
Is this the right version?
Does it deserve to show?
Then check the evidence.
Google Search Central documentation is linked in the notes for each topic.