It names a topic, not a behavior. No URL, no intended state, no way to tell when the work is done. This engineering note is about how to do a technical SEO audit so the output is testable: sample by template, exclude private routes before you start, and turn each finding into a ticket that separates what was observed, what is suspected and what was tested.

Disclosure: this post comes from GuaranteedSEOServices.com, an SEO services company. It adapts our guide How to Do a Technical SEO Audit: A Step-by-Step Process, which is set as the canonical source.

Context: how to do a technical SEO audit on one template (hypothetical)

A practitioner has a crawl export for an online store and one week to investigate the category template. The export also contains account screens, internal search results and tracking variants. The engineers who will receive the tickets want each one to be reproducible and closable.

Tool output alone cannot establish indexing, the selected canonical, root cause or business impact. It shows where to look.

Option A: work from the full crawl export

Group warnings by type and file one ticket per warning type across all URLs. Fast to produce, and nothing in the export is ignored.

Option B: sample by template, with private routes excluded up front

Before reading warnings, write a page-purpose matrix: for each route type, state whether it should be discoverable, crawlable, indexable and in the sitemap. Exclude account screens, internal search results and other utility routes from the public landing-page fix queue. Audit their access controls, crawling directives and sitemap treatment separately against their intended purpose. Then sample representative URLs from the target template and verify findings manually before filing.

Decision

Option B. The deciding constraint is the private and utility routes in the export. Without explicit exclusions, a warning such as "page has noindex" turns into a ticket to index account screens. The matrix makes those exceptions deliberate instead of accidental.

When Option A becomes reasonable: when the crawl is already limited to intended public landing pages, for example a small site whose crawl configuration excludes private routes. Then the export and the scope match, and grouping by warning is a sensible first pass.

The sampling record

For each sampled URL on the category template, record:

Keeping evidence, hypothesis and validation in separate columns is what lets an engineer trust the ticket.

Eight ticket fields

1. Observation: reproducible behavior and test date
2. Scope: example URLs and affected templates
3. Intended behavior: the specific page or system outcome
4. Proposed change: a bounded implementation recommendation
5. Dependencies: teams, access and related releases
6. Acceptance: checks that establish correct delivery
7. Monitoring: search and customer signals to follow afterward
8. Rollback: trigger and accountable owner

Worked example (hypothetical)

Sampling shows that a new category is missing from its parent hub. The ticket:

• Observation: the new category page returns its content, but no link to it appears on the parent hub on desktop or mobile.
• Scope: the category template's parent-hub link list.
• Intended behavior: each active category has a visible, descriptive link from its parent hub.
• Proposed change: generate the hub's link list from active categories in the template; exclude discontinued categories.
• Dependencies: the category data must mark discontinued items reliably.
• Acceptance: the link appears on desktop and mobile, points to the correct destination, and discontinued categories are absent.
• Monitoring: indexing of the new category in Search Console, tracked separately.
• Rollback: the release owner reverts if the hub template breaks navigation for existing categories.

Acceptance vs. monitoring

Acceptance is delivery. The ticket closes when the acceptance checks pass on the deployed site. Indexing is a later outcome, monitored separately. A live URL Inspection test that shows corrected HTML establishes current delivery; it does not prove Google has processed the change or selected the page. Keep those in separate columns so a ticket isn't held open waiting on search processing.

A canonical defect repeated across thousands of URLs may be one engineering task. File it once, with the template as scope, not once per URL.

Gotchas and testing approach

• Template changes reach unaffected pages. The hub change touches every category. Retest a sample of affected and unaffected pages after deployment, not only the one that triggered the ticket.
• Staging artifacts in production. A noindex or access rule used on staging can ship with the release. Check directives on production directly after deployment.
• Structured data drifting from source data. Review markup against visible content and confirm that it follows the underlying product or business data. A hand-corrected snippet that goes stale next week is not a durable fix.

Edge cases

• Blocked and noindexed at once. A robots.txt block can stop Google from seeing a page's directives. Google's guidance: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file" (block indexing). The matrix should say which control applies to which route type.
• Missing pages that return success. Include a deliberately missing URL in every sample. An empty template with a success status is a delivery defect, separate from content quality.

Template

Observation:
Scope:
Intended behavior:
Proposed change:
Dependencies:
Acceptance:
Monitoring:
Rollback:

Your turn

How would you adapt this workflow if you had no access to Search Console and only a crawl export plus server logs? Share the ticket or sampling design you would use under that constraint in the comments. The full step-by-step audit process is in the original guide.

The ticket looked ready: high crawler severity, one template, a clear spec. What it lacked was a dependency line saying the source data had to be corrected first. This decision record shows how to prioritize technical SEO fixes when the sprint has room for one template-level change, and how to write that change as a ticket with dependencies and a regression boundary.

Disclosure: this post comes from GuaranteedSEOServices.com, an SEO services company. It adapts our guide How to Prioritize Technical SEO Fixes When Developer Time Is Limited, which is set as the canonical source.

Context (hypothetical)

A crawler report lists many warnings. The developer team has capacity for one template-level change this sprint. After consolidating the audit backlog, three candidate findings remain. They are not equivalent crawler warnings, just the items the team could plausibly ship:

1. Generate product structured data on the product template. Highest warning count.
2. Fix canonical output on the category template. Several active category landing pages point their canonical at a filtered variant.
3. Implement redirects for a planned URL restructure of the blog.

Severity labels and issue counts are tool output, not validated business impact estimates. They tell you where to look, not what to ship.

Option A: ship the item with the most warnings

Product structured data has the biggest count, so it goes first. The appeal is obvious: one change, a large drop in reported issues.

Option B: ship the item whose dependencies are met

Check each candidate for prerequisites before ranking it. Ship the highest-value change that can actually be completed and verified this sprint. File the others as blocked, with the blocker named.

Decision

Option B: fix the category canonical. In this hypothetical, the category pages are the entry point for the main product-browsing journey, and the canonical defect was reproduced on representative category URLs. The deciding constraint is a dependency. In this hypothetical, the availability field in the product feed is out of sync with what the product page shows. Generating markup from that feed would publish the wrong values in a machine-readable format. Google's guidance is that structured data should describe the content of the page and not information that isn't visible to users (structured data introduction).

When Option A becomes reasonable: once the source data is corrected and a check confirms that the feed matches the visible page. At that point the structured data ticket is unblocked and competes on value like any other.

How to prioritize technical SEO fixes by dependency

Three dependencies from the source guide, each phrased as a ticket rule:

• Dependency 1: source data before generated output. Correct the data before generating structured data from it. The ticket names the data owner and the check that proves the data is right.
• Dependency 2: URL structure before redirect rules. Settle the intended URL structure before writing redirects. In the hypothetical, the blog restructure is still under discussion, so candidate 3 waits. Redirects written against a moving target get rewritten.
• Dependency 3: page purpose before indexing controls. Confirm what a page type is for before changing noindex or canonical behavior. For candidate 2, the team confirms that the clean category URLs are the intended landing pages before touching the template.

Bundling vs. coupling

Bundle related changes only when it reduces risk. Fixing the canonical tag and the internal links that point to the filtered variant in one release is a reasonable bundle: both serve one intended state, and testing them together is simpler.

Coupling independent work is different. Adding the blog redirects to the same release because "we're touching URLs anyway" means one failure can block or roll back an unrelated fix.

The ticket

Testing approach and regression boundary

Effort includes review and testing, not only the lines changed. A one-line template change that affects every category type may need more validation than a larger isolated component.

The regression boundary lists what must not change:

• Routes intentionally excluded, such as account screens with an intended noindex, keep their current directives.
• Filtered URLs that are deliberately canonicalized to their parent keep that behavior.
• Product and blog templates show no diff in canonical output.

Write these as checks before development starts. After release, the developer who shipped the change checks the three clean category URLs, the designated filtered variants and the product and blog template output, records the result on the ticket, and the SEO owner confirms it. Record the release date so later search observations can be compared with the change, keeping in mind that timing alone does not prove cause.

Rollback: the release owner rolls back if a check on the boundary fails in production. Review: at the next planning cycle, confirm what passed acceptance and whether the structured data blocker has cleared.

Edge cases

• Similar symptoms, different causes. Missing content on two templates may come from a render failure on one and unpublished source data on the other. Group tickets only when the evidence supports a shared mechanism.
• The blocker never clears. If the feed owner cannot fix the data this quarter, keep the structured data ticket blocked with that reason visible rather than shipping markup that disagrees with the page.

Your turn

How would you sequence these three candidates if the canonical fix needed a change in a shared component used by every template? Share the ticket you would write under that constraint in the comments. The full prioritization method, including the decision table and priority bands, is in the original guide.

That line is hypothetical, but it illustrates one failure mode in a replatform: not the rules you wrote, but the routes nobody listed. If your team is working out how to do an SEO migration for a CMS or framework change, this decision record covers the part engineers own: what to monitor after launch, and which conditions trigger a rollback instead of a fix-forward.

Disclosure: this post comes from GuaranteedSEOServices.com, an SEO services company. It adapts our guide SEO Site Migration: A Practical Plan for Changing a Website, which is set as the canonical source.

Context: a hypothetical relaunch

A company moves from an old CMS to a new framework. URL patterns change from /products.php?id=123 to /products/blue-widget/. At the planning review, the redirect map covers product and category pages, analytics tags are not yet migrated, and nobody has written down what would cause a rollback.

Constraints:

1. The old platform can stay online for a limited time after launch.
2. Old URLs are linked from places the team does not control: partner sites, emails, bookmarks.
3. A plan reduces avoidable errors. It cannot guarantee unchanged rankings or traffic, so monitoring must separate "our mistake" from normal processing.

Option A: one launch that changes everything

New platform, new URL patterns and new design in the same release. One cutover, one freeze, one round of testing.

Option B: staged changes with written rollback conditions

Ship the URL and platform change first with the existing layout and content as close to unchanged as practical. Ship the redesign in a later release. Google's site-move guidance recommends changing one thing at a time, with the example of moving the domain first and changing the layout later (site moves with URL changes).

Decision

Option B. When something breaks after launch, the team needs to narrow down the cause. Separating major changes reduces the number of possible causes. Comparing affected and unaffected templates can then help narrow the investigation.

When Option A becomes reasonable: when the constraint that the old platform can stay online disappears. If the old system must be shut down on a fixed date and cannot hold the old layout on new routes, a combined launch may be the only option. In that case, tighten the launch gates below and treat the first weeks as an incident watch.

Six gotchas to check before launch

1. Linked assets. Inventory images, PDFs and other downloads, not only HTML pages. Google says to include URLs of embedded content such as images, videos, JavaScript and CSS in the move plan (same source).
2. Failure-case routes. Test missing routes and invalid product identifiers, not only happy paths. An unknown ID on the new platform should return a proper missing-resource response, not a blank template with a success status.
3. Alternate hostnames. Check www and non-www, http and https, and any historical entry routes in scope. Each should reach the final destination without avoidable intermediate hops.
4. Staging noindex shipped to production. Protect staging with access controls and configure production indexing separately. Google's troubleshooting advice includes removing noindex or robots.txt blocks that were only needed for the migration.
5. Simultaneous changes. This is the decision above. If the redesign creeps into the platform release, write that down as a scope change.
6. Removing redirects early. Google recommends keeping redirects "for as long as possible, generally at least 1 year". A stable report after a few weeks is not a reason to delete them.

Working record per URL group

How to do an SEO migration: verification approach

• Before launch: test a representative sample of old URLs per template against the planned routing, including the failure cases and hostnames above. Agree which defects block launch, for example a broken inquiry form or an inaccessible category template.
• At launch: check production directly. Confirm critical templates return the expected responses and that staging-only restrictions are gone.
• After launch: monitor old and new URLs side by side. Annotate the release date. Expect temporary fluctuation, which Google's guide also describes, and investigate by comparing affected templates with unaffected ones before blaming the move itself.

Rollback and review conditions

Write these before launch and name who decides:

• Rollback (release owner decides, with the SEO owner consulted): a critical template is inaccessible or returns errors in production, staging restrictions reached public templates and cannot be corrected quickly, or the redirect layer fails for a whole template group.
• Fix-forward (developers, tracked as tickets): individual mapping errors, missing assets found after launch, a chain on a low-traffic route.
• Review (SEO and analytics owners): a template group shows unexplained errors, indexing problems or measurement discrepancies in repeated post-launch checks. Compare against the unaffected templates and check access, mapping relevance and measurement first.

Before cutover, verify whether the old release, route configuration and required data state can actually be restored. If reversal is unavailable, document that limitation and define a fix-forward escalation path instead. Rollback gets more expensive with every day the new URLs are live and linked, so decide the rollback window explicitly as well.

Edge cases

• Analytics not migrated at launch: without measurement on the new platform you cannot tell a tracking gap from a traffic drop. Treat working analytics as a launch gate.
• A large site moving in sections: Google suggests that large sites can move one section first to test the effects, while noting that a single section may not represent the whole site. Use the section move to validate your monitoring, not to predict the outcome.

Your turn

How would you set rollback conditions if the old platform has to be switched off on launch day? Share the design you would use under that constraint in the comments. The full migration plan, including inventory and mapping decisions, is in the original guide.

What is technical SEO, seen from a release pipeline? A workable answer: the systems that decide whether a public page can be discovered, fetched, rendered and considered for indexing. Those systems change in ordinary releases: a router refactor, a header middleware, a config copied from staging. None of those tickets mention search.

This is a decision record for adding search checks to a release without turning every deploy into an SEO review.

Disclosure: this post comes from GuaranteedSEOServices.com, an SEO services company. It adapts our guide What Is Technical SEO? Core Areas and Responsibilities, which is set as the canonical source.

Context: what is technical SEO in a release pipeline?

Search-relevant pages usually come from a few templates: home, service or category, detail, article. Three constraints shape the design:

1. Pipeline time is a shared budget. A check that makes every deploy noticeably slower will get bypassed.
2. Search failures are often silent. The page still loads for the developer who tests it.
3. Some inputs are owned outside engineering. Redirect maps, product data and structured data fields change without a code review.

Option A: a blocking full-site crawl on every deploy

Crawl every URL and fail the build on any new issue. Thorough, but runtime grows with URL count, and a changed meta description fails as loudly as a site-wide noindex. Teams learn to override noisy gates.

Option B: template sampling with a small blocking set

Check a fixed sample of representative URLs per template. Block the release only when a small set of critical access, indexing-directive, rendering or discovery assertions fails. Report everything else without blocking.

Decision

Option B. The deciding constraint is the time budget: if you deploy often, the check must be fast enough that nobody wants to skip it.

Option A becomes reasonable when a full crawl fits within that budget, or when it runs on a schedule outside the deploy path.

Testing approach: sample by template

Per template, sample one long-standing important URL, one recently published URL and one URL with an edge condition (parameters, pagination or a localized version). Add the homepage.

Blocking assertions for each sampled URL:

• Response: the intended status for a live page, not an error or an unexpected redirect.
• Indexing directive: no unexpected noindex in the server's HTML or the X-Robots-Tag header. Check the server response, not only the rendered DOM: Google says using JavaScript to remove a noindex "may not work as expected" (JavaScript SEO basics).
• robots.txt: the production file does not disallow public templates.
• Rendered content: the main heading and primary text exist after rendering.
• Discovery route: the URL is linked from its expected parent with a normal link. Google states it "can only discover your links if they are <a> HTML elements with an href attribute" (same source).

Non-blocking report: titles, descriptions, structured data warnings, performance changes.

Gotcha 1: staging rules leaking to production

Staging environments sometimes use noindex or robots.txt restrictions to manage search engine behavior. Neither is access control. If those settings leak into production, public pages may become blocked from crawling or indexing. Keep the staging access decision separate from the production indexing decision. Protect staging with authentication: Google says robots.txt "is not a mechanism for keeping a web page out of Google" and points to noindex or password protection instead (robots.txt introduction). Then assert the production values explicitly.

Gotcha 2: redirect maps that rot after launch

A migration redirect map goes stale as URLs keep changing. Treat it as maintained product data with an owner and a validation step: validate each entry against its intended destination, flag unexpected chains, loops and broken targets, and require a justified exception for an intentional multi-hop path. Make that blocking for releases that touch routing and a report for everything else.

Gotcha 3: structured data drifting from the source of truth

Hand-maintained markup diverges when the commerce or CMS data changes. Google's guidance is that structured data describes the page's content and should not cover information that isn't visible to the user (structured data introduction). Prefer generating markup from the same data that renders the page. Otherwise, compare key marked-up values with the rendered values and report mismatches.

Edge case 1: the page loads but has no route in (hypothetical)

Hypothetical: a team ships a new service page. It returns the intended status, has no noindex and renders fully. Because it is new, it is not in the fixed sample, so the release passes. Nothing on the homepage or service hub links to it. A sitemap can help discovery, but it does not replace a route through the site.

Fix the process, not just the page: any release that adds a public URL adds it to the sample for that release and names the parent page that links to it, so the discovery-route assertion has something to check.

Keep the boundary clear: passing these checks shows that the tested access, rendering and discovery conditions were met. It does not establish indexing eligibility in every respect, actual indexing or selection in search.

Edge case 2: an intentional noindex

A release that deliberately noindexes a template will fail the blocking check. Use an allowlist entry with an owner and a reason, approved by the SEO owner, so intended changes pass and accidental ones do not.

Ownership and rollback

• SEO owner: defines the sample and the intended state per template; reviews non-blocking warnings; approves any intentional indexing change.
• Developers: maintain the checks as regression tests and approve changes to the pre-release checks with the SEO owner.
• Release or DevOps owner: holds rollback. Decide before launch who can roll back a release that unintentionally blocks public pages.
• Business owner: decides which templates justify a blocking check.

Rollback trigger: a blocking assertion fails unexpectedly in the post-deploy check on production. The release owner assesses the affected public URLs and restores the intended state through rollback or a verified hotfix, then records the incident and investigates the cause.

Review trigger: revisit this record when a regression reaches production that the sample missed (add that URL pattern), or when a full crawl fits inside the pipeline budget.

Acceptance record (one per sampled or new URL)

• URL:
• Discovery route: parent page and link that must exist
• Response: expected status
• Rendered content: heading and main text that must appear
• Indexing directive: expected value in HTML and header
• Owner: who fixes it if it fails
• Acceptance test: the assertion that closes the ticket

Checklist

• Sample list per template, with an owner; new public URLs added per release
• Blocking assertions and non-blocking report defined
• Staging protected by authentication; production indexing values asserted explicitly
• Redirect map with an owner and a validation step
• Allowlist for intentional noindex, with a reason
• Named rollback owner

Your turn

How would you design this if you deploy dozens of times a day, or have no staging environment at all? Share the checks you would use under that constraint in the comments. The underlying SEO requirements are in the original guide.

Open a page built with a JavaScript framework. The article appears, the navigation works, and a calculator responds to your input.

Now fetch the same URL without executing JavaScript.

Do you receive the article—or just the container where the article will eventually appear?

That distinction matters whenever software needs to retrieve your content. Search engines, AI retrieval systems, link preview services, and browsers do not necessarily process a page in the same way.

Prerendering addresses part of that problem: it generates HTML before the visitor requests the page, so the response can already contain the information someone came to read.

A JavaScript framework does not have to send an empty page

A client-rendered application might initially return something like this:

<!doctype html>
<html lang="en">
<head>
  <title>Installation guide</title>
</head>
<body>
  <div id="root"></div>
  <script src="/assets/app.js"></script>
</body>
</html>

The browser downloads the script, executes it, requests the article data, and builds the visible document.

A prerendered response can contain the document immediately:

<!doctype html>
<html lang="en">
<head>
  <title>Installation guide</title>
</head>
<body>
  <main>
    <article>
      <h1>Installation guide</h1>
      <p>Install the client before configuring authentication.</p>
      <pre><code>npm install example-client</code></pre>
      <a href="/docs/authentication">
        Configure authentication
      </a>
    </article>
  </main>
</body>
</html>

JavaScript can still add interaction afterward. A copy button, search interface, or interactive example does not require the article itself to be absent from the response.

The important architectural decision is when and where the main content becomes HTML.

Prerendering, server rendering, and client rendering

These terms are sometimes used loosely. For this article, prerendering means generating HTML ahead of a request, usually during a build.

Frameworks can combine these approaches across routes and components.

Astro, for example, prerenders routes by default and supports request-time rendering where needed. Next.js also supports delivering server-generated HTML; its initial page load is not inherently an empty React container. See the official Astro rendering guide and Next.js rendering explanation.

A framework name alone does not tell you what your deployed URL returns.

Why this matters for SEO

Google Search can execute JavaScript. Saying that JavaScript websites cannot be indexed would be wrong.

Google documents crawling, rendering, and indexing as distinct stages. When the initial response lacks the main content, that content depends on successful rendering. Google also recommends server rendering or prerendering and notes that not all bots execute JavaScript. See its JavaScript SEO guidance.

For a documentation page, consider the dependencies involved:

1. The HTML request succeeds.
2. The script downloads.
3. The script executes successfully.
4. The content API responds.
5. The application inserts the article.

Prerendering can move the article-generation work into the publishing process. A successful request can then receive the article without repeating that browser-side sequence.

This reduces a particular set of delivery dependencies. It does not establish that every prerendered site will outperform every client-rendered site.

There are still basic requirements to get right: public access, appropriate status codes, discoverable links, indexing directives, canonical URLs, and useful content.

Why this matters for AEO

AEO usually means answer engine optimization: making information suitable for discovery and use in answer-oriented search experiences.

Consider a troubleshooting article answering:

Why does this API return an authentication error?

If the response contains only a loading interface, a system that reads that response without rendering cannot extract the explanation from it.

If the response includes the error condition, likely causes, diagnostic commands, and limitations, those details are available for processing.

That is an engineering inference about content availability—not proof of a ranking or answer-selection benefit.

The answer also needs to make sense when read outside your interface. For example:

This command refreshes the token.

Which token? For which version? Does it replace an existing credential or create a new one?

Prerendering delivers whatever you wrote. It cannot supply missing context.

My editorial recommendation is to place the answer near its question, identify prerequisites, and explain exceptions where they affect the reader’s decision. That helps a human evaluating an extracted passage, too.

Why this matters for GEO

GEO usually means generative engine optimization: work intended to improve visibility in AI-generated answers.

There is no single “AI crawler” with one documented rendering behavior.

OpenAI distinguishes its search crawler, potential training-data collection, and certain user-triggered visits. Those functions have different purposes and controls. Its crawler documentation does not establish a universal JavaScript-rendering capability for every agent. See OpenAI’s crawler overview.

It would therefore be misleading to promise:

Prerender your website and ChatGPT will cite it.

The more defensible argument is narrower:

Delivering the main content as HTML makes it available to clients that fetch and parse HTML without executing the application.

Whether a particular system retrieves, retains, selects, or cites that content is a separate question.

For Google’s AI Overviews and AI Mode, Google states that supporting pages must be indexed and eligible to appear in Search with a snippet. It does not guarantee inclusion when those conditions are met. See AI features and your website.

Prerendering can support the content-delivery layer of that process. It does not control the final answer.

Keep the useful content outside optional interactions

Imagine a comparison page with three subscription plans.

The initial HTML contains the plan names. Prices, billing periods, and cancellation conditions appear only after a browser-side request.

You have delivered the labels, but not the information needed to compare the plans.

For public pages, I would usually include the following in the initial document:

• The main explanation or answer.
• Important qualifications and exclusions.
• Public product or service details.
• Links to related public resources.
• Source references needed to evaluate claims.

Interactive controls can enhance that content.

A calculator can show its formula and assumptions before its sliders become active. A documentation page can contain code before its copy button loads.

Account balances and private user data require a different design. They should remain behind the appropriate access controls.

Prerendering introduces a freshness problem

Suppose a pricing page is generated on Monday. The source system changes on Tuesday, but nothing triggers regeneration.

The page remains easy to retrieve. Its price is now wrong.

Before choosing build-time output, define:

• Which events trigger a rebuild.
• How long publication can take.
• What happens when regeneration fails.
• How cached responses are invalidated.
• How deleted content stops being served.

A practical acceptance test is to change one source value and follow it through to the public response.

Do not stop when the CMS says “published.” Check when the new value becomes visible at the actual URL.

For rapidly changing data, request-time rendering or a supported revalidation mechanism may be a better fit than rebuilding the entire site.

Check your output directly

Fetch a representative page:

curl -sS \
  -D response-headers.txt \
  -o response-body.html \
  https://example.com/docs/installation

Inspect the body for a distinctive sentence from the article, a meaningful link, and any important qualification.

Then compare it with the browser’s live DOM.

Repeat the check without authentication. Test more than the homepage: article, category, product, and documentation templates can behave differently.

Also request a nonexistent URL. A prerendered deployment still needs correct routing and missing-page responses.

These checks establish observable behavior for your requests. They do not prove how every crawler sees the site or whether a page will be indexed.

Avoid adding a separate maintenance problem

A bot-specific rendering service is sometimes proposed as a shortcut: serve rendered output to selected crawlers while keeping the original application for everyone else.

Before introducing another serving path, ask how you will keep its content, status codes, and freshness consistent with the public page.

Where practical, my preference is to deliver useful HTML through the normal route for all visitors. It gives developers one public document to inspect and maintain.

The same caution applies to creating alternate “AI versions” of every article. Google explicitly says that special AI files, including llms.txt, are not required for its Search visibility. See its AI optimization guide.

An alternate format should solve an identified consumer need and have a maintenance owner.

Choose the rendering strategy around the document

A stable tutorial is a good candidate for build-time generation. A frequently changing public listing may need server rendering or controlled regeneration. A private dashboard has different requirements again.

The useful question is not which framework has the strongest SEO reputation.

It is whether the public response contains the right information, whether that information stays current, and whether its delivery works under realistic conditions.

Prerendering matters because it can make the document available before the application starts. SEO, AEO, and GEO still depend on what that document says—and what each downstream system chooses to do with it.

Sources

• Google: JavaScript SEO basics
• Google: AI features and your website
• Google: AI optimization guide
• OpenAI: Overview of crawlers
• Astro: On-demand rendering
• Next.js: Server and Client Components

The page loads. Then the console reports a hydration mismatch.

You add a browser check, move something into useEffect, or suppress the warning. Sometimes it disappears. Sometimes it returns on a different device.

A better starting point is to identify exactly what changed between the server-rendered page and the browser’s first render.

What hydration expects

Hydration connects React’s client-side behavior to HTML already generated on the server.

The initial client render needs to match that server output. Later updates are fine: a button can change state, a request can load fresh data, and a clock can advance.

The problem is a difference at the moment React takes over. React explicitly says to treat hydration mismatches as bugs; it does not guarantee that mismatched attributes will be corrected. See the hydrateRoot documentation.

Before changing code, ask:

What value, element, or attribute differs on the first render—and why?

Start with the reported component

Reproduce the error with a direct page reload. Read the hydration message and component stack before investigating unrelated components.

Then narrow the problem:

1. Find the smallest component containing the reported difference.
2. Replace its dynamic output with a fixed value.
3. Reload.
4. Restore one input at a time.

For a product card, that might mean temporarily replacing the price, availability, and personalized label separately.

If a fixed price removes the error, investigate the price’s inputs and formatting. You have a smaller problem than “the page has hydration errors.”

Fix browser-only state

This component renders different content depending on where it runs:

export default function SavedItems() {
  const count =
    typeof window === 'undefined'
      ? 0
      : Number(localStorage.getItem('savedCount') || 0);

  return <span>Saved items: {count}</span>;
}

The server might render 0, while the browser’s first render produces 3.

The typeof window check prevents a server-side reference error. It does not make the output match.

For browser-owned state, start with shared initial content, then read storage after mounting:

'use client';

import { useEffect, useState } from 'react';

export default function SavedItems() {
  const [count, setCount] = useState(null);

  useEffect(() => {
    try {
      const stored = localStorage.getItem('savedCount');
      const parsed = Number(stored ?? 0);

      setCount(
        Number.isInteger(parsed) && parsed >= 0 ? parsed : 0
      );
    } catch {
      setCount(0);
    }
  }, []);

  return (
    <span>
      Saved items: {count === null ? '—' : count}
    </span>
  );
}

The server and initial client render both show the placeholder. The browser updates afterward.

React documents that effects run only on the client. It also explains this approach for displaying different server and client content in its useEffect reference.

There is a visible tradeoff: the placeholder changes after mounting. For an important value the server can already know, pass that value into the component instead.

Stop generating fresh values during rendering

Look for values such as:

<p>{Date.now()}</p>
<p>{Math.random()}</p>

Independent server and browser executions can produce different output.

For a publication timestamp, use the timestamp attached to the article. Do not generate a new “publication time” when rendering it.

A stable initial representation can be simple:

export default function PublishedAt({ publishedAt }) {
  return (
    <time dateTime={publishedAt}>
      {publishedAt}
    </time>
  );
}

Here, publishedAt should be the same serialized ISO timestamp in both environments.

If you want a reader-local date, keep a stable initial label and localize it afterward:

'use client';

import { useEffect, useState } from 'react';

export default function LocalDate({ iso }) {
  const [label, setLabel] = useState(iso);

  useEffect(() => {
    const date = new Date(iso);

    if (!Number.isNaN(date.getTime())) {
      setLabel(date.toLocaleString());
    }
  }, [iso]);

  return <time dateTime={iso}>{label}</time>;
}

This example assumes iso stays unchanged for the mounted component. If it needs to represent a different document, reset the component appropriately—for example, by keying it to that document.

Use the same initial data

Suppose the server renders an inventory count of 12. Before hydration, the browser obtains a newer count of 11 and uses that as its initial state.

Both values may be legitimate. They still disagree.

My preferred investigation is to trace the initial snapshot:

• Which response supplied the server’s value?
• Which data initializes the browser?
• Does the client refetch before hydration completes?
• Are sorting, filtering, or feature flags different?

Use the server-provided snapshot for the first client render. Refresh it afterward through the framework or data library’s supported mechanism.

Also examine personalized content. An anonymous server response and a signed-in browser response need an intentional transition, rather than two incompatible initial trees.

Fix invalid HTML

Check the structure around the error:

// Invalid nesting
<p>
  Installation instructions
  <div>Run the command below.</div>
</p>

Use valid sibling or parent elements:

<section>
  <p>Installation instructions</p>
  <div>Run the command below.</div>
</section>

Nested links, nested buttons, and block elements inside paragraphs are worth checking. Next.js lists invalid nesting, browser-only APIs, time-dependent rendering, extensions, and HTML transformations among common causes in its hydration error guide.

Inspect what your component library actually renders. A component named Text might produce a paragraph, making an innocent-looking composition invalid.

Give accessibility IDs a stable mechanism

Do not generate a fresh random ID during rendering to connect a label and input.

Use React’s useId:

'use client';

import { useId } from 'react';

export default function EmailField() {
  const id = useId();

  return (
    <div>
      <label htmlFor={id}>Email</label>
      <input id={id} name="email" type="email" />
    </div>
  );
}

React designed useId to work with server rendering when the server and client component trees match. It does not fix an otherwise different tree, and it should not supply list keys. List keys should come from your data. See the useId documentation.

Investigate changes outside React

If you cannot reproduce the error consistently, compare conditions:

These observations narrow the search. They do not establish the cause on their own.

Don’t suppress the entire problem

suppressHydrationWarning is an escape hatch for a narrowly understood difference. It works only one level deep, and React does not patch suppressed mismatched text.

It is unsuitable for hiding a changing component tree, incorrect account state, or broken HTML.

Likewise, disabling server rendering for a component can be appropriate for a browser-only widget. It should be a deliberate boundary decision. Next.js documents component-level SSR disabling as an option, but moving an entire page to client-only rendering can avoid the mismatch while leaving the original data problem unexplained.

Verify the fix

After making the change:

• Reload the exact URL directly.
• Test a production build.
• Repeat the state that triggered the mismatch.
• Check both the console and visible content.
• Exercise the affected interaction.

For the saved-items example, test empty storage, a saved number, invalid data, and unavailable storage.

A useful fix explains why the initial outputs now agree. “The warning disappeared” is an observation. “Both environments start with the same placeholder, and storage is read after mounting” explains the mechanism.

Sources

• React: hydrateRoot
• React: useEffect
• React: useId
• Next.js: Hydration errors

The release looks healthy. Pages load, navigation works, and the application returns 200 OK.

But the public response includes this:

X-Robots-Tag: noindex

A restriction intended for staging has reached production. Nothing about the page’s appearance makes that obvious.

Consider this a debugging scenario, not a report of a particular incident. The important question is how to detect it before search traffic becomes the alert.

Check the response, not just the source code

An indexing instruction can come from more than one layer.

Your application might generate:

<meta name="robots" content="index, follow">

Meanwhile, a reverse proxy adds:

X-Robots-Tag: noindex

For Google, adding index does not cancel an applicable noindex. Conflicting applicable rules are resolved restrictively. The HTML and HTTP header both need inspection. Google documents this in its robots directives specification.

Start with an unauthenticated GET against an actual production page:

curl -sS \
  -D headers.txt \
  -o page.html \
  https://example.com/docs/installation

Read both files. Look for every X-Robots-Tag header and every relevant robots meta element.

Using GET matters here: it retrieves the document body as well as the headers. A HEAD-only check cannot inspect HTML directives.

If the URL redirects, inspect the redirect and its destination separately. Establish which response you are diagnosing before drawing conclusions.

Look for duplicates and crawler-specific instructions

These elements can coexist:

<meta name="robots" content="index, follow">
<meta name="googlebot" content="noindex">

A check that reads only the first robots element would miss the restriction targeting Google.

Likewise, checking only HTML misses headers. Checking only headers misses HTML.

For this investigation, collect:

• Generic robots meta elements.
• Google-specific googlebot meta elements.
• Every X-Robots-Tag response header.
• The final URL, status code, and content type.

Also look for none, which includes a restriction on indexing.

Do not treat every crawler-specific instruction as applying to every crawler. Record its scope. A restriction intended for another bot needs interpretation, rather than an automatic conclusion about Google.

Find which layer owns the restriction

Once you have reproduced the response, trace the instruction back to its source.

Search the repository:

rg -n -i 'noindex|x-robots-tag|googlebot' .

A repository search helps locate application rules. It cannot reveal a header added exclusively in a CDN dashboard.

Compare the built artifact, application response, and public response where your infrastructure allows it. If the instruction appears only at the public endpoint, investigate the layers between that endpoint and the application.

If the generated HTML already contains it, investigate what configuration the build used.

A deployment can run with production settings while serving files generated earlier with staging settings.

Don’t remove it with client-side JavaScript

This is a fragile recovery strategy:

document
  .querySelector('meta[name="robots"]')
  ?.remove();

Google warns that an initial noindex may cause it to skip rendering and JavaScript execution. The script intended to remove the restriction might never run. See its JavaScript indexing guidance.

Correct the original response. Remove the unintended header at its source or regenerate the HTML with the right settings.

Then verify the deployed URL again.

Turn the expectation into a deployment check

A release pipeline usually knows which environment it is deploying. It should also know which representative pages must remain indexable.

For example:

https://example.com/
https://example.com/docs/installation
https://example.com/blog/example-article

Choose URLs covering different templates and serving paths. A homepage-only check can miss a restriction attached to an article layout.

Here is a small Python check using only the standard library. Save it as check_indexability.py:

import re
import sys
from html.parser import HTMLParser
from urllib.request import Request, urlopen

BLOCK = re.compile(
    r'(?<![\w-])(?:noindex|none)(?![\w-])',
    re.I
)


class RobotsParser(HTMLParser):
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.rules = []

    def handle_starttag(self, tag, attrs):
        if tag.lower() != 'meta':
            return

        attrs = dict(attrs)
        name = attrs.get('name', '').lower().strip()

        if name in {'robots', 'googlebot'}:
            self.rules.append(
                (name, attrs.get('content', ''))
            )


def blockers(headers, html):
    found = []

    for value in headers:
        if BLOCK.search(value):
            found.append(f'X-Robots-Tag: {value}')

    parser = RobotsParser()
    parser.feed(html)

    for name, value in parser.rules:
        if BLOCK.search(value):
            found.append(f'meta {name}: {value}')

    return found


def check(url):
    request = Request(
        url,
        headers={'User-Agent': 'DeploymentIndexCheck/1.0'}
    )

    with urlopen(request, timeout=20) as response:
        if response.url != url:
            raise ValueError(
                f'unexpected redirect to {response.url}'
            )

        if response.status != 200:
            raise ValueError(f'HTTP {response.status}')

        if response.headers.get_content_type() != 'text/html':
            raise ValueError('expected text/html')

        html = response.read().decode(
            'utf-8', errors='replace'
        )

        found = blockers(
            response.headers.get_all('X-Robots-Tag', []),
            html
        )

        if found:
            raise ValueError('; '.join(found))

        print(f'PASS {url}')


if __name__ == '__main__':
    if len(sys.argv) < 2:
        sys.exit(
            'Usage: python check_indexability.py URL [URL ...]'
        )

    failed = False

    for url in sys.argv[1:]:
        try:
            check(url)
        except Exception as error:
            failed = True
            print(
                f'FAIL {url}: {error}',
                file=sys.stderr
            )

    sys.exit(1 if failed else 0)

Run it after deployment:

python check_indexability.py \
  https://example.com/ \
  https://example.com/docs/installation \
  https://example.com/blog/example-article

A failing URL produces a nonzero exit code, so the pipeline can mark the release unhealthy.

Use final canonical URLs as inputs. This script rejects redirects deliberately: a move to a login page or another hostname should require investigation.

The header check is conservative. It flags noindex or none even inside a header scoped to another crawler. That can produce a reviewable false alarm. It does not implement a complete crawler-specific directives evaluator.

The HTML parser handles ordinary markup and ignores commented-out tags. It does not execute JavaScript or reproduce a browser’s complete parsing behavior.

Put the check at two points

I would run this check against an isolated candidate deployment before promotion, then against the public hostname after promotion.

The first catches restrictions in the candidate response. The second catches differences introduced by production routing, caching, or header rules.

If your deployment system has no promotion stage, run it immediately after deployment and connect failure to a clear response: alert the release owner, investigate, and roll back when appropriate.

Do not describe a post-deployment check as preventing exposure. It detects a problem after the endpoint has changed.

Avoid automatically removing all restrictions when the check fails. Some pages should remain excluded. The URL list expresses the pages whose public indexing eligibility you intended to preserve.

Keep staging access separate

For confidential staging environments, use authentication or network restrictions. A noindex instruction does not prevent someone from opening the page.

Also avoid using robots.txt as a substitute for diagnosing production directives. Google must fetch a page to see its noindex; a crawl block can prevent that. Google explains this interaction in its noindex documentation.

Define staging and production behavior explicitly. Missing or unrecognized environment configuration should fail deployment rather than silently choose an indexing policy.

Verify recovery without promising a date

After fixing the source:

1. Deploy or regenerate the affected pages.
2. Invalidate relevant cached responses where necessary.
3. Repeat the unauthenticated public GET.
4. Check multiple affected templates.
5. Use Search Console’s live URL inspection and monitor subsequent indexing reports.

Keep the current response separate from the last indexed state. A clean response today does not mean Google has already revisited every affected URL.

The deployment check also has a limited purpose. It does not test robots.txt, canonical selection, meaningful content, or directives added later by JavaScript. It cannot guarantee indexing.

What it can do is catch a release that says “public production page” in its configuration while telling crawlers noindex in its response.

That is a small enough mistake to test for—and a consequential one to leave unnoticed.

Your documentation endpoint returns 200 OK. The firewall dashboard shows bot traffic. Someone pastes a paragraph from the site into an AI assistant and gets a reasonable answer.

Which of those observations proves that an AI search system retrieved the document?

None, by itself.

The HTTP response might contain a challenge page. The bot might be an unrelated scraper. The assistant might be answering from the text the user supplied.

Before changing your content or adding another machine-readable file, establish what happened at the request boundary.

Name the system and the action

“AI can’t read our site” is too broad to debug.

Try rewriting it as a specific question:

Can the search crawler operated by this provider retrieve the public migration guide at this URL?

That gives you an actor, an action, and a resource.

A training crawler collects material for a different purpose from a search crawler. A user-triggered fetch happens in a different context again. An AI answer may also rely on a search index rather than a fresh visit to your origin.

OpenAI’s crawler documentation distinguishes search, potential training collection, and certain user-triggered visits. Its search and training controls are independent.

Here is a deliberately limited comparison, checked on October 23, 2026:

“Not documented” is not a finding that JavaScript never runs. These agents’ rendering capabilities were not tested for this article.

Google states that supporting links in AI Overviews and AI Mode require a page to be indexed and eligible for a Search snippet. Meeting those requirements does not guarantee inclusion. See AI features and your website.

The practical consequence is simple: investigate the system you care about. Do not generalize a result from one assistant to every AI product.

Investigate a migration guide, not the homepage

Suppose a public site publishes a guide at:

https://docs.example.com/guides/migrate-v2

The guide explains a breaking API change. A developer says an assistant keeps recommending the old method.

Several explanations remain possible:

• The guide cannot be fetched.
• The guide can be fetched, but the response lacks its main content.
• The system has an older copy.
• Retrieval selects another document.
• The answer is generated without retrieving this guide.
• The guide itself leaves the old and new methods ambiguous.

These are different problems. Rewriting the introduction does not distinguish them.

Start with the exact URL:

curl -sS \
  -L \
  --max-redirs 5 \
  --connect-timeout 10 \
  --max-time 30 \
  -D migration-headers.txt \
  -o migration-body.html \
  -w 'status=%{http_code}\nurl=%{url_effective}\n' \
  https://docs.example.com/guides/migrate-v2

This follows redirects and records the final status and URL. The header file can contain multiple response header blocks because redirects produce additional responses.

Read the body.

Does it contain the explanation of the new method? Or does it contain a sign-in form, a challenge, a loading shell, or a generic error?

Do not stop at the status line.

For a first pass, choose three distinctive pieces of content:

1. The guide’s main heading.
2. The exact replacement API method.
3. A sentence explaining when the old method stops working.

Their presence is a useful content-delivery check. Their absence tells you where to investigate next. It does not explain every possible retrieval failure.

Your session can hide an access problem

Repeat the browser visit in a clean session without signing in.

A public-looking page may still depend on a cookie left by an earlier visit. An API may accept requests from your browser while rejecting requests without a session. A CDN may treat different networks differently.

Record those conditions rather than collapsing them into “works for me.”

A useful investigation note might say:

The article text is present in an unauthenticated GET from this network. The final response is HTML with status 200. Access from the provider’s infrastructure has not been verified.

That is a narrower claim than “AI accessible,” and much more useful to the next developer.

If the document is intentionally private, preserve that boundary. Accessibility to a public crawler is not a reason to expose private documentation.

A User-Agent string is a test input

You can repeat the request with a crawler-style User-Agent value:

curl -sS \
  -A 'OAI-SearchBot' \
  -D simulated-headers.txt \
  -o simulated-body.html \
  https://docs.example.com/guides/migrate-v2

This can reveal a rule that treats that header differently.

It does not turn curl into the provider’s crawler.

The request still comes from your network, with your HTTP client’s behavior. A firewall may consider source IP, request history, protocol characteristics, or other signals.

Do not create a broad firewall bypass for anyone who sends a familiar bot name.

For OpenAI agents, compare observed source addresses against the provider’s current published ranges. For Google requests, follow its documented verification process, which includes published IP lists or reverse DNS followed by forward confirmation. See Google’s verification guide.

When a proxy sits in front of your application, make sure you understand which logged address represents the requester. An arbitrary forwarded header is not trustworthy evidence.

Read edge logs alongside application logs

If a request is blocked before reaching your application, your application logs will not explain the rejection.

For a relevant time window, examine:

• Requested hostname and path.
• Timestamp and source address.
• Claimed User-Agent.
• Edge action, such as allow, challenge, or block.
• Response status and bytes sent.
• Cache status, where available.
• Request identifier linking edge and origin events.

Treat this as a proposed logging checklist, not a requirement to collect every possible field indefinitely.

A verified crawler request with a successful response is evidence of access at that time. It does not show that the document was retained, indexed, selected for a particular question, or cited in an answer.

Likewise, no request in a short log window does not establish that the site is blocked. The system may not have attempted a visit during that period.

Write down the observation separately from the explanation you suspect.

Make crawler policy explicit

A site owner might allow OpenAI’s search crawler while declining GPTBot collection:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This illustrates the separation documented by OpenAI. It is not a universal recommendation for every website.

Review the complete robots.txt file before changing it. Existing agent groups and path rules matter.

Robots.txt also does not authenticate visitors or secure private resources. Google’s robots.txt introduction explains that disallowed URLs may still be discovered and appear in Search without their blocked content.

Use authentication for private information. Use crawler directives to express the relevant crawling policy.

Deliver a document that survives extraction

For the migration guide, I would make these relationships explicit in the public document:

• Which product and version the guide covers.
• Which method is deprecated.
• Which method replaces it.
• Whether the replacement applies immediately or after an upgrade.
• What the developer must change in an existing implementation.

Put that information in accessible text. Use headings to separate steps, and label code examples by version.

This is a documentation recommendation. It is not a claim that a particular heading pattern increases citations.

If a warning exists only inside a screenshot or a client-side interaction, a text-only fetch may omit the distinction that makes the advice correct.

Compare the extracted response with the browser page. Ask whether someone reading only that response could safely perform the migration.

Decide what llms.txt would solve

The llms.txt proposal describes a Markdown guide that helps agents find useful information and linked resources. It can be worth evaluating for a documentation workflow whose consumers use it.

It does not repair a blocked article endpoint. It does not replace access controls. Publishing it does not establish that every provider will read it.

Google explicitly says its Search systems do not use llms.txt as a special visibility mechanism. See its generative AI optimization guide.

If you maintain alternate Markdown documents, generate them from the same source as the HTML where possible. Otherwise, the migration guide may be corrected in one format while continuing to recommend the old API in another.

The maintenance question is concrete: who checks that both representations still agree after the next release?

Close the investigation with a bounded result

For each important document, record four things:

If you test an assistant’s answer, record the question, product, date, and any cited URLs. Treat that as one observation under those conditions.

A page can be accessible without being selected. A correct answer can appear without a fresh fetch. A crawler can retrieve a document that an answer later misinterprets.

The part you can debug directly is what your infrastructure delivers. Start there, and make the next claim only when the evidence supports it.

Sources

• OpenAI: Overview of crawlers
• Google: AI features and your website
• Google: Verifying crawler requests
• Google: Robots.txt introduction
• The llms.txt proposal
• Google: Generative AI optimization guide

Open a documentation page in your browser. The heading appears, the navigation works, and the code examples look right.

Now request the same URL from a terminal:

curl -sS https://docs.example.com/docs/installation

The response contains a root element, a loading message, and a JavaScript bundle. The installation instructions are missing.

That difference is where the investigation starts.

It does not prove that a search engine cannot index the page. It does show that reading the instructions requires more than fetching its HTML.

The browser has done work you haven’t inspected

Consider this simplified React component:

import { useEffect, useState } from 'react';

export default function InstallationPage() {
  const [doc, setDoc] = useState(null);
  const [error, setError] = useState(false);

  useEffect(() => {
    let cancelled = false;

    async function load() {
      try {
        const response = await fetch('/api/docs/installation');

        if (!response.ok) {
          throw new Error(`HTTP ${response.status}`);
        }

        const data = await response.json();

        if (!cancelled) setDoc(data);
      } catch {
        if (!cancelled) setError(true);
      }
    }

    load();
    return () => { cancelled = true; };
  }, []);

  if (error) return <p>Could not load this document.</p>;
  if (!doc) return <p>Loading documentation…</p>;

  return (
    <article>
      <h1>{doc.title}</h1>
      <p>{doc.text}</p>
    </article>
  );
}

The component can display a perfectly readable page. Its main content still depends on a successful browser-side request.

That request might work because you are logged in. It might fail without a cookie, behind a firewall rule, or after an API deployment changes its response. A screenshot cannot tell you which conditions were necessary.

Google documents separate crawling, rendering, and indexing stages. Google Search can execute JavaScript, but content in an application shell must first become available during rendering. Google also notes that other bots may not execute JavaScript. See its JavaScript documentation.

For this example, the useful question is narrower than “Can Google read React?”

Can the main document be obtained reliably from its public URL?

Compare three views of the same page

Start with the network response:

curl -sS \
  -D response-headers.txt \
  -o response-body.html \
  https://docs.example.com/docs/installation

This makes a GET request and saves its headers separately from its body. Inspect both files.

Then open the page in a browser and inspect the live DOM. The Elements panel shows what exists after scripts have had an opportunity to change the document.

Finally, disable JavaScript in the browser’s developer tools and reload the URL directly. Do not navigate there through an already-running application.

These checks answer different questions:

Disabling JavaScript is a dependency check. It does not reproduce Google’s renderer.

Look for one specific sentence from the article in each view. Searching for the main heading alone can miss a page that delivers its title but fetches everything useful later.

Fix the content boundary

For a small documentation site, I would put the article in the HTML response and add interactive features afterward.

That does not require abandoning JavaScript. Node.js can send an empty application shell or a complete document. The runtime is not the deciding factor.

Here is a complete example using Node.js 24’s built-in HTTP server. There are no package dependencies.

Save this as server.mjs:

import { createServer } from 'node:http';

const docs = new Map([
  [
    'installation',
    {
      title: 'Installation',
      text: 'Install the package with npm install example-client.'
    }
  ]
]);

const escapeHTML = value =>
  String(value).replace(/[&<>"']/g, char => ({
    '&': '&amp;',
    '<': '&lt;',
    '>': '&gt;',
    '"': '&quot;',
    "'": '&#39;'
  }[char]));

function handler(req, res) {
  const path = new URL(req.url, 'http://localhost').pathname;
  const slug = path.startsWith('/docs/') ? path.slice(6) : '';
  const doc = docs.get(slug);

  res.setHeader('Content-Type', 'text/html; charset=utf-8');

  if (!doc) {
    res.writeHead(404);
    res.end(
      '<!doctype html><title>Not found</title><h1>Not found</h1>'
    );
    return;
  }

  const canonical = new URL(
    '/docs/' + slug,
    'https://docs.example.com'
  ).href;

  res.writeHead(200);
  res.end(`<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>${escapeHTML(doc.title)}</title>
  <link rel="canonical" href="${escapeHTML(canonical)}">
</head>
<body>
  <nav>
    <a href="/docs/installation">Installation</a>
  </nav>
  <main>
    <article>
      <h1>${escapeHTML(doc.title)}</h1>
      <p>${escapeHTML(doc.text)}</p>
    </article>
  </main>
</body>
</html>`);
}

createServer(handler).listen(3000, '127.0.0.1', () => {
  console.log('Open http://127.0.0.1:3000/docs/installation');
});

Run it:

node server.mjs

Then inspect the document and a missing route:

curl -i http://127.0.0.1:3000/docs/installation
curl -i http://127.0.0.1:3000/docs/missing

The first response contains the article. The second returns a 404.

Replace the example canonical origin before using this code on a real site. The fixed origin is deliberate: the canonical URL should come from trusted site configuration.

This example supports exact paths without trailing slashes. A production application should also define its preferred trailing-slash behavior and redirect the alternative consistently.

The code escapes plain text before inserting it into HTML. It is not a Markdown renderer or an HTML sanitizer. Those are separate requirements if your content includes rich markup.

The relevant server APIs are documented in the Node.js 24 HTTP reference.

Apply the same decision inside your framework

In Express, this boundary usually sits inside the route handler and template rendering. Express provides explicit response status and HTML response methods in its 5.x response API.

In a React framework, inspect whether the route delivers the main article through server rendering or build-time generation, or waits for a browser-side effect.

Choose based on the content’s lifecycle:

• Documentation released with your code can often be generated during the build.
• Public information that changes frequently may need request-time rendering or a defined revalidation strategy.
• Personal account data belongs behind authentication and may reasonably depend on client-side requests.

Server rendering introduces its own operational questions. What happens when the content database is slow? What gets cached? How does a content update invalidate that cache?

Moving a request to the server is useful only if its failure behavior is understood.

A missing document needs a missing-document response

A common deployment rule sends the application shell for every path.

That makes direct visits to client-side routes work. It can also make /docs/this-page-never-existed return 200 OK.

In the example above, the server looks up the document before sending its response. Unknown documents receive 404, rather than a successful response containing an error message.

Google’s JavaScript guidance specifically discusses meaningful status codes and soft 404 problems. This is worth checking on the deployed URL, where proxy and hosting rules may differ from local behavior.

Also test navigation. Use real links:

<a href="/docs/installation">Installation</a>

An element that changes routes only through a click handler gives you another dependency to investigate.

Check directives before rewriting the application

A rendering fix will not resolve an accidental indexing restriction.

Check for both an HTML robots directive and an HTTP header:

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

If the public page should be indexable, find where the restriction originates. It may come from the application, a preview-environment setting, or infrastructure configuration.

Google must be able to crawl a page to read its noindex instruction. Blocking the URL in robots.txt can prevent that instruction from being seen. See Google’s noindex documentation.

Keep canonical URLs consistent with your preferred public URLs. Canonical annotations are signals about duplicate versions, not commands that guarantee indexing. Google explains the distinction in its canonicalization guide.

A sitemap helps with URL discovery. It cannot supply content missing from the page response or guarantee that submitted URLs will be indexed. See the sitemap overview.

Use evidence to choose the next fix

After deployment, use Search Console’s URL Inspection tool to examine Google’s reported state and available rendered output. Keep the live test separate from the indexed-page record: checking the current implementation does not establish that the index has already changed.

Server logs can help establish whether requests reached your infrastructure and what responses were returned. They do not show that the content was selected for indexing.

A useful release check is small: request one real document and one nonexistent document, then inspect their status, headers, and body. Repeat it after changing hosting rules.

That catches a surprisingly consequential class of mistakes before anyone needs to debate whether the framework is “SEO friendly.”

Sources

• Google: JavaScript SEO basics
• Node.js 24: HTTP API
• Express 5.x: Response API
• Google: Blocking indexing
• Google: Canonical URLs
• Google: Sitemaps