Technical SEO: make sure Google can find, read and store your site

Technical SEO is everything that helps search engines access and understand your website as a whole. It deals with the structure and the code of the site, where on-page work deals with the content of a single page.

Think of a shop. Your content is the product on the shelves. Technical SEO is the building: the doors open, the lights work, the aisles are labeled. A great product behind a locked door sells nothing.

The good news for a small business: a site with a few dozen pages has few technical needs. Get the basics in this chapter right once, check them a few times a year, and spend the rest of your energy on content.

1. AccessSearch engines can reach every important page.
2. UnderstandingThe site sends clear signals about which URL is the right one for each piece of content.
3. ExperiencePages are secure, work on phones and load quickly. Chapter 6 covers speed.

Google's documentation 'How Search works' describes three stages: crawling, indexing and serving results. For modern sites there is a rendering step in between. Technical SEO removes the obstacles at each stage.

Crawling is done by Googlebot, the program that fetches pages. It discovers URLs by following links and by reading sitemaps. Google uses its smartphone crawler for indexing, so the mobile version of your page is the one that counts.

Indexing means Google analyzes the page and stores it in its database, the index. Google chooses which pages earn a place. Pages with little unique value, or pages that duplicate others, are often left out.

Start every technical review with one question: are my important pages in the index? Two free methods give the answer within minutes.

The quick method is the site: search. Type site:yourdomain.com into Google. You see a sample of indexed pages. Add a word to check a section, such as site:yourdomain.com/services.

The precise method is Google Search Console, Google's free tool for site owners. Under Indexing, open the Pages report, officially called the Page indexing report. It shows the number of indexed pages and lists reasons for every page left out.

For a single page, paste its address into the URL Inspection tool at the top of Search Console. You see when Google last crawled it, which canonical URL Google chose, and whether the page is indexed.

1. Step 1Verify your site in Search Console, ideally as a domain property through your DNS settings.
2. Step 2Open Indexing, then Pages, and note the totals.
3. Step 3Write down your 20 most important URLs and inspect any that are missing from the index.
4. Step 4After a fix, use 'Request indexing' in URL Inspection and 'Validate fix' in the report.

The report can look alarming, since nearly every site has pages outside the index. Many of those are fine: redirected URLs, duplicates with a correct canonical tag and pages you excluded on purpose.

Focus on important pages that are missing. The table lists common statuses in the report and the action each one calls for.

The robots.txt file is a small text file at the root of your site, at yourdomain.com/robots.txt. It tells crawlers which parts of the site they may request. Every well-behaved crawler reads it before anything else.

Google's documentation stresses one point: robots.txt manages crawling, and it is a poor tool for keeping a page out of Google. A blocked URL can still appear in results, shown as a bare link, when other sites link to it.

Most small business sites need only a few lines. Allow everything, block a handful of private or endless areas such as internal search results and shopping carts, and state where your sitemap lives.

1. User-agent: *The rules below apply to all crawlers.
2. Disallow: /cart/Crawlers skip every URL that starts with /cart/.
3. Disallow:An empty Disallow line means the whole site is open.
4. Disallow: /A single slash blocks the entire site. This one line has removed many sites from Google after a launch.
5. Sitemap: https://yourdomain.com/sitemap.xmlTells crawlers where to find your sitemap.

Keep CSS and JavaScript files open for crawling. Google needs them to render your pages the way a visitor sees them.

A noindex rule is a line in the HTML head of a page, or in the HTTP response header, that asks search engines to leave the page out of the index. Google must be able to crawl the page to see that request.

This creates a classic mistake: blocking a page in robots.txt and adding noindex. Google then stays away, misses the noindex and may keep the URL in the index. Choose one tool per goal.

An XML sitemap is a file that lists the URLs you want search engines to index. XML is simply a structured text format. The file usually lives at yourdomain.com/sitemap.xml, and most website platforms generate it automatically.

A sitemap helps discovery. It is most valuable for new sites, large sites and pages with few links. It is a suggestion to Google and gives zero guarantee of indexing.

Google's sitemap documentation gives the limits: 50,000 URLs or 50 MB uncompressed per file. Google ignores the priority and changefreq values. It does use the lastmod date when that date is consistently accurate.

1. IncludeOnly indexable pages that return status 200 and are their own canonical version.
2. ExcludeRedirected URLs, noindex pages, error pages and duplicate versions.
3. SubmitIn Search Console, open Sitemaps, enter the address and press Submit.
4. ReferenceAdd the Sitemap line to your robots.txt so every crawler finds it.
5. MonitorCheck the Sitemaps report for errors and for the number of discovered pages.

Site architecture is the way your pages are organized and linked. Google's crawler travels along links, so a page that sits far from the home page gets visited less often and receives less authority.

Aim for a flat structure. A visitor should reach any important page from the home page in three clicks or fewer. Main menu, category pages and contextual links in the text make that possible.

Add breadcrumbs, the small trail of links that shows the path to the current page. They help visitors orient themselves and give crawlers a clear picture of your hierarchy.

1. Level 1Home page.
2. Level 2Main sections such as services, shop, about, blog and contact.
3. Level 3Individual service pages, categories and articles.
4. Level 4Detail pages such as products, cases and sub-services.

Every time a browser or crawler requests a URL, your server answers with a three-digit status code. The code tells Google how to treat the page. You can see it in URL Inspection or in free online header checkers.

A 404 for a page that truly ended is healthy. The problems are 404s on pages that still have links or traffic, and soft 404s: empty or error-like pages that still return code 200.

A redirect sends visitors and crawlers from one URL to another. Use a 301 redirect whenever a page moves for good: a new URL, a merged page, a new domain or a switch from http to https.

Google's documentation on site moves advises keeping redirects in place as long as possible, and at least one year. That gives Google time to transfer all signals and catches visitors who use old links.

Redirect each old URL to the most relevant new URL. Sending every old page to the home page confuses visitors, and Google may treat such redirects as soft 404s.

1. Map before you moveList every old URL next to its new destination before a redesign or migration.
2. Avoid chainsA to B to C wastes time. Point A directly to C. Googlebot follows up to 10 hops and then stops.
3. Avoid loopsA to B and B back to A makes the page unreachable.
4. Update internal linksChange links on your own site to the final URL so crawlers skip the detour.
5. Test after launchCrawl the old URL list and confirm each one returns a 301 to a page with status 200.

The same content is often reachable at several addresses. Think of a product in two categories, a URL with tracking codes, or one page with a trailing slash and a twin that omits it. Google sees each address as a separate page.

A canonical tag is a line in the HTML head that names the preferred address: rel='canonical' with the full URL. Google then combines the signals of all versions on that one URL.

Google's documentation calls the canonical tag a strong hint, and Google makes the final choice. It also weighs redirects, internal links, the sitemap and https. So keep all those signals pointing at the same URL.

1. Self-referenceGive every indexable page a canonical tag that points to itself.
2. Use full URLsWrite https://yourdomain.com/page/ in full, including the protocol.
3. One per pageTwo canonical tags on a page cancel each other out.
4. Point to live pagesThe canonical URL returns status 200 and is free of noindex.
5. Stay consistentLink internally to the canonical version and list only that version in the sitemap.

Duplicate content means the same or nearly the same content at more than one URL. Google's guidance is calm about it: there is zero penalty for ordinary duplication. The cost is diluted signals and wasted crawling.

Most duplication is created by the website system itself. Check the usual sources below and fix each with a redirect or a canonical tag.

1. http and httpsBoth versions load. Fix: 301 redirect everything to https.
2. www and bare domainBoth versions load. Fix: pick one and redirect the other.
3. Trailing slash and capitals/Page and /page/ both work. Fix: one format, redirect the rest.
4. URL parametersSorting, filters and tracking codes create endless versions. Fix: canonical tag to the clean URL.
5. Print and staging copiesTest sites and print versions get indexed. Fix: password protection for staging, canonical for print.

HTTPS encrypts the connection between visitor and website. Google confirmed it as a ranking signal in 2014, and browsers label plain http pages as insecure. Every page, image and script should load over https.

Mobile-first indexing means Google crawls and judges the mobile version of your site. Google completed that switch for the web as a whole. Whatever is missing on mobile is missing for Google.

1. CertificateInstall a valid SSL certificate and renew it automatically.
2. RedirectSend every http URL to its https twin with a 301.
3. Mixed contentReplace http links to images and scripts inside https pages.
4. Same content on mobileShow the same text, headings, links and structured data on phone and desktop.
5. Responsive designUse one URL per page with a layout that adapts to the screen, as Google recommends.

Hreflang is a tag that tells Google which language and country version of a page to show. You need it only when the same content exists for several languages or regions, such as English for the US and for the UK.

Each version lists all versions, itself included. Google's documentation requires this to be mutual: when page A names page B, page B names page A. Missing return links are the most common hreflang error.

Use standard codes: a two-letter language code such as en or nl, optionally with a country code such as en-GB or nl-BE. Add x-default for the page that serves everyone else.

You can place hreflang in the HTML head, in the HTTP header or in the XML sitemap. Pick one method and apply it to every language version.

JavaScript is the programming language that makes pages interactive. Many modern sites use it to build the whole page in the browser. JavaScript SEO is making sure search engines still see that content.

Google can run JavaScript. Its documentation describes a second phase: Google first reads the raw HTML, then places the page in a queue for rendering with a current version of Chrome. Content in the raw HTML is processed soonest.

Google's crawler loads pages and skips clicking, scrolling and typing. Content that appears only after an action stays unseen. Links must be standard anchor elements with an href address to be followed.

The dependable solution is server-side rendering or static generation: the server sends complete HTML, and JavaScript adds interaction afterward. Google's documentation recommends these approaches over workarounds.

1. Test 1Open 'View page source'. Is your main text visible in the code?
2. Test 2Use URL Inspection, 'Test live URL', then view the rendered HTML and screenshot.
3. Test 3Check that menu and content links are real anchor elements with an href.
4. Test 4Confirm that titles, canonical tags and structured data are in the initial HTML.

AI assistants such as ChatGPT, Claude and Perplexity use their own crawlers, with names like GPTBot, ClaudeBot and PerplexityBot. Your robots.txt decides whether they may read your site.

An analysis published by Vercel in December 2024 found that the major AI crawlers fetched JavaScript files and skipped executing them. So content that exists only after rendering may be invisible to those assistants.

The practical rule is the same as for Google, only stricter: put the content in the HTML. Chapter 12 of this guide covers optimization for AI search in depth.

Crawl budget is the number of URLs Google is willing and able to crawl on your site in a given period. Google's guide on the subject is aimed at very large sites, from roughly a million pages, or large sites that change daily.

A business site with a few hundred pages is crawled comfortably. Your attention belongs elsewhere, unless the site generates endless URLs through filters, calendars or internal search.

You can see Google's activity under Settings, Crawl stats in Search Console. Look at the share of requests that end in errors and at the average response time of your server.

A technical seo checklist is most useful as a routine you repeat every quarter and after every redesign. The steps below need only Search Console and your browser.

For a deeper look, a crawler tool visits every page the way Googlebot does and lists errors. Screaming Frog SEO Spider is a widely used one, with a free mode for small sites. Chapter 10 compares SEO tools.

1. Minute 0 to 10Open the Page indexing report. Confirm your 20 key pages are indexed.
2. Minute 10 to 20Read your robots.txt line by line and open your XML sitemap. Remove wrong rules and dead URLs.
3. Minute 20 to 30Test http, https, www and bare domain. All should land on one version with a single 301.
4. Minute 30 to 40Inspect five key URLs. Check the Google-selected canonical and the rendered HTML.
5. Minute 40 to 50Review pages with status 404 and 5xx. Redirect or repair those with links or traffic.
6. Minute 50 to 60Check the site on your phone: same content, readable text, working menu and forms.

Delta Six builds websites on its own platform with hand-written code. That gives us direct control over the HTML each page sends, the sitemap, canonical tags, redirects and https.

Monthly SEO and optimization for AI search are included as standard. You see your complete website in a 15-minute online showcase, and you pay only when you are happy.

• Verify your site in Google Search Console and open the Page indexing report.
• Confirm that your 20 most important pages are indexed, using URL Inspection.
• Read your robots.txt and remove any rule that blocks pages, CSS or JavaScript you need indexed.
• Use noindex for pages that should stay out of search, and keep those pages crawlable.
• Submit an XML sitemap that lists only indexable pages with status 200, and add the Sitemap line to your robots.txt.
• Redirect http to https and choose either www or the bare domain, with a single 301.
• Give every indexable page a self-referencing canonical tag with the full URL.
• Redirect every moved or merged URL with a 301 to its closest replacement.
• Remove redirect chains and update internal links to the final URL.
• Repair or redirect 404 pages that still receive links or visitors.
• Make sure every important page is reachable within three clicks from the home page.
• Show the same content and links on mobile as on desktop.
• Confirm that main content and links are present in the page source.
• Add mutual hreflang tags if you serve several languages or countries.
• Repeat this audit every quarter and after every redesign or migration.