Why this matters for an insurance agent
For the agent, this is insurance against wasted work. A single technical mistake, a stray noindex or a slow mobile page, can hide their whole site and cost them every lead it should have produced. Getting this right means the pages we build actually get seen, so the agent's investment turns into ranked pages and inbound calls.
robots.txt and the noindex tag
Two controls decide whether a page can be crawled and indexed. Getting them wrong is one of the most damaging mistakes in SEO.
- robots.txt: a file at the site root that tells bots which paths they may crawl. A stray
Disallow: /blocks the entire site. - meta robots:
index,followon pages we want ranked.noindexkeeps a page out of search (fine for staging, thank-you pages).
The number one launch disaster
Never ship a staging noindex or Disallow: / to production. It tells Google to ignore the whole site. Staging noindex is fine, but it must be removed at launch. Always check this before and after go-live.
Sitemaps and canonicals
These two help Google find every page and avoid duplicate confusion.
The list of pages
An XML file listing the pages we want ranked: home, service and money pages, location pages, hubs. Keep utility pages (privacy, terms, thank-you, login) out of it so it stays focused. Submit it in Google Search Console.
The one true URL
A self-referencing rel="canonical" tag on each page points to its own final production URL. This prevents duplicate-content confusion when the same page is reachable by multiple URLs.
Mobile and page speed
Google indexes the mobile version of a site first, and it measures real-world load experience.
- Mobile-first: the site must look and work great on a phone. Most local searches happen there.
- Core Web Vitals: Google's speed and stability metrics (largest contentful paint, interaction delay, layout shift). Slow or janky pages get held back.
- Check both in PageSpeed Insights and Search Console, and fix what they flag.
Technical SEO is mostly about not shooting ourselves in the foot. A fast, crawlable, correctly-indexed site is the floor. Content and authority build the rest.
* Key takeaways
- robots.txt controls what bots may crawl. A stray Disallow: / blocks the whole site.
- Use index,follow on pages we want ranked and noindex only for staging or utility pages.
- Never let a staging noindex or Disallow: / reach production. Verify before and after launch.
- Give Google a focused sitemap.xml (money and ranking pages only) and a self-referencing canonical on each page.
- The site must be mobile-first and pass Core Web Vitals. Check PageSpeed Insights and Search Console.
Module 4 quiz
Score 75% or higher to complete this module and unlock the next one.
5 questions
Not completed-
What does Disallow: / in robots.txt do?
Why: Disallow: / tells crawlers to stay out of everything. Shipping it to production hides the whole site from search.
-
Which pages belong in sitemap.xml?
Why: The sitemap should list ranking targets only. Utility pages stay crawlable but out of the sitemap so it stays focused.
-
What is the single most damaging launch mistake in technical SEO?
Why: A leftover staging noindex or Disallow: / tells Google to ignore the site, wiping out all rankings. Always verify at launch.
-
A self-referencing canonical tag exists to:
Why: The canonical names the one true URL for a page so duplicate URLs do not split or confuse indexing.
-
Google primarily indexes which version of a site?
Why: Google is mobile-first, so the phone experience and Core Web Vitals directly affect how the site is indexed and ranked.