How to get your website indexed by Google: a step-by-step guide

Why some pages do not get indexed

Getting your pages into Google's index is the prerequisite for ranking. A page that is not indexed does not appear in search results at all, regardless of its content quality, backlinks, or technical optimisation. Before assuming an indexation problem, it helps to understand the most common reasons Google excludes pages.

The most frequent cause is a noindex directive. A noindex tag in the page's meta robots element or in the HTTP response headers tells Google not to include the page in its index. This is the correct setting for pages you do not want ranked (login pages, thank-you pages, admin areas), but it is also one of the most common accidental mistakes in web development. A staging environment setting carried over to production, a plugin update that changes default meta tags, or a CMS misconfiguration can apply noindex to pages you need ranked.

The second most common cause is a robots.txt disallow rule. If your robots.txt file blocks the directory or URL pattern of the page you want indexed, Googlebot will not crawl it. A crawl is required before indexing can happen. Check your robots.txt at yoursite.com/robots.txt and look for any disallow rules that could match your page's URL.

Other causes include the page being behind a login or paywall (Google cannot crawl authenticated content), the page being considered thin or duplicate content (Google chooses not to index pages it considers low-quality), and the page simply not being linked to from anywhere on the site (orphan pages are rarely discovered through crawling).

The fastest way to diagnose why a specific page is not indexed is the URL Inspection tool in Google Search Console. Enter the page URL, and Search Console will show you the last crawl date, the current indexation status, and the specific reason for exclusion if the page is not indexed. This single tool eliminates most of the guesswork from indexation troubleshooting.

Indexation problems are part of the wider technical audit process. The technical SEO audit guide covers indexation as step two and provides the full checklist for identifying and resolving indexation issues across an entire site.

How Google crawls and indexes pages

Understanding the crawl-to-index pipeline helps you intervene at the right point when pages are missing from search results.

Google's process has four stages. First, discovery: Googlebot learns that a URL exists, either by following a link from another page it has already crawled, by reading a sitemap you have submitted, or by processing a manual URL submission in Search Console. Second, crawling: Googlebot fetches the page's HTML and processes its content, links, and metadata. Third, rendering: Google's systems execute any JavaScript on the page to build the full DOM, since many modern sites generate content via JavaScript that is not present in the raw HTML. Fourth, indexing: Google processes the rendered page, evaluates its content quality and signals, and decides whether to add it to the search index.

The important implication here is that discovery and crawling are prerequisites for indexing. A page that has never been linked to and has never appeared in a sitemap may never be discovered at all. Even if it is discovered, crawling can be delayed on sites with large crawl queues or low crawl budgets. And even after crawling, Google makes a quality judgement before indexing.

This pipeline explains why three interventions work for improving indexation: building internal links (improves discovery and signals importance), submitting sitemaps (improves discovery for new pages), and improving content quality (improves the quality judgement at the final stage).

How to submit your site and pages to Google

There are two mechanisms for telling Google about your pages: sitemap submission and URL-level submission via Search Console.

Sitemap submission is the right approach for submitting all your pages in bulk. Your XML sitemap lists every URL you want Google to consider indexing. Submitting it in Google Search Console tells Google where to find the list and allows Search Console to report on how many of the submitted URLs have been indexed.

To submit your sitemap:

1. Log in to Google Search Console at search.google.com/search-console and select your property.
2. Navigate to Sitemaps in the left-hand menu under Index.
3. Enter your sitemap URL in the 'Add a new sitemap' field. For most sites this is yoursite.com/sitemap.xml. Platforms like Webflow, WIX, and Squarespace generate sitemaps automatically at this path.
4. Click Submit. Search Console will immediately attempt to fetch the sitemap and report back on how many URLs it found.
5. Check back after 24 to 48 hours to see how many of the submitted URLs have been indexed and whether any errors were reported.

Individual URL submission via the URL Inspection tool is the right approach for priority pages you need indexed quickly, or for pages that have been updated significantly and need Google to re-crawl them.

1. Open the URL Inspection tool in Search Console.
2. Paste the full URL of the page you want indexed.
3. Review the current status. If the page is not indexed, check the coverage tab for the specific reason.
4. If the page is accessible to Google (no blocks, correct canonical, no noindex), click 'Request indexing'. This places the URL in a priority crawl queue.
5. Repeat for any other priority pages. Note that this method is rate-limited and is not practical for submitting hundreds of pages at once.

Using Google Search Console to check indexation

Google Search Console provides three reports that are essential for monitoring and diagnosing indexation across your entire site.

The Coverage (or Indexing) report is the primary view. It shows the total number of indexed pages and breaks down all excluded pages by reason. The reasons are grouped into four categories: Error (pages that could not be indexed due to a technical problem), Valid with warning (pages that are indexed but have an issue worth addressing), Valid (pages that are indexed correctly), and Excluded (pages that Google has chosen not to index or that are deliberately excluded).

The most important exclusion reasons to investigate:

• Excluded by noindex tag: These pages carry a noindex directive. If any of these are pages you want ranked, remove the tag immediately and request re-crawling.
• Crawled, currently not indexed: Google crawled the page and chose not to index it. This usually means thin content, near-duplicate content, or low perceived quality. Improving the content depth and uniqueness is the fix.
• Discovered, currently not indexed: Google found the page but has not crawled it yet. This can indicate a crawl budget problem on large sites or simply a queue delay on new content. Adding internal links from high-authority pages speeds up crawling.
• Duplicate, Google chose different canonical than user: You specified a canonical URL but Google disagreed and indexed a different version. This usually means the content is too similar to another page and Google is choosing the one it considers more authoritative.

The URL Inspection tool provides page-level detail. For any URL you enter, it shows the last crawl date, the crawled HTML, the rendered page (after JavaScript execution), and the indexation status with specific reason codes. Use this for investigating individual pages flagged in the Coverage report.

The Sitemaps report shows the submission history and indexation rate for each submitted sitemap. A low ratio of indexed URLs to submitted URLs is a signal that many of your submitted pages have quality issues preventing indexation.

Tools like Semrush and Ahrefs also track indexed page counts over time and can alert you to sudden drops, which might indicate a technical change has accidentally applied noindex tags or blocked a directory in robots.txt. Pairing Search Console with one of these platforms gives you both the granular page-level detail and the trend monitoring needed for ongoing indexation management.

Common indexation problems and how to fix them

Most indexation problems fall into one of six categories. Each has a specific diagnosis and fix.

Accidental noindex tags. Check every page you want indexed for a meta robots tag containing 'noindex'. This is most common on sites that recently migrated from a staging environment (where noindex is deliberately applied) or on pages managed by CMS plugins that have their own indexation settings. In WordPress, Rank Math has a site-wide indexation status check that flags any posts or pages set to noindex.

Robots.txt blocking critical directories. Open your robots.txt file and review every disallow rule. A rule like Disallow: /blog/ blocks all pages in that directory, not just one. Remove any disallow rules that apply to directories containing pages you need indexed, and use noindex tags instead for individual pages you want excluded.

Orphan pages with no internal links. Pages with no internal links pointing to them are rarely discovered through crawling. The fix is to add the page to at least one relevant hub or category page and to link to it contextually from related content. Review the site architecture guide for the correct approach to hub-and-spoke internal linking.

Thin or duplicate content. Google excludes pages it considers too thin (insufficient content to be useful) or too similar to other pages already in the index. For thin pages, add substantive content that serves a specific search intent. For duplicate pages created by URL parameters or trailing slash inconsistencies, implement canonical tags pointing to the preferred version.

Slow crawl budget consumption. Very large sites with thousands of low-value pages (pagination pages, filtered product URLs, tag pages) can exhaust Google's crawl budget before it reaches important content. Noindex low-value pages, block parameter-generated URLs via canonical tags or robots.txt, and consolidate thin tag and category pages to free up crawl budget for content that matters.

JavaScript-dependent content. Pages that rely entirely on client-side JavaScript to render their primary content may not be indexed promptly because Google's rendering pipeline operates separately from its crawl pipeline and can introduce a lag of days or weeks. Where possible, serve primary content in the raw HTML response rather than generating it with JavaScript after load.

What this means for your new content

Every piece of content you publish is wasted if Google cannot find it, crawl it, and decide it is worth indexing. Building good indexation habits from the start costs far less time than diagnosing indexation failures after the fact.

For new sites, submit your sitemap immediately after launch and use the URL Inspection tool to request indexing for your most important pages. Build internal links throughout the site from the first day of publishing, since each new link helps Google discover new pages faster. Monitor the Coverage report in Search Console weekly for the first three months, as this is when indexation patterns are established and problems are easiest to catch.

For established sites, set up email alerts in Search Console for coverage errors and run a monthly review of the Coverage report to catch any pages that have dropped from the index unexpectedly. A sudden drop in indexed pages almost always has a technical cause: a plugin update, a CMS setting change, or a robots.txt modification that accidentally blocked a directory.

Indexation is one layer of the technical foundation covered in the full technical SEO audit guide. Running that audit quarterly catches the crawl and indexation problems that accumulate on any active site over time. Pair Search Console with Google Analytics to correlate indexation changes with traffic movements, and with Semrush or Ahrefs for trend monitoring and alerts when indexed page counts change unexpectedly.