You launched the site, searched the company name, and found nothing. So you submitted the sitemap again. Then again.

That may not help. A sitemap is a list of preferred URLs. It is not an order to index them, and indexing is not an order to rank them.

First prove that the page can be found, fetched, understood, and chosen.

Separate discovery, crawling, indexing, and ranking

Founders often use “SEO” for four different events:

  1. A search engine discovers the URL.
  2. Its crawler is allowed to request the URL.
  3. The system decides to store the page in its index.
  4. The page is selected for a particular search.

A page can pass one stage and fail the next. Diagnose the stage before rewriting the headline or buying links.

Google’s crawling and indexing documentation groups the relevant controls: sitemaps, crawler access, canonicalization, metadata, HTTP responses, links, and JavaScript rendering. Most launch failures are in that plumbing or in the page itself.

Decide whether the site is merely new

New domains and new pages are not indexed on a fixed schedule. Google’s indexing FAQ says that being new is a common reason a site is absent and that a sitemap does not guarantee indexing or ranking.

Verify the property in Search Console. Inspect the exact canonical URL. Request indexing once when appropriate, then watch the reported state. Repeated requests do not make a weak or inaccessible page stronger.

Waiting is reasonable only after the technical path is sound. “It probably needs time” is not a substitute for checking.

Fetch the public URL like a stranger

Open the page in a private browser and request it from outside your account. Confirm that the final response is 200, the certificate is valid, and every public hostname resolves to one intended HTTPS URL.

Look for these launch mistakes:

  • a preview password or bot challenge blocks unfamiliar visitors;
  • the apex domain and www version disagree;
  • an old redirect points to staging;
  • a missing page returns 200 with an error message;
  • the server returns 500 while the client paints a cached shell;
  • a geo or device rule changes the result.

Search systems read HTTP outcomes. A beautiful client-side error page with a success status is still a false success. Google documents this problem for JavaScript sites as a soft 404 risk.

Read robots.txt and page directives together

These controls do different jobs.

robots.txt tells a crawler which URLs it may request. A robots meta tag or X-Robots-Tag can tell a search engine not to index a resource it was allowed to fetch. If crawling is blocked, the crawler may never see the page-level instruction.

Check the live response, not the repository. Hosting defaults, framework middleware, and stale edge configuration can change what is actually served.

Common launch residue includes:

User-agent: *
Disallow: /

or:

<meta name="robots" content="noindex">

Remove a block only when the page is genuinely public. Do not make account pages, private reports, internal search results, or customer data indexable to improve a score. Access control protects private material; robots.txt does not.

Google’s robots meta specification is the useful reference when directives conflict or non-HTML files need an X-Robots-Tag.

Check which URL claims to be canonical

Open the rendered HTML and find the canonical link. It should normally identify the final public URL for that page, not localhost, a preview deployment, the homepage, or a different article.

Then compare:

  • the URL in the browser;
  • the final URL after redirects;
  • the canonical annotation;
  • the URL in the sitemap;
  • internal links pointing to the page;
  • social and structured-data URLs.

Consistency does not force a search engine to accept your preference. Google describes canonical annotations, redirects, HTTPS, internal links, and sitemap entries as signals used in canonical selection. Contradictory signals make the preferred page harder to identify.

Do not canonicalize every thin or duplicate route to the homepage. Remove, redirect, consolidate, or improve it according to what the route actually represents.

Inspect what exists before JavaScript runs

View the raw response or disable JavaScript. Can you still find the product name, plain description, primary heading, important article text, and ordinary links to other public pages?

Google can render JavaScript, but rendering adds failure modes and other crawlers may behave differently. Google recommends server-side or static rendering rather than dynamic rendering as a long-term workaround for JavaScript visibility problems.

A page that initially serves an empty application shell asks every crawler to do more work. A page whose main content appears only after a click, scroll, permission prompt, or failed API call may never present its answer.

Render the stable public claim in HTML. Hydrate the interactive product around it.

Give the page a reason to exist

Technical access makes indexing possible. It does not make indexing deserved.

Each intended search page should answer one durable question better than the surrounding alternatives. Use a descriptive title and heading. State the answer early. Include concrete scope, examples, limits, ownership, and a useful next path.

Do not publish fifty location, integration, or comparison pages whose only difference is a substituted noun. A crawler may reach them all and still decide that none adds enough value.

This is also the foundation of SEO for AI search: stable source pages, explicit facts, accessible HTML, and consistent claims are easier for both search engines and agents to retrieve.

Create crawlable paths

Link the page from another public HTML page using an ordinary anchor with a real href. Menu items, topic hubs, documentation, and related articles all help discovery and give the URL context.

A URL that exists only in a JavaScript event, submitted form, private dashboard, email, or sitemap is structurally lonely. The sitemap can help discovery; it cannot explain why the page matters within the site.

For a small startup site, every indexable URL should have at least one sensible internal route leading to it.

Submit one clean sitemap

Include absolute canonical URLs that return success and are meant to be indexed. Exclude redirects, errors, duplicates, private pages, and noindex pages. Keep the file available at a stable location and reference it from robots.txt when useful.

Submit it in Search Console. Google’s recrawl guidance notes that sitemaps are especially useful for a newly launched site, while still making no promise of immediate inclusion.

Do not update lastmod unless the page changed meaningfully. A timestamp is evidence only when it is true.

Use this launch diagnosis

For one missing page, record:

  1. Final public URL and HTTP status.
  2. Search Console URL Inspection state.
  3. Live robots.txt result.
  4. Robots meta and X-Robots-Tag values.
  5. Declared and search-selected canonical.
  6. Main text present in the server response.
  7. Crawlable internal page linking to it.
  8. Sitemap inclusion using the same URL.
  9. Any redirect, soft 404, or rendering error.
  10. What unique question the page answers.

Fix contradictions first. Request another crawl after the fix. Then wait long enough to observe the result.

Indexing is not a launch switch. It is a chain of permissions, signals, and editorial choices. Make each link in that chain plain.