User-agent: *
# `Allow: /` is load-bearing. Do not remove it, do not narrow it, and never add
# a site-wide `Disallow:` — not even to "keep staging out of Google". Google
# only honours a `noindex` if it may FETCH the page, so a `Disallow` cannot
# deindex anything: it stops the crawl and strands any already-indexed URL in
# the index permanently, with no mechanism left to remove it. Deindexing a host
# means serving `X-Robots-Tag: noindex` AND keeping it crawlable. See O-4821.
#
# NOTE — there is NO `X-Robots-Tag: noindex` on any host (measured O-4982,
# 2026-07-30). An earlier version of this comment claimed non-production hosts
# set one; they never have. What actually keeps them out of the index is a
# self-referencing apex `rel="canonical"` on every page, plus staging never
# advertising itself (its sitemap is 100% apex URLs). Measured exposure is nil:
# `site:` on staging.marcora.ai, staging--marcora-main.netlify.app and
# marcora-main.netlify.app all returned 0 on Google (against 46 for
# site:marcora.ai, so the query path works), and nothing public links to the
# staging host.
#
# If any of those `site:` queries ever returns a real URL, that arrangement has
# stopped holding and a real `noindex` is needed. Emit it AT BUILD TIME behind a
# positive context allowlist (branch-deploy/deploy-preview/dev) — never
# `!== 'production'`, which would deindex the live site on any unknown or
# mistyped context. Not via netlify.toml: `[context.*.headers]` is silently
# ignored, and the top-level `[[headers]]` that does parse is GLOBAL, so it
# would deindex production. Keep `Allow: /` regardless — crawlability is a
# precondition of the fix, not a contradiction of it.
Allow: /

# THE DEMO / THEME PAGES ARE DELIBERATELY NOT LISTED HERE ANY MORE (O-6166).
#
# There used to be 22 `Disallow:` lines below this point — /demo/, /home-2
# through /home-10, /service, /team, /career, /portfolio, /faq, /coming-soon and
# the /single-* set. They were removed on purpose, and re-adding one would
# reverse the fix rather than reinforce it.
#
# They were doing the exact thing the comment above warns against. Those pages
# answer 200 and were reachable; the `Disallow` stopped Google fetching them,
# which is precisely what prevents Google from ever seeing a `noindex` on them.
# So the pages could be indexed but never DE-indexed — /home-2 was live on the
# apex serving `Crypto App | MarketCore` with no directive a crawler could act
# on. Their `'<path>': { index: false }` routeRules were inert (no
# @nuxtjs/robots installed), and DemoPageWrapper's `noindex` ran client-side
# only, behind an access check, so a crawler received nothing from either.
#
# What suppresses them now: server/middleware/noindex.ts sets
# `X-Robots-Tag: noindex, nofollow` on the response for every path declared
# `index: false` in routeRules — read back from that same declaration, so there
# is no second list to drift. DemoPageWrapper also emits the meta tag at setup
# scope now, so it is in the SSR'd HTML rather than applied after hydration.
#
# That arrangement REQUIRES these paths to stay crawlable. Deindexing means
# `noindex` + fetchable, never `Disallow` on its own — see the O-4821 paragraph
# at the top of this file.
#
# Adding a new page that should not be indexed? Give it
# `'<path>': { index: false, robots: false }` in nuxt.config.ts routeRules and
# nothing else. The middleware and the sitemap drift guard both read that key.

Sitemap: https://marcora.ai/sitemap.xml