# Crawlers that take and return nothing. AhrefsBot alone was 998 requests and # 72.9 CPU-seconds in a 31-minute window on 2026-09-09 — the third largest # consumer on the box, behind only Googlebot and Claude-SearchBot, and it feeds # no search or answer engine. A full `Disallow: /` group is safe to write here # precisely because it grants nothing: the group-replacement caveat below only # bites when a named group is meant to still inherit the `*` rules. User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: LinkupBot Disallow: / # AI search + answer engines are deliberately NOT given their own group. # A named `User-agent:` group replaces the `*` group for that bot, so listing # OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, BraveBot, DuckDuckBot # etc. with `Allow: /` would silently drop every Disallow below for them. # They fall through to `*` and are allowed everywhere a browser is. Training # crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, # Meta-ExternalAgent) are allowed on purpose — owner decision 2026-09-03: # maximum reach for free reference content. Add a `Disallow: /` group for one # of them here if that ever changes. # # No `Crawl-delay` group for any of them either, for the same structural reason # plus a cost one. Structural: a `Crawl-delay` group is still a named group, so # it would drop every Disallow below for that bot unless the whole list were # repeated inside it. Cost: after the 2026-09-10 prayer work an anonymous city # page is a disk read in Caddy, not an SSR — crawl volume on the family that # was 64.8% of prayer's CPU no longer buys anything back, while throttling an # answer engine costs reach directly. General-purpose throttling belongs at the # edge, where it sheds before the origin is touched: one free Cloudflare # rate-limit rule on /prayer/* for unverified bots. User-agent: * Allow: / # Home Disallow: /admin # Quran Disallow: /quran/playground Disallow: /quran/search Disallow: /quran/take-notes Disallow: /quran/product-updates # The offline app shell. Renders nothing (ssr = false) and redirects on the # client, so a non-rendering crawler would only ever see a blank document — # this is the line that actually keeps it out. Disallow: /quran/app-shell # Chat Disallow: /chat/admin # Blog Disallow: /blog/admin Disallow: /blog/api/ # Adhkar Disallow: /adhkar/search # Meet — ephemeral rooms; invite links get shared externally, never index them Disallow: /meet/r/ # Prayer — utility and embed surfaces. Every one of these is either a duplicate # of an indexed page or a frame/board with no standalone value, and all of them # currently accept crawls. City and country pages, and all seven locales of # them, stay fully crawlable: the 2026-09-10 program widened that surface to # ~492k advertised URLs on purpose. # # `/prayer/widget` is a prefix match, so it would also swallow `/prayer/widgets` # — a real, indexed page. The longer `Allow` wins under the longest-match rule # and puts it back. The `/prayer/*/…` twins cover the six locale-prefixed # spellings (`/prayer/ar/embed`, `/prayer/fr/pakistan/lahore/display`, …). Disallow: /prayer/widget Allow: /prayer/widgets Disallow: /prayer/*/widget Disallow: /prayer/embed Disallow: /prayer/*/embed # `/prayer/tracking` itself is indexed (since 2026-10-26): its signed-out view # documents what the tracker records, and that page is what ChatGPT cites for # tracker questions. Only the per-day history (`/prayer/tracking/2026-10-26`) # stays out — the trailing slash keeps the page itself crawlable. Disallow: /prayer/tracking/ Disallow: /prayer/*/tracking/ Disallow: /prayer/*/display # Retired 2026-08-21; answered by a 301 at the edge since 2026-09-10. Disallow: /prayer/og.png Disallow: /prayer/*/og.png # SvelteKit client-navigation payloads. These are the JSON twin of every page — # no rendering value, and Search Console measured JSON at 27% of all crawl # requests. No `$` anchor: the real URLs carry a `?x-sveltekit-invalidated=` # query string, which a `$` would let through. Nothing user-facing ends in # `__data.json`, so the unanchored form cannot over-match. Disallow: /*__data.json # Locale-prefixed spellings of this file and the sitemaps. Paraglide's # catch-all URL pattern makes `/ar/sitemap.xml` resolve to a byte-identical copy # of the English one (verified: 200, 63 KB), so every sitemap has 6 duplicate # addresses. Listed per-locale on purpose — a `/*/sitemap.xml` wildcard would # also match `/quran/sitemap.xml` and blackhole the real ones. Disallow: /ar/sitemap Disallow: /ur/sitemap Disallow: /id/sitemap Disallow: /tr/sitemap Disallow: /bn/sitemap Disallow: /fr/sitemap # The same duplicates one level down, inside every path-prefixed app: # `/listen/ar/sitemap.xml`, `/quran/ar/sitemap.xml`, `/prayer/ar/sitemap-index.xml` # … all verified 200 and byte-identical to their English originals. The # host-root lines above are prefix matches from `/`, so they do not reach these. # Requiring the literal `//sitemap` after the app segment is what keeps # `/quran/sitemap.xml` itself out of the match. Disallow: /*/ar/sitemap Disallow: /*/ur/sitemap Disallow: /*/id/sitemap Disallow: /*/tr/sitemap Disallow: /*/bn/sitemap Disallow: /*/fr/sitemap # NOTE: pages we want de-indexed are NOT listed above. A `Disallow` hides the # page from the crawler, which means the crawler never reads its `noindex` and # the URL can linger in the index indefinitely. De-indexing is done with # `noindex, follow` in the page head instead — see # packages/ui/src/seo/seo.svelte. # Sitemaps — one line per app entrypoint. robots.txt is only honoured at the # host root (qurangallery.com/robots.txt), so this single file is the # authoritative discovery list for every path-prefixed app. Add new apps here. # # Deliberately NOT collapsed behind one suite-wide index. A sitemap index may # only list sitemap FILES, never other index files, and quran, prayer, listen # and rijal each publish their own index — wrapping those in a parent index # would nest two levels deep and silently strand the four biggest surfaces on # the site. Flat list, one level each, is the supported shape. # # `/chat/sitemap.xml` is absent on purpose: chat is auth-gated on every route # except /share/[token], so its sitemap is a valid but empty urlset and # submitting it only earns a Search Console error row. Re-add it when chat # grows a public, server-rendered landing page. # The per-app `sitemap-articles.xml` files are absent for the same reason: # every one is an empty urlset until an app publishes an article. Re-add an # app's line when its file lists URLs. Sitemap: https://qurangallery.com/sitemap.xml Sitemap: https://qurangallery.com/quran/sitemap.xml Sitemap: https://qurangallery.com/prayer/sitemap-index.xml Sitemap: https://qurangallery.com/listen/sitemap.xml Sitemap: https://qurangallery.com/blog/sitemap.xml Sitemap: https://qurangallery.com/docs/sitemap.xml Sitemap: https://qurangallery.com/rijal/sitemap.xml Sitemap: https://qurangallery.com/adhkar/sitemap.xml Sitemap: https://qurangallery.com/mirath/sitemap.xml Sitemap: https://qurangallery.com/madrasah/sitemap.xml Sitemap: https://qurangallery.com/studio/sitemap.xml Sitemap: https://qurangallery.com/mcp/sitemap.xml Sitemap: https://qurangallery.com/maktabah/sitemap.xml Sitemap: https://accounts.qurangallery.com/sitemap.xml # Machine-readable site overview for LLM agents (llmstxt.org). # Not a robots directive; listed so agents that read robots.txt find it. # llms.txt: https://qurangallery.com/llms.txt