diff --git a/kb/services/kb-site.md b/kb/services/kb-site.md index ea3caa7..2dfe0f5 100644 --- a/kb/services/kb-site.md +++ b/kb/services/kb-site.md @@ -63,8 +63,14 @@ passes. Three layers keep the site out of casual sight: a query-parameter token enforced in the nginx/NPM layer (advanced config held in NPM only — the secret is not in -this repository), an unguessable subdomain, and `noindex, nofollow` on every page -alongside a blanket `robots.txt`. +this repository), an unguessable subdomain, and `noindex, nofollow` on every page. + +A blanket `robots.txt` was part of this and has been withdrawn. It turned away +every client that honours robots.txt — including the fetch tools of AI +assistants — even when the caller already held the access token, so it pushed +out precisely the readers the token exists to let in. It also worked against its +own goal: a crawler blocked before it can read the page never sees the `noindex` +tag, and a search engine may still list a bare URL it was forbidden to fetch. This is obscurity, not access control. A URL token is written to access logs, browser history and outbound `Referer` headers, so anyone who obtains a link diff --git a/services/kb-site/docker-compose.yml b/services/kb-site/docker-compose.yml index 660c97e..ba5adb6 100644 --- a/services/kb-site/docker-compose.yml +++ b/services/kb-site/docker-compose.yml @@ -15,11 +15,11 @@ services: # through the helper-container procedure in kb/runbooks/kb-site-deploy.md # (docker cp cannot write into a :ro mount). - kb-site_content:/usr/share/nginx/html:ro - # robots.txt nie pochodzi z generatora — nie jest dokumentem KB, więc - # jedzie z repo i nie ginie przy podmianie zawartości wolumenu. Mount - # pliku jest głębszy niż mount katalogu, więc Docker montuje go później - # i plik przykrywa (nieistniejące) robots.txt z wolumenu. - - ./static/robots.txt:/usr/share/nginx/html/robots.txt:ro + # Celowo brak mounta robots.txt: "Disallow: /" blokował nie tylko + # wyszukiwarki, ale każdy klient respektujący robots.txt (m.in. fetch + # asystentów AI) — także ten, który znał token dostępu. Za ochronę przed + # indeksowaniem odpowiada `noindex, nofollow` w każdej strony + # (scripts/kb/gen_pages.py). # Docker on PIHA has exhausted its default address pools (~30 live stacks: # "all predefined address pools have been fully subnetted"), so this stack # must not ask for a subnet of its own. Declaring a network here suppresses diff --git a/services/kb-site/static/robots.txt b/services/kb-site/static/robots.txt deleted file mode 100644 index 1f53798..0000000 --- a/services/kb-site/static/robots.txt +++ /dev/null @@ -1,2 +0,0 @@ -User-agent: * -Disallow: /