From 19548d888e0bc6adec382f19680c10d37248d572 Mon Sep 17 00:00:00 2001 From: oskar Date: Wed, 5 Aug 2026 13:11:43 +0200 Subject: [PATCH] =?UTF-8?q?fix(kb-site):=20wycofaj=20robots.txt=20?= =?UTF-8?q?=E2=80=94=20blokowal=20legalny=20fetch=20z=20tokenem?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "User-agent: * / Disallow: /" odbijalo nie tylko wyszukiwarki, ale kazdego klienta respektujacego robots.txt (m.in. fetch asystentow AI) — takze takiego, ktory znal token dostepu. Token mial WPUSZCZAC znajacych go, a robots.txt ich WYPYCHAL. Dzialalo tez przeciwko wlasnemu celowi: crawler zablokowany przed pobraniem strony nigdy nie widzi meta noindex, a wyszukiwarka i tak potrafi wylistowac sam URL, ktorego nie wolno jej bylo pobrac. Ochrona przed indeksowaniem zostaje bez zmian: w kazdej strony (scripts/kb/gen_pages.py). Plik mial jedna linie polityki, wiec bez niej tracil sens — usuniety razem z bind-mountem w compose (katalog static/ zniknal jako pusty). Test: docker compose config (exit 0, zostaje tylko named volume); gen_pages.py --check — CZYSTO, 0 trafien. Co-Authored-By: Claude Opus 5 (1M context) --- kb/services/kb-site.md | 10 ++++++++-- services/kb-site/docker-compose.yml | 10 +++++----- services/kb-site/static/robots.txt | 2 -- 3 files changed, 13 insertions(+), 9 deletions(-) delete mode 100644 services/kb-site/static/robots.txt diff --git a/kb/services/kb-site.md b/kb/services/kb-site.md index ea3caa7..2dfe0f5 100644 --- a/kb/services/kb-site.md +++ b/kb/services/kb-site.md @@ -63,8 +63,14 @@ passes. Three layers keep the site out of casual sight: a query-parameter token enforced in the nginx/NPM layer (advanced config held in NPM only — the secret is not in -this repository), an unguessable subdomain, and `noindex, nofollow` on every page -alongside a blanket `robots.txt`. +this repository), an unguessable subdomain, and `noindex, nofollow` on every page. + +A blanket `robots.txt` was part of this and has been withdrawn. It turned away +every client that honours robots.txt — including the fetch tools of AI +assistants — even when the caller already held the access token, so it pushed +out precisely the readers the token exists to let in. It also worked against its +own goal: a crawler blocked before it can read the page never sees the `noindex` +tag, and a search engine may still list a bare URL it was forbidden to fetch. This is obscurity, not access control. A URL token is written to access logs, browser history and outbound `Referer` headers, so anyone who obtains a link diff --git a/services/kb-site/docker-compose.yml b/services/kb-site/docker-compose.yml index 660c97e..ba5adb6 100644 --- a/services/kb-site/docker-compose.yml +++ b/services/kb-site/docker-compose.yml @@ -15,11 +15,11 @@ services: # through the helper-container procedure in kb/runbooks/kb-site-deploy.md # (docker cp cannot write into a :ro mount). - kb-site_content:/usr/share/nginx/html:ro - # robots.txt nie pochodzi z generatora — nie jest dokumentem KB, więc - # jedzie z repo i nie ginie przy podmianie zawartości wolumenu. Mount - # pliku jest głębszy niż mount katalogu, więc Docker montuje go później - # i plik przykrywa (nieistniejące) robots.txt z wolumenu. - - ./static/robots.txt:/usr/share/nginx/html/robots.txt:ro + # Celowo brak mounta robots.txt: "Disallow: /" blokował nie tylko + # wyszukiwarki, ale każdy klient respektujący robots.txt (m.in. fetch + # asystentów AI) — także ten, który znał token dostępu. Za ochronę przed + # indeksowaniem odpowiada `noindex, nofollow` w każdej strony + # (scripts/kb/gen_pages.py). # Docker on PIHA has exhausted its default address pools (~30 live stacks: # "all predefined address pools have been fully subnetted"), so this stack # must not ask for a subnet of its own. Declaring a network here suppresses diff --git a/services/kb-site/static/robots.txt b/services/kb-site/static/robots.txt deleted file mode 100644 index 1f53798..0000000 --- a/services/kb-site/static/robots.txt +++ /dev/null @@ -1,2 +0,0 @@ -User-agent: * -Disallow: /