oskar/homelab-codex-ws
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

fix(kb-site): wycofaj robots.txt — blokowal legalny fetch z tokenem

Browse source 
"User-agent: * / Disallow: /" odbijalo nie tylko wyszukiwarki, ale kazdego
klienta respektujacego robots.txt (m.in. fetch asystentow AI) — takze takiego,
ktory znal token dostepu. Token mial WPUSZCZAC znajacych go, a robots.txt ich
WYPYCHAL.

Dzialalo tez przeciwko wlasnemu celowi: crawler zablokowany przed pobraniem
strony nigdy nie widzi meta noindex, a wyszukiwarka i tak potrafi wylistowac
sam URL, ktorego nie wolno jej bylo pobrac.

Ochrona przed indeksowaniem zostaje bez zmian: <meta name="robots"
content="noindex, nofollow"> w <head> kazdej strony (scripts/kb/gen_pages.py).

Plik mial jedna linie polityki, wiec bez niej tracil sens — usuniety razem
z bind-mountem w compose (katalog static/ zniknal jako pusty).

Test: docker compose config (exit 0, zostaje tylko named volume);
gen_pages.py --check — CZYSTO, 0 trafien.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
oskar 2026-08-05 13:11:43 +02:00
parent 67e49a0952
commit 19548d888e
3 changed files with 13 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
kb/services/kb-site.md
View file

@ -63,8 +63,14 @@ passes.
Three layers keep the site out of casual sight: a query-parameter token enforced
in the nginx/NPM layer (advanced config held in NPM only — the secret is not in
this repository), an unguessable subdomain, and `noindex, nofollow` on every page
alongside a blanket `robots.txt`.
this repository), an unguessable subdomain, and `noindex, nofollow` on every page.
A blanket `robots.txt` was part of this and has been withdrawn. It turned away
every client that honours robots.txt — including the fetch tools of AI
assistants — even when the caller already held the access token, so it pushed
out precisely the readers the token exists to let in. It also worked against its
own goal: a crawler blocked before it can read the page never sees the `noindex`
tag, and a search engine may still list a bare URL it was forbidden to fetch.
This is obscurity, not access control. A URL token is written to access logs,
browser history and outbound `Referer` headers, so anyone who obtains a link

10
services/kb-site/docker-compose.yml
View file

@ -15,11 +15,11 @@ services:
# through the helper-container procedure in kb/runbooks/kb-site-deploy.md
# (docker cp cannot write into a :ro mount).
- kb-site_content:/usr/share/nginx/html:ro
# robots.txt nie pochodzi z generatora — nie jest dokumentem KB, więc
# jedzie z repo i nie ginie przy podmianie zawartości wolumenu. Mount
# pliku jest głębszy niż mount katalogu, więc Docker montuje go później
# i plik przykrywa (nieistniejące) robots.txt z wolumenu.
- ./static/robots.txt:/usr/share/nginx/html/robots.txt:ro
# Celowo brak mounta robots.txt: "Disallow: /" blokował nie tylko
# wyszukiwarki, ale każdy klient respektujący robots.txt (m.in. fetch
# asystentów AI) — także ten, który znał token dostępu. Za ochronę przed
# indeksowaniem odpowiada `noindex, nofollow` w <head> każdej strony
# (scripts/kb/gen_pages.py).
# Docker on PIHA has exhausted its default address pools (~30 live stacks:
# "all predefined address pools have been fully subnetted"), so this stack
# must not ask for a subnet of its own. Declaring a network here suppresses

2
services/kb-site/static/robots.txt
View file

@ -1,2 +0,0 @@
User-agent: *
Disallow: /