In February 1994 the web was small enough to keep all its robots in mind. They were a handful of programs jumping from link to link to build indexes, and some of them did it badly: they requested pages too fast and brought servers to their knees. Martijn Koster, then working for the British company Nexor, proposed a modest solution on the www-talk mailing list: a text file, placed at the root of the site, in which the owner writes which robots are not welcome and which paths should not be visited. The writer Charles Stross has said he provoked it, with a badly written crawler that overloaded Koster's own server.
The file was called robots.txt. By June it had become a de facto standard, honoured by the engines of the time, WebCrawler, Lycos, AltaVista. It became an IETF standard only in September 2022, as RFC 9309. For twenty-eight years it rested on one thing alone: the good manners of whoever reads it.
A sign, not a lock
The words in the file sound like commands: Allow, Disallow. But the protocol is only advisory. It cannot prevent anything; it asks. A serious search engine reads it before any other page and obeys. A malicious robot reads it for a different reason.
The concrete case is banal and frequent. A site wants to keep its admin area, a test copy and a backup folder out of search results. It writes Disallow: /old-admin/, Disallow: /backup/ in robots.txt. The file is public by definition: anyone can open it by typing the site's name followed by /robots.txt. So the list of rooms nobody was meant to see becomes the first page a curious visitor reads. RFC 9309 says it plainly: listing paths in robots.txt exposes them publicly and makes them discoverable; controlling access requires a real security measure, such as HTTP authentication. NIST, in its guide to server security, points the same way: system security should not depend on the secrecy of its implementation.
What the file really says
The myth here touches a sign hung on a door. "No entry" makes sense where there is also a lock, or a law, or a community that respects it. On its own, the sign does something else: it says where the door is. Robots.txt was born for a web where everyone knew each other and the problem was haste, not theft. It still works well for that: guiding crawlers, pointing to the sitemap, sparing the server. It becomes a risk when it is loaded with a job it never had.
That is why the site's Robots.txt Security Review asks the pasted file a simple question: is there anything in here you are trying to hide? Admin directories, staging, backups, configuration files: if they appear, they should be protected with authentication and server permissions, and then removed from the list. Robots.txt should stay short and boring. It is the same logic as Security as an Operating Habit: protection does not live in the formula that sounds like an order, but in the check that is repeated and the lock that is really there.
Whoever comes later and finds an old robots.txt will be able to read it as a map in negative: the rooms a web civilization wanted to keep in shadow, written in plain text on the front door.