Description
Path disclosure in robots.txt occurs when sensitive paths or directories are inadvertently exposed in the robots.txt file. The robots.txt file is used to instruct web robots on which parts of a website to avoid crawling or indexing. While intended for cooperation with search engine crawlers, disclosing sensitive paths can provide malicious actors with valuable information that could be exploited for unauthorized access or to identify potential attack vectors.
Recommendation
To mitigate the risk of path disclosure in robots.txt, carefully review and sanitize the contents of the file to avoid revealing sensitive paths or directories. Ensure that only necessary and safe paths are included in the robots.txt file, and regularly review and update it to remove any inadvertently disclosed information.
References
- Is your robots.txt file vulnerable? Here’s how to check and secure it
- Wikipedia: Robots exclusion standard
- CWE-200
- CAPEC-118
- OWASP 2021-A5