WellKnownBot
The well-known resource scans are performed by WellKnownBot. All requests contain a User-Agent header that includes the string WellKnownBot and the URL of this page.While WellKnownBot makes automated requests, it is not a web crawler/spider. It only ever requests a small number of specific resources that are intended for public, programmatic consumption. | Because it is not a crawler, WellKnownBot does not follow generic User-Agent: * crawling rules in robots.txt files. However, it does obey robots.txt rules in groups that specifically target it.If you wish to restrict the resources WellKnownBot scans, add a User-Agent: WellKnownBot group to your site’s robots.txt. |
