root@oco:~$ curl https://www.facebook.com/robots.txt
* robots.txt is a simple text file placed in the root directory of a website
- it contains instructions in the form of "directives" that tell bots which parts of the website they can and cannot crawl
- may contain hidden directories
