SPIDERING/WEB CRAWLING

#step 1: configure the browser to intercept traffic
root@oco:~$ BROWSER > Settings
 Search: Proxy
 
root@oco:~$ BROWSER > Network Settings > Settings
 Configure Proxy Access to the Internet
  Manual Proxy Configuration: enabled
  HTTP Proxy: 127.0.0.1
  Port: 8080 {default Burp listening port}
  Also Use this Proxy for HTTPS: enabled
  
 * it is advisable to also check the option of "Also use this proxy for FTP and HTTPS" IOT have all requests go through BurpSuite
 
step 2: #disable Burp Intercept
root@oco:~$ burpsuite &
BURP > Proxy > Intercept
 Intercept is on: disabled

 * Burp's intercept is enabled by default
 
step 3: passively crawl the target site
root@oco:~$ BROWSER > {targetSite:port}

 * can also simply refresh the page if already on the target site
 
step 4: identify pertinent pages in burpsuite
BURP > Target > Site Map > ...