Facebook

Why is there an issue in crawling the website even if it is not protected by any Authentication?

IP Whitelisting

Sometimes, the website's server firewall, defenders, and security plugins implemented may affect Elite Site Optimizer's crawler for crawling the website and providing reports on it. To avoid the situation, Elite Site Optimizer's crawler IPs should be whitelisted in the website's server firewall, defenders, and security plugins implemented for the website. 

Elite Site Optimizer's crawler IPs:

  • 52.43.194.129 
  • 54.201.178.165
  • 210.18.181.48
  • 123.201.136.234
  • 168.91.70.56
  • 168.91.70.67
  • 168.91.88.2
  • 168.91.88.94
  • 168.91.89.15
  • 168.91.90.192
  • 168.91.90.232
  • 168.91.91.174
  • 168.91.91.49
  • 172.102.209.53
  • 172.102.209.59
  • 172.84.112.207
  • 172.84.113.138
  • 172.84.113.248
  • 172.84.114.150
  • 172.84.114.6
  • 172.84.115.221
  • 172.84.115.235
  • 172.98.185.236
  • 172.98.185.89
  • 207.230.104.192
  • 207.230.104.5

The IPs mentioned should be whitelisted in all the website's server firewall, defenders, and security plugins for crawling the website without any issues.