web crawling spiders aka robots.
Wikipedia
Architecture of a Web crawler
A Web crawler, sometimes called a spider, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).
Web search engines and some other sites use Web...