web crawler
internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering)
en-GB/ˈwɛb ˈkɹɔːlə/
en-US/ˈwɛb ˈkɹɑlɚ/
internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering)