Spider Also known as Crawler or Robot, a spider is a search engine program that "crawls" the web, collecting data, following links, making copies of new and updated sites, and storing URLs in the search engine's Index. This allows search engines to provide faster and more up-to-date listings.