Your Search Results


    A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages.  Typically search engines use crawlers to build indexes.

    Learn more

    Web crawler on Wikipedia

    Document Tags and Contributors

    Contributors to this page: jsx, klez, Andrew_Pfeiffer, hbloomer
    Last updated by: Andrew_Pfeiffer,