Join MDN and developers like you at Mozilla's View Source conference, 12-14 September in Berlin, Germany. Learn more at https://viewsourceconf.org

A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages.  Typically search engines use crawlers to build indexes.

Learn more

Document Tags and Contributors

 Contributors to this page: PetiPandaRou, Andrew_Pfeiffer, klez, hbloomer, jsx
 Last updated by: PetiPandaRou,