Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of 145 TB of data from 1.81 billion webpages as of August 2015. It completes four crawls a year.Common Crawl was founded by Gil Elbaz. Advisors to the non-profit include Peter Norvig and Joi Ito. The organization's crawlers respect nofollow and robots.txt policies.
| Attributes | Values |
|---|---|
| rdfs:comment |
|
| foaf:name |
|
| foaf:homepage | |
| founded by | |
| key person | |
| language | |
| location | |
| type |