We are actively working on crawling and saving all the domain data available on the web
Our database contains over 1.1 billion sites and expanding.
For some categories, we possess information even for 2011.
Who owns the data, owns the world
In the table below you can find categories of data collected, time period of the first entries and the date when the specified frequency of updates was reached for every category, as well as the update frequency itself.
For some categories there is more detailed description.
Website Extract
Learn what languages, plugins, CMS, etc. are used on any given site. Our crawlers will scan the pages for you in the blink of an eye helping you to conduct comprehensive research and collecting all the data about the website's systems.
| is_alive | Shows if domain is alive |
| robots.txt | |
| ads.txt | |
| humans.txt | |
| Sitemap | |
| CMS | Apps categories* |
| Geo IP data | Country code from GeoIP database |
| City | |
| Program | Google Analytics, Google Ads, Yandex ID, etc. |
| SSL cert | All data stored in SSL cert: – issuer – subject – start date – expire date – authority information access – rsa public key |
| HTML index page tags | “<title> tag <meta> tags <h1> tag text <html lang=X> value <link> tags” |
| Apps | See the list of collected categories below |
| Autonomous System Number | |
| Autonomous System Organization | |
| DNS | A, AAA, AAAA, CNAME, MX, NS, SOA, TXT |
≤ 14 days
Ownership
Find all needed information regarding domain ownership with our solutions. From the name of the entity that owns a domain name to the date of re-registration or expiration - we will help you to get data for the benefit of your partners.
| Domain registration date |
| Domain expiration date |
| Domain drop date |
| Domain last update date |
| Domain transfer date |
| Registrar ID |
| Registrar |
| Registrant |
≤ 45 days
Internet Time-Machine
Discover how the website looked through its history. Taking snapshots of every webpage on the Internet, we have created a tool that helps your clients to see what were the topic and content on the site and to make the decision about the trustworthiness of a partner.
| Website snapshot | Update frequency is not stable yet without new servers |
N/A
CIPA “Apps” categories list
- Advertising
- CDN
- Caching
- Databases
- Font scripts
- JavaScript frameworks
- LMS
- Marketing automation
- Mobile frameworks
- Page builders
- UI frameworks
- Web servers
- Wikis
- Analytics
- CMS
- Captchas
- Development
- Hosting panels
- JavaScript graphics
- Load balancers
- Message boards
- Operating systems
- Programming languages
- Web frameworks
- Webmail
- Blogs
- CRM
- DMS
- Ecommerce
- Issue trackers
- JavaScript libraries
- Maps
- Miscellaneous
- PaaS
- Reverse proxies
- Web server extensions
- Widgets
Our goal is to download all the Internet
Due to regular updates of data you may track the changes of various parameters of a given domain, even see a snapshot of the website on a particular day.
More filtering possibilities are coming soon.
More filtering possibilities are coming soon.
Scale your business wisely with data from Domain Crawler
Request a demo
Ready to start your journey in the world of data?
Contact us and we’ll show all the benefits
and perks of Domain Crawler
