Data Points

Who owns the data, owns the world
We are actively working on crawling and saving all the domain data available on the web
Our database contains over 1.1 billion sites and expanding. For some categories, we possess information even for 2011.
domain crawler_icon_idea
Who owns the data, owns the world

In the table below you can find categories of data collected, time period of the first entries and the date when the specified frequency of updates was reached for every category, as well as the update frequency itself.

For some categories there is more detailed description.

Website Extract

Learn what languages, plugins, CMS, etc. are used on any given site. Our crawlers will scan the pages for you in the blink of an eye helping you to conduct comprehensive research and collecting all the data about the website's systems.
is_aliveShows if domain is alive
robots.txt 
ads.txt 
humans.txt 
Sitemap 
CMSApps categories*
Geo IP dataCountry code from GeoIP database
City 
ProgramGoogle Analytics, Google Ads, Yandex ID, etc.
SSL certAll data stored in SSL cert:
– issuer
– subject
– start date
– expire date
– authority information access
– rsa public key
HTML index page tags“<title> tag
<meta> tags
<h1> tag text
<html lang=X> value
<link> tags”
AppsSee the list of collected categories below
Autonomous System Number 
Autonomous System Organization 
DNSA, AAA, AAAA, CNAME, MX, NS, SOA, TXT

≤ 14 days

Ownership

Find all needed information regarding domain ownership with our solutions. From the name of the entity that owns a domain name to the date of re-registration or expiration - we will help you to get data for the benefit of your partners.
Domain registration date
Domain expiration date
Domain drop date
Domain last update date
Domain transfer date
Registrar ID
Registrar
Registrant

≤ 45 days

Internet Time-Machine

Discover how the website looked through its history. Taking snapshots of every webpage on the Internet, we have created a tool that helps your clients to see what were the topic and content on the site and to make the decision about the trustworthiness of a partner.
Website snapshotUpdate frequency is not stable yet without
new servers
CIPA “Apps” categories list
  • Advertising
  • CDN
  • Caching
  • Databases
  • Font scripts
  • JavaScript frameworks
  • LMS
  • Marketing automation
  • Mobile frameworks
  • Page builders
  • UI frameworks
  • Web servers
  • Wikis
  • Analytics
  • CMS
  • Captchas
  • Development
  • Hosting panels
  • JavaScript graphics
  • Load balancers
  • Message boards
  • Operating systems
  • Programming languages
  • Web frameworks
  • Webmail
  • Blogs
  • CRM
  • DMS
  • Ecommerce
  • Issue trackers
  • JavaScript libraries
  • Maps
  • Miscellaneous
  • PaaS
  • Reverse proxies
  • Web server extensions
  • Widgets
domain crawler_icon_globe
Our goal is to download all the Internet
Due to regular updates of data you may track the changes of various parameters of a given domain, even see a snapshot of the website on a particular day.
More filtering possibilities are coming soon.
Scale your business wisely with data from Domain Crawler
Request a demo

Ready to start your journey in the world of data?
Contact us and we’ll show all the benefits
and perks of Domain Crawler