I try to keep an up-to-date list of all registered .de domains. Since the Denic does not publish the .de zone file, it “has” to be reverse engineered . In this article i am going to cover one approach i recently tested. I wanted to see how many .com domains are also registered as .de domains and vice versa.
Lets see the results:
Out of 15.123.523 registered .de Domains , 4.510.542 are also registered as .com domain, around 30% of all .de domains. Thats cool! By scanning the .com zonefile it is possible to reverse engineer one third of all registered .de Domains. Thats a nice start for further exploration.
Since i did not want to waste too much time and ressources with whois requests i relied on the DNS to check the status of each domain. Surely this is not as accurate as doing real whois querys, but it is ok to get the picture.
I took the .com zone file from 9th July 2012 and parsed a list of all registered slds in the .com zone to use as a foundation. Beeing equipped with a list of slds i was ready to start the hunt for existing .de domains. The “Powerdns Recursor” is a very powerful and fast resolver and a beautiful piece of software. It is used by ISPs around the world and easily handles thousands of parallel dns queries.
I fed the recursor with a custom C-Programm that takes a list of hostnames and tries to resolve them in parallel threads. Combined with Powerdns its pretty fast and gives me enough speed to handle millions of queries in an acceptable timeframe. I use this a lot, for example as a prefilter for availability / whois checks of large lists.
After some filtering out of the errors and edge cases 104.003.552 .com Domains have been checked for their .de counterparts. This is just one of many possible approaches, and i am probably going to cover some other ways to reconstruct the .de zone for fun and profit.
I hope you liked this article! Let me know if you want more of this stuff or have any suggestions!