Testing domain-name availability with pythonwhois

Question

I'm using successfully pythonwhois (installed with pip install ...) to check the availability of .com domains:

import pythonwhois
for domain in ['aaa.com', 'bbb.com', ...]:
    details = pythonwhois.get_whois(domain)
    if 'No match for' in str(details):   # simple but it works!
        print domain 

But:

• it is a little bit slow (2 requests per second on average)
• won't I be blacklisted by the whois server if I do 26*26*26 ~ 17000 requests?
  (I'm testing availability of ???mail.com with ? being a..z)

Question: is there a better way to check availability than doing one whois request per domain?

---

Edit: The job finished in 9572 seconds, and here is the full list of all domains available of the form ???mail.com, as of November 2017, if anyone is interested to start an email service!

Answer 1

You should parallelize what you are doing. Since most of the time spent by your function is waiting, you can verify a lot of works at once (not limited to your number of processors). Example:

import pythonwhois
from joblib import Parallel, delayed, cpu_count
n_jobs = 100 # works in parallel
def f(domain):
    details = pythonwhois.get_whois(domain)
    if 'No match for' in str(details):   # simple but it works!
        print(domain)
        return domain
    else:
        return None

domains= ['aaa.com', 'bbb.com', 'ccc.com', 'bbbaohecraoea.com']
result = Parallel(n_jobs=n_jobs, verbose=10)(delayed(f)(domain) for domain in domains)

# create a list with the available domains
available_domains=[domains[idx] for idx,r in enumerate(result) if r!=None]
print(available_domains)
# Result
# ['bbbaohecraoea.com']