Feature: Allow to increase `WEBDRIVER_TIMEOUT`?

[olifre]

Currently, the WEBDRIVER_TIMEOUT seems hardcoded in dnstwist.

This causes flakiness with one of the pages I test, which sometimes has very complex javascript-rich content (sadly), and dnstwist errors out with:

fetching content from: http://uni-bonn.de/ > https://www.uni-bonn.de/en [110.4 KB]
rendering web page: http://uni-bonn.de/
Message: timeout: Timed out receiving message from renderer: 4.154
  (Session info: headless chrome=116.0.5845.180)

Would it be possible to add an option to increase this timeout when required?

(I hope this is the correct timeout I am looking at)

[olifre]

Note: In case you want to reproduce, this was executed using the latest official docker container (phash flavour), on GitHub actions. The same test worked fine a day before, so it seems to me that this is related to site complexity and server performance on the site end, which could match hitting a timeout.

[elceef]

Although I wasn't able to reproduce this error with that particular domain name, I agree this might be "fixed" with setting a longer timeout in the web driver. I will figure out a way to make it configurable - probably with environment variables.

[olifre]

Thanks! In fact, I also can not reproduce with that domain "now", it seems to be time-dependent — I likely hit a period of high load when running my tests. So having this configurable (environment variables is fine of course) for "flaky servers" seems like a good "solution".

[elceef]

After a year it's finally done (－‸ლ)

$ WEBDRIVER_PAGELOAD_TIMEOUT=0 ./dnstwist.py -f csv --phash ulikowski.pl
usage: ./dnstwist.py [OPTION]... DOMAIN
dnstwist.py: error: Message: timeout: Timed out receiving message from renderer: 0.000
  (Session info: chrome=135.0.7049.84)