DNS Engine - Offload DNS to Dedicated Process

[TheTechromancer]

This PR adds an abstract Engine system in BBOT which uses pyzmq in ROUTER/DEALER mode to farm out resource-intensive tasks to a dedicated process. Pyzmq communicates using binary data over a local unix socket, which is very fast.

The DNS Engine is built on top of the abstract Engine class, but the same class should be able to be used for HTTP.

Original issue: #1223

Speed Benchmark

Command:

bbot -t tesla.com -f subdomain-enum -rf passive -em massdns

dev: 1 minute 57 seconds

dns-engine: 1 minute 0 seconds

TODO:

• remove massdns queue logic added when dns parallelization was nerfed
• ensure processes are always cleaned up properly
• ensure zmq sockets are always cleaned up properly
• Only start process when needed
• reintroduce dns rate limit (or let it run free)?

Will be merged into #1058.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 86.97027% with 206 lines in your changes are missing coverage. Please review.

Project coverage is 91%. Comparing base (1dd8ab2) to head (18694c1).

Files	Patch %	Lines
bbot/core/engine.py	57%	71 Missing ⚠️
bbot/scanner/scanner.py	61%	58 Missing ⚠️
bbot/core/core.py	53%	11 Missing ⚠️
bbot/core/helpers/dns/engine.py	91%	11 Missing ⚠️
bbot/modules/base.py	90%	9 Missing ⚠️
bbot/core/helpers/dns/mock.py	84%	6 Missing ⚠️
bbot/modules/massdns.py	73%	6 Missing ⚠️
bbot/modules/internal/dns.py	98%	5 Missing ⚠️
bbot/core/config/logger.py	88%	4 Missing ⚠️
bbot/core/helpers/dns/dns.py	95%	4 Missing ⚠️
... and 11 more

Additional details and impacted files

@@           Coverage Diff            @@
##           presets   #1231    +/-   ##
========================================
- Coverage       92%     91%    -0%     
========================================
  Files          316     325     +9     
  Lines        21213   21611   +398     
========================================
+ Hits         19374   19662   +288     
- Misses        1839    1949   +110     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stryker2k2]

Author walked me through this Pull Request. All questions were answered. Code looks good. Test(s) look legit. Realtime BBOT Demo of multi-processor usage was beneficial.