rm stem_examples
This commit is contained in:
parent
6029ebfd77
commit
58937cfe7f
@ -1,37 +0,0 @@
|
|||||||
up:: up.phantompy up.badexits
|
|
||||||
|
|
||||||
up.phantompy::
|
|
||||||
cp -up phantompy.md /o/var/local/src/phantompy.git/README.md
|
|
||||||
cp -up phantompy.setup /o/var/local/src/phantompy.git/setup.py
|
|
||||||
cp -up setup.cfg lookupdns.py qasync_phantompy.py phantompy.py support_phantompy.py \
|
|
||||||
/o/var/local/src/phantompy.git/
|
|
||||||
|
|
||||||
up.badexits:: refresh
|
|
||||||
cp -up exclude_badExits.md /o/var/local/src/exclude_badExits.git/README.md
|
|
||||||
cp -up setup.cfg exclude_badExits.py exclude_badExits.bash \
|
|
||||||
support_onions.py trustor_poc.py \
|
|
||||||
/o/var/local/src/exclude_badExits.git
|
|
||||||
|
|
||||||
lint.phantompy::
|
|
||||||
/var/local/bin/pydev_flake8.bash lookupdns.py qasync_phantompy.py phantompy.py support_phantompy.py
|
|
||||||
|
|
||||||
lint.badexits::
|
|
||||||
/var/local/bin/pydev_flake8.bash exclude_badExits.py \
|
|
||||||
support_onions.py trustor_poc.py
|
|
||||||
isort -c -diff exclude_badExits.py \
|
|
||||||
support_onions.py trustor_poc.py
|
|
||||||
|
|
||||||
lint:: lint.badexits lint.phantompy
|
|
||||||
sh .pylint.sh
|
|
||||||
|
|
||||||
refresh::
|
|
||||||
/var/local/bin/python3.bash -c \
|
|
||||||
'import exclude_badExits; print(exclude_badExits.__doc__)' \
|
|
||||||
> exclude_badExits.md
|
|
||||||
echo "\n## Usage \n\`\`\`\n" \
|
|
||||||
>> exclude_badExits.md
|
|
||||||
/var/local/bin/python3.bash exclude_badExits.py --help \
|
|
||||||
| sed -e '/^[^uo ]/d' \
|
|
||||||
>> exclude_badExits.md
|
|
||||||
echo "\n\`\`\`\n" \
|
|
||||||
>> exclude_badExits.md
|
|
||||||
@ -1,113 +0,0 @@
|
|||||||
This extends nusenu's basic idea of using the stem library to
|
|
||||||
dynamically exclude nodes that are likely to be bad by putting them
|
|
||||||
on the ExcludeNodes or ExcludeExitNodes setting of a running Tor.
|
|
||||||
* https://github.com/nusenu/noContactInfo_Exit_Excluder
|
|
||||||
* https://github.com/TheSmashy/TorExitRelayExclude
|
|
||||||
|
|
||||||
The basic idea is to exclude Exit nodes that do not have ContactInfo:
|
|
||||||
* https://github.com/nusenu/ContactInfo-Information-Sharing-Specification
|
|
||||||
|
|
||||||
That can be extended to relays that do not have an email in the contact,
|
|
||||||
or to relays that do not have ContactInfo that is verified to include them.
|
|
||||||
But there's a problem, and your Tor notice.log will tell you about it:
|
|
||||||
you could exclude the relays needed to access hidden services or mirror
|
|
||||||
directories. So we need to add to the process the concept of a whitelist.
|
|
||||||
In addition, we may have our own blacklist of nodes we want to exclude,
|
|
||||||
or use these lists for other applications like selektor.
|
|
||||||
|
|
||||||
So we make two files that are structured in YAML:
|
|
||||||
```
|
|
||||||
/etc/tor/yaml/torrc-goodnodes.yaml
|
|
||||||
{sGOOD_NODES}
|
|
||||||
|
|
||||||
By default all sections of the goodnodes.yaml are used as a whitelist.
|
|
||||||
|
|
||||||
Use the GoodNodes/Onions list to list onion services you want the
|
|
||||||
Introduction Points whitelisted - these points may change daily
|
|
||||||
Look in tor's notice.log for warnings of 'Every introduction point for service'
|
|
||||||
|
|
||||||
```--hs_dir``` ```default='/var/lib/tor'``` will make the program
|
|
||||||
parse the files named ```hostname``` below this dir to find
|
|
||||||
Hidden Services to whitelist.
|
|
||||||
|
|
||||||
The Introduction Points can change during the day, so you may want to
|
|
||||||
rerun this program to freshen the list of Introduction Points. A full run
|
|
||||||
that processes all the relays from stem can take 30 minutes, or run with:
|
|
||||||
|
|
||||||
```--saved_only``` will run the program with just cached information
|
|
||||||
on the relats, but will update the Introduction Points from the Services.
|
|
||||||
|
|
||||||
/etc/tor/yaml/torrc-badnodes.yaml
|
|
||||||
{sBAD_NODES}
|
|
||||||
```
|
|
||||||
That part requires [PyYAML](https://pyyaml.org/wiki/PyYAML)
|
|
||||||
https://github.com/yaml/pyyaml/ or ```ruamel```: do
|
|
||||||
```pip3 install ruamel``` or ```pip3 install PyYAML```;
|
|
||||||
the advantage of the former is that it preserves comments.
|
|
||||||
|
|
||||||
(You may have to run this as the Tor user to get RW access to
|
|
||||||
/run/tor/control, in which case the directory for the YAML files must
|
|
||||||
be group Tor writeable, and its parent's directories group Tor RX.)
|
|
||||||
|
|
||||||
Because you don't want to exclude the introduction points to any onion
|
|
||||||
you want to connect to, ```--white_onions``` should whitelist the
|
|
||||||
introduction points to a comma sep list of onions; we fixed stem to do this:
|
|
||||||
* https://github.com/torproject/stem/issues/96
|
|
||||||
* https://gitlab.torproject.org/legacy/trac/-/issues/25417
|
|
||||||
|
|
||||||
Use the GoodNodes/Onions list in goodnodes.yaml to list onion services
|
|
||||||
you want the Introduction Points whitelisted - these points may change daily.
|
|
||||||
Look in tor's notice.log for 'Every introduction point for service'
|
|
||||||
|
|
||||||
```notice_log``` will parse the notice log for warnings about relays and
|
|
||||||
services that will then be whitelisted.
|
|
||||||
|
|
||||||
```--torrc``` will read a file like /etc/tor/torrc and make some
|
|
||||||
suggestions based on what it finds; it will not edit or change the file.
|
|
||||||
|
|
||||||
```--torrc_output``` will write the torrc ExcludeNodes configuration to a file.
|
|
||||||
|
|
||||||
```--good_contacts``` will write the contact info as a ciiss dictionary
|
|
||||||
to a YAML file. If the proof is uri-rsa, the well-known file of fingerprints
|
|
||||||
is downloaded and the fingerprints are added on a 'fps' field we create
|
|
||||||
of that fingerprint's entry of the YAML dictionary. This file is read at the
|
|
||||||
beginning of the program to start with a trust database, and only new
|
|
||||||
contact info from new relays are added to the dictionary.
|
|
||||||
|
|
||||||
Now for the final part: we lookup the Contact info of every relay
|
|
||||||
that is currently in our Tor, and check it the existence of the
|
|
||||||
well-known file that lists the fingerprints of the relays it runs.
|
|
||||||
If it fails to provide the well-know url, we assume its a bad
|
|
||||||
relay and add it to a list of nodes that goes on ```ExcludeNodes```
|
|
||||||
(not just ExcludeExitNodes```). If the Contact info is good, we add the
|
|
||||||
list of fingerprints to ```ExitNodes```, a whitelist of relays to use as exits.
|
|
||||||
|
|
||||||
```--bad_on``` We offer the users 3 levels of cleaning:
|
|
||||||
1. clean relays that have no contact ```=Empty```
|
|
||||||
2. clean relays that don't have an email in the contact (implies 1)
|
|
||||||
```=Empty,NoEmail```
|
|
||||||
3. clean relays that don't have "good' contactinfo. (implies 1)
|
|
||||||
```=Empty,NoEmail,NotGood```
|
|
||||||
|
|
||||||
The default is ```Empty,NoEmail,NotGood``` ; ```NoEmail``` is inherently imperfect
|
|
||||||
in that many of the contact-as-an-email are obfuscated, but we try anyway.
|
|
||||||
|
|
||||||
To be "good" the ContactInfo must:
|
|
||||||
1. have a url for the well-defined-file to be gotten
|
|
||||||
2. must have a file that can be gotten at the URL
|
|
||||||
3. must support getting the file with a valid SSL cert from a recognized authority
|
|
||||||
4. (not in the spec but added by Python) must use a TLS SSL > v1
|
|
||||||
5. must have a fingerprint list in the file
|
|
||||||
6. must have the FP that got us the contactinfo in the fingerprint list in the file.
|
|
||||||
|
|
||||||
```--wait_boot``` is the number of seconds to wait for Tor to booststrap
|
|
||||||
|
|
||||||
```--wellknown_output``` will make the program write the well-known files
|
|
||||||
(```/.well-known/tor-relay/rsa-fingerprint.txt```) to a directory.
|
|
||||||
|
|
||||||
```--relays_output write the download relays in json to a file. The relays
|
|
||||||
are downloaded from https://onionoo.torproject.org/details
|
|
||||||
|
|
||||||
For usage, do ```python3 exclude_badExits.py --help`
|
|
||||||
See [exclude_badExits.txt](./exclude_badExits.txt)
|
|
||||||
|
|
||||||
@ -1,55 +0,0 @@
|
|||||||
# -*-mode: python; py-indent-offset: 2; indent-tabs-mode: nil; coding: utf-8-unix -*-
|
|
||||||
|
|
||||||
# http://vt5hknv6sblkgf22.onion/tutorials/examples/check_digests.html
|
|
||||||
import sys
|
|
||||||
|
|
||||||
import stem.descriptor.remote
|
|
||||||
import stem.util.tor_tools
|
|
||||||
|
|
||||||
|
|
||||||
def download_descriptors(fingerprint):
|
|
||||||
"""
|
|
||||||
Downloads the descriptors we need to validate this relay. Downloads are
|
|
||||||
parallelized, providing the caller with a tuple of the form...
|
|
||||||
|
|
||||||
(router_status_entry, server_descriptor, extrainfo_descriptor)
|
|
||||||
"""
|
|
||||||
|
|
||||||
conensus_query = stem.descriptor.remote.get_consensus()
|
|
||||||
server_desc_query = stem.descriptor.remote.get_server_descriptors(fingerprint)
|
|
||||||
extrainfo_query = stem.descriptor.remote.get_extrainfo_descriptors(fingerprint)
|
|
||||||
|
|
||||||
router_status_entries = filter(lambda desc: desc.fingerprint == fingerprint, conensus_query.run())
|
|
||||||
|
|
||||||
if len(router_status_entries) != 1:
|
|
||||||
raise IOError("Unable to find relay '%s' in the consensus" % fingerprint)
|
|
||||||
|
|
||||||
return (
|
|
||||||
router_status_entries[0],
|
|
||||||
server_desc_query.run()[0],
|
|
||||||
extrainfo_query.run()[0],
|
|
||||||
)
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
fingerprint = input("What relay fingerprint would you like to validate?\n")
|
|
||||||
print('') # blank line
|
|
||||||
|
|
||||||
if not stem.util.tor_tools.is_valid_fingerprint(fingerprint):
|
|
||||||
print("'%s' is not a valid relay fingerprint" % fingerprint)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
try:
|
|
||||||
router_status_entry, server_desc, extrainfo_desc = download_descriptors(fingerprint)
|
|
||||||
except Exception as exc:
|
|
||||||
print(exc)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if router_status_entry.digest == server_desc.digest():
|
|
||||||
print("Server descriptor digest is correct")
|
|
||||||
else:
|
|
||||||
print("Server descriptor digest invalid, expected %s but is %s" % (router_status_entry.digest, server_desc.digest()))
|
|
||||||
|
|
||||||
if server_desc.extra_info_digest == extrainfo_desc.digest():
|
|
||||||
print("Extrainfo descriptor digest is correct")
|
|
||||||
else:
|
|
||||||
print("Extrainfo descriptor digest invalid, expected %s but is %s" % (server_desc.extra_info_digest, extrainfo_desc.digest()))
|
|
||||||
@ -1,51 +0,0 @@
|
|||||||
import collections
|
|
||||||
|
|
||||||
import stem.descriptor
|
|
||||||
import stem.descriptor.remote
|
|
||||||
import stem.directory
|
|
||||||
|
|
||||||
# Query all authority votes asynchronously.
|
|
||||||
|
|
||||||
downloader = stem.descriptor.remote.DescriptorDownloader(
|
|
||||||
document_handler = stem.descriptor.DocumentHandler.DOCUMENT,
|
|
||||||
)
|
|
||||||
|
|
||||||
# An ordered dictionary ensures queries are finished in the order they were
|
|
||||||
# added.
|
|
||||||
|
|
||||||
queries = collections.OrderedDict()
|
|
||||||
|
|
||||||
for name, authority in stem.directory.Authority.from_cache().items():
|
|
||||||
if authority.v3ident is None:
|
|
||||||
continue # authority doesn't vote if it lacks a v3ident
|
|
||||||
|
|
||||||
queries[name] = downloader.get_vote(authority)
|
|
||||||
|
|
||||||
# Wait for the votes to finish being downloaded, this produces a dictionary of
|
|
||||||
# authority nicknames to their vote.
|
|
||||||
|
|
||||||
votes = dict((name, query.run()[0]) for (name, query) in queries.items())
|
|
||||||
|
|
||||||
# Get a superset of all the fingerprints in all the votes.
|
|
||||||
|
|
||||||
all_fingerprints = set()
|
|
||||||
|
|
||||||
for vote in votes.values():
|
|
||||||
all_fingerprints.update(vote.routers.keys())
|
|
||||||
|
|
||||||
# Finally, compare moria1's votes to maatuska's votes.
|
|
||||||
|
|
||||||
for fingerprint in all_fingerprints:
|
|
||||||
moria1_vote = votes['moria1'].routers.get(fingerprint)
|
|
||||||
maatuska_vote = votes['maatuska'].routers.get(fingerprint)
|
|
||||||
|
|
||||||
if not moria1_vote and not maatuska_vote:
|
|
||||||
print("both moria1 and maatuska haven't voted about %s" % fingerprint)
|
|
||||||
elif not moria1_vote:
|
|
||||||
print("moria1 hasn't voted about %s" % fingerprint)
|
|
||||||
elif not maatuska_vote:
|
|
||||||
print("maatuska hasn't voted about %s" % fingerprint)
|
|
||||||
elif 'Running' in moria1_vote.flags and 'Running' not in maatuska_vote.flags:
|
|
||||||
print("moria1 has the Running flag but maatuska doesn't: %s" % fingerprint)
|
|
||||||
elif 'Running' in maatuska_vote.flags and 'Running' not in moria1_vote.flags:
|
|
||||||
print("maatuska has the Running flag but moria1 doesn't: %s" % fingerprint)
|
|
||||||
@ -1,25 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# -*- mode: sh; fill-column: 75; tab-width: 8; coding: utf-8-unix -*-
|
|
||||||
ROLE=toxcore
|
|
||||||
|
|
||||||
PROG=exclude_badExits
|
|
||||||
build=build
|
|
||||||
dist=dist
|
|
||||||
# pyinstaller
|
|
||||||
if [ ! -e ${dist}/${PROG}.pyi -o ! ${dist}/${PROG}.pyi -nt ./${PROG}.py ] ; then
|
|
||||||
[ -f ${PROG}.spec ] || pyi-makespec ./${PROG}.py -F -c
|
|
||||||
[ -d ${build} ] || mkdir -p ${build}
|
|
||||||
[ -d ${dist} ] || mkdir -p ${dist}
|
|
||||||
[ -e ${dist}/${PROG}.pyi -a ${dist}/${PROG}.pyi -nt ./${PROG}.py ] || \
|
|
||||||
pyinstaller --distpath ${dist} --workpath ${build} \
|
|
||||||
--exclude tkinter --exclude matplotlib \
|
|
||||||
--exclude twisted --exclude jedi --exclude jaraco \
|
|
||||||
--exclude sphinx --exclude coverage --exclude nose \
|
|
||||||
--exclude PIL --exclude numpy --exclude OpenGL \
|
|
||||||
--exclude PySide2 --exclude PyQt5 --exclude IPython \
|
|
||||||
--onefile -c --ascii \
|
|
||||||
$PROG.py
|
|
||||||
# AttributeError: 'NoneType' object has no attribute 'groups'
|
|
||||||
# utils.py #400
|
|
||||||
fi
|
|
||||||
# cx_Freeze exclude_badExits.py
|
|
||||||
@ -1,26 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# -*- mode: sh; fill-column: 75; tab-width: 8; coding: utf-8-unix -*-
|
|
||||||
|
|
||||||
ROLE=toxcore
|
|
||||||
|
|
||||||
PROG=exclude_badExits
|
|
||||||
build=build
|
|
||||||
dist=dist
|
|
||||||
# pyinstaller
|
|
||||||
if [ ! -e ${dist}/${PROG}.pyi -o ! ${dist}/${PROG}.pyi -nt ./${PROG}.py ] ; then
|
|
||||||
[ -f ${PROG}.spec ] || pyi-makespec ./${PROG}.py -F -c
|
|
||||||
[ -d ${build} ] || mkdir -p ${build}
|
|
||||||
[ -d ${dist} ] || mkdir -p ${dist}
|
|
||||||
[ -e ${dist}/${PROG}.pyi -a ${dist}/${PROG}.pyi -nt ./${PROG}.py ] || \
|
|
||||||
pyinstaller --distpath ${dist} --workpath ${build} \
|
|
||||||
--exclude tkinter --exclude matplotlib \
|
|
||||||
--exclude twisted --exclude jedi --exclude jaraco \
|
|
||||||
--exclude sphinx --exclude coverage --exclude nose \
|
|
||||||
--exclude PIL --exclude numpy --exclude OpenGL \
|
|
||||||
--exclude PySide2 --exclude PyQt5 --exclude IPython \
|
|
||||||
--onefile -c --ascii \
|
|
||||||
$PROG.py
|
|
||||||
# AttributeError: 'NoneType' object has no attribute 'groups'
|
|
||||||
# utils.py #400
|
|
||||||
fi
|
|
||||||
# cx_Freeze exclude_badExits.py
|
|
||||||
@ -1,42 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
# -*- mode: sh; fill-column: 75; tab-width: 8; coding: utf-8-unix -*-
|
|
||||||
|
|
||||||
PROG=exclude_badExits.py
|
|
||||||
SOCKS_PORT=9050
|
|
||||||
CAFILE=/etc/ssl/certs/ca-certificates.crt
|
|
||||||
ROLE=toxcore
|
|
||||||
|
|
||||||
# an example of running exclude_badExits with full debugging
|
|
||||||
# expected to take an hour or so
|
|
||||||
declare -a LARGS
|
|
||||||
LARGS=(
|
|
||||||
--log_level 10
|
|
||||||
)
|
|
||||||
# you may have a special python for installed packages
|
|
||||||
EXE=`which python3.bash`
|
|
||||||
LARGS+=(
|
|
||||||
--strict_nodes 1
|
|
||||||
--points_timeout 120
|
|
||||||
--proxy-host 127.0.0.1
|
|
||||||
--proxy-port $SOCKS_PORT
|
|
||||||
--https_cafile $CAFILE
|
|
||||||
)
|
|
||||||
|
|
||||||
if [ -f '/run/tor/control' ] ; then
|
|
||||||
LARGS+=(--proxy-ctl '/run/tor/control' )
|
|
||||||
else
|
|
||||||
LARGS+=(--proxy-ctl 9051 )
|
|
||||||
fi
|
|
||||||
|
|
||||||
ddg=duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad
|
|
||||||
# for example, whitelist the introduction points to DuckDuckGo
|
|
||||||
LARGS+=( --white_onions $ddg )
|
|
||||||
|
|
||||||
# you may need to be the tor user to read /run/tor/control
|
|
||||||
grep -q ^debian-tor /etc/group && TORU=debian-tor || {
|
|
||||||
grep -q ^tor /etc/group && TORU=tor
|
|
||||||
}
|
|
||||||
sudo -u $TORU $EXE exclude_badExits.py "${LARGS[@]}" \
|
|
||||||
2>&1|tee exclude_badExits6.log
|
|
||||||
|
|
||||||
# The DEBUG statements contain the detail of why the relay was considered bad.
|
|
||||||
@ -1,151 +0,0 @@
|
|||||||
|
|
||||||
This extends nusenu's basic idea of using the stem library to
|
|
||||||
dynamically exclude nodes that are likely to be bad by putting them
|
|
||||||
on the ExcludeNodes or ExcludeExitNodes setting of a running Tor.
|
|
||||||
* https://github.com/nusenu/noContactInfo_Exit_Excluder
|
|
||||||
* https://github.com/TheSmashy/TorExitRelayExclude
|
|
||||||
|
|
||||||
The basic idea is to exclude Exit nodes that do not have ContactInfo:
|
|
||||||
* https://github.com/nusenu/ContactInfo-Information-Sharing-Specification
|
|
||||||
|
|
||||||
That can be extended to relays that do not have an email in the contact,
|
|
||||||
or to relays that do not have ContactInfo that is verified to include them.
|
|
||||||
But there's a problem, and your Tor notice.log will tell you about it:
|
|
||||||
you could exclude the relays needed to access hidden services or mirror
|
|
||||||
directories. So we need to add to the process the concept of a whitelist.
|
|
||||||
In addition, we may have our own blacklist of nodes we want to exclude,
|
|
||||||
or use these lists for other applications like selektor.
|
|
||||||
|
|
||||||
So we make two files that are structured in YAML:
|
|
||||||
```
|
|
||||||
/etc/tor/yaml/torrc-goodnodes.yaml
|
|
||||||
GoodNodes:
|
|
||||||
Relays:
|
|
||||||
IntroductionPoints:
|
|
||||||
- NODEFINGERPRINT
|
|
||||||
...
|
|
||||||
By default all sections of the goodnodes.yaml are used as a whitelist.
|
|
||||||
|
|
||||||
/etc/tor/yaml/torrc-badnodes.yaml
|
|
||||||
BadNodes:
|
|
||||||
ExcludeExitNodes:
|
|
||||||
BadExit:
|
|
||||||
# $0000000000000000000000000000000000000007
|
|
||||||
```
|
|
||||||
That part requires [PyYAML](https://pyyaml.org/wiki/PyYAML)
|
|
||||||
https://github.com/yaml/pyyaml/ or ```ruamel```: do
|
|
||||||
```pip3 install ruamel``` or ```pip3 install PyYAML```;
|
|
||||||
the advantage of the former is that it preserves comments.
|
|
||||||
|
|
||||||
(You may have to run this as the Tor user to get RW access to
|
|
||||||
/run/tor/control, in which case the directory for the YAML files must
|
|
||||||
be group Tor writeable, and its parents group Tor RX.)
|
|
||||||
|
|
||||||
Because you don't want to exclude the introduction points to any onion
|
|
||||||
you want to connect to, ```--white_onions``` should whitelist the
|
|
||||||
introduction points to a comma sep list of onions; we fixed stem to do this:
|
|
||||||
* https://github.com/torproject/stem/issues/96
|
|
||||||
* https://gitlab.torproject.org/legacy/trac/-/issues/25417
|
|
||||||
|
|
||||||
```--torrc_output``` will write the torrc ExcludeNodes configuration to a file.
|
|
||||||
|
|
||||||
```--good_contacts``` will write the contact info as a ciiss dictionary
|
|
||||||
to a YAML file. If the proof is uri-rsa, the well-known file of fingerprints
|
|
||||||
is downloaded and the fingerprints are added on a 'fps' field we create
|
|
||||||
of that fingerprint's entry of the YAML dictionary. This file is read at the
|
|
||||||
beginning of the program to start with a trust database, and only new
|
|
||||||
contact info from new relays are added to the dictionary.
|
|
||||||
|
|
||||||
Now for the final part: we lookup the Contact info of every relay
|
|
||||||
that is currently in our Tor, and check it the existence of the
|
|
||||||
well-known file that lists the fingerprints of the relays it runs.
|
|
||||||
If it fails to provide the well-know url, we assume its a bad
|
|
||||||
relay and add it to a list of nodes that goes on ```ExcludeNodes```
|
|
||||||
(not just ExcludeExitNodes```). If the Contact info is good, we add the
|
|
||||||
list of fingerprints to ```ExitNodes```, a whitelist of relays to use as exits.
|
|
||||||
|
|
||||||
```--bad_on``` We offer the users 3 levels of cleaning:
|
|
||||||
1. clean relays that have no contact ```=Empty```
|
|
||||||
2. clean relays that don't have an email in the contact (implies 1)
|
|
||||||
```=Empty,NoEmail```
|
|
||||||
3. clean relays that don't have "good' contactinfo. (implies 1)
|
|
||||||
```=Empty,NoEmail,NotGood```
|
|
||||||
|
|
||||||
The default is ```=Empty,NotGood``` ; ```NoEmail``` is inherently imperfect
|
|
||||||
in that many of the contact-as-an-email are obfuscated, but we try anyway.
|
|
||||||
|
|
||||||
To be "good" the ContactInfo must:
|
|
||||||
1. have a url for the well-defined-file to be gotten
|
|
||||||
2. must have a file that can be gotten at the URL
|
|
||||||
3. must support getting the file with a valid SSL cert from a recognized authority
|
|
||||||
4. (not in the spec but added by Python) must use a TLS SSL > v1
|
|
||||||
5. must have a fingerprint list in the file
|
|
||||||
6. must have the FP that got us the contactinfo in the fingerprint list in the file,
|
|
||||||
|
|
||||||
For usage, do ```python3 exclude_badExits.py --help`
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
```
|
|
||||||
|
|
||||||
usage: exclude_badExits.py [-h] [--https_cafile HTTPS_CAFILE]
|
|
||||||
[--proxy_host PROXY_HOST] [--proxy_port PROXY_PORT]
|
|
||||||
[--proxy_ctl PROXY_CTL] [--torrc TORRC]
|
|
||||||
[--timeout TIMEOUT] [--good_nodes GOOD_NODES]
|
|
||||||
[--bad_nodes BAD_NODES] [--bad_on BAD_ON]
|
|
||||||
[--bad_contacts BAD_CONTACTS]
|
|
||||||
[--strict_nodes {0,1}] [--wait_boot WAIT_BOOT]
|
|
||||||
[--points_timeout POINTS_TIMEOUT]
|
|
||||||
[--log_level LOG_LEVEL]
|
|
||||||
[--bad_sections BAD_SECTIONS]
|
|
||||||
[--white_onions WHITE_ONIONS]
|
|
||||||
[--torrc_output TORRC_OUTPUT]
|
|
||||||
[--relays_output RELAYS_OUTPUT]
|
|
||||||
[--good_contacts GOOD_CONTACTS]
|
|
||||||
|
|
||||||
optional arguments:
|
|
||||||
-h, --help show this help message and exit
|
|
||||||
--https_cafile HTTPS_CAFILE
|
|
||||||
Certificate Authority file (in PEM)
|
|
||||||
--proxy_host PROXY_HOST, --proxy-host PROXY_HOST
|
|
||||||
proxy host
|
|
||||||
--proxy_port PROXY_PORT, --proxy-port PROXY_PORT
|
|
||||||
proxy control port
|
|
||||||
--proxy_ctl PROXY_CTL, --proxy-ctl PROXY_CTL
|
|
||||||
control socket - or port
|
|
||||||
--torrc TORRC torrc to check for suggestions
|
|
||||||
--timeout TIMEOUT proxy download connect timeout
|
|
||||||
--good_nodes GOOD_NODES
|
|
||||||
Yaml file of good info that should not be excluded
|
|
||||||
--bad_nodes BAD_NODES
|
|
||||||
Yaml file of bad nodes that should also be excluded
|
|
||||||
--bad_on BAD_ON comma sep list of conditions - Empty,NoEmail,NotGood
|
|
||||||
--bad_contacts BAD_CONTACTS
|
|
||||||
Yaml file of bad contacts that bad FPs are using
|
|
||||||
--strict_nodes {0,1} Set StrictNodes: 1 is less anonymous but more secure,
|
|
||||||
although some sites may be unreachable
|
|
||||||
--wait_boot WAIT_BOOT
|
|
||||||
Seconds to wait for Tor to booststrap
|
|
||||||
--points_timeout POINTS_TIMEOUT
|
|
||||||
Timeout for getting introduction points - must be long
|
|
||||||
>120sec. 0 means disabled looking for IPs
|
|
||||||
--log_level LOG_LEVEL
|
|
||||||
10=debug 20=info 30=warn 40=error
|
|
||||||
--bad_sections BAD_SECTIONS
|
|
||||||
sections of the badnodes.yaml to use, comma separated,
|
|
||||||
'' BROKEN
|
|
||||||
--white_onions WHITE_ONIONS
|
|
||||||
comma sep. list of onions to whitelist their
|
|
||||||
introduction points - BROKEN
|
|
||||||
--torrc_output TORRC_OUTPUT
|
|
||||||
Write the torrc configuration to a file
|
|
||||||
--relays_output RELAYS_OUTPUT
|
|
||||||
Write the download relays in json to a file
|
|
||||||
--good_contacts GOOD_CONTACTS
|
|
||||||
Write the proof data of the included nodes to a YAML
|
|
||||||
file
|
|
||||||
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -1,52 +0,0 @@
|
|||||||
# -*-mode: python; py-indent-offset: 2; indent-tabs-mode: nil; coding: utf-8-unix -*-
|
|
||||||
# https://stem.torproject.org/tutorials/examples/exit_used.html
|
|
||||||
|
|
||||||
import functools
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import getpass
|
|
||||||
|
|
||||||
from stem import StreamStatus
|
|
||||||
from stem.control import EventType, Controller
|
|
||||||
|
|
||||||
def main():
|
|
||||||
print("Tracking requests for tor exits. Press 'enter' to end.")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
if os.path.exists('/var/run/tor/control'):
|
|
||||||
controller = Controller.from_socket_file(path='/var/run/tor/control')
|
|
||||||
else:
|
|
||||||
controller = Controller.from_port(port=9051)
|
|
||||||
try:
|
|
||||||
|
|
||||||
sys.stdout.flush()
|
|
||||||
p = getpass.unix_getpass(prompt='Controller Password: ', stream=sys.stderr)
|
|
||||||
controller.authenticate(p)
|
|
||||||
|
|
||||||
stream_listener = functools.partial(stream_event, controller)
|
|
||||||
controller.add_event_listener(stream_listener, EventType.STREAM)
|
|
||||||
print('Press Enter')
|
|
||||||
input() # wait for user to press enter
|
|
||||||
except Exception as e:
|
|
||||||
print(e)
|
|
||||||
finally:
|
|
||||||
del controller
|
|
||||||
|
|
||||||
def stream_event(controller, event):
|
|
||||||
if event.status == StreamStatus.SUCCEEDED and event.circ_id:
|
|
||||||
circ = controller.get_circuit(event.circ_id)
|
|
||||||
|
|
||||||
exit_fingerprint = circ.path[-1][0]
|
|
||||||
exit_relay = controller.get_network_status(exit_fingerprint)
|
|
||||||
|
|
||||||
print("Exit relay for our connection to %s" % (event.target))
|
|
||||||
print(" address: %s:%i" % (exit_relay.address, exit_relay.or_port))
|
|
||||||
print(" fingerprint: %s" % exit_relay.fingerprint)
|
|
||||||
print(" nickname: %s" % exit_relay.nickname)
|
|
||||||
print(" locale: %s" % controller.get_info("ip-to-country/%s" % exit_relay.address, 'unknown'))
|
|
||||||
print("")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
main()
|
|
||||||
|
|
||||||
@ -1,42 +0,0 @@
|
|||||||
# -*- mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*-
|
|
||||||
# http://vt5hknv6sblkgf22.onion/tutorials/over_the_river.html
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import getpass
|
|
||||||
from stem.control import Controller
|
|
||||||
from stem.connection import MissingPassword
|
|
||||||
|
|
||||||
if len(sys.argv) <= 1:
|
|
||||||
sys.argv += ['']
|
|
||||||
|
|
||||||
if os.path.exists('/run/tor/control'):
|
|
||||||
controller = Controller.from_socket_file(path='/run/tor/control')
|
|
||||||
else:
|
|
||||||
controller = Controller.from_port(port=9051)
|
|
||||||
try:
|
|
||||||
controller.authenticate()
|
|
||||||
except (Exception, MissingPassword):
|
|
||||||
sys.stdout.flush()
|
|
||||||
p = getpass.unix_getpass(prompt='Controller Password: ', stream=sys.stderr)
|
|
||||||
controller.authenticate(p)
|
|
||||||
try:
|
|
||||||
|
|
||||||
for elt in sys.argv[1:]:
|
|
||||||
desc = controller.get_hidden_service_descriptor(elt, await_result=True, timeout=None)
|
|
||||||
print(f"{desc} get_hidden_service_descriptor\n")
|
|
||||||
l = desc.introduction_points()
|
|
||||||
if l:
|
|
||||||
print(f"{elt} NO introduction points\n")
|
|
||||||
continue
|
|
||||||
print(f"{elt} introduction points are...\n")
|
|
||||||
|
|
||||||
for introduction_point in l:
|
|
||||||
print(' %s:%s => %s' % (introduction_point.address,
|
|
||||||
introduction_point.port,
|
|
||||||
introduction_point.identifier))
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(e)
|
|
||||||
finally:
|
|
||||||
del controller
|
|
||||||
@ -1,41 +0,0 @@
|
|||||||
# -*-mode: python; py-indent-offset: 2; indent-tabs-mode: nil; coding: utf-8-unix -*-
|
|
||||||
|
|
||||||
# http://vt5hknv6sblkgf22.onion/tutorials/examples/list_circuits.html
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import getpass
|
|
||||||
from stem import CircStatus
|
|
||||||
from stem.control import Controller
|
|
||||||
|
|
||||||
# port(port = 9051)
|
|
||||||
if os.path.exists('/var/run/tor/control'):
|
|
||||||
controller = Controller.from_socket_file(path='/var/run/tor/control')
|
|
||||||
else:
|
|
||||||
controller = Controller.from_port(port=9051)
|
|
||||||
try:
|
|
||||||
sys.stdout.flush()
|
|
||||||
p = getpass.unix_getpass(prompt='Controller Password: ', stream=sys.stderr)
|
|
||||||
controller.authenticate(p)
|
|
||||||
|
|
||||||
for circ in sorted(controller.get_circuits()):
|
|
||||||
if circ.status != CircStatus.BUILT:
|
|
||||||
continue
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print("Circuit %s (%s)" % (circ.id, circ.purpose))
|
|
||||||
|
|
||||||
for i, entry in enumerate(circ.path):
|
|
||||||
div = '+' if (i == len(circ.path) - 1) else '|'
|
|
||||||
fingerprint, nickname = entry
|
|
||||||
|
|
||||||
desc = controller.get_network_status(fingerprint, None)
|
|
||||||
address = desc.address if desc else 'unknown'
|
|
||||||
|
|
||||||
print(" %s- %s (%s, %s)" % (div, fingerprint, nickname, address))
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(e)
|
|
||||||
finally:
|
|
||||||
del controller
|
|
||||||
|
|
||||||
@ -1,84 +0,0 @@
|
|||||||
#!/usr/local/bin/python3.sh
|
|
||||||
# -*-mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*
|
|
||||||
|
|
||||||
"""
|
|
||||||
Looks for urls https://dns.google/resolve?
|
|
||||||
https://dns.google/resolve?name=domain.name&type=TXT&cd=true&do=true
|
|
||||||
and parses them to extract a magic field.
|
|
||||||
|
|
||||||
A good example of how you can parse json embedded in HTML with phantomjs.
|
|
||||||
|
|
||||||
"""
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
|
|
||||||
from phantompy import Render
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
class LookFor(Render):
|
|
||||||
|
|
||||||
def __init__(self, app, do_print=True, do_save=False):
|
|
||||||
app.lfps = []
|
|
||||||
self._app = app
|
|
||||||
self.do_print = do_print
|
|
||||||
self.do_save = do_save
|
|
||||||
self.progress = 0
|
|
||||||
self.we_run_this_tor_relay = None
|
|
||||||
Render.__init__(self, app, do_print, do_save)
|
|
||||||
|
|
||||||
def _exit(self, val):
|
|
||||||
Render._exit(self, val)
|
|
||||||
self.percent = 100
|
|
||||||
LOG.debug(f"phantom.py: Exiting with val {val}")
|
|
||||||
i = self.uri.find('name=')
|
|
||||||
fp = self.uri[i+5:]
|
|
||||||
i = fp.find('.')
|
|
||||||
fp = fp[:i]
|
|
||||||
# threadsafe?
|
|
||||||
self._app.lfps.append(fp)
|
|
||||||
|
|
||||||
def _html_callback(self, *args):
|
|
||||||
"""print(self, QPrinter, Callable[[bool], None])"""
|
|
||||||
if type(args[0]) is str:
|
|
||||||
self._save(args[0])
|
|
||||||
i = self.ilookfor(args[0])
|
|
||||||
self._onConsoleMessage(i, "__PHANTOM_PY_SAVED__", 0 , '')
|
|
||||||
|
|
||||||
def ilookfor(self, html):
|
|
||||||
import json
|
|
||||||
marker = '<pre style="word-wrap: break-word; white-space: pre-wrap;">'
|
|
||||||
if marker not in html: return -1
|
|
||||||
i = html.find(marker) + len(marker)
|
|
||||||
html = html[i:]
|
|
||||||
assert html[0] == '{', html
|
|
||||||
i = html.find('</pre')
|
|
||||||
html = html[:i]
|
|
||||||
assert html[-1] == '}', html
|
|
||||||
LOG.debug(f"Found {len(html)} json")
|
|
||||||
o = json.loads(html)
|
|
||||||
if "Answer" not in o.keys() or type(o["Answer"]) != list:
|
|
||||||
LOG.warn(f"FAIL {self.uri}")
|
|
||||||
return 1
|
|
||||||
for elt in o["Answer"]:
|
|
||||||
assert type(elt) == dict, elt
|
|
||||||
assert 'type' in elt, elt
|
|
||||||
if elt['type'] != 16: continue
|
|
||||||
assert 'data' in elt, elt
|
|
||||||
if elt['data'] == 'we-run-this-tor-relay':
|
|
||||||
LOG.info(f"OK {self.uri}")
|
|
||||||
self.we_run_this_tor_relay = True
|
|
||||||
return 0
|
|
||||||
self.we_run_this_tor_relay = False
|
|
||||||
LOG.warn(f"BAD {self.uri}")
|
|
||||||
return 2
|
|
||||||
|
|
||||||
def _loadFinished(self, result):
|
|
||||||
LOG.debug(f"phantom.py: Loading finished {self.uri}")
|
|
||||||
self.toHtml(self._html_callback)
|
|
||||||
|
|
||||||
@ -1,42 +0,0 @@
|
|||||||
# -*-mode: python; py-indent-offset: 2; indent-tabs-mode: nil; coding: utf-8-unix -*-
|
|
||||||
# https://stem.torproject.org/tutorials/examples/exit_used.html
|
|
||||||
|
|
||||||
import functools
|
|
||||||
import sys
|
|
||||||
import getpass
|
|
||||||
import os
|
|
||||||
|
|
||||||
from stem import StreamStatus
|
|
||||||
from stem.control import EventType, Controller
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
LOG = logging.getLogger('app.'+'tox_factory')
|
|
||||||
|
|
||||||
def sMapaddressResolv(target, iPort=9051):
|
|
||||||
if os.path.exists('/var/run/tor/control'):
|
|
||||||
controller = Controller.from_socket_file(path='/var/run/tor/control')
|
|
||||||
else:
|
|
||||||
controller = Controller.from_port(port=iPort)
|
|
||||||
|
|
||||||
try:
|
|
||||||
sys.stdout.flush()
|
|
||||||
p = getpass.unix_getpass(prompt='Controller Password: ', stream=sys.stderr)
|
|
||||||
controller.authenticate(p)
|
|
||||||
|
|
||||||
map_dict = {"0.0.0.0": target}
|
|
||||||
map_ret = controller.map_address(map_dict)
|
|
||||||
|
|
||||||
return map_ret
|
|
||||||
except Exception as e:
|
|
||||||
LOG.exception(e)
|
|
||||||
finally:
|
|
||||||
del controller
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
if len(sys.argv) < 2:
|
|
||||||
target = "l2ct3xnuaiwwtoybtn46qp2av4ndxcguwupzyv6xrsmnwi647vvmwtqd"
|
|
||||||
else:
|
|
||||||
target = sys.argv[1]
|
|
||||||
print(sMapaddressResolv(target))
|
|
||||||
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
|
|
||||||
from stem.descriptor.remote import DescriptorDownloader
|
|
||||||
from stem.version import Version
|
|
||||||
|
|
||||||
downloader = DescriptorDownloader()
|
|
||||||
count, with_contact = 0, 0
|
|
||||||
|
|
||||||
print("Checking for outdated relays...")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
for desc in downloader.get_server_descriptors():
|
|
||||||
if desc.tor_version < Version('0.2.3.0'):
|
|
||||||
count += 1
|
|
||||||
|
|
||||||
if desc.contact:
|
|
||||||
print(' %-15s %s' % (desc.tor_version, desc.contact.decode("utf-8", "replace")))
|
|
||||||
with_contact += 1
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print("%i outdated relays found, %i had contact information" % (count, with_contact))
|
|
||||||
# http://vt5hknv6sblkgf22.onion/tutorials/examples/outdated_relays.htmlhttp://vt5hknv6sblkgf22.onion/tutorials/examples/outdated_relays.html
|
|
||||||
@ -1,97 +0,0 @@
|
|||||||
# phantompy
|
|
||||||
|
|
||||||
A simple replacement for phantomjs using PyQt.
|
|
||||||
|
|
||||||
This code is based on a brilliant idea of
|
|
||||||
[Michael Franzl](https://gist.github.com/michaelfranzl/91f0cc13c56120391b949f885643e974/raw/a0601515e7a575bc4c7d4d2a20973b29b6c6f2df/phantom.py)
|
|
||||||
that he wrote up in his blog:
|
|
||||||
* https://blog.michael.franzl.name/2017/10/16/phantomjs-alternative-write-short-pyqt-scripts-instead-phantom-py/
|
|
||||||
* https://blog.michael.franzl.name/2017/10/16/phantom-py/
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
* Generate a PDF screenshot of the web page after it is completely loaded.
|
|
||||||
* Optionally execute a local JavaScript file specified by the argument
|
|
||||||
```javascript-file``` after the web page is completely loaded, and before the
|
|
||||||
PDF is generated. (YMMV - it segfaults for me. )
|
|
||||||
* Generate a HTML save file screenshot of the web page after it is
|
|
||||||
completely loaded and the javascript has run.
|
|
||||||
* console.log's will be printed to stdout.
|
|
||||||
* Easily add new features by changing the source code of this script,
|
|
||||||
without compiling C++ code. For more advanced applications, consider
|
|
||||||
attaching PyQt objects/methods to WebKit's JavaScript space by using
|
|
||||||
QWebFrame::addToJavaScriptWindowObject().
|
|
||||||
|
|
||||||
If you execute an external ```javascript-file```, phantompy has no
|
|
||||||
way of knowing when that script has finished doing its work. For this
|
|
||||||
reason, the external script should execute at the end
|
|
||||||
```console.log("__PHANTOM_PY_DONE__");``` when done. This will trigger
|
|
||||||
the PDF generation or the file saving, after which phantompy will exit.
|
|
||||||
If you do not want to run any javascipt file, this trigger is provided
|
|
||||||
in the code by default.
|
|
||||||
|
|
||||||
It is important to remember that since you're just running WebKit, you can
|
|
||||||
use everything that WebKit supports, including the usual JS client
|
|
||||||
libraries, CSS, CSS @media types, etc.
|
|
||||||
|
|
||||||
Qt picks up proxies from the environment, so this will respect
|
|
||||||
```https_proxy``` or ```http_proxy``` if set.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
* Python3
|
|
||||||
* PyQt5 (this should work with PySide2 and PyQt6 - let us know.)
|
|
||||||
* [qasnyc](https://github.com/CabbageDevelopment/qasync) for the
|
|
||||||
standalone program ```qasync_phantompy.py```
|
|
||||||
|
|
||||||
## Standalone
|
|
||||||
|
|
||||||
A standalone program is a little tricky as PyQt PyQt5.QtWebEngineWidgets'
|
|
||||||
QWebEnginePage uses callbacks at each step of the way:
|
|
||||||
1) loading the page = ```Render.run```
|
|
||||||
2) running javascript in and on the page = ```Render._loadFinished```
|
|
||||||
3) saving the page = ```Render.toHtml and _html_callback```
|
|
||||||
4) printing the page = ```Render._print```
|
|
||||||
|
|
||||||
The steps get chained by printing special messages to the Python
|
|
||||||
renderer of the JavaScript console: ```Render. _onConsoleMessage```
|
|
||||||
|
|
||||||
So it makes it hard if you want the standalone program to work without
|
|
||||||
a GUI, or in combination with another Qt program that is responsible
|
|
||||||
for the PyQt ```app.exec``` and the exiting of the program.
|
|
||||||
|
|
||||||
We've decided to use the best of the shims that merge the Python
|
|
||||||
```asyncio``` and Qt event loops:
|
|
||||||
[qasyc](https://github.com/CabbageDevelopment/qasync). This is seen as
|
|
||||||
the successor to the sorta abandoned [quamash](https://github.com/harvimt/quamash).
|
|
||||||
The code is based on a
|
|
||||||
[comment](https://github.com/CabbageDevelopment/qasync/issues/35#issuecomment-1315060043)
|
|
||||||
by [Alex March](https://github.com/hosaka) who's excellent code helped me.
|
|
||||||
As this is my first use of ```asyncio``` and ```qasync``` I may have
|
|
||||||
introduced some errors and it may be improved on, but it works, and
|
|
||||||
it not a monolithic Qt program, so it can be used as a library.
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
The standalone program is ```quash_phantompy.py```
|
|
||||||
|
|
||||||
|
|
||||||
### Arguments
|
|
||||||
|
|
||||||
```
|
|
||||||
--js_input (optional) Path and name of a JavaScript file to execute on the HTML
|
|
||||||
--html_output <html-file> (optional) Path a HTML output file to generate after JS is applied
|
|
||||||
--pdf_output <pdf-file> (optional) Path and name of PDF file to generate after JS is applied
|
|
||||||
--log_level 10=debug 20=info 30=warn 40=error
|
|
||||||
html_or_url - required argument, a http(s) URL or a path to a local file.
|
|
||||||
```
|
|
||||||
Setting ```DEBUG=1``` in the environment will give debugging messages
|
|
||||||
on ```stderr```.
|
|
||||||
|
|
||||||
## Postscript
|
|
||||||
|
|
||||||
When I think of all the trouble people went to compiling and
|
|
||||||
maintaining the tonnes of C++ code that went into
|
|
||||||
[phantomjs](https://github.com/ariya/phantomjs), I am amazed that it
|
|
||||||
can be replaced with a couple of hundred lines of Python!
|
|
||||||
|
|
||||||
@ -1,275 +0,0 @@
|
|||||||
#!/usr/local/bin/python3.sh
|
|
||||||
# -*-mode: python; indent-tabs-mode: nil; py-indent-offset: 2; coding: utf-8 -*-
|
|
||||||
# https://gist.github.com/michaelfranzl/91f0cc13c56120391b949f885643e974/raw/a0601515e7a575bc4c7d4d2a20973b29b6c6f2df/phantom.py
|
|
||||||
# https://blog.michael.franzl.name/2017/10/16/phantomjs-alternative-write-short-pyqt-scripts-instead-phantom-py/
|
|
||||||
# https://blog.michael.franzl.name/2017/10/16/phantom-py/
|
|
||||||
"""
|
|
||||||
# phantom.py
|
|
||||||
|
|
||||||
Simple but fully scriptable headless QtWebKit browser using PyQt5 in Python3,
|
|
||||||
specialized in executing external JavaScript and generating PDF files. A lean
|
|
||||||
replacement for other bulky headless browser frameworks.
|
|
||||||
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
If you have a display attached:
|
|
||||||
|
|
||||||
./phantom.py [--pdf_output <pdf-file>] [--js_input <javascript-file>] <url-or-html-file>
|
|
||||||
|
|
||||||
If you don't have a display attached (i.e. on a remote server), you can use
|
|
||||||
xvfb-run, or don't add --show_gui - it should work without a display.
|
|
||||||
|
|
||||||
Arguments:
|
|
||||||
|
|
||||||
[--pdf_output <pdf-file>] (optional) Path and name of PDF file to generate
|
|
||||||
[--html_output <html-file>] (optional) Path and name of HTML file to generate
|
|
||||||
[--js_input <javascript-file>] (optional) Path and name of a JavaScript file to execute
|
|
||||||
--log_level 10=debug 20=info 30=warn 40=error
|
|
||||||
<url> Can be a http(s) URL or a path to a local file
|
|
||||||
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
* Generate a PDF screenshot of the web page after it is completely loaded.
|
|
||||||
* Optionally execute a local JavaScript file specified by the argument
|
|
||||||
<javascript-file> after the web page is completely loaded, and before
|
|
||||||
the PDF is generated.
|
|
||||||
* console.log's will be printed to stdout.
|
|
||||||
* Easily add new features by changing the source code of this script, without
|
|
||||||
compiling C++ code. For more advanced applications, consider attaching
|
|
||||||
PyQt objects/methods to WebKit's JavaScript space by using
|
|
||||||
`QWebFrame::addToJavaScriptWindowObject()`.
|
|
||||||
|
|
||||||
If you execute an external <javascript-file>, phantom.py has no way of knowing
|
|
||||||
when that script has finished doing its work. For this reason, the external
|
|
||||||
script should execute `console.log("__PHANTOM_PY_DONE__");` when done. This will
|
|
||||||
trigger the PDF generation, after which phantom.py will exit. If no
|
|
||||||
`__PHANTOM_PY_DONE__` string is seen on the console for 10 seconds, phantom.py
|
|
||||||
will exit without doing anything. This behavior could be implemented more
|
|
||||||
elegantly without console.log's but it is the simplest solution.
|
|
||||||
|
|
||||||
It is important to remember that since you're just running WebKit, you can use
|
|
||||||
everything that WebKit supports, including the usual JS client libraries, CSS,
|
|
||||||
CSS @media types, etc.
|
|
||||||
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
* Python3
|
|
||||||
* PyQt5
|
|
||||||
* [qasnyc](https://github.com/CabbageDevelopment/qasync) for the
|
|
||||||
standalone program ```qasnyc_phantompy.py```
|
|
||||||
|
|
||||||
* xvfb (optional for display-less machines)
|
|
||||||
|
|
||||||
Installation of dependencies in Debian Stretch is easy:
|
|
||||||
|
|
||||||
apt-get install xvfb python3-pyqt5 python3-pyqt5.qtwebkit
|
|
||||||
|
|
||||||
Finding the equivalent for other OSes is an exercise that I leave to you.
|
|
||||||
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
Given the following file /tmp/test.html
|
|
||||||
|
|
||||||
<html>
|
|
||||||
<body>
|
|
||||||
<p>foo <span id="id1">foo</span> <span id="id2">foo</span></p>
|
|
||||||
</body>
|
|
||||||
<script>
|
|
||||||
document.getElementById('id1').innerHTML = "bar";
|
|
||||||
</script>
|
|
||||||
</html>
|
|
||||||
|
|
||||||
... and the following file /tmp/test.js:
|
|
||||||
|
|
||||||
document.getElementById('id2').innerHTML = "baz";
|
|
||||||
console.log("__PHANTOM_PY_DONE__");
|
|
||||||
|
|
||||||
... and running this script (without attached display) ...
|
|
||||||
|
|
||||||
xvfb-run python3 phantom.py /tmp/test.html /tmp/out.pdf /tmp/test.js
|
|
||||||
|
|
||||||
... you will get a PDF file /tmp/out.pdf with the contents "foo bar baz".
|
|
||||||
|
|
||||||
Note that the second occurrence of "foo" has been replaced by the web page's own
|
|
||||||
script, and the third occurrence of "foo" by the external JS file.
|
|
||||||
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
Copyright 2017 Michael Karl Franzl
|
|
||||||
|
|
||||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
|
||||||
this software and associated documentation files (the "Software"), to deal in
|
|
||||||
the Software without restriction, including without limitation the rights to
|
|
||||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
|
||||||
of the Software, and to permit persons to whom the Software is furnished to do
|
|
||||||
so, subject to the following conditions:
|
|
||||||
|
|
||||||
The above copyright notice and this permission notice shall be included in all
|
|
||||||
copies or substantial portions of the Software.
|
|
||||||
|
|
||||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
|
||||||
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
|
||||||
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
|
||||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
||||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import importlib
|
|
||||||
import os
|
|
||||||
import sys # noqa
|
|
||||||
|
|
||||||
from qasync import QtModuleName
|
|
||||||
from qasync.QtCore import QUrl
|
|
||||||
|
|
||||||
QPrinter = importlib.import_module(QtModuleName + ".QtPrintSupport.QPrinter", package=QtModuleName)
|
|
||||||
QWebEnginePage = importlib.import_module(QtModuleName + ".QtWebEngineWidgets.QWebEnginePage", package=QtModuleName)
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
def prepare(sdir='/tmp'):
|
|
||||||
sfile = os.path.join(sdir, 'test.js')
|
|
||||||
if not os.path.exists(sfile):
|
|
||||||
with open(sfile, 'wt') as ofd:
|
|
||||||
ofd.write("""
|
|
||||||
document.getElementById('id2').innerHTML = "baz";
|
|
||||||
console.log("__PHANTOM_PY_DONE__");
|
|
||||||
""")
|
|
||||||
LOG.debug(f"wrote {sfile} ")
|
|
||||||
sfile = os.path.join(sdir, 'test.html')
|
|
||||||
if not os.path.exists(sfile):
|
|
||||||
with open(sfile, 'wt') as ofd:
|
|
||||||
ofd.write("""
|
|
||||||
<html>
|
|
||||||
<body>
|
|
||||||
<p>foo <span id="id1">foo</span> <span id="id2">foo</span></p>
|
|
||||||
</body>
|
|
||||||
<script>
|
|
||||||
document.getElementById('id1').innerHTML = "bar";
|
|
||||||
</script>
|
|
||||||
</html>
|
|
||||||
""")
|
|
||||||
LOG.debug(f"wrote {sfile} ")
|
|
||||||
|
|
||||||
class Render(QWebEnginePage):
|
|
||||||
def __init__(self, app, do_print=False, do_save=True):
|
|
||||||
app.ldone = []
|
|
||||||
self._app = app
|
|
||||||
self.do_print = do_print
|
|
||||||
self.do_save = do_save
|
|
||||||
self.percent = 0
|
|
||||||
self.uri = None
|
|
||||||
self.jsfile = None
|
|
||||||
self.htmlfile = None
|
|
||||||
self.pdffile = None
|
|
||||||
QWebEnginePage.__init__(self)
|
|
||||||
|
|
||||||
def run(self, url, pdffile, htmlfile, jsfile):
|
|
||||||
self._app.lstart.append(id(self))
|
|
||||||
self.percent = 10
|
|
||||||
self.uri = url
|
|
||||||
self.jsfile = jsfile
|
|
||||||
self.htmlfile = htmlfile
|
|
||||||
self.pdffile = pdffile
|
|
||||||
self.outfile = pdffile or htmlfile
|
|
||||||
LOG.debug(f"phantom.py: URL={url} htmlfile={htmlfile} pdffile={pdffile} JSFILE={jsfile}")
|
|
||||||
qurl = QUrl.fromUserInput(url)
|
|
||||||
|
|
||||||
# The PDF generation only happens when the special string __PHANTOM_PY_DONE__
|
|
||||||
# is sent to console.log(). The following JS string will be executed by
|
|
||||||
# default, when no external JavaScript file is specified.
|
|
||||||
self.js_contents = "setTimeout(function() { console.log('__PHANTOM_PY_DONE__') }, 5000);"
|
|
||||||
|
|
||||||
if jsfile:
|
|
||||||
try:
|
|
||||||
with open(self.jsfile, 'rt') as f:
|
|
||||||
self.js_contents = f.read()
|
|
||||||
except Exception as e: # noqa
|
|
||||||
LOG.exception(f"error reading jsfile {self.jsfile}")
|
|
||||||
|
|
||||||
self.loadFinished.connect(self._loadFinished)
|
|
||||||
self.percent = 20
|
|
||||||
self.load(qurl)
|
|
||||||
self.javaScriptConsoleMessage = self._onConsoleMessage
|
|
||||||
LOG.debug(f"phantom.py: loading 10")
|
|
||||||
|
|
||||||
def _onConsoleMessage(self, *args):
|
|
||||||
if len(args) > 3:
|
|
||||||
level, txt, lineno, filename = args
|
|
||||||
else:
|
|
||||||
level = 1
|
|
||||||
txt, lineno, filename = args
|
|
||||||
LOG.debug(f"CONSOLE {lineno} {txt} {filename}")
|
|
||||||
if "__PHANTOM_PY_DONE__" in txt:
|
|
||||||
self.percent = 40
|
|
||||||
# If we get this magic string, it means that the external JS is done
|
|
||||||
if self.do_save:
|
|
||||||
self.toHtml(self._html_callback)
|
|
||||||
return
|
|
||||||
# drop through
|
|
||||||
txt = "__PHANTOM_PY_SAVED__"
|
|
||||||
if "__PHANTOM_PY_SAVED__" in txt:
|
|
||||||
self.percent = 50
|
|
||||||
if self.do_print:
|
|
||||||
self._print()
|
|
||||||
return
|
|
||||||
txt = "__PHANTOM_PY_PRINTED__"
|
|
||||||
if "__PHANTOM_PY_PRINTED__" in txt:
|
|
||||||
self.percent = 60
|
|
||||||
self._exit(level)
|
|
||||||
|
|
||||||
def _loadFinished(self, result):
|
|
||||||
# RenderProcessTerminationStatus ?
|
|
||||||
self.percent = 30
|
|
||||||
LOG.info(f"phantom.py: _loadFinished {result} {self.percent}")
|
|
||||||
LOG.debug(f"phantom.py: Evaluating JS from {self.jsfile}")
|
|
||||||
self.runJavaScript("document.documentElement.contentEditable=true")
|
|
||||||
self.runJavaScript(self.js_contents)
|
|
||||||
|
|
||||||
def _html_callback(self, *args):
|
|
||||||
"""print(self, QPrinter, Callable[[bool], None])"""
|
|
||||||
if type(args[0]) is str:
|
|
||||||
self._save(args[0])
|
|
||||||
self._onConsoleMessage(0, "__PHANTOM_PY_SAVED__", 0, '')
|
|
||||||
|
|
||||||
def _save(self, html):
|
|
||||||
sfile = self.htmlfile
|
|
||||||
# CompleteHtmlSaveFormat SingleHtmlSaveFormat MimeHtmlSaveFormat
|
|
||||||
with open(sfile, 'wt') as ofd:
|
|
||||||
ofd.write(html)
|
|
||||||
LOG.debug(f"Saved {sfile}")
|
|
||||||
|
|
||||||
def _printer_callback(self, *args):
|
|
||||||
"""print(self, QPrinter, Callable[[bool], None])"""
|
|
||||||
if args[0] is False:
|
|
||||||
i = 1
|
|
||||||
else:
|
|
||||||
i = 0
|
|
||||||
self._onConsoleMessage(i, "__PHANTOM_PY_PRINTED__", 0, '')
|
|
||||||
|
|
||||||
def _print(self):
|
|
||||||
sfile = self.pdffile
|
|
||||||
printer = QPrinter()
|
|
||||||
printer.setPageMargins(10, 10, 10, 10, QPrinter.Millimeter)
|
|
||||||
printer.setPaperSize(QPrinter.A4)
|
|
||||||
printer.setCreator("phantom.py by Michael Karl Franzl")
|
|
||||||
printer.setOutputFormat(QPrinter.PdfFormat)
|
|
||||||
printer.setOutputFileName(sfile)
|
|
||||||
self.print(printer, self._printer_callback)
|
|
||||||
LOG.debug("phantom.py: Printed")
|
|
||||||
|
|
||||||
def _exit(self, val):
|
|
||||||
self.percent = 100
|
|
||||||
LOG.debug(f"phantom.py: Exiting with val {val}")
|
|
||||||
# threadsafe?
|
|
||||||
self._app.ldone.append(self.uri)
|
|
||||||
@ -1,128 +0,0 @@
|
|||||||
#!/usr/local/bin/python3.sh
|
|
||||||
# -*-mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
# let qasync figure out what Qt we are using - we dont care
|
|
||||||
from qasync import QApplication, QEventLoop, QtWidgets
|
|
||||||
|
|
||||||
from phantompy import Render
|
|
||||||
# if you want an example of looking for things in downloaded HTML:
|
|
||||||
# from lookupdns import LookFor as Render
|
|
||||||
from support_phantompy import omain_argparser, vsetup_logging
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
try:
|
|
||||||
import shtab
|
|
||||||
except:
|
|
||||||
shtab = None
|
|
||||||
|
|
||||||
class Widget(QtWidgets.QWidget):
|
|
||||||
def __init__(self):
|
|
||||||
QtWidgets.QWidget.__init__(self)
|
|
||||||
self._label = QtWidgets.QLabel()
|
|
||||||
box = QtWidgets.QHBoxLayout()
|
|
||||||
self.setLayout(box)
|
|
||||||
box.addWidget(self._label)
|
|
||||||
self.progress = QtWidgets.QProgressBar()
|
|
||||||
self.progress.setRange(0, 99)
|
|
||||||
box.addWidget(self.progress)
|
|
||||||
|
|
||||||
def update(self, text):
|
|
||||||
i = len(asyncio.all_tasks())
|
|
||||||
self._label.setText(str(i))
|
|
||||||
self.progress.setValue(int(text))
|
|
||||||
|
|
||||||
class ContextManager:
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self._seconds = 0
|
|
||||||
|
|
||||||
async def __aenter__(self):
|
|
||||||
LOG.debug("ContextManager enter")
|
|
||||||
return self
|
|
||||||
|
|
||||||
async def __aexit__(self, *args):
|
|
||||||
LOG.debug("ContextManager exit")
|
|
||||||
|
|
||||||
async def tick(self):
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
self._seconds += 1
|
|
||||||
return self._seconds
|
|
||||||
|
|
||||||
async def main(widget, app, ilen):
|
|
||||||
LOG.debug("Task started")
|
|
||||||
try:
|
|
||||||
async with ContextManager() as ctx:
|
|
||||||
for i in range(1, 120):
|
|
||||||
seconds = await ctx.tick()
|
|
||||||
if widget:
|
|
||||||
widget.update(str(i))
|
|
||||||
if len(app.ldone) == ilen:
|
|
||||||
LOG.info(f"Finished with {app.ldone}")
|
|
||||||
print('\n'.join(app.ldone))
|
|
||||||
app.exit()
|
|
||||||
# raise asyncio.CancelledError
|
|
||||||
return
|
|
||||||
LOG.debug(f"{app.ldone} {seconds}")
|
|
||||||
except asyncio.CancelledError as ex: # noqa
|
|
||||||
LOG.debug("Task cancelled")
|
|
||||||
|
|
||||||
def iMain(largs):
|
|
||||||
parser = omain_argparser()
|
|
||||||
if shtab:
|
|
||||||
shtab.add_argument_to(parser, ["-s", "--print-completion"]) # magic!
|
|
||||||
oargs = parser.parse_args(largs)
|
|
||||||
bgui = oargs.show_gui
|
|
||||||
|
|
||||||
try:
|
|
||||||
d = int(os.environ.get('DEBUG', 0))
|
|
||||||
if d > 0:
|
|
||||||
oargs.log_level = 10
|
|
||||||
vsetup_logging(oargs.log_level, logfile='', stream=sys.stderr)
|
|
||||||
except: pass
|
|
||||||
|
|
||||||
app = QApplication([])
|
|
||||||
app.lstart = []
|
|
||||||
if bgui:
|
|
||||||
widget = Widget()
|
|
||||||
widget._app = app
|
|
||||||
widget.show()
|
|
||||||
else:
|
|
||||||
widget = None
|
|
||||||
|
|
||||||
loop = QEventLoop(app)
|
|
||||||
asyncio.set_event_loop(loop)
|
|
||||||
|
|
||||||
url = oargs.html_url
|
|
||||||
htmlfile = oargs.html_output
|
|
||||||
pdffile = oargs.html_output
|
|
||||||
jsfile = oargs.js_input
|
|
||||||
# run only starts the url loading
|
|
||||||
r = Render(app,
|
|
||||||
do_print=True if pdffile else False,
|
|
||||||
do_save=True if htmlfile else False)
|
|
||||||
uri = url.strip()
|
|
||||||
r.run(uri, pdffile, htmlfile, jsfile)
|
|
||||||
LOG.debug(f"{r.percent} {app.lstart}")
|
|
||||||
|
|
||||||
LOG.info(f"queued {len(app.lstart)} urls")
|
|
||||||
|
|
||||||
task = loop.create_task(main(widget, app, 1))
|
|
||||||
loop.run_forever()
|
|
||||||
|
|
||||||
# cancel remaining tasks and wait for them to complete
|
|
||||||
task.cancel()
|
|
||||||
tasks = asyncio.all_tasks()
|
|
||||||
loop.run_until_complete(asyncio.gather(*tasks))
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
iMain(sys.argv[1:])
|
|
||||||
@ -1,140 +0,0 @@
|
|||||||
#!/usr/local/bin/python3.sh
|
|
||||||
# -*-mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import atexit
|
|
||||||
import traceback
|
|
||||||
import functools
|
|
||||||
import asyncio
|
|
||||||
import time
|
|
||||||
import qasync
|
|
||||||
import threading
|
|
||||||
|
|
||||||
from PyQt5.QtWidgets import (QProgressBar, QWidget, QVBoxLayout)
|
|
||||||
# from PySide2.QtWidgets import QApplication, QProgressBar
|
|
||||||
from qasync import QEventLoop, QThreadExecutor
|
|
||||||
from qasync import asyncSlot, asyncClose, QApplication
|
|
||||||
|
|
||||||
from phantompy import Render
|
|
||||||
from lookupdns import LookFor
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
class MainWindow(QWidget):
|
|
||||||
"""Main window."""
|
|
||||||
def __init__(self):
|
|
||||||
super().__init__()
|
|
||||||
|
|
||||||
self.setLayout(QVBoxLayout())
|
|
||||||
self.progress = QProgressBar()
|
|
||||||
self.progress.setRange(0, 99)
|
|
||||||
self.layout().addWidget(self.progress)
|
|
||||||
|
|
||||||
async def main(app):
|
|
||||||
def close_future(future, loop):
|
|
||||||
loop.call_later(10, future.cancel)
|
|
||||||
future.cancel()
|
|
||||||
|
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
future = asyncio.Future()
|
|
||||||
app.ldone = []
|
|
||||||
|
|
||||||
getattr(app, "aboutToQuit").connect(
|
|
||||||
functools.partial(close_future, future, loop)
|
|
||||||
)
|
|
||||||
|
|
||||||
if False:
|
|
||||||
progress = QProgressBar()
|
|
||||||
progress.setRange(0, 99)
|
|
||||||
progress.show()
|
|
||||||
else:
|
|
||||||
mw = MainWindow()
|
|
||||||
progress = mw.progress
|
|
||||||
mw.show()
|
|
||||||
# LOG.info(f"calling first_50 {r}")
|
|
||||||
# await first_50(progress, r)
|
|
||||||
LOG.info(f"calling last_50 {r}")
|
|
||||||
o = QThreadExecutor(max_workers=1)
|
|
||||||
app.o = o
|
|
||||||
with o as executor:
|
|
||||||
await loop.run_in_executor(executor, functools.partial(last_50, progress, sys.argv[1:], app), loop)
|
|
||||||
LOG.info(f" {dir(o)}")
|
|
||||||
|
|
||||||
LOG.info(f"awaiting {future}")
|
|
||||||
await future
|
|
||||||
return True
|
|
||||||
|
|
||||||
async def first_50(progress, r=None):
|
|
||||||
progress.setValue(5)
|
|
||||||
LOG.info(f"first_50 {r}")
|
|
||||||
if r is not None:
|
|
||||||
# loop = asyncio.get_running_loop()
|
|
||||||
# LOG.info(f"first_50.r.run {r}")
|
|
||||||
# loop.call_soon_threadsafe(r.run, r.url, r.outfile, r.jsfile)
|
|
||||||
# r.run( r.url, r.outfile, r.jsfile)
|
|
||||||
for i in range(50):
|
|
||||||
# LOG.info(f"first_50 {r.progress} {i}")
|
|
||||||
# if r.progress >= 100: break
|
|
||||||
# progress.setValue(max(r.progress,i))
|
|
||||||
progress.setValue(i)
|
|
||||||
await asyncio.sleep(.1)
|
|
||||||
return
|
|
||||||
for i in range(50):
|
|
||||||
LOG.info(f"first_50 {r} {i}")
|
|
||||||
loop.call_soon_threadsafe(progress.setValue, i)
|
|
||||||
time.sleep(1)
|
|
||||||
|
|
||||||
def last_50(progress, largs, app, loop):
|
|
||||||
url = largs[0]
|
|
||||||
outfile = largs[1]
|
|
||||||
jsfile = largs[2] if len(largs) > 2 else None
|
|
||||||
r = Render(app, do_print=False, do_save=True)
|
|
||||||
uri = url.strip()
|
|
||||||
loop.call_soon_threadsafe(r.run, uri, outfile, jsfile)
|
|
||||||
time.sleep(1)
|
|
||||||
for i in range(50, 100):
|
|
||||||
j = len(app.ldone) # r.progress
|
|
||||||
if j == 100:
|
|
||||||
LOG.info(f"last_50 None {i} {j}")
|
|
||||||
else:
|
|
||||||
LOG.debug(f"last_50 None {i} {j}")
|
|
||||||
loop.call_soon_threadsafe(progress.setValue, i)
|
|
||||||
time.sleep(1)
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
url = 'https://dns.google/resolve?name=6D6EC2A2E2ED8BFF2D4834F8D669D82FC2A9FA8D.for-privacy.net&type=TXT&cd=true&do=true'
|
|
||||||
outfile = '/tmp/test1.pdf'
|
|
||||||
jsfile = '/tmp/test1.js'
|
|
||||||
from exclude_badExits import vsetup_logging
|
|
||||||
vsetup_logging(10)
|
|
||||||
app = QApplication([])
|
|
||||||
#?
|
|
||||||
loop = qasync.QEventLoop(app)
|
|
||||||
#NOT loop = asyncio.get_event_loop()
|
|
||||||
asyncio._set_running_loop(loop)
|
|
||||||
asyncio.events._set_running_loop(loop)
|
|
||||||
r = Render(app, do_print=False, do_save=True)
|
|
||||||
#loop.call_soon_threadsafe(r.run, url, outfile, jsfile)
|
|
||||||
r.run(url, outfile, jsfile)
|
|
||||||
app.rs = [r]
|
|
||||||
for i in range(20):
|
|
||||||
for elt in app.rs:
|
|
||||||
print (elt.percent)
|
|
||||||
time.sleep(2)
|
|
||||||
try:
|
|
||||||
qasync.run(main(app))
|
|
||||||
except asyncio.exceptions.CancelledError:
|
|
||||||
sys.exit(0)
|
|
||||||
except RuntimeError as e:
|
|
||||||
LOG.debug('Fixme')
|
|
||||||
sys.exit(0)
|
|
||||||
except KeyboardInterrupt:
|
|
||||||
sys.exit(0)
|
|
||||||
else:
|
|
||||||
val = 0
|
|
||||||
sys.exit(val)
|
|
||||||
@ -1,49 +0,0 @@
|
|||||||
#!/usr/local/bin/python3.sh
|
|
||||||
# -*-mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import traceback
|
|
||||||
|
|
||||||
from phantompy import Render
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import asyncio
|
|
||||||
import time
|
|
||||||
|
|
||||||
from PyQt5.QtWidgets import QApplication, QProgressBar
|
|
||||||
from quamash import QEventLoop, QThreadExecutor
|
|
||||||
|
|
||||||
app = QApplication(sys.argv)
|
|
||||||
loop = QEventLoop(app)
|
|
||||||
asyncio.set_event_loop(loop) # NEW must set the event loop
|
|
||||||
asyncio.events._set_running_loop(loop)
|
|
||||||
|
|
||||||
progress = QProgressBar()
|
|
||||||
progress.setRange(0, 99)
|
|
||||||
progress.show()
|
|
||||||
|
|
||||||
async def master():
|
|
||||||
await first_50()
|
|
||||||
with QThreadExecutor(1) as executor:
|
|
||||||
await loop.run_in_executor(exec, last_50)
|
|
||||||
# TODO announce completion?
|
|
||||||
|
|
||||||
async def first_50():
|
|
||||||
for i in range(50):
|
|
||||||
progress.setValue(i)
|
|
||||||
await asyncio.sleep(.1)
|
|
||||||
|
|
||||||
def last_50():
|
|
||||||
for i in range(50,100):
|
|
||||||
loop.call_soon_threadsafe(progress.setValue, i)
|
|
||||||
time.sleep(.1)
|
|
||||||
|
|
||||||
with loop: ## context manager calls .close() when loop completes, and releases all resources
|
|
||||||
loop.run_until_complete(master())
|
|
||||||
@ -1,137 +0,0 @@
|
|||||||
# -*-mode: python; py-indent-offset: 2; indent-tabs-mode: nil; coding: utf-8-unix -*-
|
|
||||||
|
|
||||||
# http://vt5hknv6sblkgf22.onion/tutorials/examples/relay_connections.html
|
|
||||||
import argparse
|
|
||||||
import collections
|
|
||||||
import time
|
|
||||||
|
|
||||||
import stem.connection
|
|
||||||
import stem.util.system
|
|
||||||
import stem.util.str_tools
|
|
||||||
|
|
||||||
from stem.control import Listener
|
|
||||||
from stem.control import Controller
|
|
||||||
from stem.util.connection import get_connections, port_usage, is_valid_ipv4_address
|
|
||||||
|
|
||||||
HEADER_LINE = " {version} uptime: {uptime} flags: {flags}\n"
|
|
||||||
|
|
||||||
DIV = '+%s+%s+%s+' % ('-' * 30, '-' * 6, '-' * 6)
|
|
||||||
COLUMN = '| %-28s | %4s | %4s |'
|
|
||||||
|
|
||||||
INBOUND_ORPORT = 'Inbound to our ORPort'
|
|
||||||
INBOUND_DIRPORT = 'Inbound to our DirPort'
|
|
||||||
INBOUND_CONTROLPORT = 'Inbound to our ControlPort'
|
|
||||||
|
|
||||||
OUTBOUND_ORPORT = 'Outbound to a relay'
|
|
||||||
OUTBOUND_EXIT = 'Outbound exit traffic'
|
|
||||||
OUTBOUND_UNKNOWN = 'Outbound uncategorized'
|
|
||||||
|
|
||||||
def main(controller):
|
|
||||||
parser = argparse.ArgumentParser()
|
|
||||||
parser.add_argument("--ctrlport", help="default: 9051 or 9151")
|
|
||||||
parser.add_argument("--resolver", help="default: autodetected")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
control_port = int(args.ctrlport) if args.ctrlport else 'default'
|
|
||||||
controller = stem.connection.connect(control_port = ('127.0.0.1', control_port))
|
|
||||||
|
|
||||||
if not controller:
|
|
||||||
return
|
|
||||||
|
|
||||||
desc = controller.get_network_status(default = None)
|
|
||||||
pid = controller.get_pid()
|
|
||||||
version = str(controller.get_version()).split()[0],
|
|
||||||
uptime = stem.util.str_tools.short_time_label(time.time() - stem.util.system.start_time(pid))
|
|
||||||
|
|
||||||
print(HEADER_LINE.format(
|
|
||||||
version=version,
|
|
||||||
uptime=uptime,
|
|
||||||
flags = ', '.join(desc.flags if desc else ['none']),
|
|
||||||
))
|
|
||||||
|
|
||||||
policy = controller.get_exit_policy()
|
|
||||||
relays = {} # address => [orports...]
|
|
||||||
|
|
||||||
for desc in controller.get_network_statuses():
|
|
||||||
relays.setdefault(desc.address, []).append(desc.or_port)
|
|
||||||
|
|
||||||
# categorize our connections
|
|
||||||
|
|
||||||
categories = collections.OrderedDict((
|
|
||||||
(INBOUND_ORPORT, []),
|
|
||||||
(INBOUND_DIRPORT, []),
|
|
||||||
(INBOUND_CONTROLPORT, []),
|
|
||||||
(OUTBOUND_ORPORT, []),
|
|
||||||
(OUTBOUND_EXIT, []),
|
|
||||||
(OUTBOUND_UNKNOWN, []),
|
|
||||||
))
|
|
||||||
|
|
||||||
exit_connections = {} # port => [connections]
|
|
||||||
|
|
||||||
for conn in get_connections(resolver = args.resolver, process_pid = pid):
|
|
||||||
if conn.protocol == 'udp':
|
|
||||||
continue
|
|
||||||
|
|
||||||
if conn.local_port in controller.get_ports(Listener.OR, []):
|
|
||||||
categories[INBOUND_ORPORT].append(conn)
|
|
||||||
elif conn.local_port in controller.get_ports(Listener.DIR, []):
|
|
||||||
categories[INBOUND_DIRPORT].append(conn)
|
|
||||||
elif conn.local_port in controller.get_ports(Listener.CONTROL, []):
|
|
||||||
categories[INBOUND_CONTROLPORT].append(conn)
|
|
||||||
elif conn.remote_port in relays.get(conn.remote_address, []):
|
|
||||||
categories[OUTBOUND_ORPORT].append(conn)
|
|
||||||
elif policy.can_exit_to(conn.remote_address, conn.remote_port):
|
|
||||||
categories[OUTBOUND_EXIT].append(conn)
|
|
||||||
exit_connections.setdefault(conn.remote_port, []).append(conn)
|
|
||||||
else:
|
|
||||||
categories[OUTBOUND_UNKNOWN].append(conn)
|
|
||||||
|
|
||||||
print(DIV)
|
|
||||||
print(COLUMN % ('Type', 'IPv4', 'IPv6'))
|
|
||||||
print(DIV)
|
|
||||||
|
|
||||||
total_ipv4, total_ipv6 = 0, 0
|
|
||||||
|
|
||||||
for label, connections in categories.items():
|
|
||||||
if len(connections) == 0:
|
|
||||||
continue
|
|
||||||
|
|
||||||
ipv4_count = len([conn for conn in connections if is_valid_ipv4_address(conn.remote_address)])
|
|
||||||
ipv6_count = len(connections) - ipv4_count
|
|
||||||
|
|
||||||
total_ipv4, total_ipv6 = total_ipv4 + ipv4_count, total_ipv6 + ipv6_count
|
|
||||||
print(COLUMN % (label, ipv4_count, ipv6_count))
|
|
||||||
|
|
||||||
print(DIV)
|
|
||||||
print(COLUMN % ('Total', total_ipv4, total_ipv6))
|
|
||||||
print(DIV)
|
|
||||||
print('')
|
|
||||||
|
|
||||||
if exit_connections:
|
|
||||||
print(DIV)
|
|
||||||
print(COLUMN % ('Exit Port', 'IPv4', 'IPv6'))
|
|
||||||
print(DIV)
|
|
||||||
|
|
||||||
total_ipv4, total_ipv6 = 0, 0
|
|
||||||
|
|
||||||
for port in sorted(exit_connections):
|
|
||||||
connections = exit_connections[port]
|
|
||||||
ipv4_count = len([conn for conn in connections if is_valid_ipv4_address(conn.remote_address)])
|
|
||||||
ipv6_count = len(connections) - ipv4_count
|
|
||||||
total_ipv4, total_ipv6 = total_ipv4 + ipv4_count, total_ipv6 + ipv6_count
|
|
||||||
|
|
||||||
usage = port_usage(port)
|
|
||||||
label = '%s (%s)' % (port, usage) if usage else port
|
|
||||||
|
|
||||||
print(COLUMN % (label, ipv4_count, ipv6_count))
|
|
||||||
|
|
||||||
print(DIV)
|
|
||||||
print(COLUMN % ('Total', total_ipv4, total_ipv6))
|
|
||||||
print(DIV)
|
|
||||||
print('')
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
with Controller.from_socket_file(path='/var/run/tor/control') as controller:
|
|
||||||
main(controller)
|
|
||||||
|
|
||||||
@ -1,58 +0,0 @@
|
|||||||
[metadata]
|
|
||||||
classifiers =
|
|
||||||
License :: OSI Approved
|
|
||||||
License :: OSI Approved :: BSD 1-clause
|
|
||||||
Intended Audience :: Web Developers
|
|
||||||
Operating System :: Microsoft :: Windows
|
|
||||||
Operating System :: POSIX :: BSD :: FreeBSD
|
|
||||||
Operating System :: POSIX :: Linux
|
|
||||||
Programming Language :: Python :: 3 :: Only
|
|
||||||
Programming Language :: Python :: 3.6
|
|
||||||
Programming Language :: Python :: 3.7
|
|
||||||
Programming Language :: Python :: 3.8
|
|
||||||
Programming Language :: Python :: 3.9
|
|
||||||
Programming Language :: Python :: Implementation :: CPython
|
|
||||||
Framework :: AsyncIO
|
|
||||||
|
|
||||||
[options]
|
|
||||||
zip_safe = false
|
|
||||||
python_requires = ~=3.6
|
|
||||||
packages = find:
|
|
||||||
include_package_data = false
|
|
||||||
install_requires =
|
|
||||||
qasync
|
|
||||||
cryptography
|
|
||||||
rsa
|
|
||||||
stem
|
|
||||||
|
|
||||||
[options.entry_points]
|
|
||||||
console_scripts =
|
|
||||||
phantompy = phantompy.__main__:iMain
|
|
||||||
|
|
||||||
[easy_install]
|
|
||||||
zip_ok = false
|
|
||||||
|
|
||||||
[flake8]
|
|
||||||
jobs = 1
|
|
||||||
max-line-length = 88
|
|
||||||
ignore =
|
|
||||||
E111
|
|
||||||
E114
|
|
||||||
E128
|
|
||||||
E225
|
|
||||||
E225
|
|
||||||
E261
|
|
||||||
E302
|
|
||||||
E305
|
|
||||||
E402
|
|
||||||
E501
|
|
||||||
E502
|
|
||||||
E541
|
|
||||||
E701
|
|
||||||
E704
|
|
||||||
E722
|
|
||||||
E741
|
|
||||||
F508
|
|
||||||
F541
|
|
||||||
W503
|
|
||||||
|
|
||||||
@ -1,89 +0,0 @@
|
|||||||
# -*- mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*-
|
|
||||||
# https://stackoverflow.com/questions/5239797/python-smtplib-proxy-support
|
|
||||||
# https://stackoverflow.com/questions/19642726/testing-python-smtp-email-service
|
|
||||||
import socket
|
|
||||||
import smtplib
|
|
||||||
|
|
||||||
import socks
|
|
||||||
|
|
||||||
class ProxySMTP(smtplib.SMTP):
|
|
||||||
def __init__(self, host='', port=0, local_hostname=None,
|
|
||||||
timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
|
|
||||||
source_address=None, proxy_addr=None, proxy_port=None):
|
|
||||||
"""Initialize a new instance.
|
|
||||||
|
|
||||||
If specified, `host' is the name of the remote host to which to
|
|
||||||
connect. If specified, `port' specifies the port to which to connect.
|
|
||||||
By default, smtplib.SMTP_PORT is used. If a host is specified the
|
|
||||||
connect method is called, and if it returns anything other than a
|
|
||||||
success code an SMTPConnectError is raised. If specified,
|
|
||||||
`local_hostname` is used as the FQDN of the local host in the HELO/EHLO
|
|
||||||
command. Otherwise, the local hostname is found using
|
|
||||||
socket.getfqdn(). The `source_address` parameter takes a 2-tuple (host,
|
|
||||||
port) for the socket to bind to as its source address before
|
|
||||||
connecting. If the host is '' and port is 0, the OS default behavior
|
|
||||||
will be used.
|
|
||||||
|
|
||||||
"""
|
|
||||||
self._host = host
|
|
||||||
self.timeout = timeout
|
|
||||||
self.esmtp_features = {}
|
|
||||||
self.command_encoding = 'ascii'
|
|
||||||
self.source_address = source_address
|
|
||||||
self.proxy_addr = proxy_addr
|
|
||||||
self.proxy_port = proxy_port
|
|
||||||
|
|
||||||
if host:
|
|
||||||
(code, msg) = self.connect(host, port)
|
|
||||||
if code != 220:
|
|
||||||
self.close()
|
|
||||||
raise smtplib.SMTPConnectError(code, msg)
|
|
||||||
if local_hostname is not None:
|
|
||||||
self.local_hostname = local_hostname
|
|
||||||
else:
|
|
||||||
# RFC 2821 says we should use the fqdn in the EHLO/HELO verb, and
|
|
||||||
# if that can't be calculated, that we should use a domain literal
|
|
||||||
# instead (essentially an encoded IP address like [A.B.C.D]).
|
|
||||||
fqdn = socket.getfqdn()
|
|
||||||
if '.' in fqdn:
|
|
||||||
self.local_hostname = fqdn
|
|
||||||
else:
|
|
||||||
# We can't find an fqdn hostname, so use a domain literal
|
|
||||||
addr = '127.0.0.1'
|
|
||||||
try:
|
|
||||||
addr = socket.gethostbyname(socket.gethostname())
|
|
||||||
except socket.gaierror:
|
|
||||||
pass
|
|
||||||
self.local_hostname = '[%s]' % addr
|
|
||||||
|
|
||||||
def _get_socket(self, host, port, timeout):
|
|
||||||
# This makes it simpler for SMTP_SSL to use the SMTP connect code
|
|
||||||
# and just alter the socket connection bit.
|
|
||||||
if self.debuglevel > 0:
|
|
||||||
self._print_debug('connect: to', (host, port), self.source_address)
|
|
||||||
return socks.create_connection((host, port),
|
|
||||||
proxy_type=socks.PROXY_TYPE_SOCKS5,
|
|
||||||
timeout=timeout,
|
|
||||||
proxy_addr=self.proxy_addr,
|
|
||||||
proxy_port=self.proxy_port)
|
|
||||||
|
|
||||||
# And to use:
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
user_email, user_pass = 'foo', 'bar'
|
|
||||||
email_server = ProxySMTP('smtp.gmail.com', 587,
|
|
||||||
proxy_addr='127.0.0.1',
|
|
||||||
proxy_port=9050,
|
|
||||||
timeout=20)
|
|
||||||
email_server.starttls()
|
|
||||||
try:
|
|
||||||
email_server.login(user_email, user_pass)
|
|
||||||
except smtplib.SMTPAuthenticationError as e:
|
|
||||||
if len(e.args) > 1:
|
|
||||||
code = e.args[0]
|
|
||||||
if code = 535:
|
|
||||||
# 5.7.8 Username and Password not accepted
|
|
||||||
pass
|
|
||||||
raise
|
|
||||||
email_server.sendmail(user_email, recipient_list, msg.as_string())
|
|
||||||
email_server.quit()
|
|
||||||
@ -1,445 +0,0 @@
|
|||||||
# -*- mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*-
|
|
||||||
|
|
||||||
import getpass
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import select
|
|
||||||
import shutil
|
|
||||||
import socket
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
|
|
||||||
if False:
|
|
||||||
import cepa as stem
|
|
||||||
from cepa.connection import MissingPassword
|
|
||||||
from cepa.control import Controller
|
|
||||||
from cepa.util.tor_tools import is_valid_fingerprint
|
|
||||||
else:
|
|
||||||
import stem
|
|
||||||
from stem.connection import MissingPassword
|
|
||||||
from stem.control import Controller
|
|
||||||
from stem.util.tor_tools import is_valid_fingerprint
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
bHAVE_TORR = shutil.which('tor-resolve')
|
|
||||||
|
|
||||||
# we check these each time but we got them by sorting bad relays
|
|
||||||
# in the wild we'll keep a copy here so we can avoid restesting
|
|
||||||
yKNOWN_NODNS = """
|
|
||||||
---
|
|
||||||
- for-privacy.net
|
|
||||||
- backup.spekadyon.org
|
|
||||||
- verification-for-nusenu.net
|
|
||||||
- prsv.ch
|
|
||||||
- ezyn.de
|
|
||||||
- dfri.se
|
|
||||||
- dtf.contact
|
|
||||||
- galtland.network
|
|
||||||
- dotsrc.org
|
|
||||||
- nicdex.com
|
|
||||||
- unzane.com
|
|
||||||
- a9.wtf
|
|
||||||
- tor.skankhunt42.pw
|
|
||||||
- tor-exit-3.aa78i2efsewr0neeknk.xyz
|
|
||||||
- privacysvcs.net
|
|
||||||
- apt96.com
|
|
||||||
- mkg20001.io
|
|
||||||
- kryptonit.org
|
|
||||||
- sebastian-elisa-pfeifer.eu
|
|
||||||
- nx42.de
|
|
||||||
- www.defcon.org
|
|
||||||
- 0x0.is
|
|
||||||
- transliberation.today
|
|
||||||
- tor-exit-2.aa78i2efsewr0neeknk.xyz
|
|
||||||
- interfesse.net
|
|
||||||
- axims.net
|
|
||||||
- a9.wtf
|
|
||||||
- heraldonion.org
|
|
||||||
- linkspartei.org
|
|
||||||
- pineapple.cx
|
|
||||||
- privacylayer.xyz
|
|
||||||
- prsv.ch
|
|
||||||
- thingtohide.nl
|
|
||||||
- tor-exit-2.aa78i2efsewr0neeknk.xyz
|
|
||||||
- tor-exit-3.aa78i2efsewr0neeknk.xyz
|
|
||||||
- tor.dlecan.com
|
|
||||||
- tuxli.org
|
|
||||||
- verification-for-nusenu.net
|
|
||||||
"""
|
|
||||||
# - 0x0.is
|
|
||||||
# - aklad5.com
|
|
||||||
# - artikel5ev.de
|
|
||||||
# - arvanode.net
|
|
||||||
# - dodo.pm
|
|
||||||
# - erjan.net
|
|
||||||
# - galtland.network
|
|
||||||
# - lonet.sh
|
|
||||||
# - moneneis.de
|
|
||||||
# - olonet.sh
|
|
||||||
# - or-exit-2.aa78i2efsewr0neeknk.xyz
|
|
||||||
# - or.wowplanet.de
|
|
||||||
# - ormycloud.org
|
|
||||||
# - plied-privacy.net
|
|
||||||
# - rivacysvcs.net
|
|
||||||
# - redacted.org
|
|
||||||
# - rofl.cat
|
|
||||||
# - sv.ch
|
|
||||||
# - tikel10.org
|
|
||||||
# - tor.wowplanet.de
|
|
||||||
# - torix-relays.org
|
|
||||||
# - tse.com
|
|
||||||
# - w.digidow.eu
|
|
||||||
# - w.cccs.de
|
|
||||||
|
|
||||||
def oMakeController(sSock='', port=9051):
|
|
||||||
import getpass
|
|
||||||
if sSock and os.path.exists(sSock):
|
|
||||||
controller = Controller.from_socket_file(path=sSock)
|
|
||||||
else:
|
|
||||||
controller = Controller.from_port(port=port)
|
|
||||||
sys.stdout.flush()
|
|
||||||
p = getpass.unix_getpass(prompt='Controller Password: ', stream=sys.stderr)
|
|
||||||
controller.authenticate(p)
|
|
||||||
return controller
|
|
||||||
|
|
||||||
oSTEM_CONTROLER = None
|
|
||||||
def oGetStemController(log_level=10, sock_or_pair='/run/tor/control'):
|
|
||||||
|
|
||||||
global oSTEM_CONTROLER
|
|
||||||
if oSTEM_CONTROLER: return oSTEM_CONTROLER
|
|
||||||
import stem.util.log
|
|
||||||
# stem.util.log.Runlevel = 'DEBUG' = 20 # log_level
|
|
||||||
|
|
||||||
if os.path.exists(sock_or_pair):
|
|
||||||
LOG.info(f"controller from socket {sock_or_pair}")
|
|
||||||
controller = Controller.from_socket_file(path=sock_or_pair)
|
|
||||||
else:
|
|
||||||
if type(sock_or_pair) == int:
|
|
||||||
port = sock_or_pair
|
|
||||||
elif ':' in sock_or_pair:
|
|
||||||
port = sock_or_pair.split(':')[1]
|
|
||||||
else:
|
|
||||||
port = sock_or_pair
|
|
||||||
try:
|
|
||||||
port = int(port)
|
|
||||||
except: port = 9051
|
|
||||||
LOG.info(f"controller from port {port}")
|
|
||||||
# stem.SocketError
|
|
||||||
controller = Controller.from_port(port=port)
|
|
||||||
try:
|
|
||||||
controller.authenticate()
|
|
||||||
except (Exception, MissingPassword):
|
|
||||||
sys.stdout.flush()
|
|
||||||
p = getpass.unix_getpass(prompt='Controller Password: ', stream=sys.stderr)
|
|
||||||
controller.authenticate(p)
|
|
||||||
oSTEM_CONTROLER = controller
|
|
||||||
LOG.debug(f"{controller}")
|
|
||||||
return oSTEM_CONTROLER
|
|
||||||
|
|
||||||
def bAreWeConnected():
|
|
||||||
# FixMe: Linux only
|
|
||||||
sFile = f"/proc/{os.getpid()}/net/route"
|
|
||||||
if not os.path.isfile(sFile): return None
|
|
||||||
i = 0
|
|
||||||
for elt in open(sFile, "r").readlines():
|
|
||||||
if elt.startswith('Iface'): continue
|
|
||||||
if elt.startswith('lo'): continue
|
|
||||||
i += 1
|
|
||||||
return i > 0
|
|
||||||
|
|
||||||
def sMapaddressResolv(target, iPort=9051, log_level=10):
|
|
||||||
if not stem:
|
|
||||||
LOG.warn('please install the stem Python package')
|
|
||||||
return ''
|
|
||||||
|
|
||||||
try:
|
|
||||||
controller = oGetStemController(log_level=log_level)
|
|
||||||
|
|
||||||
map_dict = {"0.0.0.0": target}
|
|
||||||
map_ret = controller.map_address(map_dict)
|
|
||||||
|
|
||||||
return map_ret
|
|
||||||
except Exception as e:
|
|
||||||
LOG.exception(e)
|
|
||||||
return ''
|
|
||||||
|
|
||||||
def vwait_for_controller(controller, wait_boot=10):
|
|
||||||
if bAreWeConnected() is False:
|
|
||||||
raise SystemExit("we are not connected")
|
|
||||||
percent = i = 0
|
|
||||||
# You can call this while boostrapping
|
|
||||||
while percent < 100 and i < wait_boot:
|
|
||||||
bootstrap_status = controller.get_info("status/bootstrap-phase")
|
|
||||||
progress_percent = re.match('.* PROGRESS=([0-9]+).*', bootstrap_status)
|
|
||||||
percent = int(progress_percent.group(1))
|
|
||||||
LOG.info(f"Bootstrapping {percent}%")
|
|
||||||
time.sleep(5)
|
|
||||||
i += 5
|
|
||||||
|
|
||||||
def bin_to_hex(raw_id, length=None):
|
|
||||||
if length is None: length = len(raw_id)
|
|
||||||
res = ''.join('{:02x}'.format(raw_id[i]) for i in range(length))
|
|
||||||
return res.upper()
|
|
||||||
|
|
||||||
def lIntroductionPoints(controller=None, lOnions=[], itimeout=120, log_level=10):
|
|
||||||
"""now working !!! stem 1.8.x timeout must be huge >120
|
|
||||||
'Provides the descriptor for a hidden service. The **address** is the
|
|
||||||
'.onion' address of the hidden service '
|
|
||||||
What about Services?
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
from cryptography.utils import int_from_bytes
|
|
||||||
except ImportError:
|
|
||||||
import cryptography.utils
|
|
||||||
|
|
||||||
# guessing - not in the current cryptography but stem expects it
|
|
||||||
def int_from_bytes(**args): return int.to_bytes(*args)
|
|
||||||
cryptography.utils.int_from_bytes = int_from_bytes
|
|
||||||
# this will fai if the trick above didnt work
|
|
||||||
from stem.prereq import is_crypto_available
|
|
||||||
is_crypto_available(ed25519=True)
|
|
||||||
|
|
||||||
from queue import Empty
|
|
||||||
|
|
||||||
from stem import Timeout
|
|
||||||
from stem.client.datatype import LinkByFingerprint
|
|
||||||
from stem.descriptor.hidden_service import HiddenServiceDescriptorV3
|
|
||||||
|
|
||||||
if type(lOnions) not in [set, tuple, list]:
|
|
||||||
lOnions = list(lOnions)
|
|
||||||
if controller is None:
|
|
||||||
controller = oGetStemController(log_level=log_level)
|
|
||||||
l = []
|
|
||||||
for elt in lOnions:
|
|
||||||
LOG.info(f"controller.get_hidden_service_descriptor {elt}")
|
|
||||||
try:
|
|
||||||
desc = controller.get_hidden_service_descriptor(elt,
|
|
||||||
await_result=True,
|
|
||||||
timeout=itimeout)
|
|
||||||
# LOG.log(40, f"{dir(desc)} get_hidden_service_descriptor")
|
|
||||||
# timeouts 20 sec
|
|
||||||
# mistakenly a HSv2 descriptor
|
|
||||||
hs_address = HiddenServiceDescriptorV3.from_str(str(desc)) # reparse as HSv3
|
|
||||||
oInnerLayer = hs_address.decrypt(elt)
|
|
||||||
# LOG.log(40, f"{dir(oInnerLayer)}")
|
|
||||||
|
|
||||||
# IntroductionPointV3
|
|
||||||
n = oInnerLayer.introduction_points
|
|
||||||
if not n:
|
|
||||||
LOG.warn(f"NO introduction points for {elt}")
|
|
||||||
continue
|
|
||||||
LOG.info(f"{elt} {len(n)} introduction points")
|
|
||||||
lp = []
|
|
||||||
for introduction_point in n:
|
|
||||||
for linkspecifier in introduction_point.link_specifiers:
|
|
||||||
if isinstance(linkspecifier, LinkByFingerprint):
|
|
||||||
# LOG.log(40, f"Getting fingerprint for {linkspecifier}")
|
|
||||||
if hasattr(linkspecifier, 'fingerprint'):
|
|
||||||
assert len(linkspecifier.value) == 20
|
|
||||||
lp += [bin_to_hex(linkspecifier.value)]
|
|
||||||
LOG.info(f"{len(lp)} introduction points for {elt}")
|
|
||||||
l += lp
|
|
||||||
except (Empty, Timeout,) as e: # noqa
|
|
||||||
LOG.warn(f"Timed out getting introduction points for {elt}")
|
|
||||||
continue
|
|
||||||
except Exception as e:
|
|
||||||
LOG.exception(e)
|
|
||||||
return l
|
|
||||||
|
|
||||||
def zResolveDomain(domain):
|
|
||||||
try:
|
|
||||||
ip = sTorResolve(domain)
|
|
||||||
except Exception as e: # noqa
|
|
||||||
ip = ''
|
|
||||||
if ip == '':
|
|
||||||
try:
|
|
||||||
lpair = getaddrinfo(domain, 443)
|
|
||||||
except Exception as e:
|
|
||||||
LOG.warn(f"{e}")
|
|
||||||
lpair = None
|
|
||||||
if lpair is None:
|
|
||||||
LOG.warn(f"TorResolv and getaddrinfo failed for {domain}")
|
|
||||||
return ''
|
|
||||||
ip = lpair[0]
|
|
||||||
return ip
|
|
||||||
|
|
||||||
def sTorResolve(target,
|
|
||||||
verbose=False,
|
|
||||||
sHost='127.0.0.1',
|
|
||||||
iPort=9050,
|
|
||||||
SOCK_TIMEOUT_SECONDS=10.0,
|
|
||||||
SOCK_TIMEOUT_TRIES=3,
|
|
||||||
):
|
|
||||||
MAX_INFO_RESPONSE_PACKET_LENGTH = 8
|
|
||||||
if '@' in target:
|
|
||||||
LOG.warn(f"sTorResolve failed invalid hostname {target}")
|
|
||||||
return ''
|
|
||||||
target = target.strip('/')
|
|
||||||
seb = b"\x04\xf0\x00\x00\x00\x00\x00\x01\x00"
|
|
||||||
seb += bytes(target, 'US-ASCII') + b"\x00"
|
|
||||||
assert len(seb) == 10 + len(target), str(len(seb)) + repr(seb)
|
|
||||||
|
|
||||||
# LOG.debug(f"0 Sending {len(seb)} to The TOR proxy {seb}")
|
|
||||||
|
|
||||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
|
||||||
sock.connect((sHost, iPort))
|
|
||||||
|
|
||||||
sock.settimeout(SOCK_TIMEOUT_SECONDS)
|
|
||||||
oRet = sock.sendall(seb) # noqa
|
|
||||||
|
|
||||||
i = 0
|
|
||||||
data = ''
|
|
||||||
while i < SOCK_TIMEOUT_TRIES:
|
|
||||||
i += 1
|
|
||||||
time.sleep(3)
|
|
||||||
lReady = select.select([sock.fileno()], [], [],
|
|
||||||
SOCK_TIMEOUT_SECONDS)
|
|
||||||
if not lReady[0]: continue
|
|
||||||
try:
|
|
||||||
flags=socket.MSG_WAITALL
|
|
||||||
data = sock.recv(MAX_INFO_RESPONSE_PACKET_LENGTH, flags)
|
|
||||||
except socket.timeout:
|
|
||||||
LOG.warn(f"4 The TOR proxy {(sHost, iPort)}" \
|
|
||||||
+" didnt reply in " + str(SOCK_TIMEOUT_SECONDS) + " sec."
|
|
||||||
+" #" +str(i))
|
|
||||||
except Exception as e:
|
|
||||||
LOG.error("4 The TOR proxy " \
|
|
||||||
+repr((sHost, iPort)) \
|
|
||||||
+" errored with " + str(e)
|
|
||||||
+" #" +str(i))
|
|
||||||
sock.close()
|
|
||||||
return ''
|
|
||||||
else:
|
|
||||||
if len(data) > 0: break
|
|
||||||
|
|
||||||
if len(data) == 0:
|
|
||||||
if i > SOCK_TIMEOUT_TRIES:
|
|
||||||
sLabel = "5 No reply #"
|
|
||||||
else:
|
|
||||||
sLabel = "5 No data #"
|
|
||||||
LOG.warn(f"sTorResolve: {sLabel} {i} on {sHost}:{iPort}")
|
|
||||||
sock.close()
|
|
||||||
return ''
|
|
||||||
|
|
||||||
assert len(data) >= 8
|
|
||||||
packet_sf = data[1]
|
|
||||||
if packet_sf == 90:
|
|
||||||
# , "%d" % packet_sf
|
|
||||||
assert f"{packet_sf}" == "90", f"packet_sf = {packet_sf}"
|
|
||||||
return f"{data[4]}.{data[5]}.{data[6]}.{data[7]}"
|
|
||||||
else:
|
|
||||||
# 91
|
|
||||||
LOG.warn(f"tor-resolve failed for {target} on {sHost}:{iPort}")
|
|
||||||
|
|
||||||
os.system(f"tor-resolve -4 {target} > /tmp/e 2>/dev/null")
|
|
||||||
# os.system("strace tor-resolve -4 "+target+" 2>&1|grep '^sen\|^rec'")
|
|
||||||
|
|
||||||
return ''
|
|
||||||
|
|
||||||
def getaddrinfo(sHost, sPort):
|
|
||||||
# do this the explicit way = Ive seen the compact connect fail
|
|
||||||
# >>> sHost, sPort = 'l27.0.0.1', 33446
|
|
||||||
# >>> sock.connect((sHost, sPort))
|
|
||||||
# socket.gaierror: [Errno -2] Name or service not known
|
|
||||||
try:
|
|
||||||
lElts = socket.getaddrinfo(sHost, int(sPort), socket.AF_INET)
|
|
||||||
lElts = list(filter(lambda elt: elt[1] == socket.SOCK_DGRAM, lElts))
|
|
||||||
assert len(lElts) == 1, repr(lElts)
|
|
||||||
lPair = lElts[0][-1]
|
|
||||||
assert len(lPair) == 2, repr(lPair)
|
|
||||||
assert type(lPair[1]) == int, repr(lPair)
|
|
||||||
except (socket.gaierror, OSError, BaseException) as e:
|
|
||||||
LOG.error(e)
|
|
||||||
return None
|
|
||||||
return lPair
|
|
||||||
|
|
||||||
def icheck_torrc(sFile, oArgs):
|
|
||||||
l = open(sFile, 'rt').readlines()
|
|
||||||
a = {}
|
|
||||||
for elt in l:
|
|
||||||
elt = elt.strip()
|
|
||||||
if not elt or ' ' not in elt: continue
|
|
||||||
(k, v,) = elt.split(' ', 1)
|
|
||||||
a[k] = v
|
|
||||||
keys = a
|
|
||||||
|
|
||||||
if 'HashedControlPassword' not in keys:
|
|
||||||
LOG.info('Add HashedControlPassword for security')
|
|
||||||
print('run: tor --hashcontrolpassword <TopSecretWord>')
|
|
||||||
if 'ExcludeExitNodes' in keys:
|
|
||||||
elt = 'BadNodes.ExcludeExitNodes.BadExit'
|
|
||||||
LOG.warn(f"Remove ExcludeNodes and move then to {oArgs.bad_nodes}")
|
|
||||||
print(f"move to the {elt} section as a list")
|
|
||||||
if 'GuardNodes' in keys:
|
|
||||||
elt = 'GoodNodes.GuardNodes'
|
|
||||||
LOG.warn(f"Remove GuardNodes and move then to {oArgs.good_nodes}")
|
|
||||||
print(f"move to the {elt} section as a list")
|
|
||||||
if 'ExcludeNodes' in keys:
|
|
||||||
elt = 'BadNodes.ExcludeNodes.BadExit'
|
|
||||||
LOG.warn(f"Remove ExcludeNodes and move then to {oArgs.bad_nodes}")
|
|
||||||
print(f"move to the {elt} section as a list")
|
|
||||||
if 'ControlSocket' not in keys and os.path.exists('/run/tor/control'):
|
|
||||||
LOG.info('Add ControlSocket /run/tor/control for us')
|
|
||||||
print('ControlSocket /run/tor/control GroupWritable RelaxDirModeCheck')
|
|
||||||
if 'UseMicrodescriptors' not in keys or keys['UseMicrodescriptors'] != '1':
|
|
||||||
LOG.info('Add UseMicrodescriptors 0 for us')
|
|
||||||
print('UseMicrodescriptors 0')
|
|
||||||
if 'AutomapHostsSuffixes' not in keys:
|
|
||||||
LOG.info('Add AutomapHostsSuffixes for onions')
|
|
||||||
print('AutomapHostsSuffixes .exit,.onion')
|
|
||||||
if 'AutoMapHostsOnResolve' not in keys:
|
|
||||||
LOG.info('Add AutoMapHostsOnResolve for onions')
|
|
||||||
print('AutoMapHostsOnResolve 1')
|
|
||||||
if 'VirtualAddrNetworkIPv4' not in keys:
|
|
||||||
LOG.info('Add VirtualAddrNetworkIPv4 for onions')
|
|
||||||
print('VirtualAddrNetworkIPv4 172.16.0.0/12')
|
|
||||||
return 0
|
|
||||||
|
|
||||||
def lExitExcluder(oArgs, iPort=9051, log_level=10):
|
|
||||||
"""
|
|
||||||
https://raw.githubusercontent.com/nusenu/noContactInfo_Exit_Excluder/main/exclude_noContactInfo_Exits.py
|
|
||||||
"""
|
|
||||||
if not stem:
|
|
||||||
LOG.warn('please install the stem Python package')
|
|
||||||
return ''
|
|
||||||
LOG.debug('lExcludeExitNodes')
|
|
||||||
|
|
||||||
try:
|
|
||||||
controller = oGetStemController(log_level=log_level)
|
|
||||||
# generator
|
|
||||||
relays = controller.get_server_descriptors()
|
|
||||||
except Exception as e:
|
|
||||||
LOG.error(f'Failed to get relay descriptors {e}')
|
|
||||||
return None
|
|
||||||
|
|
||||||
if controller.is_set('ExcludeExitNodes'):
|
|
||||||
LOG.info('ExcludeExitNodes is in use already.')
|
|
||||||
return None
|
|
||||||
|
|
||||||
exit_excludelist=[]
|
|
||||||
LOG.debug("Excluded exit relays:")
|
|
||||||
for relay in relays:
|
|
||||||
if relay.exit_policy.is_exiting_allowed() and not relay.contact:
|
|
||||||
if is_valid_fingerprint(relay.fingerprint):
|
|
||||||
exit_excludelist.append(relay.fingerprint)
|
|
||||||
LOG.debug("https://metrics.torproject.org/rs.html#details/%s" % relay.fingerprint)
|
|
||||||
else:
|
|
||||||
LOG.warn('Invalid Fingerprint: %s' % relay.fingerprint)
|
|
||||||
|
|
||||||
try:
|
|
||||||
controller.set_conf('ExcludeExitNodes', exit_excludelist)
|
|
||||||
LOG.info('Excluded a total of %s exit relays without ContactInfo from the exit position.' % len(exit_excludelist))
|
|
||||||
except Exception as e:
|
|
||||||
LOG.exception('ExcludeExitNodes ' +str(e))
|
|
||||||
return exit_excludelist
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
target = 'duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad'
|
|
||||||
controller = oGetStemController(log_level=10)
|
|
||||||
lIntroductionPoints(controller, [target], itimeout=120)
|
|
||||||
@ -1,48 +0,0 @@
|
|||||||
#!/usr/local/bin/python3.sh
|
|
||||||
# -*-mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -*
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
def omain_argparser(_=None):
|
|
||||||
|
|
||||||
try:
|
|
||||||
from OpenSSL import SSL
|
|
||||||
lCAfs = SSL._CERTIFICATE_FILE_LOCATIONS
|
|
||||||
except:
|
|
||||||
lCAfs = []
|
|
||||||
|
|
||||||
CAfs = []
|
|
||||||
for elt in lCAfs:
|
|
||||||
if os.path.exists(elt):
|
|
||||||
CAfs.append(elt)
|
|
||||||
if not CAfs:
|
|
||||||
CAfs = ['']
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(add_help=True,
|
|
||||||
epilog=__doc__)
|
|
||||||
parser.add_argument('--https_cafile', type=str,
|
|
||||||
help="Certificate Authority file (in PEM) (unused)",
|
|
||||||
default=CAfs[0])
|
|
||||||
parser.add_argument('--log_level', type=int, default=20,
|
|
||||||
help="10=debug 20=info 30=warn 40=error")
|
|
||||||
parser.add_argument('--js_input', type=str, default='',
|
|
||||||
help="Operate on the HTML file with javascript")
|
|
||||||
parser.add_argument('--html_output', type=str, default='',
|
|
||||||
help="Write loaded and javascripted result to a HTML file")
|
|
||||||
parser.add_argument('--pdf_output', type=str, default='',
|
|
||||||
help="Write loaded and javascripted result to a PDF file")
|
|
||||||
parser.add_argument('--show_gui', type=bool, default=False, store_action=True,
|
|
||||||
help="show a progress meter that doesn't work")
|
|
||||||
parser.add_argument('html_url', type=str, nargs='?',
|
|
||||||
required=True,
|
|
||||||
help='html file or url')
|
|
||||||
return parser
|
|
||||||
@ -1,37 +0,0 @@
|
|||||||
#!/usr/bin/python3 -u
|
|
||||||
|
|
||||||
## Copyright (C) 2012 - 2020 ENCRYPTED SUPPORT LP <adrelanos@riseup.net>
|
|
||||||
## See the file COPYING for copying conditions.
|
|
||||||
|
|
||||||
import sys
|
|
||||||
from stem.connection import connect
|
|
||||||
import re
|
|
||||||
|
|
||||||
controller = connect()
|
|
||||||
|
|
||||||
if not controller:
|
|
||||||
sys.exit(255)
|
|
||||||
|
|
||||||
bootstrap_status = controller.get_info("status/bootstrap-phase")
|
|
||||||
|
|
||||||
## Possible answer, if network cable has been removed:
|
|
||||||
## 250-status/bootstrap-phase=WARN BOOTSTRAP PROGRESS=80 TAG=conn_or SUMMARY="Connecting to the Tor network" WARNING="No route to host" REASON=NOROUTE COUNT=26 RECOMMENDATION=warn
|
|
||||||
|
|
||||||
## Possible answer:
|
|
||||||
## 250-status/bootstrap-phase=NOTICE BOOTSTRAP PROGRESS=85 TAG=handshake_or SUMMARY="Finishing handshake with first hop"
|
|
||||||
|
|
||||||
## Possible answer, when done:
|
|
||||||
## 250-status/bootstrap-phase=NOTICE BOOTSTRAP PROGRESS=100 TAG=done SUMMARY="Done"
|
|
||||||
|
|
||||||
## TODO: parse the messages above.
|
|
||||||
## 0
|
|
||||||
|
|
||||||
print(format(bootstrap_status))
|
|
||||||
|
|
||||||
progress_percent = re.match('.* PROGRESS=([0-9]+).*', bootstrap_status)
|
|
||||||
|
|
||||||
exit_code = int(progress_percent.group(1))
|
|
||||||
|
|
||||||
controller.close()
|
|
||||||
|
|
||||||
sys.exit(exit_code)
|
|
||||||
@ -1,613 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Tor Contact Info Parser - A tool/Python Class for parsing Tor ContactInfo Information Sharing v2 specification contacts
|
|
||||||
Written by Eran Sandler (https://twitter.com/erans) (C) 2018
|
|
||||||
|
|
||||||
Turned into a proper command-line tool with sub-commands and flags by @Someguy123 at Privex Inc. (C) 2021
|
|
||||||
(https://www.privex.io) (https://github.com/PrivexInc)
|
|
||||||
|
|
||||||
This is a parser for the Tor ContactInfo Information Sharing Specification v2 (https://nusenu.github.io/ContactInfo-Information-Sharing-Specification/).
|
|
||||||
|
|
||||||
The parser can parse the ContactInfo field of Tor relays based on the specification.
|
|
||||||
|
|
||||||
Official Repo: https://github.com/erans/torcontactinfoparser
|
|
||||||
Privex Fork: https://github.com/Privex/torcontactinfoparser
|
|
||||||
|
|
||||||
Released under the MIT License.
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import json
|
|
||||||
import requests
|
|
||||||
import textwrap
|
|
||||||
try:
|
|
||||||
from rich import print as rprint
|
|
||||||
HAS_RICH = True
|
|
||||||
except ImportError:
|
|
||||||
def rprint(value='', *args, **kwargs):
|
|
||||||
if value not in [None, False, True] and isinstance(value, (dict, list, set, tuple)):
|
|
||||||
value = json.dumps(value, indent=4)
|
|
||||||
return print(value, *args, **kwargs)
|
|
||||||
# rprint = print
|
|
||||||
HAS_RICH = False
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
class TorContactInfoParser(object):
|
|
||||||
email_regex = "^[a-zA-Z0-9.!#$%&’*+/=?^_`{|}~-]+@[a-zA-Z0-9-]+(?:\\.[a-zA-Z0-9-]+)*$"
|
|
||||||
|
|
||||||
def _parse_string_value(self, value, min_length, max_length, valid_chars, raise_exception=False, field_name=None, deobfuscate_email=False):
|
|
||||||
value_length = len(value)
|
|
||||||
if value_length < min_length:
|
|
||||||
if raise_exception:
|
|
||||||
raise ValueError("value of field '{0}' is too short".format(field_name))
|
|
||||||
return None
|
|
||||||
|
|
||||||
if value_length > max_length:
|
|
||||||
if raise_exception:
|
|
||||||
raise ValueError("value of field '{0}' is too long".format(field_name))
|
|
||||||
return None
|
|
||||||
|
|
||||||
if valid_chars != "*":
|
|
||||||
m = re.search(valid_chars, value)
|
|
||||||
if not m:
|
|
||||||
if raise_exception:
|
|
||||||
raise ValueError("value of field '{0}' doesn't match valid chars restrictions".format(field_name))
|
|
||||||
else:
|
|
||||||
return None
|
|
||||||
|
|
||||||
return value
|
|
||||||
|
|
||||||
def _parse_email_value(self, value, field_name, raise_exception, deobfuscate_email):
|
|
||||||
if value:
|
|
||||||
v = value.replace("[]", "@")
|
|
||||||
if re.search(self.email_regex, v):
|
|
||||||
if not deobfuscate_email:
|
|
||||||
return v.replace("@", "[]")
|
|
||||||
|
|
||||||
return v
|
|
||||||
|
|
||||||
return None
|
|
||||||
|
|
||||||
_supported_fields_parsers = {
|
|
||||||
"email" : {
|
|
||||||
"fn": _parse_email_value,
|
|
||||||
"args": {}
|
|
||||||
},
|
|
||||||
"url" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 4,
|
|
||||||
"max_length" : 399,
|
|
||||||
"valid_chars" : "[_%/:a-zA-Z0-9.-]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"proof" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 7,
|
|
||||||
"max_length" : 7,
|
|
||||||
"valid_chars" : "[adinrsu-]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"ciissversion" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[12]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"pgp" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 40,
|
|
||||||
"max_length" : 40,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"abuse" : {
|
|
||||||
"fn": _parse_email_value,
|
|
||||||
"args": {}
|
|
||||||
},
|
|
||||||
"keybase" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 50,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"twitter" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 15,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9_]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"mastodon" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 254,
|
|
||||||
"valid_chars" : "*"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"matrix" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 254,
|
|
||||||
"valid_chars" : "*"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"xmpp" : {
|
|
||||||
"fn": _parse_email_value,
|
|
||||||
"args": {}
|
|
||||||
},
|
|
||||||
"otr3" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 40,
|
|
||||||
"max_length" : 40,
|
|
||||||
"valid_chars" : "[a-z0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"hoster" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 254,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9.-]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"cost" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 13,
|
|
||||||
"valid_chars" : "[A-Z0-9.]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"uplinkbw" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 7,
|
|
||||||
"valid_chars" : "[0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"trafficacct" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 9,
|
|
||||||
"valid_chars" : "[unmetrd0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"memory" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 10,
|
|
||||||
"valid_chars" : "[0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"cpu" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 50,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9_-]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"virtualization" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 15,
|
|
||||||
"valid_chars" : "[a-z-]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"donationurl" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 254,
|
|
||||||
"valid_chars" : "*"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"btc" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 26,
|
|
||||||
"max_length" : 99,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"zec" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 95,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"xmr" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 99,
|
|
||||||
"valid_chars" : "[a-zA-Z0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"offlinemasterkey" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"signingkeylifetime" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 6,
|
|
||||||
"valid_chars" : "[0-9]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"sandbox" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 2,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"os" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 20,
|
|
||||||
"valid_chars" : "[A-Za-z0-9/.]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"tls" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 0,
|
|
||||||
"max_length" : 14,
|
|
||||||
"valid_chars" : "[a-z]+"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"aesni" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"autoupdate" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"confmgmt" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 15,
|
|
||||||
"valid_chars" : "[a-zA-Z-]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"dnslocation" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 5,
|
|
||||||
"max_length" : 100,
|
|
||||||
"valid_chars" : "[a-z,]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"dnsqname" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"dnssec" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"dnslocalrootzone" : {
|
|
||||||
"fn" : _parse_string_value,
|
|
||||||
"args" : {
|
|
||||||
"min_length" : 1,
|
|
||||||
"max_length" : 1,
|
|
||||||
"valid_chars" : "[yn]"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
def __init__(self):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def parse(self, value: str, raise_exception_on_invalid_value=False, deobfuscate_email=True) -> dict:
|
|
||||||
# the ciissversion field is mandatory
|
|
||||||
if not 'ciissversion:' in value:
|
|
||||||
return None
|
|
||||||
|
|
||||||
result = {}
|
|
||||||
parts = value.split(" ")
|
|
||||||
for p in parts:
|
|
||||||
field_parts = p.split(":", 1)
|
|
||||||
if len(field_parts) <= 1:
|
|
||||||
continue
|
|
||||||
name, data = field_parts
|
|
||||||
if name in self._supported_fields_parsers:
|
|
||||||
field_parser = self._supported_fields_parsers[name]
|
|
||||||
if field_parser is None:
|
|
||||||
result[name] = data
|
|
||||||
continue
|
|
||||||
if callable(field_parser):
|
|
||||||
value = field_parser(self, data)
|
|
||||||
else:
|
|
||||||
field_parser["args"]["field_name"] = name
|
|
||||||
field_parser["args"]["value"] = data
|
|
||||||
field_parser["args"]["raise_exception"] = raise_exception_on_invalid_value
|
|
||||||
field_parser["args"]["deobfuscate_email"] = deobfuscate_email
|
|
||||||
|
|
||||||
value = field_parser["fn"](self, **field_parser["args"])
|
|
||||||
|
|
||||||
if not result.get(name, None):
|
|
||||||
result[name] = value
|
|
||||||
|
|
||||||
return result
|
|
||||||
|
|
||||||
def cmd_parse(opts: argparse.Namespace):
|
|
||||||
"""
|
|
||||||
ArgParser function for parsing a single ContactInfo string, and outputting it as JSON (or python-style dict's)
|
|
||||||
"""
|
|
||||||
|
|
||||||
if opts.contact is None or len(opts.contact) == 0 or opts.contact[0] == '-':
|
|
||||||
contact = sys.stdin.read()
|
|
||||||
else:
|
|
||||||
contact = ' '.join(opts.contact).strip()
|
|
||||||
|
|
||||||
tparser = TorContactInfoParser()
|
|
||||||
res = tparser.parse(contact)
|
|
||||||
if not opts.pretty:
|
|
||||||
return print(json.dumps(res))
|
|
||||||
if opts.json:
|
|
||||||
res = json.dumps(res, indent=4) if opts.pretty else json.dumps(res)
|
|
||||||
# if not HAS_RICH: res = json.dumps(res, indent=4)
|
|
||||||
rprint(res)
|
|
||||||
|
|
||||||
def vsetup_logging(log_level, logfile='', stream=sys.stderr):
|
|
||||||
global LOG
|
|
||||||
add = True
|
|
||||||
|
|
||||||
try:
|
|
||||||
if 'COLOREDLOGS_LEVEL_STYLES' not in os.environ:
|
|
||||||
os.environ['COLOREDLOGS_LEVEL_STYLES'] = 'spam=22;debug=28;verbose=34;notice=220;warning=202;success=118,bold;error=124;critical=background=red'
|
|
||||||
# https://pypi.org/project/coloredlogs/
|
|
||||||
import coloredlogs
|
|
||||||
except ImportError:
|
|
||||||
coloredlogs = False
|
|
||||||
|
|
||||||
logging._defaultFormatter = logging.Formatter(datefmt='%m-%d %H:%M:%S')
|
|
||||||
logging._defaultFormatter.default_time_format = '%m-%d %H:%M:%S'
|
|
||||||
logging._defaultFormatter.default_msec_format = ''
|
|
||||||
|
|
||||||
kwargs = dict(level=log_level,
|
|
||||||
force=True,
|
|
||||||
format='%(levelname)s %(message)s')
|
|
||||||
|
|
||||||
if logfile:
|
|
||||||
add = logfile.startswith('+')
|
|
||||||
sub = logfile.startswith('-')
|
|
||||||
if add or sub:
|
|
||||||
logfile = logfile[1:]
|
|
||||||
kwargs['filename'] = logfile
|
|
||||||
|
|
||||||
if coloredlogs:
|
|
||||||
# https://pypi.org/project/coloredlogs/
|
|
||||||
aKw = dict(level=log_level,
|
|
||||||
logger=LOG,
|
|
||||||
stream=stream,
|
|
||||||
fmt='%(levelname)s %(message)s'
|
|
||||||
)
|
|
||||||
coloredlogs.install(**aKw)
|
|
||||||
if logfile:
|
|
||||||
oHandler = logging.FileHandler(logfile)
|
|
||||||
LOG.addHandler(oHandler)
|
|
||||||
LOG.info(f"CSetting log_level to {log_level} {stream}")
|
|
||||||
else:
|
|
||||||
logging.basicConfig(**kwargs)
|
|
||||||
if add and logfile:
|
|
||||||
oHandler = logging.StreamHandler(stream)
|
|
||||||
LOG.addHandler(oHandler)
|
|
||||||
LOG.info(f"SSetting log_level to {log_level!s}")
|
|
||||||
|
|
||||||
def cmd_scan(opts: argparse.Namespace, adata=None):
|
|
||||||
"""
|
|
||||||
ArgParser function for scanning all ContactInfo strings from ``https://onionoo.torproject.org/details`` ,
|
|
||||||
and outputting each one as a Python-style Dict, or JSON.
|
|
||||||
"""
|
|
||||||
parser = TorContactInfoParser()
|
|
||||||
surl = "https://onionoo.torproject.org/details"
|
|
||||||
|
|
||||||
if not adata:
|
|
||||||
LOG.info(f"Getting relays from {surl}")
|
|
||||||
jdata = requests.get(surl)
|
|
||||||
try:
|
|
||||||
adata = jdata.json()
|
|
||||||
except Exception as e:
|
|
||||||
# simplejson.errors.JSONDecodeError
|
|
||||||
LOG.exception(f"JSON error {e}")
|
|
||||||
return
|
|
||||||
elts = adata["relays"]
|
|
||||||
else:
|
|
||||||
elts = json.loads(adata)['relays']
|
|
||||||
|
|
||||||
if not elts:
|
|
||||||
LOG.warn(f"NO relays - are we connected?")
|
|
||||||
return
|
|
||||||
LOG.info(f"{len(elts)} relays")
|
|
||||||
for relay in elts:
|
|
||||||
if 'fingerprint' not in relay.keys():
|
|
||||||
LOG.warn(f"fingerprint not in relay for {relay}")
|
|
||||||
continue
|
|
||||||
fp = relay['fingerprint']
|
|
||||||
verified_host_names = relay.get('verified_host_names', [])
|
|
||||||
contact = relay.get("contact", None)
|
|
||||||
if not contact:
|
|
||||||
LOG.warn(f"No contact for {fp} {verified_host_names}")
|
|
||||||
continue
|
|
||||||
if 'ciissversion' not in contact:
|
|
||||||
LOG.debug(f"No ciissversion in contact in {fp}")
|
|
||||||
continue
|
|
||||||
LOG.debug(f"parsing {fp}")
|
|
||||||
result = parser.parse(contact, False)
|
|
||||||
if not result:
|
|
||||||
LOG.warn(f"No result for {contact} in {fp}")
|
|
||||||
continue
|
|
||||||
if len(result) > 0:
|
|
||||||
if opts.json: result = json.dumps(result, indent=4) if opts.pretty else json.dumps(result)
|
|
||||||
if opts.pretty:
|
|
||||||
rprint(result)
|
|
||||||
else:
|
|
||||||
print(result)
|
|
||||||
|
|
||||||
ETC_DIR = '/etc/tor/yaml'
|
|
||||||
def oparser():
|
|
||||||
cparser = argparse.ArgumentParser(
|
|
||||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
|
||||||
description=textwrap.dedent(f"""
|
|
||||||
Examples:
|
|
||||||
|
|
||||||
# 'scan' is the original behaviour of this script. It iterates over the data
|
|
||||||
# from https://onionoo.torproject.org/details , parses each contact, and prints it as Python dict-style JSON.
|
|
||||||
{sys.argv[0]} scan
|
|
||||||
|
|
||||||
# Same as previous. With no arguments, it's equivalent to running 'scan'.
|
|
||||||
{sys.argv[0]}
|
|
||||||
|
|
||||||
# If you pass '-p' after scan, it will enable pretty printing. For best pretty printing,
|
|
||||||
# make sure you have 'rich' installed from pypi.
|
|
||||||
{sys.argv[0]} scan -p
|
|
||||||
|
|
||||||
# If you need real JSON with double quotes, rather than Python dict-style JSON, you can
|
|
||||||
# use the '-j' flag to enable "real JSON" mode (you can combine with '-p' if you want pretty printed real json)
|
|
||||||
{sys.argv[0]} scan -j
|
|
||||||
|
|
||||||
# Using 'parse', you can parse an arbitrary ContactInfo string, and it will output the parsed result
|
|
||||||
# with pretty printing by default.
|
|
||||||
|
|
||||||
{sys.argv[0]} parse "contact Privex Inc. email:noc[]privex.io url:https://www.privex.io " \\
|
|
||||||
"proof:uri-rsa pgp:288DD1632F6E8951 keybase:privexinc twitter:PrivexInc hoster:www.privex.io " \\
|
|
||||||
"uplinkbw:500 memory:4096 virtualization:kvm btc:bc1qpst9uscvd8rpjjhzz9rau3trylh6e0wh76qrlhw3q9nj89ua728sn3t6a2 " \\
|
|
||||||
"xmr:89tukP3wfpH4FZAmC1D2GfArWwfPTz8Ap46NZc54Vyhy9YxEUYoFQ7HGQ74LrCMQTD3zxvwM1ewmGjH9WVmeffwR72m1Pps"
|
|
||||||
|
|
||||||
{{
|
|
||||||
'email': 'noc@privex.io',
|
|
||||||
'url': 'https://www.privex.io',
|
|
||||||
'proof': 'uri-rsa',
|
|
||||||
'pgp': None,
|
|
||||||
'keybase': 'privexinc',
|
|
||||||
'twitter': 'PrivexInc',
|
|
||||||
'hoster': 'www.privex.io',
|
|
||||||
'uplinkbw': '500',
|
|
||||||
'memory': '4096',
|
|
||||||
'virtualization': 'kvm',
|
|
||||||
'btc': 'bc1qpst9uscvd8rpjjhzz9rau3trylh6e0wh76qrlhw3q9nj89ua728sn3t6a2',
|
|
||||||
'xmr': '89tukP3wfpH4FZAmC1D2GfArWwfPTz8Ap46NZc54Vyhy9YxEUYoFQ7HGQ74LrCMQTD3zxvwM1ewmGjH9WVmeffwR72m1Pps'
|
|
||||||
}}
|
|
||||||
|
|
||||||
# You can also pipe a contact string into 'parse', and it will work just the same.
|
|
||||||
|
|
||||||
echo "Privex Inc. email:noc[]privex.io url:https://www.privex.io proof:uri-rsa pgp:288DD1632F6E8951 keybase:privexinc twitter:PrivexInc" | {sys.argv[0]} parse
|
|
||||||
{{'email': 'noc@privex.io', 'url': 'https://www.privex.io', 'proof': 'uri-rsa', 'pgp': None, 'keybase': 'privexinc', 'twitter': 'PrivexInc\n'}}
|
|
||||||
|
|
||||||
# If you need real JSON outputted, rather than Python dict-style output, you can pass -j to either 'parse' or 'scan'
|
|
||||||
|
|
||||||
{sys.argv[0]} parse -j "Privex Inc. email:noc[]privex.io url:https://www.privex.io proof:uri-rsa pgp:288DD1632F6E8951 keybase:privexinc twitter:PrivexInc"
|
|
||||||
{{
|
|
||||||
"email": "noc@privex.io",
|
|
||||||
"url": "https://www.privex.io",
|
|
||||||
"proof": "uri-rsa",
|
|
||||||
"pgp": null,
|
|
||||||
"keybase": "privexinc",
|
|
||||||
"twitter": "PrivexInc"
|
|
||||||
}}
|
|
||||||
|
|
||||||
# You can use '-np' to disable pretty printing for 'parse' - you can combine it with '-j' to get flat, plain JSON.
|
|
||||||
|
|
||||||
{sys.argv[0]} parse -np -j "Privex Inc. email:noc[]privex.io url:https://www.privex.io proof:uri-rsa pgp:288DD1632F6E8951 keybase:privexinc twitter:PrivexInc"
|
|
||||||
{{"email": "noc@privex.io", "url": "https://www.privex.io", "proof": "uri-rsa", "pgp": null, "keybase": "privexinc", "twitter": "PrivexInc"}}
|
|
||||||
"""))
|
|
||||||
cparser.set_defaults(func=cmd_scan, json=False, pretty=False)
|
|
||||||
subparse = cparser.add_subparsers()
|
|
||||||
subparse.required = False
|
|
||||||
sp_parse = subparse.add_parser('parse',
|
|
||||||
help="Parse a single contact string, either as an argument, or piped into stdin")
|
|
||||||
sp_parse.add_argument('contact', nargs='*')
|
|
||||||
sp_parse.add_argument('-np', '--no-pretty',
|
|
||||||
action='store_false', default=False, dest='pretty',
|
|
||||||
help="Disable pretty printing JSON")
|
|
||||||
sp_parse.add_argument('--relays_output', type=str,
|
|
||||||
dest='relays_output',
|
|
||||||
default=os.path.join(ETC_DIR, 'relays.json'),
|
|
||||||
help="Write the download relays in json to a file")
|
|
||||||
sp_parse.add_argument('-j', '--json', action='store_true',
|
|
||||||
default=False, dest='json',
|
|
||||||
help="Output real JSON, not Python dict format.")
|
|
||||||
sp_parse.set_defaults(func=cmd_parse)
|
|
||||||
|
|
||||||
sp_scan = subparse.add_parser('scan', help="Parse all contacts from https://onionoo.torproject.org/details")
|
|
||||||
sp_scan.add_argument('-p', action='store_true', default=False, dest='pretty', help="Enable pretty printing JSON")
|
|
||||||
sp_scan.add_argument('-j', '--json', action='store_true', default=False, dest='json', help="Output real JSON, not Python dict format.")
|
|
||||||
|
|
||||||
# sp_scan.set_defaults(func=cmd_scan)
|
|
||||||
|
|
||||||
return cparser
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
if os.environ.get('DEBUG', ''):
|
|
||||||
log_level = 10
|
|
||||||
else:
|
|
||||||
log_level = 20
|
|
||||||
vsetup_logging(log_level)
|
|
||||||
try:
|
|
||||||
cparser = oparser()
|
|
||||||
opts = cparser.parse_args(sys.argv[1:])
|
|
||||||
data = None
|
|
||||||
if opts.relays_output and os.path.exists(opts.relays_output):
|
|
||||||
data = open(opts.relays_output, 'rt').read()
|
|
||||||
cmd_scan(opts, data)
|
|
||||||
except (requests.exceptions.ProxyError, Exception,) as e:
|
|
||||||
LOG.error(f"{e}")
|
|
||||||
i = 0
|
|
||||||
# else:
|
|
||||||
# args = cparser.parse_args(sys.argv[1:])
|
|
||||||
# i = args.func(args)
|
|
||||||
|
|
||||||
sys.exit(i)
|
|
||||||
@ -1,627 +0,0 @@
|
|||||||
# -*- mode: python; indent-tabs-mode: nil; py-indent-offset: 4; coding: utf-8 -
|
|
||||||
|
|
||||||
# from https://github.com/nusenu/trustor-poc
|
|
||||||
# with minor refactoring to make the code more Pythonic.
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import ipaddress
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
|
|
||||||
import urllib3.util
|
|
||||||
from urllib3.util import parse_url as urlparse
|
|
||||||
|
|
||||||
from stem.control import Controller
|
|
||||||
# from stem.util.tor_tools import *
|
|
||||||
|
|
||||||
try:
|
|
||||||
# unbound is not on pypi
|
|
||||||
from unbound import RR_CLASS_IN, RR_TYPE_TXT, ub_ctx
|
|
||||||
except:
|
|
||||||
ub_ctx = RR_TYPE_TXT = RR_CLASS_IN = None
|
|
||||||
|
|
||||||
global LOG
|
|
||||||
import logging
|
|
||||||
warnings.filterwarnings('ignore')
|
|
||||||
LOG = logging.getLogger()
|
|
||||||
|
|
||||||
logging.getLogger("urllib3").setLevel(logging.INFO)
|
|
||||||
# import urllib3.contrib.pyopenssl
|
|
||||||
# urllib3.contrib.pyopenssl.inject_into_urllib3()
|
|
||||||
|
|
||||||
# download this python library from
|
|
||||||
# https://github.com/erans/torcontactinfoparser
|
|
||||||
# sys.path.append('/home/....')
|
|
||||||
try:
|
|
||||||
from torcontactinfo import TorContactInfoParser
|
|
||||||
except:
|
|
||||||
TorContactInfoParser = None
|
|
||||||
|
|
||||||
class TrustorError(Exception): pass
|
|
||||||
|
|
||||||
# https://stackoverflow.com/questions/2532053/validate-a-hostname-string
|
|
||||||
# FIXME this check allows non-fqdn names
|
|
||||||
def is_valid_hostname(hostname):
|
|
||||||
if len(hostname) > 255:
|
|
||||||
return False
|
|
||||||
if hostname[-1] == ".":
|
|
||||||
hostname = hostname[:-1] # strip exactly one dot from the right, if present
|
|
||||||
allowed = re.compile("(?!-)[A-Z0-9-]{1,63}(?<!-)$", re.IGNORECASE)
|
|
||||||
return all(allowed.match(x) for x in hostname.split("."))
|
|
||||||
|
|
||||||
def read_local_trust_config(trust_config):
|
|
||||||
'''
|
|
||||||
reads a local configuration file containing trusted domains
|
|
||||||
and returns them in an array
|
|
||||||
'''
|
|
||||||
|
|
||||||
result = []
|
|
||||||
|
|
||||||
# for now we support max_depth = 0 only
|
|
||||||
# this PoC version has no support for recursion
|
|
||||||
# https://github.com/nusenu/tor-relay-operator-ids-trust-information#trust-information-consumers
|
|
||||||
supported_max_depths = ['0']
|
|
||||||
|
|
||||||
if (os.path.isfile(trust_config)):
|
|
||||||
f = open(trust_config)
|
|
||||||
for line in f:
|
|
||||||
line = line.strip()
|
|
||||||
if line[0] == '#':
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
domain, max_depth = line.split(':')
|
|
||||||
except:
|
|
||||||
LOG.error('invalid trust config line detected: %s aborting!' % line)
|
|
||||||
sys.exit(8)
|
|
||||||
|
|
||||||
if max_depth in supported_max_depths:
|
|
||||||
if is_valid_hostname(domain) and domain not in result:
|
|
||||||
result.append(domain)
|
|
||||||
else:
|
|
||||||
LOG.error('invalid duplicate domain in trust config file: %s: %s aborting!' % (trust_config, domain))
|
|
||||||
sys.exit(9)
|
|
||||||
else:
|
|
||||||
LOG.error('unsupported max_depth value (%s) used, aborting!' % line)
|
|
||||||
sys.exit(10)
|
|
||||||
|
|
||||||
return result
|
|
||||||
else:
|
|
||||||
LOG.error("trust config file %s missing, aborting!" % trust_config)
|
|
||||||
sys.exit(11)
|
|
||||||
|
|
||||||
|
|
||||||
def read_local_validation_cache(validation_cache_file, trusted_domains=[]):
|
|
||||||
'''
|
|
||||||
reads the local validation cache and returns all fingerprints in the cache
|
|
||||||
for trusted domains
|
|
||||||
|
|
||||||
format of each entry in the cache:
|
|
||||||
domain:fingerprint:prooftype:date
|
|
||||||
'''
|
|
||||||
|
|
||||||
result = []
|
|
||||||
if trusted_domains == []:
|
|
||||||
return result
|
|
||||||
if os.path.isfile(validation_cache_file):
|
|
||||||
with open(validation_cache_file, 'rt') as f:
|
|
||||||
for line in f:
|
|
||||||
line = line.strip()
|
|
||||||
if line[0] == '#':
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
domain, fingerprint, prooftype, dt = line.split(':')
|
|
||||||
except:
|
|
||||||
LOG.error('invalid trust cache entry detected: %s aborting!' % line)
|
|
||||||
sys.exit(12)
|
|
||||||
|
|
||||||
if domain in trusted_domains:
|
|
||||||
result.append(fingerprint)
|
|
||||||
else:
|
|
||||||
LOG.warn('ignoring cached entry for untrusted domain %s' % domain)
|
|
||||||
|
|
||||||
else:
|
|
||||||
LOG.info("Validation cache file not present. It will be created.")
|
|
||||||
return result
|
|
||||||
|
|
||||||
def get_controller(address='127.0.0.1', port=9151, password=''):
|
|
||||||
'''
|
|
||||||
connects to a local tor client via the tor ControlPort
|
|
||||||
and returns a controller that allows us to easily set specific tor
|
|
||||||
configuration options or read tor relay ContactInfo strings for validation
|
|
||||||
'''
|
|
||||||
|
|
||||||
try:
|
|
||||||
# controller = Controller.from_socket_file(path=torsocketpath)
|
|
||||||
controller = Controller.from_port(address=address, port=port)
|
|
||||||
controller.authenticate(password=password)
|
|
||||||
except Exception as e:
|
|
||||||
LOG.error(f"Failed to connect to the tor process, {e}")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if not controller.is_set('UseMicrodescriptors'):
|
|
||||||
LOG.error('"UseMicrodescriptors 0" is required in your torrc configuration. Exiting.')
|
|
||||||
sys.exit(2)
|
|
||||||
|
|
||||||
return controller
|
|
||||||
|
|
||||||
def find_validation_candidates(controller,
|
|
||||||
trusted_domains=[],
|
|
||||||
validation_cache=[],
|
|
||||||
CAfile='/etc/ssl/certs/ca-certificates.crt',
|
|
||||||
accept_all=False):
|
|
||||||
'''
|
|
||||||
connect to a tor client via controlport and return a dict of all
|
|
||||||
not yet validated fingerprints per trusted operators
|
|
||||||
format:
|
|
||||||
{ trusted_domain: { prooftype: [fingerprint, fingerprint, ...]} }
|
|
||||||
|
|
||||||
example content:
|
|
||||||
{ 'emeraldonion.org' : { 'uri-rsa': ['044600FD968728A6F220D5347AD897F421B757C0', '09DCA3360179C6C8A5A20DDDE1C54662965EF1BA']}}
|
|
||||||
'''
|
|
||||||
# https://github.com/nusenu/ContactInfo-Information-Sharing-Specification#proof
|
|
||||||
accepted_proof_types = ['uri-rsa', 'dns-rsa']
|
|
||||||
|
|
||||||
# https://github.com/nusenu/ContactInfo-Information-Sharing-Specification#ciissversion
|
|
||||||
accepted_ciissversions = ['2']
|
|
||||||
|
|
||||||
result = {}
|
|
||||||
|
|
||||||
try:
|
|
||||||
relays = controller.get_server_descriptors()
|
|
||||||
except:
|
|
||||||
LOG.error('Failed to get relay descriptors via tor\'s ControlPort. Exiting.')
|
|
||||||
sys.exit(3)
|
|
||||||
|
|
||||||
ci = TorContactInfoParser()
|
|
||||||
|
|
||||||
for relay in relays:
|
|
||||||
if relay.contact:
|
|
||||||
fingerprint = relay.fingerprint
|
|
||||||
# skip fingerprints we have already successfully validated in the past
|
|
||||||
# a future version would check the cache age as well
|
|
||||||
if fingerprint in validation_cache:
|
|
||||||
continue
|
|
||||||
contactstring = relay.contact.decode('utf-8')
|
|
||||||
parsed_ci = ci.parse(contactstring)
|
|
||||||
if len(parsed_ci) > 0:
|
|
||||||
if 'ciissversion' in parsed_ci and 'proof' in parsed_ci and 'url' in parsed_ci:
|
|
||||||
prooftype = parsed_ci['proof']
|
|
||||||
ciurl = parsed_ci['url']
|
|
||||||
if parsed_ci['ciissversion'] in accepted_ciissversions and prooftype in accepted_proof_types:
|
|
||||||
if ciurl.startswith('http://') or ciurl.startswith('https://'):
|
|
||||||
try:
|
|
||||||
domain = urlparse(ciurl).netloc
|
|
||||||
except:
|
|
||||||
LOG.warning('failed to parse domain %s' % ciurl)
|
|
||||||
domain = 'error'
|
|
||||||
continue
|
|
||||||
else:
|
|
||||||
domain = ciurl
|
|
||||||
if not is_valid_hostname(domain):
|
|
||||||
domain = 'error'
|
|
||||||
continue
|
|
||||||
# we can ignore relays that do not claim to be operated by a trusted operator
|
|
||||||
# if we do not accept all
|
|
||||||
if domain not in trusted_domains and not accept_all:
|
|
||||||
continue
|
|
||||||
if domain in result.keys():
|
|
||||||
if prooftype in result[domain].keys():
|
|
||||||
result[domain][prooftype].append(fingerprint)
|
|
||||||
else:
|
|
||||||
result[domain] = {prooftype: [fingerprint]}
|
|
||||||
# mixed proof types are not allowd as per spec but we are not strict here
|
|
||||||
LOG.warning('%s is using mixed prooftypes %s' % (domain, prooftype))
|
|
||||||
else:
|
|
||||||
result[domain] = {prooftype: [fingerprint]}
|
|
||||||
return result
|
|
||||||
|
|
||||||
def oDownloadUrlRequests(uri, sCAfile, timeout=30, host='127.0.0.1', port=9050, content_type='text/plain', session=None):
|
|
||||||
import requests
|
|
||||||
# socks proxy used for outbound web requests (for validation of proofs)
|
|
||||||
proxy = {'https': "socks5h://{host}:{port}"}
|
|
||||||
# we use this UA string when connecting to webservers to fetch rsa-fingerprint.txt proof files
|
|
||||||
# https://nusenu.github.io/ContactInfo-Information-Sharing-Specification/#uri-rsa
|
|
||||||
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'}
|
|
||||||
|
|
||||||
LOG.debug("fetching %s...." % uri)
|
|
||||||
try:
|
|
||||||
# grr. fix urllib3
|
|
||||||
# urllib3.connection WARNING Certificate did not match expected hostname:
|
|
||||||
head = requests.head(uri, timeout=timeout, proxies=proxy, headers=headers)
|
|
||||||
except Exception as e:
|
|
||||||
LOG.exception(f"{e}")
|
|
||||||
raise TrustorError(f"HTTP HEAD request failed for {uri} {e}")
|
|
||||||
|
|
||||||
if head.status_code >= 300:
|
|
||||||
raise TrustorError(f"HTTP Errorcode {head.status_code}")
|
|
||||||
if not head.headers['Content-Type'].startswith('text/plain'):
|
|
||||||
raise TrustorError(f"HTTP Content-Type != text/plain")
|
|
||||||
if not os.path.exists(sCAfile):
|
|
||||||
raise TrustorError(f"File not found CAfile {sCAfile}")
|
|
||||||
|
|
||||||
if session is None: session = requests.sessions.Session()
|
|
||||||
try:
|
|
||||||
oReqResp = session.request(method="get", url=uri,
|
|
||||||
proxies=proxy,
|
|
||||||
timeout=timeout,
|
|
||||||
headers=headers,
|
|
||||||
allow_redirects=False,
|
|
||||||
verify=True
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
LOG.warn("HTTP GET request failed for %s" % uri)
|
|
||||||
raise
|
|
||||||
if oReqResp.status_code != 200:
|
|
||||||
raise TrustorError(f"HTTP Errorcode {head.status_code}")
|
|
||||||
if not oReqResp.headers['Content-Type'].startswith('text/plain'):
|
|
||||||
raise TrustorError(f"HTTP Content-Type != text/plain")
|
|
||||||
|
|
||||||
# check for redirects (not allowed as per spec)
|
|
||||||
if oReqResp.url != uri:
|
|
||||||
LOG.error(f'Redirect detected {uri} vs %s (final)' % (oReqResp.url))
|
|
||||||
raise TrustorError(f'Redirect detected {uri} vs %s (final)' % (oReqResp.url))
|
|
||||||
return oReqResp
|
|
||||||
|
|
||||||
# There's no point in using asyncio because of duplicate urls in the tasks
|
|
||||||
async def oDownloadUrlHttpx(uri, sCAfile, timeout=30, host='127.0.0.1', port=9050, content_type='text/plain'):
|
|
||||||
import httpcore
|
|
||||||
import asyncio
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
# socks proxy used for outbound web requests (for validation of proofs)
|
|
||||||
if host and port:
|
|
||||||
proxy = "socks5://{host}:{port}"
|
|
||||||
else:
|
|
||||||
proxy = ''
|
|
||||||
# we use this UA string when connecting to webservers to fetch rsa-fingerprint.txt proof files
|
|
||||||
# https://nusenu.github.io/ContactInfo-Information-Sharing-Specification/#uri-rsa
|
|
||||||
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'}
|
|
||||||
|
|
||||||
LOG.debug("fetching %s...." % uri)
|
|
||||||
async with httpx.AsyncClient(proxies=proxy) as client:
|
|
||||||
try:
|
|
||||||
# https://www.python-httpx.org/advanced/
|
|
||||||
head = await client.head(uri, timeout=timeout, headers=headers)
|
|
||||||
except Exception as e:
|
|
||||||
LOG.exception(f"{e}")
|
|
||||||
raise TrustorError(f"HTTP HEAD request failed for {uri} {e}")
|
|
||||||
|
|
||||||
if head.status_code >= 300:
|
|
||||||
raise TrustorError(f"HTTP Errorcode {head.status_code}")
|
|
||||||
if content_type and not head.headers['Content-Type'].startswith(content_type):
|
|
||||||
raise TrustorError(f"HTTP Content-Type != {content_type}" )
|
|
||||||
if not os.path.exists(sCAfile):
|
|
||||||
raise TrustorError(f"File not found CAfile {sCAfile}")
|
|
||||||
|
|
||||||
try:
|
|
||||||
oReqResp = await client.get(url=uri,
|
|
||||||
timeout=timeout,
|
|
||||||
headers=headers,
|
|
||||||
max_redirects=0,
|
|
||||||
verify=sCAfile,
|
|
||||||
)
|
|
||||||
except (asyncio.exceptions.CancelledError,
|
|
||||||
httpcore.PoolTimeout,
|
|
||||||
Exception,) as e:
|
|
||||||
LOG.warn(f"HTTP GET request failed for %s {e}" % uri)
|
|
||||||
raise
|
|
||||||
if oReqResp.status_code != 200:
|
|
||||||
LOG.warn(f"HTTP Errorcode {head.status_code}")
|
|
||||||
raise TrustorError(f"HTTP Errorcode {head.status_code}")
|
|
||||||
if not oReqResp.headers['Content-Type'].startswith('text/plain'):
|
|
||||||
LOG.warn(f"HTTP Content-Type != text/plain")
|
|
||||||
raise TrustorError(f"HTTP Content-Type != text/plain")
|
|
||||||
|
|
||||||
# check for redirects (not allowed as per spec)
|
|
||||||
if oReqResp.url != uri:
|
|
||||||
LOG.error(f'Redirect detected {uri} vs %s (final)' % (oReqResp.url))
|
|
||||||
raise TrustorError(f'Redirect detected {uri} vs %s (final)' % (oReqResp.url))
|
|
||||||
return oReqResp
|
|
||||||
|
|
||||||
|
|
||||||
def ballow_subdomain_matching(hostname, dnsnames):
|
|
||||||
for elt in dnsnames:
|
|
||||||
if len(hostname.split('.')) > len(elt.split('.')) and hostname.endswith(elt):
|
|
||||||
# parent
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
from urllib3.util.ssl_match_hostname import (CertificateError, _dnsname_match,
|
|
||||||
_ipaddress_match)
|
|
||||||
|
|
||||||
def my_match_hostname(cert, hostname):
|
|
||||||
"""Verify that *cert* (in decoded format as returned by
|
|
||||||
SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 and RFC 6125
|
|
||||||
rules are followed, but IP addresses are not accepted for *hostname*.
|
|
||||||
|
|
||||||
CertificateError is raised on failure. On success, the function
|
|
||||||
returns nothing.
|
|
||||||
"""
|
|
||||||
if not cert:
|
|
||||||
raise ValueError(
|
|
||||||
"empty or no certificate, match_hostname needs a "
|
|
||||||
"SSL socket or SSL context with either "
|
|
||||||
"CERT_OPTIONAL or CERT_REQUIRED"
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
# Divergence from upstream: ipaddress can't handle byte str
|
|
||||||
host_ip = ipaddress.ip_address(hostname)
|
|
||||||
except (UnicodeError, ValueError):
|
|
||||||
# ValueError: Not an IP address (common case)
|
|
||||||
# UnicodeError: Divergence from upstream: Have to deal with ipaddress not taking
|
|
||||||
# byte strings. addresses should be all ascii, so we consider it not
|
|
||||||
# an ipaddress in this case
|
|
||||||
host_ip = None
|
|
||||||
except AttributeError:
|
|
||||||
# Divergence from upstream: Make ipaddress library optional
|
|
||||||
if ipaddress is None:
|
|
||||||
host_ip = None
|
|
||||||
else: # Defensive
|
|
||||||
raise
|
|
||||||
dnsnames = []
|
|
||||||
san = cert.get("subjectAltName", ())
|
|
||||||
for key, value in san:
|
|
||||||
if key == "DNS":
|
|
||||||
if host_ip is None and _dnsname_match(value, hostname):
|
|
||||||
return
|
|
||||||
dnsnames.append(value)
|
|
||||||
elif key == "IP Address":
|
|
||||||
if host_ip is not None and _ipaddress_match(value, host_ip):
|
|
||||||
return
|
|
||||||
dnsnames.append(value)
|
|
||||||
if not dnsnames:
|
|
||||||
# The subject is only checked when there is no dNSName entry
|
|
||||||
# in subjectAltName
|
|
||||||
for sub in cert.get("subject", ()):
|
|
||||||
for key, value in sub:
|
|
||||||
# XXX according to RFC 2818, the most specific Common Name
|
|
||||||
# must be used.
|
|
||||||
if key == "commonName":
|
|
||||||
if _dnsname_match(value, hostname):
|
|
||||||
return
|
|
||||||
dnsnames.append(value)
|
|
||||||
if len(dnsnames) > 1:
|
|
||||||
# soften this to allow subdomain matching
|
|
||||||
if ballow_subdomain_matching(hostname, dnsnames):
|
|
||||||
LOG.warn(f"Allowing {hostname} in {dnsnames}")
|
|
||||||
return
|
|
||||||
raise CertificateError(
|
|
||||||
"hostname %r "
|
|
||||||
"doesn't match any of %s" % (hostname, ", ".join(map(repr, dnsnames)))
|
|
||||||
)
|
|
||||||
elif len(dnsnames) == 1:
|
|
||||||
raise CertificateError("hostname %r doesn't match %r" % (hostname, dnsnames[0]))
|
|
||||||
else:
|
|
||||||
raise CertificateError(
|
|
||||||
"no appropriate commonName or subjectAltName fields were found"
|
|
||||||
)
|
|
||||||
urllib3.util.ssl_match_hostname.match_hostname = my_match_hostname
|
|
||||||
from urllib3.util.ssl_ import is_ipaddress
|
|
||||||
|
|
||||||
|
|
||||||
def _my_match_hostname(cert, asserted_hostname):
|
|
||||||
# Our upstream implementation of ssl.match_hostname()
|
|
||||||
# only applies this normalization to IP addresses so it doesn't
|
|
||||||
# match DNS SANs so we do the same thing!
|
|
||||||
stripped_hostname = asserted_hostname.strip("u[]")
|
|
||||||
if is_ipaddress(stripped_hostname):
|
|
||||||
asserted_hostname = stripped_hostname
|
|
||||||
try:
|
|
||||||
my_match_hostname(cert, asserted_hostname)
|
|
||||||
except CertificateError as e:
|
|
||||||
LOG.warning(
|
|
||||||
"Certificate did not match hostname: %s. Certificate: %s",
|
|
||||||
asserted_hostname,
|
|
||||||
cert,
|
|
||||||
)
|
|
||||||
# Add cert to exception and reraise so client code can inspect
|
|
||||||
# the cert when catching the exception, if they want to
|
|
||||||
e._peer_cert = cert
|
|
||||||
raise
|
|
||||||
urllib3.connection._match_hostname = _my_match_hostname
|
|
||||||
|
|
||||||
from urllib3.contrib.socks import SOCKSProxyManager
|
|
||||||
|
|
||||||
# from urllib3 import Retry
|
|
||||||
def oDownloadUrlUrllib3Socks(uri,
|
|
||||||
sCAfile,
|
|
||||||
timeout=30,
|
|
||||||
host='127.0.0.1',
|
|
||||||
port=9050,
|
|
||||||
session=None,
|
|
||||||
content_type='text/plain'):
|
|
||||||
"""Theres no need to use requests here and it
|
|
||||||
adds too many layers on the SSL to be able to get at things
|
|
||||||
"""
|
|
||||||
domain = urlparse(uri).netloc
|
|
||||||
# socks proxy used for outbound web requests (for validation of proofs)
|
|
||||||
proxy = SOCKSProxyManager(f'socks5h://{host}:{port}/',
|
|
||||||
num_pools=1,
|
|
||||||
timeout=timeout,
|
|
||||||
cert_reqs='CERT_REQUIRED',
|
|
||||||
assert_hostname=domain,
|
|
||||||
ca_certs=sCAfile)
|
|
||||||
|
|
||||||
# we use this UA string when connecting to webservers to fetch rsa-fingerprint.txt proof files
|
|
||||||
# https://nusenu.github.io/ContactInfo-Information-Sharing-Specification/#uri-rsa
|
|
||||||
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'}
|
|
||||||
|
|
||||||
LOG.debug("fetching %s...." % uri)
|
|
||||||
try:
|
|
||||||
# grr. fix urllib3
|
|
||||||
# Errors will be wrapped in :class:`~urllib3.exceptions.MaxRetryError` unless
|
|
||||||
# retries are disabled, in which case the causing exception will be raised.
|
|
||||||
head = proxy.request('HEAD', uri,
|
|
||||||
headers=headers,
|
|
||||||
redirect=False,
|
|
||||||
retries=False)
|
|
||||||
except Exception as e:
|
|
||||||
LOG.error(f"HTTP HEAD request failed for {uri} {e}")
|
|
||||||
raise
|
|
||||||
|
|
||||||
if head.status >= 300:
|
|
||||||
raise TrustorError(f"HTTP Errorcode {head.status}")
|
|
||||||
if content_type and not head.headers['Content-Type'].startswith(content_type):
|
|
||||||
raise TrustorError(f"HTTP Content-Type != {content_type}")
|
|
||||||
if not os.path.exists(sCAfile):
|
|
||||||
raise TrustorError(f"File not found CAfile {sCAfile}")
|
|
||||||
|
|
||||||
try:
|
|
||||||
oReqResp = proxy.request("GET", uri,
|
|
||||||
headers=headers,
|
|
||||||
redirect=False,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
LOG.warn(f"HTTP GET request failed for {uri} {e}")
|
|
||||||
raise
|
|
||||||
if oReqResp.status != 200:
|
|
||||||
raise TrustorError(f"HTTP Errorcode {head.status}")
|
|
||||||
if content_type and not oReqResp.headers['Content-Type'].startswith(content_type):
|
|
||||||
raise TrustorError(f"HTTP Content-Type != {content_type}")
|
|
||||||
|
|
||||||
# check for redirects (not allowed as per spec)
|
|
||||||
if oReqResp.geturl() != uri:
|
|
||||||
LOG.error(f'Redirect detected %s vs %s (final)' % (uri, oReqResp.geturl()))
|
|
||||||
raise TrustorError(f'Redirect detected %s vs %s (final)' % (uri, oReqResp.geturl()))
|
|
||||||
oReqResp.decode_content = True
|
|
||||||
|
|
||||||
return oReqResp
|
|
||||||
|
|
||||||
import urllib3.connectionpool
|
|
||||||
from urllib3.connection import HTTPSConnection
|
|
||||||
|
|
||||||
urllib3.connectionpool.VerifiedHTTPSConnection = HTTPSConnection
|
|
||||||
|
|
||||||
def lDownloadUrlFps(domain, sCAfile, timeout=30, host='127.0.0.1', port=9050):
|
|
||||||
uri = f"https://{domain}/.well-known/tor-relay/rsa-fingerprint.txt"
|
|
||||||
o = oDownloadUrlRequests(uri, sCAfile, timeout=timeout, host=host, port=port)
|
|
||||||
well_known_content = o.text.upper().strip().split('\n')
|
|
||||||
well_known_content = [i for i in well_known_content if i and len(i) == 40]
|
|
||||||
return well_known_content
|
|
||||||
|
|
||||||
def validate_proofs(candidates, validation_cache_file, timeout=20, host='127.0.0.1', port=9050):
|
|
||||||
'''
|
|
||||||
This function takes the return value of find_validation_candidates()
|
|
||||||
and validated them according to their proof type (uri-rsa, dns-rsa)
|
|
||||||
and writes properly validated relay fingerprints to the local validation cache
|
|
||||||
'''
|
|
||||||
dt_utc = datetime.datetime.now(datetime.timezone.utc).date()
|
|
||||||
|
|
||||||
f = open(validation_cache_file, mode='a')
|
|
||||||
count = 0
|
|
||||||
|
|
||||||
for domain in candidates.keys():
|
|
||||||
for prooftype in candidates[domain].keys():
|
|
||||||
if prooftype == 'uri-rsa':
|
|
||||||
well_known_content = lDownloadUrlFps(domain, timeout=timeout, host=host, port=port)
|
|
||||||
for fingerprint in candidates[domain][prooftype]:
|
|
||||||
if fingerprint in well_known_content:
|
|
||||||
# write cache entry
|
|
||||||
count += 1
|
|
||||||
f.write('%s:%s:%s:%s\n' % (domain, fingerprint, prooftype, dt_utc))
|
|
||||||
else:
|
|
||||||
LOG.error('%s:%s:%s' % (fingerprint, domain, prooftype))
|
|
||||||
elif prooftype == 'dns-rsa' and ub_ctx:
|
|
||||||
for fingerprint in candidates[domain][prooftype]:
|
|
||||||
fp_domain = fingerprint + '.' + domain
|
|
||||||
if idns_validate(fp_domain,
|
|
||||||
libunbound_resolv_file='resolv.conf',
|
|
||||||
dnssec_DS_file='dnssec-root-trust',
|
|
||||||
) == 0:
|
|
||||||
count += 1
|
|
||||||
f.write('%s:%s:%s:%s\n' % (domain, fingerprint, prooftype, dt_utc))
|
|
||||||
else:
|
|
||||||
LOG.error('%s:%s:%s' % (fingerprint, domain, prooftype))
|
|
||||||
f.close()
|
|
||||||
LOG.info('successfully validated %s new (not yet validated before) relays' % count)
|
|
||||||
|
|
||||||
def idns_validate(domain,
|
|
||||||
libunbound_resolv_file='resolv.conf',
|
|
||||||
dnssec_DS_file='dnssec-root-trust',
|
|
||||||
):
|
|
||||||
'''
|
|
||||||
performs DNS TXT lookups and verifies the reply
|
|
||||||
- is DNSSEC valid and
|
|
||||||
- contains only a single TXT record
|
|
||||||
- the DNS record contains a hardcoded string as per specification
|
|
||||||
https://nusenu.github.io/ContactInfo-Information-Sharing-Specification/#dns-rsa
|
|
||||||
'''
|
|
||||||
if not ub_ctx: return -1
|
|
||||||
|
|
||||||
# this is not the system wide /etc/resolv.conf
|
|
||||||
# use dnscrypt-proxy to encrypt your DNS and route it via tor's SOCKSPort
|
|
||||||
|
|
||||||
ctx = ub_ctx()
|
|
||||||
if (os.path.isfile(libunbound_resolv_file)):
|
|
||||||
ctx.resolvconf(libunbound_resolv_file)
|
|
||||||
else:
|
|
||||||
LOG.error('libunbound resolv config file: "%s" is missing, aborting!' % libunbound_resolv_file)
|
|
||||||
return 5
|
|
||||||
if (os.path.isfile(dnssec_DS_file)):
|
|
||||||
ctx.add_ta_file(dnssec_DS_file)
|
|
||||||
else:
|
|
||||||
LOG.error('DNSSEC trust anchor file "%s" is missing, aborting!' % dnssec_DS_file)
|
|
||||||
return 6
|
|
||||||
|
|
||||||
status, result = ctx.resolve(domain, RR_TYPE_TXT, RR_CLASS_IN)
|
|
||||||
if status == 0 and result.havedata:
|
|
||||||
if len(result.rawdata) == 1 and result.secure:
|
|
||||||
# ignore the first byte, it is the TXT length
|
|
||||||
if result.data.as_raw_data()[0][1:] == b'we-run-this-tor-relay':
|
|
||||||
return 0
|
|
||||||
return 1
|
|
||||||
|
|
||||||
def configure_tor(controller, trusted_fingerprints, exitonly=True):
|
|
||||||
'''
|
|
||||||
takes the list of trusted fingerprints and configures a tor client
|
|
||||||
to only use trusted relays in a certain position
|
|
||||||
for now we only set exits.
|
|
||||||
we refuse to set the configuration if there are less then 40 trusted relays
|
|
||||||
'''
|
|
||||||
|
|
||||||
relay_count = len(trusted_fingerprints)
|
|
||||||
|
|
||||||
if relay_count < 41:
|
|
||||||
LOG.error('Too few trusted relays (%s), aborting!' % relay_count)
|
|
||||||
sys.exit(15)
|
|
||||||
|
|
||||||
try:
|
|
||||||
controller.set_conf('ExitNodes', trusted_fingerprints)
|
|
||||||
LOG.error('limited exits to %s relays' % relay_count)
|
|
||||||
except Exception as e: # noqa
|
|
||||||
LOG.exception('Failed to set ExitNodes tor config to trusted relays')
|
|
||||||
sys.exit(20)
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
CAfile = '/etc/ssl/certs/ca-certificates.crt'
|
|
||||||
trust_config = 'trust_config'
|
|
||||||
assert os.path.exists(trust_config)
|
|
||||||
trusted_domains = read_local_trust_config(trust_config)
|
|
||||||
|
|
||||||
validation_cache_file = 'validation_cache'
|
|
||||||
trusted_fingerprints = read_local_validation_cache(validation_cache_file,
|
|
||||||
trusted_domains=trusted_domains)
|
|
||||||
# tor ControlPort password
|
|
||||||
controller_password = ''
|
|
||||||
# tor ControlPort IP
|
|
||||||
controller_address = '127.0.0.1'
|
|
||||||
timeout = 20
|
|
||||||
port = 9050
|
|
||||||
controller = get_controller(address=controller_address, password=controller_password)
|
|
||||||
|
|
||||||
r = find_validation_candidates(controller,
|
|
||||||
validation_cache=trusted_fingerprints,
|
|
||||||
trusted_domains=trusted_domains,
|
|
||||||
CAfile=CAfile)
|
|
||||||
validate_proofs(r, validation_cache_file,
|
|
||||||
timeout=timeout,
|
|
||||||
host=controller_address,
|
|
||||||
port=port)
|
|
||||||
|
|
||||||
# refresh list with newly validated fingerprints
|
|
||||||
trusted_fingerprints = read_local_validation_cache(validation_cache_file,
|
|
||||||
trusted_domains=trusted_domains)
|
|
||||||
configure_tor(controller, trusted_fingerprints)
|
|
||||||
Loading…
Reference in New Issue
Block a user