emdee/tox_profile
2
3
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: emdee:525446cc4a31ca59a0b1524ca5b227be04174dcb
Branches Tags
emdee:main
..
compare: emdee:1e7a1cec2d247b1a12d02402ee0575dbc89eedca
Branches Tags
emdee:main

No commits in common. "525446cc4a31ca59a0b1524ca5b227be04174dcb" and "1e7a1cec2d247b1a12d02402ee0575dbc89eedca" have entirely different histories.
525446cc4a ... 1e7a1cec2d

3 changed files with 133 additions and 389 deletions
Show Stats Expand all files Collapse all files

63
README.md
View File

@ -3,7 +3,7 @@
Read and manipulate tox profile files. It started as a simple script from
<https://stackoverflow.com/questions/30901873/what-format-are-tox-files-stored-in>
For the moment tox_savefile.py just reads a Tox profile and
For the moment logging_tox_savefile.py just reads a Tox profile and
prints to stdout various things that it finds. Then it writes what it
found in YAML to stderr. Later it can be extended to print out JSON
or YAML, and then extended to accept JSON or YAML to write a profile.
@ -11,8 +11,8 @@ or YAML, and then extended to accept JSON or YAML to write a profile.
## Usage
Reads a tox profile and prints out information on what's in there to stderr.
Call it with one argument, the filename of the profile for the decrypt, edit
or info commands, or the filename of the nodes file for the nodes command.
Call it with one argument, the filename of the profile for the decrypt or info
commands, or the filename of the nodes file for the nodes command.
3 commands are supported:
1. ```--command decrypt``` decrypts the profile and writes to the result
@ -22,7 +22,7 @@ to stdout
a profile
```
usage: tox_savefile.py [-h] [--output OUTPUT]
usage: logging_tox_savefile.py [-h] [--output OUTPUT]
[--command {info,decrypt,nodes}]
[--indent INDENT]
[--info {info,repr,yaml,json,pprint,nmap_udp,nmap_tcp}]
@ -52,60 +52,28 @@ Optional arguments:
```info``` will output the profile on stdout, or to a file with ```--output```
Choose one of ```{info,repr,yaml,json,pprint,save}```
Choose one of ```{info,repr,yaml,json,pprint}```
for the format for info command.
Choose one of ```{nmap_udp,nmap_tcp}```
to run tests using ```nmap``` for the ```DHT``` and ```TCP_RELAY```
sections of the profile. Reguires ```nmap``` and uses ```sudo```.
#### Saving a copy
The code now can generate a saved copy of the profile as it parses the profile.
Use the command ```--command info --info save``` with ```--output```
and a filename, to process the file with info to stderr, and it will
save an copy of the file to the ```--output``` (unencrypted).
It may be shorter than the original profile by up to 512 bytes, as the
original toxic profile is padded at the end with nulls (or maybe in the
decryption).
### --command nodes
Takes a DHTnodes.json file as an argument.
Choose one of ```{select_tcp,select_udp,select_version}```
for ```--nodes``` to select TCP nodes, UDP nodes,
or nodes with the latest version. Requires ```jq```.
for ```--nodes``` to select TCP nodes, UDP nodes or nodes with the latest version.
Requires ```jq```.
Choose one of ```{nmap_tcp,nmap_udp}``` to run tests using ```nmap```
for the ```status_tcp==True``` and ```status_udp==True``` nodes.
Reguires ```nmap``` and uses ```sudo```.
Choose one of ```{nmap_tcp,nmap_udp}```
to run tests using ```nmap``` for the ```tcp``` and ```udp```
nodes. Reguires ```nmap``` and uses ```sudo```.
### --command decrypt
Decrypt a profile.
### --command edit
The code now can generate an edited copy of the profile.
Use the command ```--command edit --edit section,key,val``` with
```--output``` and a filename, to process the file with info to stderr,
and it will save an copy of the edited file to the
```--output``` file (unencrypted). There's not much editing yet; give
```--command edit --edit help``` to get a list of what Available Sections,
and Supported Quads (section,num,key,type) that can be edited.
Currently it is:
```
NAME,.,Nick_name,str
STATUSMESSAGE,.,Status_message,str
STATUS,.,Online_status,int
```
The ```num``` field is to accomodate sections that have lists:
* ```.``` is a placeholder for sections that don't have lists.
* ```<int>``` is for the nth element of the list, zero-based.
* ```*``` is for all elements of the list.
## Requirements
If you want to read encrypted profiles, you need to download
@ -125,22 +93,11 @@ If you want to write in YAML, you need Python yaml:
If you have coloredlogs installed it will make use of it:
<https://pypi.org/project/coloredlogs/>
For the ```select``` and ```nmap``` commands, the ```jq``` utility is
required. It's available in most distros, or <https://stedolan.github.io/jq/>
For the ```nmap``` commands, the ```nmap``` utility is
required. It's available in most distros, or <https://nmap.org/>
## Future Directions
This has not been tested on Windwoes, but is should be simple to fix.
Because it's written in Python it is easy to extend to, for example,
rekeying a profile when copying a profile to a new device:
<https://git.plastiras.org/emdee/tox_profile/wiki/MultiDevice-Announcements-POC>
## Specification
There is a copy of the Tox [spec](https://toktok.ltd/spec.html)
in the repo - it is missing any description of the groups section.

303
tox_savefile.py → logging_tox_savefile.py
View File

@ -20,7 +20,7 @@ commands, or the filename of the nodes file for the nodes command.
"""
--output Destination for info/decrypt - defaults to stdout
--info default='info',
choices=['info', 'save', 'repr', 'yaml','json', 'pprint']
choices=['info', 'repr', 'yaml','json', 'pprint']
with --info=info prints info about the profile to stderr
nmap_udp - test DHT nodes with nmap
nmap_tcp - test TCP_RELAY nodes with nmap
@ -76,7 +76,6 @@ except ImportError as e:
try:
# https://git.plastiras.org/emdee/toxygen_wrapper
from wrapper.toxencryptsave import ToxEncryptSave
from wrapper_tests.support_http import download_url
except ImportError as e:
print(f"Import Error {e}")
print("Download toxygen_wrapper to deal with encrypted tox files, from:")
@ -86,31 +85,24 @@ except ImportError as e:
print("and libtoxencryptsave.so into wrapper/../libs/")
print("Link all 3 from libtoxcore.so if you have only libtoxcore.so")
ToxEncryptSave = None
try:
from wrapper_tests.support_http import download_url
except:
try:
from support_http import download_url
except ImportError as e:
print(f"Import Error {e}")
print("Download toxygen_wrapper to deal with encrypted tox files, from:")
print("https://git.plastiras.org/emdee/toxygen_wrapper")
download_url = None
LOG = logging.getLogger('TSF')
# Fix for Windows
sDIR = os.environ.get('TMPDIR', '/tmp')
sTOX_VERSION = "1000002018"
bHAVE_NMAP = shutil.which('nmap')
sDIR = os.environ.get('TMPDIR', '/tmp')
# nodes
sTOX_VERSION = "1000002018"
bHAVE_JQ = shutil.which('jq')
bMARK = b'\x00\x00\x00\x00\x1f\x1b\xed\x15'
bDEBUG = 'DEBUG' in os.environ and os.environ['DEBUG'] != 0
def trace(s): LOG.log(LOG.level, '+ ' +s)
LOG.trace = trace
global lOUT, bOUT, aOUT, sENC
lOUT = []
aOUT = {}
bOUT = b''
sENC = 'utf-8'
# grep '#''#' logging_tox_savefile.py|sed -e 's/.* //'
sEDIT_HELP = """
NAME,.,Nick_name,str
STATUSMESSAGE,.,Status_message,str
STATUS,.,Online_status,int
"""
#messenger.c
MESSENGER_STATE_TYPE_NOSPAMKEYS = 1
@ -207,35 +199,30 @@ Length Contents
o = delta+1+32+1024+1+2+128; l = 2
nsize = struct.unpack_from(">H", result, o)[0]
o = delta+1+32+1024+1+2; l = 128
name = str(result[o:o+nsize], sENC)
name = str(result[o:o+nsize], 'utf-8')
o = delta+1+32+1024+1+2+128+2+1007; l = 2
msize = struct.unpack_from(">H", result, o)[0]
o = delta+1+32+1024+1+2+128+2; l = 1007
mame = str(result[o:o+msize], sENC)
mame = str(result[o:o+msize], 'utf-8')
LOG.info(f"Friend #{i} {dStatus[status]} {name} {pk}")
lIN += [{"Status": dStatus[status],
"Name": name,
"Pk": pk}]
return lIN
def lProcessGroups(state, index, length, result, label="GROUPS"):
"""
No GROUPS description in spec.html
"""
global sENC
def lProcessGroups(state, index, length, result):
lIN = []
i = 0
if not msgpack:
LOG.warn(f"process_chunk Groups = NO msgpack bytes={length}")
LOG.debug(f"TODO process_chunk Groups = no msgpack bytes={length}")
return []
try:
groups = msgpack.loads(result, raw=True)
LOG.info(f"{label} {len(groups)} groups")
LOG.debug(f"TODO process_chunk Groups len={len(groups)}")
for group in groups:
assert len(group) == 7, group
i += 1
state_values, \
state_bin, \
topic_info, \
@ -254,8 +241,7 @@ def lProcessGroups(state, index, length, result, label="GROUPS"):
topic_lock, \
voice_state = state_values
LOG.info(f"lProcessGroups #{i} version={version}")
dBINS = {"Version": version,
"Privacy_state": privacy_state}
dBINS = {"Version": version}
lIN += [{"State_values": dBINS}]
assert len(state_bin) == 5, state_bin
@ -269,14 +255,14 @@ def lProcessGroups(state, index, length, result, label="GROUPS"):
lIN += [{"State_bin": dBINS}]
assert len(topic_info) == 6, topic_info
topic_info_topic = str(topic_info[3], sENC)
topic_info_topic = str(topic_info[3], 'utf-8')
LOG.info(f"lProcessGroups #{i} topic_info_topic={topic_info_topic}")
dBINS = {"topic_info_topic": topic_info_topic}
lIN += [{"Topic_info": dBINS}]
assert len(mod_list) == 2, mod_list
num_moderators = mod_list[0]
LOG.info(f"lProcessGroups #{i} num moderators={mod_list[0]}")
LOG.debug(f"lProcessGroups #{i} num moderators={mod_list[0]}")
#define CRYPTO_SIGN_PUBLIC_KEY_SIZE 32
mods = mod_list[1]
assert len(mods) % 32 == 0, len(mods)
@ -286,7 +272,7 @@ def lProcessGroups(state, index, length, result, label="GROUPS"):
mod = mods[j*32:j*32 + 32]
LOG.info(f"lProcessGroups group#{i} mod#{j} sig_pk={bin_to_hex(mod)}")
lMODS += [{"Sig_pk": bin_to_hex(mod)}]
lIN += [{"Moderators": lMODS}]
if lMODS: lIN += [{"Moderators": lMODS}]
assert len(keys) == 4, keys
LOG.debug(f"lProcessGroups #{i} {repr(list(map(len, keys)))}")
@ -308,7 +294,7 @@ def lProcessGroups(state, index, length, result, label="GROUPS"):
assert len(self_info) == 4, self_info
self_nick_len, self_role, self_status, self_nick = self_info
self_nick = str(self_nick, sENC)
self_nick = str(self_nick, 'utf-8')
LOG.info(f"lProcessGroups #{i} self_nick={self_nick}")
dBINS = {"Self_nick": self_nick}
lIN += [{"Self_info": dBINS}]
@ -341,10 +327,10 @@ The Node Info data structure contains a Transport Protocol, a Socket
while length > 0:
status = struct.unpack_from(">B", result, delta)[0]
if status >= 128:
prot = 'TCP'
ipv = 'TCP'
af = status - 128
else:
prot = 'UDP'
ipv = 'UDP'
af = status
if af == 2:
af = 'IPv4'
@ -357,10 +343,10 @@ The Node Info data structure contains a Transport Protocol, a Socket
total = 1 + alen + 2 + 32
port = int(struct.unpack_from(">H", result, delta+1+alen)[0])
pk = bin_to_hex(result[delta+1+alen+2:delta+1+alen+2+32], 32)
LOG.info(f"{label} #{relay} bytes={length} status={status} prot={prot} af={af} ip={ipaddr} port={port} pk={pk}")
LOG.info(f"{label} #{relay} bytes={length} status={status} ip={ipv} af={af} ip={ipaddr} port={port} pk={pk}")
lIN += [{"Bytes": length,
"Status": status,
"Prot": prot,
"Ip": ipv,
"Af": af,
"Ip": ipaddr,
"Port": port,
@ -389,7 +375,7 @@ def lProcessDHTnodes(state, index, length, result, label="DHTnode"):
while offset < slen: #loop over nodes
status = struct.unpack_from(">B", result, offset+8)[0]
assert status < 12
prot = 'UDP'
ipv = 'UDP'
if status == 2:
af = 'IPv4'
alen = 4
@ -403,40 +389,32 @@ def lProcessDHTnodes(state, index, length, result, label="DHTnode"):
pk = bin_to_hex(result[offset+8+1+alen+2:offset+8+1+alen+2+32], 32)
LOG.info(f"{label} #{relay} status={status} ipaddr={ipaddr} port={port} {pk}")
lIN += [{
"Status": status,
"Prot": prot,
"Af": af,
"Ip": ipaddr,
"Port": port,
"Pk": pk}]
lIN += [{"status": status,
"af": af,
"ipaddr": ipaddr,
"port": port,
"pk": pk}]
offset += subtotal
delta += total
length -= total
relay += 1
return lIN
def process_chunk(index, state, oArgs=None):
global lOUT, bOUT, aOUT
global sENC
def process_chunk(index, state):
global lOUT, bOUT, iTOTAL, aOUT
length = struct.unpack_from("<I", state, index)[0]
length = struct.unpack_from("<H", state, index)[0]
data_type = struct.unpack_from("<H", state, index + 4)[0]
check = struct.unpack_from("<H", state, index + 6)[0]
assert check == 0x01CE, check
new_index = index + length + 8
result = state[index + 8:index + 8 + length]
iTOTAL += length + 8
# plan on repacking as we read - this is just a starting point
# We'll add the results back to bOUT to see if we get what we started with.
# Then will will be able to selectively null sections or selectively edit.
bOUT += struct.pack("<H", length) + struct.pack("<H", data_type) + result
label = dSTATE_TYPE[data_type]
if oArgs.command == 'edit' and oArgs.edit:
section,num,key,val = oArgs.edit.split(',',3)
diff = index - len(bOUT)
if bDEBUG and diff > 0:
LOG.warn(f"PROCESS_CHUNK {label} index={index} bOUT={len(bOUT)} delta={diff} length={length}")
elif bDEBUG:
LOG.trace(f"PROCESS_CHUNK {label} index={index} bOUT={len(bOUT)} delta={diff} length={length}")
if data_type == MESSENGER_STATE_TYPE_NOSPAMKEYS:
nospam = bin_to_hex(result[0:4])
public_key = bin_to_hex(result[4:36])
@ -447,114 +425,71 @@ def process_chunk(index, state, oArgs=None):
aIN = {"Nospam": f"{nospam}",
"Public_key": f"{public_key}",
"Private_key": f"{private_key}"}
lOUT += [{label: aIN}]; aOUT.update({label: aIN})
lOUT += [{"NOSPAMKEYS": aIN}]; aOUT.update({"NOSPAMKEYS": aIN})
elif data_type == MESSENGER_STATE_TYPE_DHT:
LOG.debug(f"process_chunk {label} length={length}")
LOG.debug(f"process_chunk {dSTATE_TYPE[data_type]} length={length}")
lIN = lProcessDHTnodes(state, index, length, result)
lOUT += [{label: lIN}]; aOUT.update({label: lIN})
if lIN: lOUT += [{"DHT": lIN}]; aOUT.update({"DHT": lIN})
elif data_type == MESSENGER_STATE_TYPE_FRIENDS:
LOG.info(f"{label} {length // 2216} FRIENDS {length % 2216}")
LOG.debug(f"TODO process_chunk {length // 2216} FRIENDS {length} {length % 2216}")
lIN = lProcessFriends(state, index, length, result)
lOUT += [{label: lIN}]; aOUT.update({label: lIN})
if lIN: lOUT += [{"FRIENDS": lIN}]; aOUT.update({"FRIENDS": lIN})
elif data_type == MESSENGER_STATE_TYPE_NAME:
name = str(result, sENC)
LOG.info(f"{label} Nick_name = " +name)
aIN = {"Nick_name": name}
lOUT += [{label: aIN}]; aOUT.update({label: aIN})
if oArgs.command == 'edit' and section == label:
## NAME,.,Nick_name,str
if key == "Nick_name":
result = bytes(val, sENC)
length = len(result)
LOG.info(f"{label} {key} EDITED to {val}")
name = str(state[index + 8:index + 8 + length], 'utf-8')
LOG.info("Nick_name = " +name)
aIN = {"NAME": name}
lOUT += [{"Nick_name": aIN}]; aOUT.update({"Nick_name": aIN})
elif data_type == MESSENGER_STATE_TYPE_STATUSMESSAGE:
mess = str(result, sENC)
LOG.info(f"{label} StatusMessage = " +mess)
mess = str(state[index + 8:index + 8 + length], 'utf-8')
LOG.info(f"StatusMessage = " +mess)
aIN = {"Status_message": mess}
lOUT += [{label: aIN}]; aOUT.update({label: aIN})
if oArgs.command == 'edit' and section == label:
## STATUSMESSAGE,.,Status_message,str
if key == "Status_message":
result = bytes(val, sENC)
length = len(result)
LOG.info(f"{label} {key} EDITED to {val}")
lOUT += [{"STATUSMESSAGE": aIN}]; aOUT.update({"STATUSMESSAGE": aIN})
elif data_type == MESSENGER_STATE_TYPE_STATUS:
# 1 uint8_t status (0 = online, 1 = away, 2 = busy)
dStatus = {0: 'online', 1: 'away', 2: 'busy'}
status = struct.unpack_from(">b", state, index)[0]
status = dStatus[status]
LOG.info(f"{label} = " +status)
LOG.info(f"{dSTATE_TYPE[data_type]} = " +status)
aIN = {f"Online_status": status}
lOUT += [{"STATUS": aIN}]; aOUT.update({"STATUS": aIN})
if oArgs.command == 'edit' and section == label:
## STATUS,.,Online_status,int
if key == "Online_status":
result = struct.pack(">b", int(val))
length = len(result)
LOG.info(f"{label} {key} EDITED to {val}")
elif data_type == MESSENGER_STATE_TYPE_GROUPS:
if length > 0:
lIN = lProcessGroups(state, index, length, result, label)
else:
lIN = []
LOG.info(f"NO {label}")
lOUT += [{label: lIN}]; aOUT.update({label: lIN})
lIN = lProcessGroups(state, index, length, result)
if lIN: lOUT += [{"GROUPS": lIN}]; aOUT.update({"GROUPS": lIN})
elif data_type == MESSENGER_STATE_TYPE_TCP_RELAY:
if length > 0:
lIN = lProcessNodeInfo(state, index, length, result, "TCPnode")
else:
lIN = []
LOG.info(f"NO {label}")
lOUT += [{label: lIN}]; aOUT.update({label: lIN})
if lIN: lOUT += [{"TCP_RELAY": lIN}]; aOUT.update({"TCP_RELAY": lIN})
elif data_type == MESSENGER_STATE_TYPE_PATH_NODE:
#define NUM_SAVED_PATH_NODES 8
assert length % 8 == 0, length
LOG.debug(f"process_chunk {label} bytes={length}")
LOG.debug(f"TODO process_chunk {dSTATE_TYPE[data_type]} bytes={length}")
lIN = lProcessNodeInfo(state, index, length, result, "PATHnode")
lOUT += [{label: lIN}]; aOUT.update({label: lIN})
if lIN: lOUT += [{label: lIN}]; aOUT.update({label: lIN})
elif data_type == MESSENGER_STATE_TYPE_CONFERENCES:
lIN = []
if length > 0:
LOG.debug(f"TODO process_chunk {label} bytes={length}")
LOG.debug(f"TODO process_chunk {dSTATE_TYPE[data_type]} bytes={length}")
else:
LOG.info(f"NO {label}")
lOUT += [{label: []}]; aOUT.update({label: []})
LOG.info(f"NO {dSTATE_TYPE[data_type]}")
lOUT += [{"CONFERENCES": []}]; aOUT.update({"CONFERENCES": []})
elif data_type != MESSENGER_STATE_TYPE_END:
LOG.error("UNRECOGNIZED datatype={datatype}")
sys.exit(1)
LOG.warn("UNRECOGNIZED datatype={datatype}")
else:
LOG.info("END") # That's all folks...
# drop through
# We repack as we read: or edit as we parse; simply edit result and length.
# We'll add the results back to bOUT to see if we get what we started with.
# Then will will be able to selectively null sections or selectively edit.
assert length == len(result), length
bOUT += struct.pack("<I", length) + \
struct.pack("<H", data_type) + \
struct.pack("<H", check) + \
result
if data_type == MESSENGER_STATE_TYPE_END or \
index + 8 >= len(state):
diff = len(bSAVE) - len(bOUT)
if oArgs.command != 'edit' and diff > 0:
# if short repacking as we read - tox_profile is padded with nulls
LOG.warn(f"PROCESS_CHUNK bSAVE={len(bSAVE)} bOUT={len(bOUT)} delta={diff}")
return
process_chunk(new_index, state, oArgs)
# failsafe
if index + 8 >= len(state): return
process_chunk(new_index, state)
def bAreWeConnected():
# FixMe: Linux
@ -625,7 +560,7 @@ def vBashFileNmapUdp():
def vOsSystemNmapUdp(l, oArgs):
iErrs = 0
for elt in aOUT["DHT"]:
cmd = f"sudo nmap -Pn -n -sU -p U:{elt['Port']} {elt['Ip']}"
cmd = f"sudo nmap -Pn -n -sU -p U:{elt['port']} {elt['ipaddr']}"
iErrs += os.system(cmd +f" >> {oArgs.output} 2>&1")
if iErrs:
LOG.warn(f"{oArgs.info} {iErrs} ERRORs to {oArgs.output}")
@ -637,7 +572,7 @@ def vOsSystemNmapUdp(l, oArgs):
def vOsSystemNmapTcp(l, oArgs):
iErrs = 0
for elt in l:
cmd = f"sudo nmap -Pn -n -sT -p T:{elt['Port']} {elt['Ip']}"
cmd = f"sudo nmap -Pn -n -sT -p T:{elt['port']} {elt['ipaddr']}"
print(f"{oArgs.info} NO errors to {oArgs.output}")
iErrs += os.system(cmd +f" >> {oArgs.output} 2>&1")
if iErrs:
@ -677,14 +612,12 @@ def oMainArgparser(_=None):
parser.add_argument('--output', type=str, default='',
help='Destination for info/decrypt - defaults to stderr')
parser.add_argument('--command', type=str, default='info',
choices=['info', 'decrypt', 'nodes', 'edit'],
required=True,
choices=['info', 'decrypt', 'nodes'],
# required=True,
help='Action command - default: info')
parser.add_argument('--edit', type=str, default='',
help='comma seperated SECTION,key,value - unfinished')
parser.add_argument('--indent', type=int, default=2,
help='Indent for yaml/json/pprint')
choices=['info', 'save', 'repr', 'yaml','json', 'pprint']
choices=['info', 'repr', 'yaml','json', 'pprint']
if bHAVE_NMAP: choices += ['nmap_tcp', 'nmap_udp', 'nmap_onion']
parser.add_argument('--info', type=str, default='info',
choices=choices,
@ -700,26 +633,19 @@ def oMainArgparser(_=None):
help='Action for nodes command (requires jq)')
parser.add_argument('--download_nodes_url', type=str,
default='https://nodes.tox.chat/json')
parser.add_argument('--encoding', type=str, default=sENC)
parser.add_argument('profile', type=str, nargs='?', default=None,
help='tox profile file - may be encrypted')
return parser
if __name__ == '__main__':
iTOTAL = 0
lArgv = sys.argv[1:]
parser = oMainArgparser()
oArgs = parser.parse_args(lArgv)
if oArgs.command in ['edit'] and oArgs.edit == 'help':
l = list(dSTATE_TYPE.values())
l.remove('END')
print('Available Sections: ' +repr(l))
print('Supported Quads: section,num,key,type ' +sEDIT_HELP)
sys.exit(0)
sFile = oArgs.profile
assert os.path.isfile(sFile), sFile
sENC = oArgs.encoding
vSetupLogging()
bSAVE = open(sFile, 'rb').read()
@ -733,11 +659,11 @@ if __name__ == '__main__':
oStream = None
if oArgs.command == 'decrypt':
assert oArgs.output, "--output required for this command"
oStream = open(oArgs.output, 'wb')
iRet = oStream.write(bSAVE)
LOG.info(f"Wrote {iRet} to {oArgs.output}")
iRet = 0
if oArgs.output:
oStream = open(oArgs.output, 'rb')
else:
oStream = sys.stdout
oStream.write(bSAVE)
elif oArgs.command == 'nodes':
iRet = -1
@ -795,7 +721,7 @@ if __name__ == '__main__':
oStream.write(bSAVE)
else:
oStream = sys.stdout
oStream.write(str(bSAVE, sENC))
oStream.write(str(bSAVE, 'utf-8'))
iRet = -1
LOG.info(f"downloaded list of nodes saved to {oStream}")
@ -804,68 +730,49 @@ if __name__ == '__main__':
elif iRet == 0:
LOG.info(f"{oArgs.nodes} iRet={iRet} to {oArgs.output}")
elif oArgs.command in ['info', 'edit']:
if oArgs.command in ['edit']:
assert oArgs.output, "--output required for this command"
assert oArgs.edit != '', "--edit required for this command"
elif oArgs.command == 'info':
# assert oArgs.info != '', "--info required for this command"
if oArgs.info in ['save', 'yaml', 'json', 'repr', 'pprint']:
assert oArgs.output, "--output required for this command"
bOUT = b'\x00\x00\x00\x00\x1f\x1b\xed\x15'
# toxEsave
assert bSAVE[:8] == bMARK, "Not a Tox profile"
bOUT = bMARK
assert bSAVE[:8] == bOUT, "Not a Tox profile"
iErrs = 0
process_chunk(len(bOUT), bSAVE, oArgs)
if not bOUT:
LOG.error(f"{oArgs.command} NO bOUT results")
lOUT = []; aOUT = {}
process_chunk(len(bOUT), bSAVE)
if lOUT:
if oArgs.output:
oStream = open(oArgs.output, 'wb')
else:
oStream = None
LOG.debug(f"command={oArgs.command} len bOUT={len(bOUT)} results")
if oArgs.command in ['edit'] or oArgs.info in ['save']:
LOG.debug(f"{oArgs.command} saving to {oArgs.output}")
oStream = open(oArgs.output, 'wb', encoding=None)
if oStream.write(bOUT) > 0: iRet = 0
LOG.info(f"{oArgs.info}ed iRet={iRet} to {oArgs.output}")
oStream = sys.stdout
if oArgs.info == 'yaml' and yaml:
yaml.dump(aOUT, stream=oStream, indent=oArgs.indent)
oStream.write('\n')
elif oArgs.info == 'json' and json:
json.dump(aOUT, oStream, indent=oArgs.indent)
oStream.write('\n')
elif oArgs.info == 'repr':
oStream.write(repr(aOUT))
oStream.write('\n')
elif oArgs.info == 'pprint':
pprint(aOUT, stream=oStream, indent=oArgs.indent, width=80)
elif oArgs.info == 'info':
pass
elif oArgs.info == 'yaml' and yaml:
LOG.debug(f"{oArgs.command} saving to {oArgs.output}")
oStream = open(oArgs.output, 'wt', encoding=sENC)
yaml.dump(aOUT, stream=oStream, indent=oArgs.indent)
if oStream.write('\n') > 0: iRet = 0
LOG.info(f"{oArgs.info}ing iRet={iRet} to {oArgs.output}")
elif oArgs.info == 'json' and json:
LOG.debug(f"{oArgs.command} saving to {oArgs.output}")
oStream = open(oArgs.output, 'wt', encoding=sENC)
json.dump(aOUT, oStream, indent=oArgs.indent)
if oStream.write('\n') > 0: iRet = 0
LOG.info(f"{oArgs.info}ing iRet={iRet} to {oArgs.output}")
elif oArgs.info == 'repr':
LOG.debug(f"{oArgs.command} saving to {oArgs.output}")
oStream = open(oArgs.output, 'wt', encoding=sENC)
if oStream.write(repr(bOUT)) > 0: iRet = 0
if oStream.write('\n') > 0: iRet = 0
LOG.info(f"{oArgs.info}ing iRet={iRet} to {oArgs.output}")
elif oArgs.info == 'pprint':
LOG.debug(f"{oArgs.command} saving to {oArgs.output}")
oStream = open(oArgs.output, 'wt', encoding=sENC)
pprint(aOUT, stream=oStream, indent=oArgs.indent, width=80)
iRet = 0
LOG.info(f"{oArgs.info}ing iRet={iRet} to {oArgs.output}")
elif oArgs.info == 'nmap_tcp' and bHAVE_NMAP:
assert oArgs.output, "--output required for this command"
oStream.close()
vOsSystemNmapTcp(aOUT["TCP_RELAY"], oArgs)
elif oArgs.info == 'nmap_udp' and bHAVE_NMAP:
assert oArgs.output, "--output required for this command"
oStream.close()
vOsSystemNmapUdp(aOUT["DHT"], oArgs)
elif oArgs.info == 'nmap_onion' and bHAVE_NMAP:
assert oArgs.output, "--output required for this command"
oStream.close()
vOsSystemNmapUdp(aOUT["PATH_NODE"], oArgs)
# were short repacking as we read - 446 bytes missing
LOG.debug(f"len bSAVE={len(bSAVE)} bOUT={len(bOUT)} delta={len(bSAVE) - len(bOUT)} iTOTAL={iTOTAL}")
if oStream and oStream != sys.stdout and oStream != sys.stderr:
oStream.close()

120
tox_savefile_test.bash
View File

@ -1,120 +0,0 @@
#!/bin/sh -e
# -*- mode: sh; fill-column: 75; tab-width: 8; coding: utf-8-unix -*-
PREFIX=/o/var/local/src
EXE=python3.sh
WRAPPER=$PREFIX/toxygen_wrapper
[ -f /usr/local/bin/usr_local_tput.bash ] && \
. /usr/local/bin/usr_local_tput.bash || {
DEBUG() { echo DEBUG $* ; }
INFO() { echo INFO $* ; }
WARN() { echo WARN $* ; }
ERROR() { echo ERROR $* ; }
}
set -- -e
target=$PREFIX/tox_profile/logging_tox_savefile.py
[ -s $target ] || exit 1
tox=$HOME/.config/tox/toxic_profile.tox
[ -s $tox ] || exit 2
json=$HOME/.config/tox/DHTnodes.json
[ -s $json ] || exit 3
[ -d $WRAPPER ] || { ERROR wrapper is required https://git.plastiras.org/emdee/toxygen_wrapper ; exit 5 ; }
export PYTHONPATH=$WRAPPER
which jq > /dev/null && HAVE_JQ=1 || HAVE_JQ=0
which nmap > /dev/null && HAVE_NMAP=1 || HAVE_NMAP=0
sudo rm -f /tmp/toxic_profile.* /tmp/toxic_nodes.*
[ "$HAVE_JQ" = 0 ] || \
jq . < $json >/tmp/toxic_nodes.json || { ERROR jq $json ; exit 4 ; }
[ -f /tmp/toxic_nodes.json ] || cp -p $json /tmp/toxic_nodes.json
json=/tmp/toxic_nodes.json
# required password
INFO decrypt /tmp/toxic_profile.bin
$EXE $target --command decrypt --output /tmp/toxic_profile.bin $tox || exit 11
[ -s /tmp/toxic_profile.bin ] || exit 12
tox=/tmp/toxic_profile.bin
INFO info $tox
$EXE $target --command info --info info $tox 2>/tmp/toxic_profile.info || exit 13
[ -s /tmp/toxic_profile.info ] || exit 14
INFO /tmp/toxic_profile.save
$EXE $target --command info --info save --output /tmp/toxic_profile.save $tox 2>/dev/null || exit 15
[ -s /tmp/toxic_profile.save ] || exit 16
for the_tox in /tmp/toxic_profile.save ; do
the_base=`echo $the_tox | sed -e 's/.save$//' -e 's/.tox$//'`
for elt in json yaml pprint repr ; do
INFO $the_base.$elt
[ "$DEBUG" != 1 ] || echo DEBUG $EXE $target \
--command info --info $elt \
--output $the_base.$elt $the_tox
$EXE $target --command info --info $elt \
--output $the_base.$elt $the_tox 2>/dev/null || exit 20
[ -s $the_base.$elt ] || exit 21
done
$EXE $target --command edit --edit help $the_tox 2>/dev/null || exit 22
INFO $the_base.edit1 'STATUSMESSAGE,.,Status_message,Toxxed on Toxic'
$EXE $target --command edit --edit 'STATUSMESSAGE,.,Status_message,Toxxed on Toxic' \
--output $the_base.edit1.tox $the_tox 2>&1|grep EDIT
[ -s $the_base.edit1.tox ] || exit 23
$EXE $target --command info $the_base.edit1.tox 2>&1|grep Toxxed || exit 24
INFO $the_base.edit2 'NAME,.,Nick_name,FooBar'
$EXE $target --command edit --edit 'NAME,.,Nick_name,FooBar' \
--output $the_base.edit2.tox $the_tox 2>&1|grep EDIT
[ -s $the_base.edit2.tox ] || exit 25
$EXE $target --command info $the_base.edit2.tox 2>&1|grep FooBar || exit 26
done
the_tox=$json
the_base=`echo $the_tox | sed -e 's/.save$//' -e 's/.json$//'`
[ "$HAVE_JQ" = 0 ] || \
for nmap in select_tcp select_udp select_version ; do
INFO $the_base.$nmap
$EXE $target --command nodes --nodes $nmap \
--output $the_base.$nmap.json $the_tox || exit 31
[ -s $the_base.$nmap.json ] || exit 32
done
grep '"status_tcp": false' $the_base.select_tcp.json && exit 33
grep '"status_udp": false' $the_base.select_udp.json && exit 34
ls -l /tmp/toxic_profile.* /tmp/toxic_nodes.*
/usr/local/bin/proxy_ping_test.bash tor || exit 0
the_tox=$tox
the_base=`echo $the_tox | sed -e 's/.save$//' -e 's/.tox$//'`
[ "$HAVE_JQ" = 0 ] || \
[ "$HAVE_NMAP" = 0 ] || \
for nmap in nmap_tcp nmap_udp nmap_onion ; do
INFO $the_base.$nmap
$EXE $target --command info --info $nmap \
--output $the_base.$nmap $the_tox.json || exit 40
[ -s $the_base.$nmap.json ] || exit 41
done
the_json=$json
the_base=`echo $json | sed -e 's/.save$//' -e 's/.json$//'`
[ "$HAVE_JQ" = 0 ] || \
for nmap in nmap_tcp nmap_udp ; do
INFO $the_base.$nmap
$EXE $target --command nodes --nodes $nmap \
--output $the_base.$nmap.json $the_json || exit 51
[ -s $the_base.$nmap.json ] || exit 52
done
exit 0