Commit Graph

271 Commits

Author SHA1 Message Date
de3b8f059e
import os and zstdfile2 bugfixes 2024-04-13 21:06:51 +02:00
7b4af58544
prep convert tool for msgpack transition 2024-04-13 19:47:28 +02:00
f287348550
introduce message fragments version 2 (msgpack)
more smaller refactors
2024-04-13 19:13:18 +02:00
498b4435c7
refactor message contexts 2024-04-13 11:38:13 +02:00
a5e67d0ee8
move uuid gen to util 2024-04-12 19:44:24 +02:00
dfbb1dea68
move os and backend to sub 2024-04-12 13:42:08 +02:00
2597edd579
new messages objcomp names 2024-04-12 13:42:08 +02:00
85a29372f4
adding message frag object version and conversion work, but comp name changes incoming 2024-04-12 13:42:07 +02:00
a9f6a5d763
move mfs to os, works, convert tool still incomplete 2024-04-12 13:42:07 +02:00
73180195fe
some more backend interface changes i realized i had to do 2024-04-12 13:42:07 +02:00
10b689ca95
refactor the serializer again 2024-04-12 13:42:07 +02:00
3796841961
conversion improvements 2024-04-12 13:42:07 +02:00
854ed851b4
fixes missing comps and conversion tool is almost working (most of meta is transfered) 2024-04-12 13:42:07 +02:00
5c3b797a99
new object (os + fsb mostly done) 2024-04-12 13:42:06 +02:00
8a580e2fbb
backend fs read 2024-04-12 13:42:06 +02:00
3cede91aa0
more work on backend and moving frags to objs 2024-04-12 13:42:06 +02:00
0610a6a64a
continue os refactor, start with fs backend 2024-04-12 13:42:06 +02:00
26d07b06db
start refactoring in the name of object store 2024-04-12 13:42:06 +02:00
fd0b210bbb
fragment store fully file2 zstd. no more zstd.h in fragment store 2024-04-12 13:42:06 +02:00
268cbe137e
move file stack creation to separate file, removing some scope 2024-04-12 13:42:05 +02:00
31bb0d3e61
refactor meta data decompression using file2 and mem 2024-04-12 13:42:05 +02:00
248f68f6a2
minimal stack creation refactor (wip) 2024-04-12 13:42:05 +02:00
3b010bd16f
catch write errors (eg unsanitzed strings) 2024-04-12 13:42:05 +02:00
7e285290fe
test in memory comp 2024-04-12 13:42:05 +02:00
5767834f71
since its working, remove old commented code 2024-04-12 13:42:05 +02:00
53ce292e82
fix stack overflow (i am stupid) and some file stack error handling 2024-04-12 13:42:04 +02:00
84bd24807d
small change to flush work space. writing new frags is broken rn 2024-04-12 13:42:04 +02:00
8d0518c2e3
harden against some parsing exceptions
(by disabling them, since the error case is already handled)
2024-04-12 13:42:04 +02:00
6d150ba441
roll back meta comp (did it wrong) and enable data comp 2024-04-12 13:42:04 +02:00
c737715c66
more refactor and transition meta write 2024-04-12 13:42:04 +02:00
b640b5a06b
variant to span helper 2024-04-12 13:42:04 +02:00
1b9363e7b5
tested and works, cleaning up commented code 2024-04-12 13:42:03 +02:00
16d2238f35
fixes for ci 2024-04-12 13:42:03 +02:00
19844a9423
use file2 zstd wrapper to read frag data (untested) 2024-04-12 13:42:03 +02:00
19fd99f713
refactor out uuidgenerator 2024-04-12 13:42:03 +02:00
f22f523774
minor frag store refactor 2024-04-12 13:42:03 +02:00
8b17ed195f
more testing and file2 zstd now passes tests with varying frame sizes
and 1.5gig files
2024-04-12 13:42:03 +02:00
def7fc1959
add file2 impl for zstd (lightly tested and not integrated yet) 2024-04-12 13:42:02 +02:00
318be9cd62
throw update 2024-04-12 13:42:02 +02:00
2772c8ee69
reduce excessive message frag saving (queue dedup + waiting 10sec)
prepare for frag updates
2024-04-12 13:42:02 +02:00
eac2927379
try to tame log spam 2024-04-12 13:42:02 +02:00
77a0ae6acd
fix accel structure being wrong and mark empty frags and dont count them 2024-04-12 13:42:02 +02:00
7879a0927b
combat memory leaks with smart pointers 2024-04-12 13:42:02 +02:00
88ea3e177d
refactor saving and save on exit 2024-04-12 13:42:01 +02:00
bc22451524
dirty frag on message updates (if still open) 2024-04-12 13:42:01 +02:00
7b8e93eec3
refactor message fuid -> fid
save alot of memory by using fid instead of fuid
2024-04-12 13:42:01 +02:00
71be5c3c6e
reduce log spam 2024-04-12 13:42:01 +02:00
2b8cee6a29
remove old code 2024-04-12 13:42:01 +02:00
5bf4640d61
forgot to throw update on read 2024-04-12 13:42:01 +02:00
0e0e81720b
dont sync messages we dont know enough about 2024-04-12 13:42:00 +02:00
592a4cb9cf
make adjacency loading work, extend range and use loops 2024-04-12 13:42:00 +02:00
93f60bd073
replace old bad prev/next code with way better code 2024-04-12 13:42:00 +02:00
6a6de77ae9
smaller contact frag fixes 2024-04-12 13:42:00 +02:00
89f065a610
impl new acceleration structure for components, not exploited yet
disable funky load at first msg
2024-04-12 13:42:00 +02:00
52e95ca654
forgot to check contact 2024-04-12 13:41:59 +02:00
eaa316a2aa
rework cursers for cg, keep views between switching. will be refactored later 2024-04-12 13:41:59 +02:00
bdf4e60f2f
fix one inverted comparator 2024-04-12 13:41:59 +02:00
2e3c779bec
stop ignoring mfs interval and sort after 2024-04-12 13:41:59 +02:00
461a4f1aa7
make inital curser a range 2024-04-12 13:41:59 +02:00
78488daa9b
loading logic implemented but broken (very funky and sometimes even out of contact) 2024-04-12 13:41:59 +02:00
22f2c8f514
load based on view cursers (untested and not used yet) 2024-04-12 13:41:58 +02:00
e442191aad
msg frag before and after helper 2024-04-12 13:41:58 +02:00
795ab2d4e1
fix potential tsrange errors and deduplicate state 2024-04-12 13:41:58 +02:00
0896038dd6
make writing safe (by using a tmp file and moving to actual location) 2024-04-12 13:41:58 +02:00
67c6f9adb0
make empty contacts from ids on message load 2024-04-12 13:41:58 +02:00
6aac44cda9
change binary meta format and add zstd to metadata 2024-04-12 13:41:58 +02:00
4fb2b51b7d
switch to streaming compressor for data to drastically improve ratio.
would still benefit from a abstract file refactor
2024-04-12 13:41:57 +02:00
182d844e32
update fs readme a little 2024-04-12 13:41:57 +02:00
6f511016bc
save msg json zstd compressed (3x compression) 2024-04-12 13:41:57 +02:00
fb885b5c21
simplify array cast a little 2024-04-12 13:41:57 +02:00
d21dbb43e2
comp refactor and make groups work 2024-04-12 13:41:57 +02:00
7ac62274f4
move json around and disable files for now 2024-04-12 13:41:56 +02:00
4ec87337c8
reverse message write order 2024-04-12 13:41:56 +02:00
20f7c6d011
add dup check, would work for ngc if we saved tox group msg id yet 2024-04-12 13:41:56 +02:00
24dc5a03f3
fix dup on write 2024-04-12 13:41:56 +02:00
3d0863ff9a
basically working, but some dup glitch is still there 2024-04-12 13:41:56 +02:00
97aedca844
scan laters 2024-04-12 13:41:56 +02:00
2e7d5538d1
fragment events + 256bit uuids 2024-04-12 13:41:55 +02:00
1bfd04680e
refactor message serializer to allow access to eg contacts 2024-04-12 13:41:55 +02:00
73d1d65142
further serializer refactoring 2024-04-12 13:41:55 +02:00
f6e55851cc
improve deserialization and provide message comp deserl 2024-04-12 13:41:55 +02:00
0b0245d844
loading fragments mostly working (not notifying anyone yet) 2024-04-12 13:41:55 +02:00
d278391528
add contact id to meta 2024-04-12 13:41:55 +02:00
aa7a5d6013
more comps 2024-04-12 13:41:54 +02:00
84987216cb
handle empty type 2024-04-12 13:41:54 +02:00
58e9fd5514
dump messages to data (some comps) 2024-04-12 13:41:54 +02:00
3d41eedf48
message fragment meta is saved, but still empty data 2024-04-12 13:41:54 +02:00
2bc30ffcdc
start with messages (no fragments get created yet) 2024-04-12 13:41:54 +02:00
e67d7d37b5
refactoring, add to mainscreen 2024-04-12 13:41:54 +02:00
98ab974515
random ids 2024-04-12 13:41:53 +02:00
267f8dffc1
working prototpying code 2024-04-12 13:41:53 +02:00
78c7ef5b46
fix main screen destruction order,
so plugins can cleanup before their facilities get destroyed
2024-04-06 11:59:05 +02:00
bde0f2c7c3
imgui debug stuff 2024-03-15 13:02:49 +01:00
83bbac2cd1
proper cleanup procedure for main 2024-03-15 13:02:37 +01:00
d5dede5a61
properly cleanup in main 2024-03-12 18:43:36 +01:00
719400068a
cache the localtime result in the message
localtime uses global state AND spams allocations like cazy
2024-03-12 18:09:42 +01:00
aaf8c6adc1
save cooldown for tox profile (10sec after last) and save on exit 2024-03-12 18:09:21 +01:00
dc081ae2aa
fix webp mem leak 2024-03-11 20:46:28 +01:00
3cf3097094
toxcore update + private interface update 2024-03-11 11:46:18 +01:00
887705969f
make texture streaming actually work 2024-03-09 16:26:35 +01:00