Commit Graph

439 Commits

Author SHA1 Message Date
e0d873d41c
Merge branch 'master' into fragment_store 2024-04-14 14:32:31 +02:00
5728432b76
everything put into their respective repos 2024-04-14 14:30:42 +02:00
0030487613
move converter tool and pics out 2024-04-14 14:16:53 +02:00
f932f5ffb4
mfs and ms moved to their own repo, now only a few files left to clean up 2024-04-14 13:58:31 +02:00
da83065024
make sure we have the right json version in the flake 2024-04-14 12:03:58 +02:00
a6614e76ce
move cursers to public api 2024-04-14 11:13:05 +02:00
cdd67f4779
fix flake and pull in os zstd fixes 2024-04-14 10:57:07 +02:00
a845609660
actually allow loading v2 and enable in converter 2024-04-14 10:23:13 +02:00
de3b8f059e
import os and zstdfile2 bugfixes 2024-04-13 21:06:51 +02:00
7b4af58544
prep convert tool for msgpack transition 2024-04-13 19:47:28 +02:00
f287348550
introduce message fragments version 2 (msgpack)
more smaller refactors
2024-04-13 19:13:18 +02:00
498b4435c7
refactor message contexts 2024-04-13 11:38:13 +02:00
a5e67d0ee8
move uuid gen to util 2024-04-12 19:44:24 +02:00
9e30983b22
Merge branch 'master' into fragment_store 2024-04-12 19:24:20 +02:00
195a87b8ab
add object store and expose to plugins 2024-04-12 19:03:30 +02:00
dfbb1dea68
move os and backend to sub 2024-04-12 13:42:08 +02:00
2597edd579
new messages objcomp names 2024-04-12 13:42:08 +02:00
85a29372f4
adding message frag object version and conversion work, but comp name changes incoming 2024-04-12 13:42:07 +02:00
a9f6a5d763
move mfs to os, works, convert tool still incomplete 2024-04-12 13:42:07 +02:00
73180195fe
some more backend interface changes i realized i had to do 2024-04-12 13:42:07 +02:00
10b689ca95
refactor the serializer again 2024-04-12 13:42:07 +02:00
3796841961
conversion improvements 2024-04-12 13:42:07 +02:00
854ed851b4
fixes missing comps and conversion tool is almost working (most of meta is transfered) 2024-04-12 13:42:07 +02:00
5c3b797a99
new object (os + fsb mostly done) 2024-04-12 13:42:06 +02:00
8a580e2fbb
backend fs read 2024-04-12 13:42:06 +02:00
3cede91aa0
more work on backend and moving frags to objs 2024-04-12 13:42:06 +02:00
0610a6a64a
continue os refactor, start with fs backend 2024-04-12 13:42:06 +02:00
26d07b06db
start refactoring in the name of object store 2024-04-12 13:42:06 +02:00
fd0b210bbb
fragment store fully file2 zstd. no more zstd.h in fragment store 2024-04-12 13:42:06 +02:00
268cbe137e
move file stack creation to separate file, removing some scope 2024-04-12 13:42:05 +02:00
31bb0d3e61
refactor meta data decompression using file2 and mem 2024-04-12 13:42:05 +02:00
248f68f6a2
minimal stack creation refactor (wip) 2024-04-12 13:42:05 +02:00
3b010bd16f
catch write errors (eg unsanitzed strings) 2024-04-12 13:42:05 +02:00
7e285290fe
test in memory comp 2024-04-12 13:42:05 +02:00
5767834f71
since its working, remove old commented code 2024-04-12 13:42:05 +02:00
53ce292e82
fix stack overflow (i am stupid) and some file stack error handling 2024-04-12 13:42:04 +02:00
84bd24807d
small change to flush work space. writing new frags is broken rn 2024-04-12 13:42:04 +02:00
8d0518c2e3
harden against some parsing exceptions
(by disabling them, since the error case is already handled)
2024-04-12 13:42:04 +02:00
6d150ba441
roll back meta comp (did it wrong) and enable data comp 2024-04-12 13:42:04 +02:00
c737715c66
more refactor and transition meta write 2024-04-12 13:42:04 +02:00
b640b5a06b
variant to span helper 2024-04-12 13:42:04 +02:00
1b9363e7b5
tested and works, cleaning up commented code 2024-04-12 13:42:03 +02:00
16d2238f35
fixes for ci 2024-04-12 13:42:03 +02:00
19844a9423
use file2 zstd wrapper to read frag data (untested) 2024-04-12 13:42:03 +02:00
19fd99f713
refactor out uuidgenerator 2024-04-12 13:42:03 +02:00
f22f523774
minor frag store refactor 2024-04-12 13:42:03 +02:00
8b17ed195f
more testing and file2 zstd now passes tests with varying frame sizes
and 1.5gig files
2024-04-12 13:42:03 +02:00
def7fc1959
add file2 impl for zstd (lightly tested and not integrated yet) 2024-04-12 13:42:02 +02:00
318be9cd62
throw update 2024-04-12 13:42:02 +02:00
2772c8ee69
reduce excessive message frag saving (queue dedup + waiting 10sec)
prepare for frag updates
2024-04-12 13:42:02 +02:00