History

Green Sky 6a6de77ae9 smaller contact frag fixes		2024-04-12 13:42:00 +02:00
..
fragment_store_i.cpp	fragment events + 256bit uuids	2024-04-12 13:41:55 +02:00
fragment_store_i.hpp	fragment events + 256bit uuids	2024-04-12 13:41:55 +02:00
fragment_store.cpp	load based on view cursers (untested and not used yet)	2024-04-12 13:41:58 +02:00
fragment_store.hpp	basically working, but some dup glitch is still there	2024-04-12 13:41:56 +02:00
fs_binary_msgpack1.png	change binary meta format and add zstd to metadata	2024-04-12 13:41:58 +02:00
fs_binary_msgpack2.png	change binary meta format and add zstd to metadata	2024-04-12 13:41:58 +02:00
message_fragment_store.cpp	smaller contact frag fixes	2024-04-12 13:42:00 +02:00
message_fragment_store.hpp	load based on view cursers (untested and not used yet)	2024-04-12 13:41:58 +02:00
message_serializer.cpp	make empty contacts from ids on message load	2024-04-12 13:41:58 +02:00
message_serializer.hpp	refactor message serializer to allow access to eg contacts	2024-04-12 13:41:55 +02:00
meta_components_id.inl	refactoring, add to mainscreen	2024-04-12 13:41:54 +02:00
meta_components.hpp	refactoring, add to mainscreen	2024-04-12 13:41:54 +02:00
README.md	change binary meta format and add zstd to metadata	2024-04-12 13:41:58 +02:00
register_mfs_json_message_components.cpp	comp refactor and make groups work	2024-04-12 13:41:57 +02:00
register_mfs_json_message_components.hpp	comp refactor and make groups work	2024-04-12 13:41:57 +02:00
register_mfs_json_tox_message_components.cpp	comp refactor and make groups work	2024-04-12 13:41:57 +02:00
register_mfs_json_tox_message_components.hpp	comp refactor and make groups work	2024-04-12 13:41:57 +02:00
serializer.hpp	further serializer refactoring	2024-04-12 13:41:55 +02:00
test_fragstore.cpp	further serializer refactoring	2024-04-12 13:41:55 +02:00
types.hpp	change binary meta format and add zstd to metadata	2024-04-12 13:41:58 +02:00

README.md

Fragment Store

Fragments are are pieces of information split into Metadata and Data. They can be stored seperated or together. They can be used as a Transport protocol/logic too.

Store types

Object Store

Fragment files are stored with the first 2 hex chars as sub folders: eg: objects/ (object store root)

5f/ (first 2hex subfolder)
- 4fffffff (the fragment file without the first 2 hexchars)

Split Object Store

Same as Object Store, but medadata and data stored in seperate files. Metadata files have the .meta suffix. They also have a filetype specific suffix, like .json, .msgpack etc.

Memory Store

Just keeps the Fragments in memory.

File formats

Files can be compressed and encrypted. Since compression needs the data's structure to work properly, it is applied before it is encrypted.

Text Json

Text json only makes sense for metadata if it's neither compressed nor encrypted. (otherwise its binary on disk anyway, so why waste bytes). Since the content of data is not looked at, nothing stops you from using text json and ecrypt it, but atleast basic compression is advised.

A Metadata json object can have arbitrary keys, some are predefined:

FragComp::DataEncryptionType (uint) Encryption type of the data, if any
FragComp::DataCompressionType (uint) Compression type of the data, if any

Binary file headers

Split Metadata

msgpack array:

[0]: file magic string SOLMET (6 bytes)
[1]: uint8 encryption type (0x00 is none)
[2]: uint8 compression type (0x00 is none, 0x01 is zstd)
[3]: binary metadata (optionally compressed and encrypted)

note that the encryption and compression are for the metadata only. The metadata itself contains encryption and compression info about the data.

Split Data

All the metadata is in the metadata file. (like encryption and compression) This is mostly to allow direct storage for files in the Fragment store without excessive duplication. Keep in mind to not use the actual file name as the data/meta file name.

Single fragment

Note: this format is unused for now

file magic bytes SOLFIL (6 bytes)

1 byte encryption type (0x00 is none)

1 byte compression type (0x00 is none)

...metadata here...

...data here...

Compression types

0x00 none
0x01 zstd (without dict)