From cff0c100ec4c0fecedec4c3e31c70df96bcde9c0 Mon Sep 17 00:00:00 2001 From: Green Sky Date: Sun, 11 Feb 2024 19:01:48 +0100 Subject: [PATCH] add fragment store draft doc --- src/fragment_store/README.md | 72 ++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 src/fragment_store/README.md diff --git a/src/fragment_store/README.md b/src/fragment_store/README.md new file mode 100644 index 0000000..1687fae --- /dev/null +++ b/src/fragment_store/README.md @@ -0,0 +1,72 @@ +# Fragment Store + +Fragments are are pieces of information split into Metadata and Data. +They can be stored seperated or together. +They can be used as a Transport protocol/logic too. + +# Store types + +### Object Store + +Fragment files are stored with the first 2 hex chars as sub folders: +eg: +objects/ (object store root) + - 5f/ (first 2hex subfolder) + - 4fffffff (the fragment file without the first 2 hexchars) + +### Split Object Store + +Same as Object Store, but medadata and data stored in seperate files. +Metadata files have the `.meta` suffix. They also have a filetype specific suffix, like `.json`, `.msgpack` etc. + +### Memory Store + +Just keeps the Fragments in memory. + + +# File formats + +Files can be compressed and encrypted. Since compression needs the data structure to funcion, it is applied before it is encrypted. + +### Text Json + +Text json only makes sense for metadata if it's neither compressed nor encrypted. (otherwise its binary on disk anyway, so why waste bytes). +Since the content of data is not looked at, nothing stops you from using text json and ecrypt it, but atleast basic compression is advised. + +A Metadata json object has the following keys: +- `enc` Encryption type of the data, if any +- `comp` Compression type of the data, if any + +## Binary file headers + +### Split Metadata + +file magic bytes `SOLMET` (6 bytes) + +1 byte encryption type (0x00 is none) + +1 byte compression type (0x00 is none) + +...metadata here... + +note that the ecnryption and compression are for the metadata only. +The metadata itself contains encryption and compression about the data. + +### Split Data + +(none) all the data is in the metadata file. +This is mostly to allow direct storage for files in the Fragment store without excessive duplication. +Keep in mind to not use the actual file name as the data/meta file name. + +### Single fragment + +file magic bytes `SOLFIL` (6 bytes) + +1 byte encryption type (0x00 is none) + +1 byte compression type (0x00 is none) + +...metadata here... + +...data here... +