add fragment store draft doc
This commit is contained in:
parent
010c49d100
commit
cff0c100ec
72
src/fragment_store/README.md
Normal file
72
src/fragment_store/README.md
Normal file
@ -0,0 +1,72 @@
|
||||
# Fragment Store
|
||||
|
||||
Fragments are are pieces of information split into Metadata and Data.
|
||||
They can be stored seperated or together.
|
||||
They can be used as a Transport protocol/logic too.
|
||||
|
||||
# Store types
|
||||
|
||||
### Object Store
|
||||
|
||||
Fragment files are stored with the first 2 hex chars as sub folders:
|
||||
eg:
|
||||
objects/ (object store root)
|
||||
- 5f/ (first 2hex subfolder)
|
||||
- 4fffffff (the fragment file without the first 2 hexchars)
|
||||
|
||||
### Split Object Store
|
||||
|
||||
Same as Object Store, but medadata and data stored in seperate files.
|
||||
Metadata files have the `.meta` suffix. They also have a filetype specific suffix, like `.json`, `.msgpack` etc.
|
||||
|
||||
### Memory Store
|
||||
|
||||
Just keeps the Fragments in memory.
|
||||
|
||||
|
||||
# File formats
|
||||
|
||||
Files can be compressed and encrypted. Since compression needs the data structure to funcion, it is applied before it is encrypted.
|
||||
|
||||
### Text Json
|
||||
|
||||
Text json only makes sense for metadata if it's neither compressed nor encrypted. (otherwise its binary on disk anyway, so why waste bytes).
|
||||
Since the content of data is not looked at, nothing stops you from using text json and ecrypt it, but atleast basic compression is advised.
|
||||
|
||||
A Metadata json object has the following keys:
|
||||
- `enc` Encryption type of the data, if any
|
||||
- `comp` Compression type of the data, if any
|
||||
|
||||
## Binary file headers
|
||||
|
||||
### Split Metadata
|
||||
|
||||
file magic bytes `SOLMET` (6 bytes)
|
||||
|
||||
1 byte encryption type (0x00 is none)
|
||||
|
||||
1 byte compression type (0x00 is none)
|
||||
|
||||
...metadata here...
|
||||
|
||||
note that the ecnryption and compression are for the metadata only.
|
||||
The metadata itself contains encryption and compression about the data.
|
||||
|
||||
### Split Data
|
||||
|
||||
(none) all the data is in the metadata file.
|
||||
This is mostly to allow direct storage for files in the Fragment store without excessive duplication.
|
||||
Keep in mind to not use the actual file name as the data/meta file name.
|
||||
|
||||
### Single fragment
|
||||
|
||||
file magic bytes `SOLFIL` (6 bytes)
|
||||
|
||||
1 byte encryption type (0x00 is none)
|
||||
|
||||
1 byte compression type (0x00 is none)
|
||||
|
||||
...metadata here...
|
||||
|
||||
...data here...
|
||||
|
Loading…
Reference in New Issue
Block a user