forked from Green-Sky/tomato
76 lines
2.1 KiB
Markdown
76 lines
2.1 KiB
Markdown
# Fragment Store
|
|
|
|
Fragments are are pieces of information split into Metadata and Data.
|
|
They can be stored seperated or together.
|
|
They can be used as a Transport protocol/logic too.
|
|
|
|
# Store types
|
|
|
|
### Object Store
|
|
|
|
Fragment files are stored with the first 2 hex chars as sub folders:
|
|
eg:
|
|
`objects/` (object store root)
|
|
- `5f/` (first 2hex subfolder)
|
|
- `4fffffff` (the fragment file without the first 2 hexchars)
|
|
|
|
### Split Object Store
|
|
|
|
Same as Object Store, but medadata and data stored in seperate files.
|
|
Metadata files have the `.meta` suffix. They also have a filetype specific suffix, like `.json`, `.msgpack` etc.
|
|
|
|
### Memory Store
|
|
|
|
Just keeps the Fragments in memory.
|
|
|
|
# File formats
|
|
|
|
Files can be compressed and encrypted. Since compression needs the data's structure to work properly, it is applied before it is encrypted.
|
|
|
|
### Text Json
|
|
|
|
Text json only makes sense for metadata if it's neither compressed nor encrypted. (otherwise its binary on disk anyway, so why waste bytes).
|
|
Since the content of data is not looked at, nothing stops you from using text json and ecrypt it, but atleast basic compression is advised.
|
|
|
|
A Metadata json object can have arbitrary keys, some are predefined:
|
|
- `FragComp::DataEncryptionType` (uint) Encryption type of the data, if any
|
|
- `FragComp::DataCompressionType` (uint) Compression type of the data, if any
|
|
|
|
## Binary file headers
|
|
|
|
### Split Metadata
|
|
|
|
file magic bytes `SOLMET` (6 bytes)
|
|
|
|
1 byte encryption type (`0x00` is none)
|
|
|
|
1 byte compression type (`0x00` is none)
|
|
|
|
...metadata here...
|
|
|
|
note that the encryption and compression are for the metadata only.
|
|
The metadata itself contains encryption and compression info about the data.
|
|
|
|
### Split Data
|
|
|
|
(none) all the data is in the metadata file.
|
|
This is mostly to allow direct storage for files in the Fragment store without excessive duplication.
|
|
Keep in mind to not use the actual file name as the data/meta file name.
|
|
|
|
### Single fragment
|
|
|
|
file magic bytes `SOLFIL` (6 bytes)
|
|
|
|
1 byte encryption type (`0x00` is none)
|
|
|
|
1 byte compression type (`0x00` is none)
|
|
|
|
...metadata here...
|
|
|
|
...data here...
|
|
|
|
## Compression types
|
|
|
|
- `0x00` none
|
|
- `0x01` zstd (without dict)
|