diff --git a/README.md b/README.md new file mode 100644 index 0000000..2692a45 --- /dev/null +++ b/README.md @@ -0,0 +1,63 @@ +# Object Store + +Objects are are pieces of information split into Metadata and Data. +They can be stored seperated or together. +They can be used as a Transport protocol/logic too. + +[see wikipedia](https://en.wikipedia.org/wiki/Object_store) + +# Store types + +### Filesystem Store + +Fragment files are stored with the first 2 hex chars as sub folders: +eg: +`objects/` (object store root) + - `5f/` (first 2hex subfolder) + - `4fffffff` (the fragment file without the first 2 hexchars) + +Metadata and data stored in seperate files. +Metadata files have the `.meta` suffix. They also have a filetype specific suffix, like `.json`, `.msgpack` etc. +(single file objects are planned but no ETA) + +### Memory Store + +Just keeps the Fragments in memory. + +# File formats + +Files can be compressed and encrypted. Since compression needs the data's structure to work properly, it is applied before it is encrypted. + +### Text Json + +Text json only makes sense for metadata if it's neither compressed nor encrypted. (otherwise its binary on disk anyway, so why waste bytes). +Since the content of data is not looked at, nothing stops you from using text json and ecrypt it, but atleast basic compression is advised. + +A Metadata json object can have arbitrary keys, some are predefined: +- `FragComp::DataEncryptionType` (uint) Encryption type of the data, if any +- `FragComp::DataCompressionType` (uint) Compression type of the data, if any + +### Binary msgpack + +msgpack array: + +- `[0]`: file magic string `SOLMET` (6 bytes) +- `[1]`: uint8 encryption type (`0x00` is none) +- `[2]`: uint8 compression type (`0x00` is none, `0x01` is zstd) +- `[3]`: binary metadata (optionally compressed and encrypted) + +note that the encryption and compression are for the metadata only. +The metadata itself contains encryption and compression info about the data. + +### Split Data + +All the metadata is in the metadata file. (like encryption and compression) +This is mostly to allow direct storage for files in the Fragment store without excessive duplication. +Keep in mind to not use the actual file name as the data/meta file name. + +## Compression types + +- `0x00` none +- `0x01` zstd (without dict) + +zstd with hardcoded dictionaries will come later.