Documentation on tiered storage use

[arend-melissant-tnt]

Hi,

I want to run garnet on aks to replace redis. I esp. like the tiered storage option, but digging through the sites I am unable to find any detailed info on how to use tiered storage. Why is there no documentation describing real world usage of garnet with tiered storage, esp to blob storage.
Also, what are the real world implications of using tiered storage, is this transparent as in the data gets written to blob storage and retrieved when reading. I can find some bits and pieces of detailed technical info, but no pragmatic use cases. I can find configuration options, but the explanation behind these options is also very limited.

Hoping someone can shed some light on this so I can start a POC for our use case...

Best regards
Arend

[shanipribadi]

basically just configure --storage-tier and --logdir (or optionally --use-azure-storage for the azure blob storage)
https://microsoft.github.io/garnet/docs/getting-started/configuration

it is mostly transparent, but you may want to read the compaction and readcache part for additional configuration/optimization
https://microsoft.github.io/garnet/docs/getting-started/compaction
https://microsoft.github.io/garnet/docs/getting-started/memory

[arend-melissant-tnt]

Hi,

the documentation is good if you use garnet as a key-value store, then it works as expected.
I use garnet to store sets of data, and that's where things fall apart.

if i start garnet with the following parameters (it took some digging and experimenting to find the right values):

garnet-server -l logs --storage-tier true --memory 128m --page 4k --obj-page 512 --obj-heap-memory 128m --obj-log-memory 1k

then it works as expected, assuming i create sets of 1000 entries or less (memory < 1 GByte). If i run my tests and add 10000 entries per set, things fall apart and the memory usage goes through the roof (>5 GBytes). each entry in the set is a string of 1024 characters