Clarify documentation regarding packsize defaults

[simonsan]

I think this is more a documentation/clarification issue. Rustic has a dynamic targeted packsize depending on the repository size.
This means, if you increase the repository size, rustic does create large pack files and prune will try to repack packs with non-matching sizes. The algorithm about determining the target pack size and the thresholds when rustic will try to repack too small or too large sized packs are saved in the repository config file and can be set using rustic config (add --help to see the possible options).
However, I'm not sure if the chosen defaults in the for the tuning parameters can be improved, so feedback about it is highly welcome!

The idea about pack sizes depending on the repository size is that you get a good fit for individual repsoitories without the need to set/guess something optimal. In general, rustic's pack sizes are larger than restics and can go up to almost 4GiB.
The idea about repacking too small / or too large packs is that for a stable situation (periodic backups, periodic forget/prune), your repository size should stay almost constant, while this usually only changes at one-time events like adding new sources etc. And then a one-time prune to correct some pack sizes IMO is OK.

To ignore too small or too large packs, there is prune --no-resize which should behave exactly like restic.

Originally posted by @aawsome in rustic-rs/rustic#1017 (comment)

[ultimatespirit]

I came here to file a bug about packsizes specifically, but I believe this issue can serve as a good catch-all. Currently the documentation page for the --set-{tree,data}pack-size related settings appears to consist entirely of the following:

Options to specify the target pack size:

    --set-treepack-size, --set-datapack-size specify the default target pack size for tree and data pack files. Arguments can given using TODO For example, valid sizes are “4048kiB”, “2MB”, “30MiB”, etc. If not specified, the default is 4 MiB for tree packs and 32 MiB for data packs.

    --set-treepack-growfactor, --set-datapack-growfactor specify how much the target pack size should be increased per square root of the total pack size in bytes of the given type. This equals to 32kiB per square root of the total pack size in GiB.

Doing a source search for the terms didn't give any further results in the docs, and in the main rustic repo it only gave the zsh (and other shells) autocomplete command text (which was, oddly enough, almost the same as the above just without the TODO / missing text): https://github.com/rustic-rs/rustic/blob/d4420c3dce6e1ad85082e0ed128d774aed6e1d26/tests/completions-fixtures/zsh.txt#L795

So these parameters seem pretty under-documented, and there's no explanation of what even is a treepack versus a datapack, a distinction restic does not make.

Things that would be helpful to document, that also double as questions I have right now so certainly wouldn't mind even a brief tl;dr response in the issues here until documentation can get updated...:

• What these are and more precisely how they affect repository size. I'm guessing treepacks are packs of metadata to form the tree-structure restic/rustic presumably use internally, while datapacks are the data, and therefore it's similar to how ZFS or BTRFS function with datapacks mainly mediating how inefficient large filesize files and small filesize files are (larger pack size -> larger files are more eficient, but smaller ones are less efficient). Not really sure what the treepack side of things modulates there though, larger number means better handling of having lots of individual files perhaps?
• Fix the example documentation above, it looks like it really just needs to get the explanation that's in the shell completion files copied in. Currently they're pretty confusing due to the missing words.
• Since this feature appears to be related to repository size / expected repository size, a table or basic heuristic of say "if you're at X TB of space you probably want Y datapack size or higher" would be very helpful, especially since even with the new knowledge of how those grow-factors work, I am still left with little idea of what tweaking said grow factor would actually do for dynamically resizing the repo versus pessimistically setting the amount directly and not worrying about the grow factor.
• This is technically an issue for the main repository, but it would be good to have some level of information about this option (and several others it looks like) in the rustic --help output (or rustic init --help) at least, preferably a man page too, since currently it appears rustic has no man page at all. Perhaps the mdbook can be auto converted to a man page with one of the man page generating backends as a first step?

[aawsome]

@ultimatespirit Thanks for your input. I'll try to answer (hopefully all) points here and hope that someone is able to create a good description for the docu from it (or, if good enough, we can just copy it to the docu)

General:

• Treepacks are the packs containing tree blobs (i.e. metadata for files and dirs) whereas datapacks are containing the data/content of files. Both are stored independently in rustic and restic (at least using new version from the last ~5 years or so). But restic in principle supports mixed packs, whereas rustic doesn't. treepacks are obviously used in completely different process steps (e.g. ls) compared to datapacks (e.g. restore). treepacks are cached (if using a cache) and in case of a hot/cold repository only treepacks are stored in the hot repository. This all make fundamental differences which justifies different treatment also with respect to optimal sizing. Regarding the sizes, typical repositories have more than 99% of their repository size in datapacks.
• It should be understood that all sizes we are talking about in the configuration are "targeted pack sizes". Packs do pack several blobs which can be of different size, so it might be that without a certain blob we arrive below the targeted pack size, but including it we arrive quite some size above it. Real sizes can be typically about 1MiB - in extreme cases 8 or even more MiB - greater than the targeted pack size. There is an absolute maximum of a 4 GiB pack size due to internal 32 bit addressing within the packs.

Pro/cons of large/small packs:

In principle, the pack size does not matter for the functionality or consistency of your backups. Also there is basically no effect on total repository size when choosing another pack size.
Reasons for smaller/larger packfiles are mainly due to outer restrictions, performance or memory consumption.

• The main point why people want to tune is basically about storage backend restrictions which may be a limit of objects (larger packfiles => less objects needed), size bounds or higher costs for smaller/larger objects.
• Larger packs typically allow a more optimized access between rustic and a (remote) storage backend which can give performance benefits. Note that for treepacks this may be compensated by a local cache.
• Larger packs need a bit less memory for the in-memory-index which for most commands yield a big part of memory consumption.
• Larger packs need more memory when they are created or completely read. backup, prune and check are here affected in rustic.
• Larger packs tend to need more repacking than smaller packs as during regular backup and forget operations they get partly used with higher probability. This can imply slower prune runs or higher data costs for a prune. There are pruneoptions to reduce this for the cost of more unused data being left after the prune run, though. (e.g. for hot/cold repos, typically no repacking is done at all)
• Larger packs can also be a bit more performant in restore as they allow certain optimizations to work.

How to finetune "targeted pack sizes" in rustic?

• rustic implements a dynamic pack sizing, i.e. larger repositories automatically get larger pack sizes. The formula is min(size + grow_factor * sqrt(repo_size), max_size)
• For explanation: The sqrt dependency on the repository size ensures that the number of files in the respository also grows with sqrt(repo_size). For example if you have a large repository A and a repository B is 4 times the size of A, then B has about 2 times the number of files in the repository and pack sizes in B are about 2 times the pack sizes in repository A. So, this balances a growth in pack sizes and number of files in the repository for growing repository sizes.
• We give defaults for these size, grow_factor and max_size parameters. (Note: The defaults might change in future rustic versions if we have reasons/expirences to do so). So, the answer to the "I have a repo sized XX TB. What is the optimal pack size to use and how can I apply this?"-question is: Use the default and the formula will calculate the proposed pack size. Only adapt, if you have specific reasons to do so.
• To use a constant "targeted pack size", either set grow_factor = 0 or max_size = size.
• This is done for treepacks and datapacks separately, i.e. in fact it is not repo_size but size of all treepacks in the repo (or same with datapacks).
• In prune, rustic also uses a tolerance to not repack packs which have a almost fitting size. This is given as minimum and maximum tolerance in percentage of the `targeted pack size"
• All these settings are set in the in-repository config file and can be set either in rustic init or using rustic config and then hold for every consecutiv operation on this repo.