Cache all root metadata versions

[jku]

Making this ready for review based on discussion in community meeting. There's no rush in getting this merged: let's make sure the review / testing is adequate.

Problem

Currently python-tuf clients typically initialize once from the "bootstrap" root (the root metadata shipped with the client software), and subsequent startups will use the cached root as starting point. This is mostly good but there is a downside: if the bootstrap root is more secure than the cache, then we lose some security (as cache could get poisoned between the first cache initialization and the actual use). This situation can arise if the client software (and bootstrap root) is shipped in a Operating System update, container image or e.g. a Debian package: in this case the client cache is writable by user but the bootstrap root and the client software may not be.

Solution

• Cache all root versions
• Always initialize the client starting with the secure bootstrap root
• When loading subsequent root versions, load from local cache and from remote repository

This should give us best of both worlds:

• client always initializes from most securely stored root
• client does not download any more data than now
• client still uses local cache for roots so repository cannot serve obsolete data

The only downside is that client will now verify every root version on startup: I think this perfectly acceptable as client can and should periodically update the bootstrap root as well -- the added computation is not an issue in practice.

Details

The ngclient cache directory looked like this before:

cachedir/
    root.json
    timestamp.json
    snapshot.json
    targets.json
    somedelegatedrole.json

Now it looks like this:

cachedir/
    root.json        #  This is a symlink to "root_history/3.root.json" for compatibility
    timestamp.json
    snapshot.json
    targets.json
    somedelegatedrole.json
    root_history/
        1.root.json
        2.root.json
        3.root.json

• non-versioned root.json is preserved as symlink so that older python-tuf versions can coexist (and so it's easier to check current state for humans and tests)
• root_history directory is used to make name clashes impossible (imagine a delegated role named "1.root") : as long as the client is not susceptible to path traversal issues, this should be safe
• tests have been added to verify that roots are cached, cached roots are used and that poisoned cache is handled correctly (remote metadata is used if cache is not valid)

[jku]

I think I'll refactor the large test cleanups to a different PR so this doesn't look so big (it isn't)

[kairoaraujo]

I think I'll refactor the large test cleanups to a different PR so this doesn't look so big (it isn't)

I'm ok with reviewing it if you want to add to this PR (separate commit).

[jku]

not planning to add anything: I just meant the test file removals that are already in the PR: "+268 −1,253" looks like a big change but the actual PR is really manageable

[jku]

OK:
there is now #2768 that should be handled before this one: this changes this PR into a fairly small one: 235 insertions(+), 89 deletions(-)

[jku]

rebased on main to build on top of #2768 to make this PR a reasonable size

[kairoaraujo]

I added a nit comment as I feel that the note could be more explicit.

[kairoaraujo]

Should we log a warning to the user here?

[jku]

I don't think that would be helpful:

• warnings are useful if they allow the user to react somehow. This is not the case here
• the note is there to prevent someone from copy-pasting example code to a real client that should be shipping an embedded root instead

[jku]

Thanks for reviews. I'll let this sit for now as it will clash with #2762 which I was hoping to merge soonish.

[kairoaraujo]

@jku I guess we can rebase it and merge, right?

[jku]

uh, kind of but it will still clash with #2773: I think I'd rather resolve after that one