Add ability to schedule snapshot, prune and check after another (not using cron or similar)

[iluvatyr]

Is your feature request related to a problem? Please describe.
As of now, scheduling does not allow to just specify that I want them to run after each other.
This is quite limiting, as I have to somehow time operations in my head and have to take into account the snapshot duration etc. with the cronjobs timings.
Additionally, in the repository options, it only allows to set the hours, which, if I have backup, prune, and check operations have to be seperated by minimum one hour.

Describe the solution you'd like
Add functionality to let the prune and check and snapshot be run on condition that another operation ran

snapshot --> Options: 1) Run after check, 2) Run after prune
prune --> Options 1) Run after snapshot, 2) Run after check
check --> Options 1) Run after snapshot, 2) Run after prune

This gives great additional functionality while not affecting anything else. Ultimately giving more flexibility

[garethgeorge]

I think there's definitely something that could be improved here, adding even more flexible scheduling of some sort is something I'm keeping in mind.

I want to avoid hard coding to solve the specific flows and instead provide some sort of general scheduling. Brainstorming out loud here, what I have in my head at the moment is to possibly replace the concepts of specific scheduled operations with a concept of a "flow".

Flows would be configured at the repo level OR at the plan level, and each flow includes

• A schedule (e.g. when to run the flow)
• A list of operations to run.
  • At the repo level this can be be any of check, prune, or forget operation.
  • At the plan level this can be a backup operation and optionally a forget operation.
• Possibly hooks?

Existing configs would be migrated converting the options you see on the repo today into a flow that runs a check and then a prune as you say. Plans would be migrated to contain a flow that run a backup followed by a forget.

The major caveat here is that this is complicated and somewhat violates Backrest's goals of simplicity / ease to setup. Which is much of why I've not pulled the trigger on this change yet. It does add a lot of flexibility, however and will become increasingly important as I implement multihost management solutions.

Other open questions are ...

• how should hooks interact with this? Do they become a part of the flow config?
• what refactorings are needed in the UI framework to keep such an implementation maintainable

so it's a longer-term project.

[adamhl8]

I'd love for this to be implemented as well. Before moving to backrest, my backup script looked like:

restic backup /path
restic forget --prune --keep-within 1m
restic check

I'd like to do the same thing with backrest but that doesn't really seem possible.

It seems like it should be possible with command hooks but none of the environment variables that are set in the repository config are available in the hook.

[garethgeorge]

In addition to my long comment above, I just want to callout that backrest today will ...

1. Only ever run a single task at a time, so don't worry about overlapping crons.
2. Prioritizes operations such that prunes are highest priority, then check, then backups.

So if you schedule a backup, prune, and check to run at the same time with cron schedules you will get the behavior that

1. The prune runs
2. Then the check runs
3. Finally the backup runs

[iluvatyr]

Hey thanks already for the fast replies.

A schedule (e.g. when to run the flow)
A list of operations to run.
At the repo level this can be be any of check, prune, or forget operation.
At the plan level this can be a backup operation and optionally a forget operation.
Possibly hooks?

I would just like to note that additionally I would like also connectors between plan level and repo level, so that after a successful snapshot, the repo level commands (check, prune) can be run and vice versa.
This could for example easily be done by just using a hook

ON_SUCCESSFUL_SNAPSHOT --> CONTINUE REPO FLOW (check, prune)
ON_SUCCESSFUL_CHECK --> CONTINUE PLAN FLOW (snapshot)

The flow idea sounds great.
Just some thoughts from me what I would do to implement it (though I lack the ability to do it)

1. leave all scheduling like it is right now and dont touch existing configs (do not migrate anything and leave as is and let people chose to migrate themselves if they actively choose to.
2. I would actually not integrate flows into Plan or repo level but rather make a 3rd point called "flow scheduling", which then can just be used to create a procedure for the plan and repo level commands and lets schedule through it.

So under this 3rd point "Flows" positioned in the GUI under the "Plans " and "Repos" one could set up a flow.
This flow should by design automatically see which plan backs up into which flow and ask if you want to create a flow for that combination and when accepting, scheduling at Plan level & repo level gets disabled automatically. The user can be asked if he wants to proceed or something like that before this happens.

Then to define a flow one can just use the operations that are available in restic and the hooks defined at the plan and the repo level and then drag them into a sequence you like.

I think this could be done by creating little GUI elements from the available operations (snapshot, prune... plan hook1, plan hook2, repo hook 1 ...) that can be connected via "ON SUCCESS" and "ON FAILURE" like already implemented in other parts...

Something along those lines

Apart from all of that, wouldnt it be possible relatively easy to create some hook that gives commands from repo level
and that can have a condition like "ON SUCCESSFUL SNAPSHOT" as the other ones here?