Expose API for updating the crawl scope + exclusions of a running crawl

[ikreymer]

Thinking perhaps the most flexible way to do this is to update the crawl template in place, and then restart the crawl?
This would avoid the need for a custom update API on browsertrix crawler, for now..

The way this would work:

• Update the crawl template (via k8s config map)
• Send SIGINT to all the crawlers, causing to finish current page gracefully, and then restart
• restarted crawlers now using updated config.

This could allow for potentially more flexible updates beyond exclusions, and ensure the crawler is shutdown and restarted in a graceful way..

Downside:

• need to restart container, but should be fairly quick.
• Possibly more tricky to support in non-k8s setup, but those will only be for dev-only..

[edsu]

I think this makes sense. I guess the history of the crawl having started with a particular configuration is being preserved in some way in browsertrix cloud? I think it would be useful to know the options that went into the construction of a web archive, which seems related to webrecorder/specs#127 ?

[ikreymer]

Turns out it was simpler to follow the same approach as updating crawl configs: a new crawl config is created, and old one is deactivated (if first crawl) or deactivated (if other crawl attempts were made).

Then, the crawl statefulset is updated to point to the new crawl config, which causes k8s to gracefully restart the crawler pods at the end of the current page, exactly the desired behavior, w/o any extra signaling.

The crawl queue filtering is only part that's a little bit tricky: have to remove JSON block strings by value of the Redis list, as there's no other atomic way to do it (can't remove by index in list, as the queue may get updated).

The new API is: /api/archives/{aid}/crawls/{crawl_id}/addExclusion?regex=... to keep it simple, however, could of course also extend support to existing update crawl config approach.

[SuaYoo]

Implemented here: #347

[ikreymer]

To support also remove exclusions, changing API to:

POST /api/archives/{aid}/crawls/{crawl_id}/exclusions?regex=...

DELETE /api/archives/{aid}/crawls/{crawl_id}/exclusions?regex=...