Add support for teardown handler to pedantic mode

[winpat]

First of all, thanks for this amazing package 🙏 , I use it in all of my projects when I need to benchmark a piece of code.

Sometimes benchmarks have side effects which need to be cleaned up after every round. For example if a benchmark writes to a file, then you might want to delete it in between rounds and start from a clean slate.

It's already possible to pass a setup function to pedantic mode, this PR introduces a similar mechanism but for cleaning up resources after a round has been competed by passing a cleanup function.

Here is an example on how it would look like when benchmarking a REST API client:

def test_api_create_resource(benchmark, api_client, resource):
    benchmark(
        api_client.create_resource, 
        args=(resource,), 
        cleanup=lambda: api_client.delete_resource(resource)
    )

Before I properly implement and polish it, I wanted to ask you @ionelmc whether this is something you would be interested in adding 😄.

[ionelmc]

Hello, yes. We can have a cleanup callback. What is this pr missing?

[winpat]

Hello, yes. We can have a cleanup callback. What is this pr missing?

Awesome, let me test the changes once more. I will try to get this MR into a mergable state in the next couple of days. The only thing missing from my point of view are docs.

[winpat]

There are couple of design tradeoffs in this MR:

1. It mirrors the behaviour of the setup function in that it only runs once per round.
2. The cleanup function receives the same arguments as the target function.

@ionelmc what do you think about this? Is this reasonable or should the cleanup handler be invoked on every iteration instead?

[ionelmc]

My opinion is that the cleanup should have parity with setup (so once per round). You have fine control over the number of iterations in a round in pedantic anyway.

[ionelmc]

This is pretty much ready right? I was thinking that maybe the option should be named "teardown" (eg: init goes with cleanup, setup goes with teardown). There should be a test that tests that cleanup is only run once regardless of how many iterations (and a warning somewhere in docs about that I guess).

[winpat]

This is pretty much ready right? I was thinking that maybe the option should be named "teardown" (eg: init goes with cleanup, setup goes with teardown). There should be a test that tests that cleanup is only run once regardless of how many iterations (and a warning somewhere in docs about that I guess).

Yes, 👍 . I will rename "cleanup" to "teardown", add the missing test/docs and fix the linting issues. This should be ready by tomorrow.

[winpat]

@ionelmc thanks for your feedback. I've addressed it.

The docs tox env is still failing in CI and I don't know why. Maybe you can give me a hint on what needs to be fixed?

Otherwise this MR is ready to be merged from my point of view.

[adiroiban]

There is a broken URL in the docs ... see https://github.com/ionelmc/pytest-benchmark/actions/runs/11596401651/job/32288391979?pr=264#step:5:212

(         authors: line    6) broken    http://marc-abramowitz.com/ - 500 Server Error: Internal Server Error for url: http://marc-abramowitz.com/

I don't think this is related to this PR ... maybe the remote server is down...and if you retry the job in a few hours, it will work

[ionelmc]

Ignore the docs, and that windows failure - it just takes too long to run.

I was wondering about one thing: would it be useful (and possible) to make the teardown take the result as the first argument? I would expect someone will come later on and complain that the result of the function is gone and he needs it for cleanup - then we'd need to make a backwards incompatible api change, something that I am not a fan of.

[winpat]

I think it would be useful. However it would likely require us to call the teardown function after every iteration. The problem is that the function being benchmark isn't required to be pure/side effect free. So it is allowed to return a different result on every invocation even if the arguments stay the same...

[ionelmc]

Ok that's a fair argument - only needs to be explained somewhere in the docs :-)