Feature Request: run autopkgtest tests post-build

[hlieberman]

In the same vein as #9, it would be super awesome to run the autopkgtest suite after the build. (Probably using the autopkgtest-virt-null backend, since we're already inside an ephemeral container, so adding another layer of containerization is probably not worth it).

[ximion]

This is definitely on the roadmap and a thing I wanted to do for quite a while :-)
It hasn't been a high priority yet though (and I haven't looked into what is required to properly support autopkgtest - when it's added, we would highly likely want to integrate this with Laniakea and have the tests run regularly on e.g. PureOS packages via debspawn. But even without that, being able to run it locally for regular packaging would be very neat).

[hlieberman]

So, I've been looking into this a bit tonight, and I have a patch that will make it work (at least on the individual package level). There's one pretty substantial caveat though. Because it's using autopkgtest-virt-null, autopkgtest makes an assumption that the tests have VM level isolation, rather than the container level isolation that they actually do. This means some tests will either flake, fail, or potentially cause undefined behavior that could affect the host. The only way to work around that, looking at the code, is to patch autopkgtest itself, or use one of the other backends. I'm going to see if the autopkgtest upstream would be willing to take a patch that does that. Alternately, there's a years-old bug request to use systemd-nspawn inside autopkgtest ( https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809443) that we could try and fix to simply run an nspawn container inside the nspawn container. I'm hesitant to push up the code that does this because I'm not sure what the ramifications are fully if you run this on arbitrary packages, and I really don't want to accidentally brick someone's box.

…

On Sun, Jul 5, 2020 at 7:33 PM Matthias Klumpp ***@***.***> wrote: This is definitely on the roadmap and a thing I wanted to do for quite a while :-) It hasn't been a high priority yet though (and I haven't looked into what is required to properly support autopkgtest - when it's added, we would highly likely want to integrate this with Laniakea and have the tests run regularly on e.g. PureOS packages via debspawn. But even without that, being able to run it locally for regular packaging would be very neat). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFNWXPFNKWKSGBPXJGR63LR2EEVNANCNFSM4OQR4FVQ> .

-- Harlan Lieberman-Berg ~hlieberman

[ximion]

Because it's using
autopkgtest-virt-null, autopkgtest makes an assumption that the tests have
VM level isolation, rather than the container level isolation that they
actually do. This means some tests will either flake, fail, or potentially
cause undefined behavior that could affect the host.

That pretty much was the reason why I didn't touch this yet. But I think it may be possible to detect whether a certain test requires VM-level isolation and then skip it when run inside of the container (if that's not possible, I think such a feature would be useful in autopkgtest).