Post-merge jobs failing

[fitzthum]

I think the Azure e2e jobs have been failing ever since #689

Error:

./client: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./client)

cc @seungukshin @mkulke

[seungukshin]

It seems like client is built on ubuntu-24.04 with GLIBC-2.39. And the execution environment (azure-cvm-tdx and azure-cvm) use ubuntu-22.04 with GLIBC-2.35. Can we change the ubuntu version for the build environemnt?

trustee/.github/workflows/kbs-e2e.yml

Line 25 in f026344

runs-on: ubuntu-24.04

[mythi]

Can we change the ubuntu version for the build environemnt?

#687 moved to use 24.04 but it had been like that since ubuntu-latest had defaulted to it for some time. What's the workflow that builds the artifacts?

[seungukshin]

@mythi , kbs-e2e.yml's build-binaries job creates the artifact. It was runned on ubuntu-22.04 before this commit

[fitzthum]

Building the binaries is here

trustee/.github/workflows/kbs-e2e.yml

Line 25 in f026344

runs-on: ubuntu-24.04

And the test runs on this

trustee/.github/workflows/kbs-e2e-azure-vtpm.yml

Line 63 in f026344

runs-on: '["self-hosted","azure-cvm-tdx"]'

This was working fine before #689 tho which is odd. I looked through the history again and it does seem like that's when the error showed up (although we did have a different problem after #687)

[fitzthum]

I don't really see how this could be caused by #689. We build the binaries for the az tests from the Makefile, which should be totally unrelated to the Dockerfiles that we changed. So maybe we should just downgrade those builder images in the CI. wdyt @mkulke, @mythi

[mkulke]

github is rotating their -latest runners slowly to 24.04 I think, so that we probably would have to look at the logs closely to see which lts ubuntu the artifacts were built on.

I assume the self-hosted pool can be configured to use ubuntu 24.04 images, however it's not great that we break compatibility with LTS distros that are still widely used, so I would suggest to the fix the builder images to 22.04 for the time being.

[mythi]

So maybe we should just downgrade those builder images in the CI

I'd be OK with that but I'd still like to understand the core problem. Why make e2e-test does not retrigger a new build if test.tar.gz isn't reusable?

[mkulke]

I'd be OK with that but I'd still like to understand the core problem. Why make e2e-test does not retrigger a new build if test.tar.gz isn't reusable?

the binaries are not built on self-hosted runners. the self-hosted runners just provide a real TEE runtime environment for the tests. as said above, I would think that bumping the runtime env to 24.04 would probably help, but otoh it's not great that binaries built on the repo will not work on 22.04 runners.

Maybe we can also remove this e2e test suite? the workflows are set up to be generic to allow running on a bare-metal TDX or SNP instances, but so far, the only real TEEs that are tested remain Azure CVMs. If we don't plan to extend this, it might be sufficient to have tests on TEE hardware somewhere downstream.

[mythi]

Let me submit a PR that moves the e2e building back to 22.04. AFAUI, the default kbs-client build is statically linked since it does not enable vTPM attesters.