Create a proof-of-concept self-hosted runner for Github Actions

[tstellar]

We should create a proof-of-concept self-hosted runner for Github Actions and enable it for some of the existing CI jobs in the release branch. Once this is working, we should document how to do this, so that community members can easily create their own self-hosted runners.

[lazyparser]

I'm just curious that how many workers/bots are sufficient for LLVM project. I had built a few internal CI for LLVM. It taks ~15mins for clean build and ~10mins to run the regression tests on 52 core Xeon 2.5GHz / 200GB RAM VM.

[ChristianKuehnel]

I guess that mostly depends on which and how many configurations we want to build...

…

On Fri, Aug 20, 2021 at 9:31 AM Wei Wu ***@***.***> wrote: I'm just curious that how many workers/bots are sufficient for LLVM project. I had built a few internal CI for LLVM. It taks ~15mins for clean build and ~10mins to run the regression tests on 52 core Xeon 2.5GHz / 200GB RAM VM. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYJMCAWKNQPVHIUKZMKNBTT5YAERANCNFSM5CPTBHMA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[lazyparser]

So I've create two type of runners for a downstream llvm repo.

Turns out the runner for github actions (or post commit runners) are kinda easy to set up: I just followed the github docs and the commands showed in the setting tab of the repo.

CI for pull requests are a bit complicated. I forked the runner that the riscv v8 team used, and set up a CI runner for pull requests successfully. the repo is here. It written in JavaScript, and the source code is kinda straight forward. feel free to check it out.

[tstellar]

So I've create two type of runners for a downstream llvm repo.

Turns out the runner for github actions (or post commit runners) are kinda easy to set up: I just followed the github docs and the commands showed in the setting tab of the repo.

Are you interested in using your runner for some release branch testing?

[tstellar]

There is some interested in setting up a self-hosted runner on Mac hardware, so we are going to request Mac resources from the LLVM Foundation.

[lazyparser]

So I've create two type of runners for a downstream llvm repo.
Turns out the runner for github actions (or post commit runners) are kinda easy to set up: I just followed the github docs and the commands showed in the setting tab of the repo.

Are you interested in using your runner for some release branch testing?

Yes. Will do. I can set up 2~3 runners in next week.

[lazyparser]

There is some interested in setting up a self-hosted runner on Mac hardware, so we are going to request Mac resources from the LLVM Foundation.

I have two mac mini for (RISC-V) CI purpose. turned out that neither the x86 mini or M1 mini with the 8GB memory could not build LLVM quickly enough. for pre or post commit CI. IMHO the iMac or Mac Pro might suit.

If we just want to a nightly build, then it would be ok to use mac mini as a start.

[tstellar]

@lazyparser The Github runners take 2hr 20min to build and run make check for llvm (and no other sub-projects). Anything that is faster than that would be an improvement.

[lazyparser]

@lazyparser The Github runners take 2hr 20min to build and run make check for llvm (and no other sub-projects). Anything that is faster than that would be an improvement.

It takes 7-10 minutes for building (clang+llvm on x86 and RISCV) and 8-12 minutes for regression testing on the 128c200g VM. For a 32c128g VM it takes 30-35 minutes.

These workers(VMs) are idle most of the time, because my team had turned to upstream directly. I'd like to use the VMs for LLVM building. Either adding IWG members to root access or config actions on the VMs myself are good to me.

btw thete is yet another daily CI for LLVM created by me:
https://ci.rvperf.org/job/upstream-llvm-main-branch/

[lazyparser]

Not sure If one project can has multiple build workers. If so, then perhaps I can connect 2 big workers and a few medium workers to github actions.

[lazyparser]

So what should to be done next?

[tstellar]

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

[lazyparser]

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

Yes. For the action, just copy the command list on the action tab (Please send it in private email or other non-public channel).

If we want to use self-built runner for PRs, then

1. a webhook need to be added.
2. a github token that can access and comment on the repo should be created.

We can start from the actions.

[tstellar]

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

Yes. For the action, just copy the command list on the action tab (Please send it in private email or other non-public channel).

I don't understand what this is, can you clarify?

[lazyparser]

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

Yes. For the action, just copy the command list on the action tab (Please send it in private email or other non-public channel).

I don't understand what this is, can you clarify?

If we want to add a new runner for github actions, we can follow

https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository

For example:

Press the add new self-runner button, then

You can see several hash string in the command script. That is the secrets/token for the runner which should be keep private.

The other way is that I can provide an account on the runner VM, so that you (or other maintainer) can log into the runner VM and run the commands directly.

[ChristianKuehnel]

meeting on 2021-10-26:

• How do we handle security around runners?
  • How do we track (audit trail) the actions triggered by the token used in a self-hosted runner.
  • Access with the runner token is not worse than people having commit access.
  • What requirements do we have for the folks owning a runner?
    • Proposal: Owners should have commit access.
  • What happens if a malicious commit snoops the token from the runner?
• Created follow-up issues to investigate further:
  • Security: Figure out what an attacker could do with a github runner token #76
  • Security: Figure out if there is an audit trail for actions triggered by a Github runner token #77