Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instruction counter to benches #559

Closed
wants to merge 1 commit into from

Conversation

rukai
Copy link
Member

@rukai rukai commented Mar 8, 2022

Unfortunately it seems unusable at the moment.
I am getting instruction counts of ~7000 per message sent which sounds way too low. It must be failing to count instructions that occur in other threads.
I raised jbreitbart/criterion-perf-events#19

I'll leave this around as a draft in case we want to refer back to it.

@rukai rukai force-pushed the instruction_counting_benches branch from 84d1b74 to cd5f5c4 Compare March 8, 2022 02:20
@rukai
Copy link
Member Author

rukai commented Mar 8, 2022

I was suggested to use the inherit() method on the builder which certainly improves things.
I'm now getting values around ~200,000.
But there are still some strange results such as passthrough_tls_select being 87,000 while passthrough_select is 198,000.

So looks like this might be doable but will require some investigation.

@rukai rukai force-pushed the instruction_counting_benches branch from cd5f5c4 to 568923c Compare March 8, 2022 09:48
@rukai rukai closed this Mar 10, 2022
@rukai rukai deleted the instruction_counting_benches branch March 10, 2022 05:43
@rukai rukai restored the instruction_counting_benches branch March 10, 2022 05:45
@rukai rukai reopened this Mar 10, 2022
@rukai rukai force-pushed the instruction_counting_benches branch from 568923c to 221214e Compare March 10, 2022 06:02
@rukai
Copy link
Member Author

rukai commented Mar 21, 2022

actions/runner-images#4974
https://github.community/t/getting-access-to-perf-events-on-github-hosted-ubuntu-latest/149164/4
https://stackoverflow.com/questions/60988282/perf-alternate-performance-management-tools-that-can-run-on-google-cloud-platf

I found these which show that getting instruction count on GA VMs will be impossible.
The only solution would be providing our own hardware or maybe there is some cloud provider that supports it.

@benbromhead
Copy link
Member

This will probably be something that folks can run locally on their own machines and not worth making part of CI. I would see generaly perf testing we do elsewhere as a "good enough" smell vs instruction count.

@rukai
Copy link
Member Author

rukai commented Feb 2, 2023

There could be some value in instruction counting locally, but the main goal was to make CI bench results stable.
benchmarking on a local machine should be stable enough that instruction counts arent as useful.

I'm going to close this, our benchmarks need a lot of infrastructure improvements but we'll have to go a different direction.

@rukai rukai closed this Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants