Build failure dataset script #4617
Conversation
There was a problem hiding this comment.
I like this "in general". Some specifics. I think collecting this data and storing it longer term is good. Can you store in .json artifacts- then we could ingest in the future? Maybe fixed_by_commit revision or bugid as the filename. Then these exist and can be ingested overtime.
The risk is that there are multiple backouts for a given bugid/original_changeset.
It would be a bonus if we could collect test regressions as well- it is ok if these are in a separate location/db/filesystem/etc. The primary use case I can see is to determine if we are finding regressions in code we ship, or just forgetting or dealing with old test cases. This is pretty easy by looking at the interdiff and determining which files were changed
| main() | ||
|
|
||
| # 0. Download databases | ||
| # 1. Identify bugs in Bugzilla that have a backout due to build failures X |
There was a problem hiding this comment.
I normally get my regression (backed out) data from treeherder database. The main reason why is we can get all the tasks that failed, which could help determine if this is failing ALL builds, or a certain platform or build failure. Not required, but the additional information is good to know.
| for key in ["node", "bug_id", "pushdate", "backedoutby", "backsout", "desc"] | ||
| } | ||
|
|
||
| bug_commits.setdefault(commit["bug_id"], []).append(commit_data) |
There was a problem hiding this comment.
how will you make this work when there are multiple landings and backouts? In general do you care about each transaction, or the end state? Sometimes the end state is a secondary patch that hot fixes the situation, or a rebase which changes other files/code blocks not originally edited.
I think any solution should account for >1 backout.
| and "for causing" in desc.lower() | ||
| and "build" in desc.lower() | ||
| ): | ||
| return commit |
There was a problem hiding this comment.
in this case why do you need bugzilla data?
There was a problem hiding this comment.
We use the Bugzilla entry to identify backed out commits. Is there a more accurate method that you may be aware of to identify backed out commits (aside from the Treeherder DB)?
There was a problem hiding this comment.
I thought commits via hg had metadata:
https://hg.mozilla.org/mozilla-central/rev/99a8f2b2b00d85148c743f16db75d9abefb33513
if you want a short list of commits that are backed out, then bugzilla comment parsing would do.
One value add method would be to propose a change to the backout process where there is queryable metadata so everytime there is a backout, data is stored in bugzilla/treeherder/some_random_db with original commit, backed out commit, related links, list of failing tasks and related error messages. Thinking more, developers relanding would have to go through a process to document what they were relanding, and confirm the interdiff :)
There was a problem hiding this comment.
my original comment here was assuming you were doing a massive hg log to look at commit history and parse out "backed out" commit messages to build a list.
| if not backing_out_commit: | ||
| continue | ||
|
|
||
| logger.info("Backing out commit found!") |
There was a problem hiding this comment.
it would be nice to mention the commit id :)
There was a problem hiding this comment.
I've included the commit information when identifying the backing out commit: 460e669
| failed_tasks.add(task["status"]["taskId"]) | ||
|
|
||
| # 6. find intersection between build tasks and failed tasks | ||
| failed_build_tasks = list(build_tasks & failed_tasks) |
There was a problem hiding this comment.
if you use treeherder db (search for fixed_by_commit) instead of bugzilla to find backouts you get most of this stuff for free.
| # continue | ||
|
|
||
| commit_diff = repository.get_diff( | ||
| repo_path="hg_dir", original_hash=commits[0], fix_hash=commits[1] |
There was a problem hiding this comment.
as many fixed revisions do not have the original code referenced, this might have more edge cases than you think. Also for multiple cycles, it is probably best to compare commits[0] <-> commits[-1]
(STILL A WORK IN PROGRESS)
Script to create the build failure dataset. Collects the Revision ID, Initial and Fix Commit Hash, Interdiff and Error Logs.