Skip to content

Add nightly benchmarks for remote caching & execution #1864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ayewo wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add nightly benchmarks for remote caching & execution #1864

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,20 @@
the GitHub issue that you want to work on it. This is to prevent duplicated
efforts from contributors on the same issue.

## Benchmarks

To catch performance regressions early in NativeLink's [RBE](https://bazel.build/remote/rbe) implementation, we maintain daily benchmarks in a separate repo:

Check failure on line 20 in CONTRIBUTING.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'repo'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'repo'?", "location": {"path": "CONTRIBUTING.md", "range": {"start": {"line": 20, "column": 153}}}, "severity": "ERROR"}
- [`nativelink-rbe-benchmarks`](https://github.com/ayewo/nativelink-rbe-benchmarks)

The use cases that are benchmarked include:
- build with remote cache only;
- build with remote cache and execution.

At the end of each daily run of the benchmarks, the results of each commit up to the most recent commit are analyzed by a GitHub Actions workflow using an open source [Change Point Detection](https://en.wikipedia.org/wiki/Change_point_detection) (CPD) tool, currently [Apache Otava](https://github.com/apache/otava).

Check failure on line 27 in CONTRIBUTING.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Otava'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Otava'?", "location": {"path": "CONTRIBUTING.md", "range": {"start": {"line": 27, "column": 277}}}, "severity": "ERROR"}

Another workflow triggers the publishing to GitHub Pages of an Astro static site visualizing the benchmark results:
- https://ayewo.github.io/nativelink-rbe-benchmarks/

## Git setup

NativeLink has a somewhat specific contribution process to ensure consistent
Expand Down
Loading