The Chromium Chronicle #14: Adding Tests to the Waterfall
Episode 14: by Zhaoyang Li in MTV, and Eric Aleshire in TOK (October, 2020)
Want to detect regressions for your new feature in Chrome? Add your tests to the waterfall (Chrome's continuous build and test infrastructure)!
There are many builders on Chrome's waterfall that run tests on a variety of platforms. This article describes how to add a test suite to an existing builder. Before proceeding, consider these questions:
Should the new tests live in a brand new suite, or just be added to an existing one?
- Tests are organized in test suites by proximity of source location and theme. If your new tests can't logically fit into any existing suite, you probably need a new suite.
Should the tests run on a public builder or an internal builder?
- Use an internal builder if the code lives in an internal repo, or the tests involve confidential data.
Should the tests run in FYI CI, main CI or commit queue(CQ)?
- FYI CI needs your self-monitoring and is used for test refinement or experimentation.
- Main CI tests are regularly monitored by sheriffs.
- CQ blocks CL submission at failure but takes more infra resources. A new suite should always start from CI before being promoted to CQ.
- If you're not sure, your platform's EngProd team can help you decide.
I already have a test suite running in CI, how do I add it to CQ? / What if I need a new builder?
- File a bug in Infra>Client>Chrome component so
chrome-browser-infra@team can start evaluations and help you set up.
How to add a test suite to an existing builder #
To add a test suite to an existing builder, you need to config some files in
Create a key in
gn_isolate_map.pylfor the new test suite with test target label and type info.
More fine tunings.
Regenerate configuration files by running
After these, it is a simple matter of checking in your config changes; the builders running this suite will pick up the new tests automatically, and the results will begin to flow in on the web interface for the builder on the waterfall - complete with plenty of debug info in the case of failures!