`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.
As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.
This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
Adds note about Jest maxWorkers as well as the base image to start with. This is an improvement over the previous section I wrote, from me banging my head against a YAML file all week 🙃.
This came from personal difficulties in running Puppeteer tests on CircleCI. I tried to keep the note as brief as possible, while being helpful for an entire CI platform.
This patch:
- updates Flakiness Dashboard format to define version per-build
and to pass COMMIT information
- drops the README.md generation - we'll move on to a designated flakiness
dashboard viewer
We used to track API Coverage for public events, but this was regressed in the refactoring that
introduced `//lib/Events.js`.
This patch:
- Brings back API Coverage for events
- Combines all coverage-generated tests into a single one. This way
we can generate less data for flakiness dashboard.
- fix `FLAKINESS_DASHBOARD_BUILD_URL` to point to a task instead of a build
- do not pretty-print `dashboard.json` when serializing flakiness results
- filter out 'COVERAGE' test(s) so that they don't add up to `dashboard.json` payload. These are useless
- validate certain important options of flakiness dashboard
- more logging to STDOUT to actually say which repo and what branch is getting used
- enhance commit message with a build URL
- use a more compact format for JSON. For 100 runs of 700 tests it yields 21MB json instead of 23MB.
- bump default builds number to 100
Make eval test resilient to variations in error message format between browsers.This will make the test pass without alternations in WebKit as well as Crhrome and Firefox.
This patch introduces a dashboard that records test results and
uploads them to https://github.com/aslushnikov/puppeteer-flakiness-dashboard
Since many bots might push results in parallel, each bot pushes
results to its own git branch.
FlakinessDashboard also generates a simple README.md with a flakiness
summary. If this proves to be not enough, we can build a website that
fetches flakiness data and renders it nicely.
This patch introduces a page.waitForFileChooser() method
that adds a watchdog to wait for file chooser dialogs.
This lets Puppeteer users to capture file chooser requests
and fulfill/cancel them if necessary.
Fixes#2946
Don't ignore stdout and stderr when using pipe for remote debugging and dumpio is true. In that case puppeteer process connects to the stdout/stderr streams of the child process and it will not hang.
This patch teaches TestRunner to support async suite
descriptions. This is needed to require tests using ES6 dynamic
imports:
```js
const t = new TestRunner();
await t.describe('tests', async () => {
(await import('./some.spec.js')).addTests(t);
(await import('./other.spec.js')).addTests(t);
});
```
This patch adds new TestRunner options:
- `disableTimeoutWhenInspectorIsEnabled` - disable test timeout if
testrunner detects enabled inspector.
- `breakOnFailure` - if testrunner should terminate test running on
first test failure
This patch improves the logic for test runner termination.
With this patch:
- TestRunner runs all afterEach/afterAll hooks when a
termination happens, properly terminating browser instances
- TestRunner cleans up all dangling timeout timers so that node.js
process is not retained and is free to exit
Chrome has a set of component extensions - e.g. CryptoTokenExtension
that helps with 2FA.
These extensions are loaded regardless of the `--disable-extensions`
flag we already pass. To disable these extensions, we need to pass additional
`--disable-component-extensions-with-background-pages` flag.
Fix#4300
Check message prefix rather than strict equality when detecting circular JSON error. The message format has changed in Node 12 which broke the condition and failed a test.
I got a KaiOS phone with some awesome buttons on it I hadn't seen before on the web. This adds the following buttons to our default keyboard layout:
`SoftLeft`
`SoftRight`
`Camera`
`Call`
`EndCall`
`VolumeUp`
`VolumeDown`
