`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.
As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.
This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
This patch teaches TestRunner to support async suite
descriptions. This is needed to require tests using ES6 dynamic
imports:
```js
const t = new TestRunner();
await t.describe('tests', async () => {
(await import('./some.spec.js')).addTests(t);
(await import('./other.spec.js')).addTests(t);
});
```
This patch adds new TestRunner options:
- `disableTimeoutWhenInspectorIsEnabled` - disable test timeout if
testrunner detects enabled inspector.
- `breakOnFailure` - if testrunner should terminate test running on
first test failure
This patch improves the logic for test runner termination.
With this patch:
- TestRunner runs all afterEach/afterAll hooks when a
termination happens, properly terminating browser instances
- TestRunner cleans up all dangling timeout timers so that node.js
process is not retained and is free to exit
This patch:
- introduces new testRunner methods `addTestDSL` and `addSuiteDSL`
to add annotated test / suite.
- introduces new test/suite declaration methods: `it_fails_ffox` and
`describe_fails_ffox`. These are equal to `it`/`describe` for chromium
tests and to `xit`/`xdescribe` for firefox.
- marks all unsupported tests with `it_fails_ffox`
- adds a new command-line flag `'--firefox-status'` to `//test/test.js`.
This flag dumps current amount of tests that are intentionally skipped
for Firefox.
End goal: get rid of all `it_fails_ffox` and `describe_fails_ffox`
tests.
Drive-By: remove cookie tests "afterEach" hook that was removing
cookies - it's not needed any more since every test is run in a
designated browser context.
References #3889
Since Node 10, `console.assert` no longer throws an AssertionError.
(This is generally good since it aligns Node.js with Browsers.)
This patch migrates all usages of `console.assert` in our codebase.
- All the `lib/` and testing code is migrated onto a handmade `assert`
function. This is to make Puppeteer transpilation / bundling easier.
- All the tooling is switched to use Node's `assert` module.
Fixes#2547.
Scrollbars look different on different platforms, so must be made invisible in tests. As a drive-by, xdescribe was broken with the new test runner.
References #2524
This patch migrates tests so that they can be run concurrently.
- By default, tests still run in one thread.
- To run tests in 4 parallel threads, run node test/test.js -j 4 or npm run unit -- -j 4
- Environment variable PPTR_PARALLEL_TESTS could be set to override default parallelization
Every test gets passed in a state. State is set up in the beforeAll and beforeEach callbacks,
and should be teared down in the afterAll and afterEach callbacks.
By default, state has a parallelIndex variable initialized that defines the thread index that runs the execution.
This patch introduces a tiny test runner to run puppeteer tests.
The test runner is self-container and allows parallel (wrt IO) test execution.
It will also allow us to split tests into multiple files if necessary.
Comparing to the jasmine, the testrunner supports parallel execution, properly
handles "unhandled promise rejection" event and signals.
Comparing to ava/jest, the testrunner doesn't run multiple node processes,
which makes it simpler but sufficient for our goals.
