puppeteer/utils/testrunner
Andrey Lushnikov f753ec6b04
chore(testrunner): distinguish between TERMINATED and CRASHED (#4821)
`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
  VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
  some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.

As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.

This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
2019-08-08 15:15:09 -07:00
..
examples chore: testrunner's ".not" should print appropriate message (#2459) 2018-04-26 11:13:22 -07:00
test chore(testrunner): distinguish between TERMINATED and CRASHED (#4821) 2019-08-08 15:15:09 -07:00
.npmignore chore: add .npmignore for testrunner (#3290) 2018-09-21 17:27:10 -07:00
index.js tests: drop jasmine test runner (#1519) 2017-12-07 16:37:22 -08:00
LICENSE chore: prepare testrunner to be published to npm (#3289) 2018-09-21 14:51:22 -07:00
Matchers.js chore(testrunner): newline for toEqual matcher (#3959) 2019-02-08 16:18:45 -08:00
Multimap.js tests: drop jasmine test runner (#1519) 2017-12-07 16:37:22 -08:00
package.json chore(testrunner): introduce tests for TestRunner (#4773) 2019-07-30 13:19:12 -07:00
README.md chore(testrunner): fix readme description (#3293) 2018-09-21 20:44:43 -07:00
Reporter.js chore(testrunner): distinguish between TERMINATED and CRASHED (#4821) 2019-08-08 15:15:09 -07:00
TestRunner.js chore(testrunner): distinguish between TERMINATED and CRASHED (#4821) 2019-08-08 15:15:09 -07:00

TestRunner

This test runner is used internally by Puppeteer to test Puppeteer itself.

  • testrunner is a library: tests are node.js scripts
  • parallel wrt IO operations
  • supports async/await
  • modular
  • well-isolated state per execution thread

Installation

npm install --save-dev @pptr/testrunner

Example

Save the following as test.js and run using node:

node test.js
const {TestRunner, Reporter, Matchers} = require('@pptr/testrunner');

// Runner holds and runs all the tests
const runner = new TestRunner({
  parallel: 2, // run 2 parallel threads
  timeout: 1000, // setup timeout of 1 second per test
});
// Simple expect-like matchers
const {expect} = new Matchers();

// Extract jasmine-like DSL into the global namespace
const {describe, xdescribe, fdescribe} = runner;
const {it, fit, xit} = runner;
const {beforeAll, beforeEach, afterAll, afterEach} = runner;

// Test hooks can be async.
beforeAll(async state => {
  state.parallelIndex; // either 0 or 1 in this example, depending on the executing thread
  state.foo = 'bar'; // set state for every test
});

describe('math', () => {
  it('to be sane', async (state, test) => {
    state.parallelIndex; // Very first test will always be ran by the 0's thread
    state.foo; // this will be 'bar'
    expect(2 + 2).toBe(4);
  });
});

// Reporter subscribes to TestRunner events and displays information in terminal
const reporter = new Reporter(runner);

// Run all tests.
runner.run();