`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.
As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.
This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
This patch:
- updates Flakiness Dashboard format to define version per-build
and to pass COMMIT information
- drops the README.md generation - we'll move on to a designated flakiness
dashboard viewer
- fix `FLAKINESS_DASHBOARD_BUILD_URL` to point to a task instead of a build
- do not pretty-print `dashboard.json` when serializing flakiness results
- filter out 'COVERAGE' test(s) so that they don't add up to `dashboard.json` payload. These are useless
- validate certain important options of flakiness dashboard
- more logging to STDOUT to actually say which repo and what branch is getting used
- enhance commit message with a build URL
- use a more compact format for JSON. For 100 runs of 700 tests it yields 21MB json instead of 23MB.
- bump default builds number to 100
This patch introduces a dashboard that records test results and
uploads them to https://github.com/aslushnikov/puppeteer-flakiness-dashboard
Since many bots might push results in parallel, each bot pushes
results to its own git branch.
FlakinessDashboard also generates a simple README.md with a flakiness
summary. If this proves to be not enough, we can build a website that
fetches flakiness data and renders it nicely.
This patch teaches TestRunner to support async suite
descriptions. This is needed to require tests using ES6 dynamic
imports:
```js
const t = new TestRunner();
await t.describe('tests', async () => {
(await import('./some.spec.js')).addTests(t);
(await import('./other.spec.js')).addTests(t);
});
```
This patch adds new TestRunner options:
- `disableTimeoutWhenInspectorIsEnabled` - disable test timeout if
testrunner detects enabled inspector.
- `breakOnFailure` - if testrunner should terminate test running on
first test failure
This patch improves the logic for test runner termination.
With this patch:
- TestRunner runs all afterEach/afterAll hooks when a
termination happens, properly terminating browser instances
- TestRunner cleans up all dangling timeout timers so that node.js
process is not retained and is free to exit
These getters are introduced as a more convenient substitute for
a `require('puppeteer/Errors')` and
`require('puppeteer/DeviceDescriptors')`.
This way we can make cross-browser story nicer - a single require
of `puppeteer` or `puppeteer-firefox` fully defines Puppeteer
environment.
This patch makes sure header overrides in request interception are
functioning as expected.
Drive-by: teach test server to use utf-8 charset header for text files.
This patch:
- implements Response.buffer() and other methods
- splits out relevant tests into a separate test suites
- implements `testServer.enableGzip()` method to optionally gzip
certain routes in tests
- adds tests to make sure `Response.text()` returns expected results
for binary and compressed responses.
This patch:
* unifies assets between tests
* enables a few puppeteer tests on Puppeteer-Firefox
Drive-by: beautify failing output of `expect.toEqual` matcher.
References #3889
This patch:
- introduces new testRunner methods `addTestDSL` and `addSuiteDSL`
to add annotated test / suite.
- introduces new test/suite declaration methods: `it_fails_ffox` and
`describe_fails_ffox`. These are equal to `it`/`describe` for chromium
tests and to `xit`/`xdescribe` for firefox.
- marks all unsupported tests with `it_fails_ffox`
- adds a new command-line flag `'--firefox-status'` to `//test/test.js`.
This flag dumps current amount of tests that are intentionally skipped
for Firefox.
End goal: get rid of all `it_fails_ffox` and `describe_fails_ffox`
tests.
Drive-By: remove cookie tests "afterEach" hook that was removing
cookies - it's not needed any more since every test is run in a
designated browser context.
References #3889
Introduce `//lib/api.js` that declares a list of publicly exposed
classes.
The `//lib/api.js` list superceedes dynamic `helper.tracePublicAPI()` calls
and is used in the following places:
- [ASYNC STACKS]: generate "async stacks" for publicy exposed API in `//index.js`
- [COVERAGE]: move coverage support from `//lib/helper` to `//test/utils`
- [DOCLINT]: get rid of 'exluded classes' hardcoded list
This will help us to re-use our coverage and doclint infrastructure
for Puppeteer-Firefox.
Drive-By: it turns out we didn't run coverage for `SecurityDetails`
class, so we lack coverage for a few methods there. These are excluded
for now, sanity tests will be added in a follow-up.
This patch splits out `IsolatedWorld` class from Frame.
The `IsolatedWorld` abstraction is an execution context
with a designated set of DOM wrappers.
References #2671
This patch teaches `page.setContent` to await resources in
the new document.
**NOTE**: This patch changes behavior: currently, `page.setContent`
awaits the `"domcontentloaded"` event; with this patch, we can now await
other lifecycle events, and switched default to the `"load"` event.
The change is justified since current behavior made `page.setContent`
unusable for its main designated usecases, pushing our client
to use [dataURL workaround](https://github.com/GoogleChrome/puppeteer/issues/728#issuecomment-334301491).
Fixes#728
This patch adds a new utility - `utils/bisect.js` - that accepts
a range of Chromium revisions and a pptr script and bisects the
range to figure when the script breaks.
The Puppeteer Script, given to the tool, should be exiting
with non-zero code to signify malfunctioning.
Example:
```
$ node utils/bisect.js --good 577361 --bad 599821 a.js
```
Also, new Chrome now exposes a new type in its protocol - binary.
It becomes a raw C++ array once used through C++ bindings, but for
us it's still a base64 string.
This adds `page.accessibility.snapshot()`. It serializes and returns the accessibility tree for the page. By default, uninteresting nodes are filtered out of the snapshot.
fixes#2033
This patch:
- adds experimental "transport" option to pptr.connect
- uses "transport" option to make sure Puppeteer-Web works with
Target.exposeDevToolsProtocol
Drive-by: add `browser.target()` to access browser target.
This patch:
- adds "browser" field to the package.json with default
bundling options.
- introduces "bundle" and "unit-bundle" commands to
create bundle and test bundle
- starts running bundle tests on Travis Node 8 bots
Fixes#2374.
Currently connection assumes that transport is a websocket
and tries to handle websocket-related errors.
This patch:
- moves ConnectionTransport interface to use callbacks instead
of events. This way it could be used in browser context as well.
- introduces WebSocketTransport that implements ConnectionTransport
interface for ws.
This is a preparation step for 2 things:
- exposing `transport` option in the `puppeteer.connect` method
- better support for `browserify`
References #2119
When we merge commits to master, Travis kicks job to build a new commit
and to publish new version of puppeteer@next.
If two commits are landed in almost the same time, then travis starts
two parallel jobs to build each commit. This race condition results
in the incorrect puppeteer@next revision.
This patch teaches apply_next_version.js to verify if current HEAD
is matching upstream HEAD. If it doesn't, the predeploy hook fails
which (hopefully) aborts deployment.
Fixes#2925.
One of our checks makes sure all links from README.md to API.md
point to the last-released version of the API.
This sometimes doesn't work: when we refer to a section
in api.md that is just added, we should be able to reference
the "master" version of the api.md
This patch:
- teaches the doclint check to keep links to tip-of-tree version
of api.md in README.md intact.
- starts refering to tip-of-tree version of api.md in `puppeter-core` section
This patch adds a new require, `puppeteer/Errors`, that
holds all the Puppeteer-specific error classes.
Currently, the only custom error class we use is `TimeoutError`. We'll
expand in future with `CrashError` and some others.
Fixes#1694.
This patch:
- updates `utils/fetch_devices.js` script to format UAs for Chrome UAs
and to add iPhone 6/7/8 as separate devices.
- re-generates `DeviceDescriptors.js` with the new script
Fixes#2730.
Since Node 10, `console.assert` no longer throws an AssertionError.
(This is generally good since it aligns Node.js with Browsers.)
This patch migrates all usages of `console.assert` in our codebase.
- All the `lib/` and testing code is migrated onto a handmade `assert`
function. This is to make Puppeteer transpilation / bundling easier.
- All the tooling is switched to use Node's `assert` module.
Fixes#2547.
This patch drops the markdown-toc module and instead rolls out
our own simple markdown table-of-contents generator.
As a side effect, it fixes links to `page.$` and `page.$$`.
This patch allows logging the output of the Chromium process to be enabled in tests by passing in the environment variable `DUMPIO=true`.
Additionally, the `stderr` of the Chromium process will always be logged in the the "Output" section of failing page tests.
Scrollbars look different on different platforms, so must be made invisible in tests. As a drive-by, xdescribe was broken with the new test runner.
References #2524
Previously protocol.d.ts was generated on `npm run tsc`. This was inconvenient because it meant that vscode checking was wrong until type checking was run manually, and was inefficient because it necessarily regenerated the types even if no new Chromium was downloaded. This patch generates the types when npm install is run from the github checkout, assuming a new Chromium revision was downloaded.
SourceFactory was meant to cache Sources so that they could be used
in different preprocessor tasks.
This turned out to be over-engineering. This patch kills the layer.
Last release v1.3.0 had an error in the documentation, claiming
it wasn't released.
This patch makes sure we have a little bit of automation in place
to save us from this in future.
This uses the `/json/protocol` endpoint to generate type definitions for the protocol.
Currently it is lacking protocol events and commands, but I will add those later.
This patch introduces `BrowserFetcher` class that manages
downloaded versions of products.
This patch:
- shapes Downloader API to be minimal yet usable for our needs. This
includes removing such methods as `Downloader.supportedPlatforms` and
`Downloader.defaultRevision`.
- makes most of the fs-related methods in Downloader async. The only
exception is the `Downloader.revisionInfo`: it has stay sync due to the
`pptr.executablePath()` method being sync.
- updates `install.js` and `utils/check_availability.js` to use new API
- finally, renames `Downloader` into `BrowserFetcher`
Fixes#1748.
feat: expose raw devtools protocol connection
This patch introduces `target.createCDPSession` method that
allows directly communicating with the target over the
Chrome DevTools Protocol.
Fixes#31.
This patch adds two new methods to the `page.coverage` namespace:
- `page.coverage.startCSSCoverage()` - to initiate css coverage
- `page.coverage.stopCSSCoverage()` - to stop css coverage
The coverage format is consistent with the JavaScript coverage.
This patch introduces a new `page.coverage` namespace with two methods:
- `page.coverage.startJSCoverage` to initiate JavaScript coverage
recording
- `page.coverage.stopJSCoverage` to stop JavaScript coverage and get
results
This patch migrates tests so that they can be run concurrently.
- By default, tests still run in one thread.
- To run tests in 4 parallel threads, run node test/test.js -j 4 or npm run unit -- -j 4
- Environment variable PPTR_PARALLEL_TESTS could be set to override default parallelization
Every test gets passed in a state. State is set up in the beforeAll and beforeEach callbacks,
and should be teared down in the afterAll and afterEach callbacks.
By default, state has a parallelIndex variable initialized that defines the thread index that runs the execution.
This patch unifies node6 transpilation:
- instead of generating multiple top-level directories, prefixed with
`node6-`, all transpiled code gets placed under single `node6/` folder
- transpilation doesn't change require paths of transpiled modules any
more
This patch:
- renames ChromiumDownloader into just Downloader (this is in
preparation for different products download)
- moves Downloader from utils/ to lib/. This unifies all of the
production-critical code in the lib/.
Drive-by: make Downloader a regular class.
This patch introduces a tiny test runner to run puppeteer tests.
The test runner is self-container and allows parallel (wrt IO) test execution.
It will also allow us to split tests into multiple files if necessary.
Comparing to the jasmine, the testrunner supports parallel execution, properly
handles "unhandled promise rejection" event and signals.
Comparing to ava/jest, the testrunner doesn't run multiple node processes,
which makes it simpler but sufficient for our goals.
In Blink, frames don't necesserily have execution context all the time.
DevTools Protocol precisely reports this situation, which results in
Puppeteer's frame.executionContext() being null occasionally.
However, from puppeteer point of view every frame will have at least a
default executions context, sooner or later:
- frame's execution context might be created naturally to run frame's
javascript
- if frame has no javascript, devtools protocol will issue execution
context creation
This patch builds up on this assumption and makes frame.executionContext()
to be a promise.
As a result, all the evaluations await for the execution context to be created first.
Fixes#827, #1325
BREAKING CHANGE: this patch changes frame.executionContext() method to return a promise.
To migrate onto a new behavior, await the context first before using it.
