The `Puppeteer` class had two concerns:
* connect to an existing browser
* launch a new browser
The first of those concerns is needed in all environments, but the
second is only needed in Node.
https://github.com/puppeteer/puppeteer/pull/6484 landing enabled us to
pull the `Puppeteer` class apart into two:
1. `Puppeteer` which hosts the behaviour for connecting to existing
browsers.
2. `PuppeteerNode`, which extends `Puppeteer` and also adds the ability
to launch a new browser.
This is a non-breaking change, because Node users will still get an
instance of a class with all the methods they expect, but it'll be a
`PuppeteerNode` rather than `Puppeteer`. I don't expect this to cause
people any issues.
We also now have new files that are effectively the entry points for
Puppeteer:
* `node.ts`: the main entry point for Puppeteer on Node.
* `web.ts`: the main entry point for Puppeteer on the web.
* `node-puppeteer-core.ts`: for those using puppeteer-core (which only
exists in Node, not on the web).
The `Launcher` class was serving two purposes:
1. Launch browsers
2. Connect to browsers
Number 1) only needs to be done in Node land, but 2) is agnostic; in a
browser version of Puppeteer we'll need the ability to connect over a
websocket to send commands back and forth.
As part of the agnostification work we needed to split the `Launcher` up
so that the connection part can be made agnostic. Additionally, I
removed dependencies on `https`, `http` and `URL` from Node, instead
leaning on fetch (via `node-fetch` if in Node land) and the browser
`URL` API (which was added to Node in Node 10).
This commit also removes our own custom type for defining the vision
deficiencies and uses the protocol's type. Now we generate docs for
those we get the docs generated for free for these. This is better than
us duplicating values for types in doc comments and having them become
outdated. If we use the protocol types directly then we ensure we're up
to date and in-sync.
Long term the docs will also link to the devtools-protocol viewer.
This corresponds to Chromium 85.0.4182.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure
https://crrev.com/c/2231445
- [FlexNG] Enable FlexNG by default
https://crrev.com/c/2216595Closes#6151.
* chore: vendor Mitt into src/common/third-party
As discussed in #6203 we need to vendor our common dependencies in so
that when we ship an ESM build all imports point to file paths and do
not rely on Node resolution (e.g. a browser does not understand `import
mitt from 'mitt'`).
* feat(chromium): roll Chromium to r768783
* fix: update unit test for crrev:2135046
* chore: update devtools-protocol revision
Co-authored-by: Changhao Han <changhaohan@chromium.org>
* chore: Use devtools-protocol package
Rather than maintain our own protocol we can instead use the devtools-protocol package and pin it to the version of Chromium that Puppeteer is shipping with.
The only changes are naming changes between the bespoke protocol that Puppeteer created and the devtools-protocol one.
Now the async hooks helper is gone api.ts was only used by the coverage
tools and by doclint.
DocLint is nearing the end of its lifespan with the TSDoc work, so I
focused on how best to define a list of modules for the coverage
tooling. They define an object of classes, and the path to that module.
They need the full path because we also check if the module exports any
events that need to be emitted - the coverage tool asserts that the
emitting of those events is also tested.
It's not _great_ that DocLint relies on a constant defined in the
coverage utils, but it should only be this way for a short period of
time and no one is actively working on DocLint (bar the effort to remove
it) so I don't think this is worth worrying about.
This change also broke the DocLint tests; based on the fact that DocLint is on its way out it doesn't feel worth fixing the tests, so this commit also removes them.
* chore(docs): reduce warnings when generating docs
This is a bunch of small miscellaneous fixes that reduce the amount of
warnings logged when generating our new docs. The long term goal is to
get this list down to 0 warnings, but I'll do it in multiple PRs.
* satisfy doclint
This pulls in the types (based on the DefinitelyTyped repo) for
`page.$eval` (and the `$eval` method on other classes). The `$eval`
method is quite hard to type due to the way we wrap and unwrap
ElementHandles that are passed to / returned from the `pageFunction`
that users provide.
Longer term we can improve the types by providing type overloads as
DefinitelyTyped does but I've deferred that for now (see the `TODO` in
the code for more details).
* chore: Don't store revisions in `package.json`
It's quite messy to have to require the `package.json` file in multiple
places purely to find out what revision of a given browser we want to
use. We can also achieve better type safety by placing it in an actual
source file.
This commit makes that change and also tidies up our reliance on
`package.json` within the source code generally; we now only use it to
find the location of the Puppeteer root such that we know where to
install downloaded browsers to.
To avoid using `package.json` to parse the name of the module, we also
now explicitly have an entry point for the Puppeteer module and the
Puppeter Core module. This will make it easier in the future to ship
less code as part of core (e.g. core never needs to download a browser,
so why ship that code?). Core can also then not have any revisions based
info contained in it.
The test install script has also been updated to ensure that
puppeteer-core can be installed correctly too.
Finally, the `install` script has been moved to TypeScript for nicer
typechecking and safety. The functionality of it has not changed.
* chore(agnostic): ship CJS and ESM builds
For our work to enable Puppeteer in other environments (e.g. a browser)
we need to ship an ESM build. This commit changes our config to ship to
`lib/cjs` and `lib/esm` accordingly. The majority of our code stays the
same, with one small fix for the CJS build to ensure that we ship a
version that lets you `require('puppeteer')` rather than have to
`require('puppeteer').default`. We do this with the `cjs-entry.js` which
is what the `main` field in our `package.json` points to.
We also swap to `read-pkg-up` to find the `package.json` file. This is
because the folder structure of `lib/` does not match `src/` now we ship
to `cjs` and `esm`, so you cannot rely on exact paths. This module works
up from the file to find the nearest `package.json` so it will always
find Puppeteer's `package.json`.
Note that we *do not* point any users to the ESM build. We happen to
ship those files so people who know about them can get at them but it's
not expected (nor will we actively support) that people will rely on
them. The CommonJS build is considered our main build.
We may make breaking changes to the structure of the ESM build which we
will do without requiring new major versions. For example the ESM build
currently ships all files that the CJS build does, but given we are
working on the ESM build being able to run in the browser this may
change over time.
Long term once the Node versions catch up we can ditch CJS and ship
exclusively ESM but we are not there yet.
This CL migrates all the tests to TypeScript. The main benefits of this is that we start consuming our TypeScript definitions and therefore find errors in them. The act of migrating found some bugs in our definitions and now we can be sure to avoid them going forwards.
You'll notice the addition of some `TODO`s in the code; I didn't want this CL to get any bigger than it already is but I intend to follow those up once this lands. It's mostly figuring out how to extend the `expect` types with our `toBeGolden` helpers and some other slight confusions with types that the tests exposed.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
These files will be used by both the web and node versions of Puppeteer.
Another name for this might be "core" but I don't want to cause
confusion with the puppeteer-core package that we publish at the moment.
This is another step towards making Puppeteer agnostic of environment
and being able to run in Node or a browser.
The files in the `node` directory are ones that would only be needed in
the Node build - e.g. the code that downloads and launches a local
browser instance.
The long term vision here is to have three folders:
* node - Node only code
* web - Web only code
* common - code that is shared
But rather than do that in one PR I'm going to split it up to make it
easier to review and deal with.
Replacing the Node EventEmitter with Mitt caused more problems than
anticipated for end users due to the API differences and the amount of
people who relied on the EventEmitter API. In hindsight this clearly
should have been explored more and then released as a breaking v4.
This commit rolls us back to the built in Node EventEmitter library
which we can release to get everyone back on stable builds. We can then
consider our approach to migrating to Mitt and when we do do that we can
release it as a breaking change and properly document the migration
strategy and approach.
* chore: migrate to Mitt as the EventEmitter
This commit moves us to using Mitt [1] for the event emitter in
Puppeteer. This removes our dependency to Node's EventEmitter which is
part of a larger stream of work to enable a Puppeteer-web version that
doesn't depend on Node.
There are no large breaking changes as we support the main methods that
EventEmitter had, but it also provides some methods that Puppeteer
didn't use. Technically end users could depend on this but it's
unlikely.
[1]: https://github.com/developit/mitt
It conflicts with an inbuilt TypeScript `Request` type so can cause
confusion when in TS land. Note: `Response.ts` and `Worker.ts` also
suffer from this; PRs to rename them are incoming.
This script generated an `index.d.ts` file but that file was never
commited to git nor included in Puppeteer when we ship. As of right now
people who want TS types can install from the DefinitelyTyped repo and
we are working on shipping types from Puppeteer itself.
Therefore this script is not adding any value and can be removed.
We should import them just like any other module. This commit makes that
change. It does not change any behaviours or the types themselves.
EXPECTED_PROTOCOL_DIFF as we're updating the structure of it.
* chore: extract `BrowserRunner` into its own module
`src/Launcher.ts` is large and hard to work in. It has multiple objects
defined in it:
* ChromeLauncher
* FirefoxLauncher
* BrowserRunner
* Launcher
This change moves BrowserRunner into its own module. More refactorings
like this will follow but this is the first step.
* chore: remove src/externs.d.ts
It defined global types that we don't want to use, and instead we move
to using interfaces that we import and reference just like with any
other interface.
This means other than Protocol (which I think is fine to leave as is),
there are no other magic global types and you have to import any types
or interfaces that you want.
* chore: migrate src/Page.js to TypeScript
The final one! This is a huge file and needs to be split up and tidied,
but for now I've left all the definitions in place and converted types
accordingly.
There's some additional tidying we can do now every `src` file is TS,
but I'll leave that for another PR to avoid this one getting any bigger.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
* chore: enforce src/protocol.d.ts is in sync
On CI we run `npm run compare-protocol-d-ts` which checks that the file
on disk is up to date with the protocol we fetch from the browser.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
We don't support it and v3 shipped without including puppeteer-web in the browser. People are welcome to manually use Browserify to try to get Puppeteer running in a browser but it ultimately isn't our primary focus right now.
Getting puppeteer-core able to run in a browser is something we'll be looking at in the future so we'll revisit this soon.
* chore: migrate src/Input to typescript
This moves `Keyboard`, `Mouse` and `Touchscreen` to TypeScript. We gain
some nice TS benefits here; by creating a type for all the keycodes we
support we can type the input args as that rather than `string` which
will hopefully save some users some debugging once we ship our TS types
in a future version.
* Remove from externs file
* Update utils/doclint/check_public_api/index.js
Co-Authored-By: Mathias Bynens <mathias@qiwi.be>
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
* chore: migrate src/JSHandle to TS
There's a few TODOs in here that all depend on typing the
`ExecutionContext.evaluateHandle` properly so that you can properly
declare what types you're expecting back. Once I've done that file (it's
next on my list) I will loop back and improve the types here, fixing
these TODOs.
* Fix doclint for {}
The codebase was incredibly inconsistent with the use of spacing around
curly braces, e.g.:
```
// this?
const a = {b: 1}
// or?
const a = { b: 1 }
```
This extended into import statements also. Google's styleguide is no
spacing, so we're going with that.
* chore: migrate `src/Connection` to TypeScript
This commit migrates `src/Connection` to TypeScript. It also changes its
exports to be ESM because TypeScript's support for exporting values to
use as types via CommonJS is poor (by design) and so rather than battle
that it made more sense to migrate the file to ESM.
The good news is that TypeScript is still outputting to `lib/` as
CommonJS, so the fact that we author in ESM is actually not a breaking
change at all.
So going forwards we will:
* migrate TS files to use ESM for importing and exporting
* continue to output to `lib/` as CommonJS
* continue to use CommonJS requires when in a `src/*.js` file
I'd also like to split `Connection.ts` into two; I think the
`CDPSession` class belongs in its own file, but I will do that in
another PR to avoid this one becoming bigger than it already is.
I also turned off `@typescript-eslint/no-use-before-define` as I don't
think it was adding value and Puppeteer's codebase seems to have a style
of declaring helper functions at the bottom which is fine by me.
Finally, I updated the DocLint tool so it knows of expected method
mismatches. It was either that or come up with a smart way to support
TypeScript generics in DocLint and given we don't want to use DocLint
that much longer that didn't feel worth it.
* Fix params being required
This PR changes `src/Dialog.js` to `src/Dialog.ts` and rewrites
accordingly. Most of the changes are straight forward; the only
interesting one from a TS point of view is the `DialogType` enum. I
expose it again as `Dialog.Type` to avoid a breaking change.
This PR also exposed some bugs with our ESLint TypeScript settings and
applying the overrides, so I fixed those too.
I also updated our DocLint tool to work on TS source files over JS lib
files if they exist. This is the minimal change to keep the existing doc
system working as we're working on moving away from this system longer
term.
I lost some time debugging before realising that I needed to run tsc. I
don't really want to put `npm run tsc` before this command else we'll
run tsc multiple times on each CI build, so I think this message is
suitable.
Travis defines `process.env.TRAVIS` and if that exists we don't want to
log this as on CI we're guaranteed to have an up to date `lib/`
directory.
This commit moves `src/DeviceDescriptors` to be authored in TypeScript. This file was chosen due to its simplicity so that we can focus on getting a mixed JS/TS codebase playing nicely before migrating the more complex files.
The file itself was a bit odd: although the array of devices was exported via `module.exports` that was never referenced by any consumers; each device was also exported via `module.exports[name] = device` and that is how it's consumed. The Puppeteer docs suggest using it like so:
```js
puppeteer.devices['iPhone 6']
```
So instead of exporting the array and then setting a bunch of properties on that, we instead define the array and export an object of keys where each key is a device. This is a breaking change (see the footer for details).
Rather than export an object I'd much rather export a Map, but that would be a larger breaking change and I'm keen to avoid those for the time being.
Note that we have to use special TypeScript specific syntax for the export that enables it to work in a CommonJS codebase [1] but again I'd rather this than move to ESM at this time. TypeScript still outputs CommonJS into `lib/` as you would expect.
BREAKING CHANGE: We no longer export an array of devices, so any users relying on doing:
```js
puppeter.devices.forEach(...)
```
…will now see a breakage. The fix is to use `Object.{keys/entries/values}` to iterate instead.
[1]: https://www.typescriptlang.org/docs/handbook/modules.html#export--and-import--require
This commit updates all the non-Puppeteer unit tests to run using Mocha and then deletes the custom test runner framework from this repository. The documentation has also been updated.
Our logic around missing methods wasn't quite right; if there is no set of missing methods for a class it _is_ an error and we still need to report it, we don't want to `continue`.
This is expected as we now alias `emulateMedia` in `index.js` which isn't a file checked by DocLint. We alias there to avoid having the function overriden by the `asyncInstallHooks` code.
This commit updates doclint to know about methods that we expect it will find are missing and in that case just skip over them. We should only do this for methods where we plan to deprecate them or we have to define them in an odd way to work around some problem (and if that's the case long term we should fix that problem so we can define them as normal).
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
Rather than use our own custom expect library, we can use expect from npm [1], which has an API almost identical to the one Puppeteer has, but with more options, better diffing, and is used by many in the community as it's the default assertions library that comes with Jest.
It's also thoroughly documented [2].
[1]: https://www.npmjs.com/package/expect
[2]: https://jestjs.io/docs/en/expect
This updates our `tsconfig.json` so it emits our JavaScript files as
well as type checking them. We compile into `./lib` which we then ship
in our npm package. The source code has moved from `./lib` into `./src`.
Because the `src/` directory is exclusively JS files, this change is a
no-op in terms of code functionality but is the first step towards being
able to replace `src/X.js` with `src/X.ts` in a way that allows us to
migrate incrementally.
The `lib` directory is gitignored, and the `src` directory is
npmignored. On `npm publish` we will now run `npm run tsc` in order to
generate the outputted code.
TypeScript seems to struggle to understand `Promise.all` when the items in the array return different types. If we were authoring in TS we could fix this with TS generics (`Promise.all<OurTypeHere>(...)`) but for now we can typecast the result. We'll fix this properly when we author in TS.
TS 3.5 got much stricter on writing changes to objects with varied types [1] so we have to do a bit of typecasting work to convince TS about the types of keys and values that we are setting.
Longer term we should think about a better data structure that avoids us having to jump through some hoops but for now I think this is a reasonable step to get us onto 3.5.
Same story regarding bindings on `window`: the easiest fix is to cast `window` to `any` for the code that adds to it. I'm sure we can come up with a more type-safe way of doing this in the future.
[1]: https://github.com/Microsoft/TypeScript/wiki/Breaking-Changes#fixes-to-unsound-writes-to-indexed-access-types
* feat: Set which browser to launch via PUPPETEER_PRODUCT
This change introduces a PUPPETEER_PRODUCT environment
variable as a first step toward using Puppeteer with
many different browsers. Setting PUPPETEER_PRODUCT=firefox, for
example, enables Firefox-specific Launcher settings.
The state is also exposed as `puppeteer.product` in the API
to support adding other product-specific behaviour as needed.
The bulk of the change is a refactoring in Launcher
to decouple generic browser start-up from product-specific
configuration.
Respecting the puppeteer-core restriction for PUPPETEER_
environment variables, lazily instantiate the Launcher
based on a `product` Puppeteer.launch option, if available.
* test: Distinguish Juggler unit tests from Firefox
The funit script is renamed to fjunit (j for Juggler, which is
used only by the experimental puppeteer-firefox package.
In contrast, the funit script now refers to running Puppeteer
unit tests against the main puppeteer package with Firefox.
To do so with Firefox Nightly, run:
`BINARY=path/to/firefox npm run funit`
A number of changes in this patch make it easier to run
Puppeteer unit tests in Mozilla's CI.
Node.js v6 was end-of-life'd in April, 2019, with AWS Lambda prohibiting updaets to the Node.js v6 runtime since June 30, 2019.
This makes it quite safe for us to remove the Node 6 support from the repository.
`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.
As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.
This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
This patch:
- updates Flakiness Dashboard format to define version per-build
and to pass COMMIT information
- drops the README.md generation - we'll move on to a designated flakiness
dashboard viewer
- fix `FLAKINESS_DASHBOARD_BUILD_URL` to point to a task instead of a build
- do not pretty-print `dashboard.json` when serializing flakiness results
- filter out 'COVERAGE' test(s) so that they don't add up to `dashboard.json` payload. These are useless
- validate certain important options of flakiness dashboard
- more logging to STDOUT to actually say which repo and what branch is getting used
- enhance commit message with a build URL
- use a more compact format for JSON. For 100 runs of 700 tests it yields 21MB json instead of 23MB.
- bump default builds number to 100
This patch introduces a dashboard that records test results and
uploads them to https://github.com/aslushnikov/puppeteer-flakiness-dashboard
Since many bots might push results in parallel, each bot pushes
results to its own git branch.
FlakinessDashboard also generates a simple README.md with a flakiness
summary. If this proves to be not enough, we can build a website that
fetches flakiness data and renders it nicely.
This patch teaches TestRunner to support async suite
descriptions. This is needed to require tests using ES6 dynamic
imports:
```js
const t = new TestRunner();
await t.describe('tests', async () => {
(await import('./some.spec.js')).addTests(t);
(await import('./other.spec.js')).addTests(t);
});
```
This patch adds new TestRunner options:
- `disableTimeoutWhenInspectorIsEnabled` - disable test timeout if
testrunner detects enabled inspector.
- `breakOnFailure` - if testrunner should terminate test running on
first test failure
This patch improves the logic for test runner termination.
With this patch:
- TestRunner runs all afterEach/afterAll hooks when a
termination happens, properly terminating browser instances
- TestRunner cleans up all dangling timeout timers so that node.js
process is not retained and is free to exit
These getters are introduced as a more convenient substitute for
a `require('puppeteer/Errors')` and
`require('puppeteer/DeviceDescriptors')`.
This way we can make cross-browser story nicer - a single require
of `puppeteer` or `puppeteer-firefox` fully defines Puppeteer
environment.
This patch makes sure header overrides in request interception are
functioning as expected.
Drive-by: teach test server to use utf-8 charset header for text files.
