* chore(agnostic): ship CJS and ESM builds
For our work to enable Puppeteer in other environments (e.g. a browser)
we need to ship an ESM build. This commit changes our config to ship to
`lib/cjs` and `lib/esm` accordingly. The majority of our code stays the
same, with one small fix for the CJS build to ensure that we ship a
version that lets you `require('puppeteer')` rather than have to
`require('puppeteer').default`. We do this with the `cjs-entry.js` which
is what the `main` field in our `package.json` points to.
We also swap to `read-pkg-up` to find the `package.json` file. This is
because the folder structure of `lib/` does not match `src/` now we ship
to `cjs` and `esm`, so you cannot rely on exact paths. This module works
up from the file to find the nearest `package.json` so it will always
find Puppeteer's `package.json`.
Note that we *do not* point any users to the ESM build. We happen to
ship those files so people who know about them can get at them but it's
not expected (nor will we actively support) that people will rely on
them. The CommonJS build is considered our main build.
We may make breaking changes to the structure of the ESM build which we
will do without requiring new major versions. For example the ESM build
currently ships all files that the CJS build does, but given we are
working on the ESM build being able to run in the browser this may
change over time.
Long term once the Node versions catch up we can ditch CJS and ship
exclusively ESM but we are not there yet.
This CL migrates all the tests to TypeScript. The main benefits of this is that we start consuming our TypeScript definitions and therefore find errors in them. The act of migrating found some bugs in our definitions and now we can be sure to avoid them going forwards.
You'll notice the addition of some `TODO`s in the code; I didn't want this CL to get any bigger than it already is but I intend to follow those up once this lands. It's mostly figuring out how to extend the `expect` types with our `toBeGolden` helpers and some other slight confusions with types that the tests exposed.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
These files will be used by both the web and node versions of Puppeteer.
Another name for this might be "core" but I don't want to cause
confusion with the puppeteer-core package that we publish at the moment.
This is another step towards making Puppeteer agnostic of environment
and being able to run in Node or a browser.
The files in the `node` directory are ones that would only be needed in
the Node build - e.g. the code that downloads and launches a local
browser instance.
The long term vision here is to have three folders:
* node - Node only code
* web - Web only code
* common - code that is shared
But rather than do that in one PR I'm going to split it up to make it
easier to review and deal with.
Replacing the Node EventEmitter with Mitt caused more problems than
anticipated for end users due to the API differences and the amount of
people who relied on the EventEmitter API. In hindsight this clearly
should have been explored more and then released as a breaking v4.
This commit rolls us back to the built in Node EventEmitter library
which we can release to get everyone back on stable builds. We can then
consider our approach to migrating to Mitt and when we do do that we can
release it as a breaking change and properly document the migration
strategy and approach.
* chore: migrate to Mitt as the EventEmitter
This commit moves us to using Mitt [1] for the event emitter in
Puppeteer. This removes our dependency to Node's EventEmitter which is
part of a larger stream of work to enable a Puppeteer-web version that
doesn't depend on Node.
There are no large breaking changes as we support the main methods that
EventEmitter had, but it also provides some methods that Puppeteer
didn't use. Technically end users could depend on this but it's
unlikely.
[1]: https://github.com/developit/mitt
It conflicts with an inbuilt TypeScript `Request` type so can cause
confusion when in TS land. Note: `Response.ts` and `Worker.ts` also
suffer from this; PRs to rename them are incoming.
This script generated an `index.d.ts` file but that file was never
commited to git nor included in Puppeteer when we ship. As of right now
people who want TS types can install from the DefinitelyTyped repo and
we are working on shipping types from Puppeteer itself.
Therefore this script is not adding any value and can be removed.
We should import them just like any other module. This commit makes that
change. It does not change any behaviours or the types themselves.
EXPECTED_PROTOCOL_DIFF as we're updating the structure of it.
* chore: extract `BrowserRunner` into its own module
`src/Launcher.ts` is large and hard to work in. It has multiple objects
defined in it:
* ChromeLauncher
* FirefoxLauncher
* BrowserRunner
* Launcher
This change moves BrowserRunner into its own module. More refactorings
like this will follow but this is the first step.
* chore: remove src/externs.d.ts
It defined global types that we don't want to use, and instead we move
to using interfaces that we import and reference just like with any
other interface.
This means other than Protocol (which I think is fine to leave as is),
there are no other magic global types and you have to import any types
or interfaces that you want.
* chore: migrate src/Page.js to TypeScript
The final one! This is a huge file and needs to be split up and tidied,
but for now I've left all the definitions in place and converted types
accordingly.
There's some additional tidying we can do now every `src` file is TS,
but I'll leave that for another PR to avoid this one getting any bigger.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
* chore: enforce src/protocol.d.ts is in sync
On CI we run `npm run compare-protocol-d-ts` which checks that the file
on disk is up to date with the protocol we fetch from the browser.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
We don't support it and v3 shipped without including puppeteer-web in the browser. People are welcome to manually use Browserify to try to get Puppeteer running in a browser but it ultimately isn't our primary focus right now.
Getting puppeteer-core able to run in a browser is something we'll be looking at in the future so we'll revisit this soon.
* chore: migrate src/Input to typescript
This moves `Keyboard`, `Mouse` and `Touchscreen` to TypeScript. We gain
some nice TS benefits here; by creating a type for all the keycodes we
support we can type the input args as that rather than `string` which
will hopefully save some users some debugging once we ship our TS types
in a future version.
* Remove from externs file
* Update utils/doclint/check_public_api/index.js
Co-Authored-By: Mathias Bynens <mathias@qiwi.be>
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
* chore: migrate src/JSHandle to TS
There's a few TODOs in here that all depend on typing the
`ExecutionContext.evaluateHandle` properly so that you can properly
declare what types you're expecting back. Once I've done that file (it's
next on my list) I will loop back and improve the types here, fixing
these TODOs.
* Fix doclint for {}
The codebase was incredibly inconsistent with the use of spacing around
curly braces, e.g.:
```
// this?
const a = {b: 1}
// or?
const a = { b: 1 }
```
This extended into import statements also. Google's styleguide is no
spacing, so we're going with that.
* chore: migrate `src/Connection` to TypeScript
This commit migrates `src/Connection` to TypeScript. It also changes its
exports to be ESM because TypeScript's support for exporting values to
use as types via CommonJS is poor (by design) and so rather than battle
that it made more sense to migrate the file to ESM.
The good news is that TypeScript is still outputting to `lib/` as
CommonJS, so the fact that we author in ESM is actually not a breaking
change at all.
So going forwards we will:
* migrate TS files to use ESM for importing and exporting
* continue to output to `lib/` as CommonJS
* continue to use CommonJS requires when in a `src/*.js` file
I'd also like to split `Connection.ts` into two; I think the
`CDPSession` class belongs in its own file, but I will do that in
another PR to avoid this one becoming bigger than it already is.
I also turned off `@typescript-eslint/no-use-before-define` as I don't
think it was adding value and Puppeteer's codebase seems to have a style
of declaring helper functions at the bottom which is fine by me.
Finally, I updated the DocLint tool so it knows of expected method
mismatches. It was either that or come up with a smart way to support
TypeScript generics in DocLint and given we don't want to use DocLint
that much longer that didn't feel worth it.
* Fix params being required
This PR changes `src/Dialog.js` to `src/Dialog.ts` and rewrites
accordingly. Most of the changes are straight forward; the only
interesting one from a TS point of view is the `DialogType` enum. I
expose it again as `Dialog.Type` to avoid a breaking change.
This PR also exposed some bugs with our ESLint TypeScript settings and
applying the overrides, so I fixed those too.
I also updated our DocLint tool to work on TS source files over JS lib
files if they exist. This is the minimal change to keep the existing doc
system working as we're working on moving away from this system longer
term.
I lost some time debugging before realising that I needed to run tsc. I
don't really want to put `npm run tsc` before this command else we'll
run tsc multiple times on each CI build, so I think this message is
suitable.
Travis defines `process.env.TRAVIS` and if that exists we don't want to
log this as on CI we're guaranteed to have an up to date `lib/`
directory.
This commit moves `src/DeviceDescriptors` to be authored in TypeScript. This file was chosen due to its simplicity so that we can focus on getting a mixed JS/TS codebase playing nicely before migrating the more complex files.
The file itself was a bit odd: although the array of devices was exported via `module.exports` that was never referenced by any consumers; each device was also exported via `module.exports[name] = device` and that is how it's consumed. The Puppeteer docs suggest using it like so:
```js
puppeteer.devices['iPhone 6']
```
So instead of exporting the array and then setting a bunch of properties on that, we instead define the array and export an object of keys where each key is a device. This is a breaking change (see the footer for details).
Rather than export an object I'd much rather export a Map, but that would be a larger breaking change and I'm keen to avoid those for the time being.
Note that we have to use special TypeScript specific syntax for the export that enables it to work in a CommonJS codebase [1] but again I'd rather this than move to ESM at this time. TypeScript still outputs CommonJS into `lib/` as you would expect.
BREAKING CHANGE: We no longer export an array of devices, so any users relying on doing:
```js
puppeter.devices.forEach(...)
```
…will now see a breakage. The fix is to use `Object.{keys/entries/values}` to iterate instead.
[1]: https://www.typescriptlang.org/docs/handbook/modules.html#export--and-import--require
This commit updates all the non-Puppeteer unit tests to run using Mocha and then deletes the custom test runner framework from this repository. The documentation has also been updated.
Our logic around missing methods wasn't quite right; if there is no set of missing methods for a class it _is_ an error and we still need to report it, we don't want to `continue`.
This is expected as we now alias `emulateMedia` in `index.js` which isn't a file checked by DocLint. We alias there to avoid having the function overriden by the `asyncInstallHooks` code.
This commit updates doclint to know about methods that we expect it will find are missing and in that case just skip over them. We should only do this for methods where we plan to deprecate them or we have to define them in an odd way to work around some problem (and if that's the case long term we should fix that problem so we can define them as normal).
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
Rather than use our own custom expect library, we can use expect from npm [1], which has an API almost identical to the one Puppeteer has, but with more options, better diffing, and is used by many in the community as it's the default assertions library that comes with Jest.
It's also thoroughly documented [2].
[1]: https://www.npmjs.com/package/expect
[2]: https://jestjs.io/docs/en/expect
This updates our `tsconfig.json` so it emits our JavaScript files as
well as type checking them. We compile into `./lib` which we then ship
in our npm package. The source code has moved from `./lib` into `./src`.
Because the `src/` directory is exclusively JS files, this change is a
no-op in terms of code functionality but is the first step towards being
able to replace `src/X.js` with `src/X.ts` in a way that allows us to
migrate incrementally.
The `lib` directory is gitignored, and the `src` directory is
npmignored. On `npm publish` we will now run `npm run tsc` in order to
generate the outputted code.
TypeScript seems to struggle to understand `Promise.all` when the items in the array return different types. If we were authoring in TS we could fix this with TS generics (`Promise.all<OurTypeHere>(...)`) but for now we can typecast the result. We'll fix this properly when we author in TS.
TS 3.5 got much stricter on writing changes to objects with varied types [1] so we have to do a bit of typecasting work to convince TS about the types of keys and values that we are setting.
Longer term we should think about a better data structure that avoids us having to jump through some hoops but for now I think this is a reasonable step to get us onto 3.5.
Same story regarding bindings on `window`: the easiest fix is to cast `window` to `any` for the code that adds to it. I'm sure we can come up with a more type-safe way of doing this in the future.
[1]: https://github.com/Microsoft/TypeScript/wiki/Breaking-Changes#fixes-to-unsound-writes-to-indexed-access-types
* feat: Set which browser to launch via PUPPETEER_PRODUCT
This change introduces a PUPPETEER_PRODUCT environment
variable as a first step toward using Puppeteer with
many different browsers. Setting PUPPETEER_PRODUCT=firefox, for
example, enables Firefox-specific Launcher settings.
The state is also exposed as `puppeteer.product` in the API
to support adding other product-specific behaviour as needed.
The bulk of the change is a refactoring in Launcher
to decouple generic browser start-up from product-specific
configuration.
Respecting the puppeteer-core restriction for PUPPETEER_
environment variables, lazily instantiate the Launcher
based on a `product` Puppeteer.launch option, if available.
* test: Distinguish Juggler unit tests from Firefox
The funit script is renamed to fjunit (j for Juggler, which is
used only by the experimental puppeteer-firefox package.
In contrast, the funit script now refers to running Puppeteer
unit tests against the main puppeteer package with Firefox.
To do so with Firefox Nightly, run:
`BINARY=path/to/firefox npm run funit`
A number of changes in this patch make it easier to run
Puppeteer unit tests in Mozilla's CI.
Node.js v6 was end-of-life'd in April, 2019, with AWS Lambda prohibiting updaets to the Node.js v6 runtime since June 30, 2019.
This makes it quite safe for us to remove the Node 6 support from the repository.
`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.
As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.
This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
This patch:
- updates Flakiness Dashboard format to define version per-build
and to pass COMMIT information
- drops the README.md generation - we'll move on to a designated flakiness
dashboard viewer
- fix `FLAKINESS_DASHBOARD_BUILD_URL` to point to a task instead of a build
- do not pretty-print `dashboard.json` when serializing flakiness results
- filter out 'COVERAGE' test(s) so that they don't add up to `dashboard.json` payload. These are useless
- validate certain important options of flakiness dashboard
- more logging to STDOUT to actually say which repo and what branch is getting used
- enhance commit message with a build URL
- use a more compact format for JSON. For 100 runs of 700 tests it yields 21MB json instead of 23MB.
- bump default builds number to 100
This patch introduces a dashboard that records test results and
uploads them to https://github.com/aslushnikov/puppeteer-flakiness-dashboard
Since many bots might push results in parallel, each bot pushes
results to its own git branch.
FlakinessDashboard also generates a simple README.md with a flakiness
summary. If this proves to be not enough, we can build a website that
fetches flakiness data and renders it nicely.
This patch teaches TestRunner to support async suite
descriptions. This is needed to require tests using ES6 dynamic
imports:
```js
const t = new TestRunner();
await t.describe('tests', async () => {
(await import('./some.spec.js')).addTests(t);
(await import('./other.spec.js')).addTests(t);
});
```
This patch adds new TestRunner options:
- `disableTimeoutWhenInspectorIsEnabled` - disable test timeout if
testrunner detects enabled inspector.
- `breakOnFailure` - if testrunner should terminate test running on
first test failure
This patch improves the logic for test runner termination.
With this patch:
- TestRunner runs all afterEach/afterAll hooks when a
termination happens, properly terminating browser instances
- TestRunner cleans up all dangling timeout timers so that node.js
process is not retained and is free to exit
These getters are introduced as a more convenient substitute for
a `require('puppeteer/Errors')` and
`require('puppeteer/DeviceDescriptors')`.
This way we can make cross-browser story nicer - a single require
of `puppeteer` or `puppeteer-firefox` fully defines Puppeteer
environment.
This patch makes sure header overrides in request interception are
functioning as expected.
Drive-by: teach test server to use utf-8 charset header for text files.
This patch:
- implements Response.buffer() and other methods
- splits out relevant tests into a separate test suites
- implements `testServer.enableGzip()` method to optionally gzip
certain routes in tests
- adds tests to make sure `Response.text()` returns expected results
for binary and compressed responses.
This patch:
* unifies assets between tests
* enables a few puppeteer tests on Puppeteer-Firefox
Drive-by: beautify failing output of `expect.toEqual` matcher.
References #3889
This patch:
- introduces new testRunner methods `addTestDSL` and `addSuiteDSL`
to add annotated test / suite.
- introduces new test/suite declaration methods: `it_fails_ffox` and
`describe_fails_ffox`. These are equal to `it`/`describe` for chromium
tests and to `xit`/`xdescribe` for firefox.
- marks all unsupported tests with `it_fails_ffox`
- adds a new command-line flag `'--firefox-status'` to `//test/test.js`.
This flag dumps current amount of tests that are intentionally skipped
for Firefox.
End goal: get rid of all `it_fails_ffox` and `describe_fails_ffox`
tests.
Drive-By: remove cookie tests "afterEach" hook that was removing
cookies - it's not needed any more since every test is run in a
designated browser context.
References #3889
Introduce `//lib/api.js` that declares a list of publicly exposed
classes.
The `//lib/api.js` list superceedes dynamic `helper.tracePublicAPI()` calls
and is used in the following places:
- [ASYNC STACKS]: generate "async stacks" for publicy exposed API in `//index.js`
- [COVERAGE]: move coverage support from `//lib/helper` to `//test/utils`
- [DOCLINT]: get rid of 'exluded classes' hardcoded list
This will help us to re-use our coverage and doclint infrastructure
for Puppeteer-Firefox.
Drive-By: it turns out we didn't run coverage for `SecurityDetails`
class, so we lack coverage for a few methods there. These are excluded
for now, sanity tests will be added in a follow-up.
This patch splits out `IsolatedWorld` class from Frame.
The `IsolatedWorld` abstraction is an execution context
with a designated set of DOM wrappers.
References #2671
