During the migration to TS we changed `jsonValue` so it returned
`<Record<string, unknown>>`. This is only true if all the JSON values it
returns are objects; but it could return an array, a string, a number,
etc. Therefore we make the type generic, setting the default to
`unknown`, so the user has control over the type.
* fix: wider compat TS types and CI checks to ensure correct type defs
This PR improves our TS types further to make sure they are usable in a
TS environment where ES Modules are the target output. Our use of
`export =` is problematic this environment as TypeScript does not allow
`export =` to be used and it errors.
The fix for the type issues to avoid `export =` is to instead define the
functions that you gain access to when you import Puppeteer as top level
functions in our `types.d.ts` file. We can do this by declaring them
explicitly in `src/node.ts`. These are then rolled into `lib/types.d.ts`
at build time. The downside to this is that we have to keep those
declarations in sync with the Puppeteer API; should we add a new method
to the `Puppeteer` class, we must add it to the `nodes.ts` declarations.
However, this could easily be automated by a small script that walks the
AST and generates these. I will do that in a follow-up PR, but I
consider this low risk given how rarely the very top level API of
Puppeteer changes. The nice thing about this approach is we no longer
need our script that hacks on changes to `lib/types.d.ts`.
To avoid yet more releases to fix issues in one particular TS
environment, this PR also includes a suite of example setups that we
test on each CI run. Each sample folder contains `good.ts`, which should
have no TS errors, and `bad.ts`, which should have some errors. The test
first packs Puppeteer into a tar, and then installs it from that tar
into each project. This should replicate how the published package
behaves when it is installed. We then check that we get no errors on
`good.ts`, and the expected errors on `bad.ts`.
We have a variety of test projects that cover both TS and JS source
code, and CJS and ESM imports and outputs.
This PR aims to vastly improve our TS types and how we ship them.
Our previous attempt at shipping TypeScript was unfortunately flawed for
many reasons when compared to the @types/puppeteer package:
* It only worked if you needed the default export. If you wanted to
import a type that Puppeteer uses, you'd have to do `import type X
from 'puppeteer/lib/...'`. This is not something we want to encourage
because that means our internal file structure becomes almost public
API.
* It gave absolutely no help to CommonJS users in JS files because it
would warn people they needed to do `const pptr =
require('puppeteer').default, which is not correct.
* I found a bug in the `evaluate` types which mean't you couldn't
override the types to provide more info, and TS would insist the types
were all `unknown`.
The goal of this PR is to support:
1. In a `ts` file, `import puppeteer from 'puppeteer'`
1. In a `ts` file, `import type {ElementHandle} from 'puppeteer'`
1. In a `ts` file, referencing a type as `puppeteer.ElementHandle`
1. In a `ts` file, you can get good type inference when running
`foo.evaluate(x => x.clientHeight)`.
1. In a `js` file using CJS, you can do `const puppeteer =
require('puppeteer')` and get good type help from VSCode.
To test this I created a new empty repository with two test files in,
one `.ts` file with this in:
https://gist.github.com/jackfranklin/22ba2f390f97c7312cd70025a2096fc8,
and a `js` file with this in:
https://gist.github.com/jackfranklin/06bed136fdb22419cb7a8a9a4d4ef32f.
These files included enough code to check that the types were behaving
as I expected.
The fix for our types was to make use of API Extractor, which we already
use for our docs, to "rollup" all the disparate type files that TS
generates into one large `types.d.ts` which contains all the various
types that we define, such as:
```ts
export declare class ElementHandle {...}
export type EvaluateFn ...
```
If we then update our `package.json` `types` field to point to that file
in `lib/types.d.ts`, this then allows a developer to write:
```
import type {ElementHandle} from 'puppeteer'
```
And get the correct type definitions. However, what the `types.d.ts`
file doesn't do out of the box is declare the default export, so
importing Puppeteer's default export to call a method such as `launch`
on it will get you an error.
That's where the `script/add-default-export-to-types.ts` comes in. It
appends the following to the auto-generated `types.d.ts` file:
```ts
declare const puppeteer: PuppeteerNode;
export = puppeteer;
```
This tells TypeScript what the default export is, and by using the
`export =` syntax, we make sure TS understands both in a TS ESM
environment and in a JS CJS environment.
Now the `build` step, which is run by GitHub Actions when we release,
will generate the `.d.ts` file and then extend it with the default
export code.
To ensure that I was generating a valid package, I created a new
repository locally with the two code samples linked in Gists above. I
then ran:
```
npm init -y
npm install --save-dev typescript
npx tsc --init
```
Which gives me a base to test from. In Puppeteer, I ran `npm pack`,
which packs the module into a tar that's almost identical to what would
be published, so I can be confident that the .d.ts files in there are
what would be published.
I then installed it:
```
npm install --save-dev ../../puppeteer/puppeteer-7.0.1-post.tgz
```
And then reloaded VSCode in my dummy project. By deliberately making
typos and hovering over the code, I could confirm that all the goals
listed above were met, and this seems like a vast improvement on our
types.
* fix(launcher): output correct error message for browser
When running `npm run release` today I got this error logged:
```
Error: Could not find browser revision 848005. Run "PUPPETEER_PRODUCT=firefox npm install" or "PUPPETEER_PRODUCT=firefox yarn install" to download a supported Firefox browser binary.
at ChromeLauncher.launch (/Users/jacktfranklin/src/puppeteer/lib/cjs/puppeteer/node/Launcher.js:80:27)
```
The error is only partially correct; I did have the browser revision
missing, but I needed the Chromium browser, not Firefox. It turns out
the logic in `Launcher.ts` didn't take this into account; it mistakenly
had been hardcoded to always log out the error as if the Firefox binary
was missing.
This PR updates the message depending on the browser:
Chrome error:
> Error: Could not find expected browser (chrome) locally. Run npm
> install or yarn install to download the correct Chromium revision
> (848005).
Firefox error:
> Error: Could not find expected browser (firefox) locally. Run
> "PUPPETEER_PRODUCT=firefox npm install" or "PUPPETEER_PRODUCT=firefox
> yarn install" to download a supported Firefox browser binary.
* Update src/node/Launcher.ts
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
This corresponds to Chromium 90.0.4403.0
This roll includes:
- Cut screenshot by ViewPort size, not position (crrev.com/c/2643792)
BREAKING CHANGE:
- `page.screenshot` cuts screenshot content by the ViewPort size, not ViewPort position.
This corresponds to Chromium 89.0.4389.0.
This roll includes:
- Add `SameParty` attribute to cookies
https://crrev.com/c/2598846
- Anchor `target=_blank` implies `rel=noopener`
https://crrev.com/c/1630010
- Don’t expect ignored elements in the AXTree
https://crrev.com/c/2505362
BREAKING CHANGE: The built-in `aria/` selector query handler doesn’t return ignored elements anymore.
Issue: #6758
This PR replaces https://github.com/puppeteer/puppeteer/pull/6289 with a simpler approach to types where we compile them all alongside the compiled code (so for every `foo.js` that is generated, we generate `foo.d.ts`).
By default that's not enough, as when you `import puppeteer from 'puppeteer'` in Node land you import `cjs-entry.js`, so we also create `cjs-entry.d.ts` which TypeScript will then pick up. This type file tells TypeScript that the thing that `cjs-entry.js` exposes is an instance of the `Puppeteer` class, which then hooks all the types up.
The previous PR (https://github.com/puppeteer/puppeteer/pull/6289) tried to merge all our typedefs into one big file via API Extractor, but this isn't really necessary a good experience for the developer.
One important consideration is that you _could_ consider this a breaking change. I'm not sure how likely it is, but it could cause problems for:
* Projects that didn't have any type info for Puppeteer and treated all its exports as `any` may now start having legitimate type failures.
* Projects that depend on the `@types/puppeteer` package may have issues if they now swap to use this one and the types aren't quite aligned.
In addition, once we do ship a release with this change in, it will mean that we have to treat any changes to any type definitions as release-note-worthy, and any breaking changes to type definitions will need to be treated as breaking code changes (nearly always a breaking type def means a breaking change anyway), but it's worth considering that once we expose these typedefs we should treat them as first class citizens of the project just like we would with the actual source code.
I also fully expect to have some bugs with our types, or have users create issues/PRs to change our types, but I think that's a good thing and it should settle down.
I've tested this locally by creating a new package, linking Puppeteer via `npm link puppeteer` and then checking that VSCode correctly shows me the right things when I use `Go to Definition` on something that comes from the Puppeteer source code.
With `nodejs@15.0.1`, install puppeteer with `https_proxy` set causes an error like:
```
> puppeteer@5.4.1 install node_modules/puppeteer
> node install.js
ERROR: Failed to set up Chromium r809590! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
TypeError [ERR_INVALID_PROTOCOL]: Protocol "https:" not supported. Expected "http:"
at new NodeError (node:internal/errors:258:15)
at new ClientRequest (node:_http_client:155:11)
at Object.request (node:https:313:10)
at httpRequest (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:488:17)
at downloadFile (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:357:21)
at BrowserFetcher.download (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:239:19)
at async downloadBrowser (node_modules/puppeteer/lib/cjs/puppeteer/node/install.js:48:5) {
code: 'ERR_INVALID_PROTOCOL'
}
```
The related issue is at https://github.com/TooTallNate/node-agent-base/pull/47, from package `agent-base` under `https-proxy-agent`
And the version bump is for `Refactor to TypeScript` is here: https://github.com/TooTallNate/node-https-proxy-agent/compare/4.0.0...5.0.0
`isNode` checks for the node version in which the process is running and it checks for the node key inside the versions. This is exactly the same as using `process.version` as they are set in the native code and `process.version` is just a shorthand for `process.versions.node`.
* chore: run unit tests on node 10.15
We saw in https://github.com/puppeteer/puppeteer/issues/6548 that the
`fs.promises` module was experimental in Node <10.17 and as such we
introduced issues for users on 10.15.
Until we can drop Node v10 (it's EOL is 30-04-20201
https://github.com/nodejs/Release#release-schedule) we should run our
tests on an old Node 10 to avoid regressing in this area.
* chore: helper for importing fs safely
This PR updates the socket transport code to swap between a Node web
socket transport or a web one based on the `isNode` environment. It also
adds unit tests to the browser tests that show we can connect in a
browser.
* chore(agnostification): common/helper.ts
The `readProtocolStream` method uses `fs` only if you want to write to a
file. So we gate at the start of the function and ensure that if we got
given a path we are not in a Node environment.
The `Puppeteer` class had two concerns:
* connect to an existing browser
* launch a new browser
The first of those concerns is needed in all environments, but the
second is only needed in Node.
https://github.com/puppeteer/puppeteer/pull/6484 landing enabled us to
pull the `Puppeteer` class apart into two:
1. `Puppeteer` which hosts the behaviour for connecting to existing
browsers.
2. `PuppeteerNode`, which extends `Puppeteer` and also adds the ability
to launch a new browser.
This is a non-breaking change, because Node users will still get an
instance of a class with all the methods they expect, but it'll be a
`PuppeteerNode` rather than `Puppeteer`. I don't expect this to cause
people any issues.
We also now have new files that are effectively the entry points for
Puppeteer:
* `node.ts`: the main entry point for Puppeteer on Node.
* `web.ts`: the main entry point for Puppeteer on the web.
* `node-puppeteer-core.ts`: for those using puppeteer-core (which only
exists in Node, not on the web).
Launching headless with a relative `userDataDir` hangs on Windows. Fix by calling `path.resolve` (idempotent) to add an absolute path instead in `defaultArgs`.
Issues: #3453
The `Launcher` class was serving two purposes:
1. Launch browsers
2. Connect to browsers
Number 1) only needs to be done in Node land, but 2) is agnostic; in a
browser version of Puppeteer we'll need the ability to connect over a
websocket to send commands back and forth.
As part of the agnostification work we needed to split the `Launcher` up
so that the connection part can be made agnostic. Additionally, I
removed dependencies on `https`, `http` and `URL` from Node, instead
leaning on fetch (via `node-fetch` if in Node land) and the browser
`URL` API (which was added to Node in Node 10).
* chore(agnostify): Create Node and Web initializer.
This PR splits `initialize.ts` into two files, one for web, and one for
Node. The Node initializer requires much more information as it needs to
know which browser(s) to download and where to store them, whereas the
web one does not.
A future PR that I'm working on will tidy up `src/common/Puppeteer.ts`
(as it contains a lot of Node specific logic around downloading and
installing browsers), but this change enables us to stop the browser
bundle attempting to use the `pkg-dir` dependency, which wouldn't work
within a browser, as well as keeping the size of the PRs down and
avoiding one mammoth PR.
This commit adds a new built-in handler for querying by accessible name and role (#6307).
Support for waitForSelector will be added in a follow-up commit.
In `src/common` we now use `fs.promises.X` which we can dynamically
`import`. In a browser environment this code will never run because it's
gated on `isNode` (in a future PR we will add tree-shaking to the bundle
step such that this code is eliminated). By using `import`, we ensure
TypeScript still can track types and give good type information.
In `src/node` we continue to use `util.promisify` but that's not a
concern as that code explicitly is never run in the browser.
This commit changes the internal representation of query handlers to contain Puppeteer-level code instead of page functions.
The interface `CustomQueryHandler` is introduced for user-defined query handlers. When a `CustomQueryHandler` is registered using `registerCustomQueryHandler` a corresponding Puppeteer-level handler is created through `makeQueryHandler` by wrapping the page functions as appropriate.
The internal query handlers (defined by the interface `QueryHandler`) contain two new functions: `waitFor` and `queryAllArray`.
- `waitFor` allows page-based handlers to make use of the `WaitTask`-backed implementation in `DOMWorld`, whereas purely Puppeteer-based handlers can define an alternative approach instead.
- `queryAllArray` is similar to `queryAll` but with a slightly different interface; it returns a `JSHandle` to an array with the results as opposed to an array of `ElementHandle`. It is used by `$$eval`.
After this change, we can introduce built-in query handlers that are not executed in the page context (#6307).
The logic for waitForXPath and waitForSelector is currently very tightly coupled. This commit tries to untangle that relationship. This is the first step towards introducing built-in query handlers that are not executed in the page context (#6307).
Debug message clarifying the problem like `Issue 1060080: Event Network.loadingFinished is not sent when a frame is loaded from another domain`: https://bugs.chromium.org/p/chromium/issues/detail?id=1060080).
It can help users to identify problem with OOPIF easier without digging into the CDP protocol implementation like in the bug `1060080` mentioned above.
To reproduce:
1. Run `DEBUG="puppeteer:frame" NODE_PATH=../ node examples/oopif.js`.
2. Verify the output contains the debug message:
`
puppeteer:frame The frame '...' moved to another session. Out-of-proccess
iframes (OOPIF) are not supported by Puppeteer yet.
https://github.com/puppeteer/puppeteer/issues/2548
`
* roll Chromium to version 86, r800071
* roll Chrome DevTools protocol version to 0.0.799653
* fix HTTPRequest.continue after
* CDP: accept post data in the binary form in Fetch.continueRequest.
https://chromium-review.googlesource.com/c/chromium/src/+/2315239
* update new-docs
By adding support for an environment variable `PUPPETEER_DOWNLOAD_PATH` it is possible to support downloading the browser binaries into a folder outside the `node_modules` folder. This makes it possible to preserve previously downloaded binaries in order to skip downloading them again.
This commit also removes our own custom type for defining the vision
deficiencies and uses the protocol's type. Now we generate docs for
those we get the docs generated for free for these. This is better than
us duplicating values for types in doc comments and having them become
outdated. If we use the protocol types directly then we ensure we're up
to date and in-sync.
Long term the docs will also link to the devtools-protocol viewer.
This corresponds to Chromium 85.0.4182.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure
https://crrev.com/c/2231445
- [FlexNG] Enable FlexNG by default
https://crrev.com/c/2216595Closes#6151.
The Node debug library uses the `DEBUG` environment variable to
configure what gets logged. Our browser version just logs everything;
this commit changes it to look for `window.__PUPPETEER_DEBUG` and matches the behaviour accordingly:
* If the value is not set, nothing is logged.
* If the value is set to `*` everything is logged.
* If the value is set to a string `foo`, messages with that prefix are
logged.
* If the value is set to a string ending in `*`, e.g. `foo*`, messages
with prefixes that start with `foo` are logged.
Also took the opportunity to pull out the PDF types into their own file
to clear up `Page.ts` slightly and give the PDF code a more natural
place to live.
* chore: vendor Mitt into src/common/third-party
As discussed in #6203 we need to vendor our common dependencies in so
that when we ship an ESM build all imports point to file paths and do
not rely on Node resolution (e.g. a browser does not understand `import
mitt from 'mitt'`).
* chore: enforce file extensions on imports
To make our output agnostic it should include file extensions in the
output, as per the ESM spec. It's a bit odd for Node packages but makes
it easier to publish a browser build.
* feat(chromium): roll Chromium to r768783
* fix: update unit test for crrev:2135046
* chore: update devtools-protocol revision
Co-authored-by: Changhao Han <changhaohan@chromium.org>
* chore: Use devtools-protocol package
Rather than maintain our own protocol we can instead use the devtools-protocol package and pin it to the version of Chromium that Puppeteer is shipping with.
The only changes are naming changes between the bespoke protocol that Puppeteer created and the devtools-protocol one.
Now the async hooks helper is gone api.ts was only used by the coverage
tools and by doclint.
DocLint is nearing the end of its lifespan with the TSDoc work, so I
focused on how best to define a list of modules for the coverage
tooling. They define an object of classes, and the path to that module.
They need the full path because we also check if the module exports any
events that need to be emitted - the coverage tool asserts that the
emitting of those events is also tested.
It's not _great_ that DocLint relies on a constant defined in the
coverage utils, but it should only be this way for a short period of
time and no one is actively working on DocLint (bar the effort to remove
it) so I don't think this is worth worrying about.
This change also broke the DocLint tests; based on the fact that DocLint is on its way out it doesn't feel worth fixing the tests, so this commit also removes them.
* chore: remove `installAsyncStackHooks` helper
This code was written when browsers/Node didn't support errors in async
functions very well. They now do a much better job of this, so we can
lose the additonal complexity from our codebase and leave it to the host
environment :)
* lazy launcher is private
* remove async stack test
This file is now deprecated and only used by the coverage tool and
DocLint - these tools will be updated to not rely on it in the future.
We now have events defined per class - e.g. all the events that `Page`
can emit are defined in the `PageEmittedEvents` enum, and similar. We
have to keep `Events.ts` around for the aforementioned tools, but don't
want its usage creeping back into our source code.
On a getter function you don't use `@returns` (as that's just for
methods). Instead we can just add to the general remarks to explain what
the property is providing.
This is part of the effort to remove `Events.ts` in favour of defining
events next to the class that emits them. In this case these events are
internal, so there's no docs changes, but it's still worth doing such
that we can remove the Events.ts file in the long term once all the
different events are migrated.
I noticed that DOMWorld was spitting a lot of warnings out when we
generated the docs. It was mostly easy tidy-ups and removing old JSDoc
comments and now the warnings are gone :)
* chore(docs): migrate & document all Page events
Rather than a generic `Events.ts` file we can instead document events as
an enum within each individual class. It's easier to document and work
with, and it's clearer where events originate from.
Previously Node.js was detected by the lack of `global.document` which doesn’t work in case JSDOM is being used in Node.js. Instead, we now detect `process.versions.node`, like here: 426943ae93 (diff-168726dbe96b3ce427e7fedce31bb0bc).
Fixes#6147.
* chore(docs): reduce warnings when generating docs
This is a bunch of small miscellaneous fixes that reduce the amount of
warnings logged when generating our new docs. The long term goal is to
get this list down to 0 warnings, but I'll do it in multiple PRs.
* satisfy doclint
This pulls in the types (based on the DefinitelyTyped repo) for
`page.$eval` (and the `$eval` method on other classes). The `$eval`
method is quite hard to type due to the way we wrap and unwrap
ElementHandles that are passed to / returned from the `pageFunction`
that users provide.
Longer term we can improve the types by providing type overloads as
DefinitelyTyped does but I've deferred that for now (see the `TODO` in
the code for more details).
This change started as a small change to pull types from DefinitelyTyped over to
Puppeteer for the `evaluateHandle` function but instead ended up also fixing
what looks to be a long standing issue with our existing documentation.
`evaluateHandle` can in fact return an `ElementHandle` rather than a `JSHandle`.
Note that `ElementHandle` extends `JSHandle` so whilst the docs are technically
correct (all ElementHandles are JSHandles) it's confusing because JSHandles
don't have methods like `click` on them, but ElementHandles do.
if you return something that is an HTML element:
```
const button = page.evaluateHandle(() => document.querySelector('button'));
// this is an ElementHandle, not a JSHandle
```
Therefore I've updated the original docs and added a large explanation to the
TSDoc for `page.evaluateHandle`.
In TypeScript land we'll assume the function will return a `JSHandle` but you
can tell TS otherwise via the generic argument, which can only be `JSHandle`
(the default) or `ElementHandle`:
```
const button = page.evaluateHandle<ElementHandle>(() => document.querySelector('button'));
```
* chore: Don't store revisions in `package.json`
It's quite messy to have to require the `package.json` file in multiple
places purely to find out what revision of a given browser we want to
use. We can also achieve better type safety by placing it in an actual
source file.
This commit makes that change and also tidies up our reliance on
`package.json` within the source code generally; we now only use it to
find the location of the Puppeteer root such that we know where to
install downloaded browsers to.
To avoid using `package.json` to parse the name of the module, we also
now explicitly have an entry point for the Puppeteer module and the
Puppeter Core module. This will make it easier in the future to ship
less code as part of core (e.g. core never needs to download a browser,
so why ship that code?). Core can also then not have any revisions based
info contained in it.
The test install script has also been updated to ensure that
puppeteer-core can be installed correctly too.
Finally, the `install` script has been moved to TypeScript for nicer
typechecking and safety. The functionality of it has not changed.
* chore(agnostic): ship CJS and ESM builds
For our work to enable Puppeteer in other environments (e.g. a browser)
we need to ship an ESM build. This commit changes our config to ship to
`lib/cjs` and `lib/esm` accordingly. The majority of our code stays the
same, with one small fix for the CJS build to ensure that we ship a
version that lets you `require('puppeteer')` rather than have to
`require('puppeteer').default`. We do this with the `cjs-entry.js` which
is what the `main` field in our `package.json` points to.
We also swap to `read-pkg-up` to find the `package.json` file. This is
because the folder structure of `lib/` does not match `src/` now we ship
to `cjs` and `esm`, so you cannot rely on exact paths. This module works
up from the file to find the nearest `package.json` so it will always
find Puppeteer's `package.json`.
Note that we *do not* point any users to the ESM build. We happen to
ship those files so people who know about them can get at them but it's
not expected (nor will we actively support) that people will rely on
them. The CommonJS build is considered our main build.
We may make breaking changes to the structure of the ESM build which we
will do without requiring new major versions. For example the ESM build
currently ships all files that the CJS build does, but given we are
working on the ESM build being able to run in the browser this may
change over time.
Long term once the Node versions catch up we can ditch CJS and ship
exclusively ESM but we are not there yet.
* feat(types): improve typing of `.evaluate()`
This is the start of the work to take the types from the
`@types/puppeteer` repository and port them into our repo so we can ship
our built-in types out the box.
This change types the `evaluate` function properly. It takes a generic
type which is the type of the function you're passing, and the arguments
and the return that you get back from the `evaluate` call are typed
correctly.
DOMWorld only needs to use Node's `fs` module if you're adding a
filepath as a script/style tag. We can detect this case and run the
`require` inline such that in a browser this code won't execute.
* Adds tsdoc to Mouse class
* Updates puppeteer class tsdoc
* docs(new): add TSDoc comments to BrowserFetcher (#6078)
* Adds tsdoc for Touchscreen (#6087)
Co-authored-by: martinsplitt <martin@geekonaut.de>
* Adds tsdoc to Mouse class
* Fixes tsdoc comment for Mouse class
Co-authored-by: martinsplitt <martin@geekonaut.de>
This CL migrates all the tests to TypeScript. The main benefits of this is that we start consuming our TypeScript definitions and therefore find errors in them. The act of migrating found some bugs in our definitions and now we can be sure to avoid them going forwards.
You'll notice the addition of some `TODO`s in the code; I didn't want this CL to get any bigger than it already is but I intend to follow those up once this lands. It's mostly figuring out how to extend the `expect` types with our `toBeGolden` helpers and some other slight confusions with types that the tests exposed.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
It was causing some infra issues when trying to migrate tests to
TypeScript (that's WIP in another branch that I'll have up soon). It's
unusual to have the entire src in TS except for the main file, which
then reaches into the compiled `lib` directory for the files it needs.
Much better is to move the entry point into TypeScript itself and update
the `main` entry in our `package.json` to point to the compiled output.
This also has the advantange of hooking up all the TS type defs that we
are shipping and will make that process easier too, along with making it
easier to port our tests to TypeScript.
These files will be used by both the web and node versions of Puppeteer.
Another name for this might be "core" but I don't want to cause
confusion with the puppeteer-core package that we publish at the moment.
This is another step towards making Puppeteer agnostic of environment
and being able to run in Node or a browser.
The files in the `node` directory are ones that would only be needed in
the Node build - e.g. the code that downloads and launches a local
browser instance.
The long term vision here is to have three folders:
* node - Node only code
* web - Web only code
* common - code that is shared
But rather than do that in one PR I'm going to split it up to make it
easier to review and deal with.
The TypeScript definition erroneously made `options` required. We can
fix it by providing a default value, which means users calling the
function will be able to leave it blank without TS complaining.
Issues like this are a +1 to porting our tests to TypeScript in order to
catch these on our own test suite, so that's something we should look into.
* chore: create new Debug module
This debug module can be used in either Node or the browser. We'll use
the `debug` module in Node land, but fallback to a simple `console.log`
solution when in the browser in an attempt to keep our browser bundle
size down.
* Use our debug wrapper rather than Node's `debug`.
A lot of the helpers in `helpers.ts` are heavily bound to NodeJS and at
the moment we're trying to make the `Connection` class be able to run in
multiple environments. Its only remaining Node dependency was its
reliance on `helpers.ts`, which it only needed for `assert`.
This is a useful change also because `helpers.ts` is quite large and
full of functions that do different things; I think we can name them
better and move them into modules with a specific purpose rather than a
generic `"helpers"` dumping ground.
Once this change lands `Connection` should be usable in the browser.
Fix child process killing when the parent process SIGINTs.
If you `ctrl + c` the Puppeteer parent process, we would sometimes not properly handle killing of the child processes. This would then leave child processes behind, with running Chromium instances. This in turn could block Puppeteer from launching again and results in
cryptic errors.
Instead of using the generic `process.kill` with the process id (which for some reason is negative the pid, which I don't get), we can kill the child process directly by calling `proc.kill`.
Fixes#5729.
Fixes#4796.
Fixes#4963.
Fixes#4333.
Fixes#1825.
This PR splits the logging for send and receive messages in to separate debug channels.
This way it is easier to filter and takes advantage of debug's automated coloring to make
it easier to visually parse on the command line
* chore: remove "Extracting..." log message
Fixes#5741.
* test: support extra Launcher options and skips
The extra Launcher options and skipping conditions enable
unit tests to be run more easily by third-parties, e.g.
browser vendors that are interested in Puppeteer support.
Extra Launcher options were previously removed as part of
switching away from the custom test harness.
* test: enable more tests for Firefox
* chore: move `index.js` into `src`
This is the first part of a series of pull requests that will slowly
make it possible to initialise Puppeteer for either a Node environment
or a web one. By creating the `initialisePuppeteer` function we can
inject dependencies in for the given version of Puppeteer (e.g. inject a
different debug library for Node vs the web).
This PR moves the initialisation into `src` and calls into it from the
root `index.js`.
This PR starts exploring the Page class and how to best document it. It explores how best to document events in the system, and I think pulling them out into an `enum` is the best solution here. It lets us end up with a page of docs that explicitly lists all the events the page class can ever emit.
Just one was used externally and I wrapped that up in a method. I think
it's a useful method to provide (I can imagine wanting to know if JS is
enabled on a page) so I think there's no harm here (I'd rather that then
have JSHandle reach into a private variable).
Replacing the Node EventEmitter with Mitt caused more problems than
anticipated for end users due to the API differences and the amount of
people who relied on the EventEmitter API. In hindsight this clearly
should have been explored more and then released as a breaking v4.
This commit rolls us back to the built in Node EventEmitter library
which we can release to get everyone back on stable builds. We can then
consider our approach to migrating to Mitt and when we do do that we can
release it as a breaking change and properly document the migration
strategy and approach.
* chore: migrate to Mitt as the EventEmitter
This commit moves us to using Mitt [1] for the event emitter in
Puppeteer. This removes our dependency to Node's EventEmitter which is
part of a larger stream of work to enable a Puppeteer-web version that
doesn't depend on Node.
There are no large breaking changes as we support the main methods that
EventEmitter had, but it also provides some methods that Puppeteer
didn't use. Technically end users could depend on this but it's
unlikely.
[1]: https://github.com/developit/mitt
It conflicts with an inbuilt TypeScript `Request` type so can cause
confusion when in TS land. Note: `Response.ts` and `Worker.ts` also
suffer from this; PRs to rename them are incoming.
* Don't use expect within Promises (#5466)
If a call to expect fails within a Promise it will not
be resolved, and causing the test to crash.
The patch aligns the code similar to what is used by all
the other tests.
We should import them just like any other module. This commit makes that
change. It does not change any behaviours or the types themselves.
EXPECTED_PROTOCOL_DIFF as we're updating the structure of it.
This corresponds to Chromium 83.0.4103.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure https://crrev.com/c/2122809
* chore: extract `BrowserRunner` into its own module
`src/Launcher.ts` is large and hard to work in. It has multiple objects
defined in it:
* ChromeLauncher
* FirefoxLauncher
* BrowserRunner
* Launcher
This change moves BrowserRunner into its own module. More refactorings
like this will follow but this is the first step.
* Warn when given unsupported product name.
Fixes#5844.
This change means when a user launches Puppeteer with a product name
that is not supported (which at the time of this commit means it's not
`firefox` or `chrome) we will warn them about it.
Decided on just a warning vs an error because the current behaviour is
that we fallback to launching Chrome and I don't think this warrants a
breaking change.
Now the first pass of migrating to TypeScript is complete I'm going
through the src files one by one to tidy up the public/private
interfaces.
Puppeteer used an underscore convention to denote privacy but violates
this in some places; we should be strict with TypeScript's `public` and
`private` keywords instead.
This means we'll get nice TS errors if you try to refer to a private
method/variable, and means when we swap to generating our TS docs the
tooling knows what method(s) are public and therefore need to be
documented vs private internals that don't.
* chore: Remove src/TaskQueue
The only place it's used is in `src/Page.ts` to have a chain of
screenshot promises. Rather than initialize a task queue in `Browser`
and pass it through a chain of constructors we instead move the class
into `src/Page` and define it inline.
In the future we might want to create a helpers folder to contain small
utilities like that (`src/Page.ts` is already far too large) but I'm
leaving that for a future PR.
`TaskQueue` isn't documented in `api.md` so I don't think this is a
breaking change.
I updated the type of `screenshot()` to return `Promise<string | Buffer
| void>` because if a promise rejects it's silently swallowed. I'd like
to change this behaviour but one step at a time. This type only had to
change as now we type the screenshot task queue correctly rather than
using `any` which then exposed the incorrect `screenshot()` types.
closes#5719
Original stream.pipeline issue has been fixed in Node 14.1.0 and above.
Meanwhile, the workaround itself caused timout failures in many CI Node 14 runs, as reported in #5719.
Removed it, as it's no longer needed, and actually blocks consumers from adding CI testing on Node 14.
* chore: remove src/externs.d.ts
It defined global types that we don't want to use, and instead we move
to using interfaces that we import and reference just like with any
other interface.
This means other than Protocol (which I think is fine to leave as is),
there are no other magic global types and you have to import any types
or interfaces that you want.
* chore: migrate src/Page.js to TypeScript
The final one! This is a huge file and needs to be split up and tidied,
but for now I've left all the definitions in place and converted types
accordingly.
There's some additional tidying we can do now every `src` file is TS,
but I'll leave that for another PR to avoid this one getting any bigger.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
This change enforces how we type arrays, e.g. choosing between:
* `string[]`
* `Array<string>`
I've gone for the `array-simple` option [1] which enforces that:
* primitive types and type references use `X[]`
* complex types use `Array<X>`
For example, we'd type an array of strings as `string[]`, but an array
of a union type as `Array<SomeUnionType>`.
[1]: https://github.com/typescript-eslint/typescript-eslint/blob/master/packages/eslint-plugin/docs/rules/array-type.md
Node's promisify function and the TS types for it give much better type understanding when wrapping a function in `promisify`. This change means we don't maintain our own implementation and our own (sub-par) types and rather lean on the tested and thorough @types/node version instead.
* chore: Useful error for Node v14 breakage
There is currently a bug with extract-zip and Node v14.0.0 that
causes extractZip to silently fail:
https://github.com/puppeteer/puppeteer/issues/5719
Rather than silenty fail if the user is on Node 14 we instead
detect that and throw an error directing the user to that bug. The
rejection message below is surfaced to the user in the command line.
The issue seems to be in streams never resolving so we wrap the call in
a timeout and give it 100ms to resolve before deciding on an error. If
the user is on Node < 14 we maintain the behaviour we had before this
patch.
Here's how this change impacts the output on Node 14 and Node 10:
Node 10:
```
npm run tsc && rm -r .local-* && node install
> puppeteer@3.0.1-post tsc /Users/jacktfranklin/src/puppeteer
> tsc --version && tsc -p . && cp src/protocol.d.ts lib/ && cp src/externs.d.ts lib/
Version 3.8.3
Downloading Chromium r737027 - 118.4 Mb [====================] 100% 0.0s
Chromium (737027) downloaded to /Users/jacktfranklin/src/puppeteer/.local-chromium/mac-737027
```
---
Node 14 without this patch:
```
npm run tsc && rm -r .local-* && node install
> puppeteer@3.0.1-post tsc /Users/jacktfranklin/src/puppeteer
> tsc --version && tsc -p . && cp src/protocol.d.ts lib/ && cp src/externs.d.ts lib/
Version 3.8.3
Downloading Chromium r737027 - 118.4 Mb [====================] 100% 0.0s
```
Note that whilst it doesn't error, it doesn't complete the install. We
don't get the success message that we saw above in the Node 10 install.
---
Node 14 with this patch:
```npm run tsc && rm -r .local-* && node install
> puppeteer@3.0.1-post tsc /Users/jacktfranklin/src/puppeteer
> tsc --version && tsc -p . && cp src/protocol.d.ts lib/ && cp src/externs.d.ts lib/
Version 3.8.3
Downloading Chromium r737027 - 118.4 Mb [====================] 100% 0.0s
ERROR: Failed to set up Chromium r737027! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
Puppeteer currently does not work on Node v14 due to an upstream bug. Please see https://github.com/puppeteer/puppeteer/issues/5719 for details.
```
The explicit message should save users a good amount of debugging time.
* chore: migrate src/Input to typescript
This moves `Keyboard`, `Mouse` and `Touchscreen` to TypeScript. We gain
some nice TS benefits here; by creating a type for all the keycodes we
support we can type the input args as that rather than `string` which
will hopefully save some users some debugging once we ship our TS types
in a future version.
* Remove from externs file
* Update utils/doclint/check_public_api/index.js
Co-Authored-By: Mathias Bynens <mathias@qiwi.be>
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
* chore: migrate src/ExecutionContext to TypeScript
I spent a while trying to decide on the best course of action for
typing the `evaluate` function.
Ideally I wanted to use generics so that as a user you could type
something like:
```
handle.evaluate<HTMLElement, number, boolean>((node, x) => true, 5)
```
And have TypeScript know the arguments of `node` and `x` based on those
generics. But I hit two problems with that:
* you have to have n overloads of `evaluate` to cope for as many number
of arguments as you can be bothered too (e.g. we'd need an overload for
1 arg, 2 args, 3 args, etc)
* I decided it's actually confusing because you don't know as a user
what those generics actually map to.
So in the end I went with one generic which is the return type of the
function:
```
handle.evaluate<boolean>((node, x) => true, 5)
```
And `node` and `x` get typed as `any` which means you can tell TS
yourself:
```
handle.evaluate<boolean>((node: HTMLElement, x: number) => true, 5)
```
I'd like to find a way to force that the arguments after the function do
match the arguments you've given (in the above example, TS would moan if
I swapped that `5` for `"foo"`), but I tried a few things and to be
honest the complexity of the types wasn't worth it, I don't think.
I'm very open to tweaking these but I'd rather ship this and tweak going
forwards rather than spend hours now tweaking. Once we ship these
typedefs and get feedback from the community I'm sure we can improve
them.
* chore: migrate src/JSHandle to TS
There's a few TODOs in here that all depend on typing the
`ExecutionContext.evaluateHandle` properly so that you can properly
declare what types you're expecting back. Once I've done that file (it's
next on my list) I will loop back and improve the types here, fixing
these TODOs.
* Fix doclint for {}
The codebase was incredibly inconsistent with the use of spacing around
curly braces, e.g.:
```
// this?
const a = {b: 1}
// or?
const a = { b: 1 }
```
This extended into import statements also. Google's styleguide is no
spacing, so we're going with that.
* chore: migrate src/helpers.ts to ESM
Doing this means we can avoid the global `types.d.ts` file and export
the interface via ESM instead.
I would ideally like to rewrite the helper module so that it doesn't
export all the functions under the `helper` namespace, but I'll leave
that for a separate PR to keep mechanical changes to one per PR and
easier to review.
* chore: migrate `src/USKeyboardLayout` to typescript
Don't think we need to expose the interface type for the keycodes so
I've left it local for now.
* retry windows unit tests
* chore: migrate `src/Connection` to TypeScript
This commit migrates `src/Connection` to TypeScript. It also changes its
exports to be ESM because TypeScript's support for exporting values to
use as types via CommonJS is poor (by design) and so rather than battle
that it made more sense to migrate the file to ESM.
The good news is that TypeScript is still outputting to `lib/` as
CommonJS, so the fact that we author in ESM is actually not a breaking
change at all.
So going forwards we will:
* migrate TS files to use ESM for importing and exporting
* continue to output to `lib/` as CommonJS
* continue to use CommonJS requires when in a `src/*.js` file
I'd also like to split `Connection.ts` into two; I think the
`CDPSession` class belongs in its own file, but I will do that in
another PR to avoid this one becoming bigger than it already is.
I also turned off `@typescript-eslint/no-use-before-define` as I don't
think it was adding value and Puppeteer's codebase seems to have a style
of declaring helper functions at the bottom which is fine by me.
Finally, I updated the DocLint tool so it knows of expected method
mismatches. It was either that or come up with a smart way to support
TypeScript generics in DocLint and given we don't want to use DocLint
that much longer that didn't feel worth it.
* Fix params being required
* chore: migrate `src/pipetransport` to typescript
Hit one bump in the fact that I want to share an interface across files.
TypeScript only lets you import/export these if you're using ESM, not
CommonJS. So the two options are:
- Migrate to ESM on a per file basis as we do this migration. This won't
affect the output as we output as CommonJS.
- Create a global `types.d.ts` file that we'll use and then migrate to
ESM after.
Right now I've gone for the second option in order to not introduce more
changes in one go. But if we end up finding we have lots of
interfaces/types/etc that we want modules to expose, we might decide
slowly introducing ESM might be a better way forwards.
* Update src/types.d.ts
Co-Authored-By: Mathias Bynens <mathias@qiwi.be>
This PR migrates the helper module to TypeScript. It's a bit of a bigger
change than others because I decided to move away from the helper class
with static methods and move towards a simpler set up where the module
is a bunch of functions. I still expose them under the `helper`
namespace to avoid this being a big change - we can update that later
when we migrate to ESM.
We do have to do some unfortunate wrangling of the promisify function.
Ideally I'd like to rely on the Node one (and the type defs) but that
doesn't work in Browserify land. I've stuck with the promisify in
`helper.ts` but pulled it into its own module to enable me to leave a
comment clarifying why we use it and the context. We can solve this with
a better web bundling story but that work is lower priority right now
than getting the `src/` directory into TypeScript.
This is a simple module but took a bit of work because:
* It wraps a Promise that can return basically anything. In a pure TS
codebase we'd solve these with generics, so you could do `new
TaskQueue<T>` where `T` will be what's returned from the queue, but
because we're calling that from JS we can't yet. I've left a TODO and
once we migrate the call sites to TS we can do a much better job than
the `void | any` type I've gone with for now.
* It was used in typedefs via `Puppeteer.TaskQueue`. I've removed that
entry from `externs.d.ts` in favour of importing it and using the type
directly. This does mean that we have imports that ESLint doesn't
realiase are actually used but I think this is better than maintaining
`externs.d.ts`.
This PR changes `src/Dialog.js` to `src/Dialog.ts` and rewrites
accordingly. Most of the changes are straight forward; the only
interesting one from a TS point of view is the `DialogType` enum. I
expose it again as `Dialog.Type` to avoid a breaking change.
This PR also exposed some bugs with our ESLint TypeScript settings and
applying the overrides, so I fixed those too.
I also updated our DocLint tool to work on TS source files over JS lib
files if they exist. This is the minimal change to keep the existing doc
system working as we're working on moving away from this system longer
term.
Fixes an edge case where Puppeteer looked for a Chromium revision when launching Firefox.
Allow appropriate Launcher to be instantiated when calling `Puppeteer.connect`.
Add an example of running Firefox.
This commit adds linting for `*.ts` files and loads up the recommended
list of TS rules from the ESLint TypeScript plugin. We can adjust the
exact rules overtime, but starting with the recommended list seems
sensible.
This commit moves `src/DeviceDescriptors` to be authored in TypeScript. This file was chosen due to its simplicity so that we can focus on getting a mixed JS/TS codebase playing nicely before migrating the more complex files.
The file itself was a bit odd: although the array of devices was exported via `module.exports` that was never referenced by any consumers; each device was also exported via `module.exports[name] = device` and that is how it's consumed. The Puppeteer docs suggest using it like so:
```js
puppeteer.devices['iPhone 6']
```
So instead of exporting the array and then setting a bunch of properties on that, we instead define the array and export an object of keys where each key is a device. This is a breaking change (see the footer for details).
Rather than export an object I'd much rather export a Map, but that would be a larger breaking change and I'm keen to avoid those for the time being.
Note that we have to use special TypeScript specific syntax for the export that enables it to work in a CommonJS codebase [1] but again I'd rather this than move to ESM at this time. TypeScript still outputs CommonJS into `lib/` as you would expect.
BREAKING CHANGE: We no longer export an array of devices, so any users relying on doing:
```js
puppeter.devices.forEach(...)
```
…will now see a breakage. The fix is to use `Object.{keys/entries/values}` to iterate instead.
[1]: https://www.typescriptlang.org/docs/handbook/modules.html#export--and-import--require
extract-zip removed support for callbacks and instead uses promises. Moreover, it has TypeScript support which allows us to remove the @types/extract-zip package.
This update allows downstream users to remove their installation of mkdirp, which uses a vulnerable version of minimist.
For more info, see https://github.com/maxogden/extract-zip/releases/tag/v2.0.0
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
Rather than use our own custom expect library, we can use expect from npm [1], which has an API almost identical to the one Puppeteer has, but with more options, better diffing, and is used by many in the community as it's the default assertions library that comes with Jest.
It's also thoroughly documented [2].
[1]: https://www.npmjs.com/package/expect
[2]: https://jestjs.io/docs/en/expect
This updates our `tsconfig.json` so it emits our JavaScript files as
well as type checking them. We compile into `./lib` which we then ship
in our npm package. The source code has moved from `./lib` into `./src`.
Because the `src/` directory is exclusively JS files, this change is a
no-op in terms of code functionality but is the first step towards being
able to replace `src/X.js` with `src/X.ts` in a way that allows us to
migrate incrementally.
The `lib` directory is gitignored, and the `src` directory is
npmignored. On `npm publish` we will now run `npm run tsc` in order to
generate the outputted code.
