* chore: run unit tests on node 10.15
We saw in https://github.com/puppeteer/puppeteer/issues/6548 that the
`fs.promises` module was experimental in Node <10.17 and as such we
introduced issues for users on 10.15.
Until we can drop Node v10 (it's EOL is 30-04-20201
https://github.com/nodejs/Release#release-schedule) we should run our
tests on an old Node 10 to avoid regressing in this area.
* chore: helper for importing fs safely
This PR updates the socket transport code to swap between a Node web
socket transport or a web one based on the `isNode` environment. It also
adds unit tests to the browser tests that show we can connect in a
browser.
* chore(agnostification): common/helper.ts
The `readProtocolStream` method uses `fs` only if you want to write to a
file. So we gate at the start of the function and ensure that if we got
given a path we are not in a Node environment.
The `Puppeteer` class had two concerns:
* connect to an existing browser
* launch a new browser
The first of those concerns is needed in all environments, but the
second is only needed in Node.
https://github.com/puppeteer/puppeteer/pull/6484 landing enabled us to
pull the `Puppeteer` class apart into two:
1. `Puppeteer` which hosts the behaviour for connecting to existing
browsers.
2. `PuppeteerNode`, which extends `Puppeteer` and also adds the ability
to launch a new browser.
This is a non-breaking change, because Node users will still get an
instance of a class with all the methods they expect, but it'll be a
`PuppeteerNode` rather than `Puppeteer`. I don't expect this to cause
people any issues.
We also now have new files that are effectively the entry points for
Puppeteer:
* `node.ts`: the main entry point for Puppeteer on Node.
* `web.ts`: the main entry point for Puppeteer on the web.
* `node-puppeteer-core.ts`: for those using puppeteer-core (which only
exists in Node, not on the web).
Launching headless with a relative `userDataDir` hangs on Windows. Fix by calling `path.resolve` (idempotent) to add an absolute path instead in `defaultArgs`.
Issues: #3453
The `Launcher` class was serving two purposes:
1. Launch browsers
2. Connect to browsers
Number 1) only needs to be done in Node land, but 2) is agnostic; in a
browser version of Puppeteer we'll need the ability to connect over a
websocket to send commands back and forth.
As part of the agnostification work we needed to split the `Launcher` up
so that the connection part can be made agnostic. Additionally, I
removed dependencies on `https`, `http` and `URL` from Node, instead
leaning on fetch (via `node-fetch` if in Node land) and the browser
`URL` API (which was added to Node in Node 10).
* chore(agnostify): Create Node and Web initializer.
This PR splits `initialize.ts` into two files, one for web, and one for
Node. The Node initializer requires much more information as it needs to
know which browser(s) to download and where to store them, whereas the
web one does not.
A future PR that I'm working on will tidy up `src/common/Puppeteer.ts`
(as it contains a lot of Node specific logic around downloading and
installing browsers), but this change enables us to stop the browser
bundle attempting to use the `pkg-dir` dependency, which wouldn't work
within a browser, as well as keeping the size of the PRs down and
avoiding one mammoth PR.
This commit adds a new built-in handler for querying by accessible name and role (#6307).
Support for waitForSelector will be added in a follow-up commit.
In `src/common` we now use `fs.promises.X` which we can dynamically
`import`. In a browser environment this code will never run because it's
gated on `isNode` (in a future PR we will add tree-shaking to the bundle
step such that this code is eliminated). By using `import`, we ensure
TypeScript still can track types and give good type information.
In `src/node` we continue to use `util.promisify` but that's not a
concern as that code explicitly is never run in the browser.
This commit changes the internal representation of query handlers to contain Puppeteer-level code instead of page functions.
The interface `CustomQueryHandler` is introduced for user-defined query handlers. When a `CustomQueryHandler` is registered using `registerCustomQueryHandler` a corresponding Puppeteer-level handler is created through `makeQueryHandler` by wrapping the page functions as appropriate.
The internal query handlers (defined by the interface `QueryHandler`) contain two new functions: `waitFor` and `queryAllArray`.
- `waitFor` allows page-based handlers to make use of the `WaitTask`-backed implementation in `DOMWorld`, whereas purely Puppeteer-based handlers can define an alternative approach instead.
- `queryAllArray` is similar to `queryAll` but with a slightly different interface; it returns a `JSHandle` to an array with the results as opposed to an array of `ElementHandle`. It is used by `$$eval`.
After this change, we can introduce built-in query handlers that are not executed in the page context (#6307).
The logic for waitForXPath and waitForSelector is currently very tightly coupled. This commit tries to untangle that relationship. This is the first step towards introducing built-in query handlers that are not executed in the page context (#6307).
Debug message clarifying the problem like `Issue 1060080: Event Network.loadingFinished is not sent when a frame is loaded from another domain`: https://bugs.chromium.org/p/chromium/issues/detail?id=1060080).
It can help users to identify problem with OOPIF easier without digging into the CDP protocol implementation like in the bug `1060080` mentioned above.
To reproduce:
1. Run `DEBUG="puppeteer:frame" NODE_PATH=../ node examples/oopif.js`.
2. Verify the output contains the debug message:
`
puppeteer:frame The frame '...' moved to another session. Out-of-proccess
iframes (OOPIF) are not supported by Puppeteer yet.
https://github.com/puppeteer/puppeteer/issues/2548
`
* roll Chromium to version 86, r800071
* roll Chrome DevTools protocol version to 0.0.799653
* fix HTTPRequest.continue after
* CDP: accept post data in the binary form in Fetch.continueRequest.
https://chromium-review.googlesource.com/c/chromium/src/+/2315239
* update new-docs
By adding support for an environment variable `PUPPETEER_DOWNLOAD_PATH` it is possible to support downloading the browser binaries into a folder outside the `node_modules` folder. This makes it possible to preserve previously downloaded binaries in order to skip downloading them again.
This commit also removes our own custom type for defining the vision
deficiencies and uses the protocol's type. Now we generate docs for
those we get the docs generated for free for these. This is better than
us duplicating values for types in doc comments and having them become
outdated. If we use the protocol types directly then we ensure we're up
to date and in-sync.
Long term the docs will also link to the devtools-protocol viewer.
This corresponds to Chromium 85.0.4182.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure
https://crrev.com/c/2231445
- [FlexNG] Enable FlexNG by default
https://crrev.com/c/2216595Closes#6151.
The Node debug library uses the `DEBUG` environment variable to
configure what gets logged. Our browser version just logs everything;
this commit changes it to look for `window.__PUPPETEER_DEBUG` and matches the behaviour accordingly:
* If the value is not set, nothing is logged.
* If the value is set to `*` everything is logged.
* If the value is set to a string `foo`, messages with that prefix are
logged.
* If the value is set to a string ending in `*`, e.g. `foo*`, messages
with prefixes that start with `foo` are logged.
Also took the opportunity to pull out the PDF types into their own file
to clear up `Page.ts` slightly and give the PDF code a more natural
place to live.
* chore: vendor Mitt into src/common/third-party
As discussed in #6203 we need to vendor our common dependencies in so
that when we ship an ESM build all imports point to file paths and do
not rely on Node resolution (e.g. a browser does not understand `import
mitt from 'mitt'`).
* chore: enforce file extensions on imports
To make our output agnostic it should include file extensions in the
output, as per the ESM spec. It's a bit odd for Node packages but makes
it easier to publish a browser build.
* feat(chromium): roll Chromium to r768783
* fix: update unit test for crrev:2135046
* chore: update devtools-protocol revision
Co-authored-by: Changhao Han <changhaohan@chromium.org>
* chore: Use devtools-protocol package
Rather than maintain our own protocol we can instead use the devtools-protocol package and pin it to the version of Chromium that Puppeteer is shipping with.
The only changes are naming changes between the bespoke protocol that Puppeteer created and the devtools-protocol one.
Now the async hooks helper is gone api.ts was only used by the coverage
tools and by doclint.
DocLint is nearing the end of its lifespan with the TSDoc work, so I
focused on how best to define a list of modules for the coverage
tooling. They define an object of classes, and the path to that module.
They need the full path because we also check if the module exports any
events that need to be emitted - the coverage tool asserts that the
emitting of those events is also tested.
It's not _great_ that DocLint relies on a constant defined in the
coverage utils, but it should only be this way for a short period of
time and no one is actively working on DocLint (bar the effort to remove
it) so I don't think this is worth worrying about.
This change also broke the DocLint tests; based on the fact that DocLint is on its way out it doesn't feel worth fixing the tests, so this commit also removes them.
