Just some code reorder. We had a describe between it calls. I'm moving that describe to the end
Co-authored-by: Jack Franklin <jacktfranklin@chromium.org>
* chore: enforce pinned dependencies
Because we don't check our `package-lock.json` in, we can end up with
different versions installed locally vs CI, or even two devs having
different versions. Let's pin and enforce we pin every version to
avoid this.
This PR updates some code to remove constant ESLint warnings. It also
upgrades those warnings to errors - so that they have to be resolved
as part of the PR, rather than landing as a warning and causing noise.
Fixes#7229.
We're seeing odd failures with Prettier on some CI branches; my hunch is that they are installing different versions of the package and therefore getting formatting conflicts. This PR updates them all and pins them to specific versions - something we should probably consider generally, or remove our `package-lock.json` from the gitignore.
The existing behavior is expected to be unchanged as the value defaults to true.
Adding such option would allow user to skip the initial wait.
Issue: #3630
* fix: make `$` and `$$` selectors generic
This means, much like TS's in built `querySelector` type, you can now do:
```ts
const listItems = page.$$<HTMLLIElement>('ul li');
```
And/or:
```ts
const h2 = page.$<HTMLHeadingElement>('h2');
```
And the return value will be of type `ElementHandle<T>|null`, where `T`
is the type you provided. By default `T` is an `Element`, so you don't
have to provide this if you don't care as a consumer about the exact
type you get back.
* chore: fix test assertions
* feat(chromium): roll Chromium to r856583
This corresponds to Chromium 90.0.4427.0
This roll includes:
- Add sourceScheme, sourcePort, and sameParty to DevTools backend (https://crbug.com/1170548, https://crbug.com/1142606)
We were blocked on doing this because API Extractor didn't support it,
but now it does, so we can bump TS and the API tooling in one go. None
of the breaking changes in TS4 cause us any issues.
During the migration to TS we changed `jsonValue` so it returned
`<Record<string, unknown>>`. This is only true if all the JSON values it
returns are objects; but it could return an array, a string, a number,
etc. Therefore we make the type generic, setting the default to
`unknown`, so the user has control over the type.
This corresponds to Chromium 90.0.4403.0
This roll includes:
- Cut screenshot by ViewPort size, not position (crrev.com/c/2643792)
BREAKING CHANGE:
- `page.screenshot` cuts screenshot content by the ViewPort size, not ViewPort position.
This corresponds to Chromium 89.0.4389.0.
This roll includes:
- Add `SameParty` attribute to cookies
https://crrev.com/c/2598846
- Anchor `target=_blank` implies `rel=noopener`
https://crrev.com/c/1630010
- Don’t expect ignored elements in the AXTree
https://crrev.com/c/2505362
BREAKING CHANGE: The built-in `aria/` selector query handler doesn’t return ignored elements anymore.
Issue: #6758
This patch enables more tests for Firefox. These tests are enabled in Mozilla's CI for Firefox. The extra error handling here prevents hangs in the test harness in that environment.
The `Puppeteer` class had two concerns:
* connect to an existing browser
* launch a new browser
The first of those concerns is needed in all environments, but the
second is only needed in Node.
https://github.com/puppeteer/puppeteer/pull/6484 landing enabled us to
pull the `Puppeteer` class apart into two:
1. `Puppeteer` which hosts the behaviour for connecting to existing
browsers.
2. `PuppeteerNode`, which extends `Puppeteer` and also adds the ability
to launch a new browser.
This is a non-breaking change, because Node users will still get an
instance of a class with all the methods they expect, but it'll be a
`PuppeteerNode` rather than `Puppeteer`. I don't expect this to cause
people any issues.
We also now have new files that are effectively the entry points for
Puppeteer:
* `node.ts`: the main entry point for Puppeteer on Node.
* `web.ts`: the main entry point for Puppeteer on the web.
* `node-puppeteer-core.ts`: for those using puppeteer-core (which only
exists in Node, not on the web).
Launching headless with a relative `userDataDir` hangs on Windows. Fix by calling `path.resolve` (idempotent) to add an absolute path instead in `defaultArgs`.
Issues: #3453
The `Launcher` class was serving two purposes:
1. Launch browsers
2. Connect to browsers
Number 1) only needs to be done in Node land, but 2) is agnostic; in a
browser version of Puppeteer we'll need the ability to connect over a
websocket to send commands back and forth.
As part of the agnostification work we needed to split the `Launcher` up
so that the connection part can be made agnostic. Additionally, I
removed dependencies on `https`, `http` and `URL` from Node, instead
leaning on fetch (via `node-fetch` if in Node land) and the browser
`URL` API (which was added to Node in Node 10).
This commit adds a new built-in handler for querying by accessible name and role (#6307).
Support for waitForSelector will be added in a follow-up commit.
In `src/common` we now use `fs.promises.X` which we can dynamically
`import`. In a browser environment this code will never run because it's
gated on `isNode` (in a future PR we will add tree-shaking to the bundle
step such that this code is eliminated). By using `import`, we ensure
TypeScript still can track types and give good type information.
In `src/node` we continue to use `util.promisify` but that's not a
concern as that code explicitly is never run in the browser.
This commit changes the internal representation of query handlers to contain Puppeteer-level code instead of page functions.
The interface `CustomQueryHandler` is introduced for user-defined query handlers. When a `CustomQueryHandler` is registered using `registerCustomQueryHandler` a corresponding Puppeteer-level handler is created through `makeQueryHandler` by wrapping the page functions as appropriate.
The internal query handlers (defined by the interface `QueryHandler`) contain two new functions: `waitFor` and `queryAllArray`.
- `waitFor` allows page-based handlers to make use of the `WaitTask`-backed implementation in `DOMWorld`, whereas purely Puppeteer-based handlers can define an alternative approach instead.
- `queryAllArray` is similar to `queryAll` but with a slightly different interface; it returns a `JSHandle` to an array with the results as opposed to an array of `ElementHandle`. It is used by `$$eval`.
After this change, we can introduce built-in query handlers that are not executed in the page context (#6307).
The logic for waitForXPath and waitForSelector is currently very tightly coupled. This commit tries to untangle that relationship. This is the first step towards introducing built-in query handlers that are not executed in the page context (#6307).
This corresponds to Chromium 85.0.4182.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure
https://crrev.com/c/2231445
- [FlexNG] Enable FlexNG by default
https://crrev.com/c/2216595Closes#6151.
* chore: vendor Mitt into src/common/third-party
As discussed in #6203 we need to vendor our common dependencies in so
that when we ship an ESM build all imports point to file paths and do
not rely on Node resolution (e.g. a browser does not understand `import
mitt from 'mitt'`).
* chore: enforce file extensions on imports
To make our output agnostic it should include file extensions in the
output, as per the ESM spec. It's a bit odd for Node packages but makes
it easier to publish a browser build.
* feat(chromium): roll Chromium to r768783
* fix: update unit test for crrev:2135046
* chore: update devtools-protocol revision
Co-authored-by: Changhao Han <changhaohan@chromium.org>
Now the async hooks helper is gone api.ts was only used by the coverage
tools and by doclint.
DocLint is nearing the end of its lifespan with the TSDoc work, so I
focused on how best to define a list of modules for the coverage
tooling. They define an object of classes, and the path to that module.
They need the full path because we also check if the module exports any
events that need to be emitted - the coverage tool asserts that the
emitting of those events is also tested.
It's not _great_ that DocLint relies on a constant defined in the
coverage utils, but it should only be this way for a short period of
time and no one is actively working on DocLint (bar the effort to remove
it) so I don't think this is worth worrying about.
This change also broke the DocLint tests; based on the fact that DocLint is on its way out it doesn't feel worth fixing the tests, so this commit also removes them.
* chore: remove `installAsyncStackHooks` helper
This code was written when browsers/Node didn't support errors in async
functions very well. They now do a much better job of this, so we can
lose the additonal complexity from our codebase and leave it to the host
environment :)
* lazy launcher is private
* remove async stack test
This pulls in the types (based on the DefinitelyTyped repo) for
`page.$eval` (and the `$eval` method on other classes). The `$eval`
method is quite hard to type due to the way we wrap and unwrap
ElementHandles that are passed to / returned from the `pageFunction`
that users provide.
Longer term we can improve the types by providing type overloads as
DefinitelyTyped does but I've deferred that for now (see the `TODO` in
the code for more details).
This change started as a small change to pull types from DefinitelyTyped over to
Puppeteer for the `evaluateHandle` function but instead ended up also fixing
what looks to be a long standing issue with our existing documentation.
`evaluateHandle` can in fact return an `ElementHandle` rather than a `JSHandle`.
Note that `ElementHandle` extends `JSHandle` so whilst the docs are technically
correct (all ElementHandles are JSHandles) it's confusing because JSHandles
don't have methods like `click` on them, but ElementHandles do.
if you return something that is an HTML element:
```
const button = page.evaluateHandle(() => document.querySelector('button'));
// this is an ElementHandle, not a JSHandle
```
Therefore I've updated the original docs and added a large explanation to the
TSDoc for `page.evaluateHandle`.
In TypeScript land we'll assume the function will return a `JSHandle` but you
can tell TS otherwise via the generic argument, which can only be `JSHandle`
(the default) or `ElementHandle`:
```
const button = page.evaluateHandle<ElementHandle>(() => document.querySelector('button'));
```
The headful one I'm permanently skipping as I don't know what the issue is and I can't debug without getting my hands on a Windows machine. If anyone has one or is able to help, that'd be great!
The other I'm deferring another month and will ping the FF folks :)
This CL migrates all the tests to TypeScript. The main benefits of this is that we start consuming our TypeScript definitions and therefore find errors in them. The act of migrating found some bugs in our definitions and now we can be sure to avoid them going forwards.
You'll notice the addition of some `TODO`s in the code; I didn't want this CL to get any bigger than it already is but I intend to follow those up once this lands. It's mostly figuring out how to extend the `expect` types with our `toBeGolden` helpers and some other slight confusions with types that the tests exposed.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
As far as I can tell these became irrelevant as of v1.15 which added
`puppeteer.errors` and `puppeteer.devices [1]. This is a breaking change
but one that's easily mitigated. We've said that we don't consider
changes to our folder/file structure a breaking change, but we can't
really do that if we have these two top level files that we've
documented.
[1]: e3abb0aa32 (diff-522b24108d7446af4c59873472a90444)
These files will be used by both the web and node versions of Puppeteer.
Another name for this might be "core" but I don't want to cause
confusion with the puppeteer-core package that we publish at the moment.
Fix child process killing when the parent process SIGINTs.
If you `ctrl + c` the Puppeteer parent process, we would sometimes not properly handle killing of the child processes. This would then leave child processes behind, with running Chromium instances. This in turn could block Puppeteer from launching again and results in
cryptic errors.
Instead of using the generic `process.kill` with the process id (which for some reason is negative the pid, which I don't get), we can kill the child process directly by calling `proc.kill`.
Fixes#5729.
Fixes#4796.
Fixes#4963.
Fixes#4333.
Fixes#1825.
* chore: remove "Extracting..." log message
Fixes#5741.
* test: support extra Launcher options and skips
The extra Launcher options and skipping conditions enable
unit tests to be run more easily by third-parties, e.g.
browser vendors that are interested in Puppeteer support.
Extra Launcher options were previously removed as part of
switching away from the custom test harness.
* test: enable more tests for Firefox
We deferred this initially because our Windows CI built wasn't stable
and so debugging this was hard. It's now much more stable so let's push
this back a month but at the same time I'll reach out to the Moz folks
as it should be easier to debug reliably now CI is stable on Windows.
* Don't use expect within Promises (#5466)
If a call to expect fails within a Promise it will not
be resolved, and causing the test to crash.
The patch aligns the code similar to what is used by all
the other tests.
* chore: fix invalid SSL assertion on Catalina
The error Chrome gives with an invalid cert changes between older Mac
versions and Catalina as detailed here:
https://support.google.com/chrome/thread/18125056?hl=en.
This PR changes Travis to run Catalina (and we think most devs run up to
date OS versions) so this fix ensures the test behaviour is consistent
locally and on Travis.
For those on older Mac versions I've left a comment by the tests to
hopefully save them debugging!
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
The coverage utils depend on `src/api.ts` being up to date and pointing to the right modules. If they aren't, you would get a cryptic error on CI:
```
1) "before all" hook in "{root}":
TypeError: Cannot read property 'prototype' of undefined
at traceAPICoverage (test/coverage-utils.js:40:54)
at Context.before (test/coverage-utils.js:103:7)
2) "after all" hook in "{root}":
TypeError: Cannot read property 'stop' of undefined
at Context.after (test/mocha-utils.js:168:22)
```
This change logs a clearer error that highlights the missing class and exits, so it's much easier to realise what's gone wrong.
Ideally the coverage wouldn't need a hardcoded list of sources, but until then this will help spot this error in the future.
This corresponds to Chromium 83.0.4103.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure https://crrev.com/c/2122809
When this test was failing, it would cause no future tests to run. This
was because the `expect` call within the `page.on` callback would throw
an error, and that would trigger a unhandled promise rejection that
caused the test framework to stop.
The fundamental issue here is making `expect` calls within callbacks.
They are brittle due to the fact that they throw, and the test framework
won't catch it, but also because you have no guarantee that they will
run. If the callback is never executed you dont' know about it.
Although it's slightly more code, using a stub is the way to do this.
Not only can we assert that the stub was called, we can make synchronous
`expect` calls that Mocha will pick up properly if they fail.
Before this change, running the tests (and making it fail on purpose)
would cause all test execution to stop:
```
> puppeteer@3.0.4-post unit /Users/jacktfranklin/src/puppeteer
> mocha --config mocha-config/puppeteer-unit-tests.js
.(node:69580) UnhandledPromiseRejectionWarning: Error: expect(received).toBe(expected) // Object.is equality
Expected: "yes."
Received: ""
at Page.<anonymous> (/Users/jacktfranklin/src/puppeteer/test/dialog.spec.js:42:37)
[snip]
(node:69580) UnhandledPromiseRejectionWarning: Unhandled promise rejection ... [snip]
```
But with this change, the rest of the tests run:
```
> puppeteer@3.0.4-post unit /Users/jacktfranklin/src/puppeteer
> mocha --config mocha-config/puppeteer-unit-tests.js
Page.Events.Dialog
✓ should fire
1) should allow accepting prompts
✓ should dismiss the prompt
2 passing (2s)
1 failing
1) Page.Events.Dialog
should allow accepting prompts:
Error: expect(received).toBe(expected) // Object.is equality
Expected: "yes."
Received: ""
at Context.<anonymous> (test/dialog.spec.js:53:35)
at processTicksAndRejections (internal/process/task_queues.js:94:5)
```
This is much better because one failing test now doesn't stop the rest
of the test suite.
This probably isn't the only instance of this in the codebase so I
propose as we encounter them we fix them usng this commit as the
template.
* Warn when given unsupported product name.
Fixes#5844.
This change means when a user launches Puppeteer with a product name
that is not supported (which at the time of this commit means it's not
`firefox` or `chrome) we will warn them about it.
Decided on just a warning vs an error because the current behaviour is
that we fallback to launching Chrome and I don't think this warrants a
breaking change.
* chore: update how we track coverage during unit tests
The old method of tracking coverage was causing issues. If a test failed
on CI, that test's failure would be lost because the test failing would
in turn cause the coverage to fail, but the `process.exit(1)` in the
coverage code caused Mocha to not output anything useful.
Instead the coverage checker now:
* tracks the coverage in memory in a Map (this hasn't changed)
* after all tests, writes that to disk in test/coverage.json (which is
gitignored)
* we then run a single Mocha test that asserts every method was called.
This means if the test run fails, the build will fail and give the error
about that test run, and that output won't be lost when the coverage
then fails too.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
The codebase was incredibly inconsistent with the use of spacing around
curly braces, e.g.:
```
// this?
const a = {b: 1}
// or?
const a = { b: 1 }
```
This extended into import statements also. Google's styleguide is no
spacing, so we're going with that.
* chore: Add Windows to Travis
This commit runs the unit tests on Windows.
There are two tests failing on Windows that we skip.
I spoke to Mathias B and we agreed to
defer debugging this for now in favour of getting tests running on
Windows. But we didn't want to ignore it forever, hence giving the test
a date at which it will start to fail.
This commit updates all the non-Puppeteer unit tests to run using Mocha and then deletes the custom test runner framework from this repository. The documentation has also been updated.
Rather than maintain our own test runner we should instead lean on the community and use Mocha which is very popular and also our test runner of choice in DevTools too.
Note that this commit doesn't remove the TestRunner source as it's still used for other unit tests, but they will be updated in a future PR and then we can remove the TestRunner.
The main bulk of this PR is updating the tests as the old TestRunner passed in contextual data via the `it` function callback whereas Mocha does not, so we introduce some helpers for the tests to make it easier.
See the large code comment in the diff for a full explanation but we can't rely on the functions being referentially equivalent so instead we test the behaviour in duplicate tests across the deprecated method and the new method.
They fail because cookies in Firefox return a `sameSite` key which the tests don't expect.
This is a solution that at least gets the Travis Firefox build (hopefully!) green again. Longer term it'd be great to allow the assertion to change based on the browser, rather than skip these tests entirely.
Rather than use our own custom expect library, we can use expect from npm [1], which has an API almost identical to the one Puppeteer has, but with more options, better diffing, and is used by many in the community as it's the default assertions library that comes with Jest.
It's also thoroughly documented [2].
[1]: https://www.npmjs.com/package/expect
[2]: https://jestjs.io/docs/en/expect
* (feat) Add option to fetch Firefox Nightly
Add Firefox support to BrowserFetcher and the install script.
By default, the latest Firefox Nightly is downloaded
directly from archive.mozilla.org (dmg, tar.bz2 and zip)
This also required changes that impact `puppeteer.launch()`
and `puppeteer.executablePath()`
Fixes#5151
* Update docs/api.md
Co-Authored-By: Mathias Bynens <mathias@qiwi.be>
* Clean up revision promise
* Improve error handling in revision check
* Remove matchAll
* Use explicit octal mode
* Update .gitignore
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
* feat: Set which browser to launch via PUPPETEER_PRODUCT
This change introduces a PUPPETEER_PRODUCT environment
variable as a first step toward using Puppeteer with
many different browsers. Setting PUPPETEER_PRODUCT=firefox, for
example, enables Firefox-specific Launcher settings.
The state is also exposed as `puppeteer.product` in the API
to support adding other product-specific behaviour as needed.
The bulk of the change is a refactoring in Launcher
to decouple generic browser start-up from product-specific
configuration.
Respecting the puppeteer-core restriction for PUPPETEER_
environment variables, lazily instantiate the Launcher
based on a `product` Puppeteer.launch option, if available.
* test: Distinguish Juggler unit tests from Firefox
The funit script is renamed to fjunit (j for Juggler, which is
used only by the experimental puppeteer-firefox package.
In contrast, the funit script now refers to running Puppeteer
unit tests against the main puppeteer package with Firefox.
To do so with Firefox Nightly, run:
`BINARY=path/to/firefox npm run funit`
A number of changes in this patch make it easier to run
Puppeteer unit tests in Mozilla's CI.
Node.js v6 was end-of-life'd in April, 2019, with AWS Lambda prohibiting updaets to the Node.js v6 runtime since June 30, 2019.
This makes it quite safe for us to remove the Node 6 support from the repository.
We'd like to pass an abortion signal inside Helper.waitForEvent in order to interrupt it when browser/page closes. Several approaches have been considered:
1. Pass CDPSession instance as a another parameter to the helper method and listen to Disconnected event on it. It would introduce undesired dependency on the session object.
2. Listen to the CDPSession closure at the call sites (e.g. waitForRequest) and pass an abortion promise which would be fulfilled when such event is fired. The listeners would have to be removed from the session on successful completion of waitForEvent so we'd have to pass some kind of DisposablePromise which would be disposed during cleanup. Such parameter looked somewhat hairy.
3. Create DisconnectPromise on CDPSession. One potential risk with that is all chained promises would hang around until the event is fired which might inadvertently cause memory leaks. On the other hand, adding such promise to Promise.race will remove dependency as soon as the race is finished. So this is the approach we're taking with one tweak: the promise is created locally inside Page.
Ideally the disconnectPromise would throw when the session is closed but it may lead to uncaught promise errors if all chained promises are resolved, to avoid that the promise is resolved with an Error and Helper.waitForEvent throws it later.
Fix#4733
Our magical node6 transpiler can't handle object spreads :(
Drive-by: use "includes" instead of "startsWith" for devtools URL
since I remember that it used to be "chrome-devtools://", and somehow
I saw it as "devtools://" somewhere.
This roll includes:
- https://crrev.com/681997 - Turn on default SiteInstance by default.
The SiteInstance by default was breaking "devtools: true" option, so
there's a new feature we disable now by default.
This keeps pressuring us towards OOPIF support since that's an
inevitable future.
`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.
As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.
This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
This patch:
- updates Flakiness Dashboard format to define version per-build
and to pass COMMIT information
- drops the README.md generation - we'll move on to a designated flakiness
dashboard viewer
We used to track API Coverage for public events, but this was regressed in the refactoring that
introduced `//lib/Events.js`.
This patch:
- Brings back API Coverage for events
- Combines all coverage-generated tests into a single one. This way
we can generate less data for flakiness dashboard.
- fix `FLAKINESS_DASHBOARD_BUILD_URL` to point to a task instead of a build
- do not pretty-print `dashboard.json` when serializing flakiness results
- filter out 'COVERAGE' test(s) so that they don't add up to `dashboard.json` payload. These are useless
- validate certain important options of flakiness dashboard
- more logging to STDOUT to actually say which repo and what branch is getting used
- enhance commit message with a build URL
- use a more compact format for JSON. For 100 runs of 700 tests it yields 21MB json instead of 23MB.
- bump default builds number to 100
Make eval test resilient to variations in error message format between browsers.This will make the test pass without alternations in WebKit as well as Crhrome and Firefox.
This patch introduces a dashboard that records test results and
uploads them to https://github.com/aslushnikov/puppeteer-flakiness-dashboard
Since many bots might push results in parallel, each bot pushes
results to its own git branch.
FlakinessDashboard also generates a simple README.md with a flakiness
summary. If this proves to be not enough, we can build a website that
fetches flakiness data and renders it nicely.
This patch introduces a page.waitForFileChooser() method
that adds a watchdog to wait for file chooser dialogs.
This lets Puppeteer users to capture file chooser requests
and fulfill/cancel them if necessary.
Fixes#2946
Don't ignore stdout and stderr when using pipe for remote debugging and dumpio is true. In that case puppeteer process connects to the stdout/stderr streams of the child process and it will not hang.
This patch adds new TestRunner options:
- `disableTimeoutWhenInspectorIsEnabled` - disable test timeout if
testrunner detects enabled inspector.
- `breakOnFailure` - if testrunner should terminate test running on
first test failure
Chrome has a set of component extensions - e.g. CryptoTokenExtension
that helps with 2FA.
These extensions are loaded regardless of the `--disable-extensions`
flag we already pass. To disable these extensions, we need to pass additional
`--disable-component-extensions-with-background-pages` flag.
Fix#4300
This patch teaches page.evaluate to do 1 hop instead of 2 hops.
As a result, things such as `page.select` will not throw an
unfortunate exception when they schedule a navigation.
Fix#4537
* feat(chromium): roll Chromium to r665405
This roll includes:
- https://crrev.com/665226 - DevTools: make interception respect cross-process frame boundaries
This fixes page loading with dynamic OOPIFs - test is added.
Fix#4442
* fix lint
This roll includes:
- [inspector_protocol:8ec18cf](8ec18cf088) Support STRING16 in the template when converting CBOR map keys
to protocol::Value.
- [inspector_protocol:37518ac](37518ac421) fix parsing of the last ASCII character
This fixes protocol handling of UTF8 in both V8 and Chromium.
Fixes#4443.
Going from `AXNode` -> `ElementHandle` is turning out to be controversial.
This patch instead adds a way to go from `ElementHandle` -> `AXNode`. If the API looks good, I'll add it into Firefox as well.
References #3641
This roll includes:
- https://crrev.com/653809 - FrameLoader: ignore failing provisional loads entirely
- https://crrev.com/654750 - DevTools: make sure Network.requestWillBeSent is emitted on time for sync xhrs
The FrameLoader patch is the reason behind the test change. It's
actually desirable to fail frame navigation if the frame detaches - and
that's consistent with Firefox.
Fixes#4337
These getters are introduced as a more convenient substitute for
a `require('puppeteer/Errors')` and
`require('puppeteer/DeviceDescriptors')`.
This way we can make cross-browser story nicer - a single require
of `puppeteer` or `puppeteer-firefox` fully defines Puppeteer
environment.
In certain cases inline element children might be positioned
outside of viewport.
In this case, we should intersect all content quads with viewport
before we pick one to click into.
Fixes#4274.
In case of multiple sessions to the same target, there's a race between
sessions to create a secondary isolated world. As a result, we might
end up having 2 execution contexts created for the needs of the
secondary isolated world.
This patch starts handling this race gracefully: instead of crashing,
we can use either of the execution contexts and ignore the rest.
Notably, the same race condition might happen if page reloads itself
in-between the calls to `page.addEvaluateOnNewDocument` and
`page.createIsolatedWorld`.
Fixes#4197.
This patch:
- refactors `NetworkManager`/`FrameManager` so that they enable all the
relevant domains themselves. This is a preparation for OOPIF support and
migration onto fetch domain.
- moves `networkManager` ownership into `FrameManager`. This way it's clear who owns what.
- stops enabling Security domain: it saves quite some traffic over
websocket since it no longer sends annoying "SecurityStateChanged" events.
Instead, use `Security.setIgnoreCertificateErrors` method.
- consolidates network cache state in network manager. This even fixes a
bug with caching and request interception interop.
This patch makes sure header overrides in request interception are
functioning as expected.
Drive-by: teach test server to use utf-8 charset header for text files.
This patch implements `page.setCookie()`, `page.deleteCookie()` and
`page.cookies()` and doubles the test coverage for cookies so that
we can feel safer on cross-browser compatibility.
This patch:
- implements Response.buffer() and other methods
- splits out relevant tests into a separate test suites
- implements `testServer.enableGzip()` method to optionally gzip
certain routes in tests
- adds tests to make sure `Response.text()` returns expected results
for binary and compressed responses.
When we started working on Puppeteer-Firefox, we forked original
Puppeteer testsuite.
This patch concludes the effort to merge testsuites back together.
Fixes#3889
Juggler now implements the same "flatten" protocol as CDP.
This patch:
* copies `Connection.js` from original Puppeteer (with a few renames, e.g. `CDPSesssion` -> `JugglerSession`).
* migrates code to support protocol-level sessions
Support browser target.
Drive-by: switch over to a more devtools'ish protocol:
- use `targetId` instead of `pageId` everywhere
- use target events instead of tab events
Firefox used to have a bug that prevented adding iframes with the
same URL as main frame. In this case, we used the EMPTY_PAGE2
so that it doesn't clash with top-level EMPTY_PAGE.
The bug seems to be fixed after we rolled onto a fresh Firefox;
there's no more need for this test asset.
This patch refactors Puppeteer-Firefox code to declare public
API in `/lib/api.js` and use it to setup async stack hooks
over the public API method calls.
This patch:
* unifies assets between tests
* enables a few puppeteer tests on Puppeteer-Firefox
Drive-by: beautify failing output of `expect.toEqual` matcher.
References #3889
This patch:
- introduces new testRunner methods `addTestDSL` and `addSuiteDSL`
to add annotated test / suite.
- introduces new test/suite declaration methods: `it_fails_ffox` and
`describe_fails_ffox`. These are equal to `it`/`describe` for chromium
tests and to `xit`/`xdescribe` for firefox.
- marks all unsupported tests with `it_fails_ffox`
- adds a new command-line flag `'--firefox-status'` to `//test/test.js`.
This flag dumps current amount of tests that are intentionally skipped
for Firefox.
End goal: get rid of all `it_fails_ffox` and `describe_fails_ffox`
tests.
Drive-By: remove cookie tests "afterEach" hook that was removing
cookies - it's not needed any more since every test is run in a
designated browser context.
References #3889
This aligns all Puppeteer tests so that they never throw the
"UnhandledPromiseRejection" when run with Puppeteer-Firefox.
With this patch, the `npm run funit` passes 275 of 460 Puppeteer tests.
References #3889.
This patch:
- adds support to `FFOX` env variable for Puppeteer testsuite
- install Firefox preferences when running tests with custom firefox
executable
References #3889
This patch:
- changes Puppeteer-Firefox plumbing of defaultBrowserOptions to align
with the way we do it for Puppeteer.
- plumbs puppeeteer-dependent Errors and DeviceDescriptors down to every
test.
- unifies a few tests between Puppeteer-Firefox and Puppeteer.
**Note:** in future, we should expose errors as `puppeteer.errors` and
device descriptors as `puppeteer.devices` to make it easy to pass around
Puppeteer/Puppeteer-Firefox instance.
References #3889.
Certain Puppeteer methods do expose the inner browser - e.g.
`browser.version()` depends on the browser we run.
Split out these tests into a vendor-specific test suites.
References #3889
Introduce a `npm run funit` script that runs puppeteer tests
against Puppeteer-Firefox.
Next steps:
- bring Puppeteer-Firefox unique tests to Puppeteer
- skip failing tests and run Puppeteer-Firefox on CI
- work through tests to pass them all with Puppeteer-Firefox
Method `page.setDefaultTimeout` overrides default 30 seconds timeout
for all `page.waitFor*` methods, including navigation and waiting
for selectors.
Fix#3319.
This patch starts executing frame.waitForSelector and frame.waitForXPath
in secondary world. As a result, websites that mutate page global
context (e.g. removing global MutationObserver) don't break Puppeteer's
behavior.
Fixes#609
`page.waitForSelector` should return `null` if waiting for `hidden:
true` and there's no matching node in DOM.
Before this patch, `page.waitForSelector` would return some JSHandle
pointing to boolean value.
Introduce `//lib/api.js` that declares a list of publicly exposed
classes.
The `//lib/api.js` list superceedes dynamic `helper.tracePublicAPI()` calls
and is used in the following places:
- [ASYNC STACKS]: generate "async stacks" for publicy exposed API in `//index.js`
- [COVERAGE]: move coverage support from `//lib/helper` to `//test/utils`
- [DOCLINT]: get rid of 'exluded classes' hardcoded list
This will help us to re-use our coverage and doclint infrastructure
for Puppeteer-Firefox.
Drive-By: it turns out we didn't run coverage for `SecurityDetails`
class, so we lack coverage for a few methods there. These are excluded
for now, sanity tests will be added in a follow-up.
This patch aligns Puppeteer testing infrastructure with the approach
we use in Puppeteer-Firefox.
This patch:
- makes all tests accept Puppeteer object as a function argument
rather than require it statically. This way we can pass either
Puppeteer or Puppeteer-Firefox to drive tests.
- renames the `puppeteer.spec.js` into `launcher.spec.js`. The
`puppeteer.spec.js` is now the entry point for all cross-browsers
tests.
Tracing is working on a per-browser level, not per-page. In order
to paralellize these tests effectively and properly, each should run
a designated browser.
Drop requirement for matching "origin" and "content-type" headers
in requests and request interceptions. This way javascript redirects
that use form submission start working.
Fix#3684.
ExecutionContext.evaluateHandle accepts arguments that are either
serializable, or JSHandles. A potential confusion is that it *does not*
accept arguments that *contain* JSHandles.
This patch adds a log message warning when it encounters that situation.
Fixes#3562
This patch teaches `page.setContent` to await resources in
the new document.
**NOTE**: This patch changes behavior: currently, `page.setContent`
awaits the `"domcontentloaded"` event; with this patch, we can now await
other lifecycle events, and switched default to the `"load"` event.
The change is justified since current behavior made `page.setContent`
unusable for its main designated usecases, pushing our client
to use [dataURL workaround](https://github.com/GoogleChrome/puppeteer/issues/728#issuecomment-334301491).
Fixes#728
NavigatorWatcher subscribes to Connection to get a `Disconnected` event,
causing us to hit the default max of 10 listeners constraint.
Technically we don't leak anything here and can safely bump
the maxListenersCount to Infinity.
However, we conveniently have `CDPSession`, and
can re-dispatch the event on it and keep the safety check in place.
This adds `page.accessibility.snapshot()`. It serializes and returns the accessibility tree for the page. By default, uninteresting nodes are filtered out of the snapshot.
fixes#2033
This patch enables cookie test. The actual upstream patch
that fixed the issue:
- https://crrev.com/599696 - Headless: support cookie encryption
Fixes#921.
This patch fixes a case in which computeQuadArea calculates the area size correctly, but returns the area as a negative number.
This occurs when DOM.getContentQuads returns quads in a specific order.
E.g. the array: [ { x: 463, y: 68.5 },{ x: 437, y: 68.5 },{ x: 437, y: 94.5 },{ x: 463, y: 94.5 } ] will receive area size of -676.
CSS stylesheets can still be parsed and added events emitted during the CSS.stopRuleUsageTracking call. It needs to be awaited before calling CSS.disable, otherwise the text content of those style sheets will be unavailable.
If nobody forces a layout, CSS coverage is inconsistent. This causes some flakes on the bots. The test in this PR fails 90% of the time on my local machine.
This adds `browser.waitForTarget` and `browserContext.waitForTarget`. It also fixes a flaky test that was incorrectly expecting targets to appear instantly.
This patch:
- adds experimental "transport" option to pptr.connect
- uses "transport" option to make sure Puppeteer-Web works with
Target.exposeDevToolsProtocol
Drive-by: add `browser.target()` to access browser target.
This patch introduces API to manage frame navigations.
As a drive-by, the `response.frame()` method is added as a shortcut
for `response.request().frame()`.
Fixes#2918.
This patch traces all public async methods and wraps them
in a helper method that tags the sync stack trace.
Later on, if the method call throws an exception, we add
a captured stack trace to the original stack trace with the "--ASYNC--"
heading.
An example of a stack trace:
```
Error: net::ERR_ABORTED at http://localhost:8907/empty.html
at navigate (/Users/lushnikov/prog/puppeteer/lib/Page.js:622:37)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at Page.<anonymous> (/Users/lushnikov/prog/puppeteer/lib/helper.js:147:27)
at fit (/Users/lushnikov/prog/puppeteer/test/page.spec.js:546:18)
at process._tickCallback (internal/process/next_tick.js:68:7)
```
If referer is passed to the options object its value will be used as the referer instead of the value set by `Page.setExtraHTTPHeaders()`.
This is the correct way to set referer header: otherwise, the `referer` header will override all the document subrequests.
Fixes#3090.
Introduce an API to manage permissions per browser context:
- BrowserContext.overridePermissions(origin, permissions)
- BrowserContext.clearPermissionOverrides()
Fixes#846.
This roll includes:
- https://crrev.com/584293 - DevTools: execute scripts in addScriptToEvaluateOnLoad in order
- https://crrev.com/585630 - DevTools: introduce Browser.grantPermissions
- https://crrev.com/587156 - Revert "[Base] Use background mode for ThreadPriority::BACKGROUND threads (behind feature) (reland)."
The "revert" patch fixes headless functionality on windows.
References #846.
Fixes#3106.
It turned out that almost any usecase requires helper methods to access
DOM inside the ExecutionContext.
Instead of exposing execution contexts as-is, we should introduce
IsolatedWorld as a first-class citizen that will hold execution contexts
inside.
This patch adds a new require, `puppeteer/Errors`, that
holds all the Puppeteer-specific error classes.
Currently, the only custom error class we use is `TimeoutError`. We'll
expand in future with `CrashError` and some others.
Fixes#1694.
This allows us:
- dogfood browser contexts the way we want them to be used
- simplifies the dance around service workers / cookies setting up and tier down.
This changes sendCharacter to use document.execCommand instead of sending a `'char'` event from the protocol. This is more aligned with how input would come in from emoji keyboards, and removes the 3ish byte limit on characters that can be sent which prevented larger emoji from being rendered correctly.
Emoji will still fail to type correctly if typing them into an iframe that is in shadow dom.
fixes#1096
This patch rolls Chromium to r579032. The patch includes:
- https://crrev.com/577366 - DevTools: report redirect responses only if response interception is enabled
- https://crrev.com/577212 - DevTools: intercept requests resulting from redirects
- https://crrev.com/578934 - DevTools: Add a protocol method to insertText
Interception Logic in DevTools protocol has changed regarding redirects;
this patch migrates interceptions to dispatch "request" events based on
requestWillBeSent event.
I have seen some flaky test failures where it would be nice to have run the tests with `DEBUG=puppeteer:error`. Instead of always running tests like that, I am redirecting `debugError` to the output category of the test. This is the same thing that we do for Chromium's stderr.
As a drive-by, I added an additional `debugError` where we were usually a try..finally pattern.
Unfortunately, disabling javascript in page prevents any microtasks
to be executed even from puppeteer-originating javascript. As a
result, the IntersectionObserver hack we use to conditionally
scroll into view doesn't work.
To workaround this, we start always scrolling before clicking if
page's javascript is disabled.
Fixes#2898
Chrome DevTools shows anonymous scripts with yellow background and names
them with `debugger://VM<scriptId>` prefix.
This patch starts reporting the same debugger:// urls for anonymous
scripts in puppeteer's JS coverage. This might simplify debugging, e.g.
using `debugger;` statement to reveal the script in DevTools and later
matching it against the one in the coverage.
This patch:
- simplifies test reusing the `offscreenbuttons.html` asset
- aligns IntersectionObserver usage with the one we have for
`ElementHandle._scrollIntoViewIfNeeded`.
This patch adds `reportAnonymousScripts` option to the `coverage.startJSCoverage` method. With this option, anonymous scripts are reported as well.
Fixes#2777
EmualationManager used to be injecting touch hooks to properly
support touch emulation.
However, these are no longer necessary, since https://crbug.com/133915
is long fixed.
Originally, we use `Element.scrollIntoViewIfNeeded` to make sure
button is on screen before trying to click it.
However, `Element.scrollIntoViewIfNeeded` doesn't work in certain
scenarios, e.g. when element is partially visible and horizontal
scrolling is required to make it fully visible.
This patch polyfills `element.scrollIntoViewIfNeeded` using
IntersectionObserver and `Element.scrollIntoView`.
Fixes#2804.
This patch:
- stops appending `undefined` to our protocol messages unnecessarily.
- rewrites `Cannot find execution context id` to `Execution context was destroyed, most likely because of a navigation.` when it occurs from a Puppeteer ExecutionContext. The error message is left alone if it occurs via a CDPSession.
This patch eliminates a common race condition with WaitTask, that
happens when predicate function gets resolved right before the execution
context gets destroyed.
This situation results in a "Cannot find context with specified id undefined"
exception.
Credits go to @jakub300 for his wonderful [investigation](https://github.com/GoogleChrome/puppeteer/issues/1325#issuecomment-395472092).
Fixes#1325.
`Data.prototype.toString` may return non-ASCII characters, which aren't accepted by `setHeader`.
E.g., on Russian locale, it might look like this:
```
> new Date().toString()
'Thu Jun 14 2018 13:11:50 GMT+0300 (Финляндия (лето))'
```
This patch:
- adds `worker.evaluate` and `worker.evaluateHandle` methods as a shortcut to their execution context equivalents.
- makes the error messages a bit nicer when interacting with a closed worker (as opposed to a closed page).
- moves the worker tests into their own spec file.
This patch disables OOPIF by default.
**NOTE**: this is a temporary bandaid for the time we're crafting
the full-fledged support for site isolation over DevTools protocol.
References #2548.
It's impossible to launch chromium without initial page.
This patch makes sure that `puppeteer.launch()` always returns a browser
with at least one page user can connect to.
Since Node 10, `console.assert` no longer throws an AssertionError.
(This is generally good since it aligns Node.js with Browsers.)
This patch migrates all usages of `console.assert` in our codebase.
- All the `lib/` and testing code is migrated onto a handmade `assert`
function. This is to make Puppeteer transpilation / bundling easier.
- All the tooling is switched to use Node's `assert` module.
Fixes#2547.
Some of the worker tests were failing on the bots. After investigating, I found one race in the test, and one race upstream in Chromium which I filed upstream as https://crbug.com/846099. But I added a small hack here as a temporary workaround.
References #2632
This patch allows logging the output of the Chromium process to be enabled in tests by passing in the environment variable `DUMPIO=true`.
Additionally, the `stderr` of the Chromium process will always be logged in the the "Output" section of failing page tests.
Scrollbars look different on different platforms, so must be made invisible in tests. As a drive-by, xdescribe was broken with the new test runner.
References #2524
This adds `page.workers()`, and two events `workercreated` and `workerdestroyed`. It also forwards logs from a worker into the page `console` event.
Only dedicated workers are supported for now, ServiceWorkers will probably work differently because they aren't necessarily associated with a single page.
Fixes#2350.
This patch introduces Browser Contexts and methods to manage them:
- `browser.createIncognitoBrowserContext()` - to create new incognito
context
- `browser.browserContext()` - to get all existing contexts
- `browserContext.dispose()` - to dispose incognito context.
Fixes#85.
Today, `page.close()` method doesn't run page's beforeunload listeners.
This way users can be sure that `page.close()` actually closes the
page.
This patch adds an optional `runBeforeUnload` option to the
`page.close()` method that would run beforeunload listeners. Note:
running beforeunload handlers might cancel page closing.
Fixes#2386.
This patch teaches Page.waitForNavigation to correctly handle navigation
to pages that have frames that might never load.
These frames include:
- frames which main resource loading was aborted due to mixed-content
error
- frames that artificially called `window.stop()` to interrupt loading
themselves
Fixes#1936.
This patch fixes puppeteer navigation primitives to work with
same-document navigation.
Same-document navigation happens when document's URL is changed,
but document instance is not re-created. Some common scenarios
for same-document navigation are:
- History API
- anchor navigation
With this patch:
- pptr starts dispatching `framenavigated` event when frame's URL gets
changed due to same-document navigation
- `page.waitForNavigation` now works with same-document navigation
- `page.goBack()` and `page.goForward()` are handled correctly.
Fixes#257.
Today, we have tests split into multiple files, with files pulling
tests from some other files.
This patch starts explicitly gathering all tests from the same
`test.js` file.
Drive-By: move one test from `browser.spec.js` into `puppeteer.spec.js`
since it starts browser itself.
This roll includes:
- https://crrev.com/549003 - DevTools: make pptr tests pass with DCHECKs.
The patch fixes a browser crash that happens during browser close.
As a result, cookies were not saved properly (and thus the flaky test we
had).
Fixes#1537.
This roll includes:
- https://crrev.com/547982 - v8 roll that includes [fixed
Runtime.callFunctionOn](1637818671) method
The upstream fix makes it possible to run frame.waitFor* functions on
pages with strict CSP.
References #1229.
This patch introduces a new `pipe` option to the launcher to connect over a pipe.
In certain environments, exposing web socket for remote debugging is a security risk.
Pipe connection eliminates this risk.
This patch adds support for `timeout: 0` to disable timeout for the following functions:
- `page.waitForFunction`
- `page.waitForXPath`
- `page.waitForSelector`
and their `frame` counterparts.
Fixes#2200
This patch:
- starts fulfilling security details for redirect responses
- changes `response.securityDetails()` to return null if the response
is served over non-secure connection
This patch introduces ExecutionContext.frame() that returns Frame
associated with this Execution Context.
This allows to associate console messages with the originating frame,
if any.
This patch:
- introduces `SecurityDetails` class that exposes a set of fields that describe properties of secure connection
- introduces method `response.securityDetails()` that returns an instance of `SecurityDetails` object.
keyboard.down() and keyboard.up() both use the _pressedKeys Set, however keyboard.down() adds and searches for the key code, whereas keyboard.up() attempts to delete based on the key rather than the key code.
Fixes#1901
This patch introduces `BrowserFetcher` class that manages
downloaded versions of products.
This patch:
- shapes Downloader API to be minimal yet usable for our needs. This
includes removing such methods as `Downloader.supportedPlatforms` and
`Downloader.defaultRevision`.
- makes most of the fs-related methods in Downloader async. The only
exception is the `Downloader.revisionInfo`: it has stay sync due to the
`pptr.executablePath()` method being sync.
- updates `install.js` and `utils/check_availability.js` to use new API
- finally, renames `Downloader` into `BrowserFetcher`
Fixes#1748.
This patch:
- introduces `test/assets/cached` folder and teaches server to cache
all the assets from the folder
- introduces `test/assets/serviceworkers` folder that stores all the
service workers and makes them register with unique URL prefix
- introduces `Response.fromCache()` and `Response.fromServiceWorker()`
methods
Fixes#1551.
This PR fixes lost functionality that is no longer on-par with the documentation for `Page.select`, namely:
> `...values` <...string> Values of options to select. If the `<select>` has the `multiple` attribute, all values are considered, **otherwise only the first one is taken into account**.
I've also added an accompanying test for this use case.
This patch:
- introduces `page.waitForXPath` method
- introduces `frame.waitForXPath` method
- amends `page.waitFor` to treat strings that start with `//` as xpath queries.
Fixes#1757.
feat: expose raw devtools protocol connection
This patch introduces `target.createCDPSession` method that
allows directly communicating with the target over the
Chrome DevTools Protocol.
Fixes#31.
If the success value of `waitForFunction` was not serializable, checking whether it was truthy with `.jsonValue()` might fail. Now I check whether it was truthy inside the page.
Fixes#1737.
This patch:
- teaches page.waitFor* methods to accept JSHandles
- starts returning JSHandles from page.waitFor* calls.
BREAKING CHANGE: this patch starts allocating `JSHandle`/`ElementHandle` instances for every call to `page.waitFor*` functions. These handles should be disposed manually to avoid memory consumption.
Fixes#1703, fixes#1654, fixes#1724.
This patch adds two new methods to the `page.coverage` namespace:
- `page.coverage.startCSSCoverage()` - to initiate css coverage
- `page.coverage.stopCSSCoverage()` - to stop css coverage
The coverage format is consistent with the JavaScript coverage.
This patch introduces a new `page.coverage` namespace with two methods:
- `page.coverage.startJSCoverage` to initiate JavaScript coverage
recording
- `page.coverage.stopJSCoverage` to stop JavaScript coverage and get
results
This patch:
- adds `puppeteer.defaultArgs()` method to get default arguments that are used to launch chrome
- adds `ignoreDefaultArgs` option to `puppeteer.launch` to avoid using default puppeteer arguments
Fixes#872
The patch converts all the getters in the codebase into the methods.
For example, the `request.url` getter becomes the `request.url()`
method.
This is done in order to unify the API and make it more predictable.
The general rule for all further changes would be:
- there are no getters/fields exposed in the api
- the only exceptions are "namespaces", e.g. `page.keyboard`
Fixes#280.
BREAKING CHANGE:
This patch ditches getters and replaces them with methods throughout
the API. The following methods were added instead of the fields:
- dialog.type()
- consoleMessage.args()
- consoleMessage.text()
- consoleMessage.type()
- request.headers()
- request.method()
- request.postData()
- request.resourceType()
- request.url()
- response.headers()
- response.ok()
- response.status()
- response.url()
This patch migrates tests so that they can be run concurrently.
- By default, tests still run in one thread.
- To run tests in 4 parallel threads, run node test/test.js -j 4 or npm run unit -- -j 4
- Environment variable PPTR_PARALLEL_TESTS could be set to override default parallelization
Every test gets passed in a state. State is set up in the beforeAll and beforeEach callbacks,
and should be teared down in the afterAll and afterEach callbacks.
By default, state has a parallelIndex variable initialized that defines the thread index that runs the execution.
This patch unifies node6 transpilation:
- instead of generating multiple top-level directories, prefixed with
`node6-`, all transpiled code gets placed under single `node6/` folder
- transpilation doesn't change require paths of transpiled modules any
more
This patch:
- renames ChromiumDownloader into just Downloader (this is in
preparation for different products download)
- moves Downloader from utils/ to lib/. This unifies all of the
production-critical code in the lib/.
Drive-by: make Downloader a regular class.
This patch introduces a tiny test runner to run puppeteer tests.
The test runner is self-container and allows parallel (wrt IO) test execution.
It will also allow us to split tests into multiple files if necessary.
Comparing to the jasmine, the testrunner supports parallel execution, properly
handles "unhandled promise rejection" event and signals.
Comparing to ava/jest, the testrunner doesn't run multiple node processes,
which makes it simpler but sufficient for our goals.
When mojo is enabled for input events, we sometimes get an extra mouse move after the page is created. Moving the mouse ourselves makes the test consistent. The same behavior also caused DevTools dispatchMouseEvent to become flaky, but that will be fixed with: https://chromium-review.googlesource.com/c/chromium/src/+/804357
Since non-promise values always win the `Promise.race`, we shouldn't
return `null` for timeout promise in NavigationWatcher.
Instead, we can return a promise that never resolved. It should be
GC'd later with the navigation watcher itself.
Fixes#1417.
With the addition of `browser.targets()` api, we now can connect to
in-flight targets.
For Puppeteer, it means that it can "miss" certain events happenning
while it wasn't attached to the target.
This patch:
- fixes this problem with NetworkManager, preparing it for the
missed `requestWillBeSent` event.
- adds a new test to ensure that not a single unhandled promise
rejection has happened during test execution.
Fixes#1363.
In Blink, frames don't necesserily have execution context all the time.
DevTools Protocol precisely reports this situation, which results in
Puppeteer's frame.executionContext() being null occasionally.
However, from puppeteer point of view every frame will have at least a
default executions context, sooner or later:
- frame's execution context might be created naturally to run frame's
javascript
- if frame has no javascript, devtools protocol will issue execution
context creation
This patch builds up on this assumption and makes frame.executionContext()
to be a promise.
As a result, all the evaluations await for the execution context to be created first.
Fixes#827, #1325
BREAKING CHANGE: this patch changes frame.executionContext() method to return a promise.
To migrate onto a new behavior, await the context first before using it.
Currently, `elementHandle.screenshot` passes bounding box into the `page.captureScreenshot`
method.
This is wrong since `captureScreenshot` accepts viewport, not bounding box.
This patch uses layout metrics to convert bounding box into viewport.
Fixes#1315.
This patch:
- starts persisting lifecycle state for every frame
- migrates NavigationWatcher to rely on these lifecycle events
- refactors Page.goto to properly return navigation errors
Fixes#1218.
This patch starts asserting that all values are of type "string".
The alternative approach to cast values to strings
might yield a hard-to-debug errors.
Fixes#1276.
This patch migrates puppeteer to support PlzNavigate chromium
project.
As a consequence of this patch, we no longer wait for both
requestWillBeSent and requestIntercepted events to happen. This should
resolve a ton of request interception bugs that "hanged" the loading.
Fixes#877.
Currently, NavigationWatcher listens to lifecycle events from Page
domain and security events from Security domain.
However, the events are dispatched from different processes in browser:
- Page's lifecycle events are dispatched from renderer process
- Security events are dispatched from browser process
This makes for the undefined order between events and results in
NavigationWatcher reporting different failuer messages, based on
the event order.
This patch stops relying on security errors in navigation watcher and
instead switches to request failure codes for the main resource.
Fixes#1195
Elements in shadow dom erroneously considered that they were detached
from document.
This patch starts using `Element.isConnected` instead of
`document.contains()` call.
Fixes#1061.
This roll includes:
- crrev.com/510651 that changes request interception methods in protocol
- s/Page.setRequestInterceptionEnabled/Page.setRequestInterception
BREAKING CHANGE
Page.setRequestInterceptionEnabled is renamed into
Page.setRequestInterception.
This patch adds "options" parameter to the `page.setContent` method. The
parameter is the same as a navigation parameter and allows to specify
maximum timeout to wait for resources to be loaded, as well as to
describe events that should be emitted before the setContent operation
would be considered successful.
Fixes#728.
This patch adds support to multiple events that could be passed inside
navigation methods:
- Page.goto
- Page.waitForNavigation
- Page.goForward
- Page.goBack
- Page.reload
Fixes#805
This patch adds a new `domcontentloaded` option to a bunch of navigation
methods:
- Page.goto
- Page.waitForNavigation
- Page.goBack
- Page.goForward
- Page.reload
Fixes#946.
This patch:
- migrates navigation watcher to use protocol-issued lifecycle events.
- removes `networkIdleTimeout` and `networkIdleInflight` options for
`page.goto` method
- adds a new `networkidle0` value to the waitUntil option of navigation
methods
References #728.
BREAKING CHANGE:
As an implication of this new approach, the `networkIdleTimeout` and
`networkIdleInflight` options are no longer supported. Interested
clients should implement the behavior themselves using the `request` and
`response` events.
BREAKING CHANGE:
This patch lets key names be code in addition to key. When specifying a code, the proper text is generated assuming a standard US keyboard layout. e.g Digit5 -> "5" or "%" depending on Shift.
* location is now specified. #777
* Using unknown key names now throws an error. #723
* Typing newlines now correctly presses enter. #681
feat(interception): Implement request.respond method
This patch implements a new Request.respond method. This
allows users to fulfill the intercepted request with a hand-crafted
response if they wish so.
References #1020.
Currently, JSHandle.jsonValue() is implemented as in-page JSON.stringify
call and consequent JSON.parse in node. This approach proved to be
unfortunate for automation purposes: if page author overrode the
Object.prototype.toJSON method, then it's harder for puppeteer to
interact with the page.
This patch switches JSHandle.jsonValue to use protocol serialization
that ignores toJSON property. THis also changes the `page.evaluate`
behavior since it is based on JSHandle.jsonValue().
Fixes#1003.
BREAKING CHANGE:
`page.evaluate` no longer calls toJSON when generating return value.
For the old behavior, do JSON.parse/JSON.stringify manually:
```js
const json = JSON.parse(await page.evaluate(() => JSON.stringify(obj)));
```
This patch:
- introduces Target class that represents any inspectable target, such as service worker or page
- emits events when targets come and go
- introduces target.page() to instantiate a page from a target
Fixes#386, fixes#443.
Similarly to the `request.response()` method, this patch adds
`request.failure()` method that returns error details for the failed
requests.
Fixes#901.
This patch:
- changes `browser.close` to terminate browser.
- introduces new `browser.disconnect` to disconnect from a browser without closing it
This patch: fixes#918, fixes#989
BREAKING CHANGE:
`browser.close()` will always close a browser, even if it was initialized with
`puppeteer.connect`. To disconnect from a remote browser, use `browser.disconnect()` instead.
This patch starts generating input events for `keyboard.down`, addressing bullet 2 of
#723. With this patch, there's no longer any difference between `keboard.press('a')` and
`keyboard.press('a', {text: 'a'})`.
BREAKING CHANGE:
`keyboard.down('a')` starts generating input event (wasn't the case before).
References #723
This patch:
- deprecates injectFile as it was confused with the addScriptTag
- accepts an options object in addScriptTag which supports properties url, path and content.
- accepts an options object in addStyleTag which supports properties url, path and content.
Fixes#949.
BREAKING CHANGE:
- the addStyleTag/addScriptTag have changed;
- the injectFile was removed in favor of (addStyleTag({path:}).
This patch introduces `Page.queryObjects` and
`ExecutionContext.queryObjects` methods to query JavaScript heap
for objects with a certain prototype.
Fixes#304.
This patch improves life of puppeteer contributor on Windows:
- Setting environment variables using cross-env since Windows requires the SET command.
- Calling Jasmine in the script debug-unit using jasmine's JavaScript binary instead of shell.
- Add /test/test-user-data-dir* to .gitignore since temporary user data directories, in case of test
fails, remains in the test directory.
The page.plainText is confusing: it's unclear what kind of text it
returns, textContent or innerText. It's also easily polyfillable and
doesn't seem to be used.
BREAKING CHANGE: the page.plainText is not existing any more.
Instead, use `page.evaluate(() => document.body.innerText)`.
This patch:
- updates JSHandle.toString to make a nicer description for primitives
- excludes JSHandle.toString from documentation to avoid its abuse
References #382
This patch moves resourceType to be all small-caps. This aligns
with our convention that all string constants should be smallcaps.
BREAKING CHANGE: this patch changes the constants of the
request.resourceType to be all small-caps.
This patch:
- adds `ElementHandle.boundingBox()` method to get bounding box of element relative
to the page
- adds `ElementHandle.screenshot()` method to capture a screenshot of an element
This patch:
- adds input methods to ElementHandle, such as ElementHandle.type and ElementHandle.press
- changes `page.type` to accept selector as the first argument
- removes `page.press` method. The `page.press` is rarely used and doesn't operate with selectors; if there's a need to press a button, `page.keyboard.press` should be used.
BREAKING CHANGE: `page.type` is changed, `page.press` is removed.
Fixes#241.
This patch:
- introduces ExecutionContext class that incapsulates javascript
execution context. An examples of execution contexts are workers and
frames
- introduces JSHandle that holds a references to the javascript
object in ExecutionContext
- inherits ElementHandle from JSHandle
Fixes#382.
Headless isn't closing gracefully, which sometimes causes data loss when Chrome closes before it finishes writing things to disk.
See https://crbug.com/771830
References #921
This patch allows passing 0 to disable timeout for the following methods:
- page.goto
- page.waitForNavigation
- page.goForward
- page.goBack
Fixes#782.
This patch introduces ConsoleMessage type and starts dispatching
it for the 'console' event.
BREAKING CHANGE: this breaks the api of the 'console' event.
Fixes#744.
This lets the user pass `...args` into `page.waitFor`. It also clarifies that the docs that `options` is not optional if `...args` are specified.
Fixes#770
Since protocol ignores all HTTP headers that don't have string
value, this patch starts validating header key-values before
sending them over the protocol.
Fixes#713.
This patch:
- makes `browser.close()` return a promise that resolves when browser gets closed
- starts closing chrome gracefully if a custom `userDataDir` is supplied
Fixes#527
This patch:
- teaches `page.evaluate` to accept ElementHandles as parameters
- removes `ElementHandle.evaluate` method since it's not needed any
more
References #382
It turns out that [undefined, 1].join(',') results in ",1" instead
of "undefined,1". This causes a syntax error when trying to pass undefined
as a first argument to `page.evaluate` method.
Fixes#572.
Currently, navigation watcher throws exception if timeout
is exceeded.
Due to the way it is used in `page.navigate`, the promise
get's rejected before it is awaited, which is considered to
be "unhandled promise rejection".
Fixes#738
It's very bad to have 'unhandled promise rejection' that can't be
handled in user code. These errors will exit node process in a near
future.
This patch avoids 'unhandled promise rejection' while sending protocol
messages.
This patch:
- introduces `puppeteer:error` debug scope and starts using it for all
swalloed errors.
- makes sure that every `client.send` method is either awaited or its
errors are handled.
- starts return promises from Request.continue() and Request.abort().
- starts swallow errors from Request.contine() and Request.abort().
The last is the most important part of the patch. Since
`Request.continue()` might try to continue canceled request, we should
disregard the error.
Fixes#627.
This patch:
- adds `page.touchscreen` namespace, similar to `page.mouse` and `page.keyboard`.
- adds tapping to multiple layers:
- `page.touchscreen.tap`
- `page.tap` - convenience method which accepts selector
- `elementHandle.tap`
Fixes#568 and #569.
This patch:
- switches to objects instead of maps for headers (in Request, Response and
page.setExtraHTTPHeaders)
- converts all header names to lower case
Fixes#547, fixes#509
It turned out that either Network.requestIntercepted or
Network.requestWillBeSent occasionally report encoded URL.
This patch starts decoding URL's when generating request hash.
Fixes#558.
This patch:
- introduces a transpiler which substitutes async/await logic with
generators.
- starts using the transpiler to generate a node6-compatible version of puppeteer
- introduces a runtime-check to decide which version of code to use
Fixes#316.
This patch rolls chromium to r496140. This includes the r496130 that
introduces multiple sessions for single target.
With this patch, it is possible to run puppeteer in headful mode
and open devtools over the automated pages without puppeteer losing
connection to the page.
Fail gracefully when chromium failed to download
This patch changes both install.js and Launcher.js to inform how
chromium could be downloaded manually.
This patch:
- removes the `page.uploadFile` method
- adds `elementHandle.uploadFile` method.
Motivation: `elementHandle.uploadFile` is rarely used, so it doesn't worth it
to keep it on page.
This patch:
- rolls chromium to r494365
- starts using Runtime.evaluate(awaitPromise: true), with new semantic
we can avoid additional Runtime.awaitPromise call
- stops resolving promises for Console event
This patch:
- refactors Connection to use a single remote debugging URL instead of a
pair of port and browserTargetId
- introduces Puppeteer.connect() method to attach to already running
browser instance.
Fixes#238.
This patch:
- teaches request interception to ignore data URLs. Currently protocol
doesn't send interceptions for data URLs.
- teaches request interception to properly process URLs with hashes.
Currently `Network.requestIntercepted` sends url with a hash, whereas
`Network.requestWillBeSent` doesn't report hashes in its urls. @see
crbug.com/755456
- skips one more header that I spotted during debugging interception on
the realworld websites.
Fixes#258, #259.
This patch starts emitting 'error' event when page crashes.
'error' events have special treatment in node, so page crashes
become observable for users.
Fixes#262.
This patch:
- split browser launching logic from Browser into `lib/Launcher.js`
- introduce `puppeteer` namespace which currently has a single `launch`
method to start a browser
With this patch, the browser is no longer created with the `new
Browser(..)` command. Instead, it should be "launched" via the
`puppeteer.launch` method:
```js
const puppeteer = require('puppeteer');
puppeteer.launch().then(async browser => {
...
});
```
With this approach browser instance lifetime matches the lifetime of
actual browser process. This helps us:
- remove proxy streams, e.g. browser.stderr and browser.stdout
- cleanup browser class and make it possible to connect to remote
browser
- introduce events on the browser instance, e.g. 'page' event. In case
of lazy-launching browser, we should've launch browser when an event
listener is added, which is unneded comlpexity.
Mouse events are no longer racy. Enabling touch no longer converts all mouse events into touches. Promises in destroyed execution contexts are rejected immediately.
The issue #168 is a protocol inconsistency which happens only
in case of HTTPS error. This patch starts refering to the
upstream bug instead of puppeteer issue.
Closes#168.
This patch
- rolls chromium to 492629
- migrates connection establishing to use browser target. This migration means
that now we have a single websocket connection to browser (implemented
in Connection class). A connection to a particular target is
incapsulated in a new Session class.
This patch:
- changes interception API so that it better aligns with what we'd like to see
in #121
- fixes the issue with redirect interception
Fixes#217.
This patch starts using "Failed" command for request interception instead of
"Aborted".
The "Aborted" status also has a side-effect of cancelling the navigation, so
there will be no error on the page and form puppeteer's standpoint, the navigation
will never complete.
The chrome-devtools://devtools/bundled/inspector.html is a perfectly
valid url in headful chromium, so we should pick another one for test
to work (and for the navigation inside the test to fail).
This patch implements 'autoRepeat' functionality for `keyboard.down`.
With this patch, the subsequent calls to `keyboard.down` would generate
an event with 'autoRepeat` flag set to true.
Closes#157
This patch makes sure that request.text() doesn't try
to fetch response body from the backend until the request is
actually finished (finished or failed).
It turns out we're not receiving 'Network.requestWillBeSent' event
for every requestId.
This patch makes sure we don't dispatch `requestfinished` and
`requestfailed` events without passing actual request.
References #168
This patch:
- teaches page.uploadFile() to resolve given file paths against
current working directory. This aligns paths handling with all the
other methods
- moves page.uploadFile() under Frame
- changes test to use relative path for file upload
The Body class was inlined in the Request and Response classes.
This patch:
- removes the Body class
- adds Request.postData public property
- adds Response.buffer(), Response.text() and Response.json() methods
Fixes#106.
The page.waitForFunction method allows to wait for a general predicate.
The predicate will be continiously polled for in page, until
it either returns true or the timeout happens.
The polling parameter could be one of the following:
- 'raf' - to poll on every animation frame
- 'mutation' - to poll on every dom mutation
- <number> - to poll every X milliseconds
References #91
This patch:
- implements a basic public API coverage based on 'helper.tracePublicAPI' methods
- adds `npm run coverage` command which reports coverage after running all of the unit tests
References #50.
This patch:
- removes Body.arrayBuffer. This method is redundant since there's
already a Body.buffer() method
- removes Body.bodyUsed getter.
References #106
This patch:
- renames page.setHTTPHeaders into page.setExtraHTTPHeaders
- starts using Map instead of Object to align with other headers
arguments
Fixes#112.
This patch improves on page.evaluate to accept a string.
The string can have a trailing '//# sourceURL=' comment which would
name the evaluation to make stacks beautiful.
In order to make sourceURL comments possible, this patch:
- removes wrapping of the client function into `Promise.resolve()`
- stops passing `awaitPromise` parameter to `Runtime.evaluate`
- starts to await promise via the `Runtime.awaitPromise` if the return type of the evaluation
is promise
closes#118
This patch renames page.evaluateOnInitialized into
page.evaluateOnNewDocument to better align with the protocol and with
what the method is actually doing.
Fixes#119.
Refactor Frame.waitForSelector to make room for Frame.waitForFunction
implementation.
This patch:
- removes AwaitedElement class which proved to be confusing, and
introduces a more straight-forward WaitTask.
- refactors the mutation observer to return true in case of successful
waiting or false in case of timeout.
References #91
This patch:
- introduces helper.addEventListener/helper.removeEventListeners
to simplify event management
- moves NavigatorWatchdog over to the helper.addEventListener to
stop leaking event listeners
This patch:
- adds Mouse class which holds mouse state and implements mouse primitives,
such as moving, button down and button up.
- implements high-level mouse api, such as `page.click` and `page.hover`.
References #40, References #89
This patch:
- introduces page.waitForSelector to wait for the selector to appear
- introduces polymorphic page.waitFor method, which accepts
either string (and in this case is a shortcut for page.waitForSelector)
or number (and in this case it's a promisified timeout).
References #91.
This patch implements timeout option for page.waitFor. The function
will throw if the selector doesn't appear during timeout milliseconds
of waittime.
References #89, #91.
This patch adds a 'visible' option to the Page.waitFor method, making
it possible to wait for the element to become actually visible.
References #89, #91.
This patch:
- gets rid of `page.emulate` and `page.emulatedDevices`
methods. Instead, it is suggested to use `page.setViewport()`
and `page.setUserAgent()` methods.
- moves DeviceDescriptors to the top level of the puppeteer so that
it is convenient to require them.
- improves on documentation to describe the suggested emulation
approach.
References #88.
The unit testing on mac is unreliable, making the last tests to hang.
This seems to be related to the upstream bug: crbug.com/741689
For now, this patch restarts the browser in the middle of unit testing.
This should be reverted once the upstream is fixed.
References #100.
Currently, it's impossible to do screenshots in parallel.
This patch:
- makes all screenshot tasks sequential inside one browser
- starts activating target before taking screenshot
- adds a test to make sure it's possible to take screenshots across
tabs
- starts waiting for a proper page closing after each test. This might
finally solve the ECONNRESET issues in tests.
References #89
This patch:
- resolves pending callbacks in connection once it gets closed
- fixes one of Page.screenshot() tests to wait for all the screenshots
- starts handling ECONNRESET error in test server to avoid throwing with no
good reason
This patch introduces Page.waitForNavigation which allows to wait
for render-initiated navigation.
This patch also does a nice refactoring, replacing Navigator with NavigatorWatcher which
is not a part of a page state.
References #89
This patch:
- renames Page.printToPDF into page.pdf
- adds a 'path' option to the page.pdf options instead of a separate
`filePath` parameter
- improves on the documentation for the `page.pdf`
References #39.
This patch makes test/golden-utils.js configurable with
output directory and the golden directory.
The plan is to reuse golden-utils for doclint tests.
This patch:
- moves doclint under utils/ folder
- adds tests to verify doclint basic functionality
This patch also drops the jasmine as a spec runner for the doclint
checks. It turned out it's hard to customize jasmine's behavior,
so instead this patch implements a dummy spec runner.
The dummy spec runner allows us:
- to format messages however we want (the custom jasmine reporter would
also allow us to do this)
- to avoid `beforeAll` functions which pollute global to pass
initialized variables over to specs
References #14
This patch:
- fixes a typo in documentation linter, which resulted in only
certain method arguments being linted.
- adds missing arguments to the documentation
References #14.
This patch introduces a general Documentation.diff method, which
produces a diff of two documentations.
With this, the patch teaches documentation linter to lint method arguments.
References #14.
This patch changes Page.navigate API:
- Page.navigate now resolves to the main page response
- Page.navigate throws errors if there's no main page response,
e.g. in case of SSL errors, max navigation timeout,
or invalid url.
This patch also adds httpsServer with a self-signed certificates
for the testing purposes.
Fixes#10.
The command runs the puppeteer testsuite with the '--inspect-brk' node
flag. This makes it possible to connect to the testsuite with
Chrome DevTools and debug it.
Fixes#57.
This patch implements documentation linter, which leaves under `test/doclint`
folder.
The documentation linter works like this:
1. Parse javascript source code with esprima and construct a "documentation" out of source code
2. Generate HTML out of `api.md` and traverse the HTML with puppeteer.
3. Make sure javascript aligns nicely with HTML
The documentation linter adds the following commands:
- `yarn doc` - to test that documentation covers all the relevant apis
- `yarn generate-toc` - to update the table-of-contents for the `api.md`
It turned out that server.close() does not shutdown server but
stops it from accepting *new* connections.
It's our responsibility to destroy all the current connections,
if any.
This line within `injectFile` wasn't doing much of anything:
```js
let expression = fs.readFile(filePath, 'utf8', (err, data) => callback({err, data}));
```
* That's fixed.
* A path error in examples/features.js is fixed.
* Test added for injectFile.
This patch implements NetworkManager, which encapsulates all the
interaction with Network domain.
The NetworkManager also uses partial implementation of Request and
Response classes, defined in the Fetch API specification.
References #26
This patch:
- Changes network idle promise to wait for 2 or fewer network requests for at least idleTime (defaults to 5s) before resolving.
- Adds timer cleanup to failure navigation case.
- Adds handling of webSocketClosed.
- Ignores unrecognized requestIds to avoid negative inflight requests.
References #10
This patch introduces the 'request' event which is fired when
page has initiated a request.
The event dispatches an instance of Request class.
References #26.
This patch does a step towards Fetch API:
- implements Request object to some extend. The Request object will be
sent in RequestWillBeSent event.
- implements InterceptedRequest which extends from Request and allows
for request modification. The InterceptedRequest does not
conform to Fetch API spec - there seems to be nothing related to
amending in-flight request.
- adds test to make sure that request can change headers.
References #26
This patch implements Basic Input api:
- Page.focus(selector) - focuses element with selector
- Page.click(selector) - clicks element with selector
- Page.type(text) - types text into a focused element
Fixed#43.
This patch:
- reformats codebase to use 2-spaces instead of 4. This will
align the project with other codebases (e.g. DevTools and Lighthouse)
- enables eslint indentation checking
References #19
