This roll includes:
- https://crrev.com/653809 - FrameLoader: ignore failing provisional loads entirely
- https://crrev.com/654750 - DevTools: make sure Network.requestWillBeSent is emitted on time for sync xhrs
The FrameLoader patch is the reason behind the test change. It's
actually desirable to fail frame navigation if the frame detaches - and
that's consistent with Firefox.
Fixes#4337
These getters are introduced as a more convenient substitute for
a `require('puppeteer/Errors')` and
`require('puppeteer/DeviceDescriptors')`.
This way we can make cross-browser story nicer - a single require
of `puppeteer` or `puppeteer-firefox` fully defines Puppeteer
environment.
In certain cases inline element children might be positioned
outside of viewport.
In this case, we should intersect all content quads with viewport
before we pick one to click into.
Fixes#4274.
In case of multiple sessions to the same target, there's a race between
sessions to create a secondary isolated world. As a result, we might
end up having 2 execution contexts created for the needs of the
secondary isolated world.
This patch starts handling this race gracefully: instead of crashing,
we can use either of the execution contexts and ignore the rest.
Notably, the same race condition might happen if page reloads itself
in-between the calls to `page.addEvaluateOnNewDocument` and
`page.createIsolatedWorld`.
Fixes#4197.
This patch:
- refactors `NetworkManager`/`FrameManager` so that they enable all the
relevant domains themselves. This is a preparation for OOPIF support and
migration onto fetch domain.
- moves `networkManager` ownership into `FrameManager`. This way it's clear who owns what.
- stops enabling Security domain: it saves quite some traffic over
websocket since it no longer sends annoying "SecurityStateChanged" events.
Instead, use `Security.setIgnoreCertificateErrors` method.
- consolidates network cache state in network manager. This even fixes a
bug with caching and request interception interop.
This patch makes sure header overrides in request interception are
functioning as expected.
Drive-by: teach test server to use utf-8 charset header for text files.
This patch implements `page.setCookie()`, `page.deleteCookie()` and
`page.cookies()` and doubles the test coverage for cookies so that
we can feel safer on cross-browser compatibility.
This patch:
- implements Response.buffer() and other methods
- splits out relevant tests into a separate test suites
- implements `testServer.enableGzip()` method to optionally gzip
certain routes in tests
- adds tests to make sure `Response.text()` returns expected results
for binary and compressed responses.
When we started working on Puppeteer-Firefox, we forked original
Puppeteer testsuite.
This patch concludes the effort to merge testsuites back together.
Fixes#3889
Juggler now implements the same "flatten" protocol as CDP.
This patch:
* copies `Connection.js` from original Puppeteer (with a few renames, e.g. `CDPSesssion` -> `JugglerSession`).
* migrates code to support protocol-level sessions
Support browser target.
Drive-by: switch over to a more devtools'ish protocol:
- use `targetId` instead of `pageId` everywhere
- use target events instead of tab events
Firefox used to have a bug that prevented adding iframes with the
same URL as main frame. In this case, we used the EMPTY_PAGE2
so that it doesn't clash with top-level EMPTY_PAGE.
The bug seems to be fixed after we rolled onto a fresh Firefox;
there's no more need for this test asset.
This patch refactors Puppeteer-Firefox code to declare public
API in `/lib/api.js` and use it to setup async stack hooks
over the public API method calls.
This patch:
* unifies assets between tests
* enables a few puppeteer tests on Puppeteer-Firefox
Drive-by: beautify failing output of `expect.toEqual` matcher.
References #3889
This patch:
- introduces new testRunner methods `addTestDSL` and `addSuiteDSL`
to add annotated test / suite.
- introduces new test/suite declaration methods: `it_fails_ffox` and
`describe_fails_ffox`. These are equal to `it`/`describe` for chromium
tests and to `xit`/`xdescribe` for firefox.
- marks all unsupported tests with `it_fails_ffox`
- adds a new command-line flag `'--firefox-status'` to `//test/test.js`.
This flag dumps current amount of tests that are intentionally skipped
for Firefox.
End goal: get rid of all `it_fails_ffox` and `describe_fails_ffox`
tests.
Drive-By: remove cookie tests "afterEach" hook that was removing
cookies - it's not needed any more since every test is run in a
designated browser context.
References #3889
This aligns all Puppeteer tests so that they never throw the
"UnhandledPromiseRejection" when run with Puppeteer-Firefox.
With this patch, the `npm run funit` passes 275 of 460 Puppeteer tests.
References #3889.
This patch:
- adds support to `FFOX` env variable for Puppeteer testsuite
- install Firefox preferences when running tests with custom firefox
executable
References #3889
This patch:
- changes Puppeteer-Firefox plumbing of defaultBrowserOptions to align
with the way we do it for Puppeteer.
- plumbs puppeeteer-dependent Errors and DeviceDescriptors down to every
test.
- unifies a few tests between Puppeteer-Firefox and Puppeteer.
**Note:** in future, we should expose errors as `puppeteer.errors` and
device descriptors as `puppeteer.devices` to make it easy to pass around
Puppeteer/Puppeteer-Firefox instance.
References #3889.
Certain Puppeteer methods do expose the inner browser - e.g.
`browser.version()` depends on the browser we run.
Split out these tests into a vendor-specific test suites.
References #3889
Introduce a `npm run funit` script that runs puppeteer tests
against Puppeteer-Firefox.
Next steps:
- bring Puppeteer-Firefox unique tests to Puppeteer
- skip failing tests and run Puppeteer-Firefox on CI
- work through tests to pass them all with Puppeteer-Firefox
Method `page.setDefaultTimeout` overrides default 30 seconds timeout
for all `page.waitFor*` methods, including navigation and waiting
for selectors.
Fix#3319.
This patch starts executing frame.waitForSelector and frame.waitForXPath
in secondary world. As a result, websites that mutate page global
context (e.g. removing global MutationObserver) don't break Puppeteer's
behavior.
Fixes#609
`page.waitForSelector` should return `null` if waiting for `hidden:
true` and there's no matching node in DOM.
Before this patch, `page.waitForSelector` would return some JSHandle
pointing to boolean value.
Introduce `//lib/api.js` that declares a list of publicly exposed
classes.
The `//lib/api.js` list superceedes dynamic `helper.tracePublicAPI()` calls
and is used in the following places:
- [ASYNC STACKS]: generate "async stacks" for publicy exposed API in `//index.js`
- [COVERAGE]: move coverage support from `//lib/helper` to `//test/utils`
- [DOCLINT]: get rid of 'exluded classes' hardcoded list
This will help us to re-use our coverage and doclint infrastructure
for Puppeteer-Firefox.
Drive-By: it turns out we didn't run coverage for `SecurityDetails`
class, so we lack coverage for a few methods there. These are excluded
for now, sanity tests will be added in a follow-up.
This patch aligns Puppeteer testing infrastructure with the approach
we use in Puppeteer-Firefox.
This patch:
- makes all tests accept Puppeteer object as a function argument
rather than require it statically. This way we can pass either
Puppeteer or Puppeteer-Firefox to drive tests.
- renames the `puppeteer.spec.js` into `launcher.spec.js`. The
`puppeteer.spec.js` is now the entry point for all cross-browsers
tests.
Tracing is working on a per-browser level, not per-page. In order
to paralellize these tests effectively and properly, each should run
a designated browser.
Drop requirement for matching "origin" and "content-type" headers
in requests and request interceptions. This way javascript redirects
that use form submission start working.
Fix#3684.
ExecutionContext.evaluateHandle accepts arguments that are either
serializable, or JSHandles. A potential confusion is that it *does not*
accept arguments that *contain* JSHandles.
This patch adds a log message warning when it encounters that situation.
Fixes#3562
This patch teaches `page.setContent` to await resources in
the new document.
**NOTE**: This patch changes behavior: currently, `page.setContent`
awaits the `"domcontentloaded"` event; with this patch, we can now await
other lifecycle events, and switched default to the `"load"` event.
The change is justified since current behavior made `page.setContent`
unusable for its main designated usecases, pushing our client
to use [dataURL workaround](https://github.com/GoogleChrome/puppeteer/issues/728#issuecomment-334301491).
Fixes#728
NavigatorWatcher subscribes to Connection to get a `Disconnected` event,
causing us to hit the default max of 10 listeners constraint.
Technically we don't leak anything here and can safely bump
the maxListenersCount to Infinity.
However, we conveniently have `CDPSession`, and
can re-dispatch the event on it and keep the safety check in place.
This adds `page.accessibility.snapshot()`. It serializes and returns the accessibility tree for the page. By default, uninteresting nodes are filtered out of the snapshot.
fixes#2033
This patch enables cookie test. The actual upstream patch
that fixed the issue:
- https://crrev.com/599696 - Headless: support cookie encryption
Fixes#921.
This patch fixes a case in which computeQuadArea calculates the area size correctly, but returns the area as a negative number.
This occurs when DOM.getContentQuads returns quads in a specific order.
E.g. the array: [ { x: 463, y: 68.5 },{ x: 437, y: 68.5 },{ x: 437, y: 94.5 },{ x: 463, y: 94.5 } ] will receive area size of -676.
CSS stylesheets can still be parsed and added events emitted during the CSS.stopRuleUsageTracking call. It needs to be awaited before calling CSS.disable, otherwise the text content of those style sheets will be unavailable.
If nobody forces a layout, CSS coverage is inconsistent. This causes some flakes on the bots. The test in this PR fails 90% of the time on my local machine.
This adds `browser.waitForTarget` and `browserContext.waitForTarget`. It also fixes a flaky test that was incorrectly expecting targets to appear instantly.
This patch:
- adds experimental "transport" option to pptr.connect
- uses "transport" option to make sure Puppeteer-Web works with
Target.exposeDevToolsProtocol
Drive-by: add `browser.target()` to access browser target.
This patch introduces API to manage frame navigations.
As a drive-by, the `response.frame()` method is added as a shortcut
for `response.request().frame()`.
Fixes#2918.
This patch traces all public async methods and wraps them
in a helper method that tags the sync stack trace.
Later on, if the method call throws an exception, we add
a captured stack trace to the original stack trace with the "--ASYNC--"
heading.
An example of a stack trace:
```
Error: net::ERR_ABORTED at http://localhost:8907/empty.html
at navigate (/Users/lushnikov/prog/puppeteer/lib/Page.js:622:37)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at Page.<anonymous> (/Users/lushnikov/prog/puppeteer/lib/helper.js:147:27)
at fit (/Users/lushnikov/prog/puppeteer/test/page.spec.js:546:18)
at process._tickCallback (internal/process/next_tick.js:68:7)
```
If referer is passed to the options object its value will be used as the referer instead of the value set by `Page.setExtraHTTPHeaders()`.
This is the correct way to set referer header: otherwise, the `referer` header will override all the document subrequests.
Fixes#3090.
Introduce an API to manage permissions per browser context:
- BrowserContext.overridePermissions(origin, permissions)
- BrowserContext.clearPermissionOverrides()
Fixes#846.
This roll includes:
- https://crrev.com/584293 - DevTools: execute scripts in addScriptToEvaluateOnLoad in order
- https://crrev.com/585630 - DevTools: introduce Browser.grantPermissions
- https://crrev.com/587156 - Revert "[Base] Use background mode for ThreadPriority::BACKGROUND threads (behind feature) (reland)."
The "revert" patch fixes headless functionality on windows.
References #846.
Fixes#3106.
It turned out that almost any usecase requires helper methods to access
DOM inside the ExecutionContext.
Instead of exposing execution contexts as-is, we should introduce
IsolatedWorld as a first-class citizen that will hold execution contexts
inside.
This patch adds a new require, `puppeteer/Errors`, that
holds all the Puppeteer-specific error classes.
Currently, the only custom error class we use is `TimeoutError`. We'll
expand in future with `CrashError` and some others.
Fixes#1694.
This allows us:
- dogfood browser contexts the way we want them to be used
- simplifies the dance around service workers / cookies setting up and tier down.
This changes sendCharacter to use document.execCommand instead of sending a `'char'` event from the protocol. This is more aligned with how input would come in from emoji keyboards, and removes the 3ish byte limit on characters that can be sent which prevented larger emoji from being rendered correctly.
Emoji will still fail to type correctly if typing them into an iframe that is in shadow dom.
fixes#1096
This patch rolls Chromium to r579032. The patch includes:
- https://crrev.com/577366 - DevTools: report redirect responses only if response interception is enabled
- https://crrev.com/577212 - DevTools: intercept requests resulting from redirects
- https://crrev.com/578934 - DevTools: Add a protocol method to insertText
Interception Logic in DevTools protocol has changed regarding redirects;
this patch migrates interceptions to dispatch "request" events based on
requestWillBeSent event.
I have seen some flaky test failures where it would be nice to have run the tests with `DEBUG=puppeteer:error`. Instead of always running tests like that, I am redirecting `debugError` to the output category of the test. This is the same thing that we do for Chromium's stderr.
As a drive-by, I added an additional `debugError` where we were usually a try..finally pattern.
Unfortunately, disabling javascript in page prevents any microtasks
to be executed even from puppeteer-originating javascript. As a
result, the IntersectionObserver hack we use to conditionally
scroll into view doesn't work.
To workaround this, we start always scrolling before clicking if
page's javascript is disabled.
Fixes#2898
Chrome DevTools shows anonymous scripts with yellow background and names
them with `debugger://VM<scriptId>` prefix.
This patch starts reporting the same debugger:// urls for anonymous
scripts in puppeteer's JS coverage. This might simplify debugging, e.g.
using `debugger;` statement to reveal the script in DevTools and later
matching it against the one in the coverage.
This patch:
- simplifies test reusing the `offscreenbuttons.html` asset
- aligns IntersectionObserver usage with the one we have for
`ElementHandle._scrollIntoViewIfNeeded`.
This patch adds `reportAnonymousScripts` option to the `coverage.startJSCoverage` method. With this option, anonymous scripts are reported as well.
Fixes#2777
EmualationManager used to be injecting touch hooks to properly
support touch emulation.
However, these are no longer necessary, since https://crbug.com/133915
is long fixed.
Originally, we use `Element.scrollIntoViewIfNeeded` to make sure
button is on screen before trying to click it.
However, `Element.scrollIntoViewIfNeeded` doesn't work in certain
scenarios, e.g. when element is partially visible and horizontal
scrolling is required to make it fully visible.
This patch polyfills `element.scrollIntoViewIfNeeded` using
IntersectionObserver and `Element.scrollIntoView`.
Fixes#2804.
This patch:
- stops appending `undefined` to our protocol messages unnecessarily.
- rewrites `Cannot find execution context id` to `Execution context was destroyed, most likely because of a navigation.` when it occurs from a Puppeteer ExecutionContext. The error message is left alone if it occurs via a CDPSession.
This patch eliminates a common race condition with WaitTask, that
happens when predicate function gets resolved right before the execution
context gets destroyed.
This situation results in a "Cannot find context with specified id undefined"
exception.
Credits go to @jakub300 for his wonderful [investigation](https://github.com/GoogleChrome/puppeteer/issues/1325#issuecomment-395472092).
Fixes#1325.
`Data.prototype.toString` may return non-ASCII characters, which aren't accepted by `setHeader`.
E.g., on Russian locale, it might look like this:
```
> new Date().toString()
'Thu Jun 14 2018 13:11:50 GMT+0300 (Финляндия (лето))'
```
This patch:
- adds `worker.evaluate` and `worker.evaluateHandle` methods as a shortcut to their execution context equivalents.
- makes the error messages a bit nicer when interacting with a closed worker (as opposed to a closed page).
- moves the worker tests into their own spec file.
This patch disables OOPIF by default.
**NOTE**: this is a temporary bandaid for the time we're crafting
the full-fledged support for site isolation over DevTools protocol.
References #2548.
It's impossible to launch chromium without initial page.
This patch makes sure that `puppeteer.launch()` always returns a browser
with at least one page user can connect to.
Since Node 10, `console.assert` no longer throws an AssertionError.
(This is generally good since it aligns Node.js with Browsers.)
This patch migrates all usages of `console.assert` in our codebase.
- All the `lib/` and testing code is migrated onto a handmade `assert`
function. This is to make Puppeteer transpilation / bundling easier.
- All the tooling is switched to use Node's `assert` module.
Fixes#2547.
Some of the worker tests were failing on the bots. After investigating, I found one race in the test, and one race upstream in Chromium which I filed upstream as https://crbug.com/846099. But I added a small hack here as a temporary workaround.
References #2632
This patch allows logging the output of the Chromium process to be enabled in tests by passing in the environment variable `DUMPIO=true`.
Additionally, the `stderr` of the Chromium process will always be logged in the the "Output" section of failing page tests.
Scrollbars look different on different platforms, so must be made invisible in tests. As a drive-by, xdescribe was broken with the new test runner.
References #2524
This adds `page.workers()`, and two events `workercreated` and `workerdestroyed`. It also forwards logs from a worker into the page `console` event.
Only dedicated workers are supported for now, ServiceWorkers will probably work differently because they aren't necessarily associated with a single page.
Fixes#2350.
This patch introduces Browser Contexts and methods to manage them:
- `browser.createIncognitoBrowserContext()` - to create new incognito
context
- `browser.browserContext()` - to get all existing contexts
- `browserContext.dispose()` - to dispose incognito context.
Fixes#85.
Today, `page.close()` method doesn't run page's beforeunload listeners.
This way users can be sure that `page.close()` actually closes the
page.
This patch adds an optional `runBeforeUnload` option to the
`page.close()` method that would run beforeunload listeners. Note:
running beforeunload handlers might cancel page closing.
Fixes#2386.
This patch teaches Page.waitForNavigation to correctly handle navigation
to pages that have frames that might never load.
These frames include:
- frames which main resource loading was aborted due to mixed-content
error
- frames that artificially called `window.stop()` to interrupt loading
themselves
Fixes#1936.
This patch fixes puppeteer navigation primitives to work with
same-document navigation.
Same-document navigation happens when document's URL is changed,
but document instance is not re-created. Some common scenarios
for same-document navigation are:
- History API
- anchor navigation
With this patch:
- pptr starts dispatching `framenavigated` event when frame's URL gets
changed due to same-document navigation
- `page.waitForNavigation` now works with same-document navigation
- `page.goBack()` and `page.goForward()` are handled correctly.
Fixes#257.
Today, we have tests split into multiple files, with files pulling
tests from some other files.
This patch starts explicitly gathering all tests from the same
`test.js` file.
Drive-By: move one test from `browser.spec.js` into `puppeteer.spec.js`
since it starts browser itself.
This roll includes:
- https://crrev.com/549003 - DevTools: make pptr tests pass with DCHECKs.
The patch fixes a browser crash that happens during browser close.
As a result, cookies were not saved properly (and thus the flaky test we
had).
Fixes#1537.
This roll includes:
- https://crrev.com/547982 - v8 roll that includes [fixed
Runtime.callFunctionOn](1637818671) method
The upstream fix makes it possible to run frame.waitFor* functions on
pages with strict CSP.
References #1229.
This patch introduces a new `pipe` option to the launcher to connect over a pipe.
In certain environments, exposing web socket for remote debugging is a security risk.
Pipe connection eliminates this risk.
This patch adds support for `timeout: 0` to disable timeout for the following functions:
- `page.waitForFunction`
- `page.waitForXPath`
- `page.waitForSelector`
and their `frame` counterparts.
Fixes#2200
This patch:
- starts fulfilling security details for redirect responses
- changes `response.securityDetails()` to return null if the response
is served over non-secure connection
This patch introduces ExecutionContext.frame() that returns Frame
associated with this Execution Context.
This allows to associate console messages with the originating frame,
if any.
This patch:
- introduces `SecurityDetails` class that exposes a set of fields that describe properties of secure connection
- introduces method `response.securityDetails()` that returns an instance of `SecurityDetails` object.
keyboard.down() and keyboard.up() both use the _pressedKeys Set, however keyboard.down() adds and searches for the key code, whereas keyboard.up() attempts to delete based on the key rather than the key code.
Fixes#1901
This patch introduces `BrowserFetcher` class that manages
downloaded versions of products.
This patch:
- shapes Downloader API to be minimal yet usable for our needs. This
includes removing such methods as `Downloader.supportedPlatforms` and
`Downloader.defaultRevision`.
- makes most of the fs-related methods in Downloader async. The only
exception is the `Downloader.revisionInfo`: it has stay sync due to the
`pptr.executablePath()` method being sync.
- updates `install.js` and `utils/check_availability.js` to use new API
- finally, renames `Downloader` into `BrowserFetcher`
Fixes#1748.
This patch:
- introduces `test/assets/cached` folder and teaches server to cache
all the assets from the folder
- introduces `test/assets/serviceworkers` folder that stores all the
service workers and makes them register with unique URL prefix
- introduces `Response.fromCache()` and `Response.fromServiceWorker()`
methods
Fixes#1551.
This PR fixes lost functionality that is no longer on-par with the documentation for `Page.select`, namely:
> `...values` <...string> Values of options to select. If the `<select>` has the `multiple` attribute, all values are considered, **otherwise only the first one is taken into account**.
I've also added an accompanying test for this use case.
This patch:
- introduces `page.waitForXPath` method
- introduces `frame.waitForXPath` method
- amends `page.waitFor` to treat strings that start with `//` as xpath queries.
Fixes#1757.
feat: expose raw devtools protocol connection
This patch introduces `target.createCDPSession` method that
allows directly communicating with the target over the
Chrome DevTools Protocol.
Fixes#31.
If the success value of `waitForFunction` was not serializable, checking whether it was truthy with `.jsonValue()` might fail. Now I check whether it was truthy inside the page.
Fixes#1737.
This patch:
- teaches page.waitFor* methods to accept JSHandles
- starts returning JSHandles from page.waitFor* calls.
BREAKING CHANGE: this patch starts allocating `JSHandle`/`ElementHandle` instances for every call to `page.waitFor*` functions. These handles should be disposed manually to avoid memory consumption.
Fixes#1703, fixes#1654, fixes#1724.
This patch adds two new methods to the `page.coverage` namespace:
- `page.coverage.startCSSCoverage()` - to initiate css coverage
- `page.coverage.stopCSSCoverage()` - to stop css coverage
The coverage format is consistent with the JavaScript coverage.
This patch introduces a new `page.coverage` namespace with two methods:
- `page.coverage.startJSCoverage` to initiate JavaScript coverage
recording
- `page.coverage.stopJSCoverage` to stop JavaScript coverage and get
results
This patch:
- adds `puppeteer.defaultArgs()` method to get default arguments that are used to launch chrome
- adds `ignoreDefaultArgs` option to `puppeteer.launch` to avoid using default puppeteer arguments
Fixes#872
The patch converts all the getters in the codebase into the methods.
For example, the `request.url` getter becomes the `request.url()`
method.
This is done in order to unify the API and make it more predictable.
The general rule for all further changes would be:
- there are no getters/fields exposed in the api
- the only exceptions are "namespaces", e.g. `page.keyboard`
Fixes#280.
BREAKING CHANGE:
This patch ditches getters and replaces them with methods throughout
the API. The following methods were added instead of the fields:
- dialog.type()
- consoleMessage.args()
- consoleMessage.text()
- consoleMessage.type()
- request.headers()
- request.method()
- request.postData()
- request.resourceType()
- request.url()
- response.headers()
- response.ok()
- response.status()
- response.url()
This patch migrates tests so that they can be run concurrently.
- By default, tests still run in one thread.
- To run tests in 4 parallel threads, run node test/test.js -j 4 or npm run unit -- -j 4
- Environment variable PPTR_PARALLEL_TESTS could be set to override default parallelization
Every test gets passed in a state. State is set up in the beforeAll and beforeEach callbacks,
and should be teared down in the afterAll and afterEach callbacks.
By default, state has a parallelIndex variable initialized that defines the thread index that runs the execution.
This patch unifies node6 transpilation:
- instead of generating multiple top-level directories, prefixed with
`node6-`, all transpiled code gets placed under single `node6/` folder
- transpilation doesn't change require paths of transpiled modules any
more
This patch:
- renames ChromiumDownloader into just Downloader (this is in
preparation for different products download)
- moves Downloader from utils/ to lib/. This unifies all of the
production-critical code in the lib/.
Drive-by: make Downloader a regular class.
This patch introduces a tiny test runner to run puppeteer tests.
The test runner is self-container and allows parallel (wrt IO) test execution.
It will also allow us to split tests into multiple files if necessary.
Comparing to the jasmine, the testrunner supports parallel execution, properly
handles "unhandled promise rejection" event and signals.
Comparing to ava/jest, the testrunner doesn't run multiple node processes,
which makes it simpler but sufficient for our goals.
