* chore: enforce pinned dependencies
Because we don't check our `package-lock.json` in, we can end up with
different versions installed locally vs CI, or even two devs having
different versions. Let's pin and enforce we pin every version to
avoid this.
This PR updates some code to remove constant ESLint warnings. It also
upgrades those warnings to errors - so that they have to be resolved
as part of the PR, rather than landing as a warning and causing noise.
Fixes#7229.
We're seeing odd failures with Prettier on some CI branches; my hunch is that they are installing different versions of the package and therefore getting formatting conflicts. This PR updates them all and pins them to specific versions - something we should probably consider generally, or remove our `package-lock.json` from the gitignore.
Currently, `npm clean-lib` fails on windows with `cmd` because it does not now about `rm`.
This change uses the already installed `rimraf` to do the job instead.
* feat(chromium): roll Chromium to r856583
This corresponds to Chromium 90.0.4427.0
This roll includes:
- Add sourceScheme, sourcePort, and sameParty to DevTools backend (https://crbug.com/1170548, https://crbug.com/1142606)
We were blocked on doing this because API Extractor didn't support it,
but now it does, so we can bump TS and the API tooling in one go. None
of the breaking changes in TS4 cause us any issues.
* fix: wider compat TS types and CI checks to ensure correct type defs
This PR improves our TS types further to make sure they are usable in a
TS environment where ES Modules are the target output. Our use of
`export =` is problematic this environment as TypeScript does not allow
`export =` to be used and it errors.
The fix for the type issues to avoid `export =` is to instead define the
functions that you gain access to when you import Puppeteer as top level
functions in our `types.d.ts` file. We can do this by declaring them
explicitly in `src/node.ts`. These are then rolled into `lib/types.d.ts`
at build time. The downside to this is that we have to keep those
declarations in sync with the Puppeteer API; should we add a new method
to the `Puppeteer` class, we must add it to the `nodes.ts` declarations.
However, this could easily be automated by a small script that walks the
AST and generates these. I will do that in a follow-up PR, but I
consider this low risk given how rarely the very top level API of
Puppeteer changes. The nice thing about this approach is we no longer
need our script that hacks on changes to `lib/types.d.ts`.
To avoid yet more releases to fix issues in one particular TS
environment, this PR also includes a suite of example setups that we
test on each CI run. Each sample folder contains `good.ts`, which should
have no TS errors, and `bad.ts`, which should have some errors. The test
first packs Puppeteer into a tar, and then installs it from that tar
into each project. This should replicate how the published package
behaves when it is installed. We then check that we get no errors on
`good.ts`, and the expected errors on `bad.ts`.
We have a variety of test projects that cover both TS and JS source
code, and CJS and ESM imports and outputs.
Else it's missed when npm publish happens. Didn't pick this up in my testing because a previous PR that changed this hadn't landed at the time of testing so the bug was missed.
This PR aims to vastly improve our TS types and how we ship them.
Our previous attempt at shipping TypeScript was unfortunately flawed for
many reasons when compared to the @types/puppeteer package:
* It only worked if you needed the default export. If you wanted to
import a type that Puppeteer uses, you'd have to do `import type X
from 'puppeteer/lib/...'`. This is not something we want to encourage
because that means our internal file structure becomes almost public
API.
* It gave absolutely no help to CommonJS users in JS files because it
would warn people they needed to do `const pptr =
require('puppeteer').default, which is not correct.
* I found a bug in the `evaluate` types which mean't you couldn't
override the types to provide more info, and TS would insist the types
were all `unknown`.
The goal of this PR is to support:
1. In a `ts` file, `import puppeteer from 'puppeteer'`
1. In a `ts` file, `import type {ElementHandle} from 'puppeteer'`
1. In a `ts` file, referencing a type as `puppeteer.ElementHandle`
1. In a `ts` file, you can get good type inference when running
`foo.evaluate(x => x.clientHeight)`.
1. In a `js` file using CJS, you can do `const puppeteer =
require('puppeteer')` and get good type help from VSCode.
To test this I created a new empty repository with two test files in,
one `.ts` file with this in:
https://gist.github.com/jackfranklin/22ba2f390f97c7312cd70025a2096fc8,
and a `js` file with this in:
https://gist.github.com/jackfranklin/06bed136fdb22419cb7a8a9a4d4ef32f.
These files included enough code to check that the types were behaving
as I expected.
The fix for our types was to make use of API Extractor, which we already
use for our docs, to "rollup" all the disparate type files that TS
generates into one large `types.d.ts` which contains all the various
types that we define, such as:
```ts
export declare class ElementHandle {...}
export type EvaluateFn ...
```
If we then update our `package.json` `types` field to point to that file
in `lib/types.d.ts`, this then allows a developer to write:
```
import type {ElementHandle} from 'puppeteer'
```
And get the correct type definitions. However, what the `types.d.ts`
file doesn't do out of the box is declare the default export, so
importing Puppeteer's default export to call a method such as `launch`
on it will get you an error.
That's where the `script/add-default-export-to-types.ts` comes in. It
appends the following to the auto-generated `types.d.ts` file:
```ts
declare const puppeteer: PuppeteerNode;
export = puppeteer;
```
This tells TypeScript what the default export is, and by using the
`export =` syntax, we make sure TS understands both in a TS ESM
environment and in a JS CJS environment.
Now the `build` step, which is run by GitHub Actions when we release,
will generate the `.d.ts` file and then extend it with the default
export code.
To ensure that I was generating a valid package, I created a new
repository locally with the two code samples linked in Gists above. I
then ran:
```
npm init -y
npm install --save-dev typescript
npx tsc --init
```
Which gives me a base to test from. In Puppeteer, I ran `npm pack`,
which packs the module into a tar that's almost identical to what would
be published, so I can be confident that the .d.ts files in there are
what would be published.
I then installed it:
```
npm install --save-dev ../../puppeteer/puppeteer-7.0.1-post.tgz
```
And then reloaded VSCode in my dummy project. By deliberately making
typos and hovering over the code, I could confirm that all the goals
listed above were met, and this seems like a vast improvement on our
types.
This PR updates our `files` list to be more specific; rather than include everything from `lib`, we include just `.js`, `.d.ts`, and their equivalent sourcemaps. This prevents noisy meta-files like `.tsbuildinfo` sneaking into the package, which are no use to anyone.
In all my excitement about shipping types, we forgot to export the
actual `d.ts` file that makes it all happen... :(
I didn't pick this up locally because I was testing with `npm link`,
which does a symbolic link and therefore doesn't mirror the list of
files that make it into the published package.
To test this change, I made the change and ran `npm pack` to generate a
tar. I then created a new empty directory and did `npm init -y` followed
by `npm install path/to/puppeteer-7.tgz`.
I then initialised TypeScript, and wrote this:
```js
import puppeteer from 'puppeteer';
async function run() {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://foo.com')
}
run();
```
As I typed I got autocompletions and if I were to make an error, I'd get
a TypeScript error.
There is follow up work to be done because this unfortunately does not
work if you use `require` rather than ES Modules. @AviVahl suggested a
change in https://github.com/puppeteer/puppeteer/issues/6807 but I
cannot get it to work, so I'd like to do more investigation here. In the
mean time, whatever the contents of `cjs-entry.d.ts`, we should
definitely be exporting it as part of the module, so this PR is still
good to land.
This corresponds to Chromium 90.0.4403.0
This roll includes:
- Cut screenshot by ViewPort size, not position (crrev.com/c/2643792)
BREAKING CHANGE:
- `page.screenshot` cuts screenshot content by the ViewPort size, not ViewPort position.
This corresponds to Chromium 89.0.4389.0.
This roll includes:
- Add `SameParty` attribute to cookies
https://crrev.com/c/2598846
- Anchor `target=_blank` implies `rel=noopener`
https://crrev.com/c/1630010
- Don’t expect ignored elements in the AXTree
https://crrev.com/c/2505362
BREAKING CHANGE: The built-in `aria/` selector query handler doesn’t return ignored elements anymore.
Issue: #6758
Introduce the source-map-support package and require it for mocha running unit tests.
Turn on the sourceMap option for tsconfig.base.json so that the sourceMappingURL= line is emitted in the generated files.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
With `nodejs@15.0.1`, install puppeteer with `https_proxy` set causes an error like:
```
> puppeteer@5.4.1 install node_modules/puppeteer
> node install.js
ERROR: Failed to set up Chromium r809590! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
TypeError [ERR_INVALID_PROTOCOL]: Protocol "https:" not supported. Expected "http:"
at new NodeError (node:internal/errors:258:15)
at new ClientRequest (node:_http_client:155:11)
at Object.request (node:https:313:10)
at httpRequest (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:488:17)
at downloadFile (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:357:21)
at BrowserFetcher.download (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:239:19)
at async downloadBrowser (node_modules/puppeteer/lib/cjs/puppeteer/node/install.js:48:5) {
code: 'ERR_INVALID_PROTOCOL'
}
```
The related issue is at https://github.com/TooTallNate/node-agent-base/pull/47, from package `agent-base` under `https-proxy-agent`
And the version bump is for `Refactor to TypeScript` is here: https://github.com/TooTallNate/node-https-proxy-agent/compare/4.0.0...5.0.0
Remove the redundant rule for dependency graph as the `src/index.ts` is not required. This was introduced in 64c9c709, but the filestructure has changed from since then and it doesn't work as `src/index.ts` is replaced with `src/node.ts` and `src/web.ts` as per the use case.
As long as we follow Conventional Commits for our commit messages (which is now enforced since #6483), we can automate the maintenance of this new changelog, which enables us to later automate the release process altogether.
This patch also eliminates the versioning decision-making process by automating it based on our commit messages.
Issue: #6482
* chore(agnostification): common/helper.ts
The `readProtocolStream` method uses `fs` only if you want to write to a
file. So we gate at the start of the function and ensure that if we got
given a path we are not in a Node environment.
They are generating a lot of noise in PRs. This commit removes them from git, but updates CI to generate them - to ensure there are no errors when generating the new documentation.
Further commits will:
1. Introduce linting to enforce methods are documented.
2. Generate previews of the new docs via GitHub actions or similar.
The `Launcher` class was serving two purposes:
1. Launch browsers
2. Connect to browsers
Number 1) only needs to be done in Node land, but 2) is agnostic; in a
browser version of Puppeteer we'll need the ability to connect over a
websocket to send commands back and forth.
As part of the agnostification work we needed to split the `Launcher` up
so that the connection part can be made agnostic. Additionally, I
removed dependencies on `https`, `http` and `URL` from Node, instead
leaning on fetch (via `node-fetch` if in Node land) and the browser
`URL` API (which was added to Node in Node 10).
This commit updates some miscellaneous dependencies to their latest
versions (with no other changes required) and also removes the `browser`
section, which was used by Browserify for the now long gone
Puppeteer-Web package that we used to publish.
This patch sets up commitlint to enforce the Conventional Commits format. This check runs with the other lint checks as part of npm run lint, and a Git commit hook is set up via Husky for automated local checks.
Issue: #6482
In `src/common` we now use `fs.promises.X` which we can dynamically
`import`. In a browser environment this code will never run because it's
gated on `isNode` (in a future PR we will add tree-shaking to the bundle
step such that this code is eliminated). By using `import`, we ensure
TypeScript still can track types and give good type information.
In `src/node` we continue to use `util.promisify` but that's not a
concern as that code explicitly is never run in the browser.
* roll Chromium to version 86, r800071
* roll Chrome DevTools protocol version to 0.0.799653
* fix HTTPRequest.continue after
* CDP: accept post data in the binary form in Fetch.continueRequest.
https://chromium-review.googlesource.com/c/chromium/src/+/2315239
* update new-docs
This corresponds to Chromium 85.0.4182.0.
This roll includes:
- Enable SameSiteByDefaultCookies and CookiesWithoutSameSiteMustBeSecure
https://crrev.com/c/2231445
- [FlexNG] Enable FlexNG by default
https://crrev.com/c/2216595Closes#6151.
* chore: vendor Mitt into src/common/third-party
As discussed in #6203 we need to vendor our common dependencies in so
that when we ship an ESM build all imports point to file paths and do
not rely on Node resolution (e.g. a browser does not understand `import
mitt from 'mitt'`).
* chore: enforce file extensions on imports
To make our output agnostic it should include file extensions in the
output, as per the ESM spec. It's a bit odd for Node packages but makes
it easier to publish a browser build.
* feat(chromium): roll Chromium to r768783
* fix: update unit test for crrev:2135046
* chore: update devtools-protocol revision
Co-authored-by: Changhao Han <changhaohan@chromium.org>
* chore: Use devtools-protocol package
Rather than maintain our own protocol we can instead use the devtools-protocol package and pin it to the version of Chromium that Puppeteer is shipping with.
The only changes are naming changes between the bespoke protocol that Puppeteer created and the devtools-protocol one.
Now the async hooks helper is gone api.ts was only used by the coverage
tools and by doclint.
DocLint is nearing the end of its lifespan with the TSDoc work, so I
focused on how best to define a list of modules for the coverage
tooling. They define an object of classes, and the path to that module.
They need the full path because we also check if the module exports any
events that need to be emitted - the coverage tool asserts that the
emitting of those events is also tested.
It's not _great_ that DocLint relies on a constant defined in the
coverage utils, but it should only be this way for a short period of
time and no one is actively working on DocLint (bar the effort to remove
it) so I don't think this is worth worrying about.
This change also broke the DocLint tests; based on the fact that DocLint is on its way out it doesn't feel worth fixing the tests, so this commit also removes them.
If you want to run TypeScript only to verify that it's typechecking correctly, this command is quicker as it doesn't output CJS and ESM to disk. Useful for checking during development.
* chore: Don't store revisions in `package.json`
It's quite messy to have to require the `package.json` file in multiple
places purely to find out what revision of a given browser we want to
use. We can also achieve better type safety by placing it in an actual
source file.
This commit makes that change and also tidies up our reliance on
`package.json` within the source code generally; we now only use it to
find the location of the Puppeteer root such that we know where to
install downloaded browsers to.
To avoid using `package.json` to parse the name of the module, we also
now explicitly have an entry point for the Puppeteer module and the
Puppeter Core module. This will make it easier in the future to ship
less code as part of core (e.g. core never needs to download a browser,
so why ship that code?). Core can also then not have any revisions based
info contained in it.
The test install script has also been updated to ensure that
puppeteer-core can be installed correctly too.
Finally, the `install` script has been moved to TypeScript for nicer
typechecking and safety. The functionality of it has not changed.
* chore(agnostic): ship CJS and ESM builds
For our work to enable Puppeteer in other environments (e.g. a browser)
we need to ship an ESM build. This commit changes our config to ship to
`lib/cjs` and `lib/esm` accordingly. The majority of our code stays the
same, with one small fix for the CJS build to ensure that we ship a
version that lets you `require('puppeteer')` rather than have to
`require('puppeteer').default`. We do this with the `cjs-entry.js` which
is what the `main` field in our `package.json` points to.
We also swap to `read-pkg-up` to find the `package.json` file. This is
because the folder structure of `lib/` does not match `src/` now we ship
to `cjs` and `esm`, so you cannot rely on exact paths. This module works
up from the file to find the nearest `package.json` so it will always
find Puppeteer's `package.json`.
Note that we *do not* point any users to the ESM build. We happen to
ship those files so people who know about them can get at them but it's
not expected (nor will we actively support) that people will rely on
them. The CommonJS build is considered our main build.
We may make breaking changes to the structure of the ESM build which we
will do without requiring new major versions. For example the ESM build
currently ships all files that the CJS build does, but given we are
working on the ESM build being able to run in the browser this may
change over time.
Long term once the Node versions catch up we can ditch CJS and ship
exclusively ESM but we are not there yet.
This CL migrates all the tests to TypeScript. The main benefits of this is that we start consuming our TypeScript definitions and therefore find errors in them. The act of migrating found some bugs in our definitions and now we can be sure to avoid them going forwards.
You'll notice the addition of some `TODO`s in the code; I didn't want this CL to get any bigger than it already is but I intend to follow those up once this lands. It's mostly figuring out how to extend the `expect` types with our `toBeGolden` helpers and some other slight confusions with types that the tests exposed.
Co-authored-by: Mathias Bynens <mathias@qiwi.be>
Without the API-* dependencies pinned different versions may be
installed on local machines vs CI. One of the checks we do is to check
that the checked in docs matches what is generated on CI. Therefore we
need to ensure devs locally run the exact version that CI runs such that
they generate the same output. So in this case we pin to a particular
version of the dependencies.
As far as I can tell these became irrelevant as of v1.15 which added
`puppeteer.errors` and `puppeteer.devices [1]. This is a breaking change
but one that's easily mitigated. We've said that we don't consider
changes to our folder/file structure a breaking change, but we can't
really do that if we have these two top level files that we've
documented.
[1]: e3abb0aa32 (diff-522b24108d7446af4c59873472a90444)
It was causing some infra issues when trying to migrate tests to
TypeScript (that's WIP in another branch that I'll have up soon). It's
unusual to have the entire src in TS except for the main file, which
then reaches into the compiled `lib` directory for the files it needs.
Much better is to move the entry point into TypeScript itself and update
the `main` entry in our `package.json` to point to the compiled output.
This also has the advantange of hooking up all the TS type defs that we
are shipping and will make that process easier too, along with making it
easier to port our tests to TypeScript.
