Commit Graph

19 Commits

Author SHA1 Message Date
Mathias Bynens
eddb23b521
chore: update URLs (#5185) 2019-11-26 13:12:25 +01:00
Andrey Lushnikov
d687c81cd5
chore(flakiness-dashboard): skip API Coverage for realz (#4843) 2019-08-13 16:23:41 -07:00
Andrey Lushnikov
763e3901d9
chore(flaky): save build result (#4819) 2019-08-08 20:53:12 -07:00
Andrey Lushnikov
f753ec6b04
chore(testrunner): distinguish between TERMINATED and CRASHED (#4821)
`testRunner.run()` might have 4 different outcomes:
- `ok` - all non-skipped tests passed
- `failed` - some tests failed or timed out
- `terminated` - process received SIGHUP/SIGINT while testrunner was running tests. This happens on CI's under certain circumstances, e.g. when
  VM is getting re-scheduled.
- `crashed` - testrunner terminated test execution due to either `UnhandledPromiseRejection` or
  some of the hooks (`beforeEach/afterEach/beforeAll/afterAll`) failures.

As an implication, there are 2 new test results: `terminated` and `crashed`.
All possible test results are:
- `ok` - test worked just fine
- `skipped` - test was skipped with `xit`
- `timedout` - test timed out
- `failed` - test threw an exception while running
- `terminated` - testrunner got terminated while running this test
- `crashed` - some `beforeEach` / `afterEach` hook corresponding to this
test timed out of threw an exception.

This patch changes a few parts of the testrunner API:
- `testRunner.run()` now returns an object `{result: string,
terminationError?: Error, terminationMessage?: string}`
- the same object is dispatched via `testRunner.on('finished')` event
- `testRunner.on('terminated')` got removed
- tests now might have `crashed` and `terminated` results
- `testRunner.on('teststarted')` dispatched before running all related
`beforeEach` hooks, and `testRunner.on('testfinished')` dispatched after
running all related `afterEach` hooks.
2019-08-08 15:15:09 -07:00
Andrey Lushnikov
c047624b68
chore: generate testIds on CIs only (#4817) 2019-08-07 10:26:53 -07:00
Andrey Lushnikov
dcff850b6f
chore(flakiness): update flakiness format (#4808)
This patch:
- updates Flakiness Dashboard format to define version per-build
  and to pass COMMIT information
- drops the README.md generation - we'll move on to a designated flakiness
  dashboard viewer
2019-08-06 15:32:55 -07:00
Andrey Lushnikov
5c7c45f0aa
chore(coverage): fix coverage for events (#4787)
We used to track API Coverage for public events, but this was regressed in the refactoring that
introduced `//lib/Events.js`.

This patch:
- Brings back API Coverage for events
- Combines all coverage-generated tests into a single one. This way
we can generate less data for flakiness dashboard.
2019-08-01 16:09:50 -07:00
Andrey Lushnikov
e252dcf200
chore(flakiness): Flakiness Dashboard fixes (#4788)
- fix `FLAKINESS_DASHBOARD_BUILD_URL` to point to a task instead of a build
- do not pretty-print `dashboard.json` when serializing flakiness results
- filter out 'COVERAGE' test(s) so that they don't add up to `dashboard.json` payload. These are useless
- validate certain important options of flakiness dashboard
- more logging to STDOUT to actually say which repo and what branch is getting used
- enhance commit message with a build URL
- use a more compact format for JSON. For 100 runs of 700 tests it yields 21MB json instead of 23MB.
- bump default builds number to 100
2019-08-01 16:09:02 -07:00
Andrey Lushnikov
932c8cbe9a
chore: introduce initial version of flakiness dashboard (#4781)
This patch introduces a dashboard that records test results and
uploads them to https://github.com/aslushnikov/puppeteer-flakiness-dashboard

Since many bots might push results in parallel, each bot pushes
results to its own git branch.

FlakinessDashboard also generates a simple README.md with a flakiness
summary. If this proves to be not enough, we can build a website that
fetches flakiness data and renders it nicely.
2019-07-31 22:23:50 -07:00
Andrey Lushnikov
e3abb0aa32
feat(puppeteer): introduce puppeteer.errors and puppeteer.devices (#4312)
These getters are introduced as a more convenient substitute for
a `require('puppeteer/Errors')` and
`require('puppeteer/DeviceDescriptors')`.

This way we can make cross-browser story nicer - a single require
of `puppeteer` or `puppeteer-firefox` fully defines Puppeteer
environment.
2019-04-19 15:33:06 -07:00
Andrey Lushnikov
1890dc04ba
feat(firefox): Page.waitForRequest/Page.waitForResponse (#3989)
Drive-by: refactor `Request.frame()` tests into a separate test suite.
2019-02-12 19:10:14 -08:00
Andrey Lushnikov
6887ad881a
chore: further unify Puppeteer-Firefox tests with Puppeteer (#3931)
This patch:
* unifies assets between tests
* enables a few puppeteer tests on Puppeteer-Firefox

Drive-by: beautify failing output of `expect.toEqual` matcher.

References #3889
2019-02-06 13:49:14 -08:00
Andrey Lushnikov
62da2366c6
chore: introduce //lib/api.js (#3835)
Introduce `//lib/api.js` that declares a list of publicly exposed
classes.

The `//lib/api.js` list superceedes dynamic `helper.tracePublicAPI()` calls
and is used in the following places:
- [ASYNC STACKS]: generate "async stacks" for publicy exposed API in `//index.js`
- [COVERAGE]: move coverage support from `//lib/helper` to `//test/utils`
- [DOCLINT]: get rid of 'exluded classes' hardcoded list

This will help us to re-use our coverage and doclint infrastructure
for Puppeteer-Firefox.

Drive-By: it turns out we didn't run coverage for `SecurityDetails`
class, so we lack coverage for a few methods there. These are excluded
for now, sanity tests will be added in a follow-up.
2019-01-25 23:21:14 -05:00
Andrey Lushnikov
5acf953104
feat(frame): introduce Frame.goto and Frame.waitForNavigation (#3276)
This patch introduces API to manage frame navigations.
As a drive-by, the `response.frame()` method is added as a shortcut
for `response.request().frame()`.

Fixes #2918.
2018-09-20 11:31:19 -07:00
Andrey Lushnikov
204c7ec8c4
feat: introduce puppeteer/Errors (#3056)
This patch adds a new require, `puppeteer/Errors`, that
holds all the Puppeteer-specific error classes.

Currently, the only custom error class we use is `TimeoutError`. We'll
expand in future with `CrashError` and some others.

Fixes #1694.
2018-08-09 16:51:12 -07:00
Andrey Lushnikov
231a2be971
feat: expose frame's execution contexts (#3048)
This patch exposes frame's execution contexts, making it possible
to debug extension's content scripts.

This is a resurrected #2812.
2018-08-09 14:57:08 -07:00
Andrey Lushnikov
c430138845
test: drop PDF tests (#2744)
We fully rely on upstream PDF functionality. Our PDF tests are inferior
to those upstream and inconvenient to write.
2018-06-14 16:42:10 -07:00
Andrey Lushnikov
06d61919ef
test: refactor utils.waitForEvents into utils.waitEvent (#2336)
This simplifies waiting for events in tests.
2018-04-09 15:46:05 -07:00
Yaniv Efraim
47481967c5 test: Break 'page.spec.js' to smaller files (#2218)
This patch breaks huge `page.spec.js` into a bunch of smaller files.
2018-03-19 20:00:12 -07:00