puppeteer/third_party/phantomjs/test/writing-tests.md
2017-05-11 00:06:41 -07:00

597 lines
25 KiB
Markdown

# How to Write Tests for PhantomJS
PhantomJS's automated tests are `.js` files located in subdirectories
of this directory. The test runner, [`run-tests.py`](run-tests.py),
executes each as a PhantomJS controller script. (Not all
subdirectories of this directory contain tests; the authoritative list
of test subdirectories is in
[the `TESTS` variable in `run-tests.py`](run-tests.py#L26).)
In addition to all of the usual PhantomJS API, these scripts have
access to a special testing API, loosely based on
[W3C testharness.js](https://github.com/w3c/testharness.js) and
defined in [`testharness.js`](testharness.js) in this directory. They
also have access to HTTP and HTTPS servers on `localhost`, which serve
the files in the [`www`](www) directory.
## The Structure of Test Scripts
Test scripts are divided into _subtests_. There are two kinds of
subtest: synchronous and asynchronous. The only difference is that a
synchronous subtest consists of a single JavaScript function (plus
anything it calls) and is considered to be complete when that function
returns. An asynchronous subtest, however, consists of many
JavaScript functions; these functions are referred to as _steps_. One
step will be invoked to start the subtest, and the others are called
in response to events. Eventually one of the steps will indicate that
the subtest is complete. (The single function used for a synchronous
subtest is also referred to as a step, when the difference doesn't
matter.)
All of the code in a test script should be part of a subtest, or part
of a setup function (see below). You may define helper functions at
top level, so long as they are only ever _called_ from subtests. You
may also initialize global variables at top level if this
initialization cannot possibly fail, e.g. with constant data or with
`require` calls for core PhantomJS modules.
The testing API is visible to a test script as a collection of global
functions and variables, documented below.
A subtest is considered to have failed if any of its steps throws a
JavaScript exception, if an `unreached_func` is called, if the
per-test timeout expires, or if `done` is called before all of its
steps have run. It is considered to succeed if `done` is called
(explicitly or implicitly) after all of its steps have run to
completion. Normally, you should use the assertion functions to detect
failure conditions; these ensure that clear diagnostic information is
printed when a test fails.
Subtests are executed strictly in the order they appear in the file,
even if some of them are synchronous and some are asynchronous. The
order of steps *within* an asynchronous subtest, however, may be
unpredictable. Also, subtests do not begin to execute until after all
top-level code in the file has been evaluated.
Some anomalous conditions are reported not as a failure, but as an
"error". For instance, if PhantomJS crashes during a test run, or if
an exception is thrown from outside a step, that is an error, and the
entire test will be abandoned.
**WARNING:** The harness will become confused if any function passed
as the `func` argument to `test`, `async_test`, `generate_tests`,
`Test.step`, `Test.step_func`, `Test.step_func_done`, or
`Test.add_cleanup` has a name that ends with `_step`.
### Accessing the HTTP and HTTPS Test Servers
The global variables `TEST_HTTP_BASE` and `TEST_HTTPS_BASE` are the
base URLs of the test HTTP and HTTPS servers, respectively. Their
values are guaranteed to match the regex `/https?:\/\/localhost:[0-9]+\//`,
but the port number is dynamically assigned for each test run, so you
must not hardwire it.
### Synchronous Subtests
There are two functions for defining synchronous subtests.
* `test(func, name, properties)`
Run `func` as a subtest; the subtest is considered to be complete as
soon as it returns. `name` is an optional descriptive name for the
subtest. `func` will be called with no arguments, and `this` set to
a `Test` object (see below). `properties` is an optional object
containing properties to apply to the test. Currently there are
three meaningful properties; any other keys in the `properties`
object are ignored:
* `timeout` - Maximum amount of time the subtest is allowed to run, in
milliseconds. If the timeout expires, the subtest will be
considered to have failed.
* `expected_fail`: If truthy, this subtest is expected to fail. It will
still be run, but a failure will be considered "normal" and the
overall test will be reported as successful. Conversely, if the
subtest *succeeds* that is considered "abnormal" and the overall
test will be reported as failing.
* `skip`: If truthy, this subtest will not be run at all, will be
reported as "skipped" rather than "succeeded" or "failed", and
will not affect the overall outcome of the test. Use `skip` only
when `expected_fail` will not do—for instance, when a test
provokes a PhantomJS crash (there currently being no way to label
a crash as "expected").
`test` returns the same `Test` object that is available as `this`
within `func`.
* `generate_tests(func, args, properties)`
Define a group of synchronous subtests, each of which will call
`func`, but with different arguments. This is easiest to explain by
example:
generate_tests(assert_equals, [
["Sum one and one", 1 + 1, 2],
["Sum one and zero", 1 + 0, 1]
]);
... is equivalent to ...
test(function() {assert_equals(1+1, 2)}, "Sum one and one");
test(function() {assert_equals(1+0, 1)}, "Sum one and zero");
Of course `func` may be as complicated as you like, and there is no
limit either to the number of arguments passed to each subtest, or
to the number of subtests.
The `properties` argument may be a single properties object, which
will be applied uniformly to all the subtests, or an array of the
same length as `args`, containing appropriate property objects for
each subtest.
`generate_tests` returns no value.
### Asynchronous Subtests
An asynchronous subtest consists of one or more _step_ functions, and
unlike a synchronous subtest, it is not considered to be complete until
the `done` method is called on its `Test` object. When this happens,
if any of the step functions have not been executed, the subtest is a
failure.
Asynchronous subtests are defined with the `async_test` function, which
is almost the same as `test`:
* `async_test(func, name, properties)`
Define an asynchronous subtest. The arguments and their
interpretation are the same as for `test`, except that `func` is
optional, and the subtest is *not* considered to be complete after
`func` returns; `func` (if present) is only the first step of the
subtest. Additional steps may be defined, either within `func` or
outside the call to `async_test`, by use of methods on the `Test`
object that is returned (and available as `this` within `func`).
Normally, an asynchronous subtest's first step will set up whatever
is being tested, and define the remainder of the steps, which will
be run in response to events.
### Test Object Methods
These methods are provided by the `Test` object which is returned by
`test` and `async_test`, and available as `this` within step
functions.
* `Test.step(func[, this_obj, ...])`
Queue one step of a subtest. `func` will eventually be called, with
`this` set to `this_obj`, or (if `this_obj` is null or absent) to
the `Test` object. Any further arguments will be passed down to
`func`. `func` will _not_ be called if a previous step has failed.
Do not use this function to define steps that should run in response
to event callbacks; only steps that should be automatically run by
the test harness.
The object returned by this function is private. Please let us know
if you think you need to use it.
* `Test.done()`
Indicate that this subtest is complete. One, and only one, step of
an asynchronous subtest must call this function, or the subtest will
never complete (and eventually it will time out).
If an asynchronous subtest has several steps, but not all of them
have run when `done` is called, the subtest is considered to have
failed.
* `Test.step_func(func[, this_obj])`
Register `func` as a step that will *not* be run automatically by
the test harness. Instead, the function *returned* by this function
(the "callback") will run `func`'s step when it is called.
(`func`'s step must still somehow get run before `done` is called,
or the subtest will be considered to have failed.)
`this_obj` will be supplied as `this` to `func`; if omitted, it
defaults to the `Test` object. Further arguments are ignored.
However, `func` will receive all of the arguments passed to the
callback, and the callback will return whatever `func` returns.
This is the normal way to register a step that should run in
response to an event. For instance, here is a minimal test of a
page load:
async_test(function () {
var p = require('webpage').create();
p.open(TEST_HTTP_BASE + 'hello.html',
this.step_func(function (status) {
assert_equals(status, 'success');
this.done();
}));
});
This also serves to illustrate why asynchronous subtests may be
necessary: this subtest is not complete when its first step returns,
only when the `onLoadFinished` event fires.
* `Test.step_func_done([func[, this_obj]])`
Same as `Test.step_func`, but the callback additionally calls `done`
after `func` returns. `func` may be omitted, in which case the
callback just calls `done`.
The example above can be shortened to
async_test(function () {
var p = require('webpage').create();
p.open(TEST_HTTP_BASE + 'hello.html',
this.step_func_done(function (status) {
assert_equals(status, 'success');
}));
});
* `Test.unreached_func([description])`
Returns a function that, if called, will call
`assert_unreached(description)` inside a step. Use this to set
event handlers for events that should _not_ happen. You need to use
this method instead of `step_func(function () { assert_unreached(); })`
so the step is properly marked as expected _not_ to run; otherwise
the test will fail whether or not the event happens.
A slightly more thorough test of a page load might read
async_test(function () {
var p = require('webpage').create();
p.onResourceError = this.unreached_func("onResourceError");
p.open(TEST_HTTP_BASE + 'hello.html',
this.step_func_done(function (status) {
assert_equals(status, 'success');
}));
});
* `Test.add_cleanup(func)`
Register `func` to be called (with no arguments, and `this` set to
the `Test` object) when `done` is called, whether or not the subtest
has failed. Use this to deallocate persistent resources or undo
changes to global state. For example, a subtest that uses a scratch
file might read
test(function () {
var fs = require('fs');
var f = fs.open('scratch_file', 'w');
this.add_cleanup(function () {
f.close();
fs.remove('scratch_file');
});
// ... more test logic here ...
});
If the step function had simply deleted the file at its end, the
file would only get deleted when the test succeeds. This example
could be rewritten using `try ... finally ...`, but that will not
work for asynchronous tests.
* `Test.fail(message)`
Explicitly mark this subtest has having failed, with failure message
`message`. You should not normally need to call this function yourself.
* `Test.force_timeout()`
Explicitly mark this subtest as having failed because its timeout has
expired. You should not normally need to call this function yourself.
### Test Script-Wide Setup
All of the subtests of a test script are normally run, even if one of
them fails. Complex tests may involve complex initialization actions
that may fail, in which case the entire test script should be aborted.
There is also a small amount of harness-wide configuration that is
possible. Both these tasks are handled by the global function
`setup`.
* `setup([func], [properties])`
One may specify either `func` or `properties` or both, but if both
are specified, `func` must be first.
`func` is called immediately, with no arguments. If it throws an
exception, the entire test script is considered to have failed and
none of the subtests are run.
`properties` is an object containing one or more of the following
keys:
* `explicit_done`: Wait for the global function `done` (not to be
confused with `Test.done` to be called, before declaring the test
script complete.
* `allow_uncaught_exception`: Don't treat an uncaught exception from
non-test code as an error. (Exceptions thrown out of test steps
are still errors.)
* `timeout`: Overall timeout for the test script, in milliseconds.
The default is five seconds. Note that `run-tests.py` imposes a
"backstop" timeout itself; if you raise this timeout you may also
need to raise that one.
* `test_timeout`: Default timeout for individual subtests. This may
be overridden by the `timeout` property on a specific subtest.
The default is not to have a timeout for individual subtests.
### Assertions
Whenever possible, use these functions to detect failure conditions.
All of them either throw a JavaScript exception (when the test fails)
or return no value (when the test succeeds). All take one or more
values to be tested, and an optional _description_. If present, the
description should be a string to be printed to clarify why the test
has failed. (The assertions all go to some length to print out values
that were not as expected in a clear format, so descriptions will
often not be necessary.)
* `assert_is_true(value[, description])`
`value` must be strictly equal to `true`.
* `assert_is_false(value[, description])`
`value` must be strictly equal to `false`.
* `assert_equals(actual, expected[, description])`
`actual` and `expected` must be shallowly, strictly equal. The
criterion used is `===` with the following exceptions:
* If `x === y`, but one of them is `+0` and the other is `-0`, they
are *not* considered equal.
* If `x !== y`, but one of the following cases holds, they *are*
considered equal:
* both are `NaN`
* both are `Date` objects and `x.getTime() === y.getTime()`
* both are `RegExp` objects and `x.toString() === y.toString()`
* `assert_not_equals(actual, expected[, description])`
`actual` and `expected` must *not* be shallowly, strictly equal,
using the same criterion as `assert_equals`.
* `assert_deep_equals(actual, expected[, description])`
If `actual` and `expected` are not objects, or if they are
`Date` or `RegExp` objects, this is the same as `assert_equals`.
Objects that are not `Date` or `RegExp` must have the same set of
own-properties (including non-enumerable own-properties), and each
pair of values for with each own-property must be `deep_equals`,
recursively. Prototype chains are ignored. Back-references are
detected and ignored; they will not cause an infinite recursion.
* `assert_approx_equals(actual, expected, epsilon[, description])`
The absolute value of the difference between `actual` and `expected`
must be no greater than `epsilon`. All three arguments must be
primitive numbers.
* `assert_less_than(actual, expected[, description])`
* `assert_less_than_equal(actual, expected[, description])`
* `assert_greater_than(actual, expected[, description])`
* `assert_greater_than_equal(actual, expected[, description])`
`actual` and `expected` must be primitive numbers, and `actual` must
be, respectively: less than, less than or equal to, greater than,
greater than or equal to `expected`.
* `assert_in_array(value, array[, description])`
`array` must contain `value` according to `Array.indexOf`.
* `assert_regexp_match(string, regexp[, description])`
The regular expression `regexp` must match the string `string`,
according to `RegExp.test()`.
* `assert_regexp_not_match(string, regexp[, description])`
The regular expression `regexp` must *not* match the string `string`,
according to `RegExp.test()`.
* `assert_type_of(object, type[, description])`
`typeof object` must be strictly equal to `type`.
* `assert_instance_of(object, type[, description])`
`object instanceof type` must be true.
* `assert_class_string(object, expected[, description])`
`object` must have the class string `expected`. The class string is
the second word in the string returned by `Object.prototype.toString`:
for instance, `({}).toString.call([])` returns `[object Array]`, so
`[]`'s class string is `Array`.
* `assert_own_property(object, name[, description])`
`object` must have an own-property named `name`.
* `assert_inherits(object, name[, description])`
`object` must inherit a property named `name`; that is,
`name in object` must be true but `object.hasOwnProperty(name)`
must be false.
* `assert_no_property(object, name[, description])`
`object` must neither have nor inherit a property named `name`.
* `assert_readonly(object, name[, description])`
`object` must have an own-property named `name` which is marked
read-only (according to `Object.getOwnPropertyDescriptor`).
* `assert_throws(code, func[, description])`
`func` must throw an exception described by `code`. `func` is
called with no arguments and no `this` (you can supply arguments
using `bind`). `code` can take one of two forms. If it is a
string, the thrown exception must either stringify to that string,
or it must be a DOMException whose `name` property is that string.
Otherwise, `code` must be an object with one or more of the
properties `code`, `name`, and `message`; whichever properties are
present must be `===` to the corresponding properties of the
exception. As a special case, if `message` is present in the
`code` object but *not* on the exception object, and the exception
stringifies to the same string as `message`, that's also considered
valid.
`assert_throws` cannot be used to catch the exception thrown by any
of the other `assert_*` functions when they fail. (You might be
looking for the `expected_fail` property on a subtest.)
* `assert_unreached([description])`
Control flow must not reach the point where this assertion appears.
(In other words, this assertion fails unconditionally.)
## Test Annotations
Some tests need to be run in a special way. You can indicate this to
the test runner with _annotations_. Annotations are lines in the test
script that begin with the three characters '`//!`'. They must be all
together at the very top of the script; `run-tests.py` stops parsing
at the first line that does _not_ begin with the annotation marker.
Annotation lines are split into _tokens_ in a shell-like fashion,
which means they are normally separated by whitespace, but you can use
backslashes or quotes to put whitespace inside a token. Backslashes
are significant inside double quotes, but not inside single quotes.
There can be any number of tokens on a line. Everything following an
unquoted, un-backslashed `#` is discarded. (The exact algorithm is
[`shlex.split`](https://docs.python.org/2/library/shlex.html), in
`comments=True`, `posix=True` mode.)
These are the recognized tokens:
* `no-harness`: Run this test script directly; the testing API
described above will not be available. This is necessary to
test PhantomJS features that `testharness.js` reserves for its
own use, such as `phantom.onError` and `phantom.exit`. No-harness
tests will usually also be output-expectations tests (see below)
but this is not required.
* `snakeoil`: Instruct PhantomJS to accept the self-signed
certificate presented by the HTTPS test server.
* `timeout:` The next token on the line must be a positive
floating-point number. `run-tests.py` will kill the PhantomJS
process, and consider the test to have failed, if it runs for longer
than that many seconds. The default timeout is seven seconds.
This timeout is separate from the per-subtest and global timeouts
enforced by the testing API. It is intended as a backstop against
bugs which cause PhantomJS to stop executing the controller script.
(In no-harness mode, it's the only timeout, unless you implement
your own.)
* `phantomjs:` All subsequent tokens on the line will be passed as
command-line arguments to PhantomJS, before the controller script.
Note that `run-tests.py` sets several PhantomJS command line options
itself; you must take care not to do something contradictory.
* `script:` All subsequent tokens on the line will be passed as
command-line arguments to the *controller script*; that is, they
will be available in
[`system.args`](http://phantomjs.org/api/system/property/args.html).
Note that your controller script will only be `system.args[0]` if
you are using no-harness mode, and that `run-tests.py` may pass
additional script arguments of its own.
* `stdin:` All subsequent tokens on the line will be concatenated
(separated by a single space) and fed to PhantomJS's standard input,
with a trailing newline. If this token is used more than once,
that produces several lines of input. If this token is not used at
all, standard input will read as empty.
* `unsupported:` The whole file will be skipped.
This directive should be used to notify that the functionality is not yet
implemented by Puppeteer. Tests could be run ignoring this directive
with the `--run-unsupported` flag.
## Output-Expectations Tests
Normally, `run-tests.py` expects each test to produce parseable output
in the [TAP](http://testanything.org/tap-specification.html) format.
This is too inflexible for testing things like `system.stdout.write`,
so there is also a mode in which you specify exactly what output the
test should produce, with additional annotations. Output-expectations
tests are not *required* to be no-harness tests, but the only reason
to use this mode for harness tests would be to test the harness
itself, and it's not sophisticated enough for that.
Using any of the following annotations makes a test an
output-expectations test:
* `expect-exit:` The next token on the line must be an integer. If it
is nonnegative, the PhantomJS process is expected to exit with that
exit code. If it is negative, the process is expected to be
terminated by the signal whose number is the absolute value of the
token. (For instance, `expect-exit: -15` for a test that is
expected to hit the backstop timeout.)
* `expect-stdout:` All subsequent tokens on the line are concatenated,
with spaces in between, and a newline is appeneded. The PhantomJS
process is expected to emit that text, verbatim, on its standard
output. If used more than once, that produces multiple lines of
expected output.
* `expect-stderr:` Same as `expect-stdout`, but the output is expected
to appear on standard error.
* `expect-exit-fails`, `expect-stdout-fails`, `expect-stderr-fails`:
The corresponding test (of the exit code, stdout, or stderr) is
expected to fail.
If some but not all of these annotations are used in a test, the
omitted ones default to exit code 0 (success) and no output on their
respective streams.
## Test Server Modules
The HTTP and HTTPS servers exposed to the test suite serve the static
files in the `www` subdirectory with URLs corresponding to their paths
relative to that directory. If you need more complicated server
behavior than that, you can write custom Python code that executes
when the server receives a request. Any `.py` file below the `www`
directory will be invoked to provide the response for that path
*without* the `.py` suffix. (For instance, `www/echo.py` provides
responses for `TEST_HTTP_BASE + 'echo'`.) Such files must define a
top-level function named `handle_request`. This function receives a
single argument, which is an instance of a subclass of
[`BaseHTTPServer.BaseHTTPRequestHandler`](https://docs.python.org/2/library/basehttpserver.html#BaseHTTPServer.BaseHTTPRequestHandler).
The request headers and body (if any) may be retrieved from this
object. The function must use the `send_response`, `send_header`, and
`end_headers` methods of this object to generate HTTP response
headers, and then return a *file-like object* (**not** a string)
containing the response body. The function is responsible for
generating appropriate `Content-Type` and `Content-Length` headers;
the server framework does not do this automatically.
Test server modules cannot directly cause a test to fail; the server
does not know which test is responsible for any given request. If
there is something wrong with a request, generate an HTTP error
response; then write your test to fail if it receives an error
response.
Python exceptions thrown by test server modules are treated as
failures *of the testsuite*, but they are all attributed to a virtual
"HTTP server errors" test.