chore: enhance markdown preprocessor (#3050)

Use preprocessor to make sure all the links in out README.md point
to the last-released version of documentation.

Fixes #3038.

README.md

@@ -6,7 +6,7 @@
 
 <img src="https://user-images.githubusercontent.com/10379601/29446482-04f7036a-841f-11e7-9872-91d1fc2ea683.png" height="200" align="right">
 
-###### <!-- gen:last-released-api -->[API](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md)<!-- gen:stop --> | [FAQ](#faq) | [Contributing](https://github.com/GoogleChrome/puppeteer/blob/master/CONTRIBUTING.md)
+###### [API](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md) | [FAQ](#faq) | [Contributing](https://github.com/GoogleChrome/puppeteer/blob/master/CONTRIBUTING.md)
 
 > Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). Puppeteer runs [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) by default, but can be configured to run full (non-headless) Chrome or Chromium.
 
@@ -37,14 +37,14 @@ npm i puppeteer
 # or "yarn add puppeteer"
 ```
 
-Note: When you install Puppeteer, it downloads a recent version of Chromium (~170Mb Mac, ~282Mb Linux, ~280Mb Win) that is guaranteed to work with the API. To skip the download, see [Environment variables](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#environment-variables).
+Note: When you install Puppeteer, it downloads a recent version of Chromium (~170Mb Mac, ~282Mb Linux, ~280Mb Win) that is guaranteed to work with the API. To skip the download, see [Environment variables](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#environment-variables).
 
 ### Usage
 
 Note: Puppeteer requires at least Node v6.4.0, but the examples below use async/await which is only supported in Node v7.6.0 or greater.
 
 Puppeteer will be familiar to people using other browser testing frameworks. You create an instance
-of `Browser`, open pages, and then manipulate them with [Puppeteer's API](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#).
+of `Browser`, open pages, and then manipulate them with [Puppeteer's API](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#).
 
 **Example** - navigating to https://example.com and saving a screenshot as *example.png*:
 
@@ -69,7 +69,7 @@ Execute script on the command line
 node example.js
 ```
 
-Puppeteer sets an initial page size to 800px x 600px, which defines the screenshot size. The page size can be customized  with [`Page.setViewport()`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetviewportviewport).
+Puppeteer sets an initial page size to 800px x 600px, which defines the screenshot size. The page size can be customized  with [`Page.setViewport()`](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#pagesetviewportviewport).
 
 **Example** - create a PDF.
 
@@ -94,7 +94,7 @@ Execute script on the command line
 node hn.js
 ```
 
-See [`Page.pdf()`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagepdfoptions) for more information about creating pdfs.
+See [`Page.pdf()`](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#pagepdfoptions) for more information about creating pdfs.
 
 **Example** - evaluate script in the context of the page
 
@@ -129,7 +129,7 @@ Execute script on the command line
 node get-dimensions.js
 ```
 
-See [`Page.evaluate()`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageevaluatepagefunction-args) for more information on `evaluate` and related methods like `evaluateOnNewDocument` and `exposeFunction`.
+See [`Page.evaluate()`](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#pageevaluatepagefunction-args) for more information on `evaluate` and related methods like `evaluateOnNewDocument` and `exposeFunction`.
 
 <!-- [END getstarted] -->
 
@@ -138,7 +138,7 @@ See [`Page.evaluate()`](https://github.com/GoogleChrome/puppeteer/blob/master/do
 
 **1. Uses Headless mode**
 
-Puppeteer launches Chromium in [headless mode](https://developers.google.com/web/updates/2017/04/headless-chrome). To launch a full version of Chromium, set the ['headless' option](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions) when launching a browser:
+Puppeteer launches Chromium in [headless mode](https://developers.google.com/web/updates/2017/04/headless-chrome). To launch a full version of Chromium, set the ['headless' option](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#puppeteerlaunchoptions) when launching a browser:
 
 ```js
 const browser = await puppeteer.launch({headless: false}); // default is true
@@ -154,7 +154,7 @@ pass in the executable's path when creating a `Browser` instance:
 const browser = await puppeteer.launch({executablePath: '/path/to/Chrome'});
 ```
 
-See [`Puppeteer.launch()`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions) for more information.
+See [`Puppeteer.launch()`](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#puppeteerlaunchoptions) for more information.
 
 See [`this article`](https://www.howtogeek.com/202825/what%E2%80%99s-the-difference-between-chromium-and-chrome/) for a description of the differences between Chromium and Chrome. [`This article`](https://chromium.googlesource.com/chromium/src/+/lkcr/docs/chromium_browser_vs_google_chrome.md) describes some differences for Linux users.
 
@@ -316,7 +316,7 @@ await page.evaluate(() => {
 
 You may find that Puppeteer does not behave as expected when controlling pages that incorporate audio and video. (For example, [video playback/screenshots is likely to fail](https://github.com/GoogleChrome/puppeteer/issues/291).) There are two reasons for this:
 
-* Puppeteer is bundled with Chromium--not Chrome--and so by default, it inherits all of [Chromium's media-related limitations](https://www.chromium.org/audio-video). This means that Puppeteer does not support licensed formats such as AAC or H.264. (However, it is possible to force Puppeteer to use a separately-installed version Chrome instead of Chromium via the [`executablePath` option to `puppeteer.launch`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions). You should only use this configuration if you need an official release of Chrome that supports these media formats.)
+* Puppeteer is bundled with Chromium--not Chrome--and so by default, it inherits all of [Chromium's media-related limitations](https://www.chromium.org/audio-video). This means that Puppeteer does not support licensed formats such as AAC or H.264. (However, it is possible to force Puppeteer to use a separately-installed version Chrome instead of Chromium via the [`executablePath` option to `puppeteer.launch`](https://github.com/GoogleChrome/puppeteer/blob/v1.6.2/docs/api.md#puppeteerlaunchoptions). You should only use this configuration if you need an official release of Chrome that supports these media formats.)
 * Since Puppeteer (in all configurations) controls a desktop version of Chromium/Chrome, features that are only supported by the mobile version of Chrome are not supported. This means that Puppeteer [does not support HTTP Live Streaming (HLS)](https://caniuse.com/#feat=http-live-streaming).
 
 #### Q: I am having trouble installing / running Puppeteer in my test environment?

utils/doclint/cli.js

@@ -37,13 +37,13 @@ async function run() {
 
   // Documentation checks.
   {
-    const mdSources = await Promise.all([
+    const readme = await Source.readFile(path.join(PROJECT_DIR, 'README.md'));
-      Source.readFile(path.join(PROJECT_DIR, 'docs', 'api.md')),
+    const api = await Source.readFile(path.join(PROJECT_DIR, 'docs', 'api.md'));
-      Source.readFile(path.join(PROJECT_DIR, 'README.md'))
+    const mdSources = [readme, api];
-    ]);
 
     const preprocessor = require('./preprocessor');
-    messages.push(...await preprocessor(mdSources, VERSION));
+    messages.push(...await preprocessor.runCommands(mdSources, VERSION));
+    messages.push(...await preprocessor.ensureReleasedAPILinks([readme], VERSION));
 
     const browser = await puppeteer.launch({args: ['--no-sandbox']});
     const page = await browser.newPage();

utils/doclint/preprocessor/index.js

@@ -16,10 +16,24 @@
 
 const Message = require('../Message');
 
-module.exports = function(sources, version) {
+module.exports.ensureReleasedAPILinks = function(sources, version) {
+  // Release version is everything that doesn't include "-".
+  const apiLinkRegex = /https:\/\/github.com\/GoogleChrome\/puppeteer\/blob\/[^/]*\/docs\/api.md/ig;
+  const lastReleasedAPI = `https://github.com/GoogleChrome/puppeteer/blob/v${version.split('-')[0]}/docs/api.md`;
+
+  const messages = [];
+  for (const source of sources) {
+    const text = source.text();
+    const newText = text.replace(apiLinkRegex, lastReleasedAPI);
+    if (source.setText(newText))
+      messages.push(Message.warning(`GEN: updated ${source.projectPath()}`));
+  }
+  return messages;
+};
+
+module.exports.runCommands = function(sources, version) {
   // Release version is everything that doesn't include "-".
   const isReleaseVersion = !version.includes('-');
-  const lastReleasedAPILink = `[API](https://github.com/GoogleChrome/puppeteer/blob/v${version.split('-')[0]}/docs/api.md)`;
 
   const messages = [];
   const commands = [];
@@ -54,8 +68,6 @@ module.exports = function(sources, version) {
       newText = isReleaseVersion ? 'v' + version : 'Tip-Of-Tree';
     else if (command.name === 'empty-if-release')
       newText = isReleaseVersion ? '' : command.originalText;
-    else if (command.name === 'last-released-api')
-      newText = lastReleasedAPILink;
     else if (command.name === 'toc')
       newText = generateTableOfContents(command.source.text().substring(command.to));
     if (newText === null)

utils/doclint/preprocessor/test.js

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-const preprocessor = require('.');
+const {runCommands, ensureReleasedAPILinks} = require('.');
 const Source = require('../Source');
 const {TestRunner, Reporter, Matchers}  = require('../../testrunner/');
 const runner = new TestRunner();
@@ -25,12 +25,39 @@ const {it, fit, xit} = runner;
 const {beforeAll, beforeEach, afterAll, afterEach} = runner;
 const {expect} = new Matchers();
 
-describe('preprocessor', function() {
+describe('ensureReleasedAPILinks', function() {
+  it('should work with non-release version', function() {
+    const source = new Source('doc.md', `
+      [API](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page)
+    `);
+    const messages = ensureReleasedAPILinks([source], '1.3.0-post');
+    expect(messages.length).toBe(1);
+    expect(messages[0].type).toBe('warning');
+    expect(messages[0].text).toContain('doc.md');
+    expect(source.text()).toBe(`
+      [API](https://github.com/GoogleChrome/puppeteer/blob/v1.3.0/docs/api.md#class-page)
+    `);
+  });
+  it('should work with release version', function() {
+    const source = new Source('doc.md', `
+      [API](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page)
+    `);
+    const messages = ensureReleasedAPILinks([source], '1.3.0');
+    expect(messages.length).toBe(1);
+    expect(messages[0].type).toBe('warning');
+    expect(messages[0].text).toContain('doc.md');
+    expect(source.text()).toBe(`
+      [API](https://github.com/GoogleChrome/puppeteer/blob/v1.3.0/docs/api.md#class-page)
+    `);
+  });
+});
+
+describe('runCommands', function() {
   it('should throw for unknown command', function() {
     const source = new Source('doc.md', `
       <!-- gen:unknown-command -->something<!-- gen:stop -->
     `);
-    const messages = preprocessor([source], '1.1.1');
+    const messages = runCommands([source], '1.1.1');
     expect(source.hasUpdatedText()).toBe(false);
     expect(messages.length).toBe(1);
     expect(messages[0].type).toBe('error');
@@ -41,7 +68,7 @@ describe('preprocessor', function() {
       const source = new Source('doc.md', `
         Puppeteer <!-- gen:version -->XXX<!-- gen:stop -->
       `);
-      const messages = preprocessor([source], '1.2.0');
+      const messages = runCommands([source], '1.2.0');
       expect(messages.length).toBe(1);
       expect(messages[0].type).toBe('warning');
       expect(messages[0].text).toContain('doc.md');
@@ -53,7 +80,7 @@ describe('preprocessor', function() {
       const source = new Source('doc.md', `
         Puppeteer <!-- gen:version -->XXX<!-- gen:stop -->
       `);
-      const messages = preprocessor([source], '1.2.0-post');
+      const messages = runCommands([source], '1.2.0-post');
       expect(messages.length).toBe(1);
       expect(messages[0].type).toBe('warning');
       expect(messages[0].text).toContain('doc.md');
@@ -64,12 +91,12 @@ describe('preprocessor', function() {
     it('should tolerate different writing', function() {
       const source = new Source('doc.md', `Puppeteer v<!--   gEn:version -->WHAT
 <!--     GEN:stop   -->`);
-      preprocessor([source], '1.1.1');
+      runCommands([source], '1.1.1');
       expect(source.text()).toBe(`Puppeteer v<!--   gEn:version -->v1.1.1<!--     GEN:stop   -->`);
     });
     it('should not tolerate missing gen:stop', function() {
       const source = new Source('doc.md', `<!--GEN:version-->`);
-      const messages = preprocessor([source], '1.2.0');
+      const messages = runCommands([source], '1.2.0');
       expect(source.hasUpdatedText()).toBe(false);
       expect(messages.length).toBe(1);
       expect(messages[0].type).toBe('error');
@@ -81,7 +108,7 @@ describe('preprocessor', function() {
       const source = new Source('doc.md', `
         <!-- gen:empty-if-release -->XXX<!-- gen:stop -->
       `);
-      const messages = preprocessor([source], '1.1.1');
+      const messages = runCommands([source], '1.1.1');
       expect(messages.length).toBe(1);
       expect(messages[0].type).toBe('warning');
       expect(messages[0].text).toContain('doc.md');
@@ -93,46 +120,20 @@ describe('preprocessor', function() {
       const source = new Source('doc.md', `
         <!-- gen:empty-if-release -->XXX<!-- gen:stop -->
       `);
-      const messages = preprocessor([source], '1.1.1-post');
+      const messages = runCommands([source], '1.1.1-post');
       expect(messages.length).toBe(0);
       expect(source.text()).toBe(`
         <!-- gen:empty-if-release -->XXX<!-- gen:stop -->
       `);
     });
   });
-  describe('gen:empty-if-release', function() {
-    it('should work with non-release version', function() {
-      const source = new Source('doc.md', `
-        <!-- gen:last-released-api -->XXX<!-- gen:stop -->
-      `);
-      const messages = preprocessor([source], '1.3.0-post');
-      expect(messages.length).toBe(1);
-      expect(messages[0].type).toBe('warning');
-      expect(messages[0].text).toContain('doc.md');
-      expect(source.text()).toBe(`
-        <!-- gen:last-released-api -->[API](https://github.com/GoogleChrome/puppeteer/blob/v1.3.0/docs/api.md)<!-- gen:stop -->
-      `);
-    });
-    it('should work with release version', function() {
-      const source = new Source('doc.md', `
-        <!-- gen:last-released-api -->XXX<!-- gen:stop -->
-      `);
-      const messages = preprocessor([source], '1.3.0');
-      expect(messages.length).toBe(1);
-      expect(messages[0].type).toBe('warning');
-      expect(messages[0].text).toContain('doc.md');
-      expect(source.text()).toBe(`
-        <!-- gen:last-released-api -->[API](https://github.com/GoogleChrome/puppeteer/blob/v1.3.0/docs/api.md)<!-- gen:stop -->
-      `);
-    });
-  });
   describe('gen:toc', function() {
     it('should work', () => {
       const source = new Source('doc.md', `<!-- gen:toc -->XXX<!-- gen:stop -->
         ### class: page
         #### page.$
         #### page.$$`);
-      const messages = preprocessor([source], '1.3.0');
+      const messages = runCommands([source], '1.3.0');
       expect(messages.length).toBe(1);
       expect(messages[0].type).toBe('warning');
       expect(messages[0].text).toContain('doc.md');
@@ -152,7 +153,7 @@ describe('preprocessor', function() {
       <!-- gen:empty-if-release -->YYY<!-- gen:stop -->
       <!-- gen:version -->ZZZ<!-- gen:stop -->
     `);
-    const messages = preprocessor([source], '1.1.1');
+    const messages = runCommands([source], '1.1.1');
     expect(messages.length).toBe(1);
     expect(messages[0].type).toBe('warning');
     expect(messages[0].text).toContain('doc.md');