Headless Chrome: an answer to server-side rendering JavaScript sites

Addy Osmani

Learn how you can use Puppeteer APIs to add server-side rendering (SSR) capabilities to an Express web server. The best part is that your app requires very small changes to code. Headless does all the heavy lifting.

In a couple of lines of code you can SSR any page and get its final markup.

import puppeteer from 'puppeteer';

async function ssr(url) {
  const browser = await puppeteer.launch({headless: true});
  const page = await browser.newPage();
  await page.goto(url, {waitUntil: 'networkidle0'});
  const html = await page.content(); // serialized HTML of page DOM.
  await browser.close();
  return html;
}

Note: This document uses ES modules (import), which requires Node 8.5.0+ running with the --experimental-modules flag. You can use require() statements if you prefer. Read more about Node's ES Modules support.

Why use Headless Chrome?

You may be interested in Headless Chrome if:

You've built a web app that isn't being indexed by search engines.
You're hoping for a quick win to optimize JavaScript performance and improve first meaningful paint.

Some frameworks like Preact ship with tools that address server-side rendering. If your framework has a prerendering solution, stick with it instead of bringing Puppeteer and Headless Chrome into your workflow.

Crawling the modern web

Search engine crawlers, social sharing platforms, even browsers have historically relied exclusively on static HTML markup to index the web and surface content. The modern web has evolved into something much different. JavaScript-based applications are here to stay, which means that in many cases, our content can be invisible to crawling tools.

The Googlebot, our Search crawler, processes JavaScript while making sure it doesn't degrade the experience of users visiting the site. There are some differences and limitations that you need to account for when designing your pages and applications to accommodate how crawlers access and render your content.

Prerender pages

All crawlers understand HTML. To ensure crawlers can index JavaScript, we need a tool that:

Knows how to run all types of modern JavaScript and generate static HTML.
Stays up to date as the web adds features.
Runs with little to no code updates to your application.

Sounds good right? That tool is the browser! Headless Chrome doesn't care what library, framework, or tool chain you use.

For example, if your application is built with Node.js, Puppeteer is an easy way to work with headless Chrome.

Start with a dynamic page that generates its HTML with JavaScript:

public/index.html

<html>
<body>
  <div id="container">
    <!-- Populated by the JS below. -->
  </div>
</body>
<script>
function renderPosts(posts, container) {
  const html = posts.reduce((html, post) => {
    return `${html}
      <li class="post">
        <h2>${post.title}</h2>
        <div class="summary">${post.summary}</div>
        <p>${post.content}</p>
      </li>`;
  }, '');

  // CAREFUL: this assumes HTML is sanitized.
  container.innerHTML = `<ul id="posts">${html}</ul>`;
}

(async() => {
  const container = document.querySelector('#container');
  const posts = await fetch('/posts').then(resp => resp.json());
  renderPosts(posts, container);
})();
</script>
</html>

SSR function

Next, take the ssr() function from earlier and beef it up a bit:

ssr.mjs

import puppeteer from 'puppeteer';

// In-memory cache of rendered pages. Note: this will be cleared whenever the
// server process stops. If you need true persistence, use something like
// Google Cloud Storage (https://firebase.google.com/docs/storage/web/start).
const RENDER_CACHE = new Map();

async function ssr(url) {
  if (RENDER_CACHE.has(url)) {
    return {html: RENDER_CACHE.get(url), ttRenderMs: 0};
  }

  const start = Date.now();

  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  try {
    // networkidle0 waits for the network to be idle (no requests for 500ms).
    // The page's JS has likely produced markup by this point, but wait longer
    // if your site lazy loads, etc.
    await page.goto(url, {waitUntil: 'networkidle0'});
    await page.waitForSelector('#posts'); // ensure #posts exists in the DOM.
  } catch (err) {
    console.error(err);
    throw new Error('page.goto/waitForSelector timed out.');
  }

  const html = await page.content(); // serialized HTML of page DOM.
  await browser.close();

  const ttRenderMs = Date.now() - start;
  console.info(`Headless rendered page in: ${ttRenderMs}ms`);

  RENDER_CACHE.set(url, html); // cache rendered page.

  return {html, ttRenderMs};
}

export {ssr as default};

The major changes:

Added caching. Caching the rendered HTML is the biggest win to speed up the response times. When the page gets re-requested, you avoid running headless Chrome altogether. I discuss other optimizations later on.
Add basic error handling if loading the page times out.
Add a call to page.waitForSelector('#posts'). This ensures that the posts exist in the DOM before we dump the serialized page.
Add science. Log how long headless takes to render the page and return the rendering time along with the HTML.
Stick the code in a module named ssr.mjs.

Example web server

Finally, here's the small express server that brings it all together. The main handler prerenders the URL http://localhost/index.html (the homepage) and serves the result as its response. Users immediately see posts when they hit the page because the static markup is now part of the response.

server.mjs

import express from 'express';
import ssr from './ssr.mjs';

const app = express();

app.get('/', async (req, res, next) => {
  const {html, ttRenderMs} = await ssr(`${req.protocol}://${req.get('host')}/index.html`);
  // Add Server-Timing! See https://w3c.github.io/server-timing/.
  res.set('Server-Timing', `Prerender;dur=${ttRenderMs};desc="Headless render time (ms)"`);
  return res.status(200).send(html); // Serve prerendered page as response.
});

app.listen(8080, () => console.log('Server started. Press Ctrl+C to quit'));

To run this example, install the dependencies (npm i --save puppeteer express) and run the server using Node 8.5.0+ and the --experimental-modules flag:

Here's what an example of the response sent back by this server:

<html>
<body>
  <div id="container">
    <ul id="posts">
      <li class="post">
        <h2>Title 1</h2>
        <div class="summary">Summary 1</div>
        <p>post content 1</p>
      </li>
      <li class="post">
        <h2>Title 2</h2>
        <div class="summary">Summary 2</div>
        <p>post content 2</p>
      </li>
      ...
    </ul>
  </div>
</body>
<script>
...
</script>
</html>

A perfect use case for the new Server-Timing API

The Server-Timing API communicates server performance metrics (such as request and response times or database lookups) back to the browser. Client code can use this information to track overall performance of a web app.

A perfect use case for Server-Timing is to report how long it takes for headless Chrome to prerender a page. To do that, just add the Server-Timing header to the server response:

res.set('Server-Timing', `Prerender;dur=1000;desc="Headless render time (ms)"`);

On the client, the Performance API and PerformanceObserver can be used to access these metrics:

const entry = performance.getEntriesByType('navigation').find(
    e => e.name === location.href);
console.log(entry.serverTiming[0].toJSON());

{
  "name": "Prerender",
  "duration": 3808,
  "description": "Headless render time (ms)"
}

Performance results

The following results incorporate most of the performance optimizations discussed later.

On an example app, headless Chrome takes about one second to render the page on the server. Once the page is cached, DevTools 3G Slow emulation puts FCP at 8.37s faster than the client-side version.

	First Paint (FP)	First Contentful Paint (FCP)
Client-side app	4s	11s
SSR version	2.3s	~2.3s

These results are promising. Users see meaningful content much quicker because the server-side rendered page no longer relies on JavaScript to load + shows posts.

Prevent re-hydration

Remember when I said "we didn't make any code changes to the client-side app"? That was a lie.

Our Express app takes a request, uses Puppeteer to load the page into headless, and serves the result as a response. But this setup has a problem.

The same JavaScript that executes in headless Chrome on the server runs again when the user's browser loads the page on the frontend. We have two places generating markup. #doublerender!

To fix this, tell the page its HTML is already in place. One solution is to have the page JavaScript check if <ul id="posts"> is already in the DOM at load time. If it is, you know the page was SSR'd and can avoid re-adding posts again. 👍

public/index.html

<html>
<body>
  <div id="container">
    <!-- Populated by JS (below) or by prerendering (server). Either way,
         #container gets populated with the posts markup:
      <ul id="posts">...</ul>
    -->
  </div>
</body>
<script>
...
(async() => {
  const container = document.querySelector('#container');

  // Posts markup is already in DOM if we're seeing a SSR'd.
  // Don't re-hydrate the posts here on the client.
  const PRE_RENDERED = container.querySelector('#posts');
  if (!PRE_RENDERED) {
    const posts = await fetch('/posts').then(resp => resp.json());
    renderPosts(posts, container);
  }
})();
</script>
</html>

Optimizations

Apart from caching the rendered results, there are plenty of interesting optimizations we can make to ssr(). Some are quick wins while others may be more speculative. The performance benefits you see may ultimately depend on the types of pages you prerender and the complexity of the app.

Abort non-essential requests

Right now, the entire page (and all of the resources it requests) is loaded unconditionally into headless Chrome. However, we're only interested in two things:

The rendered markup.
The JS requests that produced that markup.

Network requests that don't construct DOM are wasteful. Resources like images, fonts, stylesheets, and media don't participate in building the HTML of a page. They style and supplement the structure of a page but they don't explicitly create it. We should tell the browser to ignore these resources. This reduces the workload for headless Chrome, saves bandwidth, and potentially speeds up prerendering time for larger pages.

The DevTools Protocol supports a powerful feature called Network interception which can be used to modify requests before they're issued by the browser. Puppeteer supports network interception by turning on page.setRequestInterception(true) and listening for the page's request event. That allows us to abort requests for certain resources and let others continue through.

ssr.mjs

async function ssr(url) {
  ...
  const page = await browser.newPage();

  // 1. Intercept network requests.
  await page.setRequestInterception(true);

  page.on('request', req => {
    // 2. Ignore requests for resources that don't produce DOM
    // (images, stylesheets, media).
    const allowlist = ['document', 'script', 'xhr', 'fetch'];
    if (!allowlist.includes(req.resourceType())) {
      return req.abort();
    }

    // 3. Pass through all other requests.
    req.continue();
  });

  await page.goto(url, {waitUntil: 'networkidle0'});
  const html = await page.content(); // serialized HTML of page DOM.
  await browser.close();

  return {html};
}

Inline critical resources

It's common to use separate build tools (such as gulp) to process an app and inline critical CSS and JS into the page at build-time. This can speed up first meaningful paint because the browser makes fewer requests during initial page load.

Instead of a separate build tool, use the browser as your build tool! We can use Puppeteer to manipulate the page's DOM, inlining styles, JavaScript, or whatever else you want to stick in the page before prerendering it.

This example shows how to intercept responses for local stylesheets and inline those resources into the page as <style> tags:

ssr.mjs

import urlModule from 'url';
const URL = urlModule.URL;

async function ssr(url) {
  ...
  const stylesheetContents = {};

  // 1. Stash the responses of local stylesheets.
  page.on('response', async resp => {
    const responseUrl = resp.url();
    const sameOrigin = new URL(responseUrl).origin === new URL(url).origin;
    const isStylesheet = resp.request().resourceType() === 'stylesheet';
    if (sameOrigin && isStylesheet) {
      stylesheetContents[responseUrl] = await resp.text();
    }
  });

  // 2. Load page as normal, waiting for network requests to be idle.
  await page.goto(url, {waitUntil: 'networkidle0'});

  // 3. Inline the CSS.
  // Replace stylesheets in the page with their equivalent <style>.
  await page.$$eval('link[rel="stylesheet"]', (links, content) => {
    links.forEach(link => {
      const cssText = content[link.href];
      if (cssText) {
        const style = document.createElement('style');
        style.textContent = cssText;
        link.replaceWith(style);
      }
    });
  }, stylesheetContents);

  // 4. Get updated serialized HTML of page.
  const html = await page.content();
  await browser.close();

  return {html};
}

This code:

Use a page.on('response') handler to listen for network responses.
Stashes the responses of local stylesheets.
Finds all <link rel="stylesheet"> in the DOM and replaces them with an equivalent <style>. See page.$$eval API docs. The style.textContent is set to the stylesheet response.

Auto-minify resources

Another trick you can do with network interception is to modify the responses returned by a request.

As an example, say you want to minify the CSS in your app but also want to keep the convenience having it unminified when developing. Assuming you've setup another tool to pre-minify styles.css, one can use Request.respond() to rewrite the response of styles.css to be the content of styles.min.css.

ssr.mjs

import fs from 'fs';

async function ssr(url) {
  ...

  // 1. Intercept network requests.
  await page.setRequestInterception(true);

  page.on('request', req => {
    // 2. If request is for styles.css, respond with the minified version.
    if (req.url().endsWith('styles.css')) {
      return req.respond({
        status: 200,
        contentType: 'text/css',
        body: fs.readFileSync('./public/styles.min.css', 'utf-8')
      });
    }
    ...

    req.continue();
  });
  ...

  const html = await page.content();
  await browser.close();

  return {html};
}

Reuse a single Chrome instance across renders

Launching a new browser for every prerender creates a lot of overhead. Instead, you may want to launch a single instance and reuse it for rendering multiple pages.

Puppeteer can reconnect to an existing instance of Chrome by calling puppeteer.connect() and passing it the instance's remote debugging URL. To keep a long-running browser instance, we can move the code that launches Chrome from the ssr() function and into the Express server:

server.mjs

import express from 'express';
import puppeteer from 'puppeteer';
import ssr from './ssr.mjs';

let browserWSEndpoint = null;
const app = express();

app.get('/', async (req, res, next) => {
  if (!browserWSEndpoint) {
    const browser = await puppeteer.launch();
    browserWSEndpoint = await browser.wsEndpoint();
  }

  const url = `${req.protocol}://${req.get('host')}/index.html`;
  const {html} = await ssr(url, browserWSEndpoint);

  return res.status(200).send(html);
});

ssr.mjs

import puppeteer from 'puppeteer';

/**
 * @param {string} url URL to prerender.
 * @param {string} browserWSEndpoint Optional remote debugging URL. If
 *     provided, Puppeteer's reconnects to the browser instance. Otherwise,
 *     a new browser instance is launched.
 */
async function ssr(url, browserWSEndpoint) {
  ...
  console.info('Connecting to existing Chrome instance.');
  const browser = await puppeteer.connect({browserWSEndpoint});

  const page = await browser.newPage();
  ...
  await page.close(); // Close the page we opened here (not the browser).

  return {html};
}

Example: cron job to periodically prerender

TO render a number of pages all at once, you can use a shared browser instance.

import puppeteer from 'puppeteer';
import * as prerender from './ssr.mjs';
import urlModule from 'url';
const URL = urlModule.URL;

app.get('/cron/update_cache', async (req, res) => {
  if (!req.get('X-Appengine-Cron')) {
    return res.status(403).send('Sorry, cron handler can only be run as admin.');
  }

  const browser = await puppeteer.launch();
  const homepage = new URL(`${req.protocol}://${req.get('host')}`);

  // Re-render main page and a few pages back.
  prerender.clearCache();
  await prerender.ssr(homepage.href, await browser.wsEndpoint());
  await prerender.ssr(`${homepage}?year=2018`);
  await prerender.ssr(`${homepage}?year=2017`);
  await prerender.ssr(`${homepage}?year=2016`);
  await browser.close();

  res.status(200).send('Render cache updated!');
});

Also, add a clearCache() export to ssr.js:

...
function clearCache() {
  RENDER_CACHE.clear();
}

export {ssr, clearCache};

Other considerations

Create a signal for the page: "You're being rendered in headless"

When your page is being rendered by headless Chrome on the server, it may be helpful for the page's client-side logic to know that. In my app, I used this hook to "turn off" portions of my page that don't play a part in rendering the posts markup. For example, I disabled code that lazy-loads firebase-auth.js. There's no user to sign in!

Adding a ?headless parameter to the render URL is a simple way to give the page a hook:

ssr.mjs

import urlModule from 'url';
const URL = urlModule.URL;

async function ssr(url) {
  ...
  // Add ?headless to the URL so the page has a signal
  // it's being loaded by headless Chrome.
  const renderUrl = new URL(url);
  renderUrl.searchParams.set('headless', '');
  await page.goto(renderUrl, {waitUntil: 'networkidle0'});
  ...

  return {html};
}

And in the page, we can look for that parameter:

public/index.html

<html>
<body>
  <div id="container">
    <!-- Populated by the JS below. -->
  </div>
</body>
<script>
...

(async() => {
  const params = new URL(location.href).searchParams;

  const RENDERING_IN_HEADLESS = params.has('headless');
  if (RENDERING_IN_HEADLESS) {
    // Being rendered by headless Chrome on the server.
    // e.g. shut off features, don't lazy load non-essential resources, etc.
  }

  const container = document.querySelector('#container');
  const posts = await fetch('/posts').then(resp => resp.json());
  renderPosts(posts, container);
})();
</script>
</html>

Avoid inflating Analytics pageviews

Be careful if you're using Analytics on your site. Prerendering pages may result in inflated pageviews. Specifically, you'll see 2 times the number of hits, one hit when headless Chrome renders the page and another when the user's browser renders it.

So what's the fix? Use network interception to abort any request(s) that tries to load the Analytics library.

page.on('request', req => {
  // Don't load Google Analytics lib requests so pageviews aren't 2x.
  const blockist = ['www.google-analytics.com', '/gtag/js', 'ga.js', 'analytics.js'];
  if (blocklist.find(regex => req.url().match(regex))) {
    return req.abort();
  }
  ...
  req.continue();
});

Page hits never get recorded if the code never loads. Boom 💥.

Alternatively, continue to load your Analytics libraries to gain insight into how many prerenders your server is performing.

Conclusion

Puppeteer makes it easy to server-side render pages by running headless Chrome, as a companion, on your web server. My favorite "feature" of this approach is that you improve loading performance and the indexability of your app without significant code changes!

If you're curious to see a working app that uses the techniques described here, check out the devwebfeed app.

Appendix

Discussion of prior art

Server-side rendering client-side apps is hard. How hard? Just look at how many npm packages people have written which are dedicated to the topic. There are countless patterns, tools, and services available to help with SSRing JS apps.

Isomorphic / Universal JavaScript

The concept of Universal JavaScript means: the same code that runs on server also runs on the client (the browser). You share code between server and client and everyone feels a moment of zen.

Headless Chrome enables "isomorphic JS" between server and client. It's a great option if your library doesn't work on the server (Node).

Prerender tools

The Node community has built tons of tools for dealing with SSR JS apps. No surprises there! Personally, I've found that YMMV with some of these tools, so definitely do your homework before committing to one. For example, some SSR tools are older and don't use headless Chrome (or any headless browser for that matter). Instead, they use PhantomJS (aka old Safari), which means your pages aren't going to render properly if they're using newer features.

One of the notable exceptions is Prerender. Prerender is interesting in that it uses headless Chrome and comes with drop-in middleware for Express:

const prerender = require('prerender');
const server = prerender();
server.use(prerender.removeScriptTags());
server.use(prerender.blockResources());
server.start();

It's worth noting that Prerender leaves out the details of downloading and installing Chrome on different platforms. Oftentimes, that's fairly tricky to get right, which is one of the reasons why Puppeteer does for you.