Faithful E2E Testing of Nx Preset Generators

Faithful E2E Testing of Nx Preset Generators

TLDR: Here's a full working example of a faithful E2E test for an Nx preset generator since the default generated E2E harness isn't correct.

Putting it all together, here's a full sample test suite:

import {
  checkFilesExist,
  cleanup,
  runNxCommandAsync,
  tmpProjPath,
} from "@nrwl/nx-plugin/testing";
import { ChildProcess, execSync, fork } from "node:child_process";
import path from "node:path";
import { getPortPromise as getOpenPort } from "portfinder";

// These tests can take awhile to run. Modify or remove this depending on how long this takes
// on your machine or in your environment.
jest.setTimeout(60_000);

const startVerdaccio = async (port: number): Promise<ChildProcess> => {
  const configPath = path.join(__dirname, "../verdaccio.yaml");
  return new Promise((resolve, reject) => {
    const child = fork(require.resolve("verdaccio/bin/verdaccio"), [
      "-c",
      configPath,
      "-l",
      `${port}`,
    ]);

    child.on("message", (message: { verdaccio_started: boolean }) => {
      if (message.verdaccio_started) {
        resolve(child);
      }
    });
    child.on("error", (error: any) => reject([error]));
    child.on("disconnect", (error: any) => reject([error]));
  });
};

describe("nx-plugin e2e", () => {
  let verdaccioProcess: ChildProcess;

  beforeAll(async () => {
    cleanup();

    const verdaccioPort = await getOpenPort();
    verdaccioProcess = await startVerdaccio(verdaccioPort);

    const verdaccioUrl = `http://localhost:${verdaccioPort}`;

    execSync(`yarn config set registry ${verdaccioUrl}`);

    execSync(
      `npx npm-cli-login -u chiubaka -p test -e test@chiubaka.com -r ${verdaccioUrl}`,
    );
    execSync(`npm publish --registry=${verdaccioUrl}`, {
      cwd: path.join(__dirname, "../../../dist/packages/nx-plugin"),
    });

    const destination = path.join(tmpProjPath(), "..");
    const workspaceName = path.basename(tmpProjPath());

    execSync(
      `npm_config_registry=${verdaccioUrl} npx create-nx-workspace ${workspaceName} --preset=@chiubaka/nx-plugin --nxCloud=false`,
      {
        cwd: destination,
      },
    );
  });

  afterAll(async () => {
    // `nx reset` kills the daemon, and performs
    // some work which can help clean up e2e leftovers
    await runNxCommandAsync("reset");

    execSync(`yarn config set registry https://registry.yarnpkg.com`);

    verdaccioProcess.kill();
  });

  it("should not create an apps dir", () => {
    expect(() => {
      checkFilesExist("apps");
    }).toThrow();
  });
});

Be sure that you have _debug: true somewhere in your verdaccio.yml and that you have verdaccio, verdaccio-auth-memory, and verdaccio-memory installed as devDependencies.

The Problem

Nx models custom presets as ordinary generators that get treated differently by Nx's internals when generating a workspace. The suggested command for scaffolding a preset generator is the exact same command used for scaffolding generators for other use cases with only the caveat that a preset generator must be named preset.

Unfortunately, the preset generator use case is sufficiently different from other generator use cases (e.g. generating a project within a pre-existing workspace) that this mental model and a lot of the generated scaffolding turns out to be misleading.

Specifically, I've found that neither the unit nor the E2E testing harnesses for the preset generator are terribly faithful. By "faithful" I mean: does this testing set up accurately reflect the real context my code will run in?

In the case of the default E2E testing harness, the answer is a definite no. Out of the box, this harness is designed to test a generator by running it inside of a pre-existing workspace as if it were a library or app generator, which it's not.

This harness:

Generates a new workspace with the empty preset.
Patches the package.json file in the newly generated workspace to install my plugin from the local filesystem.
Invokes nx generate @my-org/my-plugin:preset project to run the preset generator as a normal generator.

For awhile this seemed fine, but as my preset generator grew more complex, I started noticing places where real workspace generation would fail, but my E2E tests were passing. It didn't take long to realize that this was because my generator gets run differently in production than it does in these E2E tests.

Tests that don't reliably tell me when my code is broken aren't doing their job 🙃!

Motivation

Lately I've been writing a custom Nx workspace preset. The goal of a custom preset is to allow customization of the workspace creation process.

I'm hoping to use this to create a batteries-included standardized monorepo generator complete with my preferred configuration for things like linting, testing, CI, and even GitHub project settings.

Unfortunately, this use case isn't very well-documented within Nx. In fact, in a lot of cases the documentation and provided scaffolding for preset generators is, in my humble opinion, seriously misleading. I've had to figure things out by reading through Nx's open source codebase and doing a lot of experimentation.

This article is my attempt at saving someone else all that time and pain :).

The Solution

It was a little tricky getting a full E2E test for workspace generation itself, but I managed to piece a solution together.

Here's the outline:

Start up a Verdaccio server before running tests.
1. Verdaccio is a lightweight registry that's easy to install and use locally.
Authenticate with the Verdaccio registry to allow publishing.
Publish the built Nx plugin locally to the Verdaccio registry.
Run create-nx-workspace with my preset, making sure to hit the local Verdaccio registry to grab the plugin.

Why Verdaccio?

The challenge in E2E testing a preset generator is that preset generators are usually invoked through create-nx-workspace and passed as a package name in the preset argument.

Behind the scenes, create-nx-workspace resolves the package name from NPM in order to run the preset.

In our tests, we obviously don't want to pull in a real published version of the plugin. We'd like to bundle the current state of the plugin and run the E2E tests against that.

Since create-nx-workspace is a CLI, often run through npx, we can't use other common local package linking methods like npm link or modifying a package.json file (because there isn't one yet!). Instead, the strategy is to actually publish the package, but to a local registry that won't affect anything outside of our tests.

Verdaccio fills this role perfectly. By default, it acts as a proxy for NPM, pulling any packages that aren't found in the local registry from the remote registry. This means we can publish our plugin registry and expect Verdaccio to return the development version of our own packages, while still correctly pulling in other dependencies from elsewhere.

Starting an ephemeral Verdaccio server in tests

One of the trickier parts of getting this E2E harness to work was figuring out how to reliably run an ephemeral Verdaccio server as part of my test set up. As it turns out, the Verdaccio documentation is a bit rough around the edges, as well. There are two relevant pages, one about End to End Testing, which doesn't provide a lot of context and another about the Node.js API, which is ostensibly not about E2E testing at all.

I was most drawn to the idea of running Verdaccio programmatically using the module API, but had trouble getting this to work. In most cases it seemed that the server would not start up properly and my tests would just hang.

Ultimately the approached that work was running Verdaccio as a child process using fork. I took cues from the example code in the Verdaccio documentation as well as from this sample repo that contains a complete example of this set up.

For my test setup and tear down I ended with something like this:

import { ChildProcess, fork } from "node:child_process";
import { getPortPromise as getOpenPort } from "portfinder";

const startVerdaccio = async (port: number): Promise<ChildProcess> => {
  const configPath = path.join(__dirname, "../verdaccio.yaml");
  return new Promise((resolve, reject) => {
    const child = fork(require.resolve("verdaccio/bin/verdaccio"), [
      "-c",
      configPath,
      "-l",
      `${port}`,
    ]);

    child.on("message", (message: { verdaccio_started: boolean }) => {
      if (message.verdaccio_started) {
        resolve(child);
      }
    });
    child.on("error", (error: any) => reject([error]));
    child.on("disconnect", (error: any) => reject([error]));
  });
};
describe("nx-plugin e2e", () => {
  let verdaccioProcess: ChildProcess;

  beforeAll(async () => {
      const verdaccioPort = await getOpenPort();
      verdaccioProcess = await startVerdaccio(verdaccioPort);
  });

  afterAll(async () => {
    verdaccioProcess.kill();
  });
});

I created a verdaccio.yaml file that looks like this:

# verdaccio-memory
store:
  memory:
    limit: 1000
# verdaccio-auth-memory plugin
auth:
  # htpasswd:
  #   file: ./htpasswd
  auth-memory:
    users:
      foo:
        name: foo
        password: bar
      admin:  
        name: foo
        password: bar
# uplinks
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
  verdacciobk:
    url: http://localhost:8000/
    auth:
      type: bearer
      token: dsyTcamuhMd8GlsakOhP5A==
packages:
  "@*/*":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
  "react":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: verdacciobk
  "**":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

# rate limit configuration
rateLimit:
  windowMs: 1000
  max: 10000

middlewares:
  audit:
    enabled: true

security: 
  api: 
    jwt: 
     sign: 
      expiresIn: 1d    

logs: { type: file, path: /dev/null, level: info }
i18n:
  web: en-US

# try to use verdaccio with child_process:fork
_debug: true

Notably, I took this file almost completely from the verdaccio-fork example repo. The only small change I made was to modify logs to send all verdaccio output to /dev/null so it wouldn't clutter my testing output.

Per the Verdaccio docs, _debug: true is very important when using Verdaccio in this way, as it's what turns on the ability to listen for the verdaccio_started message once the server is ready to go.

Authenticating with Verdaccio

Next challenge was authenticating with the new Verdaccio server from inside of tests. Initially, I thought I could just run a simple npm adduser --registry=http://my-local-registry command. It took a few confusing test failures before I realized that npm adduser is an interactive CLI and was failing my tests because it was expecting user input.

The way around this is to use npm-cli-login instead. You can either add it as a devDependency and invoke it with npm run npm-cli-login or just use npx npm-cli-login.

Here's the full command to authenticate:

import { execSync } from "node:child_process";
import { getPortPromise as getOpenPort } from "portfinder";

const verdaccioPort = await getOpenPort();
const verdaccioUrl = `http://localhost:${verdaccioPort}`;
execSync(
  `npx npm-cli-login -u chiubaka -p test -e test@chiubaka.com -r ${verdaccioUrl}`,
);

This needs to go in the beforeAll setup block of your tests.

Publishing to Verdaccio

Now that Verdaccio is running and we're authenticated, publishing is easy!

import { execSync } from "node:child_process";
import { getPortPromise as getOpenPort } from "portfinder";

const verdaccioPort = await getOpenPort();
const verdaccioUrl = `http://localhost:${verdaccioPort}`;
execSync(`npm publish --registry=${verdaccioUrl}`, {
  cwd: path.join(__dirname, "../../../dist/packages/nx-plugin"),
});

Where the cwd of the execSync here needs to be the path to the built version of your plugin, which the @nrwl/nx-plugin:e2e executor will ensure is pre-built before running your tests by default.

Running the workspace generation command

With our plugin package published to the local registry, all that's left is to run the workspace generation command to create a real generated workspace in the E2E testing directory (tmp within the Nx plugin workspace by default).

Since we're aiming to be as faithful as possible to the true experience users will have when using our plugin, we'll use the create-nx-workspace command to invoke the preset generator.

In order to get the create-nx-workspace command to use the local Verdaccio registry, we'll need to run it with npx and prefix with the npm_config_registry=[http://my-local-registry] environment variable.

Additionally, in order for a lot of the @nrwl/nx-plugin/testing utils to work properly, note that your testing workspace needs to be generated in a very specific place. At time of writing, the name of that directory is proj, but since that could change without warning it's safest to dynamically determine the name of the testing workspace using tmpProjPath().

Here's what the full command looks like:

import { tmpProjPath } from "@nrwl/nx-plugin/testing";
import { execSync } from "node:child_process";
import path from "node:path";
import { getPortPromise as getOpenPort } from "portfinder";

const verdaccioPort = await getOpenPort();
const verdaccioUrl = `http://localhost:${verdaccioPort}`;

const destination = path.join(tmpProjPath(), "..");
const workspaceName = path.basename(tmpProjPath());

execSync(
  `npm_config_registry=${verdaccioUrl} npx create-nx-workspace ${workspaceName} --preset=@chiubaka/nx-plugin --nxCloud=false`,
  {
    cwd: destination,
  },
);

Full working solution

Putting it all together, here's a full sample test suite:

import {
  checkFilesExist,
  cleanup,
  runNxCommandAsync,
  tmpProjPath,
} from "@nrwl/nx-plugin/testing";
import { ChildProcess, execSync, fork } from "node:child_process";
import path from "node:path";
import { getPortPromise as getOpenPort } from "portfinder";

// These tests can take awhile to run. Modify or remove this depending on how long this takes
// on your machine or in your environment.
jest.setTimeout(60_000);

const startVerdaccio = async (port: number): Promise<ChildProcess> => {
  const configPath = path.join(__dirname, "../verdaccio.yaml");
  return new Promise((resolve, reject) => {
    const child = fork(require.resolve("verdaccio/bin/verdaccio"), [
      "-c",
      configPath,
      "-l",
      `${port}`,
    ]);

    child.on("message", (message: { verdaccio_started: boolean }) => {
      if (message.verdaccio_started) {
        resolve(child);
      }
    });
    child.on("error", (error: any) => reject([error]));
    child.on("disconnect", (error: any) => reject([error]));
  });
};

describe("nx-plugin e2e", () => {
  let verdaccioProcess: ChildProcess;

  beforeAll(async () => {
    cleanup();

    const verdaccioPort = await getOpenPort();
    verdaccioProcess = await startVerdaccio(verdaccioPort);

    const verdaccioUrl = `http://localhost:${verdaccioPort}`;

    execSync(`yarn config set registry ${verdaccioUrl}`);

    execSync(
      `npx npm-cli-login -u chiubaka -p test -e test@chiubaka.com -r ${verdaccioUrl}`,
    );
    execSync(`npm publish --registry=${verdaccioUrl}`, {
      cwd: path.join(__dirname, "../../../dist/packages/nx-plugin"),
    });

    const destination = path.join(tmpProjPath(), "..");
    const workspaceName = path.basename(tmpProjPath());

    execSync(
      `npm_config_registry=${verdaccioUrl} npx create-nx-workspace ${workspaceName} --preset=@chiubaka/nx-plugin --nxCloud=false`,
      {
        cwd: destination,
      },
    );
  });

  afterAll(async () => {
    // `nx reset` kills the daemon, and performs
    // some work which can help clean up e2e leftovers
    await runNxCommandAsync("reset");

    execSync(`yarn config set registry https://registry.yarnpkg.com`);

    verdaccioProcess.kill();
  });

  it("should not create an apps dir", () => {
    expect(() => {
      checkFilesExist("apps");
    }).toThrow();
  });
});

Be sure to include a verdaccio.yaml file that looks something like this:

# verdaccio-memory
store:
  memory:
    limit: 1000
# verdaccio-auth-memory plugin
auth:
  # htpasswd:
  #   file: ./htpasswd
  auth-memory:
    users:
      foo:
        name: foo
        password: bar
      admin:  
        name: foo
        password: bar
# uplinks
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
  verdacciobk:
    url: http://localhost:8000/
    auth:
      type: bearer
      token: dsyTcamuhMd8GlsakOhP5A==
packages:
  "@*/*":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
  "react":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: verdacciobk
  "**":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

# rate limit configuration
rateLimit:
  windowMs: 1000
  max: 10000

middlewares:
  audit:
    enabled: true

security: 
  api: 
    jwt: 
     sign: 
      expiresIn: 1d    

logs: { type: file, path: /dev/null, level: info }
i18n:
  web: en-US

# try to use verdaccio with child_process:fork
_debug: true

And of course, make sure you've installed verdaccio, verdaccio-auth-memory, and verdaccio-memory to support this config file.