# Faithful E2E Testing of Nx Preset Generators

TLDR: Here's a full working example of a faithful E2E test for an Nx preset generator since the default generated E2E harness isn't correct.

Putting it all together, here's a full sample test suite:
```typescript
import {
  checkFilesExist,
  cleanup,
  runNxCommandAsync,
  tmpProjPath,
} from "@nrwl/nx-plugin/testing";
import { ChildProcess, execSync, fork } from "node:child_process";
import path from "node:path";
import { getPortPromise as getOpenPort } from "portfinder";

// These tests can take awhile to run. Modify or remove this depending on how long this takes
// on your machine or in your environment.
jest.setTimeout(60_000);

const startVerdaccio = async (port: number): Promise<ChildProcess> => {
  const configPath = path.join(__dirname, "../verdaccio.yaml");
  return new Promise((resolve, reject) => {
    const child = fork(require.resolve("verdaccio/bin/verdaccio"), [
      "-c",
      configPath,
      "-l",
      `${port}`,
    ]);

    child.on("message", (message: { verdaccio_started: boolean }) => {
      if (message.verdaccio_started) {
        resolve(child);
      }
    });
    child.on("error", (error: any) => reject([error]));
    child.on("disconnect", (error: any) => reject([error]));
  });
};

describe("nx-plugin e2e", () => {
  let verdaccioProcess: ChildProcess;

  beforeAll(async () => {
    cleanup();

    const verdaccioPort = await getOpenPort();
    verdaccioProcess = await startVerdaccio(verdaccioPort);

    const verdaccioUrl = `http://localhost:${verdaccioPort}`;

    execSync(`yarn config set registry ${verdaccioUrl}`);

    execSync(
      `npx npm-cli-login -u chiubaka -p test -e test@chiubaka.com -r ${verdaccioUrl}`,
    );
    execSync(`npm publish --registry=${verdaccioUrl}`, {
      cwd: path.join(__dirname, "../../../dist/packages/nx-plugin"),
    });

    const destination = path.join(tmpProjPath(), "..");
    const workspaceName = path.basename(tmpProjPath());

    execSync(
      `npm_config_registry=${verdaccioUrl} npx create-nx-workspace ${workspaceName} --preset=@chiubaka/nx-plugin --nxCloud=false`,
      {
        cwd: destination,
      },
    );
  });

  afterAll(async () => {
    // `nx reset` kills the daemon, and performs
    // some work which can help clean up e2e leftovers
    await runNxCommandAsync("reset");

    execSync(`yarn config set registry https://registry.yarnpkg.com`);

    verdaccioProcess.kill();
  });

  it("should not create an apps dir", () => {
    expect(() => {
      checkFilesExist("apps");
    }).toThrow();
  });
});
```

Be sure that you have `_debug: true` somewhere in your `verdaccio.yml` and that you have `verdaccio`, `verdaccio-auth-memory`, and `verdaccio-memory` installed as `devDependencies`.

## The Problem
Nx models custom presets as ordinary generators that get treated differently by Nx's internals when generating a workspace. The suggested command for scaffolding a preset generator is the exact same command used for scaffolding generators for other use cases with only the caveat that a preset generator _must be named `preset`_.

Unfortunately, the preset generator use case is sufficiently different from other generator use cases (e.g. generating a project within a pre-existing workspace) that this mental model and a lot of the generated scaffolding turns out to be misleading.

Specifically, I've found that neither the unit nor the E2E testing harnesses for the preset generator are terribly faithful. By "faithful" I mean: does this testing set up accurately reflect the real context my code will run in?

In the case of the default E2E testing harness, the answer is a definite no. Out of the box, this harness is designed to test a generator by running it inside of a pre-existing workspace as if it were a library or app generator, which it's not.

This harness:
1. Generates a new workspace with the `empty` preset.
2. Patches the `package.json` file in the newly generated workspace to install my plugin from the local filesystem.
3. Invokes `nx generate @my-org/my-plugin:preset project` to run the preset generator as a normal generator.

For awhile this _seemed_ fine, but as my preset generator grew more complex, I started noticing places where real workspace generation would fail, but my E2E tests were passing.  It didn't take long to realize that this was because my generator gets run differently in production than it does in these E2E tests.

Tests that don't reliably tell me when my code is broken aren't doing their job 🙃!

## Motivation
Lately I've been writing a [custom Nx workspace preset](https://nx.dev/packages/nx-plugin#preset). The goal of a custom preset is to allow customization of the workspace creation process.

I'm hoping to use this to create a batteries-included standardized monorepo generator complete with my preferred configuration for things like linting, testing, CI, and even GitHub project settings.

Unfortunately, this use case isn't very well-documented within Nx. In fact, in a lot of cases the documentation and provided scaffolding for preset generators is, in my humble opinion, seriously misleading. I've had to figure things out by reading through Nx's open source codebase and doing a lot of experimentation.

This article is my attempt at saving someone else all that time and pain :).

## The Solution

It was a little tricky getting a full E2E test for workspace generation itself, but I managed to piece a solution together.

Here's the outline:
1. Start up a [Verdaccio](https://verdaccio.org/) server before running tests.
  1. Verdaccio is a lightweight registry that's easy to install and use locally.
2. Authenticate with the Verdaccio registry to allow publishing.
3. Publish the built Nx plugin locally to the Verdaccio registry.
4. Run `create-nx-workspace` with my preset, making sure to hit the local Verdaccio registry to grab the plugin.

### Why Verdaccio?
The challenge in E2E testing a preset generator is that preset generators are usually invoked through `create-nx-workspace` and passed as a package name in the `preset` argument.

Behind the scenes, `create-nx-workspace` resolves the package name from NPM in order to run the preset.

In our tests, we obviously don't want to pull in a real published version of the plugin. We'd like to bundle the current state of the plugin and run the E2E tests against that.

Since `create-nx-workspace` is a CLI, often run through `npx`, we can't use other common local package linking methods like `npm link` or modifying a `package.json` file (because there isn't one yet!). Instead, the strategy is to actually publish the package, but to a local registry that won't affect anything outside of our tests.

Verdaccio fills this role perfectly. By default, it acts as a proxy for NPM, pulling any packages that aren't found in the local registry from the remote registry. This means we can publish our plugin registry and expect Verdaccio to return the development version of our own packages, while still correctly pulling in other dependencies from elsewhere.

### Starting an ephemeral Verdaccio server in tests
One of the trickier parts of getting this E2E harness to work was figuring out how to reliably run an ephemeral Verdaccio server as part of my test set up. As it turns out, the Verdaccio documentation is a bit rough around the edges, as well. There are two relevant pages, one about [End to End Testing](https://verdaccio.org/docs/e2e), which doesn't provide a lot of context and another about the [Node.js API](https://verdaccio.org/docs/verdaccio-programmatically), which is ostensibly not about E2E testing at all.

I was most drawn to the idea of running Verdaccio programmatically using the [module API](https://verdaccio.org/docs/verdaccio-programmatically#using-the-module-api), but had trouble getting this to work. In most cases it seemed that the server would not start up properly and my tests would just hang.

Ultimately the approached that work was [running Verdaccio as a child process using `fork`](https://verdaccio.org/docs/verdaccio-programmatically#using-fork-from-child_process-module). I took cues from the example code in the Verdaccio documentation as well as from this [sample repo](https://github.com/juanpicado/verdaccio-fork) that contains a complete example of this set up.

For my test setup and tear down I ended with something like this:
```typescript
import { ChildProcess, fork } from "node:child_process";
import { getPortPromise as getOpenPort } from "portfinder";

const startVerdaccio = async (port: number): Promise<ChildProcess> => {
  const configPath = path.join(__dirname, "../verdaccio.yaml");
  return new Promise((resolve, reject) => {
    const child = fork(require.resolve("verdaccio/bin/verdaccio"), [
      "-c",
      configPath,
      "-l",
      `${port}`,
    ]);

    child.on("message", (message: { verdaccio_started: boolean }) => {
      if (message.verdaccio_started) {
        resolve(child);
      }
    });
    child.on("error", (error: any) => reject([error]));
    child.on("disconnect", (error: any) => reject([error]));
  });
};
describe("nx-plugin e2e", () => {
  let verdaccioProcess: ChildProcess;

  beforeAll(async () => {
      const verdaccioPort = await getOpenPort();
      verdaccioProcess = await startVerdaccio(verdaccioPort);
  });

  afterAll(async () => {
    verdaccioProcess.kill();
  });
});
```

I created a `verdaccio.yaml` file that looks like this:
```yaml
# verdaccio-memory
store:
  memory:
    limit: 1000
# verdaccio-auth-memory plugin
auth:
  # htpasswd:
  #   file: ./htpasswd
  auth-memory:
    users:
      foo:
        name: foo
        password: bar
      admin:  
        name: foo
        password: bar
# uplinks
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
  verdacciobk:
    url: http://localhost:8000/
    auth:
      type: bearer
      token: dsyTcamuhMd8GlsakOhP5A==
packages:
  "@*/*":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
  "react":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: verdacciobk
  "**":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

# rate limit configuration
rateLimit:
  windowMs: 1000
  max: 10000

middlewares:
  audit:
    enabled: true

security: 
  api: 
    jwt: 
     sign: 
      expiresIn: 1d    

logs: { type: file, path: /dev/null, level: info }
i18n:
  web: en-US

# try to use verdaccio with child_process:fork
_debug: true
```

Notably, I took this file almost completely from the [`verdaccio-fork` example repo](https://github.com/juanpicado/verdaccio-fork/blob/main/verdaccio.yaml). The only small change I made was to modify `logs` to send all `verdaccio` output to `/dev/null` so it wouldn't clutter my testing output.

Per the Verdaccio docs, `_debug: true` is very important when using Verdaccio in this way, as it's what turns on the ability to listen for the `verdaccio_started` message once the server is ready to go.

### Authenticating with Verdaccio
Next challenge was authenticating with the new Verdaccio server from inside of tests. Initially, I thought I could just run a simple `npm adduser --registry=http://my-local-registry` command. It took a few confusing test failures before I realized that `npm adduser` is an interactive CLI and was failing my tests because it was expecting user input.

The way around this is to use `npm-cli-login` instead. You can either add it as a `devDependency` and invoke it with `npm run npm-cli-login` or just use `npx npm-cli-login`.

Here's the full command to authenticate:
```typescript
import { execSync } from "node:child_process";
import { getPortPromise as getOpenPort } from "portfinder";

const verdaccioPort = await getOpenPort();
const verdaccioUrl = `http://localhost:${verdaccioPort}`;
execSync(
  `npx npm-cli-login -u chiubaka -p test -e test@chiubaka.com -r ${verdaccioUrl}`,
);
```

This needs to go in the `beforeAll` setup block of your tests.

### Publishing to Verdaccio
Now that Verdaccio is running and we're authenticated, publishing is easy!

```typescript
import { execSync } from "node:child_process";
import { getPortPromise as getOpenPort } from "portfinder";

const verdaccioPort = await getOpenPort();
const verdaccioUrl = `http://localhost:${verdaccioPort}`;
execSync(`npm publish --registry=${verdaccioUrl}`, {
  cwd: path.join(__dirname, "../../../dist/packages/nx-plugin"),
});
```

Where the `cwd` of the `execSync` here needs to be the path to the built version of your plugin, which the `@nrwl/nx-plugin:e2e` executor will ensure is pre-built before running your tests by default.

### Running the workspace generation command
With our plugin package published to the local registry, all that's left is to run the workspace generation command to create a real generated workspace in the E2E testing directory (`tmp` within the Nx plugin workspace by default).

Since we're aiming to be as faithful as possible to the true experience users will have when using our plugin, we'll use the `create-nx-workspace` command to invoke the preset generator.

In order to get the `create-nx-workspace` command to use the local Verdaccio registry, we'll need to run it with `npx` and prefix with the `npm_config_registry=[http://my-local-registry]` environment variable.

Additionally, in order for a lot of the `@nrwl/nx-plugin/testing` utils to work properly, note that your testing workspace needs to be generated in a very specific place. At time of writing, the name of that directory is `proj`, but since that could change without warning it's safest to dynamically determine the name of the testing workspace using `tmpProjPath()`.

Here's what the full command looks like:
```typescript
import { tmpProjPath } from "@nrwl/nx-plugin/testing";
import { execSync } from "node:child_process";
import path from "node:path";
import { getPortPromise as getOpenPort } from "portfinder";

const verdaccioPort = await getOpenPort();
const verdaccioUrl = `http://localhost:${verdaccioPort}`;

const destination = path.join(tmpProjPath(), "..");
const workspaceName = path.basename(tmpProjPath());

execSync(
  `npm_config_registry=${verdaccioUrl} npx create-nx-workspace ${workspaceName} --preset=@chiubaka/nx-plugin --nxCloud=false`,
  {
    cwd: destination,
  },
);
```

### Full working solution
Putting it all together, here's a full sample test suite:
```typescript
import {
  checkFilesExist,
  cleanup,
  runNxCommandAsync,
  tmpProjPath,
} from "@nrwl/nx-plugin/testing";
import { ChildProcess, execSync, fork } from "node:child_process";
import path from "node:path";
import { getPortPromise as getOpenPort } from "portfinder";

// These tests can take awhile to run. Modify or remove this depending on how long this takes
// on your machine or in your environment.
jest.setTimeout(60_000);

const startVerdaccio = async (port: number): Promise<ChildProcess> => {
  const configPath = path.join(__dirname, "../verdaccio.yaml");
  return new Promise((resolve, reject) => {
    const child = fork(require.resolve("verdaccio/bin/verdaccio"), [
      "-c",
      configPath,
      "-l",
      `${port}`,
    ]);

    child.on("message", (message: { verdaccio_started: boolean }) => {
      if (message.verdaccio_started) {
        resolve(child);
      }
    });
    child.on("error", (error: any) => reject([error]));
    child.on("disconnect", (error: any) => reject([error]));
  });
};

describe("nx-plugin e2e", () => {
  let verdaccioProcess: ChildProcess;

  beforeAll(async () => {
    cleanup();

    const verdaccioPort = await getOpenPort();
    verdaccioProcess = await startVerdaccio(verdaccioPort);

    const verdaccioUrl = `http://localhost:${verdaccioPort}`;

    execSync(`yarn config set registry ${verdaccioUrl}`);

    execSync(
      `npx npm-cli-login -u chiubaka -p test -e test@chiubaka.com -r ${verdaccioUrl}`,
    );
    execSync(`npm publish --registry=${verdaccioUrl}`, {
      cwd: path.join(__dirname, "../../../dist/packages/nx-plugin"),
    });

    const destination = path.join(tmpProjPath(), "..");
    const workspaceName = path.basename(tmpProjPath());

    execSync(
      `npm_config_registry=${verdaccioUrl} npx create-nx-workspace ${workspaceName} --preset=@chiubaka/nx-plugin --nxCloud=false`,
      {
        cwd: destination,
      },
    );
  });

  afterAll(async () => {
    // `nx reset` kills the daemon, and performs
    // some work which can help clean up e2e leftovers
    await runNxCommandAsync("reset");

    execSync(`yarn config set registry https://registry.yarnpkg.com`);

    verdaccioProcess.kill();
  });

  it("should not create an apps dir", () => {
    expect(() => {
      checkFilesExist("apps");
    }).toThrow();
  });
});
```

Be sure to include a `verdaccio.yaml` file that looks something like this:
```yaml
# verdaccio-memory
store:
  memory:
    limit: 1000
# verdaccio-auth-memory plugin
auth:
  # htpasswd:
  #   file: ./htpasswd
  auth-memory:
    users:
      foo:
        name: foo
        password: bar
      admin:  
        name: foo
        password: bar
# uplinks
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
  verdacciobk:
    url: http://localhost:8000/
    auth:
      type: bearer
      token: dsyTcamuhMd8GlsakOhP5A==
packages:
  "@*/*":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
  "react":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: verdacciobk
  "**":
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

# rate limit configuration
rateLimit:
  windowMs: 1000
  max: 10000

middlewares:
  audit:
    enabled: true

security: 
  api: 
    jwt: 
     sign: 
      expiresIn: 1d    

logs: { type: file, path: /dev/null, level: info }
i18n:
  web: en-US

# try to use verdaccio with child_process:fork
_debug: true
```

And of course, make sure you've installed `verdaccio`, `verdaccio-auth-memory`, and `verdaccio-memory` to support this config file.