npm to pnpm migration

⚠️ Action Required ⚠️

Update all local scripts you use which are calling npm
- Simply replace npm with pnpm for any scripts related to:
  - gf-app-tng
  - gf-sourcers
  - tech-docs
Install pnpm globally
```
npm i -g pnpm@10.17.1
```

(MacOS users) Prepare puppeteer for the crawlerService

Install chromium via brew
```
brew install chromium --no-quarantine
```

Set & source the environment variables

echo "\nexport PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true" >> ~/.zshrc
echo "\nexport PUPPETEER_EXECUTABLE_PATH=`which chromium`" >> ~/.zshrc
source ~/.zshrc

Pull main for:
- gf-app-tng
- gf-sourcers
- tech-docs
At the root of each of the projects listed above, run:
```
rm -rf `find . -type d -name node_modules` && pnpm i
```
Report any problems you encounter to Ian

Further Info & Gotchas

Building dependencies incrementally

Say we have gf-commons as a dependency of crawler-service and we only want to build the crawler-service.

This can be achieved using selective filtering:

# Build crawler-service along with any direct or non direct dependencies
pnpm --filter crawler-service... run build

Basic steps involved in migrating a repository

Run pnpm import at the root of the project to migrate package-lock.json
Delete the package-lock.json file and node_modules folder
```
rm -rf package-lock.json node_modules
```

Add a script to enforce usage of pnpm

{
  "devEngines": {
    "packageManager": {
      "name": "pnpm",
      "onFail": "error"
    }
  },
  "engines": {
    "node": ">=20.0",
    "pnpm": ">=10.0.0"
  }
}

Install packages
```
pnpm i
```
Consider (pre/post)install scripts
Consider security
```
minimumReleaseAge: 10080 # 7 days
```
Replace usages of npm in package scripts, shell scripts & CI/CD actions

hoisted dependencies

With npm it's easy to "accidentally" use a package that isn't installed in your package.json, it will allow you to use anything it finds in node_modules.

pnpm is more strict and if a dependency isn't declared, it will cause issues when attempting to use the dependency.

You can use pnpx npm-check to find packages in your code that are not declared in your package.json.

install scripts are blocked by default

pnpm blocks preinstall and postinstall scripts by default for security.

⚠️ Exercise caution - this is a safety feature to prevent malicious scripts from running

If you need to preserve the ability for running postinstall scripts, you can specify those packages using onlyBuiltDependencies.

If you install a package which requires to run a (pre/post)install script, pnpm will guide you through the process using pnpm approve-builds which automatically updates your pnpm-workspace.yaml with the newly approved builds.

npm uses a flat node_modules structure

pnpm doesn't create a flat node_modules folder. This means any packages that don't declare a dependency

Legacy build tools or scripts that require() dependencies not declared in their own package.json. Example: A webpack plugin that assumes react is in the top-level node_modules instead of its own node_modules.
Monorepos with older tooling that doesn’t resolve modules correctly in pnpm’s nested structure.
Badly behaved packages that try to import dependencies they didn’t declare (e.g. a package that imports lodash but doesn’t list it as a dependency — it only works under npm because lodash is accidentally hoisted).

Option 1 | Nuclear: Preserve flat module structure

Using --shamefully-hoist tells pnpm to flatten node_modules like npm does, putting dependencies at the project root. This makes such packages work, but you lose some of pnpm’s isolation benefits.

pnpm install --shamefully-hoist

Option 2 | Middle ground

This hoists as much as possible but still keeps symlinks and pnpm’s store advantages. This is less extreme than --shamefully-hoist but usually solves most flat node_modules expectations.

Add this line to your .npmrc, .pnpmfile.cjs:

node-linker=hoisted
# For pnpm-workspace.yaml
nodeLinker=hoisted

Install all mono-repo deps

In a mono-repo, it's helpful to be able to install ALL deps in one go (e.g. better onboarding experience)

pnpm --recursive --filter '*' install

⚠️ Per pnpm install docs, this appears to now be redundant

Inside a workspace, pnpm install installs all dependencies in all the projects. If you want to disable this behavior, set the recursive-install setting to false.

Installing deps on CI

Example workflow

name: pnpm Example Workflow
on:
  push:

jobs:
  build:
    runs-on: ubuntu-22.04

    strategy:
      matrix:
        node-version: [20]

    steps:
      - uses: actions/checkout@v4

      - name: Install pnpm
        uses: pnpm/action-setup@v4
        with:
          version: 10

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'pnpm'

      - name: Install dependencies
        run: pnpm install

Pinning installs in CI

With npm you can run efficient CI installs ensuring we only install deps pinned to the versions listed in package-lock.json.

The equivalent to npm ci is:

pnpm i --frozen-lockfile

To get just dependencies (not devDependencies)

pnpm i --frozen-lockfile  --prod

To get just devDependencies

pnpm i --frozen-lockfile  --dev

Security

Prevent packages released within the past 24hrs (1440 minutes) from being installed:

minimumReleaseAge: 1440 # 10080 === 7days

Tab Autocompletion

OhMyZsh:

pnpm completion zsh >| ~/completion-for-pnpm.zsh
echo 'source ~/completion-for-pnpm.zsh' >> ~/.zshrc

Bash:

pnpm completion bash > ~/completion-for-pnpm.bash
echo 'source ~/completion-for-pnpm.bash' >> ~/.bashrc

Fish:

pnpm completion fish > ~/.config/fish/completions/pnpm.fish

pnpm runs commands recursively

By default, when you run a pnpm command, it runs recursively in the repo if there is no script in that folder matching the command you are running.

e.g. If you run pnpm run test in the root, and there is no test script in the package.json at the root, the command will run recursively.

pnpm i installs dependencies recursively when run from the root, whereas running at a sub-project level will install only the deps for that project.

You can control this behaviour with:

-r, --recursive: Run in all projects
--filter <package_name>: Run in specific packages

e.g.

# Run tests only in gf-app-backend
pnpm --filter gf-app-backend test

npx pnpm equivalent

We have locked down the repository to enforce pnpm as the package manager to avoid mistakes.

You can still continue to use npx elsewhere on the command line.

We can revisit this decision later should it cause any frustrations.

Fortunately pnpm has it's own version of npx. Use pnpx as a drop in replacement.

⚠️ package.json scripts should always call deps directly. Using npx or pnpx without specifying a package version leads to unpredictable results

e.g.
{
 "scripts": {
   // There is a risk npx will use a later version than that installed
   ❌ "lint": "npx eslint",
   // eslint is already a dependency; use the exact version
   ✅ "lint": "eslint"
 },
 "devDependencies": {
   "eslint": "8.7.1"
 }
}

Deduplicating dependencies

pnpm dedupe: Perform an install removing older dependencies in the lockfile if a newer version can be used.
pnpm dedupe --check: Check if running dedupe would result in changes without installing packages or editing the lockfile. Exits with a non-zero status code if changes are possible.

⚠️ Action Required ⚠️​

Further Info & Gotchas​

Building dependencies incrementally​

Basic steps involved in migrating a repository​

hoisted dependencies​

install scripts are blocked by default​

npm uses a flat node_modules structure​

Option 1 | Nuclear: Preserve flat module structure​

Option 2 | Middle ground​

Install all mono-repo deps​

Installing deps on CI​

Example workflow​

Pinning installs in CI​

Security​

Tab Autocompletion​

pnpm runs commands recursively​

npx pnpm equivalent​

Deduplicating dependencies​