Worker Threads & Child Processes

Node.js is single-threaded, but that does not mean you are limited to one CPU core. Worker Threads and Child Processes let you leverage multi-core systems for CPU-intensive operations. To understand why this matters, think of the Node.js event loop as a single chef in a kitchen. That chef can handle hundreds of orders by efficiently switching between tasks while things cook in the oven (I/O operations). But if one order requires the chef to hand-knead dough for 10 minutes straight (CPU-intensive work), every other order just waits. Worker threads are like hiring additional chefs for the heavy prep work, so the main chef stays free to handle incoming orders.

The Problem: Blocking the Event Loop

// This blocks ALL requests while computing -- the event loop cannot process
// ANY incoming connections, timer callbacks, or I/O completions until this
// for-loop finishes. With 10 billion iterations, that could be 30+ seconds.
app.get('/compute', (req, res) => {
  let result = 0;
  for (let i = 0; i < 1e10; i++) {
    result += Math.sqrt(i);
  }
  res.json({ result });
});

// Every other route handler is frozen during that computation.
// Even a trivial health check takes 30+ seconds to respond.
app.get('/health', (req, res) => {
  res.json({ status: 'ok' }); // Blocked until /compute finishes!
});

CPU-intensive operations block the event loop, making your server unresponsive to ALL requests. This is the #1 performance killer in Node.js applications.

Solutions Overview

Method	Use Case	Communication	Overhead
Worker Threads	CPU-intensive JS	Shared memory, message passing	Low
Child Process	External programs, shell commands	stdio, IPC	Medium
Cluster	Multi-instance servers	None (separate processes)	High

Worker Threads

Worker Threads run JavaScript in parallel threads within the same process, sharing memory when needed. Unlike child processes, they do not spawn a separate OS process, so they have lower startup overhead and can share data directly via SharedArrayBuffer.

Basic Usage

// main.js -- this single file serves both roles.
// isMainThread tells you which role this execution is playing.
const { Worker, isMainThread, parentPort, workerData } = require('worker_threads');

if (isMainThread) {
  // Main thread -- creates a worker and listens for results.
  // __filename means "run this same file again, but as a worker thread"
  const worker = new Worker(__filename, {
    workerData: { value: 1000000 } // Data passed to the worker at creation
  });

  // Workers communicate via message passing (like postMessage in browsers)
  worker.on('message', (result) => {
    console.log('Result:', result);
  });

  // Always handle errors -- an unhandled error in a worker does NOT
  // crash the main thread, but silently dies if you do not listen for it
  worker.on('error', (error) => {
    console.error('Worker error:', error);
  });

  worker.on('exit', (code) => {
    if (code !== 0) {
      console.error(`Worker stopped with code ${code}`);
    }
  });
} else {
  // Worker thread -- runs in a separate V8 isolate (its own heap and event loop)
  const { value } = workerData;
  let result = 0;
  
  for (let i = 0; i < value; i++) {
    result += Math.sqrt(i);
  }
  
  // Send the result back to the main thread
  parentPort.postMessage(result);
}

Separate Worker File (Recommended)

// workers/compute.js
const { parentPort, workerData } = require('worker_threads');

const heavyComputation = (n) => {
  let result = 0;
  for (let i = 0; i < n; i++) {
    result += Math.sqrt(i);
  }
  return result;
};

const result = heavyComputation(workerData.iterations);
parentPort.postMessage(result);

// main.js
const { Worker } = require('worker_threads');
const path = require('path');

const runWorker = (iterations) => {
  return new Promise((resolve, reject) => {
    const worker = new Worker(
      path.join(__dirname, 'workers/compute.js'),
      { workerData: { iterations } }
    );

    worker.on('message', resolve);
    worker.on('error', reject);
    worker.on('exit', (code) => {
      if (code !== 0) {
        reject(new Error(`Worker stopped with code ${code}`));
      }
    });
  });
};

// Usage in Express
app.get('/compute', async (req, res) => {
  try {
    const result = await runWorker(1e9);
    res.json({ result });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Worker Pool

Creating a new worker for every request is expensive — each worker spins up a new V8 isolate, which takes milliseconds and memory. A worker pool pre-creates a fixed number of workers at startup and reuses them for incoming tasks. It is the same principle as database connection pooling: amortize the creation cost across many operations. When all workers are busy, new tasks queue up and wait for the next available worker. This also naturally applies backpressure — if tasks arrive faster than workers can process them, the queue grows, which you can monitor and use to trigger alerts or scaling.

// workerPool.js
const { Worker } = require('worker_threads');
const path = require('path');

class WorkerPool {
  constructor(workerPath, poolSize = 4) {
    this.workerPath = workerPath;
    this.poolSize = poolSize;
    this.workers = [];
    this.freeWorkers = [];
    this.taskQueue = [];
    
    this.init();
  }

  init() {
    for (let i = 0; i < this.poolSize; i++) {
      this.addWorker();
    }
  }

  addWorker() {
    const worker = new Worker(this.workerPath);
    
    worker.on('message', (result) => {
      const { resolve } = worker.currentTask;
      worker.currentTask = null;
      this.freeWorkers.push(worker);
      resolve(result);
      this.processQueue();
    });

    worker.on('error', (error) => {
      if (worker.currentTask) {
        worker.currentTask.reject(error);
      }
      // Replace dead worker
      this.workers = this.workers.filter(w => w !== worker);
      this.freeWorkers = this.freeWorkers.filter(w => w !== worker);
      this.addWorker();
    });

    this.workers.push(worker);
    this.freeWorkers.push(worker);
  }

  runTask(data) {
    return new Promise((resolve, reject) => {
      this.taskQueue.push({ data, resolve, reject });
      this.processQueue();
    });
  }

  processQueue() {
    if (this.taskQueue.length === 0) return;
    if (this.freeWorkers.length === 0) return;

    const worker = this.freeWorkers.pop();
    const task = this.taskQueue.shift();
    
    worker.currentTask = task;
    worker.postMessage(task.data);
  }

  destroy() {
    for (const worker of this.workers) {
      worker.terminate();
    }
  }
}

module.exports = WorkerPool;

// Usage
const WorkerPool = require('./workerPool');
const pool = new WorkerPool('./workers/compute.js', 4);

app.get('/compute', async (req, res) => {
  const result = await pool.runTask({ iterations: 1e9 });
  res.json({ result });
});

// Cleanup on shutdown
process.on('SIGTERM', () => pool.destroy());

SharedArrayBuffer for Shared Memory

Normal message passing between workers involves copying data — the main thread serializes the message, sends a copy, and the worker deserializes it. For large data or high-frequency communication, this copying overhead is significant. SharedArrayBuffer provides true shared memory: multiple threads read and write the same underlying bytes, with no copying. However, this introduces the classic problems of concurrent programming — race conditions and data corruption — so you must use Atomics operations (atomic read-modify-write) to safely access shared data.

// main.js
const { Worker } = require('worker_threads');

// Create shared memory -- 4 bytes is enough for one 32-bit integer
const sharedBuffer = new SharedArrayBuffer(4);
const sharedArray = new Int32Array(sharedBuffer);
sharedArray[0] = 0; // Initialize counter

// Start multiple workers that increment the counter
const workers = [];
for (let i = 0; i < 4; i++) {
  const worker = new Worker('./workers/counter.js', {
    workerData: { sharedBuffer }
  });
  workers.push(worker);
}

// Wait for all workers
Promise.all(workers.map(w => 
  new Promise(resolve => w.on('exit', resolve))
)).then(() => {
  console.log('Final count:', sharedArray[0]);
});

// workers/counter.js
const { workerData } = require('worker_threads');

const sharedArray = new Int32Array(workerData.sharedBuffer);

for (let i = 0; i < 1000000; i++) {
  Atomics.add(sharedArray, 0, 1); // Thread-safe increment
}

Child Processes

While Worker Threads run JavaScript in parallel threads within a single process, Child Processes spawn entirely separate OS processes. They are heavier (each gets its own memory space and V8 instance), but they provide complete isolation and the ability to run any program — not just JavaScript.

exec - Run Shell Commands

const { exec } = require('child_process');
const util = require('util');

const execPromise = util.promisify(exec);

// Simple command
exec('ls -la', (error, stdout, stderr) => {
  if (error) {
    console.error('Error:', error.message);
    return;
  }
  console.log('Output:', stdout);
});

// Promise version
const runCommand = async (command) => {
  try {
    const { stdout, stderr } = await execPromise(command);
    return stdout;
  } catch (error) {
    throw new Error(error.stderr || error.message);
  }
};

// Usage
app.get('/git-log', async (req, res) => {
  const log = await runCommand('git log --oneline -10');
  res.json({ log: log.split('\n') });
});

spawn - Stream Output

The key difference between exec and spawn: exec buffers the entire output in memory and returns it all at once (limited to 200KB by default), while spawn streams output in real time. Use exec for short commands with small output; use spawn for long-running processes, large output, or when you need to process output incrementally.

const { spawn } = require('child_process');

// spawn streams output in real time -- ideal for long-running processes
const runProcess = (command, args) => {
  return new Promise((resolve, reject) => {
    const process = spawn(command, args);
    
    let stdout = '';
    let stderr = '';

    process.stdout.on('data', (data) => {
      stdout += data;
      console.log('stdout:', data.toString());
    });

    process.stderr.on('data', (data) => {
      stderr += data;
      console.error('stderr:', data.toString());
    });

    process.on('close', (code) => {
      if (code === 0) {
        resolve(stdout);
      } else {
        reject(new Error(stderr || `Process exited with code ${code}`));
      }
    });

    process.on('error', reject);
  });
};

// Install npm packages
app.post('/npm-install', async (req, res) => {
  try {
    await runProcess('npm', ['install', req.body.package]);
    res.json({ success: true });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

fork - Run Node.js Scripts

fork is a specialized version of spawn designed specifically for running Node.js scripts. It automatically sets up an IPC (Inter-Process Communication) channel between parent and child, so they can exchange structured JavaScript objects via process.send() and process.on('message') — no manual serialization needed.

// main.js
const { fork } = require('child_process');
const path = require('path');

const runNodeScript = (scriptPath, data) => {
  return new Promise((resolve, reject) => {
    const child = fork(scriptPath);
    
    child.send(data);
    
    child.on('message', (result) => {
      resolve(result);
      child.kill();
    });
    
    child.on('error', reject);
    
    child.on('exit', (code) => {
      if (code !== 0) {
        reject(new Error(`Process exited with code ${code}`));
      }
    });
  });
};

// scripts/process-data.js
process.on('message', async (data) => {
  // Heavy processing
  const result = await processData(data);
  process.send(result);
});

// Usage
app.post('/process', async (req, res) => {
  const result = await runNodeScript(
    path.join(__dirname, 'scripts/process-data.js'),
    req.body
  );
  res.json(result);
});

Real-World Use Cases

Image Processing with Workers

Image resizing and format conversion are classic CPU-bound tasks — the sharp library does pixel-level transformations that saturate a CPU core. Without workers, processing a single large image could block your server for 500ms+. With a worker pool, the main thread stays responsive while workers process images in parallel.

// workers/imageProcessor.js
const { parentPort, workerData } = require('worker_threads');
const sharp = require('sharp');

const processImage = async ({ inputPath, outputPath, width, height }) => {
  await sharp(inputPath)
    .resize(width, height)
    .webp({ quality: 80 })
    .toFile(outputPath);
  
  return { success: true, outputPath };
};

parentPort.on('message', async (task) => {
  try {
    const result = await processImage(task);
    parentPort.postMessage({ success: true, ...result });
  } catch (error) {
    parentPort.postMessage({ success: false, error: error.message });
  }
});

// imageService.js
const WorkerPool = require('./workerPool');
const pool = new WorkerPool('./workers/imageProcessor.js', 4);

const processImages = async (images) => {
  const tasks = images.map(img => 
    pool.runTask({
      inputPath: img.path,
      outputPath: img.outputPath,
      width: 800,
      height: 600
    })
  );
  
  return Promise.all(tasks);
};

PDF Generation

// workers/pdfGenerator.js
const { parentPort } = require('worker_threads');
const PDFDocument = require('pdfkit');
const fs = require('fs');

parentPort.on('message', async ({ data, outputPath }) => {
  try {
    const doc = new PDFDocument();
    const stream = fs.createWriteStream(outputPath);
    
    doc.pipe(stream);
    
    // Generate PDF content
    doc.fontSize(25).text(data.title, 100, 100);
    doc.fontSize(12).text(data.content, 100, 150);
    
    doc.end();
    
    stream.on('finish', () => {
      parentPort.postMessage({ success: true, path: outputPath });
    });
  } catch (error) {
    parentPort.postMessage({ success: false, error: error.message });
  }
});

Video Transcoding with FFmpeg

Video transcoding is a perfect use case for child processes: FFmpeg is an external C program (not JavaScript), it produces streaming progress output on stderr, and it can run for minutes on large files. spawn is the right choice here because it streams output incrementally — exec would buffer the entire stderr log in memory.

const { spawn } = require('child_process');

const transcodeVideo = (input, output, options = {}) => {
  return new Promise((resolve, reject) => {
    const args = [
      '-i', input,
      '-c:v', options.codec || 'libx264',
      '-preset', options.preset || 'medium',
      '-crf', options.quality || '23',
      '-c:a', 'aac',
      '-y', // Overwrite output
      output
    ];

    const ffmpeg = spawn('ffmpeg', args);
    
    let progress = '';
    
    ffmpeg.stderr.on('data', (data) => {
      progress = data.toString();
      // Parse progress for UI updates
    });

    ffmpeg.on('close', (code) => {
      if (code === 0) {
        resolve({ success: true, output });
      } else {
        reject(new Error(`FFmpeg exited with code ${code}`));
      }
    });

    ffmpeg.on('error', reject);
  });
};

Best Practices

Use Worker Pools - Don’t create new workers per request
Set appropriate pool size - Usually number of CPU cores
Handle errors properly - Workers can crash
Clean up on shutdown - Terminate workers gracefully
Don’t overuse workers - Only for CPU-intensive tasks
Consider message serialization - Large data transfers have overhead
Use SharedArrayBuffer - For shared state between threads

When to Use What

Scenario	Solution
CPU-intensive calculations	Worker Threads
Image/video processing	Worker Pool + spawn for external tools
Running shell commands	exec or spawn
Running Node.js scripts	fork
Scaling HTTP servers	Cluster or PM2
Background job processing	Worker Threads or separate process

Summary

Worker Threads run JavaScript in parallel without blocking
Worker Pools reuse threads for better performance
SharedArrayBuffer enables shared memory between threads
Child Processes run external programs or Node.js scripts
Use exec for simple commands, spawn for streaming output
Use fork for Node.js scripts with IPC
Always handle errors and cleanup properly
Match pool size to CPU cores for optimal performance

20. TypeScript with Node.js 22. Capstone Project: Full-Stack API

Overview

Testing & Code Quality

Crash Courses

AI Engineering

Math for ML - Understanding Linear Algebra

Probability & Statistics for ML

Math for ML - Understanding Calculus

ML Mastery

Deep Learning Mastery

NestJS Mastery

Microservices Mastery

Low Level Design

OOP Concepts

SOLID Principles

Design Patterns

LLD Case Studies

System Design (HLD)

Senior Level (L5+/Staff)

HLD Case Studies

Engineering Fundamentals

DevOps & Operations

Azure Cloud Engineering

AWS Cloud

AWS Monitoring & Observability

AWS Security Services

AWS Serverless

AWS Operations

AWS Advanced

AWS Case Studies

GCP Cloud Engineering

DevOps Tools

Database Engineering

HIPAA Compliance Mastery

Operating Systems

Linux Internals

Distributed Systems

Networking Mastery

Build Your Own X

Go Lang Mastery

C Programming

Classic Research Papers

Distributed System Tools

Documentation Index

​Worker Threads & Child Processes

​The Problem: Blocking the Event Loop

​Solutions Overview

​Worker Threads

​Basic Usage

​Separate Worker File (Recommended)

​Worker Pool

​SharedArrayBuffer for Shared Memory

​Child Processes

​exec - Run Shell Commands

​spawn - Stream Output

​fork - Run Node.js Scripts

​Real-World Use Cases

​Image Processing with Workers

​PDF Generation

​Video Transcoding with FFmpeg

​Best Practices

​When to Use What

​Summary

Worker Threads & Child Processes

The Problem: Blocking the Event Loop

Solutions Overview

Worker Threads

Basic Usage

Separate Worker File (Recommended)

Worker Pool

SharedArrayBuffer for Shared Memory

Child Processes

exec - Run Shell Commands

spawn - Stream Output

fork - Run Node.js Scripts

Real-World Use Cases

Image Processing with Workers

PDF Generation

Video Transcoding with FFmpeg

Best Practices

When to Use What

Summary