CI/CD for Microservices

Microservices unlock the ability to deploy services independently, but this requires sophisticated CI/CD pipelines. This chapter covers building production-grade deployment pipelines.

Learning Objectives:

Design CI/CD pipelines for microservices
Implement deployment strategies (Blue-Green, Canary, Rolling)
Set up GitOps with ArgoCD
Configure automated testing in pipelines
Build multi-service deployment orchestration

CI/CD Challenges in Microservices

┌─────────────────────────────────────────────────────────────────────────────┐
│                    MONOLITH vs MICROSERVICES CI/CD                          │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  MONOLITH:                                                                   │
│  ─────────────────────────────────────────────────────────────────────────  │
│                                                                              │
│  ┌────────────┐    ┌──────────┐    ┌───────────┐    ┌────────────┐         │
│  │   Commit   │───▶│  Build   │───▶│   Test    │───▶│   Deploy   │         │
│  │            │    │  (1 app) │    │  (1 suite)│    │  (1 target)│         │
│  └────────────┘    └──────────┘    └───────────┘    └────────────┘         │
│                                                                              │
│  ✓ Simple pipeline                                                          │
│  ✓ One build, one deploy                                                    │
│  ✗ All or nothing deployment                                               │
│  ✗ Long deployment cycles                                                   │
│                                                                              │
│  ═══════════════════════════════════════════════════════════════════════════│
│                                                                              │
│  MICROSERVICES:                                                              │
│  ─────────────────────────────────────────────────────────────────────────  │
│                                                                              │
│  ┌────────────┐    ┌──────────────────────────────────────────────┐        │
│  │  Commit    │    │              PARALLEL PIPELINES               │        │
│  │  Service A │───▶│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │        │
│  └────────────┘    │  │ Build  │▶│  Test  │▶│ Scan   │▶│ Deploy │ │        │
│                    │  └────────┘ └────────┘ └────────┘ └────────┘ │        │
│  ┌────────────┐    │                                               │        │
│  │  Commit    │───▶│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │        │
│  │  Service B │    │  │ Build  │▶│  Test  │▶│ Scan   │▶│ Deploy │ │        │
│  └────────────┘    │  └────────┘ └────────┘ └────────┘ └────────┘ │        │
│                    │                                               │        │
│  ┌────────────┐    │  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │        │
│  │  Commit    │───▶│  │ Build  │▶│  Test  │▶│ Scan   │▶│ Deploy │ │        │
│  │  Service C │    │  └────────┘ └────────┘ └────────┘ └────────┘ │        │
│  └────────────┘    └──────────────────────────────────────────────┘        │
│                                         │                                   │
│                                         ▼                                   │
│                    ┌──────────────────────────────────────────────┐        │
│                    │            INTEGRATION TESTS                  │        │
│                    │     (Contract tests, E2E tests)              │        │
│                    └──────────────────────────────────────────────┘        │
│                                                                              │
│  ✓ Independent deployments                                                  │
│  ✓ Faster feedback loops                                                    │
│  ✓ Targeted rollbacks                                                       │
│  ✗ Complex orchestration                                                    │
│  ✗ Dependency management                                                    │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Pipeline Architecture

Service-Specific Pipelines

# .github/workflows/service-pipeline.yml
name: Service Pipeline

on:
  push:
    paths:
      - 'services/order-service/**'
      - '.github/workflows/order-service.yml'
  pull_request:
    paths:
      - 'services/order-service/**'

env:
  SERVICE_NAME: order-service
  REGISTRY: ghcr.io/${{ github.repository_owner }}
  
jobs:
  # Stage 1: Build and Unit Test
  build:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
      version: ${{ steps.version.outputs.version }}
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
          cache-dependency-path: services/${{ env.SERVICE_NAME }}/package-lock.json
      
      - name: Install dependencies
        working-directory: services/${{ env.SERVICE_NAME }}
        run: npm ci
      
      - name: Run linting
        working-directory: services/${{ env.SERVICE_NAME }}
        run: npm run lint
      
      - name: Run unit tests
        working-directory: services/${{ env.SERVICE_NAME }}
        run: npm test -- --coverage
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: services/${{ env.SERVICE_NAME }}/coverage/lcov.info
          flags: ${{ env.SERVICE_NAME }}
      
      - name: Generate version
        id: version
        run: |
          VERSION=$(date +%Y%m%d)-${{ github.run_number }}-${GITHUB_SHA::8}
          echo "version=$VERSION" >> $GITHUB_OUTPUT
      
      - name: Build Docker image
        uses: docker/build-push-action@v5
        with:
          context: services/${{ env.SERVICE_NAME }}
          push: false
          tags: ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:${{ steps.version.outputs.version }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          outputs: type=docker,dest=/tmp/image.tar
      
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: docker-image
          path: /tmp/image.tar

  # Stage 2: Security Scanning
  security:
    needs: build
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Download artifact
        uses: actions/download-artifact@v4
        with:
          name: docker-image
          path: /tmp
      
      - name: Load image
        run: docker load --input /tmp/image.tar
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
      
      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'
      
      - name: Run Snyk
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high
          command: test

  # Stage 3: Integration Tests
  integration:
    needs: [build, security]
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: testpassword
          POSTGRES_DB: testdb
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      
      redis:
        image: redis:7
        ports:
          - 6379:6379
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Download artifact
        uses: actions/download-artifact@v4
        with:
          name: docker-image
          path: /tmp
      
      - name: Load image
        run: docker load --input /tmp/image.tar
      
      - name: Run integration tests
        working-directory: services/${{ env.SERVICE_NAME }}
        env:
          DATABASE_URL: postgresql://postgres:testpassword@localhost:5432/testdb
          REDIS_URL: redis://localhost:6379
        run: npm run test:integration

  # Stage 4: Contract Tests
  contract-tests:
    needs: build
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install dependencies
        working-directory: services/${{ env.SERVICE_NAME }}
        run: npm ci
      
      - name: Run contract tests
        working-directory: services/${{ env.SERVICE_NAME }}
        run: npm run test:contract
      
      - name: Publish pacts
        if: github.ref == 'refs/heads/main'
        working-directory: services/${{ env.SERVICE_NAME }}
        env:
          PACT_BROKER_URL: ${{ secrets.PACT_BROKER_URL }}
          PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}
        run: npm run pact:publish

  # Stage 5: Push to Registry
  push:
    needs: [build, security, integration, contract-tests]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    
    steps:
      - name: Download artifact
        uses: actions/download-artifact@v4
        with:
          name: docker-image
          path: /tmp
      
      - name: Load image
        run: docker load --input /tmp/image.tar
      
      - name: Login to Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Push image
        run: |
          docker push ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }}
          docker tag ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }} \
                     ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:latest
          docker push ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:latest

  # Stage 6: Deploy to Staging
  deploy-staging:
    needs: push
    runs-on: ubuntu-latest
    environment: staging
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Update Kubernetes manifest
        run: |
          cd k8s/overlays/staging
          kustomize edit set image ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}=${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }}
      
      - name: Commit and push
        run: |
          git config --global user.name 'GitHub Actions'
          git config --global user.email '[email protected]'
          git add .
          git commit -m "Deploy ${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }} to staging"
          git push

  # Stage 7: E2E Tests on Staging
  e2e-staging:
    needs: deploy-staging
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Wait for deployment
        run: |
          kubectl rollout status deployment/${{ env.SERVICE_NAME }} -n staging --timeout=300s
      
      - name: Run E2E tests
        working-directory: tests/e2e
        env:
          BASE_URL: https://staging.example.com
        run: npm run test:e2e

  # Stage 8: Deploy to Production (with approval)
  deploy-production:
    needs: e2e-staging
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Update production manifest
        run: |
          cd k8s/overlays/production
          kustomize edit set image ${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}=${{ env.REGISTRY }}/${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }}
      
      - name: Commit and push
        run: |
          git config --global user.name 'GitHub Actions'
          git config --global user.email '[email protected]'
          git add .
          git commit -m "Deploy ${{ env.SERVICE_NAME }}:${{ needs.build.outputs.version }} to production"
          git push

Deployment Strategies

Blue-Green Deployment

┌─────────────────────────────────────────────────────────────────────────────┐
│                      BLUE-GREEN DEPLOYMENT                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  STEP 1: Current State (Blue is Active)                                     │
│  ─────────────────────────────────────────                                  │
│                                                                              │
│  ┌──────────────────────────────────────────────────────────────┐           │
│  │                       Load Balancer                          │           │
│  └───────────────────────────┬──────────────────────────────────┘           │
│                              │                                               │
│                              ▼                                               │
│  ┌────────────────────────────────────┐    ┌─────────────────────────────┐  │
│  │           BLUE (v1.0)              │    │       GREEN (idle)          │  │
│  │  ┌────────┐ ┌────────┐ ┌────────┐  │    │                             │  │
│  │  │ Pod 1  │ │ Pod 2  │ │ Pod 3  │  │    │        (empty)              │  │
│  │  └────────┘ └────────┘ └────────┘  │    │                             │  │
│  │         ✅ ACTIVE                   │    │                             │  │
│  └────────────────────────────────────┘    └─────────────────────────────┘  │
│                                                                              │
│  ═══════════════════════════════════════════════════════════════════════════│
│                                                                              │
│  STEP 2: Deploy New Version to Green                                        │
│  ───────────────────────────────────────                                    │
│                                                                              │
│  ┌────────────────────────────────────┐    ┌─────────────────────────────┐  │
│  │           BLUE (v1.0)              │    │       GREEN (v2.0)          │  │
│  │  ┌────────┐ ┌────────┐ ┌────────┐  │    │ ┌────────┐ ┌────────┐      │  │
│  │  │ Pod 1  │ │ Pod 2  │ │ Pod 3  │  │    │ │ Pod 1  │ │ Pod 2  │      │  │
│  │  └────────┘ └────────┘ └────────┘  │    │ └────────┘ └────────┘      │  │
│  │         ✅ ACTIVE                   │    │    🔄 DEPLOYING            │  │
│  └────────────────────────────────────┘    └─────────────────────────────┘  │
│                                                                              │
│  ═══════════════════════════════════════════════════════════════════════════│
│                                                                              │
│  STEP 3: Switch Traffic (Instant Cutover)                                   │
│  ────────────────────────────────────────                                   │
│                                                                              │
│  ┌──────────────────────────────────────────────────────────────┐           │
│  │                       Load Balancer                          │           │
│  └───────────────────────────────────────────────┬──────────────┘           │
│                                                  │                          │
│                                                  ▼                          │
│  ┌────────────────────────────────────┐    ┌─────────────────────────────┐  │
│  │           BLUE (v1.0)              │    │       GREEN (v2.0)          │  │
│  │  ┌────────┐ ┌────────┐ ┌────────┐  │    │ ┌────────┐ ┌────────┐      │  │
│  │  │ Pod 1  │ │ Pod 2  │ │ Pod 3  │  │    │ │ Pod 1  │ │ Pod 2  │      │  │
│  │  └────────┘ └────────┘ └────────┘  │    │ └────────┘ └────────┘      │  │
│  │        🔄 STANDBY                   │    │     ✅ ACTIVE              │  │
│  └────────────────────────────────────┘    └─────────────────────────────┘  │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

# blue-green/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-green
  labels:
    app: order-service
    slot: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
      slot: green
  template:
    metadata:
      labels:
        app: order-service
        slot: green
        version: v2.0.0
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:v2.0.0
        ports:
        - containerPort: 3000
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health/live
            port: 3000
          initialDelaySeconds: 15
          periodSeconds: 10
---
# Switch service selector to point to green
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
    slot: green  # Change to 'blue' to rollback
  ports:
  - port: 80
    targetPort: 3000

Blue-Green deployment script:

// scripts/blue-green-deploy.js
const k8s = require('@kubernetes/client-node');

class BlueGreenDeployer {
  constructor(serviceName, namespace = 'default') {
    this.serviceName = serviceName;
    this.namespace = namespace;
    
    const kc = new k8s.KubeConfig();
    kc.loadFromDefault();
    this.appsApi = kc.makeApiClient(k8s.AppsV1Api);
    this.coreApi = kc.makeApiClient(k8s.CoreV1Api);
  }

  async getCurrentSlot() {
    const service = await this.coreApi.readNamespacedService(
      this.serviceName, 
      this.namespace
    );
    return service.body.spec.selector.slot;
  }

  async getInactiveSlot() {
    const current = await this.getCurrentSlot();
    return current === 'blue' ? 'green' : 'blue';
  }

  async deployToInactive(imageTag) {
    const inactiveSlot = await this.getInactiveSlot();
    const deploymentName = `${this.serviceName}-${inactiveSlot}`;
    
    console.log(`Deploying ${imageTag} to ${inactiveSlot} slot...`);
    
    // Update deployment
    const patch = {
      spec: {
        template: {
          spec: {
            containers: [{
              name: this.serviceName,
              image: `myregistry/${this.serviceName}:${imageTag}`
            }]
          }
        }
      }
    };
    
    await this.appsApi.patchNamespacedDeployment(
      deploymentName,
      this.namespace,
      patch,
      undefined,
      undefined,
      undefined,
      undefined,
      undefined,
      { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }
    );
    
    // Wait for rollout
    await this.waitForRollout(deploymentName);
    
    return inactiveSlot;
  }

  async waitForRollout(deploymentName, timeout = 300) {
    const start = Date.now();
    
    while (Date.now() - start < timeout * 1000) {
      const deployment = await this.appsApi.readNamespacedDeployment(
        deploymentName,
        this.namespace
      );
      
      const status = deployment.body.status;
      if (status.readyReplicas === status.replicas &&
          status.updatedReplicas === status.replicas) {
        console.log(`✅ Deployment ${deploymentName} is ready`);
        return;
      }
      
      console.log(`⏳ Waiting... (${status.readyReplicas}/${status.replicas} ready)`);
      await new Promise(r => setTimeout(r, 5000));
    }
    
    throw new Error(`Deployment ${deploymentName} timed out`);
  }

  async runHealthChecks(slot) {
    console.log(`Running health checks on ${slot}...`);
    
    // Get pods in the slot
    const pods = await this.coreApi.listNamespacedPod(
      this.namespace,
      undefined,
      undefined,
      undefined,
      undefined,
      `app=${this.serviceName},slot=${slot}`
    );
    
    for (const pod of pods.body.items) {
      const ready = pod.status.conditions?.find(c => c.type === 'Ready');
      if (!ready || ready.status !== 'True') {
        throw new Error(`Pod ${pod.metadata.name} is not ready`);
      }
    }
    
    console.log(`✅ All pods healthy in ${slot}`);
  }

  async switchTraffic(newSlot) {
    console.log(`Switching traffic to ${newSlot}...`);
    
    const patch = {
      spec: {
        selector: {
          app: this.serviceName,
          slot: newSlot
        }
      }
    };
    
    await this.coreApi.patchNamespacedService(
      this.serviceName,
      this.namespace,
      patch,
      undefined,
      undefined,
      undefined,
      undefined,
      undefined,
      { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }
    );
    
    console.log(`✅ Traffic switched to ${newSlot}`);
  }

  async rollback() {
    const currentSlot = await this.getCurrentSlot();
    const previousSlot = currentSlot === 'blue' ? 'green' : 'blue';
    
    console.log(`🔄 Rolling back from ${currentSlot} to ${previousSlot}...`);
    await this.switchTraffic(previousSlot);
    console.log(`✅ Rollback complete`);
  }

  async deploy(imageTag) {
    try {
      // Deploy to inactive slot
      const newSlot = await this.deployToInactive(imageTag);
      
      // Run health checks
      await this.runHealthChecks(newSlot);
      
      // Switch traffic
      await this.switchTraffic(newSlot);
      
      console.log(`\n✅ Blue-green deployment complete!`);
      console.log(`   Active slot: ${newSlot}`);
      console.log(`   Image: ${imageTag}`);
      
    } catch (error) {
      console.error('Deployment failed:', error.message);
      console.log('Run rollback to switch back to previous version');
      throw error;
    }
  }
}

// Usage
const deployer = new BlueGreenDeployer('order-service');
deployer.deploy('v2.0.0');

Canary Deployment

# canary/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-canary
spec:
  replicas: 1  # Start with 1 canary replica
  selector:
    matchLabels:
      app: order-service
      track: canary
  template:
    metadata:
      labels:
        app: order-service
        track: canary
        version: v2.0.0
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:v2.0.0
---
# Main deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-stable
spec:
  replicas: 9  # 9 stable replicas
  selector:
    matchLabels:
      app: order-service
      track: stable
  template:
    metadata:
      labels:
        app: order-service
        track: stable
        version: v1.0.0
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:v1.0.0
---
# Service routes to both
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service  # Routes to both stable and canary
  ports:
  - port: 80
    targetPort: 3000

Progressive Canary with Argo Rollouts:

# argo-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-service
spec:
  replicas: 10
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:v2.0.0
        ports:
        - containerPort: 3000
  strategy:
    canary:
      # Canary steps
      steps:
      - setWeight: 5
      - pause: { duration: 2m }
      - setWeight: 10
      - pause: { duration: 5m }
      - setWeight: 25
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }
      - setWeight: 75
      - pause: { duration: 10m }
      - setWeight: 100
      
      # Automated analysis
      analysis:
        templates:
        - templateName: success-rate
        startingStep: 2
        args:
        - name: service-name
          value: order-service
      
      # Traffic routing with Istio
      trafficRouting:
        istio:
          virtualService:
            name: order-service
            routes:
            - primary
          destinationRule:
            name: order-service
            canarySubsetName: canary
            stableSubsetName: stable
      
      # Anti-affinity for canary pods
      antiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          weight: 100
---
# Analysis template
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    count: 5
    successCondition: result[0] >= 0.99
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(
            istio_requests_total{
              destination_service_name="{{args.service-name}}",
              response_code!~"5.*"
            }[5m]
          )) / 
          sum(rate(
            istio_requests_total{
              destination_service_name="{{args.service-name}}"
            }[5m]
          ))

GitOps with ArgoCD

ArgoCD Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                        GITOPS WORKFLOW                                       │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌────────────────┐       ┌────────────────┐       ┌────────────────┐       │
│  │  Application   │       │  CI Pipeline   │       │  Git Repo      │       │
│  │  Repository    │──────▶│  (GitHub       │──────▶│  (K8s Manifests│       │
│  │                │ build │   Actions)     │ push  │   Kustomize)   │       │
│  └────────────────┘       └────────────────┘       └───────┬────────┘       │
│                                                            │                 │
│                                                     watch/sync              │
│                                                            │                 │
│                                                            ▼                 │
│                                                   ┌────────────────┐        │
│                                                   │    ArgoCD      │        │
│  ┌────────────────────────────────────────────────┤                │        │
│  │                                                │  • Watches Git │        │
│  │          KUBERNETES CLUSTER                    │  • Syncs state │        │
│  │                                                │  • Self-heals  │        │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐       │                │        │
│  │  │ Service  │ │ Service  │ │ Service  │       └────────────────┘        │
│  │  │    A     │ │    B     │ │    C     │                                 │
│  │  └──────────┘ └──────────┘ └──────────┘                                 │
│  │                                                                          │
│  └──────────────────────────────────────────────────────────────────────────│
│                                                                              │
│  ✅ BENEFITS:                                                                │
│  • Git = single source of truth                                             │
│  • Audit trail (git history)                                                │
│  • Easy rollback (git revert)                                               │
│  • Declarative configuration                                                │
│  • Self-healing (auto-sync)                                                 │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

ArgoCD Application Configuration

# argocd/applications/order-service.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: order-service
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: services/order-service/overlays/production
    
    # Kustomize configuration
    kustomize:
      images:
      - myregistry/order-service
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: true  # Delete resources not in Git
      selfHeal: true  # Revert manual changes
      allowEmpty: false
    syncOptions:
    - Validate=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  
  # Health checks
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas  # Ignore HPA-managed replicas
  
  # Notifications
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments
    notifications.argoproj.io/subscribe.on-sync-failed.slack: deployments
---
# ApplicationSet for multiple environments
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: order-service-environments
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - environment: staging
        namespace: staging
        cluster: https://staging.k8s.example.com
      - environment: production
        namespace: production
        cluster: https://production.k8s.example.com
  
  template:
    metadata:
      name: 'order-service-{{environment}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: 'services/order-service/overlays/{{environment}}'
      destination:
        server: '{{cluster}}'
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Kustomize Structure

k8s-manifests/
└── services/
    └── order-service/
        ├── base/
        │   ├── kustomization.yaml
        │   ├── deployment.yaml
        │   ├── service.yaml
        │   ├── configmap.yaml
        │   └── hpa.yaml
        └── overlays/
            ├── staging/
            │   ├── kustomization.yaml
            │   ├── replicas-patch.yaml
            │   └── config-patch.yaml
            └── production/
                ├── kustomization.yaml
                ├── replicas-patch.yaml
                └── config-patch.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml
- service.yaml
- configmap.yaml
- hpa.yaml

commonLabels:
  app: order-service

images:
- name: order-service
  newName: myregistry/order-service
  newTag: latest

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

resources:
- ../../base

patches:
- path: replicas-patch.yaml
- path: config-patch.yaml

images:
- name: myregistry/order-service
  newTag: v2.1.0  # Updated by CI pipeline

configMapGenerator:
- name: order-service-config
  behavior: merge
  literals:
  - LOG_LEVEL=warn
  - ENABLE_DEBUG=false

Monorepo vs Polyrepo CI/CD

Monorepo Strategy

# .github/workflows/monorepo-ci.yml
name: Monorepo CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.filter.outputs.changes }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            order-service:
              - 'services/order-service/**'
            user-service:
              - 'services/user-service/**'
            payment-service:
              - 'services/payment-service/**'
            shared-libs:
              - 'libs/**'

  build:
    needs: detect-changes
    if: ${{ needs.detect-changes.outputs.services != '[]' }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: ${{ fromJson(needs.detect-changes.outputs.services) }}
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Build ${{ matrix.service }}
        if: matrix.service != 'shared-libs'
        run: |
          cd services/${{ matrix.service }}
          docker build -t myregistry/${{ matrix.service }}:${{ github.sha }} .
      
      - name: Rebuild dependent services
        if: matrix.service == 'shared-libs'
        run: |
          # Find all services depending on shared libs
          for service in $(find services -name "package.json" -exec grep -l "@myorg/shared" {} \;); do
            service_name=$(dirname $service | xargs basename)
            echo "Building $service_name..."
            cd services/$service_name
            docker build -t myregistry/$service_name:${{ github.sha }} .
            cd ../..
          done

Polyrepo Strategy

# Each service has its own repo
# order-service/.github/workflows/ci.yml
name: Order Service CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Build
        run: docker build -t myregistry/order-service:${{ github.sha }} .
      
      - name: Test
        run: npm test
      
      - name: Push
        if: github.ref == 'refs/heads/main'
        run: docker push myregistry/order-service:${{ github.sha }}
      
      - name: Trigger deployment
        if: github.ref == 'refs/heads/main'
        uses: peter-evans/repository-dispatch@v2
        with:
          token: ${{ secrets.DEPLOY_REPO_TOKEN }}
          repository: myorg/k8s-manifests
          event-type: deploy-order-service
          client-payload: '{"image": "myregistry/order-service:${{ github.sha }}"}'

Interview Questions

Q1: How do you design CI/CD for microservices?

Answer:Key principles:

Independent pipelines: Each service has its own pipeline
Parallel execution: Build services in parallel
Contract testing: Verify API contracts between services
Progressive deployment: Canary → Staging → Production

Pipeline stages:

Build → Unit Test → Security Scan → Integration Test → 
Contract Test → Push → Deploy Staging → E2E Test → Deploy Prod

Best practices:

Feature flags for incomplete features
Automated rollback on failure
Environment parity (staging ≈ production)

Q2: Compare Blue-Green vs Canary deployments

Answer:

Aspect	Blue-Green	Canary
Traffic switch	Instant (100%)	Gradual (5% → 100%)
Rollback speed	Instant	Instant
Resource cost	2x capacity	1x + small %
Risk	Higher (all traffic)	Lower (partial traffic)
Complexity	Simpler	More complex

Use Blue-Green when:

Need instant rollback
Have resources for 2x capacity
Database schema is compatible

Use Canary when:

Want to test with real traffic
Need to validate gradually
Resource constrained

Q3: What is GitOps and why use it?

Answer:GitOps = Git as single source of truth for infrastructure.Principles:

Declarative configuration in Git
Version controlled (history, audit)
Automated sync (Git → Cluster)
Self-healing (drift correction)

Benefits:

Easy rollback (git revert)
Audit trail (git history)
Pull-based security (no cluster credentials in CI)
Consistent environments

Tools: ArgoCD, Flux, Jenkins X

Q4: How do you handle database migrations in CI/CD?

Answer:Strategies:

Backward-compatible migrations
- Add columns, don’t remove
- Deploy code that works with old/new schema
- Separate deployment from migration

Expand-Contract pattern

Step 1: Add new column (nullable)
Step 2: Deploy code writing to both
Step 3: Backfill old data
Step 4: Deploy code reading from new
Step 5: Remove old column

Versioned APIs
- API v1 uses old schema
- API v2 uses new schema
- Migrate gradually

Never: Rename/delete columns in one deploy

Q5: How do you test microservices in CI/CD?

Answer:Testing pyramid:

          /\
         /  \      E2E Tests (few)
        /----\     
       /      \    Integration Tests
      /--------\   
     /          \  Contract Tests (Pact)
    /------------\ 
   /              \ Unit Tests (many)
  /________________\

In pipeline:

Unit tests: Fast, isolated, run first
Contract tests: Verify API contracts
Integration tests: Test with real dependencies
E2E tests: Full user flows (staging only)

Key practices:

Use service containers in CI (Postgres, Redis)
Mock external services
Parallelize test suites
Test data isolation

Chapter Summary

Key Takeaways:

Each microservice needs its own CI/CD pipeline
Use progressive deployment strategies (Canary, Blue-Green)
GitOps provides audit trail and easy rollbacks
Contract testing catches breaking changes early
Automate everything: testing, security scanning, deployment

Next Chapter: Database Patterns - Data management strategies for microservices.

Testing & Code Quality

Crash Courses

AI Engineering

Math for ML - Understanding Linear Algebra

Probability & Statistics for ML

Math for ML - Understanding Calculus

ML Mastery

Deep Learning Mastery

NestJS Mastery

Microservices Mastery

Low Level Design

OOP Concepts

SOLID Principles

Design Patterns

LLD Case Studies

System Design (HLD)

Senior Level (L5+/Staff)

HLD Case Studies

Engineering Fundamentals

DevOps & Operations

AWS Cloud

AWS Monitoring & Observability

AWS Security Services

AWS Serverless

AWS Operations

AWS Advanced

AWS Case Studies

DevOps Tools

Database Engineering

HIPAA Compliance Mastery

Operating Systems

Linux Internals

Distributed Systems

Networking Mastery

Build Your Own X

Go Lang Mastery

C Programming

Distributed System Tools

​CI/CD for Microservices

​CI/CD Challenges in Microservices

​Pipeline Architecture

​Service-Specific Pipelines

​Deployment Strategies

​Blue-Green Deployment

​Canary Deployment

​GitOps with ArgoCD

​ArgoCD Architecture

​ArgoCD Application Configuration

​Kustomize Structure

​Monorepo vs Polyrepo CI/CD

​Monorepo Strategy

​Polyrepo Strategy

​Interview Questions

​Chapter Summary