> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Containers & Artifacts

> Master Artifact Registry, Cloud Build, and Containerization on GCP

# Chapter 9: The Container Workflow - Artifact Registry and Cloud Build

Before a container ever reaches a Kubernetes cluster, it must be built, secured, and stored. In Google Cloud, this "Supply Chain" is handled by **Cloud Build** and **Artifact Registry**, integrated with deep security features like **Artifact Analysis** and **Binary Authorization**.

***

## 1. Artifact Registry: Beyond Docker

Artifact Registry is the next generation of Google Container Registry (GCR). It is a fully managed service that supports container images, language packages, and OS packages.

### Key Architectural Features

* **Regional Repositories:** Unlike the legacy GCR (which used multi-regional buckets), Artifact Registry allows you to store images in specific regions (e.g., `europe-west1`). This reduces latency for your GKE clusters and eliminates cross-region egress costs. If you are migrating from AWS, Artifact Registry is the equivalent of Amazon ECR but with multi-format support (not just Docker images).
* **Virtual Repositories:** You can create a single "Virtual" endpoint that aggregates multiple upstream repositories. This allows developers to use one URL while pulling from both internal private repos and public mirrors. Think of it like a smart DNS that routes to the right backend based on which repo has the artifact.
* **Remote Repositories:** Act as a caching proxy for public registries like Docker Hub or npmjs.org. This protects your builds from upstream outages (like the "left-pad" incident) and avoids Docker Hub's rate limits (100 pulls/6hr for anonymous users, 200 for free accounts).

**Cost Tip:** Artifact Registry charges \$0.10/GB/month for storage and standard network egress rates for pulls. A common waste pattern is keeping every image tag forever. A single microservice with daily builds generates \~30 images/month. Over a year, that is 360 images, potentially consuming 50-100GB. Set up cleanup policies to automatically delete images older than 90 days (keeping tagged releases), and you will cut Artifact Registry costs by 70-80%.

***

## 2. Cloud Build: Serverless CI/CD

Cloud Build is a serverless execution engine. It executes your build as a series of **Build Steps**, where each step is a container.

### 2.1 Custom Build Steps

If the standard Google-managed steps (gcloud, docker, git) aren't enough, you can create your own.

* **Concept**: Any container image can be a build step.
* **Example**: Create a custom Go-based container that performs specialized security audits or database migrations.

### 2.2 Secret Manager Integration

Never hardcode API keys or credentials in your `cloudbuild.yaml`.

* **The Right Way**: Store secrets in **Secret Manager** and reference them in your build configuration.

**Example (Injecting a GitHub Token):**

```yaml theme={null}
availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/github-token/versions/latest
    env: 'GITHUB_TOKEN'

steps:
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    git clone https://$$GITHUB_TOKEN@github.com/my-org/private-repo.git
  secretEnv: ['GITHUB_TOKEN']
```

### 2.3 Layer Caching with GCS

To speed up builds, Cloud Build can cache Docker layers in a GCS bucket.

* **Mechanism**: The `--cache` flag in the `gcloud builds submit` command.
* **Benefit**: Reduces build times by 50-80% for complex apps where only the application code changes, not the base OS or dependencies.

***

## 3. The "Hardened Container" Stack

Security in containerization is about layers. GCP provides a unique stack to ensure your containers are isolated and minimal.

### Container-Optimized OS (COS)

The default node image for GKE. Think of COS as the "stripped-down race car" of operating systems -- everything non-essential has been removed to minimize the attack surface and maximize boot speed.

* **Minimal Footprint:** It includes only the essential components to run Docker/containerd. No package manager (`apt`/`yum`), no SSH daemon by default, no unnecessary services. AWS uses Amazon Linux 2 or Bottleneck as its equivalent minimal container OS, and Azure uses Azure Linux (Mariner).
* **Read-Only Rootfs:** The root file system is read-only. Even if a container escapes to the host, it cannot modify the OS. This is a critical defense-in-depth measure.
* **Auto-Updates:** It includes a built-in mechanism for safe, automatic security updates.

### gVisor (GKE Sandbox)

Standard containers share the host's Linux kernel. If a container exploits a kernel vulnerability, it can take over the entire machine.

* **The Solution:** **gVisor** is an open-source "user-space kernel" written in Go.
* **How it works:** It intercepts system calls from the container and handles them in a sandbox, providing a strong boundary between the container and the host kernel.
* **Use Case:** Running untrusted code or multi-tenant workloads.

***

## 4. Advanced Supply Chain: Remote Repositories and Private Pools

### 4.1 Artifact Registry: Remote and Virtual Repositories

Artifact Registry isn't just a place to push images; it's a proxy for the world.

* **Remote Repositories**: You can point a repository to Docker Hub or npmjs.org. When you pull an image, Artifact Registry caches it locally. This protects you from upstream outages and reduces latency.
* **Virtual Repositories**: Aggregate multiple repositories (standard, remote, and other virtual) into a single endpoint. This is the "Gold Standard" for enterprise package management, providing a unified URL for developers.

### 4.2 Cloud Build Private Pools: VPC Integration

For enterprise security, you often need to build code that resides in a private network (e.g., a private GitLab or a Bitbucket server).

* **VPC Peering**: Private Pools require a VPC peering connection between your network and the Google-managed "Service Networking" project.
* **No Internet Access**: You can configure a Private Pool to have **zero** external internet access, ensuring that your source code and build artifacts never leave your private perimeter.

***

## 5. Security: The Hardened Host and Sandbox

### 4.1 Artifact Analysis & Vulnerability Insights

Once an image is in Artifact Registry, **Artifact Analysis** kicks in.

* **Continuous Scanning:** It continuously scans images for new CVEs (Common Vulnerabilities and Exposures). If a new vulnerability is discovered in `nginx:latest`, you'll see an alert in your registry.
* **On-Demand Scanning (Vulnerability Insights API):** For CI/CD pipelines, you can use the Vulnerability Insights API to scan an image *before* it is fully deployed.
  * **Workflow:** Cloud Build builds the image -> Triggers On-Demand Scan -> If "Critical" count > 0 -> Build Fails.

```bash theme={null}
# Example: Triggering an on-demand scan via CLI
gcloud artifacts docker images scan us-central1-docker.pkg.dev/$PROJECT_ID/prod-images/my-app:v1
```

### 4.2 Artifact Registry Cleanup Policies

Storing every version of every image forever is expensive. **Cleanup Policies** allow you to automate the deletion of old or unused artifacts.

#### Tag Immutability: Protecting Production

A critical security best practice is **Tag Immutability**.

* **The Problem**: Someone accidentally pushes a broken image as `v1.0.0`, overwriting the stable production image.
* **The Solution**: Configure Artifact Registry to make tags immutable. Once `v1.0.0` is pushed, it cannot be overwritten. Updates must be pushed as `v1.0.1`.

### 4.3 Artifact Analysis & Severity Levels

Once an image is in Artifact Registry, **Artifact Analysis** scans it for CVEs.

* **Severity Levels**:
  * **CRITICAL**: Remote code execution or complete system takeover.
  * **HIGH**: Significant security risk (e.g., privilege escalation).
  * **MEDIUM/LOW**: Minor risks requiring authentication or physical access.
* **CVSS Score**: A numerical value (0-10) indicating the risk. SRE teams typically set a "Break-the-Build" threshold at 7.0 or higher.

***

## 5. Security: Binary Authorization (BinAuthz)

Binary Authorization is the ultimate gatekeeper for GKE.

### 5.1 The "Chain of Trust"

1. **Policy**: You define a policy in BinAuthz stating that every image must be signed by an **Attestor**.
2. **Attestation**: After Cloud Build successfully runs security scans and tests, it uses a PGP key to "sign" the image hash.
3. **Enforcement**: When you attempt to `kubectl apply` a deployment, the GKE admission controller checks BinAuthz. If the image doesn't have a valid signature from the required Attestor, the deployment is **rejected**.

***

## 5. Interview Preparation: Architectural Deep Dive

**1. Q: Why should an organization migrate from Container Registry (GCR) to Artifact Registry?**
**A:** Artifact Registry is the successor to GCR and offers several enterprise features:

* **Regional Repositories:** GCR is multi-regional (e.g., `us.gcr.io`), leading to egress costs when GKE clusters pull across regions. Artifact Registry is regional (e.g., `us-central1-docker.pkg.dev`), eliminating those costs.
* **Cleanup Policies:** Automated deletion of old images based on tags or age.
* **Multiple Formats:** Supports Docker, npm, Maven, Python, and OS packages (Apt/Yum) in a single service.
* **VPC Service Controls:** Native support for security perimeters.

**2. Q: Explain the security benefit of gVisor (GKE Sandbox).**
**A:** Standard containers share the host's Linux kernel. If a container escapes (e.g., via a kernel exploit), it could gain control of the host node. **gVisor** acts as a user-space kernel that intercepts and filters system calls. It provides a much stronger isolation boundary, ensuring that even if a container is compromised, it cannot interact directly with the host kernel, making it ideal for running untrusted code or multi-tenant workloads.

**3. Q: How does Binary Authorization integrate with Cloud Build?**
**A:** Binary Authorization is a deploy-time security gate. During the CI/CD process, Cloud Build (or a scanning tool) creates an **Attestation** (a digital signature) for a specific image hash. The Binary Authorization policy in GKE is configured to only allow images that have a valid signature from a specific "Attestor." This ensures that only code that has passed security scans and automated tests can reach production.

**4. Q: What is a Cloud Build "Private Pool" and why use it?**
**A:** By default, Cloud Build runs in a Google-managed multi-tenant network. A **Private Pool** allows you to run build workers inside a private VPC. This is critical for builds that need to access internal resources like a private Bitbucket server, a Cloud SQL instance with only a private IP, or an on-premise artifact repository via Interconnect, all while keeping the traffic entirely off the public internet.

**5. Q: What is the "Vulnerability Insights API" and how is it used in a pipeline?**
**A:** It is the API used for **On-Demand Scanning**. Unlike the continuous scan that happens in the background of Artifact Registry, the Insights API allows you to trigger a scan programmatically *before* an image is stored or deployed. In a CI/CD pipeline, you can call this API to get a report on CVEs and fail the build if the number of "Critical" or "High" vulnerabilities exceeds your organization's threshold.

***

## Implementation: The "Secure Build" Lab

### Creating a Secure Repository and Build Pipeline

```bash theme={null}
# 1. Create a Docker repository with Vulnerability Scanning enabled
# Why us-central1 specifically: Match the region of your GKE cluster to eliminate
# cross-region egress charges on image pulls. A busy cluster pulling 100 images/day
# from a different region costs ~$3-5/day in unnecessary egress.
gcloud artifacts repositories create prod-images \
    --repository-format=docker \
    --location=us-central1 \
    --description="Production hardened images"

# 2. Submit a build that uses Kaniko for caching
# Why Kaniko over standard Docker build: Kaniko does not require a Docker daemon,
# making it safer for multi-tenant build environments (no privileged containers needed).
# Why --cache=true: Reuses unchanged layers from previous builds, cutting build times by 50-80%.
gcloud builds submit . \
    --config=cloudbuild.yaml \
    --substitutions=_IMAGE_NAME=my-app,_VERSION=v1

# 3. cloudbuild.yaml example
# steps:
# - name: 'gcr.io/kaniko-project/executor:latest'
#   args:
#   - --destination=us-central1-docker.pkg.dev/$PROJECT_ID/prod-images/$_IMAGE_NAME:$_VERSION
#   - --cache=true          # Reuse unchanged layers -- massive build time savings
#   - --cache-ttl=24h       # Cache expires after 24h to pick up security patches in base images
```

### Pro-Tip: Vulnerability Alerts

Don't just scan; act. You can set up a **Pub/Sub topic** for Artifact Analysis. Every time a new vulnerability is found, GCS can send a message to a Cloud Function that automatically triggers a re-build or alerts your SRE team on Slack.