Microservices & gRPC

Go is the language of choice for building microservices due to its simplicity, performance, and excellent concurrency support. Kubernetes, Docker, Istio, etcd, and Prometheus are all written in Go — the infrastructure that runs microservices is itself built with Go, which tells you something about its fitness for this domain. This chapter covers gRPC, Protocol Buffers, and essential microservices patterns.

gRPC Fundamentals

gRPC is a high-performance RPC framework that uses HTTP/2 and Protocol Buffers. Think of gRPC as a phone call between services: both sides agree on the language (protobuf schema) before the conversation starts, the connection is efficient (HTTP/2 multiplexing — multiple calls share one connection), and both sides can talk simultaneously (bidirectional streaming). REST, by comparison, is more like sending letters: each message is self-contained (JSON), the format is human-readable but verbose, and there is no built-in concept of an ongoing conversation.

Why gRPC?

Feature	gRPC	REST
Protocol	HTTP/2	HTTP/1.1
Payload	Binary (Protobuf)	Text (JSON/XML)
Streaming	Bidirectional	Limited
Code Generation	Yes	Optional
Type Safety	Strong	Weak
Performance	Faster	Slower

Protocol Buffers

Protocol Buffers (protobuf) is Google’s language-neutral serialization format.

// user.proto
syntax = "proto3";

package user;
option go_package = "github.com/myapp/proto/user";

// User message
message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
  repeated string roles = 4;
  UserStatus status = 5;
  google.protobuf.Timestamp created_at = 6;
}

enum UserStatus {
  USER_STATUS_UNSPECIFIED = 0;
  USER_STATUS_ACTIVE = 1;
  USER_STATUS_INACTIVE = 2;
  USER_STATUS_BANNED = 3;
}

// Request/Response messages
message GetUserRequest {
  int64 id = 1;
}

message GetUserResponse {
  User user = 1;
}

message ListUsersRequest {
  int32 page_size = 1;
  string page_token = 2;
  string filter = 3;
}

message ListUsersResponse {
  repeated User users = 1;
  string next_page_token = 2;
  int32 total_count = 3;
}

message CreateUserRequest {
  string name = 1;
  string email = 2;
  string password = 3;
}

// Service definition
service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc ListUsers(ListUsersRequest) returns (ListUsersResponse);
  rpc CreateUser(CreateUserRequest) returns (User);
  rpc UpdateUser(User) returns (User);
  rpc DeleteUser(GetUserRequest) returns (google.protobuf.Empty);
  
  // Streaming
  rpc WatchUsers(ListUsersRequest) returns (stream User);
  rpc BatchCreateUsers(stream CreateUserRequest) returns (ListUsersResponse);
}

Generating Go Code

# Install protoc compiler and plugins
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# Generate code
protoc --go_out=. --go_opt=paths=source_relative \
       --go-grpc_out=. --go-grpc_opt=paths=source_relative \
       proto/user.proto

gRPC Server

Implementing the Service

package main

import (
    "context"
    "log"
    "net"
    
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    
    pb "github.com/myapp/proto/user"
)

type userServer struct {
    pb.UnimplementedUserServiceServer // Embed this to satisfy the interface for forward compatibility
    users map[int64]*pb.User
}

func NewUserServer() *userServer {
    return &userServer{
        users: make(map[int64]*pb.User),
    }
}

func (s *userServer) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.GetUserResponse, error) {
    user, ok := s.users[req.Id]
    if !ok {
        // Idiomatic gRPC: use status.Errorf with standard codes, not fmt.Errorf.
        // The client can switch on the code to decide how to handle the error.
        return nil, status.Errorf(codes.NotFound, "user %d not found", req.Id)
    }
    
    return &pb.GetUserResponse{User: user}, nil
}

func (s *userServer) ListUsers(ctx context.Context, req *pb.ListUsersRequest) (*pb.ListUsersResponse, error) {
    var users []*pb.User
    for _, user := range s.users {
        users = append(users, user)
        if len(users) >= int(req.PageSize) {
            break
        }
    }
    
    return &pb.ListUsersResponse{
        Users:      users,
        TotalCount: int32(len(s.users)),
    }, nil
}

func (s *userServer) CreateUser(ctx context.Context, req *pb.CreateUserRequest) (*pb.User, error) {
    // Validate
    if req.Name == "" {
        return nil, status.Error(codes.InvalidArgument, "name is required")
    }
    if req.Email == "" {
        return nil, status.Error(codes.InvalidArgument, "email is required")
    }
    
    // Create user
    user := &pb.User{
        Id:     int64(len(s.users) + 1),
        Name:   req.Name,
        Email:  req.Email,
        Status: pb.UserStatus_USER_STATUS_ACTIVE,
    }
    s.users[user.Id] = user
    
    return user, nil
}

// Server streaming
func (s *userServer) WatchUsers(req *pb.ListUsersRequest, stream pb.UserService_WatchUsersServer) error {
    // Send existing users
    for _, user := range s.users {
        if err := stream.Send(user); err != nil {
            return err
        }
    }
    
    // In a real app, watch for new users and stream them
    // This is simplified - you'd use channels/pub-sub
    ticker := time.NewTicker(time.Second)
    defer ticker.Stop()
    
    for {
        select {
        case <-stream.Context().Done():
            return stream.Context().Err()
        case <-ticker.C:
            // Check for new users and stream
        }
    }
}

// Client streaming
func (s *userServer) BatchCreateUsers(stream pb.UserService_BatchCreateUsersServer) error {
    var users []*pb.User
    
    for {
        req, err := stream.Recv()
        if err == io.EOF {
            return stream.SendAndClose(&pb.ListUsersResponse{
                Users:      users,
                TotalCount: int32(len(users)),
            })
        }
        if err != nil {
            return err
        }
        
        user, err := s.CreateUser(stream.Context(), req)
        if err != nil {
            return err
        }
        users = append(users, user)
    }
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    
    grpcServer := grpc.NewServer()
    pb.RegisterUserServiceServer(grpcServer, NewUserServer())
    
    log.Println("gRPC server starting on :50051")
    if err := grpcServer.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

Pitfall — Not Implementing UnimplementedServer: If you do not embed pb.UnimplementedUserServiceServer, your server will fail to compile when new RPC methods are added to the proto definition. The Unimplemented embedding provides default “unimplemented” responses for new methods, ensuring forward compatibility. This is a gRPC best practice and is enforced by default in newer versions of protoc-gen-go-grpc.

Pitfall — Shared Mutable State Without Synchronization: The userServer above uses a plain map which is not safe for concurrent access. In production, gRPC servers handle multiple requests concurrently (one goroutine per request). You must protect shared state with a sync.RWMutex or use a database. Without this, concurrent map access will cause a runtime panic.

gRPC Client

package main

import (
    "context"
    "log"
    "time"
    
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
    
    pb "github.com/myapp/proto/user"
)

func main() {
    // Connect to server
    conn, err := grpc.Dial("localhost:50051",
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )
    if err != nil {
        log.Fatalf("failed to connect: %v", err)
    }
    defer conn.Close()
    
    client := pb.NewUserServiceClient(conn)
    
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    
    // Create user
    user, err := client.CreateUser(ctx, &pb.CreateUserRequest{
        Name:  "John Doe",
        Email: "john@example.com",
    })
    if err != nil {
        log.Fatalf("CreateUser failed: %v", err)
    }
    log.Printf("Created user: %+v", user)
    
    // Get user
    resp, err := client.GetUser(ctx, &pb.GetUserRequest{Id: user.Id})
    if err != nil {
        log.Fatalf("GetUser failed: %v", err)
    }
    log.Printf("Got user: %+v", resp.User)
    
    // List users
    listResp, err := client.ListUsers(ctx, &pb.ListUsersRequest{PageSize: 10})
    if err != nil {
        log.Fatalf("ListUsers failed: %v", err)
    }
    log.Printf("Found %d users", listResp.TotalCount)
    
    // Server streaming
    stream, err := client.WatchUsers(ctx, &pb.ListUsersRequest{})
    if err != nil {
        log.Fatalf("WatchUsers failed: %v", err)
    }
    
    for {
        user, err := stream.Recv()
        if err == io.EOF {
            break
        }
        if err != nil {
            log.Fatalf("stream error: %v", err)
        }
        log.Printf("Received user: %+v", user)
    }
}

gRPC Interceptors (Middleware)

Server Interceptors

// Logging interceptor
func loggingInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    start := time.Now()
    
    resp, err := handler(ctx, req)
    
    log.Printf("Method: %s, Duration: %s, Error: %v",
        info.FullMethod,
        time.Since(start),
        err,
    )
    
    return resp, err
}

// Recovery interceptor
func recoveryInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (resp interface{}, err error) {
    defer func() {
        if r := recover(); r != nil {
            err = status.Errorf(codes.Internal, "panic: %v", r)
            log.Printf("Panic recovered: %v\n%s", r, debug.Stack())
        }
    }()
    
    return handler(ctx, req)
}

// Auth interceptor
func authInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    md, ok := metadata.FromIncomingContext(ctx)
    if !ok {
        return nil, status.Error(codes.Unauthenticated, "missing metadata")
    }
    
    tokens := md.Get("authorization")
    if len(tokens) == 0 {
        return nil, status.Error(codes.Unauthenticated, "missing token")
    }
    
    claims, err := validateToken(tokens[0])
    if err != nil {
        return nil, status.Error(codes.Unauthenticated, "invalid token")
    }
    
    // Add claims to context
    ctx = context.WithValue(ctx, userClaimsKey{}, claims)
    
    return handler(ctx, req)
}

// Using interceptors
func main() {
    grpcServer := grpc.NewServer(
        grpc.ChainUnaryInterceptor(
            recoveryInterceptor,
            loggingInterceptor,
            authInterceptor,
        ),
        grpc.ChainStreamInterceptor(
            // Stream interceptors
        ),
    )
}

Client Interceptors

func clientLoggingInterceptor(
    ctx context.Context,
    method string,
    req, reply interface{},
    cc *grpc.ClientConn,
    invoker grpc.UnaryInvoker,
    opts ...grpc.CallOption,
) error {
    start := time.Now()
    err := invoker(ctx, method, req, reply, cc, opts...)
    log.Printf("Method: %s, Duration: %s, Error: %v", method, time.Since(start), err)
    return err
}

func clientAuthInterceptor(token string) grpc.UnaryClientInterceptor {
    return func(
        ctx context.Context,
        method string,
        req, reply interface{},
        cc *grpc.ClientConn,
        invoker grpc.UnaryInvoker,
        opts ...grpc.CallOption,
    ) error {
        ctx = metadata.AppendToOutgoingContext(ctx, "authorization", token)
        return invoker(ctx, method, req, reply, cc, opts...)
    }
}

// Using client interceptors
conn, err := grpc.Dial("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithChainUnaryInterceptor(
        clientLoggingInterceptor,
        clientAuthInterceptor("my-token"),
    ),
)

Service Discovery

Using Consul

import (
    "github.com/hashicorp/consul/api"
)

type ServiceRegistry struct {
    client *api.Client
}

func NewServiceRegistry(addr string) (*ServiceRegistry, error) {
    config := api.DefaultConfig()
    config.Address = addr
    
    client, err := api.NewClient(config)
    if err != nil {
        return nil, err
    }
    
    return &ServiceRegistry{client: client}, nil
}

func (r *ServiceRegistry) Register(name, address string, port int) error {
    registration := &api.AgentServiceRegistration{
        ID:      fmt.Sprintf("%s-%s-%d", name, address, port),
        Name:    name,
        Address: address,
        Port:    port,
        Check: &api.AgentServiceCheck{
            GRPC:                           fmt.Sprintf("%s:%d", address, port),
            Interval:                       "10s",
            DeregisterCriticalServiceAfter: "1m",
        },
    }
    
    return r.client.Agent().ServiceRegister(registration)
}

func (r *ServiceRegistry) Discover(name string) ([]*api.ServiceEntry, error) {
    entries, _, err := r.client.Health().Service(name, "", true, nil)
    return entries, err
}

func (r *ServiceRegistry) Deregister(id string) error {
    return r.client.Agent().ServiceDeregister(id)
}

Load Balancing with gRPC

import (
    "google.golang.org/grpc/resolver"
)

// Custom resolver for service discovery
type consulResolver struct {
    registry *ServiceRegistry
    service  string
    cc       resolver.ClientConn
}

func (r *consulResolver) ResolveNow(options resolver.ResolveNowOptions) {
    entries, err := r.registry.Discover(r.service)
    if err != nil {
        return
    }
    
    var addrs []resolver.Address
    for _, entry := range entries {
        addr := fmt.Sprintf("%s:%d", entry.Service.Address, entry.Service.Port)
        addrs = append(addrs, resolver.Address{Addr: addr})
    }
    
    r.cc.UpdateState(resolver.State{Addresses: addrs})
}

// Using round-robin load balancing
conn, err := grpc.Dial("consul:///user-service",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

Health Checks

import (
    "google.golang.org/grpc/health"
    "google.golang.org/grpc/health/grpc_health_v1"
)

func main() {
    grpcServer := grpc.NewServer()
    
    // Register your services
    pb.RegisterUserServiceServer(grpcServer, NewUserServer())
    
    // Register health service
    healthServer := health.NewServer()
    grpc_health_v1.RegisterHealthServer(grpcServer, healthServer)
    
    // Set service health status
    healthServer.SetServingStatus("user.UserService", grpc_health_v1.HealthCheckResponse_SERVING)
    
    // Update health based on dependencies
    go func() {
        for {
            if dbHealthy() {
                healthServer.SetServingStatus("", grpc_health_v1.HealthCheckResponse_SERVING)
            } else {
                healthServer.SetServingStatus("", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
            }
            time.Sleep(10 * time.Second)
        }
    }()
    
    grpcServer.Serve(lis)
}

Circuit Breaker

import "github.com/sony/gobreaker"

type UserClient struct {
    client pb.UserServiceClient
    cb     *gobreaker.CircuitBreaker
}

func NewUserClient(conn *grpc.ClientConn) *UserClient {
    settings := gobreaker.Settings{
        Name:        "UserService",
        MaxRequests: 3,                      // Max requests in half-open state
        Interval:    10 * time.Second,       // Cyclic period of closed state
        Timeout:     30 * time.Second,       // Timeout in open state
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
            return counts.Requests >= 3 && failureRatio >= 0.6
        },
        OnStateChange: func(name string, from, to gobreaker.State) {
            log.Printf("Circuit breaker %s: %s -> %s", name, from, to)
        },
    }
    
    return &UserClient{
        client: pb.NewUserServiceClient(conn),
        cb:     gobreaker.NewCircuitBreaker(settings),
    }
}

func (c *UserClient) GetUser(ctx context.Context, id int64) (*pb.User, error) {
    result, err := c.cb.Execute(func() (interface{}, error) {
        resp, err := c.client.GetUser(ctx, &pb.GetUserRequest{Id: id})
        if err != nil {
            return nil, err
        }
        return resp.User, nil
    })
    
    if err != nil {
        return nil, err
    }
    
    return result.(*pb.User), nil
}

Event-Driven Architecture

Publishing Events

type Event struct {
    ID        string
    Type      string
    Payload   json.RawMessage
    Timestamp time.Time
}

type EventPublisher interface {
    Publish(ctx context.Context, topic string, event *Event) error
}

// NATS implementation
type NATSPublisher struct {
    conn *nats.Conn
}

func (p *NATSPublisher) Publish(ctx context.Context, topic string, event *Event) error {
    data, err := json.Marshal(event)
    if err != nil {
        return err
    }
    return p.conn.Publish(topic, data)
}

// Usage in service
func (s *OrderService) CreateOrder(ctx context.Context, order *Order) error {
    // Create order in DB
    if err := s.repo.Create(ctx, order); err != nil {
        return err
    }
    
    // Publish event
    event := &Event{
        ID:        uuid.New().String(),
        Type:      "order.created",
        Payload:   json.RawMessage(mustMarshal(order)),
        Timestamp: time.Now(),
    }
    
    return s.publisher.Publish(ctx, "orders", event)
}

Consuming Events

type EventHandler func(ctx context.Context, event *Event) error

type EventConsumer struct {
    conn     *nats.Conn
    handlers map[string]EventHandler
}

func (c *EventConsumer) Subscribe(topic string, handler EventHandler) error {
    _, err := c.conn.Subscribe(topic, func(msg *nats.Msg) {
        var event Event
        if err := json.Unmarshal(msg.Data, &event); err != nil {
            log.Printf("Failed to unmarshal event: %v", err)
            return
        }
        
        ctx := context.Background()
        if err := handler(ctx, &event); err != nil {
            log.Printf("Handler error: %v", err)
        }
    })
    return err
}

// Handler
func handleOrderCreated(ctx context.Context, event *Event) error {
    var order Order
    if err := json.Unmarshal(event.Payload, &order); err != nil {
        return err
    }
    
    // Process order (send confirmation email, update inventory, etc.)
    return nil
}

Distributed Tracing

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

func initTracer() (*trace.TracerProvider, error) {
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        return nil, err
    }
    
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("user-service"),
        )),
    )
    
    otel.SetTracerProvider(tp)
    return tp, nil
}

func main() {
    tp, err := initTracer()
    if err != nil {
        log.Fatal(err)
    }
    defer tp.Shutdown(context.Background())
    
    // gRPC server with tracing
    grpcServer := grpc.NewServer(
        grpc.StatsHandler(otelgrpc.NewServerHandler()),
    )
    
    // gRPC client with tracing
    conn, err := grpc.Dial("localhost:50051",
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    )
}

Interview Questions

What's the difference between gRPC and REST?

Protocol: gRPC uses HTTP/2, REST typically HTTP/1.1
Payload: gRPC uses binary Protobuf, REST uses text (JSON/XML)
Streaming: gRPC has native bidirectional streaming
Type Safety: gRPC has strong contracts, REST relies on documentation
Performance: gRPC is faster due to binary serialization and HTTP/2

What are the four types of gRPC methods?

Unary: Single request, single response
Server streaming: Single request, stream of responses
Client streaming: Stream of requests, single response
Bidirectional streaming: Stream in both directions

How do you handle errors in gRPC?

Use google.golang.org/grpc/status package with standard codes:

codes.NotFound for missing resources
codes.InvalidArgument for bad input
codes.Unauthenticated for auth failures
codes.PermissionDenied for authorization failures
codes.Internal for server errors

What patterns do you use for service-to-service communication?

Circuit Breaker: Prevent cascading failures
Retry with backoff: Handle transient failures
Timeout: Prevent hanging requests
Load balancing: Distribute traffic
Service discovery: Dynamic endpoint resolution
Health checks: Monitor service availability

Summary

Component	Purpose
Protocol Buffers	Define service contracts and messages
gRPC Server	Implement service methods
gRPC Client	Call remote services
Interceptors	Add cross-cutting concerns
Service Discovery	Find service instances
Circuit Breaker	Handle failures gracefully
Health Checks	Monitor service health
Distributed Tracing	Track requests across services

Interview Deep-Dive

Compare gRPC and REST for service-to-service communication. When would you choose each, and what are the hidden operational costs of gRPC?

Strong Answer:

gRPC excels at service-to-service (east-west) traffic: binary protobuf serialization is 5-10x smaller and faster than JSON, HTTP/2 multiplexing avoids head-of-line blocking, bidirectional streaming enables real-time data flows, and code generation from proto files gives you type-safe clients in any language with compile-time contract verification.
REST excels at client-facing (north-south) traffic: every browser, mobile client, and CLI tool speaks HTTP/JSON natively. REST is human-readable, debuggable with curl, and does not require code generation tooling. API gateways, CDNs, and load balancers all understand HTTP/REST natively.
Hidden operational costs of gRPC: you need a protobuf compilation pipeline in CI (protoc + plugins), proto file versioning and backward compatibility management, gRPC debugging is harder (cannot use curl, need grpcurl or Postman with gRPC support), load balancers need gRPC-aware configuration (because gRPC uses HTTP/2 and multiplexes requests on a single connection, naive round-robin at the TCP level does not work — you need L7 load balancing), and browser clients cannot call gRPC directly (you need grpc-web or a REST gateway like grpc-gateway).
My recommendation: use gRPC between internal services where performance and type safety matter. Expose REST/JSON for external APIs and browser clients. Use grpc-gateway to generate a REST facade over your gRPC services when both are needed.

Follow-up: gRPC uses HTTP/2 with persistent connections. How does this affect load balancing, and what pattern do you use to handle it?Because gRPC multiplexes all requests over a single HTTP/2 connection, a TCP-level (L4) load balancer routes all requests from one client to the same server. If you have 3 servers and 3 clients, each client connects to one server, and load distribution depends on which client is busiest. The fix is L7 (application-level) load balancing that inspects individual gRPC requests within the HTTP/2 stream and routes them independently. Alternatively, use client-side load balancing: the gRPC client resolves multiple server addresses (via DNS or service discovery) and distributes requests using round-robin or another policy. In Go, you configure this with grpc.WithDefaultServiceConfig specifying round_robin and a custom resolver that returns multiple addresses. In Kubernetes, use headless services so DNS returns all pod IPs, enabling client-side balancing.

How do gRPC interceptors work in Go? Compare them to HTTP middleware and explain how you would implement logging, authentication, and error recovery.

Strong Answer:

gRPC interceptors are the gRPC equivalent of HTTP middleware. There are two types: unary interceptors (for request-response RPCs) and stream interceptors (for streaming RPCs). Each interceptor has access to the request, the server info (method name, service), and calls the next handler in the chain.
The signature for a unary server interceptor is: func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error). You can inspect/modify the request before calling handler(ctx, req), and inspect/modify the response/error after.
For logging: record the start time, call the handler, then log the method name, duration, and error status. The info.FullMethod gives you the service and method name (like /user.UserService/GetUser).
For authentication: extract the token from gRPC metadata (metadata.FromIncomingContext(ctx)), validate it, and inject claims into the context with context.WithValue. Return status.Error(codes.Unauthenticated, ...) if validation fails. This is analogous to reading the Authorization header in HTTP middleware.
For error recovery: wrap the handler call in a defer func() { if r := recover(); r != nil { ... } }() to catch panics and convert them to status.Error(codes.Internal, ...) instead of crashing the server.
Key difference from HTTP middleware: gRPC interceptors chain with grpc.ChainUnaryInterceptor(), and gRPC uses structured error codes (codes.NotFound, codes.PermissionDenied) instead of HTTP status codes. The error codes are richer and more standardized than HTTP status codes for RPC semantics.

Follow-up: What are the four types of gRPC methods, and when would you use server streaming versus bidirectional streaming?Unary (single request, single response) is the default for most CRUD operations. Server streaming (single request, stream of responses) is for when the server has a large or ongoing result set — think real-time price feeds, log tailing, or paginated results streamed incrementally. Client streaming (stream of requests, single response) is for batch uploads where the client sends many items and gets a summary response. Bidirectional streaming (both sides stream simultaneously) is for real-time interactive protocols like chat, collaborative editing, or game state synchronization. In practice, unary covers 90% of use cases. Server streaming covers most of the rest. Client streaming and bidirectional are rare and significantly more complex to implement correctly (error handling, backpressure, reconnection).

Your microservice makes calls to three downstream services. One of them starts returning errors. How do you prevent this from cascading and taking down your service?

Strong Answer:

Three layers of defense: timeouts, circuit breakers, and bulkheads.
Timeouts: every downstream call must have a context timeout. If service C is slow, the request to C times out after (say) 2 seconds instead of hanging indefinitely. Without timeouts, your goroutines pile up waiting for the slow service, exhausting memory and goroutine capacity.
Circuit breaker: after N consecutive failures to service C (or a failure rate threshold), the circuit opens and subsequent calls fail immediately without even attempting the request. This gives service C time to recover and prevents your service from wasting resources on requests that will fail. After a timeout period, the circuit moves to half-open and probes C with a single request. If it succeeds, the circuit closes.
Bulkheads: isolate the failure domain. If service C’s connection pool is separate from services A and B, exhausting C’s pool does not affect A and B. In Go, this means using separate HTTP clients (with their own connection pools) or separate goroutine pools for each downstream service.
Additionally: implement graceful degradation. If service C is the recommendation engine and it is down, return a static default recommendation set instead of failing the entire request. The user gets a slightly worse experience but the service stays up.
In Go specifically: use errgroup.WithContext so that if one downstream call fails, the context is cancelled and the other concurrent calls are aborted. Combine with context.WithTimeout per call, and sony/gobreaker for circuit breaking.

Follow-up: How do you implement distributed tracing across Go microservices, and why is it essential for debugging?Distributed tracing (using OpenTelemetry + Jaeger/Zipkin) propagates a trace ID across service boundaries. Each service creates spans for its operations and links them to the parent trace. In Go, the gRPC interceptor (or HTTP middleware) extracts the trace context from incoming request headers, creates a child span, and injects the context into outgoing requests. When a request is slow or fails, you look up the trace ID and see the entire call graph: which services were involved, how long each took, and where the bottleneck or error occurred. Without tracing, debugging a problem across 5 microservices requires correlating logs from 5 different systems using timestamps and request IDs. With tracing, you have a single view of the entire request lifecycle. This is not optional for production microservices — it is the only way to understand latency and failure in a distributed system.

Overview

Testing & Code Quality

Crash Courses

AI Engineering

Math for ML - Understanding Linear Algebra

Probability & Statistics for ML

Math for ML - Understanding Calculus

ML Mastery

Deep Learning Mastery

NestJS Mastery

Microservices Mastery

Low Level Design

OOP Concepts

SOLID Principles

Design Patterns

LLD Case Studies

System Design (HLD)

Senior Level (L5+/Staff)

HLD Case Studies

Engineering Fundamentals

DevOps & Operations

Azure Cloud Engineering

AWS Cloud

AWS Monitoring & Observability

AWS Security Services

AWS Serverless

AWS Operations

AWS Advanced

AWS Case Studies

GCP Cloud Engineering

DevOps Tools

Database Engineering

HIPAA Compliance Mastery

Operating Systems

Linux Internals

Distributed Systems

Networking Mastery

Build Your Own X

Go Lang Mastery

C Programming

Classic Research Papers

Distributed System Tools

Documentation Index

​Microservices & gRPC

​gRPC Fundamentals

​Why gRPC?

​Protocol Buffers

​Generating Go Code

​gRPC Server

​Implementing the Service

​gRPC Client

​gRPC Interceptors (Middleware)

​Server Interceptors

​Client Interceptors

​Service Discovery

​Using Consul

​Load Balancing with gRPC

​Health Checks

​Circuit Breaker

​Event-Driven Architecture

​Publishing Events

​Consuming Events

​Distributed Tracing

​Interview Questions

​Summary

​Interview Deep-Dive

Microservices & gRPC

gRPC Fundamentals

Why gRPC?

Protocol Buffers

Generating Go Code

gRPC Server

Implementing the Service

gRPC Client

gRPC Interceptors (Middleware)

Server Interceptors

Client Interceptors

Service Discovery

Using Consul

Load Balancing with gRPC

Health Checks

Circuit Breaker

Event-Driven Architecture

Publishing Events

Consuming Events

Distributed Tracing

Interview Questions

Summary

Interview Deep-Dive