Troubleshooting Lab

7 minute read

Lab Overview

Apply debugging and troubleshooting skills to realistic scenarios you’ll encounter in provider development.

Exercise 1: State Drift Investigation

Scenario

Users report that terraform plan shows changes even though they haven’t modified their configuration.

$ terraform plan
Terraform will perform the following actions:

  # cachecloud_cache.example will be updated in-place
  ~ resource "cachecloud_cache" "example" {
      ~ tags = {
          - "Environment" = "production"
          + "Environment" = "Production"
        }
    }

Investigation Steps

Step 1: Enable Logging

export TF_LOG=TRACE
export TF_LOG_PATH=terraform.log
terraform plan

Step 2: Examine Logs

# Look for Read operation
grep "CacheResource.Read" terraform.log

# Check what API returned
grep "API response" terraform.log

Step 3: Identify Root Cause

// Found in logs: API returns "Production" but state has "production"
// Problem: Case inconsistency

// Investigation in code
func (r *CacheResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
    // API response
    cache, err := r.client.GetCache(ctx, state.ID.ValueString())
    
    // Direct assignment - no normalization!
    state.Tags = cache.Tags  // Problem here
    
    resp.Diagnostics.Append(resp.State.Set(ctx, &state)...)
}

Step 4: Fix

func (r *CacheResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
    cache, err := r.client.GetCache(ctx, state.ID.ValueString())
    
    // Normalize tags before setting state
    normalizedTags := make(map[string]string)
    for k, v := range cache.Tags {
        normalizedTags[k] = strings.ToLower(v)
    }
    
    // Convert to types.Map
    tagValues := make(map[string]attr.Value)
    for k, v := range normalizedTags {
        tagValues[k] = types.StringValue(v)
    }
    state.Tags, _ = types.MapValue(types.StringType, tagValues)
    
    resp.Diagnostics.Append(resp.State.Set(ctx, &state)...)
}

Step 5: Verify

# Test the fix
go test -run TestCacheResource_Read_TagsNormalization

# Verify with actual provider
terraform plan
# No changes shown

Exercise 2: Type Conversion Panic

Scenario

Provider crashes with panic during apply:

$ terraform apply

panic: interface conversion: interface {} is []interface {}, not string

goroutine 1 [running]:
github.com/example/terraform-provider-cachecloud/internal/provider.(*CacheResource).Create(...)

Investigation

Step 1: Reproduce

resource "cachecloud_cache" "test" {
  name       = "test-cache"
  subnet_ids = ["subnet-a", "subnet-b"]  # This causes panic
}

Step 2: Debug

# Run with debugger
dlv debug ./main.go -- --debug

# Set breakpoint at panic location
(dlv) break resource_cache.go:45
(dlv) continue

Step 3: Inspect Variables

(dlv) print plan.SubnetIDs
github.com/hashicorp/terraform-plugin-framework/types/basetypes.ListValue {...}

(dlv) whatis plan.SubnetIDs
Type: github.com/hashicorp/terraform-plugin-framework/types/basetypes.ListValue

# Try to access directly (this will panic)
(dlv) print plan.SubnetIDs.([]string)
panic: interface conversion

Step 4: Identify Issue

// Problematic code
func (r *CacheResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
    var plan CacheResourceModel
    resp.Diagnostics.Append(req.Plan.Get(ctx, &plan)...)
    
    // ❌ Wrong: Can't directly cast types.List to []string
    subnets := plan.SubnetIDs.([]string)  // PANIC!
}

Step 5: Fix

func (r *CacheResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
    var plan CacheResourceModel
    resp.Diagnostics.Append(req.Plan.Get(ctx, &plan)...)
    
    // ✅ Correct: Use ElementsAs
    var subnets []string
    resp.Diagnostics.Append(plan.SubnetIDs.ElementsAs(ctx, &subnets, false)...)
    if resp.Diagnostics.HasError() {
        return
    }
    
    // Now use subnets safely
    cache, err := r.client.CreateCache(ctx, &CreateCacheInput{
        SubnetIDs: subnets,
    })
}

Step 6: Test

func TestCacheResource_Create_SubnetIDs(t *testing.T) {
    resource.Test(t, resource.TestCase{
        ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
        Steps: []resource.TestStep{
            {
                Config: `
resource "cachecloud_cache" "test" {
  name       = "test"
  subnet_ids = ["subnet-a", "subnet-b"]
}
`,
                Check: resource.ComposeTestCheckFunc(
                    resource.TestCheckResourceAttr("cachecloud_cache.test", "subnet_ids.#", "2"),
                ),
            },
        },
    })
}

Exercise 3: API Rate Limiting

Scenario

Large-scale deployments fail with rate limit errors:

Error: 429 Too Many Requests

Error: Failed to create cache: rate limit exceeded

Investigation

Step 1: Analyze Pattern

# Check logs for API call frequency
grep "HTTP request" terraform.log | wc -l
# 1000 requests in 10 seconds = 100 req/s

# API limit is 50 req/s

Step 2: Identify Bottleneck

// Current code makes individual calls
func (d *CachesDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    var caches []*Cache
    
    // Gets all cache IDs
    ids, err := d.client.ListCacheIDs(ctx)  // 1 call
    
    // ❌ Individual call for each cache
    for _, id := range ids {  // 999 more calls!
        cache, err := d.client.GetCache(ctx, id)
        caches = append(caches, cache)
    }
}

Step 3: Implement Solution

// Add rate limiting
import "golang.org/x/time/rate"

type Client struct {
    httpClient *http.Client
    limiter    *rate.Limiter
}

func NewClient(apiKey string, rps float64) *Client {
    return &Client{
        httpClient: &http.Client{},
        limiter:    rate.NewLimiter(rate.Limit(rps), 10),
    }
}

func (c *Client) Do(req *http.Request) (*http.Response, error) {
    // Wait for rate limiter
    if err := c.limiter.Wait(req.Context()); err != nil {
        return nil, err
    }
    
    return c.httpClient.Do(req)
}

// Add batching
func (c *Client) BatchGetCaches(ctx context.Context, ids []string) ([]*Cache, error) {
    // Single API call for multiple caches
    body, _ := json.Marshal(map[string]interface{}{
        "ids": ids,
    })
    
    req, _ := http.NewRequestWithContext(ctx, "POST", c.baseURL+"/caches/batch", bytes.NewReader(body))
    resp, err := c.Do(req)
    if err != nil {
        return nil, err
    }
    
    var caches []*Cache
    json.NewDecoder(resp.Body).Decode(&caches)
    return caches, nil
}

Step 4: Update Data Source

func (d *CachesDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    ids, err := d.client.ListCacheIDs(ctx)  // 1 call
    
    // ✅ Batch call
    caches, err := d.client.BatchGetCaches(ctx, ids)  // 1 more call
    
    // Total: 2 calls instead of 1000!
}

Step 5: Verify

# Test with large deployment
terraform plan

# Check logs
grep "HTTP request" terraform.log | wc -l
# 2 requests - success!

Exercise 4: Memory Leak

Scenario

Provider memory usage grows continuously during long-running operations.

Investigation

Step 1: Profile Memory

// Add pprof endpoint
import _ "net/http/pprof"

func main() {
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    
    // ... rest of main
}

Step 2: Capture Heap Dumps

# Initial heap dump
curl http://localhost:6060/debug/pprof/heap > heap1.prof

# Run operations
terraform apply

# Final heap dump
curl http://localhost:6060/debug/pprof/heap > heap2.prof

# Compare
go tool pprof -base heap1.prof heap2.prof
(pprof) top
    flat  flat%   sum%        cum   cum%
 512.50MB 85.42% 85.42%   512.50MB 85.42%  (*Client).GetCache
  87.50MB 14.58%   100%    87.50MB 14.58%  (*CacheDataSource).Read

Step 3: Identify Leak

// Problem: Unbounded cache
type Client struct {
    cache map[string]*Cache  // Never cleaned!
}

func (c *Client) GetCache(ctx context.Context, id string) (*Cache, error) {
    // Check cache
    if cached, found := c.cache[id]; found {
        return cached, nil
    }
    
    // Fetch and cache
    cache, err := c.fetchCache(ctx, id)
    if err != nil {
        return nil, err
    }
    
    // ❌ Store forever - memory leak!
    c.cache[id] = cache
    
    return cache, nil
}

Step 4: Fix

import "github.com/hashicorp/golang-lru/v2"

type Client struct {
    cache *lru.Cache[string, *cacheEntry]
}

type cacheEntry struct {
    cache     *Cache
    expiresAt time.Time
}

func NewClient(apiKey string) *Client {
    cache, _ := lru.New[string, *cacheEntry](100)  // Bounded size
    
    return &Client{
        cache: cache,
    }
}

func (c *Client) GetCache(ctx context.Context, id string) (*Cache, error) {
    // Check cache
    if entry, found := c.cache.Get(id); found {
        if time.Now().Before(entry.expiresAt) {
            return entry.cache, nil
        }
        c.cache.Remove(id)  // Expired
    }
    
    // Fetch
    cache, err := c.fetchCache(ctx, id)
    if err != nil {
        return nil, err
    }
    
    // Store with TTL
    c.cache.Add(id, &cacheEntry{
        cache:     cache,
        expiresAt: time.Now().Add(5 * time.Minute),
    })
    
    return cache, nil
}

Step 5: Verify Fix

# Profile again
curl http://localhost:6060/debug/pprof/heap > heap3.prof

# Run operations
terraform apply

curl http://localhost:6060/debug/pprof/heap > heap4.prof

# Compare - should be stable now
go tool pprof -base heap3.prof heap4.prof
(pprof) top
# Memory stable

Exercise 5: Concurrency Issues

Scenario

Intermittent panics in parallel operations:

panic: concurrent map writes

goroutine 42 [running]:
github.com/example/terraform-provider-cachecloud/internal/client.(*Client).recordMetric(...)

Investigation

Step 1: Enable Race Detector

go test -race ./internal/client/
==================
WARNING: DATA RACE
Write at 0x00c000120000 by goroutine 7:
  github.com/example/terraform-provider-cachecloud/internal/client.(*Client).recordMetric()
      /path/to/client.go:42 +0x123

Previous write at 0x00c000120000 by goroutine 6:
  github.com/example/terraform-provider-cachecloud/internal/client.(*Client).recordMetric()
      /path/to/client.go:42 +0x123
==================

Step 2: Identify Issue

type Client struct {
    metrics map[string]int  // ❌ Not thread-safe!
}

func (c *Client) recordMetric(key string) {
    // ❌ Concurrent map access - race condition!
    c.metrics[key]++
}

func (c *Client) BatchFetch(ctx context.Context, ids []string) {
    var wg sync.WaitGroup
    for _, id := range ids {
        wg.Add(1)
        go func(cacheID string) {
            defer wg.Done()
            c.GetCache(ctx, cacheID)
            c.recordMetric("get_cache")  // ❌ Concurrent writes!
        }(id)
    }
    wg.Wait()
}

Step 3: Fix

import "sync"

type Client struct {
    metrics sync.Map  // ✅ Thread-safe map
    // Or use mutex
    metricsLock sync.RWMutex
    metricsData map[string]int
}

// Option 1: sync.Map
func (c *Client) recordMetric(key string) {
    val, _ := c.metrics.LoadOrStore(key, new(int64))
    atomic.AddInt64(val.(*int64), 1)
}

// Option 2: Mutex
func (c *Client) recordMetric(key string) {
    c.metricsLock.Lock()
    defer c.metricsLock.Unlock()
    c.metricsData[key]++
}

Step 4: Test

func TestConcurrentAccess(t *testing.T) {
    client := NewClient("test-key")
    
    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            client.recordMetric("test")
        }()
    }
    
    wg.Wait()
    
    // Verify count
    count := client.getMetric("test")
    assert.Equal(t, 100, count)
}

Step 5: Verify

# Run with race detector
go test -race ./...
# No warnings

Troubleshooting Checklist

Before Debugging

Can you reproduce the issue consistently?
Do you have the full error message?
Do you have the Terraform configuration?
Do you have provider logs?
What version of the provider?
What version of Terraform?

During Debugging

After Fix

Write test to prevent regression
Update documentation
Check for similar issues elsewhere
Profile to verify fix
Add defensive code if needed

Key Takeaways

✅ Reproduce issues systematically
✅ Use logging and debugger effectively
✅ Check for common patterns (state, types, concurrency)
✅ Profile performance problems
✅ Write tests for bug fixes
✅ Document troubleshooting steps

Congratulations!

You’ve completed Module 15 and gained practical debugging skills!

Next Module: Production Best Practices →

Previous: ← Common Issues
Next Module: Production Best Practices →

Last updated: July 2, 2026

Share on

Twitter Facebook LinkedIn

Troubleshooting Lab

Lab Overview

Exercise 1: State Drift Investigation

Scenario

Investigation Steps

Exercise 2: Type Conversion Panic

Scenario

Investigation

Exercise 3: API Rate Limiting

Scenario

Investigation

Exercise 4: Memory Leak

Scenario

Investigation

Exercise 5: Concurrency Issues

Scenario

Investigation

Troubleshooting Checklist

Before Debugging

During Debugging

After Fix

Key Takeaways

Congratulations!

Share on

You may also enjoy

AWS EKS Karpenter: Intelligent Kubernetes Node Autoscaling and Provisioning

Terraform Provider Development: Building Custom Providers with the Plugin Framework

EKS IRSA and Pod Identity: Fine-Grained AWS Permissions Without Static Keys

Writing Resume: Quantifying Infrastructure Impact with Metrics and Cost Savings