Data Source Basics

What Are Data Sources?

Data sources allow Terraform configurations to fetch and reference existing infrastructure that may or may not be managed by Terraform. They are read-only and do not create, modify, or delete resources.

Key characteristics:

• Read-only operations
• Query existing infrastructure
• No state lifecycle management
• Results cached during plan phase
• Refreshed on every terraform plan

Example usage:

# Query existing database
data "mycloud_database" "existing" {
  name = "production-db"
}

# Use the data source output
resource "mycloud_app" "api" {
  database_endpoint = data.mycloud_database.existing.endpoint
  database_port     = data.mycloud_database.existing.port
}

output "db_id" {
  value = data.mycloud_database.existing.id
}

Data Sources vs. Resources

Understanding the difference is crucial:

Feature	Data Source	Resource
Declaration	data "type" "name"	resource "type" "name"
Purpose	Query/reference	Create/manage
Operations	Read only	Create, Read, Update, Delete
State Impact	Cached results	Full lifecycle tracking
Modifications	Cannot change infrastructure	Manages infrastructure
Refresh	Every plan	Explicit read
Dependencies	Can depend on resources	Independent
API Calls	GET/Query	POST, PUT, DELETE

Example comparison:

# Data source: Query existing VPC
data "mycloud_vpc" "existing" {
  name = "production-vpc"
}

# Resource: Create new VPC
resource "mycloud_vpc" "new" {
  name = "development-vpc"
  cidr = "10.0.0.0/16"
}

When to Use Data Sources

✅ Use Data Sources When:

1. Referencing Existing Infrastructure

# Infrastructure created outside Terraform
data "mycloud_vpc" "legacy" {
  id = "vpc-12345"
}

resource "mycloud_subnet" "new" {
  vpc_id = data.mycloud_vpc.legacy.id
  cidr   = "10.0.1.0/24"
}

2. Fetching Dynamic Values

# Get latest AMI
data "mycloud_ami" "ubuntu" {
  most_recent = true
  
  filter {
    name   = "name"
    values = ["ubuntu-22.04-*"]
  }
}

resource "mycloud_instance" "web" {
  ami           = data.mycloud_ami.ubuntu.id
  instance_type = "t3.micro"
}

3. Service Discovery

# Find shared database
data "mycloud_database" "shared" {
  tags = {
    Shared = "true"
    Team   = "platform"
  }
}

# Use in application
resource "mycloud_app" "service" {
  database_url = data.mycloud_database.shared.connection_string
}

4. Multi-Account/Multi-Region Scenarios

# Get regions
data "mycloud_regions" "available" {
  all_regions = true
}

# Deploy to each
resource "mycloud_bucket" "regional" {
  for_each = toset(data.mycloud_regions.available.names)
  
  name   = "backup-${each.value}"
  region = each.value
}

5. Getting Account/Provider Information

# Current account details
data "mycloud_account" "current" {}

output "account_id" {
  value = data.mycloud_account.current.id
}

❌ Don’t Use Data Sources When:

1. You Need to Manage Lifecycle

# ❌ Bad: Can't create with data source
data "mycloud_database" "new" {
  name = "new-database"  # Won't create if doesn't exist
}

# ✅ Good: Use resource
resource "mycloud_database" "new" {
  name = "new-database"
}

2. You Need to Modify Resources

# ❌ Bad: Can't update via data source
data "mycloud_database" "db" {
  id = "db-123"
  instance_class = "db.t3.large"  # This won't update the database
}

# ✅ Good: Import and manage as resource
resource "mycloud_database" "db" {
  id             = "db-123"  # Import first
  instance_class = "db.t3.large"
}

Data Source Interfaces

Required Interface

All data sources must implement datasource.DataSource:

import (
    "github.com/hashicorp/terraform-plugin-framework/datasource"
)

type DataSource interface {
    // Metadata sets the data source type name
    Metadata(context.Context, MetadataRequest, *MetadataResponse)
    
    // Schema defines the structure
    Schema(context.Context, SchemaRequest, *SchemaResponse)
    
    // Read performs the query and populates state
    Read(context.Context, ReadRequest, *ReadResponse)
}

Optional Interfaces

1. DataSourceWithConfigure - Receive shared client:

type DataSourceWithConfigure interface {
    DataSource
    Configure(context.Context, ConfigureRequest, *ConfigureResponse)
}

2. DataSourceWithValidateConfig - Validate configuration:

type DataSourceWithValidateConfig interface {
    DataSource
    ValidateConfig(context.Context, ValidateConfigRequest, *ValidateConfigResponse)
}

Basic Data Source Structure

Minimal Implementation

package provider

import (
    "context"
    "fmt"
    
    "github.com/hashicorp/terraform-plugin-framework/datasource"
    "github.com/hashicorp/terraform-plugin-framework/datasource/schema"
    "github.com/hashicorp/terraform-plugin-framework/types"
    "github.com/hashicorp/terraform-plugin-log/tflog"
    
    "github.com/yourorg/terraform-provider-mycloud/internal/client"
)

// Ensure implementation satisfies interfaces
var (
    _ datasource.DataSource              = &DatabaseDataSource{}
    _ datasource.DataSourceWithConfigure = &DatabaseDataSource{}
)

// Data source struct
type DatabaseDataSource struct {
    client *client.Client
}

// Data model
type DatabaseDataSourceModel struct {
    ID       types.String `tfsdk:"id"`
    Name     types.String `tfsdk:"name"`
    Endpoint types.String `tfsdk:"endpoint"`
    Port     types.Int64  `tfsdk:"port"`
    Engine   types.String `tfsdk:"engine"`
    Status   types.String `tfsdk:"status"`
    Tags     types.Map    `tfsdk:"tags"`
}

// Constructor
func NewDatabaseDataSource() datasource.DataSource {
    return &DatabaseDataSource{}
}

// Metadata sets the data source type name
func (d *DatabaseDataSource) Metadata(ctx context.Context, req datasource.MetadataRequest, resp *datasource.MetadataResponse) {
    resp.TypeName = req.ProviderTypeName + "_database"
}

// Schema defines the structure
func (d *DatabaseDataSource) Schema(ctx context.Context, req datasource.SchemaRequest, resp *datasource.SchemaResponse) {
    resp.Schema = schema.Schema{
        MarkdownDescription: "Fetches information about a database",
        
        Attributes: map[string]schema.Attribute{
            "id": schema.StringAttribute{
                Computed:            true,
                MarkdownDescription: "Database identifier",
            },
            "name": schema.StringAttribute{
                Required:            true,
                MarkdownDescription: "Name of the database to look up",
            },
            "endpoint": schema.StringAttribute{
                Computed:            true,
                MarkdownDescription: "Database connection endpoint",
            },
            "port": schema.Int64Attribute{
                Computed:            true,
                MarkdownDescription: "Database port",
            },
            "engine": schema.StringAttribute{
                Computed:            true,
                MarkdownDescription: "Database engine",
            },
            "status": schema.StringAttribute{
                Computed:            true,
                MarkdownDescription: "Current status",
            },
            "tags": schema.MapAttribute{
                Computed:            true,
                ElementType:         types.StringType,
                MarkdownDescription: "Resource tags",
            },
        },
    }
}

// Configure receives shared client
func (d *DatabaseDataSource) Configure(ctx context.Context, req datasource.ConfigureRequest, resp *datasource.ConfigureResponse) {
    if req.ProviderData == nil {
        return
    }
    
    client, ok := req.ProviderData.(*client.Client)
    if !ok {
        resp.Diagnostics.AddError(
            "Unexpected Data Source Configure Type",
            fmt.Sprintf("Expected *client.Client, got: %T", req.ProviderData),
        )
        return
    }
    
    d.client = client
}

// Read performs the query
func (d *DatabaseDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    var config DatabaseDataSourceModel
    
    // Get configuration
    diags := req.Config.Get(ctx, &config)
    resp.Diagnostics.Append(diags...)
    if resp.Diagnostics.HasError() {
        return
    }
    
    name := config.Name.ValueString()
    
    tflog.Debug(ctx, "Reading database data source", map[string]any{
        "name": name,
    })
    
    // Query API
    db, err := d.client.GetDatabaseByName(ctx, name)
    if err != nil {
        resp.Diagnostics.AddError(
            "Failed to Read Database",
            fmt.Sprintf("Could not find database '%s': %s", name, err.Error()),
        )
        return
    }
    
    // Set all attributes
    config.ID = types.StringValue(db.ID)
    config.Endpoint = types.StringValue(db.Endpoint)
    config.Port = types.Int64Value(int64(db.Port))
    config.Engine = types.StringValue(db.Engine)
    config.Status = types.StringValue(db.Status)
    
    // Convert tags
    if len(db.Tags) > 0 {
        tagsMap := make(map[string]attr.Value)
        for k, v := range db.Tags {
            tagsMap[k] = types.StringValue(v)
        }
        mapValue, _ := types.MapValue(types.StringType, tagsMap)
        config.Tags = mapValue
    } else {
        config.Tags = types.MapNull(types.StringType)
    }
    
    // Save to state
    diags = resp.State.Set(ctx, &config)
    resp.Diagnostics.Append(diags...)
    
    tflog.Info(ctx, "Successfully read database data source", map[string]any{
        "id": db.ID,
    })
}

Registering Data Sources

Register in provider’s DataSources method:

func (p *MyCloudProvider) DataSources(ctx context.Context) []func() datasource.DataSource {
    return []func() datasource.DataSource{
        NewDatabaseDataSource,
        NewVPCDataSource,
        NewAMIDataSource,
        // ... other data sources
    }
}

Data Source Lifecycle

Plan Phase

1. Terraform parses configuration
2. Identifies data source blocks
3. Calls Read() for each data source
4. Caches results for use in plan

Apply Phase

1. Uses cached data source results
2. Does NOT call Read() again
3. Data sources evaluated before resources

Important: Data sources are read during plan, not apply.

data "mycloud_ami" "latest" {
  most_recent = true
}

resource "mycloud_instance" "web" {
  ami = data.mycloud_ami.latest.id  # Uses cached value from plan
}

Common Patterns

1. Lookup by ID

func (d *DatabaseDataSource) Schema(ctx context.Context, req datasource.SchemaRequest, resp *datasource.SchemaResponse) {
    resp.Schema = schema.Schema{
        Attributes: map[string]schema.Attribute{
            "id": schema.StringAttribute{
                Required: true,
                MarkdownDescription: "Database ID to look up",
            },
            "name": schema.StringAttribute{
                Computed: true,
            },
            "endpoint": schema.StringAttribute{
                Computed: true,
            },
        },
    }
}

func (d *DatabaseDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    var config DatabaseDataSourceModel
    req.Config.Get(ctx, &config)
    
    db, err := d.client.GetDatabase(ctx, config.ID.ValueString())
    if err != nil {
        resp.Diagnostics.AddError("Read Failed", err.Error())
        return
    }
    
    config.Name = types.StringValue(db.Name)
    config.Endpoint = types.StringValue(db.Endpoint)
    
    resp.State.Set(ctx, config)
}

2. Lookup by Name

func (d *DatabaseDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    var config DatabaseDataSourceModel
    req.Config.Get(ctx, &config)
    
    name := config.Name.ValueString()
    
    // Search by name
    dbs, err := d.client.ListDatabases(ctx, &client.ListOptions{
        Filter: fmt.Sprintf("name eq '%s'", name),
    })
    
    if err != nil {
        resp.Diagnostics.AddError("Query Failed", err.Error())
        return
    }
    
    if len(dbs) == 0 {
        resp.Diagnostics.AddError(
            "Database Not Found",
            fmt.Sprintf("No database found with name: %s", name),
        )
        return
    }
    
    if len(dbs) > 1 {
        resp.Diagnostics.AddError(
            "Multiple Databases Found",
            fmt.Sprintf("Found %d databases with name '%s'. Name must be unique", len(dbs), name),
        )
        return
    }
    
    db := dbs[0]
    config.ID = types.StringValue(db.ID)
    config.Endpoint = types.StringValue(db.Endpoint)
    
    resp.State.Set(ctx, config)
}

3. Lookup by Tags

type DatabaseDataSourceModel struct {
    ID       types.String `tfsdk:"id"`
    Name     types.String `tfsdk:"name"`
    Tags     types.Map    `tfsdk:"tags"`
    Endpoint types.String `tfsdk:"endpoint"`
}

func (d *DatabaseDataSource) Schema(ctx context.Context, req datasource.SchemaRequest, resp *datasource.SchemaResponse) {
    resp.Schema = schema.Schema{
        Attributes: map[string]schema.Attribute{
            "tags": schema.MapAttribute{
                Required:            true,
                ElementType:         types.StringType,
                MarkdownDescription: "Tags to filter by (all must match)",
            },
            "id": schema.StringAttribute{
                Computed: true,
            },
            "name": schema.StringAttribute{
                Computed: true,
            },
            "endpoint": schema.StringAttribute{
                Computed: true,
            },
        },
    }
}

func (d *DatabaseDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    var config DatabaseDataSourceModel
    req.Config.Get(ctx, &config)
    
    var filterTags map[string]string
    config.Tags.ElementsAs(ctx, &filterTags, false)
    
    // Find databases matching all tags
    dbs, err := d.client.FindDatabasesByTags(ctx, filterTags)
    
    if err != nil {
        resp.Diagnostics.AddError("Query Failed", err.Error())
        return
    }
    
    if len(dbs) == 0 {
        resp.Diagnostics.AddError(
            "No Database Found",
            fmt.Sprintf("No database found matching tags: %v", filterTags),
        )
        return
    }
    
    if len(dbs) > 1 {
        resp.Diagnostics.AddError(
            "Multiple Databases Found",
            fmt.Sprintf("Found %d databases matching tags. Use more specific tags", len(dbs)),
        )
        return
    }
    
    db := dbs[0]
    config.ID = types.StringValue(db.ID)
    config.Name = types.StringValue(db.Name)
    config.Endpoint = types.StringValue(db.Endpoint)
    
    resp.State.Set(ctx, config)
}

Error Handling

Not Found Errors

func (d *DatabaseDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
    var config DatabaseDataSourceModel
    req.Config.Get(ctx, &config)
    
    db, err := d.client.GetDatabaseByName(ctx, config.Name.ValueString())
    
    if err != nil {
        if client.IsNotFoundError(err) {
            resp.Diagnostics.AddError(
                "Database Not Found",
                fmt.Sprintf(
                    "Could not find database with name '%s'.\n\n"+
                    "Please check:\n"+
                    "  • Database name is correct\n"+
                    "  • Database exists in the configured region\n"+
                    "  • API credentials have permission to read databases",
                    config.Name.ValueString(),
                ),
            )
            return
        }
        
        resp.Diagnostics.AddError(
            "Failed to Query Database",
            fmt.Sprintf("API error: %s", err.Error()),
        )
        return
    }
    
    // Set state...
}

Multiple Results

if len(dbs) > 1 {
    resp.Diagnostics.AddError(
        "Multiple Databases Found",
        fmt.Sprintf(
            "Found %d databases matching the criteria.\n\n"+
            "Please refine your search by:\n"+
            "  • Using a more specific name\n"+
            "  • Adding more tag filters\n"+
            "  • Using the database ID instead\n\n"+
            "Found databases: %s",
            len(dbs),
            formatDatabaseList(dbs),
        ),
    )
    return
}

Best Practices

1. Clear Attribute Documentation

"name": schema.StringAttribute{
    Required: true,
    MarkdownDescription: "Exact name of the database to look up. " +
        "Lookup is case-sensitive.",
},

2. Descriptive Error Messages

resp.Diagnostics.AddError(
    "Database Not Found",
    fmt.Sprintf(
        "No database found with name '%s' in region '%s'.\n\n"+
        "Verify the database exists and you have read permissions.",
        name, region,
    ),
)

3. Log Debug Information

tflog.Debug(ctx, "Querying database", map[string]any{
    "name":   name,
    "region": region,
})

4. Validate Configuration

func (d *DatabaseDataSource) ValidateConfig(ctx context.Context, req datasource.ValidateConfigRequest, resp *datasource.ValidateConfigResponse) {
    var config DatabaseDataSourceModel
    resp.Diagnostics.Append(req.Config.Get(ctx, &config)...)
    
    // Require at least one identifier
    if config.ID.IsNull() && config.Name.IsNull() {
        resp.Diagnostics.AddError(
            "Missing Required Attribute",
            "Either 'id' or 'name' must be specified",
        )
    }
    
    // Don't allow both
    if !config.ID.IsNull() && !config.Name.IsNull() {
        resp.Diagnostics.AddError(
            "Conflicting Attributes",
            "Only one of 'id' or 'name' can be specified",
        )
    }
}

5. Set All Computed Attributes

// ✅ Good: Set all attributes
config.ID = types.StringValue(db.ID)
config.Name = types.StringValue(db.Name)
config.Endpoint = types.StringValue(db.Endpoint)
config.Port = types.Int64Value(int64(db.Port))

// ❌ Bad: Leaving attributes unset
config.ID = types.StringValue(db.ID)
// Forgot to set other attributes!

Key Takeaways

• ✅ Data sources are read-only and query existing infrastructure
• ✅ Implement datasource.DataSource interface (Metadata, Schema, Read)
• ✅ Use DataSourceWithConfigure to receive shared client
• ✅ All fetched attributes should be Computed: true
• ✅ Search criteria are Required or Optional
• ✅ Handle not-found errors with clear messages
• ✅ Handle multiple result scenarios appropriately
• ✅ Data sources are evaluated during plan phase
• ✅ Results are cached and reused in apply phase
• ✅ Register data sources in provider’s DataSources() method

What’s Next?

Now let’s dive deeper into designing effective data source schemas!

Next: Data Source Schema →

---

Previous: ← Data Sources Overview

Last updated: June 30, 2026