ELK Stack — Indices & Documents

4 minute read

Indices and Documents

Every piece of data in Elasticsearch is stored as a document inside an index. These are the two most fundamental concepts you need to understand.

What Is a Document?

A document is a single JSON record stored in Elasticsearch. Think of it as a row in a database table.

Example — a log event document:

{
  "_index": "nginx-logs-2025.03.15",
  "_id": "abc123",
  "_source": {
    "timestamp": "2025-03-15T10:15:30Z",
    "client_ip": "192.168.1.50",
    "method": "GET",
    "path": "/api/users",
    "status_code": 200,
    "bytes_sent": 1234,
    "user_agent": "Mozilla/5.0",
    "host": "web-server-01"
  }
}

Key parts of a document:

  • _index — which index this document belongs to
  • _id — unique identifier (auto-generated or custom)
  • _source — the actual JSON data you indexed

Documents can contain any valid JSON — strings, numbers, booleans, arrays, nested objects. There’s no fixed schema required (though mappings are recommended).

What Is an Index?

An index is a collection of related documents. Think of it as a database table or a folder that groups documents together.

┌──────────────────────────────────────────────┐
│           Elasticsearch Cluster               │
│                                               │
│  ┌─────────────────────────────────────────┐ │
│  │  Index: nginx-logs-2025.03.15           │ │
│  │                                          │ │
│  │  Doc 1: { status: 200, path: /api/... } │ │
│  │  Doc 2: { status: 500, path: /api/... } │ │
│  │  Doc 3: { status: 200, path: /login  }  │ │
│  │  ... (2,500,000 documents)               │ │
│  └─────────────────────────────────────────┘ │
│                                               │
│  ┌─────────────────────────────────────────┐ │
│  │  Index: app-logs-2025.03.15             │ │
│  │                                          │ │
│  │  Doc 1: { level: ERROR, msg: "NullP.."} │ │
│  │  Doc 2: { level: INFO, msg: "Started"}  │ │
│  │  ... (500,000 documents)                 │ │
│  └─────────────────────────────────────────┘ │
└──────────────────────────────────────────────┘

Naming conventions for log indices:

Pattern Example Use Case
Daily nginx-logs-2025.03.15 High-volume (roll over daily)
Monthly app-logs-2025.03 Medium-volume
Static products Reference data (not time-based)

Daily indices are the most common for logs — they make it easy to manage retention (delete old indices) and optimize performance.

Database Analogy

If you’re coming from a relational database background:

Relational DB Elasticsearch Description
Database Cluster Top-level container
Table Index Collection of related records
Row Document Single record
Column Field Individual data element
Schema Mapping Structure definition
SQL Query DSL Query language

Important difference: In Elasticsearch, all data within an index is stored as flat JSON documents. There are no joins (no SELECT ... JOIN). If you need related data together, you denormalize it into a single document.

Create an Index

$ curl -X PUT -u elastic:changeme123 \
  "http://localhost:9200/my-first-index" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }'

Expected output:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "my-first-index"
}
  • number_of_shards: 1 — one primary shard (fine for learning)
  • number_of_replicas: 0 — no replicas (we only have one node)

Index a Document

$ curl -X POST -u elastic:changeme123 \
  "http://localhost:9200/my-first-index/_doc" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Hello from Elasticsearch!",
    "level": "info",
    "timestamp": "2025-03-15T10:00:00Z"
  }'

Expected output:

{
  "_index": "my-first-index",
  "_id": "random_id_here",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Elasticsearch auto-generated an _id for the document. You can also specify your own:

$ curl -X PUT -u elastic:changeme123 \
  "http://localhost:9200/my-first-index/_doc/my-custom-id" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Document with custom ID",
    "level": "debug"
  }'

Retrieve a Document

$ curl -u elastic:changeme123 \
  "http://localhost:9200/my-first-index/_doc/my-custom-id?pretty"

Expected output:

{
  "_index": "my-first-index",
  "_id": "my-custom-id",
  "_version": 1,
  "found": true,
  "_source": {
    "message": "Document with custom ID",
    "level": "debug"
  }
}

List All Indices

$ curl -u elastic:changeme123 "http://localhost:9200/_cat/indices?v"

Expected output:

health status index          uuid                   pri rep docs.count store.size
green  open   my-first-index xY1234...              1   0          2       8.2kb

Column meanings:

  • health — green (all shards allocated), yellow (replicas missing), red (primary shards missing)
  • pri — number of primary shards
  • rep — number of replica shards
  • docs.count — number of documents

Delete an Index

$ curl -X DELETE -u elastic:changeme123 \
  "http://localhost:9200/my-first-index"

Expected output:

{
  "acknowledged": true
}

⚠️ Warning: Deleting an index permanently removes all its documents. There’s no undo.

Automatic Index Creation

When you index a document into a non-existent index, Elasticsearch creates it automatically:

$ curl -X POST -u elastic:changeme123 \
  "http://localhost:9200/auto-created-index/_doc" \
  -H "Content-Type: application/json" \
  -d '{"message": "This index was created automatically"}'

Elasticsearch creates auto-created-index with default settings (1 shard, 1 replica). In production, you should create indices explicitly with proper settings and mappings.

Index Information

Get detailed information about an index:

$ curl -u elastic:changeme123 \
  "http://localhost:9200/my-first-index?pretty"

This shows settings, mappings, and aliases for the index.

Document Count

$ curl -u elastic:changeme123 \
  "http://localhost:9200/my-first-index/_count"

Expected output:

{
  "count": 2,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  }
}

Quick Reference

Concept Description
Document A single JSON record (like a database row)
Index A collection of documents (like a database table)
_id Unique document identifier (auto-generated or custom)
_source The original JSON data
Daily indices logs-2025.03.15 — common pattern for time-series data
Create index PUT /my-index
Index document POST /my-index/_doc
Get document GET /my-index/_doc/<id>
Delete index DELETE /my-index

Next Step

Now that you understand indices and documents, let’s learn about mappings — how Elasticsearch defines the schema for your data.

You may also enjoy