config/kibana.md

Purpose

Create a small fake application log environment with realistic log entries, use Kibana's Discover interface to practise KQL (Kibana Query Language). By the end, you'll have:

1. A test Elasticsearch index containing application logs.
2. Different log levels: INFO, WARN, ERROR, DEBUG.
3. Different application services.
4. HTTP request and response logs.
5. User login and authentication logs.
6. Sample KQL queries to practise searching, filtering, and troubleshooting.

We'll use a fictional application called myapp.

Folow the following steps,

Step 1. Create test log data

You can insert test data directly into Elasticsearch using the Kibana Dev Tools console.

Open Kibana:

http://localhost:5601

Navigate to:

Management → Dev Tools → Console

Paste the following command.

Create test logs

POST /_bulk
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:00:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User john successfully logged in","user":{"name":"john","id":"1001"},"source_ip":"192.168.1.10","event_type":"login","status":"success"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:01:00Z","level":"INFO","service":"web-service","host":"server02","message":"HTTP request received","http":{"method":"GET","status_code":200,"url":"/api/products"},"response_time":120,"source_ip":"192.168.1.10"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:02:00Z","level":"WARN","service":"payment-service","host":"server03","message":"Payment processing took longer than expected","response_time":3500,"transaction_id":"TX10001","amount":150.50,"currency":"USD"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:03:00Z","level":"ERROR","service":"payment-service","host":"server03","message":"Payment transaction failed","error":{"type":"PaymentGatewayException","message":"Connection timeout"},"transaction_id":"TX10002","amount":250.00}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:04:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User alice successfully logged in","user":{"name":"alice","id":"1002"},"source_ip":"192.168.1.20","event_type":"login","status":"success"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:05:00Z","level":"ERROR","service":"auth-service","host":"server01","message":"Failed login attempt","user":{"name":"admin","id":"0"},"source_ip":"203.0.113.50","event_type":"login","status":"failed"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:06:00Z","level":"DEBUG","service":"web-service","host":"server02","message":"Database connection pool initialized","connection_pool_size":20}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:07:00Z","level":"WARN","service":"web-service","host":"server02","message":"High response time detected","http":{"method":"GET","status_code":504,"url":"/api/orders"},"response_time":5000}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:08:00Z","level":"INFO","service":"inventory-service","host":"server04","message":"Product inventory updated","product_id":"P1001","quantity":150}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:09:00Z","level":"ERROR","service":"inventory-service","host":"server04","message":"Database connection failed","error":{"type":"DatabaseException","message":"Connection refused"}}

Click Run.

You should receive a response indicating that 11 documents were indexed successfully.

Note: If your Elasticsearch container requires authentication, Kibana Dev Tools normally handles the connection automatically.

---

Step 2. Create a Data View in Kibana

Elasticsearch stores the logs, but Kibana needs a Data View to display them in Discover.

1. Open Kibana.
2. Navigate to Stack Management.
3. Select Data Views.
4. Click Create data view.
5. Enter:

Setting	Value
Name	My Application Logs
Index pattern	logs-lab*
Timestamp field	@timestamp

6. Click Save data view.

Now navigate to:

Analytics → Discover

Select My Application Logs.

You should see your 11 test log entries.

Important: Your sample timestamps are in UTC. In Discover, change the time picker (top-right) to Last 24 hours if you don't immediately see the logs.

Step 3. Practise KQL searches

Now comes the interesting part!

In Discover, locate the KQL search bar at the top. Try these queries one by one.

Exercise 1: Search by log level

Find all ERROR logs:

level: "ERROR"

Find all INFO logs:

level: "INFO"

Find all WARNING logs:

level: "WARN"

Exercise 2: Search by application service

Find logs generated by the payment service:

service: "payment-service"

Find logs generated by the authentication service:

service: "auth-service"

Find logs generated by either payment or inventory services:

service: ("payment-service" OR "inventory-service")

Exercise 3: Combine multiple conditions

Find ERROR logs generated by the payment service:

level: "ERROR" AND service: "payment-service"

Find all ERROR or WARN logs:

level: ("ERROR" OR "WARN")

Find failed login attempts:

event_type: "login" AND status: "failed"

Find logs from a particular IP address:

source_ip: "203.0.113.50"

Exercise 4: Search using numeric conditions

Find requests taking longer than 1 second:

response_time > 1000

Find requests returning HTTP 504:

http.status_code: 504

Find requests taking longer than 1 second and returning HTTP errors:

response_time > 1000 AND http.status_code >= 400

Find transactions with amounts greater than 200:

amount > 200

Exercise 5: Search for specific text

Find logs containing the word timeout:

message: *timeout*

Find logs containing database:

message: *database*

Find logs containing either failed or failure:

message: (*failed* OR *failure*)

Exercise 6: Search for missing fields

Find logs without a response time:

NOT response_time: *

Find logs containing an error field:

error.type: *

Find logs containing a transaction ID:

transaction_id: *

---

Step 4. Practise more realistic troubleshooting scenarios

Now let's simulate some real-world scenarios that you might encounter when monitoring production systems.

Scenario	KQL Query
Find all application errors	level: "ERROR"
Find payment failures	service: "payment-service" AND level: "ERROR"
Find failed user authentication	event_type: "login" AND status: "failed"
Find slow HTTP requests	response_time >= 3000
Find HTTP server errors	http.status_code >= 500
Find database problems	error.type: "DatabaseException"
Find logs from server03	host: "server03"
Find all logs except DEBUG	NOT level: "DEBUG"
Find payment errors or warnings	service: "payment-service" AND level: ("ERROR" OR "WARN")

These are useful patterns to understand because production troubleshooting often involves combining multiple conditions.

---

Step 5. Practise using the Discover interface

Besides writing KQL, I recommend practising these Kibana features:

1. Add fields to the table

In Discover, expand a log entry and add these fields to your displayed columns:

• @timestamp
• level
• service
• host
• message
• response_time

2. Sort logs

Click the @timestamp column to sort logs by time.

3. Filter using field values

Expand a document and click a field value to add it as a filter.

For example, clicking level: ERROR will automatically create a filter.

4. Save searches

Save your KQL searches so you can reuse them later.

For example:

• All Application Errors
• Payment Service Errors
• Failed Login Attempts
• Slow HTTP Requests

Step 6. Understand KQL vs Elasticsearch SQL

One important thing to remember:

KQL is not SQL.

KQL is primarily a filtering language. It helps you find documents matching particular conditions.

For example:

level: "ERROR" AND service: "payment-service"

If you want to perform SQL-like operations such as:

• GROUP BY
• COUNT
• AVG
• SUM
• ORDER BY

You should explore ES|QL (Elasticsearch Query Language) as well.

For example, ES|QL can perform aggregations:

FROM logs-lab
| WHERE level == "ERROR"
| STATS error_count = COUNT(*) BY service
| SORT error_count DESC

This counts errors grouped by application service.

---

Advanced topics

I suggest learning Elasticsearch and Kibana in this order:

Level	Topic	What to learn
1	Discover	Search and explore logs
2	KQL	Filtering and searching
3	ES|QL	Aggregation and data analysis
4	Visualize	Create charts and dashboards
5	Dashboard	Build monitoring dashboards
6	Alerts	Configure alerts for ERROR logs
7	Ingestion	Use Logstash, Beats or Elastic Agent
8	Advanced	Index lifecycle management and Elasticsearch architecture

One particularly useful next exercise: Create a dashboard showing:

• Total log count
• ERROR count
• ERROR count grouped by service
• HTTP response time
• Failed login attempts over time

This will give you a practical introduction to how Elasticsearch and Kibana are used in real production environments.