diff --git a/kibana.md b/kibana.md new file mode 100644 index 0000000..b3029b3 --- /dev/null +++ b/kibana.md @@ -0,0 +1,344 @@ + +# Purpose +Create a small **fake application log environment** with realistic log entries, use Kibana's Discover interface to practise KQL (Kibana Query Language). By the end, you'll have: +1. A test Elasticsearch index containing application logs. +2. Different log levels: INFO, WARN, ERROR, DEBUG. +3. Different application services. +4. HTTP request and response logs. +5. User login and authentication logs. +6. Sample KQL queries to practise searching, filtering, and troubleshooting. + +We'll use a fictional application called `myapp`. + +Folow the following steps, + +# Step 1. Create test log data + +You can insert test data directly into Elasticsearch using the Kibana Dev Tools console. + +Open Kibana: + +`http://localhost:5601` + +Navigate to: + +**Management → Dev Tools → Console** + +Paste the following command. + +### Create test logs + +```http +POST /_bulk +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:00:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User john successfully logged in","user":{"name":"john","id":"1001"},"source_ip":"192.168.1.10","event_type":"login","status":"success"} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:01:00Z","level":"INFO","service":"web-service","host":"server02","message":"HTTP request received","http":{"method":"GET","status_code":200,"url":"/api/products"},"response_time":120,"source_ip":"192.168.1.10"} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:02:00Z","level":"WARN","service":"payment-service","host":"server03","message":"Payment processing took longer than expected","response_time":3500,"transaction_id":"TX10001","amount":150.50,"currency":"USD"} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:03:00Z","level":"ERROR","service":"payment-service","host":"server03","message":"Payment transaction failed","error":{"type":"PaymentGatewayException","message":"Connection timeout"},"transaction_id":"TX10002","amount":250.00} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:04:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User alice successfully logged in","user":{"name":"alice","id":"1002"},"source_ip":"192.168.1.20","event_type":"login","status":"success"} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:05:00Z","level":"ERROR","service":"auth-service","host":"server01","message":"Failed login attempt","user":{"name":"admin","id":"0"},"source_ip":"203.0.113.50","event_type":"login","status":"failed"} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:06:00Z","level":"DEBUG","service":"web-service","host":"server02","message":"Database connection pool initialized","connection_pool_size":20} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:07:00Z","level":"WARN","service":"web-service","host":"server02","message":"High response time detected","http":{"method":"GET","status_code":504,"url":"/api/orders"},"response_time":5000} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:08:00Z","level":"INFO","service":"inventory-service","host":"server04","message":"Product inventory updated","product_id":"P1001","quantity":150} +{"index":{"_index":"logs-lab"}} +{"@timestamp":"2026-11-09T01:09:00Z","level":"ERROR","service":"inventory-service","host":"server04","message":"Database connection failed","error":{"type":"DatabaseException","message":"Connection refused"}} +``` + +Click **Run**. + +You should receive a response indicating that 11 documents were indexed successfully. + +Note: If your Elasticsearch container requires authentication, Kibana Dev Tools normally handles the connection automatically. + +--- + +# Step 2. Create a Data View in Kibana + +Elasticsearch stores the logs, but Kibana needs a Data View to display them in Discover. + +1. Open Kibana. +2. Navigate to **Stack Management**. +3. Select **Data Views**. +4. Click **Create data view**. +5. Enter: + +| Setting | Value | +|---|---| +| Name | My Application Logs | +| Index pattern | `logs-lab*` | +| Timestamp field | `@timestamp` | + +6. Click **Save data view**. + +Now navigate to: + +**Analytics → Discover** + +Select `My Application Logs`. + +You should see your 11 test log entries. + +**Important:** Your sample timestamps are in UTC. In Discover, change the time picker (top-right) to **Last 24 hours** if you don't immediately see the logs. + +# Step 3. Practise KQL searches + +Now comes the interesting part! + +In Discover, locate the KQL search bar at the top. Try these queries one by one. + +### Exercise 1: Search by log level + +Find all ERROR logs: + +```kql +level: "ERROR" +``` + +Find all INFO logs: + +```kql +level: "INFO" +``` + +Find all WARNING logs: + +```kql +level: "WARN" +``` + +### Exercise 2: Search by application service + +Find logs generated by the payment service: + +```kql +service: "payment-service" +``` + +Find logs generated by the authentication service: + +```kql +service: "auth-service" +``` + +Find logs generated by either payment or inventory services: + +```kql +service: ("payment-service" OR "inventory-service") +``` + +### Exercise 3: Combine multiple conditions + +Find ERROR logs generated by the payment service: + +```kql +level: "ERROR" AND service: "payment-service" +``` + +Find all ERROR or WARN logs: + +```kql +level: ("ERROR" OR "WARN") +``` + +Find failed login attempts: + +```kql +event_type: "login" AND status: "failed" +``` + +Find logs from a particular IP address: + +```kql +source_ip: "203.0.113.50" +``` + +### Exercise 4: Search using numeric conditions + +Find requests taking longer than 1 second: + +```kql +response_time > 1000 +``` + +Find requests returning HTTP 504: + +```kql +http.status_code: 504 +``` + +Find requests taking longer than 1 second and returning HTTP errors: + +```kql +response_time > 1000 AND http.status_code >= 400 +``` + +Find transactions with amounts greater than 200: + +```kql +amount > 200 +``` + +### Exercise 5: Search for specific text + +Find logs containing the word `timeout`: + +```kql +message: *timeout* +``` + +Find logs containing `database`: + +```kql +message: *database* +``` + +Find logs containing either `failed` or `failure`: + +```kql +message: (*failed* OR *failure*) +``` + +### Exercise 6: Search for missing fields + +Find logs without a response time: + +```kql +NOT response_time: * +``` + +Find logs containing an error field: + +```kql +error.type: * +``` + +Find logs containing a transaction ID: + +```kql +transaction_id: * +``` + +--- + +# Step 4. Practise more realistic troubleshooting scenarios + +Now let's simulate some real-world scenarios that you might encounter when monitoring production systems. + +| Scenario | KQL Query | +|---|---| +| Find all application errors | `level: "ERROR"` | +| Find payment failures | `service: "payment-service" AND level: "ERROR"` | +| Find failed user authentication | `event_type: "login" AND status: "failed"` | +| Find slow HTTP requests | `response_time >= 3000` | +| Find HTTP server errors | `http.status_code >= 500` | +| Find database problems | `error.type: "DatabaseException"` | +| Find logs from server03 | `host: "server03"` | +| Find all logs except DEBUG | `NOT level: "DEBUG"` | +| Find payment errors or warnings | `service: "payment-service" AND level: ("ERROR" OR "WARN")` | + +These are useful patterns to understand because production troubleshooting often involves combining multiple conditions. + +--- + +# Step 5. Practise using the Discover interface + +Besides writing KQL, I recommend practising these Kibana features: + +**1. Add fields to the table** + +In Discover, expand a log entry and add these fields to your displayed columns: + +- `@timestamp` +- `level` +- `service` +- `host` +- `message` +- `response_time` + +**2. Sort logs** + +Click the `@timestamp` column to sort logs by time. + +**3. Filter using field values** + +Expand a document and click a field value to add it as a filter. + +For example, clicking `level: ERROR` will automatically create a filter. + +**4. Save searches** + +Save your KQL searches so you can reuse them later. + +For example: +- All Application Errors +- Payment Service Errors +- Failed Login Attempts +- Slow HTTP Requests + +# Step 6. Understand KQL vs Elasticsearch SQL + +One important thing to remember: + +**KQL is not SQL.** + +KQL is primarily a filtering language. It helps you find documents matching particular conditions. + +For example: + +```kql +level: "ERROR" AND service: "payment-service" +``` + +If you want to perform SQL-like operations such as: +- GROUP BY +- COUNT +- AVG +- SUM +- ORDER BY + +You should explore **ES|QL (Elasticsearch Query Language)** as well. + +For example, ES|QL can perform aggregations: + +```sql +FROM logs-lab +| WHERE level == "ERROR" +| STATS error_count = COUNT(*) BY service +| SORT error_count DESC +``` + +This counts errors grouped by application service. + +--- + +## Advanced topics + +I suggest learning Elasticsearch and Kibana in this order: + +| Level | Topic | What to learn | +|---|---|---| +| 1 | Discover | Search and explore logs | +| 2 | KQL | Filtering and searching | +| 3 | ES\|QL | Aggregation and data analysis | +| 4 | Visualize | Create charts and dashboards | +| 5 | Dashboard | Build monitoring dashboards | +| 6 | Alerts | Configure alerts for ERROR logs | +| 7 | Ingestion | Use Logstash, Beats or Elastic Agent | +| 8 | Advanced | Index lifecycle management and Elasticsearch architecture | + +**One particularly useful next exercise:** Create a dashboard showing: +- Total log count +- ERROR count +- ERROR count grouped by service +- HTTP response time +- Failed login attempts over time + +This will give you a practical introduction to how Elasticsearch and Kibana are used in real production environments.