DevOps Project Explanation: Step 5 Setting Up Monitoring and Logging
Step 5: Setting Up Monitoring and Logging
Promethus and Grafana
Azure Monitor
AWS CloudWatch
ELK -Kibana
Datadog
Nagios
a) Install Prometheus
Helm (for Kubernetes):
If you're using Kubernetes, you can install Prometheus using Helm:
helm install prometheus prometheus-community/kube-prometheus-stack
Direct Installation (for other clusters):
kubectl apply -f https://raw.githubusercontent.com/pro...
Configure Prometheus: Customize the prometheus.yml to scrape metrics from your cluster nodes or services.
b) Set Up Grafana for Visualization
Install Grafana (with Helm for Kubernetes):
helm install grafana grafana/Grafana
Connect Grafana to Prometheus:
In Grafana, go to "Data Sources" and choose Prometheus as the source.
Enter the Prometheus server URL and save.
Create Dashboards: Use pre-built Kubernetes dashboards or create your own to visualize cluster metrics (CPU, memory usage, pod status, etc.).
3. Set Up Logging with EFK (Elasticsearch, Fluentd, Kibana)
a) Install Elasticsearch and Kibana
Install Elasticsearch (via Helm for Kubernetes):
helm install elasticsearch elastic/elasticsearch
Install Kibana:
helm install kibana elastic/kibana
b) Configure Fluentd for Log Collection
Install Fluentd:
Fluentd collects logs from nodes and sends them to Elasticsearch for indexing.
In Kubernetes, you can deploy Fluentd as a DaemonSet to collect logs from all nodes.
Fluentd Configuration:
Set up Fluentd to forward logs to Elasticsearch. A sample configuration might look like:
yaml
Copy
match
@type elasticsearch
host elasticsearch.default.svc.cluster.local
port 9200
logstash_format true
flush_interval 5s
/match
c) Access Kibana for Log Visualization
Once logs are being collected and indexed by Elasticsearch, you can access Kibana to search, filter, and visualize your logs.
4. Set Up Alerts and Notifications
Prometheus Alerts: You can configure alerting rules in Prometheus to notify you when something goes wrong. Example Prometheus alert rule:
yaml
Copy
groups:
name: example-alerts
rules:
alert: HighCPUUsage
expr: sum(rate(container_cpu_usage_seconds_total{image!="", container!="POD"}[1m])) by (container) = 0.8
for: 1m
labels:
severity: critical
annotations:
summary: "Container CPU usage is too high"
Alertmanager: Prometheus integrates with Alertmanager, which can send notifications to various channels like Slack, email, or PagerDuty.
Grafana Alerts: Set up alerts based on the visualized metrics to notify via email, Slack, or other channels.
5. Set Up Cloud-native Monitoring (Optional)
If your cluster is hosted on AWS, GCP, or Azure, you can leverage their cloud-native tools for monitoring and logging.
AWS CloudWatch: Use CloudWatch to monitor EC2 instances, EKS, and other AWS resources. CloudWatch can also collect logs and metrics from EC2, Kubernetes, and Lambda.
Google Stackdriver: Use Stackdriver for monitoring and logging in Google Cloud.
Azure Monitor: Azure's native monitoring platform can be used for monitoring AKS and other Azure resources.
6. Configure Log Rotation and Retention
Ensure that logs are rotated regularly and older logs are archived or deleted based on your retention policy. For Kubernetes clusters, tools like Loki (a lightweight log aggregation system) can be used with Grafana.
Example Architecture:
For a Kubernetes cluster, the monitoring and logging setup would look like:
Prometheus for metrics collection.
Grafana for visualization of metrics.
Fluentd for log collection.
Elasticsearch + Kibana (EFK) for log storage and visualization.
Alertmanager for sending notifications.
#devopsmadeeasy
#monitoring
#devops
#sre