Demo 1

What we are going to build

• We are building a prometheus installation demo.
• We'll:
  • run an ec2 instance in AWS (for a node exporter)
  • install node-exporter on it
  • run Prometheus using Docker on out personal computer

Run an EC2 instance

• Run an ec2 instance globally accessible from the Internet: (so place in a public subnet)
• Configure a security group for it:
  • allow inbound SSH (port 22)
  • allow inbount connection from Prometheus (tcp port 9100)
  • allow all traffic outbound
• Off course, use a key-pair that'll allow you to connect to it

Install node_exporter

• The following will:
  • download, extract and copy node_exporter to /usr/local/bin
  • Create a user for it
  • Create, run and enable a service that will run node_exporter
• You can include this script in the user data part of an EC2 instance

 1wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
 2tar xvfz node_exporter-1.5.0.linux-amd64.tar.gz
 3cd node_exporter-1.5.0.linux-amd64
 4sudo cp node_exporter /usr/local/bin/
 5
 6# Create systemd service for node_exporter
 7sudo bash -c 'cat > /etc/systemd/system/node_exporter.service << EOF
 8[Unit]
 9Description=Node Exporter
10Wants=network-online.target
11After=network-online.target
12
13[Service]
14User=node_exporter
15Group=node_exporter
16Type=simple
17ExecStart=/usr/local/bin/node_exporter
18
19[Install]
20WantedBy=multi-user.target
21EOF'
22
23# Create user, set permissions and start service
24sudo useradd -rs /bin/false node_exporter
25sudo systemctl daemon-reload
26sudo systemctl start node_exporter
27sudo systemctl enable node_exporter

Running Prometheus on my virtual machine

• I am running a local ubuntu on VmWare Workstation
• Cretae an empty working directory somewhere, and cd into it:

1osboxes@osboxes:~$ mkdir prometheus
2osboxes@osboxes:~$ cd prometheus/

• Create a basic prometheus.yml configuration file. Here's what I was using (so the IP address is my node-exporter running in aws):

 1global:
 2  scrape_interval: 15s
 3scrape_configs:
 4  - job_name: 'prometheus'
 5    static_configs:
 6      - targets: ['localhost:9090']  # Prometheus itself
 7    
 8  - job_name: 'node_exporter'
 9    static_configs:
10      - targets: ['52.207.251.32:9100']  # Node exporter on EC2
11        labels:
12          instance: 'ec2-node'

• Run prometheus in a container, using the conf file:

1docker run -d --name prometheus \
2  -p 9090:9090 \
3  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
4  prom/prometheus

• Browse to localhost:9090 to get into Prometheus

Try prometheus

CPU Usage

• Use this query:

1100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

Explanation:

• node_cpu_seconds_total{mode="idle"}
  This metric tracks the total CPU time spent in idle mode for each CPU core.
• irate(node_cpu_seconds_total{mode="idle"}[1m])
  The irate() function calculates the per-second rate of increase of this metric over the last minute. This gives you the fraction of time each CPU core spent idle during the last minute (a value between 0 and 1).
  Really this is the cpu usage (but per core and in fraction - not percent)
• avg by(instance) (...)
  This averages the idle rates across all CPU cores for each instance, giving you the average idle rate per instance.
• (...) * 100
  This converts the idle rate from a fraction to a percentage.
• 100 - (...)
  Finally, this subtracts the idle percentage from 100 to get the CPU usage percentage. If your CPU is 30% idle, then it's 70% used.

Memory Usage

• Use this query:

1100 * (1 - ((node_memory_MemAvailable_bytes) / (node_memory_MemTotal_bytes)))

Disk Usage

• Use this query:

1100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)