-
Prometheus Consul Blackbox | export 监控实现
云和安全管理服务专家新钛云服 郭鹏超原创
前言：
• blackbox_exporter
是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。
• Consul
主要提供，服务发现，健康检查，等功能，本次集成主要使用到服务发现功能。
本文主要实现，基于consul_sd_config & consul 的 prometheus 服务发现，实现网路设备ping监控，站点可用行监控，以及证书相关信息监控。
安装环境：
• k8s
• consul
• Prometheus
• blackbox_exporter
1: Consul 安装
1.1：使用helm 安装 consul
Bash # 添加 consul helm 源 helm repo add hashicorp https://helm.releases.hashicorp.com # 安装consul helm -n consul install \ --set storageClass=alicloud-disk-efficiency \ consul hashicorp/consul \ --version=0.32.11.2：查看服务安装状态
Bash [root@xxxxxxxx consul_install]# kubectl -n consul get pods NAME READY STATUS RESTARTS AGE consul-consul-9lxfc 1/1 Running 0 6d1h consul-consul-ntqcf 1/1 Running 0 6d1h consul-consul-q7c6f 1/1 Running 0 6d1h consul-consul-server-0 1/1 Running 0 6d1h consul-consul-server-1 1/1 Running 0 6d1h consul-consul-server-2 1/1 Running 0 6d1h1.3：nginx-ingress consul
• consul_ingress.yml
Bash # consul.xxxxxx.cn -----> 替换为正确域名 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: consul-ingress namespace: consul annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: consul.xxxxxx.cn http: paths: - path: / pathType: Prefix backend: service: name: consul-consul-ui port: number: 80• 执行部署
Bash kubectl apply -f consul_ingress.yml1.4：访问测试
2: Blackbox_export
2.1：blackbox 安装
• blackbox-exporter-config.yaml
Bash apiVersion: v1 kind: ConfigMap metadata: name: blackbox-exporter labels: app: blackbox-exporter data: blackbox.yml: |- modules: ## ----------- DNS 检测配置 ----------- dns_tcp: prober: dns dns: transport_protocol: "tcp" preferred_ip_protocol: "ip4" query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址 query_type: "A" ## ----------- TCP 检测模块配置 ----------- tcp_connect: prober: tcp timeout: 5s ## ----------- ICMP 检测配置 ----------- ping: prober: icmp timeout: 5s icmp: preferred_ip_protocol: "ip4" ## ----------- HTTP GET 2xx 检测模块配置 ----------- http_get_2xx: prober: http timeout: 10s http: method: GET preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1","HTTP/2"] valid_status_codes: [200] # 验证的HTTP状态码,默认为2xx no_follow_redirects: false # 是否不跟随重定向 ## ----------- HTTP GET 3xx 检测模块配置 ----------- http_get_3xx: prober: http timeout: 10s http: method: GET preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1","HTTP/2"] valid_status_codes: [301,302,304,305,306,307] # 验证的HTTP状态码,默认为2xx no_follow_redirects: false # 是否不跟随重定向 ## ----------- HTTP POST 监测模块 ----------- http_post_2xx: prober: http timeout: 10s http: method: POST preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1", "HTTP/2"] #headers: # HTTP头设置 # Content-Type: application/json #body: '{}' # 请求体设置• blackbox-exporter-deploy.yaml
Bash apiVersion: v1 kind: Service metadata: name: blackbox-exporter labels: k8s-app: blackbox-exporter spec: type: ClusterIP ports: - name: http port: 9115 targetPort: 9115 selector: k8s-app: blackbox-exporter --- apiVersion: apps/v1 kind: Deployment metadata: name: blackbox-exporter labels: k8s-app: blackbox-exporter spec: replicas: 1 selector: matchLabels: k8s-app: blackbox-exporter template: metadata: labels: k8s-app: blackbox-exporter spec: containers: - name: blackbox-exporter image: prom/blackbox-exporter:v0.19.0 args: - --config.file=/etc/blackbox_exporter/blackbox.yml - --web.listen-address=:9115 - --log.level=info ports: - name: http containerPort: 9115 resources: limits: cpu: 3 memory: 6000Mi requests: cpu: 100m memory: 50Mi livenessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 readinessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 volumeMounts: - name: config mountPath: /etc/blackbox_exporter volumes: - name: config configMap: name: blackbox-exporter defaultMode: 420• 执行安装
Bash kubectl apply -f blackbox-exporter-deploy.yaml kubectl apply -f blackbox-exporter-config.yaml2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml
Bash apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: blackbox-ingress namespace: monitoring annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: blackbox-devops.lululemon.cn http: paths: - path: / pathType: Prefix backend: service: name: blackbox-exporter port: number: 9115• 执行安装
Bash kubectl apply -f blackbox_ingress.yml3: rometheus 添加 服务动态发现
Bash ##### http_get_2xx 数据获取 - job_name: http_get_2xx params: module: - http_get_2xx scrape_interval: 2s scrape_timeout: 2s metrics_path: /probe consul_sd_configs: # consul 服务地址 - server: consul-consul-server.consul.svc.cluster.local:8500 tag_separator: ',' services: - http_get_2xx relabel_configs: - source_labels: ['__meta_consul_service_address'] target_label: __param_target - source_labels: ['__meta_consul_service_address'] target_label: instance - target_label: __address__ ## blackbox-export 地址 replacement: blackbox-exporter.monitoring.svc.cluster.local:9115 ####### icmp 配置 - job_name: blackbox_icmp params: module: - ping scrape_interval: 2s scrape_timeout: 2s metrics_path: /probe consul_sd_configs: # consul 服务地址 - server: consul-consul-server.consul.svc.cluster.local:8500 tag_separator: ',' services: - ping relabel_configs: - source_labels: ['__meta_consul_service_address'] target_label: __param_target - source_labels: ['__meta_consul_service_address'] target_label: instance - target_label: __address__ ## blackbox-export 地址 replacement: blackbox-exporter.monitoring.svc.cluster.local:91154：添加 icmp 监控
4.1：添加监控地址到consul
• icmp_list
Bash 192.168.1.1 192.168.1.2• add_consul_service_icmp.sh
Bash #!/usr/bin/env bash ip_addr=$1 if test "$ip_addr";then curl -X PUT -d '{ "id": "icmp_'${ip_addr}'", "name": "ping", "address": "'${ip_addr}'", "port": 443, "Meta": { "env": "prod", "team": "network", "project": "network", "owner": "Mike" }, "tags": ["node"], "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \ http://consul-consul-server:8500/v1/agent/service/register else echo "请输入参数" fi• 添加service ping
Bash for i in `cat icmp_list`;do bash add_consul_service_icmp.sh $i;done4.2：查看consul 服务
4.3：删除ping 监控地址脚本
Bash #!/usr/bin/env bash ip_addr=$1 curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}5: 添加http_get_2xx
5.1：添加监控域名
• domain_name_list
Bash wwww.baidu.com wwww.1111.com wwww.2222.com• add_consul_service_http_get_2xx.sh
Bash #!/usr/bin/env bash service_name=$1 if test "$service_name";then curl -X PUT -d '{ "id": "http_get_2xx_'${service_name}'", "name": "http_get_2xx", "address": "https://'${service_name}'", "port": 443, "Meta": { "env": "prod", "team": "web", "project": "web", "owner": "Devops" }, "tags": ["node"], "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \ http://consul-consul-server:8500/v1/agent/service/register else echo "请输入参数" fi• 添加 service http_get_2xx
Bash for i in `cat domain_name_list`;do bash add_consul_service_http_get_2xx.sh $i;done5.2：查看consul 服务
5.3：删除域名监控脚本
• del_consul_service_http_get_2xx.sh
Bash #!/usr/bin/env bash ip_addr=$1 curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}6：查看prometheus 监控
总结：
使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。
云和安全管理服务专家
