Skip to content

Latest commit

 

History

History
3100 lines (2482 loc) · 82.6 KB

第七章——Promtheus监控k8s企业级应用.md

File metadata and controls

3100 lines (2482 loc) · 82.6 KB
Raw

第七章——Promtheus监控k8s企业级应用

前言：

配置是独立于程序的可配变量，同一份程序在不同配置下会有不同的行为。

云原生（Cloud Native）程序的特点：
  • 程序的配置，通过设置环境变了传递到容器内部
  • 程序的配置，通过程序启动参数配置生效
  • 程序的配置，通过集中在配置中心进行统一换了（CRUD）
Devops工程师应该做什么？
  • 容器化公司自研的应用程序（通过Docker进行二次封装）
  • 推动容器化应用，转变为云原生应用（一次构建，到处使用）
  • 使用容器编排框架（kubernetes），合理、规范、专业的编排业务容器

Prometheus监控软件概述

开源监控告警解决方案，推荐文章

当然一时半会你可能没那么快的去理解，那就跟我们先做下去你就会慢慢理解什么是时间序列数据

https://github.com/prometheus

https://prometheus.io

Prometheus的特点：

  • 多维数据模型：由度量名称和键值对标识的时间序列数据
  • 内置时间序列数据库：TSDB
  • promQL：一种灵活的查询语言，可以利用多维数据完成复杂查询
  • 基于HTTP的pull（拉取）方式采集时间序列数据
  • 同时支持PushGateway组件收集数据
  • 通过服务发现或静态配置发现目标
  • 支持作为数据源接入Grafana
我们将使用的官方架构图

1582697010557

Prometheus Server：服务核心组件，通过pull metrics从 Exporter 拉取和存储监控数据,并提供一套灵活的查询语言（PromQL）。

pushgateway：类似一个中转站，Prometheus的server端只会使用pull方式拉取数据，但是某些节点因为某些原因只能使用push方式推送数据，那么它就是用来接收push而来的数据并暴露给Prometheus的server拉取的中转站，这里我们不做它。

Exporters/Jobs：负责收集目标对象（host, container…）的性能数据，并通过 HTTP 接口供 Prometheus Server 获取。

Service Discovery：服务发现，Prometheus支持多种服务发现机制：文件，DNS，Consul,Kubernetes,OpenStack,EC2等等。基于服务发现的过程并不复杂，通过第三方提供的接口，Prometheus查询到需要监控的Target列表，然后轮训这些Target获取监控数据。

Alertmanager：从 Prometheus server 端接收到 alerts 后，会进行去除重复数据，分组，并路由到对方的接受方式，发出报警。常见的接收方式有：电子邮件，pagerduty 等。

UI页面的三种方法：

  • Prometheus web UI：自带的（不怎么好用）
  • Grafana：美观、强大的可视化监控指标展示工具
  • API clients：自己开发的监控展示工具

工作流程：Prometheus Server定期从配置好的Exporters/Jobs中拉metrics，或者来着pushgateway发过来的metrics，或者其它的metrics，收集完后运行定义好的alert.rules（这个文件后面会讲到），记录时间序列或者向Alertmanager推送警报。更多了解Prometheus、Metrics Server与Kubernetes监控体系

和zabbixc对比
Prometheus Zabbix
后端用golang开发，K8S也是go开发 后端用C开发，界面用PHP开发
更适合云环境的监控，尤其是对K8S有着更好的支持 更适合监控物理机，虚拟机环境
监控数据存储在基于时间序列的数据库内，便于对已有数据进行新的聚合 监控数据存储在关系型数据库内，如MySQL，很难从现有数据中扩展维度
自身界面相对较弱，很多配置需要修改配置文件，但可以借由Grafana出图 图形化界面相对比较成熟
支持更大的集群规模，速度也更快 集群规模上线为10000个节点
2015年后开始快速发展，社区活跃，使用场景越来越多 发展实际更长，对于很多监控场景，都有现成的解决方案

由于资源问题，我已经把不用的服务关掉了

1583464421747

1583464533840

1583465087708

交付kube-state-metric

WHAT：为prometheus采集k8s资源数据的exporter，能够采集绝大多数k8s内置资源的相关数据，例如pod、deploy、service等等。同时它也提供自己的数据，主要是资源采集个数和采集发生的异常次数统计

https://quay.io/repository/coreos/kube-state-metrics?tab=tags

# 200机器，下载包：
~]# docker pull quay.io/coreos/kube-state-metrics:v1.5.0
~]# docker images|grep kube-state
~]# docker tag 91599517197a harbor.od.com/public/kube-state-metrics:v1.5.0
~]# docker push harbor.od.com/public/kube-state-metrics:v1.5.0

1583394715770

# 200机器，准备资源配置清单：
~]# mkdir /data/k8s-yaml/kube-state-metrics
~]# cd /data/k8s-yaml/kube-state-metrics
kube-state-metrics]# vi rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

kube-state-metrics]# vi dp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  labels:
    grafanak8sapp: "true"
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      grafanak8sapp: "true"
      app: kube-state-metrics
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        grafanak8sapp: "true"
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: harbor.od.com/public/kube-state-metrics:v1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: http-metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
      serviceAccountName: kube-state-metrics

1583396103539

# 应用清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/rbac.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/dp.yaml
# 查询kube-metrics是否正常启动，curl哪个是在dashboard里看到的
~]# curl 172.7.21.8:8080/healthz
# out:ok
# 该命令是查看取出来的信息
~]# curl 172.7.21.8:8080/metric

1583396394541

1583396300678

完成

交付node-exporter

WHAT: 用来监控运算节点上的宿主机的资源信息，需要部署到所有运算节点

node-exporter官方dockerhub地址

# 200机器，下载镜像并准备资源配置清单：
~]# docker pull prom/node-exporter:v0.15.0
~]# docker images|grep node-exporter
~]# docker tag 12d51ffa2b22 harbor.od.com/public/node-exporter:v0.15.0
~]# docker push harbor.od.com/public/node-exporter:v0.15.0
~]# mkdir /data/k8s-yaml/node-exporter/
~]# cd /data/k8s-yaml/node-exporter/
node-exporter]# vi ds.yaml
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    daemon: "node-exporter"
    grafanak8sapp: "true"
spec:
  selector:
    matchLabels:
      daemon: "node-exporter"
      grafanak8sapp: "true"
  template:
    metadata:
      name: node-exporter
      labels:
        daemon: "node-exporter"
        grafanak8sapp: "true"
    spec:
      volumes:
      - name: proc
        hostPath: 
          path: /proc
          type: ""
      - name: sys
        hostPath:
          path: /sys
          type: ""
      containers:
      - name: node-exporter
        image: harbor.od.com/public/node-exporter:v0.15.0
        imagePullPolicy: IfNotPresent
        args:
        - --path.procfs=/host_proc
        - --path.sysfs=/host_sys
        ports:
        - name: node-exporter
          hostPort: 9100
          containerPort: 9100
          protocol: TCP
        volumeMounts:
        - name: sys
          readOnly: true
          mountPath: /host_sys
        - name: proc
          readOnly: true
          mountPath: /host_proc
      hostNetwork: true

# 22机器，应用：
# 先看一下宿主机有没有9100端口，发现什么都没有
~]# netstat -luntp|grep 9100
~]# kubectl apply -f http://k8s-yaml.od.com/node-exporter/ds.yaml
# 创建完再看端口，可能启动的慢些，我是刷了3次才有
~]# netstat -luntp|grep 9100
~]# curl localhost:9100
# 该命令是查看取出来的信息
~]# curl localhost:9100/metrics

1583396713104

1583396743927

完成

交付cadvisor

WHAT： 用来监控容器内部使用资源的信息

cadvisor官方dockerhub镜像

# 200机器，下载镜像：
~]# docker pull google/cadvisor:v0.28.3
~]# docker images|grep cadvisor
~]# docker tag 75f88e3ec33 harbor.od.com/public/cadvisor:v0.28.3
~]# docker push harbor.od.com/public/cadvisor:v0.28.3
~]# mkdir /data/k8s-yaml/cadvisor/
~]# cd /data/k8s-yaml/cadvisor/
cadvisor]# vi ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: kube-system
  labels:
    app: cadvisor
spec:
  selector:
    matchLabels:
      name: cadvisor
  template:
    metadata:
      labels:
        name: cadvisor
    spec:
      hostNetwork: true
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoExecute
      containers:
      - name: cadvisor
        image: harbor.od.com/public/cadvisor:v0.28.3
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        ports:
          - name: http
            containerPort: 4194
            protocol: TCP
        readinessProbe:
          tcpSocket:
            port: 4194
          initialDelaySeconds: 5
          periodSeconds: 10
        args:
          - --housekeeping_interval=10s
          - --port=4194
      terminationGracePeriodSeconds: 30
      volumes:
      - name: rootfs
        hostPath:
          path: /
      - name: var-run
        hostPath:
          path: /var/run
      - name: sys
        hostPath:
          path: /sys
      - name: docker
        hostPath:
          path: /data/docker

1583457963534

此时我们看到大多数节点都运行在21机器上，我们人为的让pod调度到22机器（当然即使你的大多数节点都运行在22机器上也没关系）

可人为影响K8S调度策略的三种方法：

  • 污点、容忍方法：
    • 污点：运算节点node上的污点（先在运算节点上打标签等 kubectl taint nodes node1 key1=value1:NoSchedule），污点可以有多个
    • 容忍度：pod是否能够容忍污点
    • 参考kubernetes官网
  • nodeName：让Pod运行再指定的node上
  • nodeSelector：通过标签选择器，让Pod运行再指定的一类node上
# 给21机器打个污点，22机器：
~]# kubectl taint node hdss7-21.host.com node-role.kubernetes.io/master=master:NoSchedule

1581470696125

1583458119938

# 21/22两个机器，修改软连接：
~]# mount -o remount,rw /sys/fs/cgroup/
~]# ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu
~]# ls -l /sys/fs/cgroup/

mount -o remount, rw /sys/fs/cgroup：重新以可读可写的方式挂载为已经挂载/sys/fs/cgroup

ln -s：创建对应的软链接

ls -l：显示不隐藏的文件与文件夹的详细信息

1583458181722

# 22机器，应用资源清单：
~]# kubectl apply -f http://k8s-yaml.od.com/cadvisor/ds.yaml
~]# kubectl get pods -n kube-system -o wide

只有22机器上有，跟我们预期一样

1583458264404

# 21机器，我们删掉污点：
~]# kubectl taint node hdss7-21.host.com node-role.kubernetes.io/master-
# out: node/hdss7-21.host.com untainted

看dashboard，污点已经没了

1583458299094

在去Pods看，污点没了，pod就自动起来了

1583458316885

完成

再修改下

1583458394302

交付blackbox-exporter

WHAT：监控业务容器存活性

# 200机器，下载镜像
~]# docker pull prom/blackbox-exporter:v0.15.1
~]# docker images|grep blackbox-exporter
~]# docker tag 81b70b6158be harbor.od.com/public/blackbox-exporter:v0.15.1
~]# docker push harbor.od.com/public/blackbox-exporter:v0.15.1
~]# mkdir /data/k8s-yaml/blackbox-exporter
~]# cd /data/k8s-yaml/blackbox-exporter
blackbox-exporter]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: blackbox-exporter
  name: blackbox-exporter
  namespace: kube-system
data:
  blackbox.yml: |-
    modules:
      http_2xx:
        prober: http
        timeout: 2s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: [200,301,302]
          method: GET
          preferred_ip_protocol: "ip4"
      tcp_connect:
        prober: tcp
        timeout: 2s

blackbox-exporter]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: blackbox-exporter
  namespace: kube-system
  labels:
    app: blackbox-exporter
  annotations:
    deployment.kubernetes.io/revision: 1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blackbox-exporter
  template:
    metadata:
      labels:
        app: blackbox-exporter
    spec:
      volumes:
      - name: config
        configMap:
          name: blackbox-exporter
          defaultMode: 420
      containers:
      - name: blackbox-exporter
        image: harbor.od.com/public/blackbox-exporter:v0.15.1
        imagePullPolicy: IfNotPresent
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --log.level=info
        - --web.listen-address=:9115
        ports:
        - name: blackbox-port
          containerPort: 9115
          protocol: TCP
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 50Mi
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3

blackbox-exporter]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: blackbox-exporter
  namespace: kube-system
spec:
  selector:
    app: blackbox-exporter
  ports:
    - name: blackbox-port
      protocol: TCP
      port: 9115

blackbox-exporter]# vi ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blackbox-exporter
  namespace: kube-system
spec:
  rules:
  - host: blackbox.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: blackbox-exporter
          servicePort: blackbox-port

1583460099035

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
blackbox           A    10.4.7.10

~]# systemctl restart named
# 22机器
~]# dig -t A blackbox.od.com @192.168.0.2 +short
# out: 10.4.7.10

1583460226033

# 22机器，应用：
~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/ingress.yaml

1583460413279

blackbox.od.com

1583460433145

完成

安装部署Prometheus-server

WHAT：服务核心组件，通过pull metrics从 Exporter 拉取和存储监控数据,并提供一套灵活的查询语言（PromQL）

prometheus-server官网docker地址

# 200机器，准备镜像、资源清单：
~]# docker pull prom/prometheus:v2.14.0
~]# docker images|grep prometheus
~]# docker tag 7317640d555e harbor.od.com/infra/prometheus:v2.14.0
~]# docker push harbor.od.com/infra/prometheus:v2.14.0
~]# mkdir /data/k8s-yaml/prometheus
~]# cd /data/k8s-yaml/prometheus
prometheus]# vi rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: prometheus
  namespace: infra
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: infra

prometheus]# vi dp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "5"
  labels:
    name: prometheus
  name: prometheus
  namespace: infra
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 7
  selector:
    matchLabels:
      app: prometheus
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: harbor.od.com/infra/prometheus:v2.14.0
        imagePullPolicy: IfNotPresent
        command:
        - /bin/prometheus
        args:
        - --config.file=/data/etc/prometheus.yml
        - --storage.tsdb.path=/data/prom-db
        - --storage.tsdb.min-block-duration=10m
        - --storage.tsdb.retention=72h
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - mountPath: /data
          name: data
        resources:
          requests:
            cpu: "1000m"
            memory: "1.5Gi"
          limits:
            cpu: "2000m"
            memory: "3Gi"
      imagePullSecrets:
      - name: harbor
      securityContext:
        runAsUser: 0
      serviceAccountName: prometheus
      volumes:
      - name: data
        nfs:
          server: hdss7-200
          path: /data/nfs-volume/prometheus

prometheus]# vi svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: infra
spec:
  ports:
  - port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus

prometheus]# vi ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  name: prometheus
  namespace: infra
spec:
  rules:
  - host: prometheus.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: prometheus
          servicePort: 9090

# 准备prometheus的配置文件：
prometheus]# mkdir /data/nfs-volume/prometheus
prometheus]# cd /data/nfs-volume/prometheus
prometheus]# mkdir {etc,prom-db}
prometheus]# cd etc/
etc]# cp /opt/certs/ca.pem .
etc]# cp -a /opt/certs/client.pem .
etc]# cp -a /opt/certs/client-key.pem .
etc]# prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s
scrape_configs:
- job_name: 'etcd'
  tls_config:
    ca_file: /data/etc/ca.pem
    cert_file: /data/etc/client.pem
    key_file: /data/etc/client-key.pem
  scheme: https
  static_configs:
  - targets:
    - '10.4.7.12:2379'
    - '10.4.7.21:2379'
    - '10.4.7.22:2379'
- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
- job_name: 'kubernetes-kubelet'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __address__
    replacement: ${1}:10255
- job_name: 'kubernetes-cadvisor'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __address__
    replacement: ${1}:4194
- job_name: 'kubernetes-kube-state'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
  - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
    regex: .*true.*
    action: keep
  - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
    regex: 'node-exporter;(.*)'
    action: replace
    target_label: nodename
- job_name: 'blackbox_http_pod_probe'
  metrics_path: /probe
  kubernetes_sd_configs:
  - role: pod
  params:
    module: [http_2xx]
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_blackbox_scheme]
    action: keep
    regex: http
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_blackbox_port,  __meta_kubernetes_pod_annotation_blackbox_path]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+);(.+)
    replacement: $1:$2$3
    target_label: __param_target
  - action: replace
    target_label: __address__
    replacement: blackbox-exporter.kube-system:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
- job_name: 'blackbox_tcp_pod_probe'
  metrics_path: /probe
  kubernetes_sd_configs:
  - role: pod
  params:
    module: [tcp_connect]
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_blackbox_scheme]
    action: keep
    regex: tcp
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_blackbox_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __param_target
  - action: replace
    target_label: __address__
    replacement: blackbox-exporter.kube-system:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
- job_name: 'traefik'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    action: keep
    regex: traefik
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

cp -a：在复制目录时使用，它保留链接、文件属性，并复制目录下的所有内容

1583461823355

# 11机器， 解析域名，有ingress就有页面就需要解析：
~]# vi /var/named/od.com.zone
serial 前滚一位
prometheus         A    10.4.7.10

~]# systemctl restart named
~]# dig -t A prometheus.od.com @10.4.7.11 +short
# out:10.4.7.10

1582704423890

# 22机器，应用配置清单：
~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/rbac.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/ingress.yaml

1583461941453

1583462134250

prometheus.od.com

这就是Prometheus自带的UI页面，现在你就知道为什么我们需要Grafana来替代了，如果你还不清楚，等下看Grafana的页面你就知道了

1583462164465

1583462217169

完成

配置Prometheus监控业务容器

先配置traefik

1583462282296

# Edit a Daemon Set，添加以下内容，记得给上面加逗号:
"annotations": {
  "prometheus_io_scheme": "traefik",
  "prometheus_io_path": "/metrics",
  "prometheus_io_port": "8080"
}
# 直接加进去update，会自动对齐

1583462379871

删掉两个对应的pod让它重启

1583462451073

# 22机器，查看下，如果起不来就用命令行的方式强制删除：
~]# kubectl get pods -n kube-system
~]# kubectl delete pods traefik-ingress-g26kw -n kube-system --force --grace-period=0

1583462566364

启动成功后，去Prometheus查看

刷新后，可以看到是traefik2/2，已经有了

1583462600531

完成

blackbox

我们起一个dubbo-service，之前我们最后做的是Apollo的版本，现在我们的Apollo已经关了（因为消耗资源），现在需要起更早之前不是Apollo的版本。

我们去harbor里面找

1583465190219

我的Apollo的版本可能比你的多一个，不用在意，那是做实验弄的

修改版本信息

1583466214230

1583466251914

在把scale改成1

1583466284890

查看POD的LOGS日志

1583466310189

翻页查看，已经启动

1583466328146

如何监控存活性，只需要修改配置

1584699708597

# Edit a Deployment（TCP），添加以下内容
"annotations": {
  "blackbox_port": "20880",
  "blackbox_scheme": "tcp"
}
# 直接加进去update，会自动对齐

1583466938931

UPDATE后，已经running起来了

1583467301614

prometheus.od.com刷新，自动发现业务

1583466979716

blackbox.od.com 刷新

1583467331128

同样的，我们把dubbo-consumer也弄进来

先去harbor找一个不是Apollo的版本（为什么要用不是Apollo的版本前面已经说了）

1583503611435

修改版本信息，并添加annotations

# Edit a Deployment(http)，添加以下内容，记得前面的逗号
"annotations":{
  "blackbox_path": "/hello?name=health",
  "blackbox_port": "8080",
  "blackbox_scheme": "http"
}
# 直接加进去update，会自动对齐

1583503670313

1583504095794

UPDATE后，把scale改成1

1583503796291

确保起来了

1583503811855

1583503829457

prometheus.od.com刷新，自动发现业务

1583503935815

blackbox.od.com 刷新

1583504112078

安装部署配置Grafana

WHAT：美观、强大的可视化监控指标展示工具

WHY：用来代替prometheus原生UI界面

# 200机器，准备镜像、资源配置清单：
~]# docker pull grafana/grafana:5.4.2
~]# docker images|grep grafana
~]# docker tag 6f18ddf9e552 harbor.od.com/infra/grafana:v5.4.2
~]# docker push harbor.od.com/infra/grafana:v5.4.2
~]# mkdir /data/k8s-yaml/grafana/ /data/nfs-volume/grafana
~]# cd /data/k8s-yaml/grafana/
grafana]# vi rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: grafana
rules:
- apiGroups:
  - "*"
  resources:
  - namespaces
  - deployments
  - pods
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: grafana
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: grafana
subjects:
- kind: User
  name: k8s-node

grafana]# vi dp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: grafana
    name: grafana
  name: grafana
  namespace: infra
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 7
  selector:
    matchLabels:
      name: grafana
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: grafana
        name: grafana
    spec:
      containers:
      - name: grafana
        image: harbor.od.com/infra/grafana:v5.4.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3000
          protocol: TCP
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: data
      imagePullSecrets:
      - name: harbor
      securityContext:
        runAsUser: 0
      volumes:
      - nfs:
          server: hdss7-200
          path: /data/nfs-volume/grafana
        name: data

grafana]# vi svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: infra
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: grafana

grafana]# vi ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana
  namespace: infra
spec:
  rules:
  - host: grafana.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: grafana
          servicePort: 3000

1583504719781

# 11机器，解析域名:
~]# vi /var/named/od.com.zone
serial 前滚一位

grafana            A    10.4.7.10
~]# systemctl restart named
~]# ping grafana.od.com

1582705048800

# 22机器，应用配置清单：
~]# kubectl apply -f http://k8s-yaml.od.com/grafana/rbac.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/grafana/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/grafana/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/grafana/ingress.yaml

1583504865941

grafana.od.com

默认账户和密码都是admin

修改密码：admin123

1583504898029

修改配置，修改如下图

1583505029816

装插件

进入容器

1583505097409

# 第一个：kubenetes App
grafana# grafana-cli plugins install grafana-kubernetes-app
# 第二个：Clock Pannel
grafana# grafana-cli plugins install grafana-clock-panel
# 第三个：Pie Chart
grafana# grafana-cli plugins install grafana-piechart-panel
# 第四个：D3Gauge
grafana# grafana-cli plugins install briangann-gauge-panel
# 第五个：Discrete
grafana# grafana-cli plugins install natel-discrete-panel

1583505305939

装完后，可以在200机器查看

# 200机器：
cd /data/nfs-volume/grafana/plugins/
plugins]# ll

1583505462177

删掉让它重启

1583505490948

重启完成后

查看grafana.od.com，刚刚安装的5个插件都在里面了（记得检查是否在里面了）

1583505547061

添加数据源：Add data source

1583505581557

1583505600031

# 填入参数：
URL:http://prometheus.od.com
TLS Client Auth✔    With CA Cert✔

1583505840252

# 填入参数对应的pem参数：
# 200机器拿ca等：
~]# cat /opt/certs/ca.pem
~]# cat /opt/certs/client.pem
~]# cat /opt/certs/client-key.pem

1583505713033

1583505856093

保存

然后我们去配置plugins里面的kubernetes

1583505923109

1583505938700

右侧就多了个按钮，点击进去

1583505969865

# 按参数填入：
Name:myk8s
URL:https://10.4.7.10:7443
Access:Server
TLS Client Auth✔    With CA Cert✔

1583506058483

# 填入参数：
# 200机器拿ca等：
~]# cat /opt/certs/ca.pem
~]# cat /opt/certs/client.pem
~]# cat /opt/certs/client-key.pem

1583506131529

save后再点击右侧框的图标，并点击Name

1583506163546

可能抓取数据的时间会稍微慢些（两分钟左右）

1583506184293

1583506503213

点击右上角的K8s Cluster，选择你要看的东西

1583506545308

由于K8s Container里面数据不全，如下图

1583506559069

我们改下，把Cluster删了

1583506631392

1583506645982

container也删了

1583506675876

deployment也删了

1583506695618

node也删了

1583506709705

1583506730713

1583506744138

把我给你准备的dashboard的json文件import进来

1583543092886

1583543584476

1583543602130

用同样的方法把node、deployment、cluster、container这4个分别import进来

1583543698727

可以都看一下，已经正常了

然后把etcd、generic、traefik也import进来

1583543809740

1583543831830

还有另外一种import的方法（使用官网的）：

grafana官网

找一个别人写好的点进去

1584241883144

这个编号可以直接用

1584241903882

如下图，我们装blackbox的编号是9965

1584242072703

1584242093513

把名字和Prometheus修改一下

1584242164621

或者，你也可以用我上传的（我用的是7587）

1583543931644

你可以两个都用，自己做对比，都留着也可以，就是占一些资源

JMX

1583544009027

这个里面还什么都没有

1583544017606

把Dubbo微服务数据弄到Grafana

dubbo-service

1583544062372

# Edit a Daemon Set，添加以下内容，注意给上一行加逗号
  "prometheus_io_scrape": "true",
  "prometheus_io_port": "12346",
  "prometheus_io_path": "/"
# 直接加进去update，会自动对齐，

1583544144136

dubbo-consumer

1583544157268

# Edit a Daemon Set，添加以下内容，注意给上一行加逗号
  "prometheus_io_scrape": "true",
  "prometheus_io_port": "12346",
  "prometheus_io_path": "/"
# 直接加进去update，会自动对齐，

1583544192459

刷新JMX（可能有点慢，我等了1分钟才出来service，我机器不行了）

1583544446817

完成

此时你可以感受到，Grafana明显比K8S自带的UI界面更加人性化

安装部署alertmanager

WHAT： 从 Prometheus server 端接收到 alerts 后，会进行去除重复数据，分组，并路由到对方的接受方式，发出报警。常见的接收方式有：电子邮件，pagerduty 等。

WHY：使得系统的警告随时让我们知道

# 200机器，准备镜像、资源清单：
~]# mkdir /data/k8s-yaml/alertmanager
~]# cd /data/k8s-yaml/alertmanager
alertmanager]# docker pull docker.io/prom/alertmanager:v0.14.0
# 注意，这里你如果不用14版本可能会报错
alertmanager]# docker images|grep alert
alertmanager]# docker tag 23744b2d645c harbor.od.com/infra/alertmanager:v0.14.0
alertmanager]# docker push harbor.od.com/infra/alertmanager:v0.14.0
# 注意下面记得修改成你自己的邮箱等信息，还有中文注释可以删掉
alertmanager]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: infra
data:
  config.yml: |-
    global:
      # 在没有报警的情况下声明为已解决的时间
      resolve_timeout: 5m
      # 配置邮件发送信息
      smtp_smarthost: 'smtp.163.com:25'
      smtp_from: '[email protected]'
      smtp_auth_username: '[email protected]'
      smtp_auth_password: 'xxxxxx'
      smtp_require_tls: false
    # 所有报警信息进入后的根路由，用来设置报警的分发策略
    route:
      # 这里的标签列表是接收到报警信息后的重新分组标签，例如，接收到的报警信息里面有许多具有 cluster=A 和 alertname=LatncyHigh 这样的标签的报警信息将会批量被聚合到一个分组里面
      group_by: ['alertname', 'cluster']
      # 当一个新的报警分组被创建后，需要等待至少group_wait时间来初始化通知，这种方式可以确保您能有足够的时间为同一分组来获取多个警报，然后一起触发这个报警信息。
      group_wait: 30s

      # 当第一个报警发送后，等待'group_interval'时间来发送新的一组报警信息。
      group_interval: 5m

      # 如果一个报警信息已经发送成功了，等待'repeat_interval'时间来重新发送他们
      repeat_interval: 5m

      # 默认的receiver：如果一个报警没有被一个route匹配，则发送给默认的接收器
      receiver: default

    receivers:
    - name: 'default'
      email_configs:
      - to: '[email protected]'
        send_resolved: true

alertmanager]# vi dp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: alertmanager
  namespace: infra
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager
  template:
    metadata:
      labels:
        app: alertmanager
    spec:
      containers:
      - name: alertmanager
        image: harbor.od.com/infra/alertmanager:v0.14.0
        args:
          - "--config.file=/etc/alertmanager/config.yml"
          - "--storage.path=/alertmanager"
        ports:
        - name: alertmanager
          containerPort: 9093
        volumeMounts:
        - name: alertmanager-cm
          mountPath: /etc/alertmanager
      volumes:
      - name: alertmanager-cm
        configMap:
          name: alertmanager-config
      imagePullSecrets:
      - name: harbor

alertmanager]# vi svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  namespace: infra
spec:
  selector: 
    app: alertmanager
  ports:
    - port: 80
      targetPort: 9093

1583547933312

# 22机器，应用清单：
~]# kubectl apply -f http://k8s-yaml.od.com/alertmanager/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/alertmanager/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/alertmanager/svc.yaml

1583545326722

1583545352951

# 200机器，配置报警规则：
~]# vi /data/nfs-volume/prometheus/etc/rules.yml
groups:
- name: hostStatsAlert
  rules:
  - alert: hostCpuUsageAlert
    expr: sum(avg without (cpu)(irate(node_cpu{mode!='idle'}[5m]))) by (instance) > 0.85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} CPU usage above 85% (current value: {{ $value }}%)"
  - alert: hostMemUsageAlert
    expr: (node_memory_MemTotal - node_memory_MemAvailable)/node_memory_MemTotal > 0.85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} MEM usage above 85% (current value: {{ $value }}%)"
  - alert: OutOfInodes
    expr: node_filesystem_free{fstype="overlay",mountpoint ="/"} / node_filesystem_size{fstype="overlay",mountpoint ="/"} * 100 < 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Out of inodes (instance {{ $labels.instance }})"
      description: "Disk is almost running out of available inodes (< 10% left) (current value: {{ $value }})"
  - alert: OutOfDiskSpace
    expr: node_filesystem_free{fstype="overlay",mountpoint ="/rootfs"} / node_filesystem_size{fstype="overlay",mountpoint ="/rootfs"} * 100 < 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Out of disk space (instance {{ $labels.instance }})"
      description: "Disk is almost full (< 10% left) (current value: {{ $value }})"
  - alert: UnusualNetworkThroughputIn
    expr: sum by (instance) (irate(node_network_receive_bytes[2m])) / 1024 / 1024 > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Unusual network throughput in (instance {{ $labels.instance }})"
      description: "Host network interfaces are probably receiving too much data (> 100 MB/s) (current value: {{ $value }})"
  - alert: UnusualNetworkThroughputOut
    expr: sum by (instance) (irate(node_network_transmit_bytes[2m])) / 1024 / 1024 > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Unusual network throughput out (instance {{ $labels.instance }})"
      description: "Host network interfaces are probably sending too much data (> 100 MB/s) (current value: {{ $value }})"
  - alert: UnusualDiskReadRate
    expr: sum by (instance) (irate(node_disk_bytes_read[2m])) / 1024 / 1024 > 50
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Unusual disk read rate (instance {{ $labels.instance }})"
      description: "Disk is probably reading too much data (> 50 MB/s) (current value: {{ $value }})"
  - alert: UnusualDiskWriteRate
    expr: sum by (instance) (irate(node_disk_bytes_written[2m])) / 1024 / 1024 > 50
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Unusual disk write rate (instance {{ $labels.instance }})"
      description: "Disk is probably writing too much data (> 50 MB/s) (current value: {{ $value }})"
  - alert: UnusualDiskReadLatency
    expr: rate(node_disk_read_time_ms[1m]) / rate(node_disk_reads_completed[1m]) > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Unusual disk read latency (instance {{ $labels.instance }})"
      description: "Disk latency is growing (read operations > 100ms) (current value: {{ $value }})"
  - alert: UnusualDiskWriteLatency
    expr: rate(node_disk_write_time_ms[1m]) / rate(node_disk_writes_completedl[1m]) > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Unusual disk write latency (instance {{ $labels.instance }})"
      description: "Disk latency is growing (write operations > 100ms) (current value: {{ $value }})"
- name: http_status
  rules:
  - alert: ProbeFailed
    expr: probe_success == 0
    for: 1m
    labels:
      severity: error
    annotations:
      summary: "Probe failed (instance {{ $labels.instance }})"
      description: "Probe failed (current value: {{ $value }})"
  - alert: StatusCode
    expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
    for: 1m
    labels:
      severity: error
    annotations:
      summary: "Status Code (instance {{ $labels.instance }})"
      description: "HTTP status code is not 200-399 (current value: {{ $value }})"
  - alert: SslCertificateWillExpireSoon
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
      description: "SSL certificate expires in 30 days (current value: {{ $value }})"
  - alert: SslCertificateHasExpired
    expr: probe_ssl_earliest_cert_expiry - time()  <= 0
    for: 5m
    labels:
      severity: error
    annotations:
      summary: "SSL certificate has expired (instance {{ $labels.instance }})"
      description: "SSL certificate has expired already (current value: {{ $value }})"
  - alert: BlackboxSlowPing
    expr: probe_icmp_duration_seconds > 2
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Blackbox slow ping (instance {{ $labels.instance }})"
      description: "Blackbox ping took more than 2s (current value: {{ $value }})"
  - alert: BlackboxSlowRequests
    expr: probe_http_duration_seconds > 2 
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Blackbox slow requests (instance {{ $labels.instance }})"
      description: "Blackbox request took more than 2s (current value: {{ $value }})"
  - alert: PodCpuUsagePercent
    expr: sum(sum(label_replace(irate(container_cpu_usage_seconds_total[1m]),"pod","$1","container_label_io_kubernetes_pod_name", "(.*)"))by(pod) / on(pod) group_right kube_pod_container_resource_limits_cpu_cores *100 )by(container,namespace,node,pod,severity) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Pod cpu usage percent has exceeded 80% (current value: {{ $value }}%)"

# 在最后面添加如下内容
~]# vi /data/nfs-volume/prometheus/etc/prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager"]
rule_files:
 - "/data/etc/rules.yml"

1583545590235

rules.yml文件：这个文件就是报警规则

这时候可以重启Prometheus的pod，但生产商因为Prometheus太庞大，删掉容易拖垮集群，所以我们用另外一种方法，平滑加载（Prometheus支持）：

# 21机器，因为我们起的Prometheus是在21机器，平滑加载:
~]# ps aux|grep prometheus
~]# kill -SIGHUP 1488

1583545718137

1583545762475

这时候报警规则就都有了

测试alertmanager报警功能

先把对应的两个邮箱的stmp都打开

1583721318338

1583721408460

我们测试一下，把dubbo-service停了，这样consumer就会报错

把service的scale改成0

1583545840349

blackbox.od.com查看，已经failure了

1583545937643

prometheus.od.com.alerts查看，两个变红了（一开始是变黄）

1583548102650

1583545983131

这时候可以在163邮箱看到已发送的报警

1583721856162

QQ邮箱收到报警

1583721899076

完成（service的scale记得改回1）

关于rules.yml：报警不能错报也不能漏报，在实际应用中，我们需要不断的修改rules的规则，以来贴近我们公司的实际需求。

资源不足时，可关闭部分非必要资源

# 22机器，也可以用dashboard操作：
~]# kubectl scale deployment grafana --replicas=0 -n infra
# out : deployment.extensions/grafana scaled
~]# kubectl scale deployment alertmanager --replicas=0 -n infra
# out : deployment.extensions/alertmanager scaled
~]# kubectl scale deployment prometheus --replicas=0 -n infra
# out : deployment.extensions/prometheus scaled

通过K8S部署dubbo微服务接入ELK架构

WHAT：ELK是三个开源软件的缩写，分别是：

  • E——ElasticSearch：分布式搜索引擎，提供搜集、分析、存储数据三大功能。
  • L——LogStash：对日志的搜集、分析、过滤日志的工具，支持大量的数据获取方式。
  • K——Kibana：为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。
  • 还有新增的FileBeat（流式日志收集器）：轻量级的日志收集处理工具，占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具，用来替代部分原本Logstash的工作。收集日志的多种方式及原理

WHY： 随着容器编排的进行，业务容器在不断的被创建、摧毁、迁移、扩容缩容等，面对如此海量的数据，又分布在各个不同的地方，我们不可能用传统的方法登录到每台机器看，所以我们需要建立一套集中的方法。我们需要这样一套日志手机、分析的系统：

  • 收集——采集多种来源的日志数据（流式日志收集器）
  • 传输——稳定的把日志数据传输到中央系统（消息队列）
  • 存储——将日志以结构化数据的形式存储起来（搜索引擎）
  • 分析——支持方便的分析、检索等，有GUI管理系统（前端）
  • 警告——提供错误报告，监控机制（监控工具）

这就是ELK

ELK Stack概述

1581729929995

c1/c2：container（容器）的缩写

filebeat：收集业务容器的日志，把c和filebeat放在一个pod里让他们一起跑，这样耦合就紧了

kafka：高吞吐量的分布式发布订阅消息系统，它可以处理消费者在网站中的所有动作流数据。filebeat收集数据以Topic形式发布到kafka。

Topic：Kafka数据写入操作的基本单元

logstash：取kafka里的topic，然后再往ElasticSearch上传（异步过程，即又取又传）

index-pattern：把数据按环境分（按prod和test分），并传到kibana

kibana：展示数据

制作tomcat容器的底包镜像

尝试用tomcat的方式，因为很多公司老项目都是用tomcat跑起来，之前我们用的是springboot

tomcat官网

1583558092305

# 200 机器：
cd /opt/src/
# 你也可以直接用我上传的，因为版本一直在变，之前的版本你是下载不下来的，如何查看新版本如上图
src]# wget https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.51/bin/apache-tomcat-8.5.51.tar.gz
src]# mkdir /data/dockerfile/tomcat
src]# tar xfv  apache-tomcat-8.5.51.tar.gz -C /data/dockerfile/tomcat
src]# cd /data/dockerfile/tomcat
# 配置tomcat-关闭AJP端口
tomcat]# vi apache-tomcat-8.5.51/conf/server.xml
# 找到AJP，注释掉相应的一行，结果如下图，8.5.51是已经自动注释掉的

1583558364369

# 200机器，删掉不需要的日志：
tomcat]# vi apache-tomcat-8.5.51/conf/logging.properties
# 删掉3manager，4host-manager的handlers，并注释掉相关的，结果如下图
# 日志级别改成INFO

1583558487445

1583558525033

1583558607700

# 200机器，准备Dockerfile：
tomcat]# vi Dockerfile
From harbor.od.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ 
    echo 'Asia/Shanghai' >/etc/timezone
ENV CATALINA_HOME /opt/tomcat
ENV LANG zh_CN.UTF-8
ADD apache-tomcat-8.5.51/ /opt/tomcat
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
WORKDIR /opt/tomcat
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]

tomcat]# vi config.yml
---
rules:
  - pattern: '-*'

tomcat]# wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar
tomcat]# vi entrypoint.sh
#!/bin/bash
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
MIN_HEAP=${MIN_HEAP:-"128m"}
MAX_HEAP=${MAX_HEAP:-"128m"}
JAVA_OPTS=${JAVA_OPTS:-"-Xmn384m -Xss256k -Duser.timezone=GMT+08  -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram  -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8"}
CATALINA_OPTS="${CATALINA_OPTS}"
JAVA_OPTS="${M_OPTS} ${C_OPTS} -Xms${MIN_HEAP} -Xmx${MAX_HEAP} ${JAVA_OPTS}"
sed -i -e "1a\JAVA_OPTS=\"$JAVA_OPTS\"" -e "1a\CATALINA_OPTS=\"$CATALINA_OPTS\"" /opt/tomcat/bin/catalina.sh

cd /opt/tomcat && /opt/tomcat/bin/catalina.sh run 2>&1 >> /opt/tomcat/logs/stdout.log

tomcat]# chmod u+x entrypoint.sh
tomcat]# ll
tomcat]# docker build . -t harbor.od.com/base/tomcat:v8.5.51
tomcat]# docker push harbor.od.com/base/tomcat:v8.5.51

Dockerfile文件解析：

  • FROM：镜像地址
  • RUN：修改时区
  • ENV：设置环境变量，把tomcat软件放到opt下
  • ENV：设置环境变量，字符集用zh_CN.UTF-8
  • ADD：把apache-tomcat-8.5.50包放到/opt/tomcat下
  • ADD：让prome基于文件的自动发现服务，这个可以不要，因为没在用prome
  • ADD：把jmx_javaagent-0.3.1.jar包放到/opt/...下，用来专门收集jvm的export，能提供一个http的接口
  • WORKDIR：工作目录
  • ADD：移动文件
  • CMD：运行文件

1583559245639

完成

交付tomcat形式的dubbo服务消费者到K8S集群

改造下dubbo-demo-web项目

由于是tomcat，我们需要多建一条Jenkins流水线

1583559296196

1583559325853

1583559348560

1

1583559819045

2

1583559830860

3

1583559838755

4

1583559908506

5

1583559948318

6

1583559958558

7

1583559972282

8

1583559985561

9

1583560000469

10

1583560009300

11

1583560038370

# 将如下内容填入pipeline：
pipeline {
  agent any 
    stages {
    stage('pull') { //get project code from repo 
      steps {
        sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
    }
    stage('build') { //exec mvn cmd
      steps {
        sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
      }
    }
    stage('unzip') { //unzip  target/*.war -c target/project_dir
      steps {
        sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && unzip *.war -d ./project_dir"
      }
    }
    stage('image') { //build image and push to registry
      steps {
        writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/tomcat/webapps/${params.root_url}"""
        sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
      }
    }
  }
}

1583560082441

save

点击构建

1583560122560

# 填入指定参数，我的gittee是有tomcat的版本的，下面我依旧用的是gitlab
app_name:       dubbo-demo-web
image_name:     app/dubbo-demo-web
git_repo:       http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver:        tomcat
add_tag:        20200214_1300
mvn_dir:        ./
target_dir:     ./dubbo-client/target
mvn_cmd:        mvn clean package -Dmaven.test.skip=true
base_image:     base/tomcat:v8.5.51
maven:          3.6.1-8u232
root_url:       ROOT
# 点击Build进行构建，等待构建完成

1583561544404

build成功后

修改版本信息，删掉20880，如下图，然后update

1583630633310

1583630769979

浏览器输入demo-test.od.com/hello?name=tomcat

1583630984480

完成

查看dashboard里的pod里

1583631035074

1583631074816

这些就是我们要收集的日志，收到ELK

二进制安装部署elasticsearch

我们这里只部一个es的节点，因为我们主要是了解数据流的方式

官网下载包右键复制链接

1581667412370

# 12机器：
~]# cd /opt/src/
src]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.6.tar.gz
src]# tar xfv elasticsearch-6.8.6.tar.gz -C /opt
src]# ln -s /opt/elasticsearch-6.8.6/ /opt/elasticsearch
src]# cd /opt/elasticsearch
# 配置
elasticsearch]# mkdir -p /data/elasticsearch/{data,logs}
# 修改以下内容
elasticsearch]# vi config/elasticsearch.yml
cluster.name: es.od.com
node.name: hdss7-12.host.com
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
bootstrap.memory_lock: true
network.host: 10.4.7.12
http.port: 9200

# 修改以下内容
elasticsearch]# vi config/jvm.options
-Xms512m
-Xmx512m

# 创建普通用户
elasticsearch]# useradd -s /bin/bash -M es
elasticsearch]# chown -R es.es /opt/elasticsearch-6.8.6/
elasticsearch]# chown -R es.es /data/elasticsearch/
# 文件描述符
elasticsearch]# vi /etc/security/limits.d/es.conf
es hard nofile 65536
es soft fsize unlimited
es hard memlock unlimited
es soft memlock unlimited

# 调整内核参数
elasticsearch]# sysctl -w vm.max_map_count=262144
elasticsearch]# echo "vm.max_map_count=262144" >> /etc/sysctl.conf
elasticsearch]# sysctl -p
# 启动
elasticsearch]# su -c "/opt/elasticsearch/bin/elasticsearch -d" es
elasticsearch]# netstat -luntp|grep 9200
# 调整ES日志模板
elasticsearch]# curl -H "Content-Type:application/json" -XPUT http://10.4.7.12:9200/_template/k8s -d '{
  "template" : "k8s*",
  "index_patterns": ["k8s*"],  
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 0
  }
}'

1583631650876

完成，你看我敲这么多遍就知道要等

安装部署kafka和kafka-manager

官网

做kafka的时候不建议用超过2.2.0的版本

# 11机器：
cd /opt/src/
src]# wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
src]# tar xfv kafka_2.12-2.2.0.tgz -C /opt/
src]# ln -s /opt/kafka_2.12-2.2.0/ /opt/kafka
src]# cd /opt/kafka
kafka]# ll

1583632474778

# 11机器，配置：
kafka]# mkdir -pv /data/kafka/logs
# 修改以下配置，其中zk是不变的，最下面两行则新增到尾部
# listeners这个配置建议写成IP:9092,有些老版本的kafa不写默认是localhost,会导致filebeat识别kafka地址错误
kafka]# vi config/server.properties
listeners=PLAINTEXT://10.4.7.11:9092
log.dirs=/data/kafka/logs
zookeeper.connect=localhost:2181
log.flush.interval.messages=10000
log.flush.interval.ms=1000
delete.topic.enable=true
host.name=hdss7-11.host.com

1583632595085

# 11机器，启动：
kafka]# bin/kafka-server-start.sh -daemon config/server.properties
kafka]# ps aux|grep kafka
kafka]# netstat -luntp|grep 80711

1583632747263

1583632764359

部署kafka-manager
# 200机器，制作docker：
~]# mkdir /data/dockerfile/kafka-manager
~]# cd /data/dockerfile/kafka-manager
kafka-manager]# vi Dockerfile 
FROM hseeberger/scala-sbt

ENV ZK_HOSTS=10.4.7.11:2181 \
     KM_VERSION=2.0.0.2

RUN mkdir -p /tmp && \
    cd /tmp && \
    wget https://github.com/yahoo/kafka-manager/archive/${KM_VERSION}.tar.gz && \
    tar xxf ${KM_VERSION}.tar.gz && \
    cd /tmp/kafka-manager-${KM_VERSION} && \
    sbt clean dist && \
    unzip  -d / ./target/universal/kafka-manager-${KM_VERSION}.zip && \
    rm -fr /tmp/${KM_VERSION} /tmp/kafka-manager-${KM_VERSION}

WORKDIR /kafka-manager-${KM_VERSION}

EXPOSE 9000
ENTRYPOINT ["./bin/kafka-manager","-Dconfig.file=conf/application.conf"]

# 因为大，build过程比较慢，也比较容易失败，20分钟左右，
kafka-manager]# docker build . -t harbor.od.com/infra/kafka-manager:v2.0.0.2
# build一直失败就用我做好的，不跟你的机器也得是10.4.7.11等，因为dockerfile里面已经写死了
# kafka-manager]# docker pull 909336740/kafka-manager:v2.0.0.2
# kafka-manager]# docker tag 29badab5ea08 harbor.od.com/infra/kafka-manager:v2.0.0.2
kafka-manager]# docker images|grep kafka
kafka-manager]# docker push harbor.od.com/infra/kafka-manager:v2.0.0.2

1583635072663

# 200机器，配置资源清单：
mkdir /data/k8s-yaml/kafka-manager
cd /data/k8s-yaml/kafka-manager
kafka-manager]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-manager
  namespace: infra
  labels: 
    name: kafka-manager
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: kafka-manager
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
  template:
    metadata:
      labels: 
        app: kafka-manager
    spec:
      containers:
      - name: kafka-manager
        image: harbor.od.com/infra/kafka-manager:v2.0.0.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
          protocol: TCP
        env:
        - name: ZK_HOSTS
          value: zk1.od.com:2181
        - name: APPLICATION_SECRET
          value: letmein
      imagePullSecrets:
      - name: harbor
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0

kafka-manager]# vi svc.yaml
kind: Service
apiVersion: v1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 9000
    targetPort: 9000
  selector: 
    app: kafka-manager

kafka-manager]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
  rules:
  - host: km.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: kafka-manager
          servicePort: 9000

1583635130773

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位

km                 A    10.4.7.10
~]# systemctl restart named
~]# dig -t A km.od.com @10.4.7.11 +short
# out:10.4.7.10

1583635171353

# 22机器，应用资源：
~]# kubectl apply -f http://k8s-yaml.od.com/kafka-manager/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/kafka-manager/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/kafka-manager/ingress.yaml

1583635645699

文件大可能起不来，需要多拉几次（当然你的资源配置高应该是没问题的）

1581673449530

启动成功后浏览器输入km.od.com

1583635710991

1583635777584

填完上面三个值后就可以下拉save了

点击

1583635809826

1583635828160

1583635897650

完成

制作filebeat底包并接入dubbo服务消费者

Filebeat官网

下载指纹

1583636131189

打开后复制，后面的不需要复制

1583636252938

开始前，请确保你的这些服务都是起来的

1583636282215

# 200机器，准备镜像，资源配置清单：
mkdir /data/dockerfile/filebeat
~]# cd /data/dockerfile/filebeat
# 刚刚复制的指纹替代到下面的FILEBEAT_SHA1来，你用的是什么版本FILEBEAT_VERSION就用什么版本，更新的很快，我之前用的是5.1现在已经是6.1了
filebeat]# vi Dockerfile
FROM debian:jessie

ENV FILEBEAT_VERSION=7.6.1 \
    FILEBEAT_SHA1=887edb2ab255084ef96dbc4c7c047bfa92dad16f263e23c0fcc80120ea5aca90a3a7a44d4783ba37b135dac76618971272a591ab4a24997d8ad40c7bc23ffabf

RUN set -x && \
  apt-get update && \
  apt-get install -y wget && \
  wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -O /opt/filebeat.tar.gz && \
  cd /opt && \
  echo "${FILEBEAT_SHA1}  filebeat.tar.gz" | sha512sum -c - && \
  tar xzvf filebeat.tar.gz && \
  cd filebeat-* && \
  cp filebeat /bin && \
  cd /opt && \
  rm -rf filebeat* && \
  apt-get purge -y wget && \
  apt-get autoremove -y && \
  apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]

filebeat]# vi docker-entrypoint.sh
#!/bin/bash

ENV=${ENV:-"test"}
PROJ_NAME=${PROJ_NAME:-"no-define"}
MULTILINE=${MULTILINE:-"^\d{2}"}

cat > /etc/filebeat.yaml << EOF
filebeat.inputs:
- type: log
  fields_under_root: true
  fields:
    topic: logm-${PROJ_NAME}
  paths:
    - /logm/*.log
    - /logm/*/*.log
    - /logm/*/*/*.log
    - /logm/*/*/*/*.log
    - /logm/*/*/*/*/*.log
  scan_frequency: 120s
  max_bytes: 10485760
  multiline.pattern: '$MULTILINE'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 100
- type: log
  fields_under_root: true
  fields:
    topic: logu-${PROJ_NAME}
  paths:
    - /logu/*.log
    - /logu/*/*.log
    - /logu/*/*/*.log
    - /logu/*/*/*/*.log
    - /logu/*/*/*/*/*.log
    - /logu/*/*/*/*/*/*.log
output.kafka:
  hosts: ["10.4.7.11:9092"]
  topic: k8s-fb-$ENV-%{[topic]}
  version: 2.0.0
  required_acks: 0
  max_message_bytes: 10485760
EOF

set -xe

# If user don't provide any command
# Run filebeat
if [[ "$1" == "" ]]; then
     exec filebeat  -c /etc/filebeat.yaml 
else
    # Else allow the user to run arbitrarily commands like bash
    exec "$@"
fi

filebeat]# chmod u+x docker-entrypoint.sh
filebeat]# docker build . -t harbor.od.com/infra/filebeat:v7.6.1
# build可能会失败很多次，我最长的是7次，下面有相关报错
filebeat]# docker images|grep filebeat
filebeat]# docker push harbor.od.com/infra/filebeat:v7.6.1
# 删掉原来的内容全部用新的，使用的两个镜像对应上你自己的镜像
filebeat]# vi /data/k8s-yaml/test/dubbo-demo-consumer/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-web:tomcat_200307_1410
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://apollo-configservice:8080
        volumeMounts:
        - mountPath: /opt/tomcat/logs
          name: logm
      - name: filebeat
        image: harbor.od.com/infra/filebeat:v7.6.1
        imagePullPolicy: IfNotPresent
        env:
        - name: ENV
          value: test
        - name: PROJ_NAME
          value: dubbo-demo-web
        volumeMounts:
        - mountPath: /logm
          name: logm
      volumes:
      - emptyDir: {}
        name: logm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

相关报错（其它问题基本都是网络不稳定的问题）：1583639164435

因为你用的指纹不是自己的，或者版本没写对。

dp.yaml文件解析： spec-containers下有两个name，对应的两个容器，这就是边车模式（sidecar）。

# 22机器，应用资源清单：
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/dp.yaml
#out: deployment.extensions/dubbo-demo-consumer configured
~]# kubectl get pods -n test

1583640124995

机器在21机器

1583640217181

# 查看filebeat日志，21机器：
~]# docker ps -a|grep consumer
~]# docker exec -ti a6adcd6e83b3 bash
:/# cd /logm
:/#/logm# ls
:/#/logm# cd ..
# 这个log，是你每一次刷新demo页面都会有数据，你把它夯在这里
:/# tail -fn 200 /logm/stdout.log
# 日志就都在这里了

1583640393605

# 浏览器输入：demo-test.com/hello?name=tomcat

1583640365512

刷新上面的页面，去21机器看log

1583640316947

刷新km.od.com/clusters/kafka-od/topics

1583640444719

完成

部署logstash镜像

# 200机器，准备镜像、资源清单：
# logstash的版本需要和es的版本一样，11机器cd /opt/目录下即可查看到
~]# docker pull logstash:6.8.6
~]# docker images|grep logstash
~]# docker tag d0a2dac51fcb harbor.od.com/infra/logstash:v6.8.6
~]# docker push harbor.od.com/infra/logstash:v6.8.6
~]# mkdir /etc/logstash
~]# vi /etc/logstash/logstash-test.conf
input {
  kafka {
    bootstrap_servers => "10.4.7.11:9092"
    client_id => "10.4.7.200"
    consumer_threads => 4
    group_id => "k8s_test"
    topics_pattern => "k8s-fb-test-.*"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["10.4.7.12:9200"]
    index => "k8s-test-%{+YYYY.MM.DD}"
  }
}

~]# vi /etc/logstash/logstash-prod.conf
input {
  kafka {
    bootstrap_servers => "10.4.7.11:9092"
    client_id => "10.4.7.200"
    consumer_threads => 4
    group_id => "k8s_prod"
    topics_pattern => "k8s-fb-prod-.*"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["10.4.7.12:9200"]
    index => "k8s-prod-%{+YYYY.MM.DD}"
  }
}

# 启动
~]# docker run -d --name logstash-test -v /etc/logstash:/etc/logstash harbor.od.com/infra/logstash:v6.8.6 -f /etc/logstash/logstash-test.conf
~]# docker ps -a|grep logstash

1583651857160

我们刷新demo页面让kafka里面更新些日志

1583651874243

有日志了

1583651931546

# 200机器，验证ES索引（可能比较慢）：
~]# curl http://10.4.7.12:9200/_cat/indices?v

1583652168302

这个反应有点慢，我等了快三分钟

完成

交付kibana到K8S集群

为什么用kibana：当然运维可以直接在dashboard里exec进去，然后命令行看情况，但是开发或者测试不行，那是机密的，我们得要一个页面供他们使用，使用需要kibana。

~]# docker pull kibana:6.8.6
~]# docker images|grep kibana
~]# docker tag adfab5632ef4 harbor.od.com/infra/kibana:v6.8.6
~]# docker push harbor.od.com/infra/kibana:v6.8.6
~]# mkdir /data/k8s-yaml/kibana
~]# cd /data/k8s-yaml/kibana/
kibana]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kibana
  namespace: infra
  labels: 
    name: kibana
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: kibana
  template:
    metadata:
      labels: 
        app: kibana
        name: kibana
    spec:
      containers:
      - name: kibana
        image: harbor.od.com/infra/kibana:v6.8.6
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5601
          protocol: TCP
        env:
        - name: ELASTICSEARCH_URL
          value: http://10.4.7.12:9200
      imagePullSecrets:
      - name: harbor
      securityContext: 
        runAsUser: 0
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

kibana]# vi svc.yaml
kind: Service
apiVersion: v1
metadata: 
  name: kibana
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 5601
    targetPort: 5601
  selector: 
    app: kibana

kibana]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kibana
  namespace: infra
spec:
  rules:
  - host: kibana.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: kibana
          servicePort: 5601

1583652777163

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
kibana             A    10.4.7.10

~]# systemctl restart named
~]# dig -t A kibana.od.com @10.4.7.11 +short

1583652835822

# 22机器（21机器还夯着log），应用资源清单：
~]# kubectl apply -f http://k8s-yaml.od.com/kibana/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/kibana/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/kibana/ingress.yaml
~]# kubectl get pods -n infra

1583653095099

kibana.od.com

我用的低配8C32G，机器快跑不动了，还会显示service not yet

1581735655839

1583653403755

点完可能会转圈转很久

1583654044637

1583654055784

去创建

1583653710404

1583653734639

1583653767919

创建后，你就能看到日志

1583654148342

把prod里的configservice和admin依次起来

1583654217058

# 200机器：
cd /data/k8s-yaml/prod/dubbo-demo-consumer/
dubbo-demo-consumer]# cp ../../test/dubbo-demo-consumer/dp.yaml .
# y
# 修改namespace为prod，fat改成pro，http地址也改了
dubbo-demo-consumer]# vi dp.yaml

1583654303335

1583654391353

config-prod.od.com

1583654579273

完成

详解Kibana生产实践方法

查看环境情况

确认Eureka有config和admin

1583654579273

确认Apollo里有两个环境

1583654655665

确认完后，我们先把service起来

1583654706752

然后到consumer，consumer需要接日志

# 22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/dp.yaml
~]# kubectl get pods -n prod

1583654764653

# 200机器：
# 启动
~]# docker run -d --name logstash-prod -v /etc/logstash:/etc/logstash harbor.od.com/infra/logstash:v6.8.6 -f /etc/logstash/logstash-prod.conf
~]# docker ps -a|grep logstash
# curl一下，这时候还只有test
~]# curl http://10.4.7.12:9200/_cat/indices?v

1583654862812

# 访问浏览器demo-prod.od.com/hello?name=prod

1583654914493

我们看一下调度到哪个节点了

1583654962325

# 调度到21节点，我们去21节点看一下：
~]# docker ps -a|grep consumer
~]# docker exec -ti 094e68c795b0 bash
:/# cd /logm
:/logm# ls
:/logm# tail -fn 200 stdout.log

1583655033176

夯住

1583655055686

去http://km.od.com

1583655088618

1583655114357

1583655130583

已经有prod了

# 200机器，curl的时候可能要等一下才有（可以去多刷一下网页产生日志）：
~]# curl http://10.4.7.12:9200/_cat/indices?v

1583655264326

去kibana配一下

1583655369240

1583655383868

如何使用kibana

时间选择

1583655477179

1583655509551

test没用数据的点下这个就有了，平常用的最多的也是today，后面突然没数据了你就可以刷新或者点时间，特别是配置差的同学

1583655706547

环境选择器

1583655530032

关键字选择器

先把message顶上来，还有log.file.path、hostname

1583655759010

我们先制造一些错误，把service scale成0

1583655838132

然后刷新一下页面，让它报错，记得是test环境

1583655858792

搜exception关键字，并可展开

1583655981692

1583656009350

现在consumer日志已经完成了，记得把service的pod还原，并删掉consumer的pod让它重启

课外作业（不是一定要完成，但是你做了我做的这些）

consumer日志已经完成，还可以做service日志

# 200机器：
# 修改一下内容
~]# cd  /data/dockerfile/jre8/
# 修改以下内容
jre8]# vi entrypoint.sh
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL} 2>&1 >> /opt/logs/stdout.log

jre8]# docker build . -t harbor.od.com/base/jre8:8u112_with_logs
jre8]# docker push harbor.od.com/base/jre8:8u112_with_logs

1584067987917

1581752521087

去修改一下Jenkins加一个底包

1584237876688

1584237900572

1584237947516

下面就要你接着做了