various alerts numbers

[ltrilety]

The number of cluster alerts does not always correspond to the other available alerts numbers. Moreover those numbers not always correspond with actual state of the cluster.

E.g. I had 2 as number of alerts on Clusters page, whilst related Volumes and Hosts shows all 0. The same for the alert list.
In another case I had 1 alert on the Clusters, related Volumes and Hosts shows all 0 and Alerts list shows 3 active alerts.
In both cases the cluster was completely up and running.

Reproduction steps:

1. kill some bricks, let the tendrl notice (especially there should be alert about degraded state of a volume)
2. stop glusterd services on all nodes and wait few seconds
3. start glusterd services on all nodes so all bricks are running again
4. If needed repeat previous steps (in my case just stopping and starting glusterd was enough)

Tendrl version:
tendrl-commons-1.6.1-1.el7.centos.noarch
tendrl-api-1.6.1-1.el7.centos.noarch
tendrl-ui-1.6.1-1.el7.centos.noarch
tendrl-grafana-selinux-1.5.4-2.el7.centos.noarch
tendrl-ansible-1.6.1-1.el7.centos.noarch
tendrl-notifier-1.6.0-1.el7.centos.noarch
tendrl-node-agent-1.6.1-1.el7.centos.noarch
tendrl-api-httpd-1.6.1-1.el7.centos.noarch
tendrl-selinux-1.5.4-2.el7.centos.noarch
tendrl-grafana-plugins-1.6.1-1.el7.centos.noarch
tendrl-monitoring-integration-1.6.1-1.el7.centos.noarch

[ltrilety]

example of clusters response:
{"clusters":[{"cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","cluster_id":"dbc52192-ce3b-4086-a16b-f536b3357827","integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","hash":"675274c7df9b2ff76f83a749018f88ab","sds_version":"4.1dev","updated_at":"2018-03-12 07:49:45.445414+00:00","sds_name":"gluster","errors":[],"globaldetails":{"connection_count":"0","volume_up_degraded":"0","integration_id":"","connection_active":"0","updated_at":"2018-03-12 12:37:51.652064+00:00","peer_count":"6","vol_count":"2","hash":"05b54438e08b265a966b8cce373ebcbc","status":"healthy"},"public_network":"","cluster_network":"","is_managed":"yes","volume_profiling_state":"disabled","alert_counters":{"hash":"36eb647522122be3c86c180b3b2f6d57","warning_count":"2","updated_at":"2018-03-12 12:00:50.093777+00:00","integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827"},"status":"syncing","current_job":"{"status": "finished", "job_id": "ba769c6d-c127-4835-ade6-348c03d160dd", "job_name": "ImportCluster"}","nodes":[{"status":"UP","hash":"9b7d427804d56669c3a885cd551d8376","tags":["detected_cluster/8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","tendrl/integration/gluster","gluster/server","tendrl/integration/dbc52192-ce3b-4086-a16b-f536b3357827","tendrl/node_e70db3d4-388a-4c39-a56a-79f61de624a6","tendrl/node"],"first_sync_done":"yes","is_managed":"yes","last_sync":"2018-03-12 12:39:06.422237+00:00","sync_status":"done","fqdn":"ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:52.059225+00:00","node_id":"e70db3d4-388a-4c39-a56a-79f61de624a6","ipv4_addr":"10.37.169.105","node_id":"e70db3d4-388a-4c39-a56a-79f61de624a6"},{"is_managed":"yes","last_sync":"2018-03-12 12:38:46.258076+00:00","status":"UP","sync_status":"in_progress","fqdn":"ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:53.939244+00:00","ipv4_addr":"10.37.169.92","hash":"58b58d6b6264106f916e5e405b1eb0ac","tags":["detected_cluster/8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","tendrl/integration/gluster","gluster/server","tendrl/integration/dbc52192-ce3b-4086-a16b-f536b3357827","tendrl/node_9eefd5d3-db1a-4559-a27e-8373abfe5ffe","tendrl/node"],"first_sync_done":"yes","node_id":"9eefd5d3-db1a-4559-a27e-8373abfe5ffe","node_id":"9eefd5d3-db1a-4559-a27e-8373abfe5ffe"},{"is_managed":"yes","hash":"3670123ebfd1a2e5624a016fd1bb6754","sync_status":"in_progress","updated_at":"2018-03-12 12:39:55.107502+00:00","first_sync_done":"yes","ipv4_addr":"10.37.169.103","status":"UP","tags":["detected_cluster/8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","tendrl/integration/gluster","tendrl/node_6731a30c-016d-4dbd-90d1-19bac16fac27","gluster/server","tendrl/integration/dbc52192-ce3b-4086-a16b-f536b3357827","tendrl/node","provisioner/dbc52192-ce3b-4086-a16b-f536b3357827"],"fqdn":"ltrilety-usm1-gl3.usmqe.lab.eng.brq.redhat.com","node_id":"6731a30c-016d-4dbd-90d1-19bac16fac27","last_sync":"2018-03-12 12:38:45.475427+00:00","node_id":"6731a30c-016d-4dbd-90d1-19bac16fac27"},{"tags":["detected_cluster/8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","tendrl/integration/gluster","tendrl/node_6a82f3d7-ee8c-4301-95f5-f1d54e95dbed","tendrl/integration/dbc52192-ce3b-4086-a16b-f536b3357827","gluster/server","tendrl/node"],"sync_status":"done","node_id":"6a82f3d7-ee8c-4301-95f5-f1d54e95dbed","is_managed":"yes","status":"UP","fqdn":"ltrilety-usm1-gl2.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:49.423388+00:00","first_sync_done":"yes","ipv4_addr":"10.37.169.99","last_sync":"2018-03-12 12:39:00.785716+00:00","hash":"a29582ff90afa4d722fffc77cd0001db","node_id":"6a82f3d7-ee8c-4301-95f5-f1d54e95dbed"},{"is_managed":"yes","last_sync":"2018-03-12 12:39:15.896963+00:00","tags":["detected_cluster/8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","tendrl/integration/gluster","tendrl/node_2c2fa7bd-cb58-4217-a2a1-02dd8ec6ea88","gluster/server","tendrl/integration/dbc52192-ce3b-4086-a16b-f536b3357827","provisioner/dbc52192-ce3b-4086-a16b-f536b3357827","tendrl/node"],"first_sync_done":"yes","sync_status":"done","fqdn":"ltrilety-usm1-gl5.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:54.422538+00:00","node_id":"2c2fa7bd-cb58-4217-a2a1-02dd8ec6ea88","ipv4_addr":"10.37.169.110","status":"UP","hash":"12f9a3a4cb1ab90d52e07dd60c18eb3c","node_id":"2c2fa7bd-cb58-4217-a2a1-02dd8ec6ea88"},{"status":"UP","hash":"e9f74c60d6f5af1214bc6b2c8fd76f89","fqdn":"ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:54.902361+00:00","is_managed":"yes","last_sync":"2018-03-12 12:39:22.236917+00:00","tags":["tendrl/node_32fb7392-3b50-4027-81f7-ccee554ed97e","detected_cluster/8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","tendrl/integration/gluster","gluster/server","tendrl/integration/dbc52192-ce3b-4086-a16b-f536b3357827","tendrl/node"],"sync_status":"done","first_sync_done":"yes","node_id":"32fb7392-3b50-4027-81f7-ccee554ed97e","ipv4_addr":"10.37.169.116","node_id":"32fb7392-3b50-4027-81f7-ccee554ed97e"}]},{"cluster_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","hash":"856ca3d01e96c882aa6e188cae78aa03","sds_version":"4.1dev","updated_at":"2018-03-12 07:49:42.990775+00:00","sds_name":"gluster","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","errors":[],"globaldetails":{"vol_count":"1","hash":"7f11836461b81993ab955c730a4bc489","connection_count":"0","status":"healthy","integration_id":"","updated_at":"2018-03-12 12:39:04.250951+00:00","peer_count":"6","volume_up_degraded":"0","connection_active":"0"},"public_network":"","cluster_network":"","is_managed":"yes","volume_profiling_state":"disabled","alert_counters":{"hash":"8cea3a2244bb53a510a5aa974afe9b4f","warning_count":"1","updated_at":"2018-03-12 12:35:15.886538+00:00","integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3"},"status":"","current_job":"{"status": "finished", "job_id": "af8d1052-449e-494b-b472-15c0bd12936b", "job_name": "ImportCluster"}","nodes":[{"status":"UP","tags":["tendrl/integration/gluster","detected_cluster/6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","gluster/server","tendrl/integration/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node_1a9a4334-de31-4eac-9077-7caa5648e2cf","tendrl/node"],"sync_status":"done","updated_at":"2018-03-12 12:39:35.012825+00:00","first_sync_done":"yes","node_id":"1a9a4334-de31-4eac-9077-7caa5648e2cf","is_managed":"yes","hash":"94978820c3fb3b5832bcb5b3c27cb1d6","fqdn":"ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com","ipv4_addr":"10.37.169.150","last_sync":"2018-03-12 12:39:34.406178+00:00","node_id":"1a9a4334-de31-4eac-9077-7caa5648e2cf"},{"fqdn":"ltrilety-usm2-gl4.usmqe.lab.eng.brq.redhat.com","ipv4_addr":"10.37.169.154","is_managed":"yes","tags":["tendrl/integration/gluster","detected_cluster/6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","gluster/server","tendrl/integration/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node_5de37956-70f9-44be-bca1-5553fbf09860","tendrl/node"],"sync_status":"done","updated_at":"2018-03-12 12:39:40.826716+00:00","first_sync_done":"yes","node_id":"5de37956-70f9-44be-bca1-5553fbf09860","last_sync":"2018-03-12 12:38:57.932957+00:00","status":"UP","hash":"ff25c1293806e97ea10e63c01656e9d1","node_id":"5de37956-70f9-44be-bca1-5553fbf09860"},{"tags":["tendrl/integration/gluster","detected_cluster/6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","tendrl/node_d2e5b057-99bc-47e0-b83f-e3183fda480f","gluster/server","tendrl/integration/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node"],"updated_at":"2018-03-12 12:39:34.666497+00:00","first_sync_done":"yes","node_id":"d2e5b057-99bc-47e0-b83f-e3183fda480f","status":"UP","hash":"1aba532b5019a2d930ce2812242ea380","sync_status":"done","fqdn":"ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com","ipv4_addr":"10.37.169.148","is_managed":"yes","last_sync":"2018-03-12 12:39:34.581028+00:00","node_id":"d2e5b057-99bc-47e0-b83f-e3183fda480f"},{"updated_at":"2018-03-12 12:39:30.142940+00:00","first_sync_done":"yes","node_id":"d2abbfbe-9c79-45b8-b31a-f2df1f480df4","is_managed":"yes","tags":["tendrl/integration/gluster","detected_cluster/6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","gluster/server","tendrl/integration/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node_d2abbfbe-9c79-45b8-b31a-f2df1f480df4","provisioner/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node"],"fqdn":"ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com","sync_status":"done","ipv4_addr":"10.37.169.152","last_sync":"2018-03-12 12:39:29.996227+00:00","status":"UP","hash":"facb043897841f58a75696f62e765415","node_id":"d2abbfbe-9c79-45b8-b31a-f2df1f480df4"},{"ipv4_addr":"10.37.169.156","tags":["tendrl/integration/gluster","detected_cluster/6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","gluster/server","tendrl/integration/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node_7d5123d7-2593-4939-836f-ebee9dc6deef","tendrl/node"],"sync_status":"in_progress","fqdn":"ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:48.809399+00:00","node_id":"7d5123d7-2593-4939-836f-ebee9dc6deef","status":"UP","hash":"a97ddb5fb1c367023ebf7d41134cdec3","first_sync_done":"yes","is_managed":"yes","last_sync":"2018-03-12 12:38:46.845360+00:00","node_id":"7d5123d7-2593-4939-836f-ebee9dc6deef"},{"hash":"124620c5f5c6761db6657bd5ead9ddce","sync_status":"done","fqdn":"ltrilety-usm2-gl6.usmqe.lab.eng.brq.redhat.com","updated_at":"2018-03-12 12:39:52.775004+00:00","node_id":"8e0399c4-9c74-44a3-84bf-eb209c50a313","status":"UP","tags":["tendrl/integration/gluster","detected_cluster/6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","gluster/server","tendrl/integration/2b0328ea-62a9-43db-9b56-84132c587ed3","tendrl/node_8e0399c4-9c74-44a3-84bf-eb209c50a313","tendrl/node"],"first_sync_done":"yes","ipv4_addr":"10.37.169.158","is_managed":"yes","last_sync":"2018-03-12 12:39:52.683004+00:00","node_id":"8e0399c4-9c74-44a3-84bf-eb209c50a313"}]}]}

[ltrilety]

example of alerts response:
[{"hash":"e58cb83e280d1b2d7fff32c795ff4ad6","severity":"INFO","resource":"brick_status","updated_at":"2018-03-12 09:19:51.263899+00:00","source":"gluster_integration","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_2/2 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_2/2"},"pid":"10519","alert_id":"6436c021-7adf-415d-8c75-cdca3a88fe4e","significance":"","time_stamp":"2018-03-12T09:19:28.883733+00:00","acked":"True","ackedby":"TENDRL","node_id":"9eefd5d3-db1a-4559-a27e-8373abfe5ffe","ack_comment":"["System acked"]","current_value":"Started","acked_at":"2018-03-12T09:19:34.662729+00:00","delivered":"True","alert_type":"STATUS","classification":"["node", "cluster"]"},{"significance":"","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","fqdn":"ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"node_id":"9eefd5d3-db1a-4559-a27e-8373abfe5ffe","time_stamp":"2018-03-12T09:19:39.368558+00:00","classification":"["node", "cluster"]","acked":"True","alert_type":"STATUS","resource":"brick_status","pid":"10519","ackedby":"TENDRL","source":"gluster_integration","ack_comment":"["System acked"]","acked_at":"2018-03-12T09:19:45.479981+00:00","delivered":"True","current_value":"Started","updated_at":"2018-03-12 09:20:13.441785+00:00","alert_id":"28d05e30-7436-46d9-9009-6c77ec8d0f99","hash":"89c9261d149796637f57a1cd1587b44c","severity":"INFO"},{"ackedby":"TENDRL","significance":"","acked_at":"2018-03-12T09:19:55.212031+00:00","pid":"10519","ack_comment":"["System acked"]","current_value":"Started","alert_id":"96abf2cd-5d05-48c8-897f-88f66ba31419","time_stamp":"2018-03-12T09:19:49.209805+00:00","alert_type":"STATUS","hash":"d93642459d7b6c960b519097bbe93d91","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","fqdn":"ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"resource":"brick_status","updated_at":"2018-03-12 09:20:14.572373+00:00","node_id":"9eefd5d3-db1a-4559-a27e-8373abfe5ffe","delivered":"True","severity":"INFO","classification":"["node", "cluster"]","acked":"True","source":"gluster_integration"},{"acked_at":"2018-03-12T11:34:01.573132+00:00","classification":"["node", "cluster"]","acked":"True","ack_comment":"["System acked"]","delivered":"True","severity":"INFO","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1"},"resource":"brick_status","pid":"10519","alert_type":"STATUS","ackedby":"TENDRL","significance":"","current_value":"Started","time_stamp":"2018-03-12T11:33:54.901943+00:00","hash":"ea688b8802c386dbcb539c79ae7b242a","updated_at":"2018-03-12 11:34:25.480576+00:00","source":"gluster_integration","alert_id":"ee46d597-b99a-4749-b651-f1ca87f125eb","node_id":"9eefd5d3-db1a-4559-a27e-8373abfe5ffe"},{"alert_id":"629cc575-e111-4801-a829-f16c6fdc0a5a","ack_comment":"["System acked"]","significance":"","classification":"["node", "cluster"]","resource":"brick_status","pid":"18227","severity":"INFO","updated_at":"2018-03-12 11:35:05.092019+00:00","ackedby":"TENDRL","delivered":"True","source":"gluster_integration","time_stamp":"2018-03-12T11:34:21.289477+00:00","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl3.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl3.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl3.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1"},"acked":"True","node_id":"6731a30c-016d-4dbd-90d1-19bac16fac27","current_value":"Started","acked_at":"2018-03-12T11:34:22.815885+00:00","alert_type":"STATUS","hash":"1fd331781d811afce31aaabcbba5a8de"},{"delivered":"True","severity":"INFO","classification":"["cluster"]","acked":"True","alert_type":"STATUS","tags":{"cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","message":"Status of peer: ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Disconnected to Connected","plugin_instance":"peer_ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com"},"source":"gluster_integration","ack_comment":"["System acked"]","node_id":"","current_value":"Connected","time_stamp":"2018-03-12T11:57:21.351657+00:00","resource":"peer_status","pid":"30984","updated_at":"2018-03-12 11:58:30.336782+00:00","alert_id":"592db0c4-4a51-4395-9bf1-c3d61e560008","acked_at":"2018-03-12T11:57:49.286923+00:00","hash":"0cb6e47acd170691645de5067243a5da","ackedby":"TENDRL","significance":""},{"current_value":"Started","pid":"9395","ackedby":"TENDRL","source":"gluster_integration","node_id":"e70db3d4-388a-4c39-a56a-79f61de624a6","ack_comment":"["System acked"]","alert_type":"STATUS","severity":"INFO","resource":"brick_status","acked":"True","alert_id":"d15d6dcf-08c3-4963-99ec-30e297f1cd26","significance":"","time_stamp":"2018-03-12T11:58:56.575655+00:00","delivered":"True","hash":"2f19cb761573c71cea40bc618436005b","classification":"["node", "cluster"]","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1"},"updated_at":"2018-03-12 11:59:50.905146+00:00","acked_at":"2018-03-12T11:59:02.623798+00:00"},{"acked_at":"2018-03-12T11:59:13.020615+00:00","delivered":"True","alert_id":"ed8c9963-8ed9-4afb-a6ad-d26ccc1b8b0a","current_value":"Started","alert_type":"STATUS","classification":"["node", "cluster"]","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_2/2 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_2/2"},"pid":"9395","resource":"brick_status","ack_comment":"["System acked"]","significance":"","source":"gluster_integration","node_id":"e70db3d4-388a-4c39-a56a-79f61de624a6","time_stamp":"2018-03-12T11:59:06.530464+00:00","hash":"015dfa61a404549e67849ec47f4a04e1","severity":"INFO","updated_at":"2018-03-12 11:59:47.918247+00:00","acked":"True","ackedby":"TENDRL"},{"classification":"["node", "cluster"]","source":"gluster_integration","acked_at":"2018-03-12T11:59:17.070032+00:00","severity":"INFO","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1"},"current_value":"Started","resource":"brick_status","updated_at":"2018-03-12 11:59:53.562921+00:00","acked":"True","ackedby":"TENDRL","ack_comment":"["System acked"]","delivered":"True","hash":"a6c637fcf109b2b95268e2d48e942715","alert_id":"98697494-c3bd-4af1-9ac8-5cc2b266f587","node_id":"32fb7392-3b50-4027-81f7-ccee554ed97e","significance":"","time_stamp":"2018-03-12T11:59:10.656849+00:00","alert_type":"STATUS","pid":"19799"},{"node_id":"e70db3d4-388a-4c39-a56a-79f61de624a6","ack_comment":"["System acked"]","significance":"","severity":"INFO","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","fqdn":"ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"acked":"True","source":"gluster_integration","pid":"9395","acked_at":"2018-03-12T11:59:24.365797+00:00","alert_type":"STATUS","classification":"["node", "cluster"]","resource":"brick_status","delivered":"True","hash":"6f1f47ceb1468ff251ffd1087b0d085d","ackedby":"TENDRL","current_value":"Started","updated_at":"2018-03-12 11:59:49.356596+00:00","alert_id":"87b338fe-cc07-48ba-8065-0daed44015f7","time_stamp":"2018-03-12T11:59:18.518706+00:00"},{"alert_type":"STATUS","resource":"brick_status","current_value":"Started","time_stamp":"2018-03-12T11:59:19.619398+00:00","acked_at":"2018-03-12T11:59:25.640880+00:00","delivered":"True","hash":"49e340da577b5ba36a948b2c86708c55","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_2/2 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_2/2"},"acked":"True","source":"gluster_integration","ackedby":"TENDRL","alert_id":"a8ae5fa0-0dc5-41d2-874c-8f27883250f6","node_id":"32fb7392-3b50-4027-81f7-ccee554ed97e","ack_comment":"["System acked"]","severity":"INFO","classification":"["node", "cluster"]","pid":"19799","updated_at":"2018-03-12 11:59:55.131788+00:00","significance":""},{"severity":"INFO","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","fqdn":"ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm1-gl4.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"resource":"brick_status","updated_at":"2018-03-12 11:59:52.167603+00:00","significance":"","hash":"a29d6e374af3443b1f3ae9f1f5e30790","acked":"True","acked_at":"2018-03-12T11:59:33.716245+00:00","delivered":"True","classification":"["node", "cluster"]","pid":"9395","source":"gluster_integration","alert_id":"51bf9b02-1269-4380-9865-0a652858a315","node_id":"e70db3d4-388a-4c39-a56a-79f61de624a6","alert_type":"STATUS","ackedby":"TENDRL","ack_comment":"["System acked"]","current_value":"Started","time_stamp":"2018-03-12T11:59:27.695233+00:00"},{"delivered":"True","alert_type":"STATUS","classification":"["node", "cluster"]","significance":"","current_value":"Started","acked_at":"2018-03-12T11:59:36.778192+00:00","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","fqdn":"ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"node_id":"32fb7392-3b50-4027-81f7-ccee554ed97e","time_stamp":"2018-03-12T11:59:30.602282+00:00","severity":"INFO","resource":"brick_status","alert_id":"affd5b19-9617-4d82-be6c-8992c4a26280","hash":"602d5d85d1ff64a52b2a89ce1dd3c3e4","pid":"19799","updated_at":"2018-03-12 12:00:33.213199+00:00","acked":"True","ackedby":"TENDRL","source":"gluster_integration","ack_comment":"["System acked"]"},{"alert_type":"STATUS","hash":"a33589346efaeaa248be464f59ddd4a9","severity":"INFO","alert_id":"037d292f-8132-41fc-b2a1-61b92791125f","current_value":"Started","acked_at":"2018-03-12T11:59:45.499086+00:00","delivered":"True","node_id":"32fb7392-3b50-4027-81f7-ccee554ed97e","pid":"19799","updated_at":"2018-03-12 12:00:31.674054+00:00","ack_comment":"["System acked"]","significance":"","classification":"["node", "cluster"]","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","fqdn":"ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm1-gl6.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"resource":"brick_status","acked":"True","ackedby":"TENDRL","source":"gluster_integration","time_stamp":"2018-03-12T11:59:39.497922+00:00"},{"alert_type":"STATUS","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","fqdn":"ltrilety-usm1-gl2.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_name":"volume_alpha_distrep_6x2","message":"Status of brick: ltrilety-usm1-gl2.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1 under volume volume_alpha_distrep_6x2 in cluster dbc52192-ce3b-4086-a16b-f536b3357827 changed from Stopped to Started","plugin_instance":"volume_volume_alpha_distrep_6x2|brick_ltrilety-usm1-gl2.usmqe.lab.eng.brq.redhat.com:/mnt/brick_alpha_distrep_1/1"},"resource":"brick_status","pid":"812","significance":"","time_stamp":"2018-03-12T11:59:48.030841+00:00","delivered":"True","hash":"7846560db421be6d5f71712d56651883","classification":"["node", "cluster"]","updated_at":"2018-03-12 12:00:34.663686+00:00","acked":"True","alert_id":"23ca9ebd-a12a-46e9-95d2-65a4a08a07ee","acked_at":"2018-03-12T11:59:49.629227+00:00","current_value":"Started","node_id":"6a82f3d7-ee8c-4301-95f5-f1d54e95dbed","ackedby":"TENDRL","source":"gluster_integration","ack_comment":"["System acked"]","severity":"INFO"},{"node_id":"","significance":"","current_value":"up","delivered":"True","severity":"INFO","classification":"["cluster"]","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_name":"volume_alpha_distrep_6x2","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_id":"655e851a-c1cc-4757-809b-7509716a573b","message":"State of volume: volume_alpha_distrep_6x2 changed from (degraded) to up","plugin_instance":"volume_volume_alpha_distrep_6x2"},"resource":"volume_state","acked_at":"2018-03-12T12:00:36.625810+00:00","source":"gluster_integration","alert_type":"STATUS","acked":"True","ackedby":"TENDRL","alert_id":"a03ced96-d8cf-4e99-bcd8-76a0769b412e","ack_comment":"["System acked"]","hash":"3d1f5b9d78850fe7966df68bc53e665f","pid":"812","updated_at":"2018-03-12 12:01:10.989693+00:00","time_stamp":"2018-03-12T12:00:11.031776+00:00"},{"acked":"True","significance":"","current_value":"up","time_stamp":"2018-03-12T12:00:23.499989+00:00","hash":"48499c4ca14cf5d70777299f446d3bb0","ackedby":"TENDRL","alert_id":"4f69b43a-542f-4ef4-868e-3d2df2e80c0c","ack_comment":"["System acked"]","updated_at":"2018-03-12 12:01:12.324700+00:00","source":"gluster_integration","delivered":"True","alert_type":"STATUS","classification":"["cluster"]","resource":"volume_state","node_id":"","acked_at":"2018-03-12T12:00:49.448495+00:00","severity":"INFO","tags":{"integration_id":"dbc52192-ce3b-4086-a16b-f536b3357827","volume_name":"volume_gama_disperse_4_plus_2x2","cluster_name":"gluster-8bbb8c073a5219ce5bcb919b0538fa4e65ac726a3ab38e97665558627b0a0b98","volume_id":"d7935c7e-10b3-4ddc-af6f-de91f0b4ab28","message":"State of volume: volume_gama_disperse_4_plus_2x2 changed from (degraded) to up","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2"},"pid":"18227"},{"significance":"","classification":"["cluster"]","pid":"10531","updated_at":"2018-03-12 12:25:34.949052+00:00","source":"gluster_integration","alert_id":"880ddc1d-ffaa-443c-b50a-688502355b53","node_id":"","acked_at":"","acked":"False","current_value":"Disconnected","delivered":"True","hash":"3d24adfc6a8cfe702bf27cbf17fe45b3","severity":"WARNING","ack_comment":"[]","time_stamp":"2018-03-12T12:24:07.346406+00:00","alert_type":"STATUS","tags":{"cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","message":"Status of peer: ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Connected to Disconnected","plugin_instance":"peer_ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com"},"resource":"peer_status","ackedby":""},{"significance":"","acked":"True","ackedby":"TENDRL","alert_id":"130b58cb-caa6-49b4-95ae-d39afa29d291","node_id":"","current_value":"Connected","acked_at":"2018-03-12T12:24:45.251741+00:00","delivered":"True","hash":"9bf57aa8d5534acd401c22335ff13710","severity":"INFO","classification":"["cluster"]","tags":{"cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","message":"Status of peer: ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Disconnected to Connected","plugin_instance":"peer_ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com"},"source":"gluster_integration","ack_comment":"["System acked"]","time_stamp":"2018-03-12T12:24:30.459842+00:00","alert_type":"STATUS","resource":"peer_status","pid":"19717","updated_at":"2018-03-12 12:25:40.382403+00:00"},{"acked":"True","ack_comment":"["System acked"]","significance":"","current_value":"Started","acked_at":"2018-03-12T12:25:09.726516+00:00","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"pid":"28373","updated_at":"2018-03-12 12:26:38.717491+00:00","alert_id":"cb517b50-9c25-402c-b6ce-fc87960cd251","alert_type":"STATUS","hash":"8d5ec2adba83ebffca06bc4ba690e261","severity":"INFO","resource":"brick_status","ackedby":"TENDRL","source":"gluster_integration","time_stamp":"2018-03-12T12:25:06.892349+00:00","delivered":"True","classification":"["node", "cluster"]","node_id":"d2e5b057-99bc-47e0-b83f-e3183fda480f"},{"alert_id":"a980bce8-b79c-46af-b3cf-c7c1edc964f7","node_id":"1a9a4334-de31-4eac-9077-7caa5648e2cf","significance":"","time_stamp":"2018-03-12T12:25:09.185863+00:00","classification":"["node", "cluster"]","acked":"True","ackedby":"TENDRL","acked_at":"2018-03-12T12:25:11.987892+00:00","alert_type":"STATUS","ack_comment":"["System acked"]","pid":"24421","source":"gluster_integration","hash":"af9405971ff4ff03fac49543ad9acc40","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"resource":"brick_status","current_value":"Started","delivered":"True","severity":"INFO","updated_at":"2018-03-12 12:25:39.124174+00:00"},{"tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl1.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"alert_type":"STATUS","hash":"e6b8d6f93728d79b2032e0db8c870147","severity":"INFO","pid":"28373","alert_id":"31556d0e-7ffe-49d0-a432-249575e95c2a","time_stamp":"2018-03-12T12:25:11.773200+00:00","delivered":"True","classification":"["node", "cluster"]","resource":"brick_status","source":"gluster_integration","acked_at":"2018-03-12T12:25:14.636684+00:00","acked":"True","ackedby":"TENDRL","node_id":"d2e5b057-99bc-47e0-b83f-e3183fda480f","ack_comment":"["System acked"]","significance":"","current_value":"Started","updated_at":"2018-03-12 12:26:40.444616+00:00"},{"pid":"24421","updated_at":"2018-03-12 12:26:46.257174+00:00","current_value":"Started","acked_at":"2018-03-12T12:25:17.220354+00:00","alert_type":"STATUS","resource":"brick_status","ack_comment":"["System acked"]","delivered":"True","hash":"3a4cbeda7922c8e27d176097981fdfa4","significance":"","time_stamp":"2018-03-12T12:25:14.471675+00:00","alert_id":"e0b19f2a-27db-4f19-aa8f-3434f71afc5e","node_id":"1a9a4334-de31-4eac-9077-7caa5648e2cf","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl2.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"acked":"True","ackedby":"TENDRL","source":"gluster_integration","severity":"INFO","classification":"["node", "cluster"]"},{"classification":"["cluster"]","pid":"28373","acked":"False","ackedby":"","significance":"","current_value":"Disconnected","hash":"3d30db9d7db78f2e28636de976530449","severity":"WARNING","tags":{"cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","message":"Status of peer: ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Connected to Disconnected","plugin_instance":"peer_ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com"},"updated_at":"2018-03-12 12:32:23.436994+00:00","source":"gluster_integration","node_id":"","ack_comment":"[]","acked_at":"","delivered":"True","time_stamp":"2018-03-12T12:30:35.799720+00:00","alert_type":"STATUS","resource":"peer_status","alert_id":"d003dc08-adc1-4b8f-91ea-8368c00062bc"},{"delivered":"True","alert_type":"STATUS","resource":"peer_status","acked":"True","node_id":"","time_stamp":"2018-03-12T12:30:46.376992+00:00","acked_at":"2018-03-12T12:31:04.953422+00:00","hash":"5568ac88fa7c2b0f9fbb13fd2da880cd","classification":"["cluster"]","ackedby":"TENDRL","source":"gluster_integration","significance":"","current_value":"Connected","updated_at":"2018-03-12 12:32:24.864727+00:00","alert_id":"d35cd3f5-5215-4c19-aa36-188318e06115","ack_comment":"["System acked"]","severity":"INFO","tags":{"cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","message":"Status of peer: ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Disconnected to Connected","plugin_instance":"peer_ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com"},"pid":"7965"},{"node_id":"d2abbfbe-9c79-45b8-b31a-f2df1f480df4","ack_comment":"[]","acked_at":"","hash":"09c0a0b7da8c98971b90d0084b24843a","pid":"28373","acked":"False","time_stamp":"2018-03-12T12:31:15.749702+00:00","alert_type":"STATUS","severity":"WARNING","significance":"","current_value":"Stopped","ackedby":"","source":"gluster_integration","alert_id":"c111d8de-58d8-4642-b723-85faf3785903","classification":"["node", "cluster"]","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Started to Stopped","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"resource":"brick_status","updated_at":"2018-03-12 12:32:20.325506+00:00","delivered":"True"},{"ackedby":"TENDRL","node_id":"d2abbfbe-9c79-45b8-b31a-f2df1f480df4","alert_type":"STATUS","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl3.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"pid":"4029","acked":"True","source":"gluster_integration","alert_id":"a20351dd-820d-4e18-8358-b5a5a84948da","hash":"b83f882ec892bfc7cf0e639be3413ded","classification":"["node", "cluster"]","resource":"brick_status","acked_at":"2018-03-12T12:32:03.201657+00:00","ack_comment":"["System acked"]","current_value":"Started","significance":"","time_stamp":"2018-03-12T12:32:00.320574+00:00","delivered":"True","severity":"INFO","updated_at":"2018-03-12 12:32:27.591637+00:00"},{"updated_at":"2018-03-12 12:34:21.925584+00:00","alert_id":"1260ecb0-f97f-4f7a-a2da-f6ec65e48928","ack_comment":"["System acked"]","delivered":"True","alert_type":"STATUS","severity":"INFO","significance":"","current_value":"Started","acked_at":"2018-03-12T12:33:24.804816+00:00","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_1/1"},"ackedby":"TENDRL","classification":"["node", "cluster"]","acked":"True","source":"gluster_integration","node_id":"7d5123d7-2593-4939-836f-ebee9dc6deef","time_stamp":"2018-03-12T12:33:22.002935+00:00","hash":"6245041ee41ae9bd63946030479d7b58","resource":"brick_status","pid":"19717"},{"severity":"INFO","acked":"True","ackedby":"TENDRL","alert_id":"0deb81a6-c77a-4049-8290-f2b15cf65427","ack_comment":"["System acked"]","time_stamp":"2018-03-12T12:33:26.821107+00:00","alert_type":"STATUS","updated_at":"2018-03-12 12:34:20.462874+00:00","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","fqdn":"ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_name":"volume_gama_disperse_4_plus_2x2","message":"Status of brick: ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2 under volume volume_gama_disperse_4_plus_2x2 in cluster 2b0328ea-62a9-43db-9b56-84132c587ed3 changed from Stopped to Started","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2|brick_ltrilety-usm2-gl5.usmqe.lab.eng.brq.redhat.com:/mnt/brick_gama_disperse_2/2"},"hash":"5fc5ae279d67ea2eca4acb9391d9ee4d","classification":"["node", "cluster"]","resource":"brick_status","node_id":"7d5123d7-2593-4939-836f-ebee9dc6deef","acked_at":"2018-03-12T12:33:29.580837+00:00","pid":"19717","source":"gluster_integration","significance":"","current_value":"Started","delivered":"True"},{"classification":"["cluster"]","resource":"volume_state","ackedby":"TENDRL","alert_id":"f5a33527-2a98-4777-b9cb-9583be2bbd2f","delivered":"True","hash":"1f8da28350f5cc7c6e161d80635dabcf","severity":"INFO","source":"gluster_integration","ack_comment":"["System acked"]","acked_at":"2018-03-12T12:35:15.321945+00:00","tags":{"integration_id":"2b0328ea-62a9-43db-9b56-84132c587ed3","volume_name":"volume_gama_disperse_4_plus_2x2","cluster_name":"gluster-6661d05430420c533a707cb027513bd93ddc695e9984c2d19235671b051203bf","volume_id":"ab24c82f-739d-4c5d-95de-4e63146409ea","message":"State of volume: volume_gama_disperse_4_plus_2x2 changed from (degraded) to up","plugin_instance":"volume_volume_gama_disperse_4_plus_2x2"},"updated_at":"2018-03-12 12:36:12.910838+00:00","pid":"4029","significance":"","current_value":"up","time_stamp":"2018-03-12T12:34:56.872047+00:00","alert_type":"STATUS","acked":"True","node_id":""}]

[ltrilety]

After a discussion moving this bug to monitoring-integration Tendrl/monitoring-integration#373