new feature: need exemplar support

[bigfish49]

As prometheus and grafana has supported exemplar feature, can mtail support this feature?
here is some doc related:
https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md
https://grafana.com/docs/grafana/latest/basics/exemplars/
https://prometheus.io/docs/prometheus/latest/feature_flags/#exemplars-storage
thanks!

[jaqx0r]

Oh sweet, yes we can. I've been wondering when they'd get around to it. Do you have an example use in mind? I could attach the entire log line but perhaps we want to only add a substring?

…

On Wed, 28 Sept 2022, 23:08 bigfish49, ***@***.***> wrote: As prometheus and grafana has supported exemplar feature, can mtail support this feature? here is some doc related: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md https://grafana.com/docs/grafana/latest/basics/exemplars/ https://prometheus.io/docs/prometheus/latest/feature_flags/#exemplars-storage thanks! — Reply to this email directly, view it on GitHub <#684>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXFX63YY7DDLIWH25N2YB3WAQ7N3ANCNFSM6AAAAAAQXYOPEA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[bigfish49]

In our use case, we need to attach substrings like trace-id, as we just want to link to the log context by using grafana.
So, i think we should support this by adding a substring. I suppose we could have done this in this way:

1. define some prefix like "exemplar_label_", "exemplar_value_", "exemplar_timestamp_",
2. when we find such variable in a pattern, we auto update exemplar using those values
3. e.g. the rule is written like this:
   counter my_counter
   /tid=(?P<exemplar_label_trace_id>[0-9A-Za-z]+)/ {
   my_counter++
   }
   the output metrics should be like this:
   my_counter 11 # {trace_id="KOO5S4vxi0o"} 0

in the example above, exemplar value is not define, so just default it to 0.