parse_aws_alb_log failing with new AWS field

[Alex-Waring]

AWS have added a new Transaction ID field to their ALB logs, and have not announced this so it's not in their docs yet. However for those who are on the rollout plan, the parse_aws_alb_log function is failing:

Vector unable to parse alb log: function call error for "parse_aws_alb_log" at (18:45): Log should be fully consumed: " TID_XXXX"

The parse_aws_alb_log function should have added to it the ability to conditionally parse this entry, if that's possible, until AWS confirm the rollout is complete.

To workaround this I have added:

strip_tid, err = split(.message, " TID_")[0]
if err != null {
    .@error = true
    .@error_msg = "Vector unable to parse alb log: " + err
} else {
    structured, err = parse_aws_alb_log(strip_tid)
    if err != null {
        .@error = true
        .@error_msg = "Vector unable to parse alb log: " + err
    } else {
        . = merge(., structured)
    }
}

[jszwedko]

Agreed. Do you have an example of one of the new logs @Alex-Waring ?

[Alex-Waring]

I have a heavily redacted version yes:

https 2024-05-22T03:48:29.047890Z app/lbid 1.1.1.1:44444 - -1 -1 -1 400 - 38 370 "GET http://2.2.2.2:443/ HTTP/1.1" "-" - - - "-" "-" "-" - 2024-05-22T03:48:29.047000Z "-" "-" "-" "-" "-" "-" "-" TID_[a-z0-9]{32}

It's identical to the old setup with the TID tacked on the end

[Alex-Waring]

Just to confirm, we should be ignoring these appended logs. This is from AWS:

When new fields are introduced, they are added to the end of the log entry. You should ignore any fields at the end of the log entry that you were not expecting.

[jszwedko]

Just to confirm, we should be ignoring these appended logs. This is from AWS:

When new fields are introduced, they are added to the end of the log entry. You should ignore any fields at the end of the log entry that you were not expecting.

Aaah, that's a good note. We should make the parsing resilient to unexpected trailing fields.

[nhlushak]

Hey, we have also been affected by this change from AWS. Do you have any plans to patch this behaviour in nearest time?

[mzupan]

for anyone dealing with this issue and waiting this is how I fixed it

. = parse_aws_alb_log!(slice!(.message, 0, -37))