S3Proxy does not return x-amz-server-side-encryption and x-amz-server-side-encryption-aws-kms-key-id headers

[Benmartin92]

Issue

S3Proxy does not return the headers x-amz-server-side-encryption and x-amz-server-side-encryption-aws-kms-key-id which results in checksum validation failure in S3 clients:

Exception in thread "main" software.amazon.awssdk.core.exception.RetryableException: Data read has a different checksum than expected. Was 0x552e21cd4cd9918678e3c1a0df491bc3, but expected 0xcf11f39f138eb50ef4f7421706dfb29d. This commonly means that the data was corrupted between the client and service. Note: Despite this error, the upload still completed and was persisted in S3

As verification, I used AWS CLI which showed the difference between directly accessing the same bucket vs. via S3Proxy:

2024-12-12 12:46:46,968 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'ofWmwrKnaAh2g5fgwwpCWcwGC9FJG73zKdX8FoAJ8VJfNEMAqN249rXeAsFoUDPfekuDoVSjqsZ1FOei2fEElA==', 'x-amz-request-id': '13H87K132861JEC6', 'Date': 'Thu, 12 Dec 2024 11:46:47 GMT', 'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'arn:aws:kms:us-east-1::key/mrk-', 'ETag': '"035e5926480cd20795670103c4bd436c"', 'Content-Length': '0', 'Server': 'AmazonS3'}

2024-12-12 12:43:20,763 - MainThread - botocore.parsers - DEBUG - Response headers: {'Date': 'Thu, 12 Dec 2024 11:43:20 GMT', 'x-amz-request-id': '4442587FB7D0A2F9', 'ETag': '"0293a30ad0c1c17929e8f8bfbe7bfac5"', 'Content-Length': '0', 'Server': 'Jetty(11.0.24)'}

Note that when Server-side encryption with AWS Key Management Service keys (SSE-KMS) is applied, S3 clients (e.g. AWS CLI, AWS Java SDK V1 and V2) intelligently skip checksum validation if the above response haeders are returned. However, S3Proxy fails to do so therefore clients attempt to verify the checksum which will always fail as encryption alters ETags.

Workaround

As a quick workaround, we can tell the clients (serverSideEncryption(ServerSideEncryption.AWS_KMS)) that we are doing server side encryption. Here is a snippet using AWS SDK for Java 2.x:

 PutObjectRequest putObjectRequest = PutObjectRequest.builder()
            .bucket(bucketName)
            .key(keyName)
            .serverSideEncryption(ServerSideEncryption.AWS_KMS)
            .build();