Sep 2025
S3 announced support for conditional writes in August of 2024. Conditional writes allow distributed systems to safely read objects, modify them, and write them back to S3 without any additional dependencies. This greatly simplifies many read-heavy systems.
However, AWS’s documentation for this feature — especially in Go — is
terrible. In this post, I’ll show you how to issue conditional writes with v2 of
AWS’s Go SDK. I’ll also show you how to write integration tests with
testcontainers and MinIO.
If you’d rather jump straight to Github, all the code in this post is available
in akshayjshah/conditionalwrite.
Issuing a conditional write is as simple as setting the If-None-Match or
If-Match HTTP headers. With a small single-object client type:
1package conditionalwrite
2
3import (
4 "context"
5 "errors"
6 "io"
7 "net/http"
8
9 "github.com/aws/aws-sdk-go-v2/aws"
10 "github.com/aws/aws-sdk-go-v2/credentials"
11 "github.com/aws/aws-sdk-go-v2/service/s3"
12 "github.com/aws/smithy-go"
13)
14
15type ETag string
16
17const None ETag = ""
18
19type Client struct {
20 client *s3.Client
21 bucket string
22 key string
23}
24
25func (c *Client) Set(
26 ctx context.Context,
27 r io.Reader,
28 previous ETag) (ETag, error) {
29
30 input := &s3.PutObjectInput{
31 Bucket: aws.String(c.bucket),
32 Key: aws.String(c.key),
33 Body: r,
34 }
35 if previous == "" {
36 input.IfNoneMatch = aws.String("*")
37 } else {
38 input.IfMatch = aws.String(string(previous))
39 }
40 res, err := c.client.PutObject(ctx, input)
41 if err != nil {
42 return "", err
43 }
44 if res == nil || res.ETag == nil {
45 return "", errors.New("no ETag")
46 }
47 return ETag(*res.ETag), nil
48}
When clients are in a read-modify-write loop (usually called “optimistic concurrency control”), it’s important to distinguish concurrency control errors. To do this in Go, we must reach down into the Smithy package:
1func IsPreconditionFailed(err error) bool {
2 return getSmithyCode(err) == "PreconditionFailed"
3}
4
5func getSmithyCode(err error) string {
6 if err == nil {
7 return ""
8 }
9 var e smithy.APIError
10 if errors.As(err, &e) {
11 return e.ErrorCode()
12 }
13 return ""
14}
Of course, we’ll also need a way to construct a Client. I found AWS’s
authentication packages difficult to grok — the documentation rightly focuses
on production use cases, but I wanted to work with local object storage.
1func NewClient(endpoint, user, pw, region, bucket, key string) *Client {
2 c := s3.New(s3.Options{
3 Region: region,
4 BaseEndpoint: aws.String(endpoint),
5 DefaultsMode: aws.DefaultsModeStandard,
6 Credentials: credentials.NewStaticCredentialsProvider(
7 user,
8 pw,
9 "", /* session */
10 ),
11 UsePathStyle: true,
12 RequestChecksumCalculation: aws.RequestChecksumCalculationWhenSupported,
13 ResponseChecksumValidation: aws.ResponseChecksumValidationWhenSupported,
14 HTTPClient: &http.Client{
15 Transport: &http.Transport{},
16 },
17 })
18 return &Client{client: c, bucket: bucket, key: key}
19}
For tests, it’s also nice to have a method to create buckets:
1func (c *Client) CreateBucket(ctx context.Context) error {
2 _, err := c.client.CreateBucket(ctx, &s3.CreateBucketInput{
3 Bucket: aws.String(c.bucket),
4 })
5 if getSmithyCode(err) == "BucketAlreadyOwnedByYou" {
6 return nil
7 }
8 return err
9}
I don’t like mocking complex dependencies like S3, so I’d prefer to run an
S3-compatible object store in my tests. MinIO is widely used and has an
excellent testcontainers module, so it’s easy to integrate
and allows each test to have an isolated object store. This does mean that tests
require Docker, but I’m happy with that tradeoff.
Here’s a simple test that creates an object, updates it successfully, and then tries to update it with an outdated ETag:
1package conditionalwrite
2
3import (
4 "fmt"
5 "strings"
6 "testing"
7
8 "github.com/testcontainers/testcontainers-go/modules/minio"
9)
10
11func TestConditionalWrite(t *testing.T) {
12 // Requires a running (or socket-activated) Docker daemon.
13 const user, password = "admin", "password"
14 mc, err := minio.Run(
15 t.Context(),
16 "minio/minio:RELEASE.2025-07-23T15-54-02Z",
17 minio.WithUsername(user),
18 minio.WithPassword(password),
19 )
20 if err != nil {
21 t.Fatalf("start MinIO container: %v", err)
22 }
23 addr, err := mc.ConnectionString(t.Context())
24 if err != nil {
25 t.Fatalf("get MinIO connection string: %v", err)
26 }
27
28 c := NewClient(
29 fmt.Sprintf("http://%s", addr), // endpoint
30 user,
31 password,
32 "us-east-1", // region
33 "test", // bucket
34 "text.txt", // key
35 )
36
37 err = c.CreateBucket(t.Context())
38 if err != nil {
39 t.Fatalf("create bucket failed: %v", err)
40 }
41
42 etag, err := c.Set(t.Context(), strings.NewReader("one"), None)
43 if err != nil {
44 t.Fatalf("initial write failed: %v", err)
45 }
46
47 _, err = c.Set(t.Context(), strings.NewReader("two"), etag)
48 if err != nil {
49 t.Fatalf("overwrite with correct ETag failed: %v", err)
50 }
51
52 _, err = c.Set(t.Context(), strings.NewReader("three"), etag)
53 if err == nil {
54 t.Fatal("overwrite with incorrect ETag succeeded")
55 }
56 if !IsPreconditionFailed(err) {
57 t.Fatalf("expected PreconditionFailed error, got %v", err)
58 }
59}
The overhead of starting a MinIO container makes this test slow enough that I’d
consider skipping it when testing.Short() is set.
After 19 years, S3’s API has grown quite a bit: its Smithy model is a forty thousand line JSON file. Because there’s no distinction between commonly-used and long-tail endpoints, the generated Go package is frustratingly enormous and cumbersome. I’d never let types from this package leak into production code. Instead, I’d write a wrapper and enforce its usage with a custom linter.
That said, I’m glad that S3 finally supports conditional writes. Optimistic
concurrency control with If-Match is dead simple, doesn’t add any moving
parts, and is efficient enough for many read-dominated workloads.
All the code in this post is available on Github.