Fix S3 PutObject missing Content-Length header and add E2E tests

[Copilot]

Fix S3 PutObject Missing Content-Length Header

• Understand the issue: S3 PutObject requires Content-Length header which is missing when using io.Reader
• Review current implementation in fs/s3.go Write method
• Fix the Write method to buffer the content and set ContentLength explicitly
• Verify code compiles successfully
• Run code review and address feedback
• Attempt security checks (CodeQL timed out but manual review completed)
• Document design decisions in memory
• Add E2E integration tests via LocalStack
• Address code review feedback
• Fix GitHub Actions test failures

Summary

Fixed S3 PutObject error "MissingContentLength: You must provide the Content-Length HTTP header" and added comprehensive E2E tests with LocalStack to validate the fix.

Bug Fix

• Modified fs/s3.go Write method to buffer content and explicitly set ContentLength

E2E Tests Added

• LocalStack service in docker-compose.yml for AWS S3 emulation
• Comprehensive test suite (s3_e2e_test.go) with 4 test scenarios testing TeeReader chains
• Tests run against both LocalStack and MinIO with correct credentials for each
• README documentation for running E2E tests
• CI automatically runs E2E tests via existing docker-compose setup

GitHub Actions Fix

• Fixed E2E test credentials: LocalStack uses "test"/"test", MinIO uses "minioadmin"/"minioadmin"

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Fix S3 Put</issue_title>
<issue_description>```
time: 2026-02-20T07:56:55.735456933Z
error: "error saving artifacts: error saving artifact to db: error writing
artifact(playbooks/019c7a0d-3f9e-1193-409a-5a632b510aff/logs.json): operation
error S3: PutObject, https response error StatusCode: 411, R
api error MissingContentLength: You must provide the Content-Length HTTP
header."
trace: 01KHX0TH1R8R3EC518D47MAHRK
context:
name: opensearch-pod-logs
run_id: 019c7a0d-3f9a-3dfb-6a9f-49b80f053f4d
action_id: 019c7a0d-3f9e-1193-409a-5a632b510aff
namespace: monitoring
action_name: Query logs
playbook_id: bef6904d-c187-47dc-b672-b151d5e2c12c
stacktrace: >-
Oops: error saving artifacts
--- at /app/playbook/runner/exec.go:106 executeAction()
--- at /app/playbook/runner/runner.go:359 ExecuteAndSaveAction()
--- at /app/playbook/runner/runner.go:486 TemplateAndExecuteAction()
--- at /app/playbook/runner/runner.go:433 RunAction()
--- at /app/playbook/run_consumer.go:311 ActionConsumer.func1()
--- at /go/pkg/mod/github.com/flanksource/duty@v1.0.1167/context/context.go:342 Context.Transaction.func1()
--- at /go/pkg/mod/gorm.io/gorm@v1.31.1/finisher_api.go:654 DB.Transaction()
--- at /go/pkg/mod/github.com/flanksource/duty@v1.0.1167/context/context.go:332 Context.Transaction()
--- at /app/playbook/run_consumer.go:290 ActionConsumer()
--- at /go/pkg/mod/github.com/flanksource/duty@v1.0.1167/postq/pg_consumer.go:76 PGConsumer.ConsumeUntilEmpty()

## Comments on the Issue (you are @copilot in this section)

<comments>
</comments>

• Fixes Fix S3 Put #39

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[moshloop]

@copilot add E2E integration tests via localstack

[Copilot]

@copilot add E2E integration tests via localstack

Added comprehensive E2E integration tests with LocalStack in commit 9339f3a.

What's included:

• LocalStack service in docker-compose.yml (S3 on port 4566)
• New test file fs/s3_e2e_test.go with 4 test scenarios:
  • Simple write operations
  • Write with TeeReader (simulates checksum calculation)
  • Write with multiple TeeReaders (full artifact processing pipeline)
  • Large file handling (1MB)
• Tests run against both LocalStack and MinIO for compatibility
• README documentation in fs/testdata/README.md with setup and usage instructions

The tests specifically validate the Content-Length fix works correctly with io.Reader chains as used in the artifact system.

[moshloop]

@copilot fix failing github actions

[Copilot]

@copilot fix failing github actions

Fixed in commit a110baa. The E2E test was using incorrect credentials for MinIO - it was using "test"/"test" (LocalStack credentials) instead of "minioadmin"/"minioadmin" (MinIO credentials). Updated the test to pass correct credentials for each service.