File Upload REST API with Go and Amazon S3

Among the things I dislike doing in web development, I think file upload is in the Top 3. It's not hard and lots of libraries in your favorite language will help you to that. But there are always some small annoyances to store files on servers. Now I deploy my code to Heroku, put my files on S3 and it's done. Far easier! The other day, I wanted to create a small HTTP service to upload files on S3 in Go but there's no packaged solution to do that, so I thought I would write an article about it. Here's how I did it.

Requirements

First we need a package to interact with Amazon Web Services. For this article I chose to use mitchellh/goamz from Mitchell Hashimoto (creator of Vagrant and HashiCorp). It's a fork of goamz/goamz. There is also an "official" package developed by Stripe with Amazon recently making it official, but it's marked as "incredibly experimental" so I prefered to use the more stable one.

For the HTTP part, I decided to use stack, my own framework.

go get -u github.com/mitchellh/goamz
go get -u github.com/nmerouze/stack

Upload Files on S3

I am not going to use a real S3 bucket to write my code, so this article will be written as an example on how to write web services in TDD with Go. First action would be to upload a file on S3. The test will start by initializing a fake S3 server and create the bucket:

package upload_test

import (
  "testing"

  "github.com/mitchellh/goamz/aws"
  "github.com/mitchellh/goamz/s3"
  "github.com/mitchellh/goamz/s3/s3test"
)

func startServer() (*s3test.Server, *s3.Bucket) {
  testServer, _ := s3test.NewServer(new(s3test.Config)) // Fake server
  auth := aws.Auth{"abc", "123", ""} // Fake credentials
  conn := s3.New(auth, aws.Region{Name: "faux-region-1", S3Endpoint: testServer.URL(), S3LocationConstraint: true}) // Needs to be true when testing
  bucket := conn.Bucket("foobar.com") // Fake bucket
  bucket.PutBucket(s3.Private) // Bucket creation
  return testServer, bucket
}

func TestPut(t *testing.T) {
  s, b := startServer()
  defer s.Quit() // Quit the fake server when the test is done
}

As we will have to start the server for each test, we can directly put the code in a function. Then we make a fake request to our web service.

func TestPut(t *testing.T) {
  // ...
  w := httptest.NewRecorder()
  r, _ := http.NewRequest("PUT", "/files/foo.txt", bytes.NewBufferString("foobar")) // We send a file named foo.txt with the content "foobar"
  r.Header.Set("Content-Type", "text/plain")
  upload.Service(b).ServeHTTP(w, r) // "upload" is the name of our package
}

Then we need to see if things go well. The status must be 200 (OK), we want to set the Location header to the URL of the file on S3, we want to check if the file has been created and finally to see if the JSON body is correct.

func TestPut(t *testing.T) {
  // ...
  expCode := 200
  if w.Code != expCode {
    t.Fatalf("Response status expected: %#v, got: %#v", expCode, w.Code)
  }

  expLoc := b.URL("/foo.txt") // We get the full URL of the file stored in the bucket
  if w.Header().Get("Location") != expLoc {
    t.Fatalf("Response Location header expected: %#v, got: %#v", expLoc, w.Header().Get("Location"))
  }

  expBody := fmt.Sprintf(`{"url":"%s"}`, expLoc)
  if w.Body.String() != expBody {
    t.Fatalf("Response body expected: %#v, got: %#v", expBody, w.Body.String())
  }

  data, _ := b.Get("/foo.txt") // We get the file stored in the bucket
  if string(data) != "foobar" {
    t.Fatalf("Stored file content expected: %#v, got: %#v", "foobar", string(data))
  }
}

Now we can see our test fail with go test ./.

# github.com/nmerouze/stack-examples/upload_test
./upload_test.go:32: undefined: upload.Service
FAIL  github.com/nmerouze/stack-examples/upload [build failed]

So now we know what to expect, let's make our first action. First we define the router of our service:

package upload

import (
  "net/http"

  "github.com/mitchellh/goamz/s3"
  "github.com/nmerouze/stack/mux"
)

func Service(bucket *s3.Bucket) http.Handler {
  m := mux.New()
  return m
}

Let's run our test again.

--- FAIL: TestPut (0.00 seconds)
  upload_test.go:36: Response status expected: 200, got: 404
FAIL
FAIL  github.com/nmerouze/stack-examples/upload 0.017s

Great our first expectation is visible, let's add the code for that:

package upload

import (
  "net/http"

  "github.com/mitchellh/goamz/s3"
  "github.com/nmerouze/stack/mux"
)

// All our handlers are methods of this struct so they are able to access the bucket.
type appContext struct {
  bucket *s3.Bucket
}

func (c *appContext) upsertFile(w http.ResponseWriter, r *http.Request) {
}

func Service(bucket *s3.Bucket) http.Handler {
  c := &appContext{bucket}
  m := mux.New()
  m.Put("/files/*path").ThenFunc(c.upsertFile)
  return m
}

What does our test says?

--- FAIL: TestPut (0.00 seconds)
  upload_test.go:41: Response Location header expected: "http://127.0.0.1:52356/foobar.com/foo.txt", got: ""
FAIL
FAIL  github.com/nmerouze/stack-examples/upload 0.016s

Ok we pass the status expectation. Now we need to write our handler. It will read the request body (content of the file), get the URL path (path of the file) and put it on S3. Then we write the Location header and the response body.

func (c *appContext) upsertFile(w http.ResponseWriter, r *http.Request) {
  content, _ := ioutil.ReadAll(r.Body)
  path := mux.Params(r).ByName("path")
  c.bucket.Put(path, content, r.Header.Get("Content-Type"), s3.PublicRead)
  url := c.bucket.URL(path)
  w.Header().Set("Location", url)
  w.Header().Set("Content-Type", "application/json")
  fmt.Fprintf(w, `{"url":"%s"}`, url)
}

The test should pass:

ok    github.com/nmerouze/upload  0.022s

I used ioutil.ReadAll() to read the body, but a better implementation would be to use io.Copy(). For more information, read this article.

There we have our upload action. We didn't handle the errors at all. Most of the possible errors aren't really testable. Though one thing will trigger a panic: no request body. In this case ioutil.ReadAll() will panic and we need to handle this case. The test will be similar to the previous one but we will just put nil instead of the body and change the expectations a bit:

func TestPutBodyNil(t *testing.T) {
  s, b := startServer()
  defer s.Quit()

  w := httptest.NewRecorder()
  r, _ := http.NewRequest("PUT", "/files/foo.txt", nil)
  upload.Service(b).ServeHTTP(w, r)

  expCode := 400
  if w.Code != expCode {
    t.Fatalf("Response status expected: %#v, got: %#v", expCode, w.Code)
  }

  expBody := "Body must be set\n"
  if w.Body.String() != expBody {
    t.Fatalf("Response body expected: %#v, got: %#v", expBody, w.Body.String())
  }
}

If we run the test, the code will panic showing we have a problem. So let's return an error if there is no request body:

func (c *appContext) upsertFile(w http.ResponseWriter, r *http.Request) {
  if r.Body == nil {
    http.Error(w, "Body must be set", http.StatusBadRequest)
    return
  }

  // ...
}

We check if the body is nil at the top of the handler and return an error if it is. The test will pass and we will have handle the main problem. There are other errors but I didn't find any simple solution to reproduce them in tests because most values passed to them will not result in errors. We could test them with mocking and stubbing but it would require more complexity in the code and for the purpose of this article, I don't think it would be useful. Let's handle the errors in the code anyway:

func (c *appContext) upsertFile(w http.ResponseWriter, r *http.Request) {
  if r.Body == nil {
    http.Error(w, "Body must be set", http.StatusBadRequest)
    return
  }

  // I didn't find what kind of body would give an error, but let's write the error code anyway.
  content, err := ioutil.ReadAll(r.Body)
  if err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

  // If S3 has any problem with our request, it will return an error. It could also be a 500 error if S3 itself has a problem.
  path := mux.Params(r).ByName("path")
  err = c.bucket.Put(path, content, r.Header.Get("Content-Type"), s3.PublicRead)
  if err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

  url := c.bucket.URL(path)
  w.Header().Set("Location", url)
  w.Header().Set("Content-Type", "application/json")
  fmt.Fprintf(w, `{"url":"%s"}`, url)
}

Every possible error is handled. Our endpoint is now complete. The code is on github.

Get and Delete files

I am not going to write anything in the article about these endpoints because there are trivial, but you can find them on github.

Binary

Now we have our package but to run the program we need to create a bainary (a file with a main function). The function will initialize a connection to S3 and pass a bucket to the app. I create a main.go file in cmd directory in my package to do so.

package main

import (
  "net/http"

  "../../upload"
  "github.com/mitchellh/goamz/aws"
  "github.com/mitchellh/goamz/s3"
)

func main() {
  auth, err := aws.EnvAuth()
  if err != nil {
    panic(err)
  }

  server := s3.New(auth, aws.USEast)
  b := server.Bucket("foobar.com")
  h := upload.Service(b)
  http.ListenAndServe(":8080", h)
}

goamz has a neat function to read the credentials from the environment, pretty useful! Replace foobar.com with an existing bucket in your S3 account and you're done with this file.

To run this code on a development machine, we are going to create a .env at the root file with our credentials of the package.

AWS_ACCESS_KEY_ID=123 AWS_SECRET_ACCESS_KEY=abc

And a Makefile to automate the command:

run:
  env $$(cat .env) go run cmd/main.go

Here we have a web service that can run with the command make run and upload files on our S3 account. When it runs, just test it with curl.

curl -d 'foobar' -X PUT -H 'Content-Type: text/plain' http://localhost:8080/files/foo.txt

foo.txt should be in the selected bucket with the content foobar.

Conclusion

Uploading files to S3 is very concise. Building a web service to do the job is also very straightforward. But this article does not deal with security. I plan to write an article about authentication and JSON Web Tokens. Something like a middleware that can be added to the routes of this service to secure the endpoints.