Photo upload api endpoint in Clojure

May 26, 2015 · 1252 words · 6 minutes read

tl;dr:

in memory (photo) file upload validation(file size and dimensions)
I’ve never written Clojure before
unhandled exceptions

Important note: this is my first take on functional programming and Clojure so you probably shouldn’t read anything written here as a guide for handling file uploads or validation.

Through the post, I’ll try stick to the least necessary amount of code, skip the parts that are explained in documentation of given dependencies and go with the top-down approach explaining the code. Full source code is available in the respository.

There are few objectives that I’d like to achieve and I’m using midje to make sure that everything works as expected. I want to create an API endpoint that will:

return response if photo size is above the limit without downloading whole file

    (fact "validates file upload size"
      (let [filecontent {:bytes-stream (input-stream (make-array Byte/TYPE 200001))
                         :content-type "image/png"
                         :filename     "test.png"}
            request (assoc
                      (mock/request :post "/photos")
                      :params {:file fiilecontent})
            response (subject/photos-routes request)]
        (:status response) => 422
        (-> response :body json/decode (get "errors")) => "File is too big, I can't take it anymore"))

validate photo dimensions without creating temporary object

    (fact "returns error about dimensions"
      (with-open [in (input-stream (file "test/photouploader/test/fixtures/image_too_small.png"))]
        (let [filecontent {:bytes-stream in
                           :content-type "image/png"
                           :filename     "test.png"}
              request (assoc
                        (mock/request :post "/photos")
                        :params {:file filecontent})
              response (subject/photos-routes request)]

          (:status response) => 422
          (-> response :body json/decode (get "errors") first) => "Wrong dimensions.")))

save photo file to the disk

    (fact "creates photo and returns its id"
      (with-open [in (input-stream (file "test/photouploader/test/fixtures/image.png"))]
        (let [filecontent {:bytes-stream in
                           :content-type "image/png"
                           :filename     "test.png"}
              request (assoc
                        (mock/request :post "/photos")
                        :params {:file filecontent})
              response (subject/photos-routes request)]

          (:status response) => 200
          (.exists (as-file "public/assets/test.png"))  => true
          (println (kcore/select photos))
          (-> (kcore/select photos) first :image_file_name) => "test.png"
          (-> response :body json/decode (get-in ["photo" "image_file_name"])) => "test.png"
          (io/delete-file "public/assets/test.png"))))))

save information about upload in database in a paperclip accessible format.

Route

Firstly, We’re going to create an API endpoint in liberator to handle POST request with a file:

(defn- handle-photo-upload [file _ctx]
  (let [photo (photos-db/create! {:image file})
        errors (:errors photo [])]
    (if (empty? errors)
      {:status 200 :body {:photo photo}}
      {:status 422 :body {:errors errors}})))

(defn render-response [ctx]
  (let [{:keys [status body]} ctx]
    (ring-response {:headers {}
                    :status  status
                    :body    (json/encode body)})))

(defresource upload-photo [file]
  :allowed-methods [:post]
  :available-media-types ["application/json"]
  :post! (partial handle-photo-upload file)
  :handle-created (partial render-response))

(defroutes photos-routes
  (POST "/photos" [file](upload-photo file)))

and here we stumble into the first issue: does Ring, on which liberator depends, help me with handling file uploads without downloading the whole file? Unfortunately that wasn’t the case because byte-array-store middleware provided with Ring (which can be used to handle multipart uploads) transforms files immediately to a byte array.

and here we stumble into the first issue, does Ring, on which liberator depends, help me with handling file uploads without downloading the whole file? Unfortunately that wasn’t the case because byte-array-store middleware provided with Ring(which can be used to handle multipart uploads) transforms files immediately to a byte array. We’d like to avoid that, so we need to create custom stream handler that’s going to return buffered input stream.

(defn stream-byte-array-store
  []
  (fn [item]
    (-> (select-keys item [:filename :content-type])
      (assoc :bytes-stream (IOUtils/toBufferedInputStream ^java.io.InputStream (:stream item))))))

Middlewares in Ring have to be passed into the site handler:

(def app
  (-> (routes photos-routes app-routes)
      (handler/site {:multipart {:store (stream-byte-array-store)}})))

Model

For the sake of simplicity, we’re going to implement the rest of the objectives in the model. Still going with the top-down approach, we know that we’ll want to:

perform validations
add record in database
save file on disk
return errors from validations

Let’s start with a main method method that’s going to be public and as you can recall, is used in the router to save a photo.

(defn create! [{:keys [image]}]
  (let [filename (:filename image)
        photo-validation (validate-photo image) ;; Perform validations
        errors (:errors photo-validation)
        photo {:image_file_name    filename
               :image_file_size    (count (:image photo-validation))
               :image_content_type "image/png"}]

    (if (empty? errors)
      (try
        (transaction
          (create-at-db! photo)
          (save-file (:image photo-validation) filename "public/assets/"))
          photo
        (catch Exception e
          (rollback)
          (throw (Exception. e))))
      (merge photo {:errors errors}))))

Right now, we’re ignoring the fact that the image_content_type and file location path are hardcoded, it can be easily fixed later by getting the configuration file.

In the let form firstly we perform validations, so let’s add a code that’s supposed to do that:

(defn- rename [obj old-name new-name]
  (merge {new-name (old-name obj)} (dissoc obj old-name)))

(defn- validate-photo [image]
  (-> {:errors [] :image image}
    (photo-presence-validator)
    (rename :image :file)
    (size-validator 200000)
    (rename :file :image)
    (dimensions-validator 200 200)))

Here, we’re doing a few of things

provide array where we’ll combine errors from validators
check if file is provided
rename temporarily the key where the image is so that the size-validator could be reused for other files uploads, not only images
validate dimensions of the photo

Before we start writing validators, it would be good to have a helper function to combine errors:

(defn add-error
  [obj msg & [additional-params]]
  {:pre [(not (nil? obj))
         (string? msg)]}
  (if-let [errors (:errors obj)]
    (merge obj {:errors (conj errors msg)} additional-params)
    {}))

Equipped with such function, we’re ready to start writing validators. At the very beginning we’ve to check for file presence. If we want to know if a file is available, we can just check if there’s any filename:

(defn- photo-presence-validator [response]
  (let [filename (:filename (:image response))]
    (if (empty? filename)
      (add-error response "Missing file")
      response)))

File validator on the other side, looks quite complex in respect to other methods because we’ve to perform byte by byte read from the stream and check if we’ve not exceeded the limit. We’d like to also save the stream while consuming it because if the file is below the limit, we’d like to use it in future validations.

(defn size-validator [response size]
  (if-let [file (:file response)]
    (let [bytes-stream (:bytes-stream file)]
      (with-open [in bytes-stream]
        (let [buffer (make-array Byte/TYPE 1)]
          (loop [g (.read in buffer)
                 r 0
                 full-file (conj [] (first buffer))]
            (if (> r size)
              (add-error response "File is too big, I can't take it anymore" {:file nil})
              (if (= g -1)
                (merge response {:file full-file})
                (recur
                  (.read in buffer)
                  (+ r g)
                  (conj full-file (first buffer)))))))))
    response))

Dimensions validation can be performed by just casting the stream as image thanks to ImageIO:

(defn dimensions-validator [response max-width max-height]
  (if-let [image (:image response)]
    (let [img (ImageIO/read ^InputStream (ByteArrayInputStream. (byte-array image)))
          width (.getWidth img)
          height (.getHeight img)]
      (if (and (>= height max-height) (>= width max-width))
        response
        (add-error response "Wrong dimensions.")))
    response))

That was the last validation needed to be performed before we could save a file. I’m using Korma to generete SQL queries

(defn create-at-db! [photo]
  {:pre [(empty? (get photo :errors []))]}
  (insert photos
    (values (merge photo {:image_updated_at (time-coerce/to-sql-time (time/now))}))))

and now the last part, saving the photo to the disk:

(defn- save-file [bytes-stream filename path]
  (with-open [out (FileOutputStream. (str path filename))]
    (.write out (into-array Byte/TYPE bytes-stream))))

exceptions are handled in #create! because we need to perform rollback if anything goes wrong with the file save.

Summary

There’re a lot of missing parts for the uploader to be production ready. As a beginner I’ve mostly struggled with the IO and not being a Java programmer wrapping my head around different kinds of bytes and streams wasn’t the pleasantest time. I’m still not sure if tests setup with midje is similar to something people do in applications at production but it worked well for such small and primitive example. I wish there were more books, blog posts about that from more experienced Clojure devs. And lastly, HUGE thanks to @annapawlicka for patience, guidance and helping me out with all kinds of beginner’s problems. 💙