Better File Uploads with Shrine: Metadata

This article is part of the “Better File Uploads with Shrine” series.

Shrine has very flexible and customizable support for saving file metadata. Whenever Shrine is about to upload a file, it extracts available metadata from the file, and adds it to the returned Shrine::UploadedFile object.

uploaded_file = uploader.upload(file)
uploaded_file #=> #<Shrine::UploadedFile>

uploaded_file.metadata #=>
# {
#   "filename" => "nature.jpg",
#   "mime_type" => "image/jpeg",
#   "size" => 2859343
# }

uploaded_file.original_filename #=> "nature.jpg"
uploaded_file.extension         #=> "jpg"
uploaded_file.mime_type         #=> "image/jpeg"
uploaded_file.size              #=> 2859343

Most file attachments libraries have support for saving file metadata into additional columns. However, that means that you need to have a database column for each metadata you want to save.

# This is lame
add_column :photos, :image_filename, :string
add_column :photos, :image_type,     :string
add_column :photos, :image_size,     :integer
add_column :photos, :image_width,    :integer
add_column :photos, :image_height,   :integer
# ...

Shrine takes a much simpler approach here. Since it uses a single <attachment>_data database column to save the serialized Shrine::UploadedFile object, any metadata included in this object will also get saved to the same column.

photo = Photo.create(image: file)
photo.image_data #=>
# {
#   "id": "ee08af.jpg",
#   "storage": "store",
#   "metadata": {
#     "filename" => "nature.jpg",
#     "mime_type" => "image/jpeg",
#     "size" => 2859343
#   }
# }

Furthermore, when you’re storing processed files alongside the main file, Shrine automatically extracts and saves metadata of each file:

class ImageUploader < Shrine
  plugin :derivatives

  Attacher.derivatives_processor do |original|
    magick = ImageProcessing::MiniMagick.source(original)

    {
      small:  magick.resize_to_limit!(300, 300),
      medium: magick.resize_to_limit!(500, 500),
      large:  magick.resize_to_limit!(800, 800),
    }
  end
end
photo.image_derivatives!
photo.image_data
# {
#   "id": "ee08af.jpg",
#   "storage": "store",
#   "metadata": { ... },
#   "derivatives": {
#     "small": { "id": "f9f67a.jpg", "storage": "store", "metadata": { ... } },
#     "medium": { "id": "5e27cf.jpg", "storage": "store", "metadata": { ... } },
#     "large": { "id": "ff1da8.jpg", "storage": "store", "metadata": { ... } },
#   }
# }

photo.image(:small).size  #=> 21496
photo.image(:medium).size #=> 98237
photo.image(:large).size  #=> 150383

MIME type

Shrine doesn’t have any mandatory dependency for extracting MIME type, so by default it is inherited from #content_type of the input file if available. However, this attribute on uploaded files is set by Rack from the Content-Type request header, which was set by the browser solely based on the file extension.

This means that by default Shrine’s mime_type metadata is not guaranteed to hold the actual MIME type of the file (e.g. the user can upload a PHP file with a .jpg extension). This might sound like Shrine is not secure by deafult, but you do get a warning in the console when #content_type is used. And in some scenarios this might be exactly what you want.

Shrine comes with a determine_mime_type plugin, which determines MIME type from file content. It uses tools that know how to read “magic headers” of files, and saves the result into mime_type (which can then be validated).

# Gemfile
gem "marcel"
Shrine.plugin :determine_mime_type, analyzer: :marcel
File.write("image.png", "<?php ... ?>") # PHP file with a .png extension
uploaded_file = uploader.upload(File.open("image.png"))
uploaded_file.mime_type #=> "text/x-php"

If the :marcel analyzer doesn’t suit you, you can choose a different analyzer, or even combine them:

Shrine.plugin :determine_mime_type, analyzer: -> (io, analyzers) do
  analyzers[:mimemagic].call(io) || analyzers[:file].call(io)
end

Image Dimensions

Image dimensions can be extracted by loading the store_dimensions plugin:

# Gemfile
gem "fastimage" # default analyzer
class ImageUploader < Shrine
  plugin :store_dimensions
end
image = image_uploader.upload(file)
image.metadata #=>
# {
#   "filename" => "nature.jpg",
#   "mime_type" => "image/jpeg",
#   "size" => 90423,
#   "width" => 500,
#   "height" => 400,
# }

image.width  #=> 500
image.height #=> 400

image.dimensions #=> [500, 400]

Custom metadata

In addition to built-in metadata, Shrine allows you to easily extract and save custom metadata, with the add_metadata plugin.

class DocumentUploader < Shrine
  plugin :add_metadata

  add_metadata :pages do |io|
    PDF::Reader.new(io.path).page_count
  end
end
pdf = document_uploader.upload(cv)
pdf.metadata #=>
# {
#   "filename" => "curriculum-vitae.pdf",
#   "mime_type" => "application/pdf",
#   "size" => 49234,
#   "pages" => 5
# }
pdf.pages #=> 5

Notice that it also generated a #pages reader method on the Shrine::UploadedFile object. I think it’s nice to be able to extend Shrine objects with methods that fit your domain.

If you’re using a tool which extracts multiple metadata at once, the add_metadata plugin supports returning a hash as well.

class VideoUploader < Shrine
  plugin :add_metadata

  add_metadata do |io|
    movie = FFMPEG::Movie.new(io.path)

    { "duration"   => movie.duration,
      "bitrate"    => movie.bitrate,
      "resolution" => movie.resolution,
      "frame_rate" => movie.frame_rate }
  end

  metadata_method :duration, :bitrate, :resolution, :frame_rate
end
video = video_uploader.upload(file)

video.duration   #=> 7.5
video.bitrate    #=> 481
video.resolution #=> "640x480"
video.frame_rate #=> 16.72

Storage metadata

In addition to extracting the metadata on your side, Shrine also gives the storage itself the ability to update the metadata after uploading. Some storages like filesystem and Amazon S3 won’t use this, but many other storage services extract file metadata during uploading.

For example, when you’re uploading images to Cloudinary, shrine-cloudinary will automatically update size, mime_type, width and height metadata values. This is especially useful if you’re processing the image on upload with Cloudinary, because then the metadata that Shrine extracted won’t match the uploaded file, since those were extracted before the upload.

uploaded_file = cloudinary_uploader.upload(image, upload_options: {
  format: "png",
  width:  800,
  height: 800,
  crop:   :limit,
})

uploaded_file.metadata #=>
# {
#   "filename" => "nature.jpg"
#   "mime_type" => "image/png",
#   "size" => 8584,
#   "width" => 800,
#   "height" => 600,
# }

Cloudinary also has the ability to automatically generate responsive breakpoints, and the ability to update the metadata allows the storage to store the information about the generated breakpoints.

uploaded_file = cloudinary_uploader.upload(image, upload_options: {
  responsive_breakpoints: {
    bytes_step: 20000,
    min_width: 200,
    max_width: 1000,
    max_images: 20,
  }
})

uploaded_file.metadata["cloudinary"]["responsive_breakpoints"] #=>
# [{
#   "breakpoints": {
#     {
#       "width": 1000,
#       "height": 667,
#       "bytes": 79821,
#       "url": "http://res.cloudinary.com/demo/image/upload/c_scale,w_1000/v1453637947/dog.jpg",
#       "secure_url": "https://res.cloudinary.com/demo/image/upload/c_scale,w_1000/v1453637947/dog.jpg"
#     },
#     ...
#   }
# }]

Some other storages that use the ability to update metadata include shrine-flickr, shrine-transloadit and shrine-uploadcare.

Summary

We’ve seen how Shrine automatically extracts metadata before upload, which is then stored into the same database column. You can determine MIME type and extract image dimensions from file content using a variety of tools, just by loading a corresponding plugin. You can also extract any custom metadata, and storage services can add their own metadata as well.

Janko Marohnić

Janko Marohnić

A passionate Ruby backend developer who fell in love with Roda & Sequel, and told Rails “it’s not me, it’s you”. He enjoys working with JSON APIs and SQL databases, while prioritizing testing, and always tries to find the best library for the job. Creator of Shrine and test.vim.

comments powered by Disqus