Publishing a Static Website on CloudFront

My first CloudFront article deals with basic S3 and CloudFront setup and with the DefaultRootObject which is the key of hosting a website on Amazon's CDN.

This second post focuses about getting your website on CloudFront by providing maintenance scripts to handle publishing and invalidation. Most of the scripts are generic, but the Rakefile targets a Jekyll-generated static site.

The Rakefile

First of all, here's my simple Rakefile. Don't try to use it yet, there's a lot of stuff missing.

task :default => :server

desc 'Start server with --auto'
task :server do
  jekyll('--server --auto')
end

desc 'Build site with Jekyll'
task :build do
  jekyll('--no-future')
end

desc 'Build and deploy'
task :publish => :build do
  bucket = 'MYBUCKET'
  puts "Publishing site to bucket #{bucket}"
  sh 'ruby aws_cf_sync.rb _site/ ' + bucket
end

def jekyll(opts = '')
  sh 'rm -rf _site/*'
  sh 'jekyll ' + opts
end

There's only 3 commands available:

• server: run the Jekyll server on localhost
• build: generate the website in Jekyll's _site folder
• publish: sync the _site folder on a S3 bucket and invalidate its content

The publish task is the only one I'll talk about, since the two others are really basic. Before continuing, make sure you have replaced "MYBUCKET" by your own S3 bucket.

Directory synchronization

I've written a script that wraps the s3cmd "sync" command with an invalidation tool to help with updates:

local   = ARGV[0]
s3_dest = ARGV[1]

if local == nil || s3_dest == nil
  puts "syntax aws_cf_sync.rb local_source s3_dest"
  exit
end

config = "#{Dir.pwd}/s3.config"
if !File.exists?(config)
  puts "please setup your s3.config file"
  exit
end

invalidate = "#{Dir.pwd}/aws_cf_invalidate.rb"
if !File.exists?(invalidate)
  puts "please download the aws_cf_invalidate.rb script"
  exit
end

s3_dest   = s3_dest.split('/')
s3_bucket = s3_dest.shift
s3_path   = s3_dest.join('/')

s3_path += '/' unless s3_dest.length == 0

%x[ $(which s3cmd) -c #{config} sync #{local} s3://#{s3_bucket}/#{s3_path} --acl-public ]

files = %x[ cd _site && find . -type f ].split("\n").map do |f|
  s3_path + f[2,f.length]
end

%x[ ruby #{invalidate} #{files.join(' ')} ]

To run this script, s3cmd has to be installed. On OSX, you can do so easily with homebrew:

$ brew install s3cmd

Now, create an s3.config file (God I hate these) with the s3cmd --configure command or copy the following configuration. Don't forget to set your own AWS credentials (rows #2,3).

[default]
access_key = S3_ACCESS_KEY
secret_key = S3_SECRET_KEY
acl_public = False
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2008-06-30/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
force = False
get_continue = False
gpg_command = None
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)sgpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)sgpg_passphrase = 
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
list_md5 = False
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0 
recursive = False
recv_chunk = 4096
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
urlencoding_mode = normal
use_https = False
verbosity = WARNING

You can test your s3cmd configuration by typing this command to list all your buckets:

$ s3cmd -c s3.config ls

Cache invalidation

The sync script invalidates the cache of the published objects by calling this script:

require 'rubygems'
require 'hmac-sha1'
require 'net/https'
require 'base64'

s3_access='S3_ACCESS_KEY'
s3_secret='S3_SECRET_KEY'
cf_distribution='CLOUDFRONT_DISTRIBUTION_ID'

if ARGV.length < 1
  puts "usage: aws_cf_invalidate.rb file1.html dir1/file2.jpg ..."
  exit
end

paths = '<Path>/' + ARGV.join('</Path><Path>/') + '</Path>'

date = Time.now.utc
date = date.strftime("%a, %d %b %Y %H:%M:%S %Z")
digest = HMAC::SHA1.new(s3_secret)
digest << date

uri = URI.parse('https://cloudfront.amazonaws.com/2010-08-01/distribution/' + cf_distribution + '/invalidation')

req = Net::HTTP::Post.new(uri.path)
req.initialize_http_header({
  'x-amz-date' => date,
  'Content-Type' => 'text/xml',
  'Authorization' => "AWS %s:%s" % [s3_access, Base64.encode64(digest.digest)]
})

req.body = "<InvalidationBatch>" + paths + "<CallerReference>ref_#{Time.now.utc.to_i}</CallerReference></InvalidationBatch>"

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = http.request(req)

puts res.code
puts res.body

Note that you have to set your CloudFront distribution ID and your AWS credentials in the previous script too. Sorry for that, but I didn't feel like refactoring around this painful s3cmd config file.

Summary and publishing

In your Jekyll working directory, you should now have these files:

• Rakefile: rake tasks to easily manage your site
• aws_cf_sync.rb: script that syncs your local files with your S3 bucket
• aws_cf_invalidate.rb: script that invalidates the cache of the updated files
• s3.config: the configuration of your s3 bucket

Before publishing, I suggest you add these exclusions in your _config.yml file:

exclude: [ Rakefile, aws_cf_sync.rb,
           aws_cf_invalidate.rb, s3.config ]

If you've been a good reader and followed every instruction, all you have to do to publish your website on your S3 bucket and invalidate the CloudFront cache is:

$ rake publish

You can now fire up your browser and refresh frantically until you see your changes.