An Introduction To OpenResty (nginx + lua) - Part 1

Overview

Last week, I started moving middleware code out of a Go application and directly into nginx via the lua-nginx-module. This was simple code, but I'm using it as an stepping-stone for possibly writing/moving more code like this.

lua-nginx-module is an nginx module which makes it possible to handle http request directly in nginx using Lua. This is a powerful combination granting you great performance and the productivity of a dynamic language. Furthermore, because we normally run our Go servers behind nginx, this simplifies our stack.

Installation

It's possible to install lua-nginx-module by compiling it into nginx. However, I suggest you install OpenResty; the parent project of lua-nginx-module. This is nginx bundled with lua-nginx-module as well as other popular nginx/lua-nginx modules.

Installing OpenResty is pretty much the same as install nginx. In other words, it can be as simple as running ./configure && make && make install. Personally though, I like to configure it. You can run ./configure --help to see available options. Here's what I do:

./configure \
  --prefix=/opt/resty \
  --conf-path=/opt/resty/nginx.conf \
  --with-cc-opt="-I/usr/local/include" \
  --with-ld-opt="-L/usr/local/lib" \
  --with-pcre-jit \
  --with-ipv6 \
  --with-http_postgres_module \
  --with-http_gunzip_module \
  --with-http_secure_link_module \
  --with-http_gzip_static_module \
  --without-http_redis_module \
  --without-http_redis2_module \
  --without-http_xss_module \
  --without-http_memc_module \
  --without-http_rds_json_module \
  --without-http_rds_csv_module \
  --without-lua_resty_memcached \
  --without-lua_resty_mysql \
  --without-http_ssi_module \
  --without-http_autoindex_module \
  --without-http_fastcgi_module \
  --without-http_uwsgi_module \
  --without-http_scgi_module \
  --without-http_memcached_module \
  --without-http_empty_gif_module

As you can see, I like to disable modules I won't be using. If you plan on using memcached or mysql, you'll want to take out the relevant lines. I've disabled the redis and redis2 modules because the recommended lua-resty-redis is enabled by default (yes, that means there's a total of 3 redis modules, I don't know why).

On OSX, you'll probably need to install PCRE. The simplest approach is via brew install pcre.

Finally, it doesn't appear to be a common problem, but I ran into problems building luaJIT (which ./configure does). I don't remember the exact steps I took, but it had to do with installing GCC 4.2.1 and, I think, forcing it as my compiler by exporting CC to point to it.

After you run make install you can go into /opt/resty/ and find a few folders, and a bunch of files. I usually delete all the files except for nginx.conf and mime.types, but leave the folders.

The actual nginx binary is located at nginx/sbin/nginx. Unrelated to OpenResty (and the rest of this post), but you can run the nginx binary with the -V flag to see how it was configured. This is pretty handy when you jump on a system and aren't 100% sure what's available. The -T flag is also worth pointing out as it'll test your configuration. It's a great idea to check the status code of ./nginx -T as part your deployment (and abort said deployment on error).

Project Structure

One of the first things you're going to have to figure out is how to organize your projects so that you and your teamates can develop and test while keeping deployments straightforward. Keeping in mind that I just started, I'm going to explain how I did it.

The trick that I use relies on the fact that nginx can be started with a -p and -c flags. The first overwrites the prefix path nginx was configured with. The second tells nginx to use a specific configuration file (instead of the one it was configured to use). Assuming we've setup our code in ~/code/proj1, the first thing we want to do is add a develop.conf file to the root of the project. This is a standard nginx config file, but it doesn't have to be tweaked since it's only going to be used for development:

worker_processes  1;
daemon off;
error_log /dev/stdout warn;

events{
  worker_connections 32;
}

http {
  default_type  text/html;
  access_log off;

  server {
    listen 3000;
    include 'src/proj1.conf';
  }
}

There's nothing special here, so let's look at src/proj1.conf:

location / {
  content_by_lua_block {
    require("handler")()
  }
}

Even if you don't know Lua or how it integrates with nginx, you can hopefully guess that our traditional nginx location block is going to execute the above Lua block, which in turns requires and executes code located in a "handler" module/file.

For the sake of completeness, we'll add our basic "handler" module, but then we'll go back to tying together our configuration files and development environment. Here's handler.lua:

local function process()
  ngx.say('the spice must flow')
end

return process

It's important to understand and maintain the relationship between develop.conf and src/proj1.conf. Essentially, src/proj1.conf is a fundamental part of your application's logic and should work regardless of its environment. develop.conf is only used for development and is therefore the place where we can enable development flags and setup our paths.

We've already setup a few development flags. daemon off; prevents nginx from running in the background and error_log /dev/stdout warn; sends errors directly to our terminal.

We can start nginx via /opt/resty/nginx/sbin/nginx -c ~/code/proj1/develop.conf but if we try to load http://localhost:3000, we'll get a 500 error. If you look at your terminal, you should see a useful error message: the handler.lua, which we required, couldn't be found.

To solve this, we'll leverage the lua_package_path directive. This adds a directory to our search path. Change your develop.conf file:

...
  lua_package_path '${prefix}../../src/?.lua;;';
  server {
    listen 3000;
    include 'src/proj1.conf';
  }
}

We're telling Lua to search for our code relative to nginx's prefix configuration. (the extra ; appends the existing search path). Unfortunately, nginx's prefix path is /opt/resty, but we can change that with the -p flag. First though, you'll need to create some directories:

mkdir -p ~/code/proj1/test/nginx/logs

We can now launch nginx via:

/opt/resty/nginx/sbin/nginx -c ~/code/proj1/develop.conf -p ~/code/proj1/test/nginx/

And reload our page to get a proper result.

You can use any prefix path you want and adjust your lua_package_path accordingly. In a later part we'll cover how to write tests so you'll end up with test folder anyways.

Tweaks

Two last things. Either before or after the lua_package_path line, we can add lua_code_cache off;. This will cause nginx to reload our Lua files on each request. It's wonderful to enable in development (but horrible to do in production).

The above line doesn't trigger a reload on changes to our develop.conf or src/proj1.conf. To achieve that, I'm using a CoffeeScript file, named develop.coffee which launches nginx and watches these two files for changes:

fs = require('fs')
crypto = require('crypto')
exec = require('child_process').exec

# maybe some teammates have their binary somewhere else
path = process.env.RESTY_PATH || '/opt/resty/nginx/sbin/nginx'

cwd = process.cwd()
console.log "running #{path} -p #{cwd}/test/nginx/ -c #{cwd}/develop.conf"
nginx = exec("#{path} -p #{cwd}/test/nginx/ -c #{cwd}/develop.conf")
nginx.on 'exit', (status, a) -> process.exit(status)
nginx.stderr.on 'data', (data) -> console.log(data.toString().trim())
nginx.stdout.on 'data', (data) -> console.log(data.toString().trim())

# on osx I noticed multiple update events being triggered for a single change
# so we'll calculate the md5 content and only reload nginx if it's actually changed
watch = (path) ->
  last = null
  fs.watch path, ->
    hash = crypto.createHash('md5').update(fs.readFileSync(path)).digest("hex")
    return if hash == last
    process.kill(nginx.pid, 'SIGHUP')
    last = hash

watch(file) for file in ['develop.conf', 'src/proj1.conf']

You can now start your server by running coffee develop.coffee.