Running Gitlab in Docker

I spent a good bit of time working out all the bits and pieces needed to make this work, so I’m going to blog about it here as a way of documentation.

Purpose

I wanted to run Gitlab and host my personal code repos myself.  Gitlab is a pretty awesome project and the Docker container they provide is the perfect way to run this on my Ubuntu 14.04 server in my basement.  I’m a total Docker amateur and this was a good reason to also learn how to manage this with Ansible 2.0.  Feel free to tell me what I’m doing wrong in the comments – this is my first guess at the best way to do this and it does work.

Where I’m Installing This

This will run on an Ubuntu 14.04 server in my basement.  This means it is behind my home router and shares an IP address on both my local network and publicly with a bunch of other services, so Gitlab can’t just assume it is the only thing running.  This comes into plan when managing the ports Gitlab uses.

Running Gitlab CE Docker Container with Ansible

Some details removed for security, but this is essentially what I run in Ansible.

---
- name: Pull Gitlab Docker image
  docker_image:
  name: gitlab/gitlab-ce
- name: Run Gitlab Docker container
  docker_container:
  name: Gitlab
  image: gitlab/gitlab-ce
  detach: yes
  restart_policy: always
  published_ports:
    - "8085:80"
    - "4443:443"
    - "2225:22"
  hostname: gitlab.thelastcicada.com
  volumes:
    - /ssd/gitlab/config:/etc/gitlab
    - /ssd/gitlab/logs:/var/log/gitlab
    - /ssd/gitlab/data:/var/opt/gitlab
  env:
    SMTP_USER: [email protected]
    SMTP_DOMAIN: smtp.mailgun.org
    SMTP_PASS: xxxxxxxxxxxxxxxxxxxxxxxxxxx
  state: started

 
You’ll see that this container will be addressed on the Ubuntu server at port 2225 for SSH, port 8085 for HTTP, and port 4443 for HTTPS. I send environmental variables to Gitlab to use my Mailgun account for sending emails.

Routing Traffic

I use Nginx to proxy_pass HTTP and HTTPS to the Gitlab container. I also do SSL termination here using a LetsEncrypt cert.

I use my /etc/hosts file on my Ubuntu server to route traffic for gitlab.thelastcicada.com to 127.0.0.1 so I can access it from this same box.

Adding the Git Origin

When I first added the git origin (using SSH not HTTP), Gitlab asked for a password every time I tried to do a git push. This ended up being because I needed to specify port 2225 for the SSH connection instead of the default of port 22. I added the following in my ~/.ssh/config file to make any connection to this host use port 2225 for SSH:

Host gitlab.thelastcicada.com
HostName gitlab.thelastcicada.com
Port 2225

I’ll need to do this on any host that I use to push code to my Gitlab server. I could solve this by having Gitlab use port 22 and setting SSH to the main Ubuntu server to use a different pot, but we’ll see how much of a pain this turns out to be.

Conclusion

I haven’t lived with this long enough to know if this is going to be my final configuration. I’ll update this post if I decide that improvements need to be made.

PHP 7 is For Real

PHP 7 is not messing around when it comes to performance.  On this WordPress blog you are reading right now, just switching to PHP 7 reduced memory usage in the Media Uploader by 78%!

How I Came to Know This…

As you may know, I’m running this WordPress multisite install on a $0.99 VM from Atlantic.net.  I’ve been super happy with this server (which I’m lucky to have since the deal is no longer available) and have really enjoyed figuring out how to make everything work with only 256M of RAM (see my previous post for more on how I’ve done this).  This week, I thought this grand experiment was going to come to an end as I ran into error after error trying to upload multiple images though the WordPress Media Uploader.

The WordPress uploader is a known to use quite a bit of memory when using the php-gd library to resize uploaded images.  With my puny 256M, it is no wonder I was running into this.  I installed the free New Relic server monitoring to get some pretty graphs of what was going on and, sure enough, PHP 5.5 was going bonkers when I tried to upload multiple images.

Don't have to be a sysadmin to know bad things are happening here.
Don’t have to be a sysadmin to know bad things are happening here.

If you look at the Physical memory graph, you can see where this goes bad.  Once the red swap space starts growing, it is an indication the machine has filled up the RAM and has started using the disk in a desperate attempt to keep things running.  This essentially locks up the server as the disk is nowhere near fast enough for this job.  This is what the user experience was in the browser as well – 1 or 2 images would upload, and the rest would fail with a generic “HTTP Error” message in red after doing nothing for many minutes.

New Relic is pretty great in that you can see exactly what server process has used all your memory, and it confirms what we suspected: PHP is the culprit.

php5-fpm struggling to deal with bulk image resizing.
php5-fpm struggling to deal with bulk image resizing.

I did a lot of optimizing to try to get this memory usage down, including changing the WordPress memory limit, the PHP memory limit, and fiddling with the caching settings.  Nothing made a difference.

I started looking for other things I could try and seriously considered just upgrading to the $4.97 per month VM with 512M of RAM.  But I came across this article about Symphony framework and how much less memory certain code used on PHP 7 compared to PHP 5.6 and thought the upgrade was worth a shot.

It was a little bit messy getting PHP 7 on Ubuntu and I longed for CentOS 7 where I could use the excellent Remi repo, but using a Digital Ocean tutorial, I was able to get PHP upgraded and was ready to run another test.

I tried uploading 4 images to start.  The results were so unbelievable, I did another test with many many more images to make sure I hadn’t done my test wrong.  The Media Uploader on PHP 7 processed as many images as I could throw at it and the memory usage barely budged.  With PHP 5.5, the CPU spiked, the load went up, and a php5-fpm process sat using 80%+ of the RAM for minutes and minutes.  With PHP 7, the php7.0-fpm process never went past 15% memory usage and the CPU barely jumped at all.  Check out the telemetry from New Relic:

PHP 7 eating images for breakfast
PHP 7 eating images for breakfast

You can see the CPU spike as PHP 7 processes the images, but there’s hardly a bump in the RAM usage at all.

An unbelievable improvement
An unbelievable improvement

According to New Relic, PHP 7 used a maximum of 41 MB to process the dozen images I threw at it.  With PHP 5.5, memory usage topped out at 190 MB for the same job.  That’s an incredible improvement and it allows me to keep enjoying my $0.99 VM!

I knew PHP 7 was good, but I had no idea I’d see gains this dramatic.  If you run a WordPress server, go install PHP 7 right now – I’m a believer.

Solving the Python 2.6 issue for LetEncrypt on CentOS 6

The Problem

I wanted to use LetsEncrypt to install a free SSL certificate on a website (call it “mywebsite.com” for this example) that runs on a CentOS 6.7 VM.  I downloaded the LetsEncrypt client and ran my command to download a cert that I would configure in Nginx.

letsencrypt-auto certonly --agree-tos --webroot --webroot-path /var/www/html/mywebsite.com -d mywebsite.com

I had done this previously on Ubuntu 14.04 without any problem.  On CentOS 6.7, I was greeted by this error:

WARNING: Python 2.6 support is very experimental at present...
if you would like to work on improving it, please ensure you have backups
and then run this script again with the --debug flag!

It isn’t actually clear what this error means.  Did it actually do anything?  Is it just informational?  Turns out, what it is saying is that the LetsEncrypt client is not compatible with Python 2.6 – it requires Python 2.7 and above.  This is a big problem for CentOS 6 as the basic tools of the operating system (‘yum’ for instance) rely on Python 2.6.  You will completely bork your system if you replace Python 2.6 with Python 2.7.  This applies for RedHat Enterprise Linux (RHEL) 6 as well.

The Solution

Software Collections!  Good luck figuring out what these are from any of the RedHat documentation.  Software Collections are actually a way to install multiple versions of software on a CentOS or RHEL server without resorting to compiling from source or Docker containers.  What it means for us is that we can install Python 2.7 on this same server and only activate it when running LetsEncrypt.  The default Python version on the system will remain 2.6.

To get started, install software collections through yum

yum install centos-release-SCL

Check your version of Python to make sure it remains 2.6.x

[~]# python -V
Python 2.6.6

Install Python 2.7 via Software Collections (referred to as SCL from here on)

yum install python27

Check and make sure the default Python version is what you started with

[~]# python -V
Python 2.6.6

Now comes the fun stuff.  If you look in /opt/rh/python27/root/, you’ll see yum has created a whole system directory structure, complete with /etc, /var, /home, /bin, etc.  This is where your simulated system for Python 2.7 lives.  Your Python 2.7 configuration files, log files, binary files – these are all within this “fake” directory structure.  Python 2.7 lives entirely in here and will not impact anything in the normal system structure.

To activate and use Python 2.7, we need to run the commands through the SCL system like so:

scl enable python27 "/root/letsencrypt/letsencrypt-auto certonly  --agree-tos --webroot --webroot-path /var/www/html/mywebsite.com -d mywebsite.com"

The basic syntax of Software Collections is simple

scl <action> [<collection>...] <command>

We need to use quotes in this case to identify the entire letsencrypt client string and configuration flags as part of the single command.  To see a full list of what scl can do, use

scl --help

At this point, the LetsEncrypt client should run just like normal, using Python 2.7.  Your certificate will be downloaded and the LetsEncrypt client will exit without error.  If you want to do some debugging and have a shell with Python 2.7 as your default Python, try

scl enable python27 bash

In that shell, everything you do will use Python 2.7.  Simply type “exit” to jump back to the regular shell and Python 2.6.

Now that we know how to activate Python 2.7, we can automate certificate renewal via a cron job.  I put this in /etc/crontab, but it would work fine if put into “crontab -e”

03 3 * * * root scl enable python27 "/root/letsencrypt/letsencrypt-auto certonly --keep-until-expiring --agree-tos --quiet --webroot --webroot-path /var/www/html/mywebsite.com -d mywebsite.com"; service nginx reload

Software Collections are great for solving all sorts of problems, from running multiple versions of PHP on a single server to creating software testing environments.

Hope this helped and you are on your way to enjoying a free SSL certificate!

Other Resources

 

 

WordPress VM on $0.99 per month

I’ve been a big fan of Digital Ocean and their $5 Droplets (aka Virtual Machines). They are 100% SSD backed and speedy, and for $5 a month, you can afford to experiment. But when I borked my Droplet by screwing up a kernel update, I started thinking…. can I do this any cheaper than $5 per month?

This is when I discovered atlantic.net and their “GO” VPS package for $0.99 per month. $0.99! I had to try this out. Admittedly, the 10 GB of disk space and 256 MB of RAM is pretty tight, but I am strangely drawn to challenges of efficiency and really enjoy doing a lot with a little. So right now, this WordPress multisite install is running on a $0.99 per month server.

So far it has been very stable, but I have had to make some adjustments to keep from overrunning the limits of this box. First thing I reigned in was MySQL. Since this blog has very few posts, I could decrease the buffers to a very low value and still get good performance for my dataset. This would be much more challenging with more content and a larger database. Here’s what I have set in my.cnf – many of these variable could probably even go lower:

max_connections         = 50
connect_timeout         = 5
wait_timeout            = 600
max_allowed_packet      = 16M
thread_cache_size       = 86
sort_buffer_size        = 64k
bulk_insert_buffer_size = 64k
tmp_table_size          = 12M
max_heap_table_size     = 12M

myisam_recover          = BACKUP
key_buffer_size         = 32k
table_open_cache        = 400
myisam_sort_buffer_size = 32k
concurrent_insert       = 2
read_buffer_size        = 32k
read_rnd_buffer_size    = 32k

query_cache_limit               = 256K
query_cache_size                = 2M

default_storage_engine  = InnoDB

innodb_buffer_pool_size = 12M
innodb_log_buffer_size  = 2M
innodb_file_per_table   = 1
innodb_open_files       = 400
innodb_io_capacity      = 400
innodb_flush_method     = O_DIRECT

All of my tables are InnoDB, so I cut way back on any MyISAM buffers. Running mysqltuner.pl shows the maximum possible RAM usage to be 62.6 MB, about 25% of our RAM. Not too shabby.

 >>  MySQLTuner 1.3.0 - Major Hayden 
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
[OK] Logged in using credentials from debian maintenance account.
[!!] Currently running unsupported MySQL version 10.1.1-MariaDB-1~trusty-wsrep-log
[OK] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MRG_MyISAM
[--] Data in InnoDB tables: 11M (Tables: 69)
[!!] Total fragmented tables: 1

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 3m 56s (639 q [2.708 qps], 63 conn, TX: 282K, RX: 71K)
[--] Reads / Writes: 87% / 13%
[--] Total buffers: 36.0M global + 544.0K per thread (50 max threads)
[OK] Maximum possible memory usage: 62.6M (25% of installed RAM)
[OK] Slow queries: 0% (0/639)
[OK] Highest usage of available connections: 4% (2/50)
[!!] Key buffer size / total MyISAM indexes: 32.0K/121.0K
[OK] Query cache efficiency: 27.0% (127 cached / 470 selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 22 sorts)
[OK] Temporary tables created on disk: 20% (54 on disk / 266 total)
[OK] Thread cache hit rate: 96% (2 created / 63 connections)
[OK] Table cache hit rate: 52% (99 open / 187 opened)
[OK] Open file limit used: 5% (60/1K)
[OK] Table locks acquired immediately: 100% (236 immediate / 236 locks)
[OK] InnoDB buffer pool / data size: 12.0M/11.0M
[OK] InnoDB log waits: 0

I then checked /etc/memcached.conf (since I switched from CentOS to Ubuntu) and made sure Memcached was set to only use 64 MB of RAM (-m 64). In the future, I plan to run some tests to see how much RAM WordPress actually uses in Memcached from a clean install (my suspicion is it isn’t much).

Besides that, the tweaks are minor. The memory_limit in php.ini is 128M and pm.max_children = 8 for php-fpm. I have swap enabled on this VPS to allow the disk to be used for overflow.

With hardly anything installed, I’m already using 2.5 GB of the 10 GB disk. Thankfully I don’t upload much on here, so this should last me for a while.

The low RAM limits what you can realistically run, but it fits my needs nicely and I’ve so far been happy with it. Now don’t expect the modern design or fancy tools like you get from Digital Ocean, but if you don’t expect too much from Atlantic.net, you might not be let down.

Update: It was pointed out to me that the feature set in the Atlantic.net dashboard and the Digital Ocean dashboard are very similar and, after checking out the Atlantic.net dashboard, I can say that it does have a pretty similar list of features. The main tools I use (snapshots, console access, and password resets) are all there.

Microcaching with Nginx for WordPress

This is by no-means an authoritative guide on the subject, only a chronicle of what I learned setting this up for myself. This caching method is only appropriate for users on a VM or dedicated server with SSH access and moderate to advanced command line skills.

Purpose

I’ve found the Batcache plugin for WordPress offers excellent performance when Memcached or APC are available for the object cache. For nearly all scenarios, simply installing Batcache with the default values will be sufficient to provide a fast and scalable website.

However, while Batcache can store keys and values in Memcached or APC (or Redis or whatever you use for object caching in WordPress), the basic code for setting and retrieving keys from the cache is written in PHP. This means every time a page is served from Batcache, PHP and 1.12% of WordPress must be loaded. PHP is not the fastest program out there and just loading it creates a non-trivial load on a server. If you have a small server, this limits the amount of concurrent traffic you can serve as you will eventually run out of resources to load PHP processes. This is not an issue for the vast majority of users out there. However, for those running WordPress at high scale with popular content (especially viral content that will generate very spikey traffic surges), or those just trying to squeeze maximum performance out of a cheap VM, finding a way to serve full pages without loading PHP can net some pretty fantastic results.

At this point, somebody usually says “why not use Varnish?”. Varnish will indeed do full page caching without loading any PHP. But Varnish will not do SSL termination, and if you want to use SPDY (which I plan to at some point), you’ll need to have the entire site loaded with SSL. Since Nginx does SSL termination and has built-in caching capabilities, it made the most sense to try to figure out how to make it work.

The Concept

Batcache will handle caching with no problems at low, medium, and high scale situations. Only at very high scale will Batcache run into the limits of PHP. Therefore, if we can “rate-limit” the amount of requests reaching Batcache, we’ll avoid running into these limits at the extremely high scale. This is where the term “microcaching” comes from – we will use Nginx to serve full page caches only for an extremely short amount of time (5 seconds for example), essentially rate-limiting the amount of requests reaching WordPress and Batcache for any single page to 1 request every 5 seconds. If a page is experiencing a large traffic surge, this can make all the difference.

I considered using Nginx for the full page cache and dropping Batcache altogether, but the difficulty with this is you need to intelligently purge the cache when posts are added or updated. The plugin repo has a number of plugins that purport to do just this, but most of what I found was designed to work with Nginx in front of Apache (not my preferred setup) or using the fastcgi_cache_purge Nginx module, which isn’t included in some common Nginx repos. I was looking for something that would work without compiling from source and could be dropped in with minimal extra configuration. This solution allows us to forget about cache purges and invalidation in the Nginx cache (cache values expire too quickly to be a bother) and take advantage of the solid logic provided by Batcache for the main page cache.

The Most Important Thing

The Nginx fastcgi cache respects the expiration values set by the X-Accel-Expires/Expires/Cache-Control headers. These headers are set by WordPress and/or Batcache.

There are a number of ways to deal with this, but the easiest thing to do is just tell the Nginx cache to ignore these headers. Add this to /etc/nginx/nginx.conf:

fastcgi_ignore_headers Cache-Control Expires;

Nginx Setup

First we define where the cache files will be saved (use a tmpfs folder for even greater performance) with this setting in the http block in the nginx.conf file:

fastcgi_cache_path /etc/nginx/cache levels=1:2 keys_zone=thelastcicada:100m inactive=10m max_size=100m;

If the /etc/nginx/cache directory doesn’t exist, create it. The keys_zone name (thelastcicada in this example) will be used later to reference this collection of settings. In the key_zone, the second parameter is the size of the “shared memory zone” used for cache. I believe this is how much RAM/Swap will be used by Nginx for caching, but I’m not entirely clear on what the “shared memory zone” actually is. The “inactive” parameter is the time before garbage collection removes an old and unused cache item – in this case, 10 minutes. The max_size is the limit for the on disk cache files. Anything above this limit will be removed by garbage collection, presumably based on inactivity. For further explanation of what these settings mean, check out this Digital Ocean tutorial and the Nginx documentation.

The cache key in Nginx is an MD5 hash based on this setting, which we add just under the fastcgi_cache_path:

fastcgi_cache_key "$scheme://$host$request_method$request_uri";

Update: fastcgi_cache_key has been altered to include $request_method based on a tip from Innoscale. Adding this field prevents a weird issue whereby Nginx will occasionally cache a redirect rather than the page contents.

Next we move to the server block to define what pages should not be cached. The logic for which pages not to cache are very similar to the logic used by Batcache and all WordPress caching solutions. These lines can be added either in the php location block or in the root of the server block:

#Cache everything by default
set $no_cache 0;

#Don't cache logged in users or commenters
if ( $http_cookie ~* "comment_author_|wordpress_(?!test_cookie)|wp-postpass_" ) {
        set $no_cache 1;
}

#Don't cache the following URLs
if ($request_uri ~* "/(wp-admin/|wp-login.php)")
{
        set $no_cache 1;
}

In the php location block, insert the fastcgi cache settings for this particular site:

#matches keys_zone in fastcgi_cache_path
fastcgi_cache thelastcicada;

#don't serve pages defined earlier
fastcgi_cache_bypass $no_cache;

#don't cache pages defined earlier
fastcgi_no_cache $no_cache;

#defines the default cache time
fastcgi_cache_valid any 10s;

#unsure what the impacts of this variable is
fastcgi_max_temp_file_size 2M;

#Use stale cache items while updating in the background
fastcgi_cache_use_stale updating error timeout invalid_header http_500;
fastcgi_cache_lock on;
fastcgi_cache_lock_timeout 10s;

One last setting to add anywhere in the server block:

add_header X-Cache $upstream_cache_status;

This will add a header that can be checked in the browser and will indicate whether the cache was hit or missed.

Restart Nginx and reload your website a few times and look for “X-Cache: HIT” in the headers (I use Chrome’s inspector to see my site’s headers). At the time of writing this post, this site uses Nginx microcaching and you should see the X-Cache header setting indicating when the Nginx cache is active.

My entire nginx.conf and server-specific conf file are available here for reference. Note that I’m using a WordPress multisite install with subfolders, which is a little different than the single-site setup.

Resources

How to Setup FastCGI Caching with Nginx on your VPS
How I built “Have Baby. Need Stuff!”
Nginx HttpFastcgiModule Documentation