How to fix Gitlab timeout

A client asked for a Gitlab server. We’re using Gitlab internally for our small projects, so we already had some expertise on it, but it’s always interesting to see how a solution that is working for you will fare in a different environment.

After the installation, everything was running smoothly except the client had a problem on pages showing a large number of commits. This is a known issue, and there’s a lot of answers spread in the web, so I wanted to provide the solution we adopted, in the hope it would be useful for some fellow admin.

First thing to remember is, if you’ve used the Omnibus package, all your modifications will go in /etc/gitlab/gitlab.rb. We added these lines:

# commit read timeout
gitlab_rails['git_timeout'] = 600

# nginx timeout
nginx['keepalive_timeout'] = 300

# unicorn timeout
unicorn['worker_timeout'] = 300

After modifying that file you’ll need to run:

# gitlab-ctl reconfigure

The command will rebuild the configuration files and restart the services with the new values. This configuration solved the problem for our client.

If you are not using the Omnibus package, you’ll need to modify gitlab-rails/etc/gitlab.yml, gitlab-rails/etc/unicorn.rb and of course nginx config file.

Advertisements

Fixing nginx random segfaults

Logs in our client’s machine were filled with messages about nginx segfaults:

Sep 24 12:02:34 hostname kernel: nginx[8290]: segfault at 8 ip 00000000004935f0 sp 00007fff6698fd40 error 4 in nginx[400000+b5000]
Sep 24 12:02:34 hostname kernel: nginx[8292]: segfault at 8 ip 00000000004935f0 sp 00007fff6698fd40 error 4 in nginx[400000+b5000]
Sep 24 12:02:36 hostname kernel: nginx[8295]: segfault at 8 ip 00000000004935f0 sp 00007fff6698fd40 error 4 in nginx[400000+b5000]
Sep 24 12:02:36 hostname kernel: nginx[8296]: segfault at 8 ip 00000000004935f0 sp 00007fff6698fd40 error 4 in nginx[400000+b5000]

If you have more than one virtual host and you are using SSL on all of them, you may have incurred in this bug.

As the report states, a workaround is to specify the SSL session cache inside the http block instead than in each server’s block. We disabled it completely for testing and the segfaults went away.

Running Sinatra (and other Rack apps) on Nginx + Unicorn

Like it often happens with Open Source projects, documentation for Unicorn is quite a mess. I was searching for a tutorial on how to deploy a simple ‘Hello, world!’ Sinatra app on Nginx + Unicorn, but all tutorials I found were somewhat lacking. With some help from vjt I managed to get it up and running, so here’s the breakdown. Most config files are adapted from original project examples.

As you can imagine, the first step is to install all the packages. I’m using an Ubuntu 10.04 virtual machine, so:

user[~]% sudo aptitude install ruby-full rubygems nginx
[...]
user[~]% sudo gem install unicorn sinatra
[...]

Then, make sure Ruby gems are in PATH:

user[~]% echo 'PATH=$PATH:/var/lib/gems/1.8/bin' >> /etc/environment
user[~]% source /etc/environment
user[~]% echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/var/lib/gems/1.8/bin
user[~]% unicorn --version
unicorn v4.1.1

Let’s create a basic Sinatra app:

user[~]% mkdir basic; cd basic/
user[~/basic/]% cat > app.rb <<EOF
require 'rubygems'
require 'sinatra'

get '/' do
  "hello world! it's #{Time.now} here!"
end

EOF

Let’s verify the app is working with WEBrick (Sinatra development webserver):

user[~/basic/]% ruby app.rb
[2012-01-01 21:03:58] INFO  WEBrick 1.3.1
[2012-01-01 21:03:58] INFO  ruby 1.8.7 (2010-01-10) [x86_64-linux]
== Sinatra/1.3.1 has taken the stage on 4567 for development with backup from WEBrick
[2012-01-01 21:03:58] INFO  WEBrick::HTTPServer#start: pid=1474 port=4567

Try connecting to the host on port 4567:

user[~]% curl http://localhost:4567
hello world! it's Sun Jan 01 21:08:14 +0100 2012 here!

Everything is working fine. Stop WEBrick (Control + C) and let’s create unicorn config files:

user[~/basic/]% cat > config.ru <<EOF
require 'sinatra'

set :env,  :production
disable :run

require './app.rb'

run Sinatra::Application

EOF
user[~/basic/]%
user[~/basic/]% cat > unicorn.conf <<EOF
worker_processes 8
working_directory "/home/USERNAME/basic"
listen 'unix:/tmp/basic.sock', :backlog => 512
timeout 120
pid "/var/run/unicorn/basic_unicorn.pid"

preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

EOF

Make sure /var/run/unicorn is writable by the user that will run unicorn. Try starting the daemon:

user[~]% cd basic
user[~/basic/]% unicorn -c unicorn.conf
I, [2012-01-10T14:13:40.886917 #6288]  INFO -- : unlinking existing socket=/tmp/basic.sock
I, [2012-01-10T14:13:40.887076 #6288]  INFO -- : listening on addr=/tmp/basic.sock fd=3
I, [2012-01-10T14:13:40.887287 #6288]  INFO -- : Refreshing Gem list
I, [2012-01-10T14:13:40.951007 #6291]  INFO -- : worker=2 spawned pid=6291
I, [2012-01-10T14:13:40.951180 #6291]  INFO -- : worker=2 ready
I, [2012-01-10T14:13:40.951426 #6289]  INFO -- : worker=0 spawned pid=6289
I, [2012-01-10T14:13:40.951563 #6289]  INFO -- : worker=0 ready
I, [2012-01-10T14:13:40.951960 #6288]  INFO -- : master process ready
[...]

You can check if the (UNIX) socket is up:

user[~]% sudo netstat -nalx
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
[...]
unix  2      [ ACC ]     STREAM     LISTENING     17669    /tmp/basic.sock
[...]

Now let’s change the nginx configuration. I modified the example from Unicorn website:

user[~]% sudo cat > /etc/nginx/nginx.conf <<EOF
worker_processes 1;

user USERNAME GROUP;

pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log;

events {
  worker_connections 1024;
  accept_mutex off;
  use epoll;
}

http {
  include mime.types;
  default_type application/octet-stream;
  access_log /tmp/nginx.access.log combined;
  sendfile on;

  tcp_nopush on;
  tcp_nodelay off;

  gzip on;
  gzip_http_version 1.0;
  gzip_proxied any;
  gzip_min_length 500;
  gzip_disable "MSIE [1-6]\.";
  gzip_types text/plain text/html text/xml text/css
             text/comma-separated-values
             text/javascript application/x-javascript
             application/atom+xml;

  upstream app_server {
    server unix:/tmp/basic.sock fail_timeout=0;
  }

  server {
    listen 80 default deferred; # for Linux

    client_max_body_size 4G;
    server_name _;

    keepalive_timeout 5;

    # path for static files
    root /home/USERNAME/basic/public;

    # Prefer to serve static files directly from nginx to avoid unnecessary
    # data copies from the application server.
    try_files $uri/index.html $uri.html $uri @app;

    location @app {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_pass http://app_server;
    }
  }
}

EOF

Restart nginx:

user[~]% sudo service nginx restart
Restarting nginx: [warn]: duplicate MIME type "text/html" in /etc/nginx/nginx.conf:68
the configuration file /etc/nginx/nginx.conf syntax is ok
configuration file /etc/nginx/nginx.conf test is successful
[warn]: duplicate MIME type "text/html" in /etc/nginx/nginx.conf:68
nginx.

Check that nginx is running:

user[~]% sudo netstat -nalt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
[...]

Try loading the website:

user[~]% curl localhost
hello world! it's Tue Jan 10 14:21:04 +0100 2012 here!

Here you go! You should work some magic (or just download one of many startup scripts) for running Unicorn at boot time, then sit down and relax :)

Edit: fixed paths and a call to sudo.