How to solve OpenVPN errors after upgrading OpenSSL

I went on upgrading OpenVPN and OpenSSL on an old production system, but after restarting the service, the clients would not connect. There were two different problems a “CRL expired” error and after fixing that a “CRL signature failed” error.

CRL expired

The OpenVPN server logs were reporting:

Mon Nov  6 10:04:22 2017 TCP connection established with [AF_INET]192.168.100.1:19347
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 TLS: Initial packet from [AF_INET]192.168.100.1:19347, sid=150b3618 b004e9a4
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 VERIFY ERROR: depth=0, error=CRL has expired: C=IT, ST=PR, L=Parma, O=domain, OU=domain.eu, CN=user, name=user, emailAddress=info@stardata.it
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 OpenSSL: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 TLS_ERROR: BIO read tls_read_plaintext error
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 TLS Error: TLS object -> incoming plaintext read error
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 TLS Error: TLS handshake failed
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 Fatal TLS error (check_tls_errors_co), restarting
Mon Nov  6 10:04:23 2017 192.168.100.1:19347 SIGUSR1[soft,tls-error] received, client-instance restarting

This is a common problem on older systems, the culprit is in the OpenSSL configuration used to generate the CRL, that is limited to just 30 days by default.

So, I had to regenerate my CRL after increasing the default_crl_days parameter in the ssl config to 180 (for our use case is more than enough), using:

$ openssl  ca  -gencrl  -keyfile keys/ca.key  \
               -cert keys/ca.crt  -out keys/crl.pem \
               -config easy-rsa/openssl-1.0.0.cnf

CRL signature failed

Due to the vulnerabilities found in MD5, this hashing routine has been disabled by default on modern SSL. Our certificates, though, were still using it, so the new error message (after fixing the CRL), became:

Mon Nov  6 10:14:40 2017 TCP connection established with [AF_INET]192.168.100.1:18463
Mon Nov  6 10:14:41 2017 192.168.100.1:18463 TLS: Initial packet from [AF_INET]192.168.100.1:18463, sid=13fdd1fe 5d82d4d6
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 VERIFY ERROR: depth=0, error=CRL signature failure: C=IT, ST=PR, L=Parma, O=domain, OU=domain.eu, CN=user, name=user, emailAddress=info@stardata.it
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 OpenSSL: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 TLS_ERROR: BIO read tls_read_plaintext error
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 TLS Error: TLS object -> incoming plaintext read error
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 TLS Error: TLS handshake failed
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 Fatal TLS error (check_tls_errors_co), restarting
Mon Nov  6 10:14:42 2017 192.168.100.1:18463 SIGUSR1[soft,tls-error] received, client-instance restarting

This one was trickier to solve. It turns out that you can re-enable MD5 as a workaround using two environment variables: NSS_HASH_ALG_SUPPORT=+MD5 and OPENSSL_ENABLE_MD5_VERIFY=1. In my case, I just added them to openvpn init script because the system is going to be decommissioned soon.

References

Advertisements

Develop a Jekyll website… without Jekyll

Containers are great for developers: when I’m messing around with code I try to keep everything neatly containerized, so I can just pull my repository on some other machine, run a few scripts and be ready to keep on developing without having to install stuff on the main Operating System.

Jekyll is a nice Static Website Generator, used prominently on GitHub. An already-made Jekyll container exists, but I couldn’t find out how (or even if) you could use it to create a Jekyll website from scratch. So I fired up a generic Ruby container and installed jekyll in it to create the base layout, then I ran the already-made jekyll container to build the website.

$ cat > Gemfile <<EOF
source 'https://rubygems.org'
gem "jekyll"
EOF

$ docker run --rm --volume=$PWD:/usr/src/app -w /usr/src/app -it ruby:latest /bin/bash
[container#1]# bundle install
[...]
Fetching jekyll 3.6.0
Installing jekyll 3.6.0
Bundle complete! 1 Gemfile dependency, 20 gems now installed.
Bundled gems are installed into /usr/local/bundle.
[container#1]# jekyll new test01
[...]
Bundler: Using jekyll 3.6.0
Bundler: Bundle complete! 1 Gemfile dependency, 20 gems now installed.
Bundler: Bundled gems are installed into /usr/local/bundle.
New jekyll site installed in /usr/src/app/test01.
[container#1]# exit
$ ls -l test01
-rw-r--r-- 1 root root  398 ott 14 16:40 404.html
-rw-r--r-- 1 root root  539 ott 14 16:40 about.md
-rw-r--r-- 1 root root 1,7K ott 14 16:40 _config.yml
-rw-r--r-- 1 root root  937 ott 14 16:40 Gemfile
-rw-r--r-- 1 root root  213 ott 14 16:40 index.md
drwxr-xr-x 2 root root 4,0K ott 14 16:40 _posts

Once I had the basic site structure ready, I ran the Jekyll container to build it:

$ cd test01
$ docker run --rm  --volume=$PWD:/srv/jekyll  -it  jekyll/jekyll:latest  jekyll build
Resolving dependencies...
The Gemfile's dependencies are satisfied
Configuration file: /srv/jekyll/_config.yml
            Source: /srv/jekyll
       Destination: /srv/jekyll/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 0.292 seconds.
 Auto-regeneration: disabled. Use --watch to enable.
$ ls -lh _site/
-rw-r--r-- 1 velenux velenux 5,5K ott 14 16:44 404.html
drwxr-xr-x 2 velenux velenux 4,0K ott 14 16:44 about
drwxr-xr-x 2 velenux velenux 4,0K ott 14 16:44 assets
-rw-r--r-- 1 velenux velenux 3,7K ott 14 16:44 feed.xml
-rw-r--r-- 1 velenux velenux 5,5K ott 14 16:44 index.html
drwxr-xr-x 3 velenux velenux 4,0K ott 14 16:44 jekyll

So… that’s it, you can now develop your Jekyll website without having Jekyll
installed on your system.

Centralize your logs on the cheap on obsolete systems

This is a tale about what you should never do, but you are often forced to do in this time and age.

I’ll explain the technical solution and then tell the story for some context.

On a recent-ish system, install multitail:

# yum -y --enablerepo=epel install multitail

multitail allows to follow multiple tail or even the output of multiple commands in one single window (or multiple windows handled by ncurses), but it also allows to save the output of those commands to another file. In my case the command line looked like:

multitail --mergeall -D -a all.log \
  -l 'ssh web01 "tail -qF /var/log/apache2/*.log /var/log/apache2/*/*.log"' \
  -l 'ssh web02 "tail -qF /var/log/apache2/*.log /var/log/apache2/*/*.log"'

This would create a file all.log containing the output from tail -qF of Apache logs from web01 and web02.

So, what’s the backstory? Why would I do something like this? Centralized logs are nothing new, right? We have Solutions[tm] for that.

Backstory

Imagine you have a time constraint of “one hour”.

Then imagine you have systems so obsolete that the signing key (valid for 10 years) for their repositories expired.

If I had more time I would try to see if rsyslog was recent enough to have the text-input file module and I would’ve tried to have rsyslog push the logs to a more recent system with logstash/ELK on.

Bonus code

I made a little script to generate the multitail commandline, here, have fun:

#!/bin/bash
HOST_LIST="web01 web02"
LOG_LIST="/var/log/apache2/*.log /var/log/apache2/*/*.log"

CMD_MULTITAIL="multitail --mergeall -D -a all.log"

for target in $HOST_LIST ; do
  CMD_MULTITAIL="$CMD_MULTITAIL -l 'ssh $target \"tail -qF $LOG_LIST\"'"
done

echo $CMD_MULTITAIL

I seriously hope nobody (else) will ever need this, but if you do, I got you covered.

How to use MS SQL Server from PHP 5.x on Linux

Short answer: don’t. It’s a world of pain.

If you really must, keep reading.

Install the required packages

I’ll assume you already have a working webserver and PHP installation, along with the EPEL and Remi repositories.

yum --enablerepo=epel --enablerepo=remi install php-mssql freetds

Make sure the mssql library is loaded

cat /etc/php.d/mssql.ini
; Enable mssql extension module
extension=mssql.so

Configure mssql

In /etc/php.ini make sure these two lines are set like this:

mssql.textlimit = 20971520
mssql.textsize = 20971520

Configure freetds

Edit /etc/freetds.conf to change the text size value and to create a new entry:

[global]
  [...]
  text size = 20971520
[...]
[mssqltest]
  host = 192.168.0.100
  instance = instance_name
  port = 1433
  client charset = UTF-8
  tds version = 8.0
  text size = 20971520

You can omit the instance parameter if you are going to use the default.

Restart Apache and PHP

service httpd configtest && service httpd restart

Test the configuration

Create a simple php script, mssql_test.php:

<?php

$host_db = "mssqltest";  // reference to the name you used in /etc/freetds.conf
$user_db = "your_user";
$pass_db = "your_pass";
$name_db = "your_db_name";

$conn_db = mssql_connect($host_db, $user_db, $pass_db)
  or die("Connection failed to $host_db: " . mssql_get_last_message() . "\n\n");

$dbh     = mssql_select_db($name_db, $conn_db)
  or die("Cannot open DB $name_db: " . mssql_get_last_message() . "\n\n");

echo 'Connected.';

$query_results = mssql_query('SELECT * from YourTable')
  or die ("Cannot run your query: " . mssql_get_last_message() . "\n\n");

$row = mssql_fetch_array($query_results)
  or die ("Cannot fetch row: " . mssql_get_last_message() . "\n\n");

echo "$row\n\n";

mssql_close($link);
?>

References

Firefox Beta PPA on Linux Mint 18.2

I wanted to test out a more recent version of Firefox on Linux Mint 18.2, so I went over to grab the Ubuntu PPA.

I ran the installation commands (as root):

add-apt-repository ppa:mozillateam/firefox-next
apt-get update
apt-get install firefox

But apt wouldn’t update the package, because the default package priorities (check /etc/apt/preferences.d/*.pref) won’t allow the use of Ubuntu packages over Mint packages.

Fair enough, I created a new file /etc/apt/preferences.d/firefox.pref containing:

Package: firefox
Pin: origin ppa.launchpad.net
Pin-Priority: 900

Package: firefox
Pin: release o=Ubuntu
Pin-Priority: 900

And updated to the beta (currently 55.0b12).

Enjoy the 10 fixed bugs and the exciting 127 new bugs to discover!

Workaround for javaws jnpl error “Cannot grant permissions to unsigned jars.”

There’s an emergency and you’re trying to log on that ancient KVM/iLO/iDRAC and you’re getting an error like:

net.sourceforge.jnlp.LaunchException: Fatal: Application Error: Cannot grant permissions to unsigned jars.

Quick workaround:

Find the java.security file. In my case it is located in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/java.security

Then find the row:

jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

Comment it out, copy it, delete the MD5 string.

#jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024
jdk.jar.disabledAlgorithms=MD2, RSA keySize < 1024

Save the file. Try again. It should work.

Some more pointers if the problem persists:

Disk I/O errors on Adaptec ASR8805 raid controller

We have an Adaptec ASR8805 controller on one of the servers we manage. For various reasons we need to shrink a logical volume that is sitting on a RAID 6 logical device created and exposed by this controller, but we can’t because we’re getting seek errors:

Buffer I/O error on device dm-2, logical block 3330419721
sd 6:0:1:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 6:0:1:0: [sdb]  Sense Key : Hardware Error [current] 
sd 6:0:1:0: [sdb]  Add. Sense: Internal target failure
sd 6:0:1:0: [sdb] CDB: Read(16): 88 00 00 00 00 06 34 11 69 00 00 00 01 00 00 00
end_request: critical target error, dev sdb, sector 26643360000
sd 6:0:1:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

From what the controller is reporting, the RAID 6 is healthy, and all the physical drives SMART information seems ok(ish).

It turns out, no background checking of the RAID 6 parity has been enabled, and that is probably the problem, as reported by this article.

To get a “quick” fix (it’s a 24T array), I started:

# arcconf task start 1 logicaldrive 1 verify_fix

when it’ll be finished, I’ll enable the background check with:

# arcconf consistencycheck 1 on

I really hope this saves time to some fellow admin out there :)