fotografia · linux

Updated photographic workflow on Linux

My current photographic workflow on Linux has become quite complicated, so I wanted to share some info about it.

First, I download and rename my photos from the memory card using my own Photofix, that currently writes in a temporary directory in my home.

The next step involves a piece of non-free software: since my new camera raw format isn’t supported yet, I run Adobe DNG converter with Wine, to convert my CR3 files to DNG.

Next I can finally edit my files with Darktable. I usually start by doing a broad selection: every new photo starts at 2 Stars (“saving it just in case”); the worst ones I downvote to 1 Star (“candidate for the trash bin”), those that have potential and should be developed further get 3 Stars, those that I consider “worth showing to the Internet” get 4 Stars and my best shots get 5 Stars.

After the initial assessment I go on developing the single images starting with the best ones. After the editing the initial vote can be revised both upwards and downwards, so I might promote a 3 Star photo to 4 Stars or demote a 4 Stars to 3 or 2.

After I’m done editing, I export all the photos ranked 4 or 5 Stars as high resolution JPGs from Darktable to a new directory (usually named YYYYMM), and then run another script that creates scaled down and watermarked versions for Flickr (smaller size) and YourShot (bigger size).

When I’m done editing and exporting I move the RAW files to my long term storage space, although I’m considering to refine this last step too: I should export all the 1 Star and 2 Stars photos to lower quality JPGs instead of keeping the raw files. That should free some space on my storage and make my backups lighter and faster too :)

Advertisements
linux · tips

How to fix Clementine (music player) playing music, but no audio

Just a quick one, in case someone out there gets the same problem: my Clementine player did start playing music, but I could not hear anything.
Turns out my very minimal Debian testing install didn’t include gstreamer1.0-pulseaudio. Installing that package fixed it for me.

Uncategorized

Android adb “unknown backup flag” problem

Apparently, at some point the syntax for using adb backup changed and it’s not really well documented…

This is how I backed up my Android phone with adb today:

adb backup -f mybackup.bkp '-apk -obb -shared -all -system'

While before you would launch your backup with:

adb backup -f oldbackup.bkp -apk -obb -shared -all -system

Notice the lack of quotes in the old version.

linux · tech · tips

Schedule one-time jobs with systemd

I rarely use at, but today I shut down crond to do some maintenance and I wanted to schedule an automatic restart for later in the day in case I forget to restart it manually.

So, I ran:

# echo "/usr/bin/service crond start" | at now +6 hours
-bash: at: command not found

Turns out, on systems running systemd you can use systemd-run as a substitute to at to schedule one-time jobs, like this:

# systemd-run --on-active=30 /bin/touch /tmp/foo

The default --on-active parameter is in seconds, but you can pass modifiers to make it more readable:

# systemd-run --on-active="4h 30m" /bin/touch /tmp/foo

If you need to restart a service, there’s a handy shortcut, the --unit parameter:

# systemd-run --on-active=6h --unit crond.service

You can check the job queue (sorta what you would have done with atq) with:

# systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
gio 2018-06-07 16:32:01 CEST 5h 18min left mer 2018-06-06 16:32:01 CEST 18h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
gio 2018-06-07 17:12:12 CEST 7h left n/a n/a crond.timer crond.service

Another poor service (atd) has been swallowed by systemd. RIP.

References:

linux · tech · tips

Barebone Kickstart setup for CentOS 7

Since I had to install a bunch of baremetal servers and I haven’t had the time to checkout Foreman yet, I created a minimal setup to be able to use a Kickstart file.

My early iterations were done in Packer, then I switched to the baremetal servers to work out the details.

Please note: this is an automated install that WILL DELETE EVERYTHING on /dev/sda !!!

The kickstart file

This kickstart file has been made iterating over CentOS 6 and CentOS 7 default install kickstart files (those generated by the installer), with a couple of changes based on the documentation and similar examples (many thanks to Jeff Geerling !).

Please note: this is an automated install that WILL DELETE EVERYTHING on /dev/sda !!! – Do not run it on the wrong system!

Also, this is just a “template”, make sure to change it where it makes sense, for example the partitioning scheme and the root password. For the network settings, see below the script to customize and serve the kickstart file over http.

template.cfg

# Run the installer
install

# Use CDROM installation media
cdrom

# System language
lang en_US.UTF-8

# Keyboard layouts - Change this!
keyboard --vckeymap=it --xlayouts='it','us' --switch='grp:alt_shift_toggle'

# Enable more hardware support
unsupported_hardware

# Network information - the --device=link option activates the specific IP address on the first interface with a link up
# the ZZNAMEZZ labels will be changed later with sed, to customize the installation
network  --bootproto=static --device=link --gateway=ZZGATEWAYZZ --ip=ZZIPADDRZZ --nameserver=ZZDNSZZ --netmask=ZZNETMASKZZ --noipv6 --activate
network  --hostname=ZZHOSTNAMEZZ

# System authorization information
auth --enableshadow --passalgo=sha512

# Root password - Change this!
rootpw YOUR_SECURE_PASSWORD

# System timezone - Change this!
timezone Europe/Rome --isUtc --nontp

# Run the text install
text

# Skip X config
skipx

# Only use a specific disk, Change the drive here!
ignoredisk --only-use=sda

# Overwrite the MBR
zerombr

# Partition clearing information
clearpart --all --initlabel --drives=sda

# System bootloader configuration - Change the drive here
bootloader --location=mbr --boot-drive=sda


# PARTITIONING
# This is our partitioning scheme, change it where required

# this might not be required
part biosboot --fstype="biosboot" --ondisk=sda --size=1

# this is required
part /boot --fstype="xfs" --ondisk=sda --size=1024

# this will create a Volume Group "VGsystem" spanning the whole disk (except for the /boot partition)
part pv.229 --fstype="lvmpv" --ondisk=sda --size=200000 --grow
volgroup VGsystem --pesize=4096 pv.229

#
logvol /         --fstype="xfs"   --size=10240  --label="ROOT"  --name=LVroot  --vgname=VGsystem
logvol /usr      --fstype="xfs"   --size=20480  --name=LVusr    --vgname=VGsystem
logvol /var      --fstype="xfs"   --size=20480  --name=LVvar    --vgname=VGsystem
logvol /var/log  --fstype="xfs"   --size=20480  --name=LVvarlog --vgname=VGsystem

logvol swap      --fstype="swap"  --size=16384  --name=LVswap   --vgname=VGsystem

logvol /tmp      --fstype="xfs"   --size=10240  --name=LVtmp    --vgname=VGsystem
logvol /home     --fstype="xfs"   --size=51200  --name=LVhome   --vgname=VGsystem
logvol /opt      --fstype="xfs"   --size=20480  --name=LVopt    --vgname=VGsystem


# Do not run the Setup Agent on first boot
firstboot --disabled

# Accept the EULA
eula --agreed

# System services - we disable chronyd because we use NTP
services --disabled="chronyd" --enabled="sshd"


# Reboot the system when the install is complete
reboot


# Packages

%packages --ignoremissing --excludedocs
@^minimal
@core
kexec-tools
%end

%addon com_redhat_kdump --disable

%end

# upgrade the system before rebooting

%post
yum -y upgrade
yum clean all
%end

Customizing and serving the kickstart file

As we mentioned earlier, I made a pretty simple script to customize the kickstart template and serve it over http.

Please note: this is an automated install that WILL DELETE EVERYTHING on /dev/sda !!!

serve_kickstart.sh

#!/bin/bash

gateway="192.168.0.1"
netmask="255.255.255.0"
dns="192.168.0.11,192.168.0.12"

# this is pretty hacky, sorry
local_ipaddr=$(ip -4 -o addr show dev eth0 | awk {'print $4'} | cut -d/ -f1)

# accepts hostname and ip address on the command line
server_hostname="$1"
server_ipaddr="$2"

if [ -z "$server_hostname" ]; then
    server_hostname="freshinstall.stardata.lan"
    echo "Using '$server_hostname' as default."
fi

if [ -z "$server_ipaddr" ]; then
    server_ipaddr="192.168.0.99"
    echo "Using '$server_ipaddr' as default IP address."
fi


# create the file to customize
/bin/cp -f template.cfg custom.cfg

# customize the kickstart file
sed -i "s/ZZGATEWAYZZ/$gateway/g" custom.cfg
sed -i "s/ZZIPADDRZZ/$server_ipaddr/g" custom.cfg
sed -i "s/ZZDNSZZ/$dns/g" custom.cfg
sed -i "s/ZZNETMASKZZ/$netmask/g" custom.cfg
sed -i "s/ZZHOSTNAMEZZ/$server_hostname/g" custom.cfg

# create the file to serve
/bin/mv -f custom.cfg c7.cfg

# write the instructions to add to the boot on screen
echo "To use this kickstart, add to the boot command line: "

echo -e "\nip=${server_ipaddr} netmask=${netmask} gateway=${gateway} dns=${dns} text ks=http://${local_ipaddr}:8000/c7.cfg\n\n"

sleep 3

python -m SimpleHTTPServer

This is what an example run looks like:

$ ./serve_kickstart.sh test01.stardata.lan 192.168.0.100
To use this kickstart, add to the boot command line:

ip=192.168.0.100 netmask=255.255.255.0 gateway=192.168.0.1 dns=192.168.0.11,192.168.0.12 text ks=http://192.168.0.200:8000/c7.cfg

Serving HTTP on 0.0.0.0 port 8000 ...

192.168.0.100 - - [20/Apr/2018 16:03:43] "GET /c7.cfg HTTP/1.1" 200 -

If you take a look at the c7.cfg that is served via http on port 8000, you’ll see that the relevant network placeholders have been swapped with the custom values from the script:

$ grep ^network c7.cfg
network  --bootproto=static --device=link --gateway=192.168.0.1 --ip=192.168.0.100 --nameserver=192.168.0.11,192.168.0.12 --netmask=255.255.255.0 --noipv6 --activate
network  --hostname=test01.stardata.lan

As usual, I hope this helps some fellow admin out there.

References

linux · tech

How to compile and install v8 and v8js on CentOS 7

A client tasked us to install v8 and v8js on a test system to play around with the server-side compilation of vue.js applications.

The v8 library available on SCL and EPEL is old, 2013-old. So we were faced with the option to either compile v8 or switch away from CentOS for Debian or Ubuntu with some third party repositories that we don’t really know much about and we don’t really trust for when the platform will go in production.

A word of caution: compiling takes a long time, I recommend you use a fast machine (I used a VM with 8 cores, 16Gb RAM and it took about 20 mins to compile).

Compiling v8

As we said, compiling takes quite a lot of time. This is the script I wrote after a few rounds of trial and error. You can (and should, really) run this as a normal, unprivileged user.

#!/bin/bash

set -x  # debug
set -e  # exit on all errors

# update and install basic packages
sudo yum -y upgrade
sudo yum -y --enablerepo=epel --enablerepo=remi-php71 install git subversion make gcc-c++ chrpath redhat-lsb-core php php-devel php-pear

mkdir -p local/src

# install the depot tools from google
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git local/depot_tools
export PATH=$PATH:local/depot_tools

# install v8
cd local/src
fetch v8
cd v8
gclient sync
./tools/dev/v8gen.py -vv x64.release -- is_component_build=true
time ninja -C out.gn/x64.release
time ./tools/run-tests.py --gn

As you can see, there’s a bit of requirements that aren’t a good choice for production (compilers, git, etc), but you can uninstall them later or, better yet, build a package.

Installing v8

For the installation I relied on the v8js build instructions, copying (as root) just the required files from my build directory:

mkdir -p /opt/v8/{lib,include}

cd /your_build/path/local/src/v8
cp -v out.gn/x64.release/lib*.so out.gn/x64.release/*_blob.bin \
   out.gn/x64.release/icudtl.dat /opt/v8/lib/
cp -vR include/* /opt/v8/include/

Now we can install v8js using pecl.

Compiling v8js from PECL

During the initial setup I installed php-pear from the Remi PHP 7.1 repository. This gives us the pecl command to install php modules easily.

When asked for the v8 library path we’re using the /opt/v8 path we created in the previous step:

# pecl install v8js
downloading v8js-2.1.0.tgz ...
Starting to download v8js-2.1.0.tgz (101,553 bytes)
.......................done: 101,553 bytes
28 source files, building
running: phpize
Configuring for:
PHP Api Version:         20160303
Zend Module Api No:      20160303
Zend Extension Api No:   320160303
Please provide the installation prefix of libv8 [autodetect] : /opt/v8
building in /var/tmp/pear-build-rootDGsvNg/v8js-2.1.0
[...]
Build process completed successfully
Installing '/usr/lib64/php/modules/v8js.so'
install ok: channel://pecl.php.net/v8js-2.1.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=v8js.so" to php.ini
# echo 'extension=v8js.so' > /etc/php.d/60-v8js.ini
# service php-fpm restart

We compiled v8js and created a new ini file to load the new module in PHP, and restarted php-fpm to apply the changes.

We can verify that v8js is installed by running a simple phpinfo check:

$ cat > phpinfo.php
<?php phpinfo();
$ php phpinfo.php | grep -i v8
/etc/php.d/60-v8js.ini
v8js
V8 Javascript Engine => enabled
V8 Engine Compiled Version => 6.8.0
V8 Engine Linked Version => 6.8.0 (candidate)
v8js.flags => no value => no value
v8js.icudtl_dat_path => no value => no value
v8js.use_array_access => 0 => 0
v8js.use_date => 0 => 0

At this point you should be good to go. I hope this helps some fellow admins out there :)

Compiling v8 < 5.5

If you need an older version, the instructions for compiling and copying the files are slightly different:

compile-v8-5.2.sh

#!/bin/bash

set -x  # debug
set -e  # exit on errors

# update the system and install required packages
sudo yum -y upgrade
sudo yum -y --enablerepo=epel --enablerepo=remi-php71 install git subversion make gcc-c++ chrpath redhat-lsb-core php php-devel php-pear

mkdir -p local/src

# install depot tools
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git local/depot_tools
export PATH=$PATH:local/depot_tools 

# install v8
cd local/src 
fetch v8 
cd local/src/v8 
git checkout 5.2
gclient sync

export GYPFLAGS="-Dv8_use_external_startup_data=0"
export GYPFLAGS="${GYPFLAGS} -Dlinux_use_bundled_gold=0"
time make x64.release native library=shared snapshot=on -j8

Please note that if you get this error:

PYTHONPATH="/path/tools/generate_shim_headers:/path/build::/path/tools/gyp/pylib:" \
GYP_GENERATORS=make \
tools/gyp/gyp --generator-output="out" gypfiles/all.gyp \
              -Igypfiles/standalone.gypi --depth=. -S.native  -Dv8_enable_backtrace=1 -Darm_fpu=default -Darm_float_abi=default
gyp: Error importing pymod_do_mainmodule (detect_v8_host_arch): No module named detect_v8_host_arch
make: *** [out/Makefile.native] Error 1

You will need to edit the Makefile as documented in this diff to fix the Python paths.

As for the installation part:

install-v8-5.2.sh

export PATH=$PATH:local/depot_tools 

# install v8
sudo mkdir -p /opt/v8/{lib,include}

cd local/src/v8

sudo cp -v out/native/lib.target/lib*.so /opt/v8/lib/
sudo cp -vR include/* /opt/v8/include/
sudo chrpath -r '$ORIGIN' /opt/v8/lib/libv8.so

# Install libv8_libplatform.a (V8 >= 5.2.51)
echo -e "create /opt/v8/lib/libv8_libplatform.a\naddlib out/native/obj.target/src/libv8_libplatform.a\nsave\nend" | sudo ar -M

As you can see the build process is a bit different, so make sure you're following the correct one for your version.

References