Automate your vCenter interactions from the Linux commandline with govmomi and govc

This is the first time in a LONG time that I found a tool I’m really happy with: govc allows you to interact with VMware vCenter without the need for the dreaded vClient, Windows machines or to write your own scripts to access the horrible VMware API. This way you can easily automate many tasks on VMware directly from the command line or your bash scripts.

Disclaimer: I don’t like VMware. I hated every single moment when I was forced to start a Windows VM just to interface with ESX/vCenter, so this tool is exactly what I desired.

First thing first, you’ll need the tool. You can compile it if you want, but there’s handy binaries already available on the project github page: https://github.com/vmware/govmomi/releases

I downloaded it and created a small wrapper for it:

 # curl -LO https://github.com/vmware/govmomi/releases/download/v0.9.0/govc_linux_amd64.gz
# gunzip govc_linux_amd64.gz
# cat >govc <<EOF
#!/bin/bash

export GOVC_URL='https://username:password@vsphere-ip-or-hostname/sdk'
export GOVC_DATACENTER=VSPHERE_DC
export GOVC_INSECURE=true

/usr/bin/govc_linux_amd64 \$@
EOF
# chmod +x govc*
# cp -i govc* /usr/bin/

Note that you only need GOVC_INSECURE=true if you are using self-signed certificates and you don’t have the CA added to your local trusted certs.

At this point by calling /usr/bin/govc you should be able to use the tool without specifying too many options on the command line. For a start, you can get some basic info about your environment:

 # govc about
Name:         VMware vCenter Server
Vendor:       VMware, Inc.
Version:      5.5.0
Build:        4180647
OS type:      linux-x64
API type:     VirtualCenter
API version:  5.5
Product ID:   vpx
UUID:         AAAA-BBBB-CCCC-DDDD-EEEE
# govc datacenter.info
Name:              VSPHERE_DC
Hosts:             10
Clusters:          0
Virtual Machines:  0
Networks:          6
Datastores:        14

What else can we do? Quite a lot of stuff, actually. To get an idea, just run govc without any argument. To see what parameters a command supports, run govc command –help.

For example you can easily get the datastore usage info for the whole datacenter with one simple command:

# govc datastore.info
Name:        ds1_SAS
  Path:      /VSPHERE_DC/datastore/ds1_SAS
  Type:      VMFS
  URL:       ds:///vmfs/volumes/AAAA1-BBBB-CCCC-DDDD/
  Capacity:  833.0 GB
  Free:      424.7 GB
Name:        ds2_SAS
  Path:      /VSPHERE_DC/datastore/ds2_SAS
  Type:      VMFS
  URL:       ds:///vmfs/volumes/AAAA2-BBBB-CCCC-DDDD/
  Capacity:  833.0 GB
  Free:      388.7 GB
[...]

Amazed by this, I thought I could as easily get all the info about the ESX hosts with host.info, but nope:

 # govc host.info
/usr/bin/govc_linux_amd64: default host resolves to multiple instances, please specify
# govc host.info -host.ip=192.168.100.1
Name:              192.168.100.1
  Path:            /VSPHERE_DC/host/192.168.100.1/192.168.100.1
  Manufacturer:    HP
  Logical CPUs:    24 CPUs @ 2294MHz
  Processor type:  Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
  CPU usage:       2061 MHz (3.7%)
  Memory:          131037MB
  Memory usage:    51020 MB (1.4%)
  Boot time:       2016-08-17 18:54:24.980915 +0000 UTC

It turns out, it doesn’t work the same way. As you see, I had to use the -host.ip parameter to specify the IP address of the ESX host.

At the time I thought the -host.ip parameter was the only way to specify the target, but it’s not accepted by the vm.info command:

 # govc vm.info -host.ip=192.168.100.100
/usr/bin/govc_linux_amd64: flag provided but not defined: -host.ip
# govc vm.info -vm.ip=192.168.100.100
Name:           Control
  Path:         /VSPHERE_DC/vm/services/Control
  UUID:         AAAA-BBBB-CCCC-DDDD-FFFF
  Guest name:   Red Hat Enterprise Linux 6 (64-bit)
  Memory:       4096MB
  CPU:          3 vCPU(s)
  Power state:  poweredOn
  Boot time:    2016-04-27 11:49:10.410135 +0000 UTC
  IP address:   192.168.100.100
  Host:         192.168.100.1

At the time I was quite puzzled because I couldn’t find a good way to iterate over the hosts or VMs. Until I noticed the ls command:

 # govc ls
/VSPHERE_DC/vm
/VSPHERE_DC/network
/VSPHERE_DC/host
/VSPHERE_DC/datastore
# govc ls /VSPHERE_DC/host
/VSPHERE_DC/host/192.168.100.1
/VSPHERE_DC/host/192.168.100.2
[...]
# govc ls /VSPHERE_DC/vm
/VSPHERE_DC/vm/database
/VSPHERE_DC/vm/template
[...]
# govc ls /VSPHERE_DC/vm/database/*
/VSPHERE_DC/vm/database/opencart/DB-OPENCART01
/VSPHERE_DC/vm/database/social/DB-SOCIAL-MASTER03
/VSPHERE_DC/vm/database/social/DB_SOCIAL_SLAVE03
[...]

And after a few more minutes fiddling and reading the –help options I realized you could invoke govc with the result from the ls command as a parameter, for example:

 # govc vm.info /VSPHERE_DC/vm/database/opencart/DB-OPENCART01
Name:           DB-OPENCART01
  Path:         /VSPHERE_DC/vm/database/opencart/DB-OPENCART01
[...]
# govc ls /VSPHERE_DC/vm/database/* | xargs govc vm.info | grep ^Name
Name:           DB-OPENCART01
Name:           DB_SOCIAL_SLAVE03
Name:           DB-SOCIAL-MASTER03
[...]

Iterating over hosts, VMs and datastores is easy!

govc also allows you to run esxcli on the specified host, for example in one of my many experiments I ran:

 # govc host.esxcli --host.ip=192.168.100.1 vm process list | grep DisplayName | awk {'print $2'} | sort
TOMCAT01
TOMCAT-TEST
WS01
WS01-TEST
WS02
WS-TEST

A very important thing to notice is that every command can be run with the parameter -json=true. This outputs a machine-parsable format that includes many more details than the normal text output.

 # govc vm.info -json /VSPHERE_DC/vm/database/opencart/DB-OPENCART01 | python -m json.tool
{
    "VirtualMachines": [
        {
            "AlarmActionsEnabled": true,
            "AvailableField": null,
            "Capability": {
                "BootOptionsSupported": true,
                "BootRetryOptionsSupported": true,
                "ChangeTrackingSupported": true,
                "ConsolePreferencesSupported": false,
[...]

This, of course, could be very handy for your scripts!

More info on the project github page: https://github.com/vmware/govmomi/tree/master/govc

A small introduction to this tool that helped me get started: http://www.virtuallyghetto.com/2014/09/govmomi-vsphere-sdk-for-go-govc-cli-kubernetes-on-vsphere-part-1.html

Step-up Firefox TLS security

For some reason, I’m using the same Firefox configuration since ages. Today, when Asana insisted that my browser was too old, I was a bit puzzled since I’m running the latest Fedora and I was reasonably sure Firefox was updated. It ended up being an old TLS setting.

To change it, open a new tab and type about:config in the address bar. In the new page, search (there’s a search box) for the key security.tls.version.min and make sure it’s set to 3.

Ethernet bonding in CentOS 7

Just a few quick notes about how I configured Ethernet bonding on CentOS 7. I want to write it down because it was subtly different from what I had on CentOS 6, so I’ll have a reference for the future ;)

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
TYPE=Bond
BONDING_MASTER=yes
IPADDR=192.168.0.100
PREFIX=24
GATEWAY=192.168.0.1
ONBOOT=yes
BOOTPROTO=static
NM_CONTROLLED=no
USERCTL=no
BONDING_OPTS="mode=1 miimon=100 downdelay=300 updelay=30000 primary=enp2s0f0"
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
DNS1=192.168.0.251
DNS2=192.168.0.252
DNS3=192.168.0.253
DOMAIN=stardata.lan

/etc/sysconfig/network-scripts/ifcfg-enp2s0f0

NAME=enp2s0f0
DEVICE=enp2s0f0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED="no"
USERCTL="no"
MASTER="bond0"
SLAVE="yes"

/etc/sysconfig/network-scripts/ifcfg-enp4s0f0

NAME=enp4s0f0
DEVICE=enp4s0f0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED="no"
USERCTL="no"
MASTER="bond0"
SLAVE="yes"

For more info and different methods to setup the bonding (nmtui is actually pretty cool), check the official RHEL7 documentation.

Create a custom profile for KDEnlive 16.4.3

Fedora recently upgraded the bundled KDEnlive to the 16.4.3 release that changes a lot of the previous export settings.

Thanks to this tutorial on the KDE website I managed to create a profile for my videos that is similar to what I was running previously:

File: /usr/share/mlt/presets/consumer/avformat/Velenux/1920x1200_60fps

description=MP4 1920x1200 60fps
f=mp4
frame_rate_num=60
frame_rate_den=1
width=1920
height=1200
progressive=1

vcodec=libx264
vb=10M
g=30
bf=2

acodec=aac
ab=384k
pix_fmt=yuv420p

threads=4
coder=1
movflags=+faststart

meta.preset.extension=mp4
meta.preset.name=MP4 1920x1200 60fps

In the editor I created a new profile with the settings suggested by the tutorial:

  • Destination: File rendering
  • Group: MP4
  • Profile Name: MP4 1920×1200 60fps
  • Extension: mp4
  • Parameters: properties=Velenux/1920x1200_60fps vb=%quality+’k’ ab=%audiobitrate+’k’
  • Video qualities: 10000,20000,30000
  • Default quality: 10000
  • Audio Bitrates: 384,256,192,160,128
  • Default Audio Bitrate: 256

For reference: the tutorial this article is based on, the KDEnlive manual about render profiles parameters and the KDEnlive docs about rendering.

Using SQL on text files to discover which ship to fly in EVE online

EVE online is a MMORPG where the players fly spaceships to blow each other to pieces and smack talk in forums and chats online.

The developers recently released a CSV data dump containing all the PvP kills (spaceships blown apart by other players) in the last month.

I thought it was an excellent excuse to test out one of the projects I starred on github and never got to try out: q, a nifty tool that allows you to use SQL queries on CSV or otherwise tabular text files.

As a first step, since q was having some problems with the text encoding in CCP release, I got rid of most of the colums I didn’t care about:

$ mkdir eve-killdump
$ cd eve-killdump/
$ unzip ~/Downloads/WWB_killdump.zip
$ cut -d, -f10-12,14,16-18 killdump.csv > onlymatters.csv

At this point my “onlymatters.csv” file looked like this:

destroyedShipType,destroyedShipGroup,killTime,solarSystemName,regionName,iskLost,iskDestroyed
Coercer,Destroyer,2016-03-10 21:29:37,Lamaa,The Bleak Lands,12585330.52,6692045.33

Then I ran my first query, I wanted to see which ships were destroyed the most:

$ q -H -d, -w none "SELECT destroyedShipType, destroyedShipGroup, COUNT(*) as nDestroyed FROM ./onlymatters.csv GROUP BY destroyedShipType ORDER BY nDestroyed" | tail -n 10
Atron,Frigate,6067
Catalyst,Destroyer,7702
Caracal,Cruiser,7903
Svipul,Tactical Destroyer,8165
Velator,Rookie ship,8481
Tristan,Frigate,8700
Mobile Tractor Unit,Mobile Tractor Unit,8712
Thrasher,Destroyer,8914
Ibis,Rookie ship,10503
Capsule,Capsule,155253

Unsurprisingly, a lot of Frigates (the cheapest/easiest to fly spaceship type) are destroyed the most. And most of them of Gallente manufacturing! Interesting.

Then I wanted to see the destroyed ships grouped by “ship group” (shall we call it “class”?) and ran:

$ q -H -d, -w none "SELECT destroyedShipType, destroyedShipGroup, COUNT(*) as nDestroyed FROM ./onlymatters.csv GROUP BY destroyedShipGroup ORDER BY nDestroyed" | tail -n 10
Manticore,Stealth Bomber,10924
Megathron,Battleship,11947
Svipul,Tactical Destroyer,15961
Drake,Combat Battlecruiser,16478
Malediction,Interceptor,22912
Ibis,Rookie ship,28716
Coercer,Destroyer,31393
Maller,Cruiser,44608
Tristan,Frigate,100919
Capsule,Capsule,158907

Again, Frigates are second only to Capsules (every pilot has a capsule to fly when his ship gets blown apart, BUT the capsule can be blown apart too, so I guess the Capsule is the most flown “ship” in EVE!) in number of ship lost.

That’s it for now, was a nice excercise to test out q a bit :)

Fly safe! o/

firewalld and nmcli – how to open a port on a specific interface on CentOS 7

For admins used to using iptables, the changes in RHEL 7.x made life a lot harder: the default config is a mess of zones, rules sending the traffic through different chains and what not. I had to spend hours tracking down how to add a single port to a single zone and switch one interface from one zone to another, I might as well document the whole experience for the sake of fellow admins out there:

First thing first, let’s see how our interfaces are configured:

# firewall-cmd --get-active-zones
public
  interfaces: ens160 ens192 ens224 ens256

In my particular case I want to switch ens224 (my management interface) from the “public” to the “work” zone, so I check what services are enabled in both zones:

# firewall-cmd --zone=public --list-services
dhcpv6-client http ssh
# firewall-cmd --zone=work --list-services
dhcpv6-client ipp-client ssh

And then I make sure I have http enabled in the “work” zone as well:

# firewall-cmd --permanent --zone=work --add-service http

Then I went to switch the ens224 interface from “public” to “work”… but it didn’t work:

# firewall-cmd --permanent --zone=public --remove-interface=ens224
# firewall-cmd --permanent --zone=work --add-interface=ens224
# firewall-cmd --reload
success
# firewall-cmd --get-active-zones
public
  interfaces: ens160 ens192 ens224 ens256

You also need to change the zone in the configuration setting, either by editing the configuration file in /etc/sysconfig/network-scripts/ or, as it was in my case, by fiddling with NetworkManager:

# nmcli c
NAME    UUID                                  TYPE            DEVICE 
nas     xxxxxxxx-yyyy-zzzz-tttt-wwwwwwwwwwww  802-3-ethernet  ens256 
cda-be  xxxxxxxx-yyyy-zzzz-tttt-wwwwwwwwwwww  802-3-ethernet  ens224 
bal     xxxxxxxx-yyyy-zzzz-tttt-wwwwwwwwwwww  802-3-ethernet  ens160 
cda-fe  xxxxxxxx-yyyy-zzzz-tttt-wwwwwwwwwwww  802-3-ethernet  ens192
# nmcli -p con show cda-be|grep connection.zone
connection.zone:                        --
# nmcli con modify cda-be connection.zone work
# nmcli -p con show cda-be|grep connection.zone
connection.zone:                        work

I reloaded the firewall configuration again and verified with iptables that the rules were now pointing to the “work” zone and that the zone did allow for http traffic:

# firewall-cmd --reload
# iptables -nvL
[...]
Chain INPUT_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 IN_public  all  --  ens256 *       0.0.0.0/0            0.0.0.0/0           [goto] 
    0     0 IN_public  all  --  ens192 *       0.0.0.0/0            0.0.0.0/0           [goto] 
    1    44 IN_public  all  --  ens160 *       0.0.0.0/0            0.0.0.0/0           [goto] 
    1    60 IN_work    all  --  ens224 *       0.0.0.0/0            0.0.0.0/0           [goto] 
    0     0 IN_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto] 
[...]
Chain IN_work_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:631 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 ctstate NEW

I hope this will be useful for someone out there :)
More info, as usual, in the official documentation: firewall-cmd, nmcli.