This is a tale about what you should never do, but you are often forced to do in this time and age.
I’ll explain the technical solution and then tell the story for some context.
On a recent-ish system, install
# yum -y --enablerepo=epel install multitail
multitail allows to follow multiple tail or even the output of multiple commands in one single window (or multiple windows handled by ncurses), but it also allows to save the output of those commands to another file. In my case the command line looked like:
multitail --mergeall -D -a all.log \ -l 'ssh web01 "tail -qF /var/log/apache2/*.log /var/log/apache2/*/*.log"' \ -l 'ssh web02 "tail -qF /var/log/apache2/*.log /var/log/apache2/*/*.log"'
This would create a file
all.log containing the output from
tail -qF of Apache logs from
So, what’s the backstory? Why would I do something like this? Centralized logs are nothing new, right? We have Solutions[tm] for that.
Imagine you have a time constraint of “one hour”.
Then imagine you have systems so obsolete that the signing key (valid for 10 years) for their repositories expired.
If I had more time I would try to see if
rsyslog was recent enough to have the text-input file module and I would’ve tried to have rsyslog push the logs to a more recent system with logstash/ELK on.
I made a little script to generate the multitail commandline, here, have fun:
#!/bin/bash HOST_LIST="web01 web02" LOG_LIST="/var/log/apache2/*.log /var/log/apache2/*/*.log" CMD_MULTITAIL="multitail --mergeall -D -a all.log" for target in $HOST_LIST ; do CMD_MULTITAIL="$CMD_MULTITAIL -l 'ssh $target \"tail -qF $LOG_LIST\"'" done echo $CMD_MULTITAIL
I seriously hope nobody (else) will ever need this, but if you do, I got you covered.