Reading logs with Splunk and Docker

Every now and then, I need to parse through some log files where there are multiple files (like in a core dump or a whole logs directory) — figuratively looking for needle in a haystack.

My work has a Splunk license but I didn’t want to add the logs in and have them be a possible red herring for another if I left them in. So, I used Docker to host a Splunk instance on my Mac. Here’s the instructions on setting it up:

Things you need:

Docker ID and installer.

Steps:

  • Install and setup Docker.
  • Get and install the Splunk repository.
    docker pull splunk/splunk:7.0.0
  • Setup a place on your Mac to drop logs and a place to store configs. I’ll use a base folder of ~/Local in this example.
  • Create the Docker container.
    docker run --name splunk7 -d -v ~/Local/SplunkWatch:/splunkwatch -v ~/Local/DockerData/splunk:/opt/splunk/etc -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=root" -p "8000:8000" splunk/splunk:7.0.0

Login to the frontend at http://localhost:8000. Any logs dropped into ~/Local/SplunkWatch are parsed. Server configs are stored to persist (and be editable) in ~/Local/DockerData/splunk.

For a more GUI interface you can use https://kitematic.com.

Some helpful docker commands:

###list running docker containers
docker ps

### list all docker containers
docker ps -a

### enter docker container
docker exec -it splunk7 /bin/bash

### stop/start docker
docker start splunk
docker stop splunk

### Clean out DB
docker exec -it splunk7 /bin/bash
/opt/splunk/bin/splunk stop
/opt/splunk/bin/splunk clean eventdata
/opt/splunk/bin/splunk start

Migrating from Ubuntu + ZFS File Server to unRAID

For years I have had a Dell Inspiron 537 with Ubuntu hosting a ZFS RAID 1 file-share. I’ve used it to be a BitTorrent Resilio Sync server, a Plex Media Server, Apache web server, ssh relay and more. But, it started to show its age with Plex. I was content to let it struggle for another year until I found unRAID from https://lime-technology.com.

After trying a demo, I decided to upgrade and shuffle the Inspiron over to my parents – retiring a hacked together computer lovingly called ‘Frankenpoot‘ –  and build a new computer to be a NAS server for me.

The main reasons to use unRAID over something like FreeNAS or Ubuntu again was ZFS’s (and like filesystems’) need for equal-sized disks. That requirement is limiting on a budget. The old Dell, called in its early days Zod (bad guy from Superman) and then Zorg from the movie The 5th Element, had a ZFS RAID1 pool of 2x 2TB drives called “ZorgZFS”. It worked great, no issues or complaints about ZFS. But the chassis of the Inspiron 537 had only room for 2x 3.5″ drives and a SSD boot drive haphazardly attached with only one screw. As I was going to build my own, I could have as many drive bays as I can afford. unRAID can handle drives of any size — as long as the parity drive is the largest.

Over the next few weeks, I’ll be documenting configuration, setup and other knowledge relating to unRAID.

OpenVPN Troubleshooting

Recently I tried installing and setting up OpenVPN on my home server. I ran into a problem I had had before. Both the installation of the .deb package and the init script would hang.

I found the –verbose switch to the /usr/local/openvpn_as/bin/ovpn-init command. it would always hang right after “Generating init scripts auto command”. As generating the init scripts should be instantaneous, this was frustrating. At one point in the script is calls /usr/local/openvpn_as/bin/_ovpn-init and attempts to list the iptables config with iptable –list and put it into a variable. My iptables list is LARGE and uses the hostnames of several IPs I want blocked. (Think asian and Russian sites.) That iptables command it uses does DNS on every hostname. It should use (and I changed the script) to use iptables –list -n which skips DNS.

Now the ovpn-init script runs and doesn’t hang.

Zabbix and Percona

I am staging a Zabbix v3 install and needed to monitor MySQL with Percona. I used the Zabbix Template from this page, and ran into errors:

Invalid tag “/zabbix_export/templates/template(1)/items/item(1)”: the tag “snmp_community” is missing.

Among others.

It looks like the template from Percona is a bit outdated for Zabbix v3. I spent a few hours fixing it. Here it is, fixed.

 

Continue reading “Zabbix and Percona”