Reading logs with Splunk and Docker

Every now and then, I need to parse through some log files where there are multiple files (like in a core dump or a whole logs directory) — figuratively looking for needle in a haystack.

My work has a Splunk license but I didn’t want to add the logs in and have them be a possible red herring for another if I left them in. So, I used Docker to host a Splunk instance on my Mac. Here’s the instructions on setting it up:

Things you need:

Docker ID and installer.

Steps:

  • Install and setup Docker.
  • Get and install the Splunk repository.
    docker pull splunk/splunk:7.0.0
  • Setup a place on your Mac to drop logs and a place to store configs. I’ll use a base folder of ~/Local in this example.
  • Create the Docker container.
    docker run --name splunk7 -d -v ~/Local/SplunkWatch:/splunkwatch -v ~/Local/DockerData/splunk:/opt/splunk/etc -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=root" -p "8000:8000" splunk/splunk:7.0.0

Login to the frontend at http://localhost:8000. Any logs dropped into ~/Local/SplunkWatch are parsed. Server configs are stored to persist (and be editable) in ~/Local/DockerData/splunk.

For a more GUI interface you can use https://kitematic.com.

Some helpful docker commands:

###list running docker containers
docker ps

### list all docker containers
docker ps -a

### enter docker container
docker exec -it splunk7 /bin/bash

### stop/start docker
docker start splunk
docker stop splunk

### Clean out DB
docker exec -it splunk7 /bin/bash
/opt/splunk/bin/splunk stop
/opt/splunk/bin/splunk clean eventdata
/opt/splunk/bin/splunk start

iOS 8.0 and excessive ‘Documents & Sync’ over LTE

“Verizon Msg: You’ve used about 50% of your 3GB data plan…”

I received this warning only 2 days before my data plan started over. I thought nothing about it at the time. Then, a few days later, after the start of the billing cycle, I got:

“Verizon Msg: You’ve used about 50% of your 3GB data plan…”

That didn’t seem right. A Gig and a half in about 20 days?! I hadn’t used any streaming service lately – especially over LTE. But, I was able to add another Gig to my plan for free, so I thought nothing of it. But, a week later, after the start of another billing cycle:

“Verizon Msg: You’ve used about 75% of your 4GB data plan”

No! This will not stand! I delved into the Cellular settings in the Settings app and found that ‘Documents and Sync’ was using way too much data – over LTE even when I was using Wifi! I called Apple and started a ticket. But, that takes time for them to come up with a solution. So, I started to document and troubleshoot. I disabled iCloud settings and found that nothing short of disabling iCloud on the device would still cause ‘data bleeding’ as I called it.

I could see it use a Meg a minute!
I could see it use a Meg a minute!

I surmised that something was wrong with some basic sync between my other iCloud-enabled devices – even over LTE – with them on the same Wifi network! At some point, I came to the Keychain.app and saw the iCloud Keychain.

Screenshot 2015-04-10 12.39.58

I emptied it out. I wasn’t able to backup anything in it. After about 30 minutes to make sure it wasn’t a convenient pause, the ‘data bleeding’ was stopped!

Here’s the interesting/scary thing: I had disabled Keychain sync on the iPhone and it still tried to sync!

I hope that was a bug and not how it really works!

Now, after 4 days after my data plan has restarted, I am seeing better cellular usage – and battery usage!

This is after 4 days!
This is after 4 days!

 

Also fixed after this, I usually had old Wifi networks in my Preferred Networks list in the System Preferences app. They were now gone!

 

iCloud, the little sync service that can’t

Beta? you Bet!
iCloud Pages, Numbers, Keynote Icons

So last night I started to notice that Pages, Numbers, Keynote iCloud documents and Day One entries were not able to sync.

I tried the standard rigamarole: turning off Document syncing, restarting computers and iOS devices to no success. Right now, my Day One entries are in a zip file on my Desktop.

Then I found a log file called ubiquity-digest.log in Console.app. Here’s what I found:

there was a http error 503 while talking to iCloud reason: ubiquity account locked

I work with web servers and mail servers, so here’s my thought on what is happening: deep in Apple’s server farms, they found or a major error occurred and they have turned off the server process by disabling the user account it runs under (ubiquity) that the Macs, iOS devices talk to, to prevent damage to existing data. Granted this is just a wild-ass guess.

This could be only related to a particular server or so, otherwise I am sure it would be in the media.

I would hope they are fixing it, but Apple’s Server Status pages are, as of this post, silent.

So, Apple, I have a question: What kind of cock-up does it take to communicate with customers via your server status pages?