Splunking My Commute

After Splunk’s .conf2016 I found myself spending my free time analyzing the data around me looking for something new to Splunk. This is actually something that happens with increasing regularity actually. After you realize what Splunk can do you can’t help look at the world in a new way. What I didn’t count on, it was my first Splunk .conf after all, was just how much spending a week at Disney World with data enthusiasts would be inspiration viagra. Seriously. There should be a warning at sign up.

Anyway, one of the themes at .conf was the Internet of Things (IoT) and I like many people have started playing with home automation. For me it comes out of a desire to automate life’s repetitive tasks (ain’t nobody got time for that) and use life’s data to make smarter decisions. I’ve got several projects in mind that leverage the power of Splunk and the data generated by by home automation, but right now the focus is on the biggest pain in the ass for anyone living and working in the Atlanta, GA area. The daily commute to the office.

I have played around with Owntracks as a way to get my location data and I liked it as a source for this project because the data was already in key-value pairs and already contained things like longitude and latitude as well as velocity and altitude. Not to mention I was already routing the data to a private mqtt server, so I just needed a way for Splunk to subscribe to my owntracks topic.

On Splunkbase I found Damien Dallimore’s mqtt Modular Input. At first glance I thought this would be perfect. But a couple of things set me in a different direction. First, it relies on having JRE installed on my Splunk server. Not totally bad, but I tend to avoid java as much as I can out of personal preference. And second, I found that I wanted to manipulate the data before it got to Splunk. Nothing big, but I was already thinking I wanted to break out the topic into devices for one. The reality was this second item could have been easily done in Splunk via regex field extractions, but given the first item it made it easy to justify building my own solution.

Since Python seems to be the tool on my belt I am have been reaching for most lately thats what I pulled out for this problem. I didn’t want to totally reinvent the wheel, so I went looking for projects already working in this space. I found an interesting project called mqttwarn that basically subscribes to mqtt topics and then route them to different services although Splunk isn’t one already built in. While it didn’t solve my need out of the box it gave me some ideas and some code I incorporated into my script.

I leveraged the maho-mqtt python library to subscribe to the mqtt topic and then opted to use Splunk Http Event Collector instead of writing the data to a log because why the HEC not. Amiright? By default the data from mqtt was formatted as a string which meant more work once it got into Splunk, so I converted it to a dict (If you didn’t know Splunk is really good at parsing key-value pairs) and then I dropped in George Starcher’s https://github.com/georgestarcher/Splunk-Class-httpevent to handle posting the json to HEC and I was all set. All I had to do was create a HEC token on my Splunk server and fire up my script.

Now comes the fun part of seeing what story the data tells me. It is not a perfect solution, but that wasn’t the end game. Besides whats the fun in that?

I think I might try and rewrite the mqtt modular input on Splunkbase so it doesn’t require JRE, but for now I’ll just let this bake. If you want to see my python solution check it out at https://github.com/thejeffreystone/owntracks-to-splunk

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.