One of the challenges with lots of data is actually figuring out how to collect and process it. At huge scale the presentation that was given by Matthew Johnson with Netflix will scratch that itch for folk. Clearly designing and building systems for something as large and distributed as Netflix is complex, you just don't realize how complex until someone walks you through some of the math involved. It is an interesting presentation and worth a watch.
I found the presentation given later in the day by Phillip Liu from SignalFx to be thought provoking and this is the presentation I am referring to about being worth a second look. At first I didn't understand what SignalFx was actually providing that was different than what Splunk or DataDog might be doing. It wasn't until later in the presentation that I figured out what they had done. The online presentation is a very different experience then being there in person believe it or not. Phillip speaks very softly so most of the participants in the room really couldn't hear him well. This all changes with the video because he wore a mic for the streaming and video recording. I didn't get the light bulb moment until 41 minutes into a 54 minute presentation. Yes, I am that dim. You can hear me catch on at about 42:19 in the video.
Here in the video jumping in a little bit before I figure that out. https://youtu.be/5pXt659j8Wk?t=41m22s
In summary, SignalFx is providing math (function) as a service! You can do a variety of functional equations on data points as they are streaming into the product. This allows you to do all sort of interesting things with the data in real time that in the past you had to run large data reports on to produce. Even more impressive is that anyone with basic math skills can use the tool or leverage what SignalFx is providing out of the box with integration with other major platforms. What was really interesting is that Cisco did a demo after that showing their IOS XR routing platform being able to stream data and have SignalFx do something with that data. This effort was possible because of Cisco's work with OpenConfig. Anees Shaikh with Google also presented talking about the efforts to standardize network management and monitoring. It builds out the foundation for why this new software first world is changing what is happening in networking so profoundly.
I'm glad I was able to finally make another Tech Field Day event and I think this was a great one to be involved with as it is an area of professional interest.
- Ed