Chicago Apache Flink Meetup (CHAF):Hands-on Apache Flink Workshop (February 21, 2017)

From the promotional materials: 

In this workshop, we will go over a hands on tutorial highlighting building a Flink stream processor.

The workshop will show you how to

1. Set up a stream consisting of Chicago Transit Authority (CTA) bus tracker data

2. Use different types of windowing techniques to generate metrics on the CTA

3. Sink the metrics to a queryable data sink

4. Turn all of the above into a deployable job that can be scaled up.

You will see all of Flink's major components in action: streams, sinks, windows, triggers, watermarks, etc.

A preconfigured virtual machine containing the workshop materials will be available for download prior to the meetup.

Pizza and Soda will be provided!

Joe Olson is a local area developer specializing in applying emerging technology to solve business problems, and is currently focusing on understanding the rapidly developing streaming space.

My personal notes:

• Joe has been working with Flink for 1 to 1.5 years now
• the only way to get community going is to get folks involved
• most meetup groups consist of PowerPoint slides
• streaming is hot right now…big data, IoT, and real time ML applying the pressure
• there are currently several emerging frameworks and philosophies that will likely converge at some point
• Flink is where Hadoop was back around 2009
• one way to possibly jump start adoption is through some hackathons
• what we are about to do:
• – ingest CTA bus tracker historical data set
• – use the Flink streaming API to transform this data set into a Kafka topic…this will provide a good overview of the basic streaming model
• – view the data inside Kafka
• – use a separate Flink job to process the data
• – load the data into keyed windows
• – fire the window on count or on timeout
• – aggregate the data when fired
• – sink the aggregated data
• it took Joe a long time to figure all of this out, so he wants to pass all of this on to folks
• the VM is Ubuntu 16.04, Flink 1.2, Kafka 0.9, Kafkatool, IntelliJ 2016.3 Community Version, Scala 2.11, Java 8, Gradle 3.2
• the data consists of one day's worth of CTA data in JSON format
• Joe commented that it is important to know the version of Scala being used when using Flink, otherwise it won't work
• the CTA has a public API that returns minute fixes of all of its buses
• a fix: {"vid": 1958, "tmstmp": "20150211 23:59", "lat": 41.880638122558594, "lon": -87.738800048828125, "hdg": 267, "pid": 949, "rt": "20", "des": "Austin", "pdist": 3429, "tablockid": "N20 -893", "tatripid": 1040830, "zone": "null”}
• just under 1m fixes are created per day…about 250MB uncompressed
• about 2 years of archived daily data also exists: https://github.com/jolson7168/ctaData
• a good deal of Flink jobs fall into this pattern:
• – connect to a data source and set up a stream
• – do processing on each element in the stream
• – agreggrating, filtering, buffering
• sink results of processing (or start another stream)
• at this point, Joe transitioned to demonstrations by first bringing up IntelliJ
• the code is in Github…first do a git pull to refresh the VM, as he made some changes after creating the image
• there are 2 sections of the project…a loader and a processor…the top of the project loads Kafka and the bottom processes the data
• Joe commented that a lot of stuff will get thrown at you in the Flink documentation
• typical steps for a job are to get data from Kafka, process the data, and then sink it
• FixProcessor is a basic job
• all of the build files are set up if you want to compile this into a JAR
• the notes provide the Gradle build command and Flink start command that are needed
• the Flink job manager can be accessed multiple ways, including via browser
• Kafka is strange in that Zookeeper needs to be started separately prior to starting Kafka
• Joe said that we can email or call him with issues, assistance etc with regard to this project
• he noted that everything is installed to the Ubuntu VM home directory
• a graphical way to see contents of Kafka is by using something called Kafka Tool
• create a Kafka topic for output
• there's not a lot you can do with the Flink UI…it's pretty primitive right now
• deploy the JAR to the Flink server via the Flink UI after it is done building
• Boris (one of the CHUG organizers) mentioned that zookeeperutils can be used to simplify
• the JSON file is in the home directory
• the CTA pushes out "half-assed XML files"…the provided JSON file is much better than the XML
• Flink is kind of unique in that it works like SQL in the sense that it determines an optimal plan as to how to run each job
• Joe commented that this is a very trivial job, but it will help you start understanding how Flink processes data
• JSON fields that are not being used are discarded on the way to Kafka
• letting this job run for a couple minutes will get up to 1m records
• to get this job to run faster, "just throw more processors at it"…cluster it
• job #1 takeaways:
• – very simple Flink job
• – connect to source (data file on OS)
• – filter (make sure valid JSON)
• – sink valid fixes to Kafka topic
• – compile into a JAR
• – deploy JAR on server
• – observe execution
• – check out the log files
• Flink logs and you can load into Kibana to search them…don't mess around with the Flink logs directly themselves
• Flink controls the connection to the source and the sink…the logging it's doing is better than anything you'll be able to do
• it will save you a lot of heartache in the long run if you just get used to how Flink logs
• Kafka takeaways
• – start/stop Kafka and Zookeeper
• – observe what is in a Kafka topic
• – use Kafka for both the source and the sink for Flink
• the new version of Kafka apparently replaces offsets with timestamps, which should be more useful
• using Flink 1.2, which just came out…it supports Kafka .10
• state management is better for 1.2
• back to IntelliJ
• Joe then progressed to the next job
• need to get timestamp of each event
• Joe commented that he didn't respect the boundary between event time and process time…when data comes in, you need to tell Flink the timestamp and the format it is in
• once the timestamp is obtained, you have an anchor of when the event took place
• Joe didn't understand the intricacies of events that took place in the past, so didn't get this job done
• there's always 2 boundaries in Flink…the watermark is the latest point in time that Flink knows about
• he he didn't know how to simulate this with historical data
• this job shows how to get the timestamp and how to set the watermark, but doesn't work completely as of yet
• there are 3 different window types and different strategies as to how you can use them
• the windowing feature in Flink is one of it's greatest strengths
• Kafka is starting to go in the same direction as Flink, but Flink has the jump on this by far
• Boris said Kafka went in the opposite, simpler direction because it was too complex
• we don't know who is going to win this contest, but the ideas will remain
• try to follow all of the open source products in this space as to what they're trying to do, and you will get a good idea as to where the next 10 years will go
• someone said that setting up Kafka was the hardest part…Boris argued that it is the easiest part (and I concur with Boris that setting up Kafka is simple)
• the 2 things to take away is that there is a trigger as to when the window fires…you can override this with your own trigger
• it took Joe a long time to figure out how to do this
• there are different ways to handle late data, out of order data…Flink handles these types of cases well
• for example, you could create a 1 hour window, but keep it open a few more minutes to catch any stragglers
• next month's meetup will be grabbing CTA data in real time and applying windows against it
• Flink will give you guarantees
• if you are managing state, you can use RocksDB to…it is optimized for this…RocksDB is the heavy hitter for managing state
• Boris commented that RocksDB is local to the CPU, and it confused the heck out of him…people want to call this state, but it isn't state
• you will run into a lot of trouble if you don't play by the rules that Flink sets…for what it wasn't designed to do…they joked that this is because it's German
• Data Artisans wants to make Flink the database itself…a queriable state feature
• the table is a stream…the stream is a table
• Boris joked that nobody knows what this means, but they all want to do it…Boris said that the stream is not a database, it's a log
• Boris said that his "head spins around" when folks start talking about "infinite streaming" etc
• for debugging purposes, you can just run within IntelliJ using Gradle
• someone asked about unit testing strategies, and he said that they are struggling with this right now
• Boris disagreed…he said that people struggled with Hadoop a few years ago in a similar way
• job #2 takeaways:
• – another simple flink job
• – connect to source (JSON in Kafka)
• – create a fix job from JSON
• – identify the timestamp from the fix
• – key the stream
• – send fixes into windows based on key
• – trigger the window by count or by timeout
• – aggregate items in window on triggering
• – sink aggregate results to PostgreSQL
• – using a custom window
• – using a custom trigger…preserving state
• – using a custom aggregation
• – sinking to a relational database
• – run from within IntelliJ…useful for rapid debugging
• Joe has Flink in production…doesn't know anyone else that does
• Joe encouraged us to present at future meetup sessions, even if it is to present something small for 5 or 10 minutes