satya - 3/13/2018, 2:50:35 PM

Apache Flume

satya - 3/13/2018, 2:55:42 PM

About Apache Flume

satya - 3/13/2018, 2:58:20 PM

Kafka, Apache Flume

satya - 3/13/2018, 3:08:18 PM

Here is how Kafka and Flume work together

satya - 3/13/2018, 3:08:48 PM

Here is kafka without flume

satya - 3/13/2018, 3:13:00 PM

Here is flume with kafka

satya - 3/13/2018, 3:17:48 PM

So whats the difference between Kafka and Flume

Flume connects data sources and sinks in a topology via channels.

It is agnostic to soruces, sinks, and channels

Data flows between them

However if the channels are not fault tolerant data loss can happen

So Kafka can be used as a channel

satya - 3/13/2018, 3:18:41 PM

So flume is the set of highways that delivers events

satya - 3/13/2018, 3:19:14 PM

Apache storm can process the events delivered flume

satya - 3/13/2018, 3:19:49 PM

So apache spark works better with Kafka, while Apache storm works better with flume