Configuring Flume to write avro events into HDFS

Recently i wanted to figure out how to configure Flume so that it can listen for Avro Events and whenever it gets event it should dump it in the HDFS. In order to do that i built this simple Flume configuration

# example.conf: A single-node Flume configuration# Name the components on this agentagent1.sources = avroagent1.sinks = logger1agent1.channels = memory1# Describe/configure the sourceagent1.sources.avro.type = avroagent1.sources.avro.bind = localhostagent1.sources.avro.port = 41414agent1.sources.avro.selector.type = replicatingagent1.sources.avro.channels = memory1# Describe the sinkagent1.sinks.hdfs1.type = hdfsagent1.sinks.hdfs1.hdfs.path=/tmp/flume/eventsagent1.sinks.hdfs1.hdfs.rollInterval=60#The number of events to be written into a file before it is rolled.agent1.sinks.hdfs1.hdfs.rollSize=0agent1.sinks.hdfs1.hdfs.batchSize=100agent1.sinks.hdfs1.hdfs.serializer=org.apache.flume.sink.hdfs.AvroEventSerializer$Builderagent1.sinks.hdfs1.hdfs.fileType = DataStreamagent1.sinks = = memory1# Use a channel which buffers events in memoryagent1.channels.memory1.type = memoryagent1.channels.memory1.capacity = 1000agent1.channels.memory1.transactionCapacity = 100

In this i have Avro source listening on local machine at port 41414, once it gets event it writes that in HDFS in /tmp/flume/events directory Once this file is saved in local machine as hellohdfsavro.conf i can start the flume agent using following command

flume-ng agent --conf conf --conf-file hellohdfsavro.conf  --name agent1 -Dflume.root.logger=DEBUG,console