Flume Hello World tutotiral

I am using flume for some time now and really like it. This is simple HelloWorld tutorial that i thought would be helpful if you want to get started with Flume. This tutorial will walk you through steps for setting up Flume that listens to messages on port 44444, once it gets message it just prints it out on console, Follow these steps

  1. First create sampleflume.properties file on your machine like this

# example.conf: A single-node Flume configuration# Name the components on this agentagent1.sources = netcat1agent1.sinks = logger1agent1.channels = memory1# Describe/configure the sourceagent1.sources.netcat1.type = netcatagent1.sources.netcat1.bind = localhostagent1.sources.netcat1.port = 44444# Describe the sinkagent1.sinks.logger1.type = logger# Use a channel which buffers events in memoryagent1.channels.memory1.type = memoryagent1.channels.memory1.capacity = 1000agent1.channels.memory1.transactionCapacity = 100# Bind the source and sink to the channelagent1.sources.netcat1.channels = memory1agent1.sinks.logger1.channel = memory1

Your flume configuration file must have at least 3 elements a source, channel and sink

  • netcat1: netcat1 source defines how flume is listening to messages. In this case type of netcat means it will listen on port that you can connect to using either netcat or telnet
  • memory: memory channel defines how flume stores messages that it has received before they are consumed by sink. In this case i am saying keep the messages in memory
  • logger1: Logger sink is for testing, it just prints the messages on console
  • Once your configuration file is ready you can start a flume agent by executing following command
    flume-ng agent --conf conf --conf-file sampleflume.properties  --name agent1 -Dflume.root.logger=DEBUG,console

    YOu will see flume printing messages on the console while it is starting like this

  • Once server is started you can connect to it using nc or telnet and send messages to it like this. Whatever messages you send will be printed to console
  • Once you send messages using nc command look at the server console and you should see the messages that you sent