本文共 1943 字，大约阅读时间需要 6 分钟。

1.实现功能：

扇入：flume可以实现从数以百计的Web servers中收集信息，然后将日志信息传送到十几个agent服务器，最后写到hdfs上。本文实例采用的就是实现这个扇入功能：

flume1：收集4666端口信息，并输送到Avro Sink

flume2：监控日志信息，并输送到Avro Sink

flume3：收集flume1 Avro Sink和flume2 Avro Sink的数据，写入到hdfs上。

 

2.代理

（1）flume1

# Name the components on this agent

a1.sources = r1 a1.sinks = k1 a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = netcat a1.sources.r1.bind = bigdata.ibeifeng.com a1.sources.r1.port = 4666

# Describe the sink

a1.sinks.k1.type = avro a1.sinks.k1.hostname = bigdata.ibeifeng.com a1.sinks.k1.port = 6666

# Use a channel which buffers events in memory

a1.channels.c1.type = memory #channe最多存储1000事件 a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

（2）flume2

a2.sources = r2

a2.sinks = k2 a2.channels = c2

#定义source的类型

a2.sources.r2.type = exec a2.sources.r2.command = tail -f  /var/log/httpd/access_log a2.sources.r2.channels = c2

#定义channel

a2.channels.c2.type = memory #存储1000个event事件 a2.channels.c2.capacity = 1000 a2.channels.c2.transactionCapacity = 100

          

#定义sink a2.sinks.k2.type = avro a2.sinks.k2.hostname = bigdata.ibeifeng.com a2.sinks.k2.port = 6666 a2.sinks.k2.channel = c2

（3）flume3

a3.sources = r3

a3.sinks = k3 a3.channels = c3

#定义source的类型

a3.sources.r3.type = avro a3.sources.r3.bind = bigdata.ibeifeng.com a3.sources.r3.port = 6666 a3.sources.r3.channels = c3

#定义channel

a3.channels.c3.type = memory #存储1000个event事件 a3.channels.c3.capacity = 1000 a3.channels.c3.transactionCapacity = 100

#定义sink a3.sinks.k3.type = hdfs a3.sinks.k3.hdfs.path = hdfs://bigdata.ibeifeng.com:8020/opo a3.sinks.k3.channel = c3

 

3.启动

bin/flume-ng agent --name a3 --conf  conf --conf-file conf/avro-hdfs.properties &

bin/flume-ng agent --name a2 --conf  conf --conf-file conf/apache-avro.properties & bin/flume-ng agent --name a1 --conf  conf --conf-file conf/flume-test2.properties &

（经测试，成功！）