首先应确定master和slave上都安装了Flume,如果slave上没有,可以将master上的传过去
scp -r /opt/flume/ root@slave1:/opt/ 配置一级Flume 在master上创建netcat-avro.conf,意思是一级flume从444端口收集信息,再从555端口以avro格式发到二级flume
cd /opt/flume/conf vi netcat-avro.conf => a1.sources = r1 a1.channels = c1 a1.sinks = k1 a1.sources.r1.type = netcat a1.sources.r1.bind = master a1.sources.r1.port = 444 a1.sinks.k1.type = avro a1.sinks.k1.channel = c1 a1.sinks.k1.hostname=slave1 a1.sinks.k1.port = 555 a1.channels.c1.type = memory a1.channels.c1.capacity = 100 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 <= 配置二级Flume 在slave上创建avro-logger.conf,意思是二级flume从555端口收集信息
cd /opt/flume/conf vi avro-logger.conf => a1.sources = r1 a1....
Flume用于收集、聚合和传输大量数据,如日志文件、事件等。
下载Flume安装包到master节点
wget --no-check-certificate https://od.fifcom.cn/misc/hadoop/apache-flume-1.8.0-bin.tar.gz 安装Flume 解压Flume
tar -zxvf apache-flume-1.8.0-bin.tar.gz -C /opt/ mv /opt/apache-flume-1.8.0-bin/ /opt/flume/ 进入Flume的配置文件目录,并从模板复制配置文件
cd /opt/flume/conf cp flume-env.sh.template flume-env.sh 编辑环境配置文件中的JAVA_HOME
vi flume-env.sh => export JAVA_HOME=/opt/jdk8u282-b08 <= 在当前目录创建netcat-logger.conf文件,用于配置一个简单的数据采集方案
vi netcat-logger.conf => a1.sources = r1 a1.sinks = k1 a1.channels = c1 a1.sources.r1.type = netcat a1.sources.r1.bind = master a1.sources.r1.port = 444 a1.sinks.k1.type = logger a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 <= 测试Flume 首先回到Flume根目录,并使用命令启用Flume...