Bii o ṣe le Fi Hadoop Single Node Cluster sori ẹrọ (Pseudonode) lori CentOS 7
Hadoop jẹ ilana orisun-ṣiṣi ti o lo ni ibigbogbo lati ba Bigdata ṣe. Pupọ julọ awọn iṣẹ akanṣe Bigdata/Awọn atupale data ti wa ni kikọ lori oke Hadoop Eco-System. O ni ipele fẹlẹfẹlẹ meji, ọkan jẹ fun Ifipamọ data ati pe omiiran jẹ fun Data Ṣiṣejade.
Ifipamọ yoo ni abojuto nipasẹ eto faili tirẹ ti a pe ni HDFS (Hadoop Distributed Filesystem) ati Processing yoo jẹ abojuto nipasẹ YARN (Sibẹsibẹ Oludunadura Ohun-elo Miran miiran). Mapreduce jẹ ẹrọ ṣiṣe aiyipada ti Hadoop Eco-System.
Nkan yii ṣapejuwe ilana lati fi sori ẹrọ fifi sori ẹrọ Pseudonode ti Hadoop, nibiti gbogbo awọn daemons (JVMs) yoo ṣe iṣupọ Ẹru Nikan ni CentOS 7.
Eyi jẹ akọkọ fun awọn alakọbẹrẹ lati kọ Hadoop. Ni akoko gidi, Hadoop yoo fi sii bi iṣupọ multinode nibiti a yoo pin data laarin awọn olupin bi awọn bulọọki ati pe iṣẹ naa ni yoo ṣe ni ọna ti o jọra.
- Fifi sori ẹrọ ti o kere ju ti olupin CentOS 7.
- Java v1.8 idasilẹ.
Lori oju-iwe yii
- Bii a ṣe le Fi Java sori CentOS 7
- Ṣeto Wiwọle Wiwọle Ọrọigbaniwọle lori CentOS 7 Bii a ṣe le Fi Node Hadoop Kan sii ni CentOS 7 Bii a ṣe le ṣe atunto Hadoop ni CentOS 7 Ṣiṣatunṣe Eto Faili HDFS nipasẹ orukọNode naa
1. Hadoop jẹ Eco-System eyiti o jẹ Java. A nilo Java ti a fi sii ninu eto wa ni aṣẹ lati fi Hadoop sori ẹrọ.
# yum install java-1.8.0-openjdk
2. Itele, ṣayẹwo ẹya ti a fi sori ẹrọ ti Java lori eto naa.
# java -version
A nilo lati ni tunto ssh ninu ẹrọ wa, Hadoop yoo ṣakoso awọn apa pẹlu lilo SSH. Ipilẹ oluwa nlo asopọ SSH lati sopọ awọn apa ẹrú rẹ ati ṣe iṣiṣẹ bi ibẹrẹ ati iduro.
A nilo lati ṣeto ọrọigbaniwọle-kere ssh ki oluwa le ṣe ibasọrọ pẹlu awọn ẹrú nipa lilo ssh laisi ọrọ igbaniwọle kan. Bibẹẹkọ fun idasile asopọ kọọkan, nilo lati tẹ ọrọ igbaniwọle sii.
Ninu oju ipade yii, Awọn iṣẹ Titunto (Namenode, Seconden Namenode & Resource Manager) ati awọn iṣẹ Slave (Datanode & Nodemanager) yoo ṣiṣẹ bi awọn JVM ọtọtọ. Botilẹjẹpe o jẹ oju ipade, a nilo lati ni ssh ọrọigbaniwọle-lati ṣe Titunto si lati ba Slave sọrọ laisi ijẹrisi.
3. Ṣeto wiwọle iwọle SSH ti ko ni ọrọigbaniwọle nipa lilo awọn ofin wọnyi lori olupin naa.
# ssh-keygen # ssh-copy-id -i localhost
4. Lẹhin ti o tunto iwọle SSH ti ko ni ọrọigbaniwọle, gbiyanju lati buwolu wọle lẹẹkansii, iwọ yoo ni asopọ laisi ọrọ igbaniwọle kan.
# ssh localhost
5. Lọ si oju opo wẹẹbu Hadoop Afun ki o gba igbasilẹ iduroṣinṣin ti Hadoop nipa lilo pipaṣẹ wget atẹle.
# wget https://archive.apache.org/dist/hadoop/core/hadoop-2.10.1/hadoop-2.10.1.tar.gz # tar xvpzf hadoop-2.10.1.tar.gz
6. Nigbamii, ṣafikun awọn oniyipada ayika Hadoop ninu ~/.bashrc faili bi o ti han.
HADOOP_PREFIX=/root/hadoop-2.10.1 PATH=$PATH:$HADOOP_PREFIX/bin export PATH JAVA_HOME HADOOP_PREFIX
7. Lẹhin fifi awọn oniyipada agbegbe kun si ~/.bashrc faili naa, orisun faili naa ki o jẹrisi Hadoop nipa ṣiṣe awọn ofin wọnyi.
# source ~/.bashrc # cd $HADOOP_PREFIX # bin/hadoop version
A nilo lati tunto ni isalẹ awọn faili iṣeto Hadoop lati le ba ẹrọ rẹ mu. Ni Hadoop, iṣẹ kọọkan ni nọmba ibudo tirẹ ati itọsọna tirẹ lati tọju data naa.
- Awọn faili iṣeto ni Hadoop - core-site.xml, hdfs-site.xml, mapred-site.xml & yarn-site.xml
8. Ni akọkọ, a nilo lati mu imudojuiwọn JAVA_HOME ati ọna Hadoop ninu faili hadoop-env.sh bi o ti han.
# cd $HADOOP_PREFIX/etc/hadoop # vi hadoop-env.sh
Tẹ laini atẹle ni ibẹrẹ faili naa.
export JAVA_HOME=/usr/lib/jvm/java-1.8.0/jre export HADOOP_PREFIX=/root/hadoop-2.10.1
9. Itele, tunṣe core-site.xml faili.
# cd $HADOOP_PREFIX/etc/hadoop # vi core-site.xml
Lẹẹmọ atẹle atẹle laarin awọn
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
10. Ṣẹda awọn ilana ni isalẹ labẹ tecmint itọsọna ile olumulo, eyi ti yoo ṣee lo fun NN ati ipamọ DN.
# mkdir -p /home/tecmint/hdata/ # mkdir -p /home/tecmint/hdata/data # mkdir -p /home/tecmint/hdata/name
10. Nigbamii, yipada hdfs-site.xml faili.
# cd $HADOOP_PREFIX/etc/hadoop # vi hdfs-site.xml
Lẹẹmọ atẹle atẹle laarin awọn
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/tecmint/hdata/name</value>
</property>
<property>
<name>dfs .datanode.data.dir</name>
<value>home/tecmint/hdata/data</value>
</property>
</configuration>
11. Lẹẹkansi, tunṣe maapu-site.xml faili.
# cd $HADOOP_PREFIX/etc/hadoop # cp mapred-site.xml.template mapred-site.xml # vi mapred-site.xml
Lẹẹmọ atẹle atẹle laarin awọn
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
12. Ni ikẹhin, ṣe atunṣe faili yarn-site.xml faili.
# cd $HADOOP_PREFIX/etc/hadoop # vi yarn-site.xml
Lẹẹmọ atẹle atẹle laarin awọn
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
13. Ṣaaju ki o to bẹrẹ Iṣupọ, a nilo lati ṣe agbekalẹ Hadoop NN ninu eto agbegbe wa nibiti o ti fi sii. Nigbagbogbo, yoo ṣee ṣe ni ipele ibẹrẹ ṣaaju ki o to bẹrẹ iṣupọ ni igba akọkọ.
Ṣiṣe kika NN yoo fa isonu data ni metastore NN, nitorinaa a ni lati ṣọra diẹ sii, a ko gbọdọ ṣe agbekalẹ NN lakoko ti iṣupọ n ṣiṣẹ ayafi ti o ba nilo imomose.
# cd $HADOOP_PREFIX # bin/hadoop namenode -format
14. Bẹrẹ daemon NameNode ati daemon DataNode: (ibudo 50070).
# cd $HADOOP_PREFIX # sbin/start-dfs.sh
15. Bẹrẹ ResourceManager daemon ati NodeManager daemon: (ibudo 8088).
# sbin/start-yarn.sh
16. Lati da gbogbo awọn iṣẹ duro.
# sbin/stop-dfs.sh # sbin/stop-dfs.sh
Akopọ
Ninu àpilẹkọ yii, a ti lọ nipasẹ igbesẹ nipasẹ ilana igbesẹ lati ṣeto Ikojọpọ Hadoop Pseudonode (Node Nikan). Ti o ba ni imoye ipilẹ ti Linux ki o tẹle awọn igbesẹ wọnyi, iṣupọ naa yoo wa ni UP ni iṣẹju 40.
Eyi le wulo pupọ fun alakọbẹrẹ lati bẹrẹ ikẹkọ ati adaṣe Hadoop tabi ẹya vanilla ti Hadoop yii le ṣee lo fun awọn idi Idagbasoke. Ti a ba fẹ lati ni iṣupọ akoko gidi, boya a nilo o kere ju awọn olupin ti ara 3 ni ọwọ tabi ni lati pese awọsanma fun nini awọn olupin pupọ.