复制因子的配置项为 <hadoop_home>/etc/hadoop/hdfs-site.xml 中的 dfs.replication
表示一个数据文件的备份数, 我们使用如下命令可以查看文件的分块信息、副本个数、存储位置
hdfs fsck <file> -blocks -files -locations -racks
[hadoop@txz-store hadoop]$ hdfs fsck /envs/py3.tar.gz -blocks -files -locations -racks
Connecting to namenode via <http://txz-data0:50070/fsck?ugi=hadoop&blocks=1&files=1&locations=1&racks=1&path=%2Fenvs%2Fpy3.tar.gz>
FSCK started by hadoop (auth:SIMPLE) from /192.168.0.250 for path /envs/py3.tar.gz at Tue Apr 02 16:35:15 CST 2019
/envs/py3.tar.gz 715590742 bytes, 6 block(s): OK
0. BP-1634922614-192.168.0.212-1545292690839:blk_1074012799_271985 len=134217728 Live_repl=1 [/default-rack/192.168.0.250:50010]
1. BP-1634922614-192.168.0.212-1545292690839:blk_1074012800_271986 len=134217728 Live_repl=1 [/default-rack/192.168.0.250:50010]
2. BP-1634922614-192.168.0.212-1545292690839:blk_1074012801_271987 len=134217728 Live_repl=1 [/default-rack/192.168.0.250:50010]
3. BP-1634922614-192.168.0.212-1545292690839:blk_1074012802_271988 len=134217728 Live_repl=1 [/default-rack/192.168.0.250:50010]
4. BP-1634922614-192.168.0.212-1545292690839:blk_1074012803_271989 len=134217728 Live_repl=1 [/default-rack/192.168.0.250:50010]
5. BP-1634922614-192.168.0.212-1545292690839:blk_1074012804_271990 len=44502102 Live_repl=1 [/default-rack/192.168.0.250:50010]
Status: HEALTHY
Total size: 715590742 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 119265123 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 1
Average block replication: 1.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Tue Apr 02 16:35:15 CST 2019 in 1 milliseconds
The filesystem under path '/envs/py3.tar.gz' is HEALTHY
从上面的信息可以看到 Default replication factor 值为1, 表示数据的复制因子为 1, 从分块的保存信息上看, 数据的分块都只有一个副本.
在我们调整 datanode 节点数量的时候, 新更新的复制因子对原来的文件不会起作用
所以当我们需要调整原来的文件的复制因子时, 执行
hadoop fs -setrep -R <复制因子数量> <dir/file>
并且复制因子数量不能大于 datanode 节点数量, 超出部分不会生效.
调整完成之后再次查看文件的复制因子, 会发现此时数据仍旧是 1个副本, 但是显示丢失副本. Missing replicas
然后等待, 让hdfs 执行自行恢复缺少的副本
通过访问 <name_node_ui>/dfshealth.html#tab-overview 找到 Under-Replicated 项可以看到当前需要复制的副本数
和扩大的方式类似, 无法重新执行