沒有 Cloudera Manager 的 CDH3 再平衡器

我們最近向集群添加了新節點，我想執行重新平衡器以在這些節點之間分配數據。我們正在使用 CDH3 並且整個系統都已安裝 - 我什至不確定這些盒子中是否安裝了 cloudera 管理器。

如果沒有 Cloudera Manager，我能否執行平衡器？進行重新分配的正確方法是什麼？

如果這是一個非常明顯的問題，請原諒，我在Google上搜尋了很多，也在這裡搜尋了一些問題，但沒有成功，因此有這些問題。

PS：我別無選擇，只能把它放在 CDH4 中，因為它不會讓我創建 CDH3 標籤。

是的，您可以在沒有 CM 的情況下執行平衡器。您將需要執行以下操作，最好從您的 namenode 執行，但它應該在任何有權訪問的節點或客戶端上執行。

首先，執行 screen 或 tmux。此過程可能需要一段時間。如果您不在 screen/tmux 中執行它，沒有什麼不好的事情發生，這只是為了保護您與遠端系統的連接斷開。

如果你沒有執行 Kerberos，你可以這樣做：

sudo su - hdfs
hadoop balancer -threshold <somevalue> > balance.out 2>&1

如果您正在執行 Kerberos，則需要執行以下操作：

sudo su - hdfs
kinit -t -k /path/to/your/hdfs.keytab hdfs/fully.qualified.hostname
hadoop balancer -threshold <somevalue> > balance.out 2>&1

然後在不同的 shell 中，您可以拖尾輸出，通過適當的 grep 將其傳遞以清除平衡器產生的無用資訊。

我使用這樣的東西：

tail -f balance.out | grep -v Moving

這意味著我將看到如下資訊：

13/11/25 05:53:38 INFO balancer.Balancer: 0 over utilized nodes:
13/11/25 05:53:38 INFO balancer.Balancer: 1 under utilized nodes:  192.168.1.151:50010
13/11/25 05:53:38 INFO balancer.Balancer: Need to move 181.53 MB bytes to make the cluster balanced.
13/11/25 05:53:38 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.1.131:50010 to 192.168.1.151:50010
13/11/25 05:53:38 INFO balancer.Balancer: Will move 10 GBbytes in this iteration
Nov 25, 2013 5:53:38 AM        2203             47.76 TB           181.53 MB              10 GB
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.147:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.122:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.137:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.128:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.145:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.126:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.149:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.146:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.153:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.156:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.151:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.134:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.135:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.154:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.144:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.125:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.148:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.139:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.152:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.133:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.132:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.136:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.150:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.129:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.130:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.142:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.123:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.127:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.160:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.158:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.131:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.138:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.124:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.140:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.159:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.121:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.155:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.141:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.157:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.143:50010
13/11/25 05:54:23 INFO balancer.Balancer: 0 over utilized nodes:
13/11/25 05:54:23 INFO balancer.Balancer: 0 under utilized nodes: 
The cluster is balanced. Exiting...
Balancing took 90.16988833333333 hours

最後，如果您覺得重新平衡花費的時間太長，您可以調整dfs.balance.bandwidthPerSec。需要在每個datanode上設置，重啟hadoop-0.20-datanode程序生效。我記得預設是 1MB/s。該值以字節為單位指定。

引用自：https://serverfault.com/questions/566557