Apache-Mesos

Apache Mesos 啟動一次,但一旦我關閉它就不會重新啟動

  • November 5, 2015

我在 redhat 6.6 機器上建構了 Apache Mesos。它已安裝到我將呼叫的目錄$INST中。

我設置$INST/etc/masters$INST/etc/slaves擁有主人和奴隸的主機名,然後$INST/etc/mesos-slave-env.sh像這樣設置:

export MESOS_work_dir=/path/to/some/directory/$HOSTNAME/work
export MESOS_log_dir=/path/to/some/directory/$HOSTNAME/log
export MESOS_master=masternodename:5050

$INST/etc/mesos-master-env.sh完全一樣,但沒有MESOS_master定義。

/path/to/some/directory可以由所有節點共享,也可以在我遇到的行為的所有節點上唯一。

然後我$INST/sbin/mesos-start-cluster第一次跑。開始。我可以打開firefox masternodename:5050並查看 Web UI,它顯示了所有附加的從站。

但是,如果我跑去$INST/sbin/mesos-stop-cluster.sh殺死集群,然後用 重新啟動它mesos-start-cluster.sh,它就會永遠掛起。curl masternodename:5050打開到埠的連接並永遠等待數據,但沒有數據。日誌顯示了這一點,並且從這裡沒有任何進展:

I1102 16:05:39.334799 27997 logging.cpp:172] INFO level logging started!
I1102 16:05:39.335925 27997 main.cpp:229] Build: 2015-11-02 20:29:24 by sbhaide
I1102 16:05:39.335942 27997 main.cpp:231] Version: 0.25.0
I1102 16:05:39.336308 27997 main.cpp:252] Using 'HierarchicalDRF' allocator
I1102 16:05:39.344976 27997 leveldb.cpp:176] Opened db in 7.787897ms
I1102 16:05:39.346916 27997 leveldb.cpp:183] Compacted db in 1.90886ms
I1102 16:05:39.347038 27997 leveldb.cpp:198] Created db iterator in 94694ns
I1102 16:05:39.347062 27997 leveldb.cpp:204] Seeked to beginning of db in 4003ns
I1102 16:05:39.347074 27997 leveldb.cpp:273] Iterated through 0 keys in the db in 513ns
I1102 16:05:39.347393 27997 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I1102 16:05:39.351538 28017 recover.cpp:449] Starting replica recovery
I1102 16:05:39.352499 27997 main.cpp:465] Starting Mesos master
I1102 16:05:39.352665 28017 recover.cpp:475] Replica is in EMPTY status
I1102 16:05:39.356853 28023 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request
I1102 16:05:39.356978 28025 master.cpp:376] Master 6fa2ccac-3527-4522-a72d-8eeba06f55eb (xxxxxx.xxx.xxxxxxxx.xxx.xxx) started on 10.148.0.101:5050
I1102 16:05:39.357002 28025 master.cpp:378] Flags at startup: --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --log_dir="/path/to/log/directory/xxxxxxxx/log" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --port="5050" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/install/directory/path/share/mesos/webui" --work_dir="..." --zk_session_timeout="10secs"
I1102 16:05:39.357393 28025 master.cpp:425] Master allowing unauthenticated frameworks to register
I1102 16:05:39.357405 28025 master.cpp:430] Master allowing unauthenticated slaves to register
I1102 16:05:39.357467 28025 master.cpp:467] Using default 'crammd5' authenticator
W1102 16:05:39.357502 28025 authenticator.cpp:505] No credentials provided, authentication requests will be refused
I1102 16:05:39.358242 28025 authenticator.cpp:512] Initializing server SASL
I1102 16:05:39.359158 28011 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1102 16:05:39.360354 28029 recover.cpp:566] Updating replica status to STARTING
I1102 16:05:39.361856 28016 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 1.183548ms
I1102 16:05:39.361889 28016 replica.cpp:323] Persisted replica status to STARTING
I1102 16:05:39.362313 28014 recover.cpp:475] Replica is in STARTING status
I1102 16:05:39.363344 28014 replica.cpp:641] Replica in STARTING status received a broadcasted recover request
I1102 16:05:39.363711 28016 recover.cpp:195] Received a recover response from a replica in STARTING status
I1102 16:05:39.364202 28007 recover.cpp:566] Updating replica status to VOTING
I1102 16:05:39.364570 28029 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 222611ns
I1102 16:05:39.364594 28029 replica.cpp:323] Persisted replica status to VOTING
I1102 16:05:39.364678 28022 recover.cpp:580] Successfully joined the Paxos group
I1102 16:05:39.364972 28022 recover.cpp:464] Recover process terminated

(數據有些匿名)

我第一次啟動它時它工作得很好,但第二次就永遠掛起,我不知道為什麼。它必須在某處儲存狀態資訊,但lsof沒有向我顯示它在執行期間可能會影響任何節點的任何文件!

任何想法在哪裡看或可能導致這種情況?

問題是系統的熵池已經耗盡,並且它在讀取到/dev/random;時被阻塞。它是通過編譯一個新版本的cyrus-sasl庫來使用/dev/urandom而不是 解決的/dev/random,並將我的 mesos 連結到那個。

引用自:https://serverfault.com/questions/733420