Coreos
為 TLS 配置 etcd2 的問題
我家裡有兩台 CoreOS stable v1122.2.0 機器。他們都在discovery.etcd.io正確註冊。（當我瀏覽到發現 url 時，我看到它們都在https://ip:2380註冊）。
第一個伺服器是 10.79.218.2，第二個是 10.79.218.3。
我正在嘗試為 tls 配置 etcd。所以我創建了所需的證書並為 tls 配置了 etcd2。所有配置（本地 IP 為 10.79.218.2 的第一台伺服器）都在 cloud-config.yml 文件中：
#cloud-config write_files: - path: "/etc/ssl/etcd/ca.pem" permissions: "0600" owner: "etcd:etcd" content: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - path: "/etc/ssl/etcd/server1.pem" permissions: "0600" owner: "etcd:etcd" content: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - path: "/etc/ssl/etcd/server1-key.pem" permissions: "0600" owner: "etcd:etcd" content: | -----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY----- - path: "/etc/kubernetes/cni/net.d/10-calico.conf" content: | { "name": "calico", "type": "flannel", "delegate": { "type": "calico", "etcd_endpoints": "https://10.79.218.2:2379,https://10.79.218.3:2379", "log_level": "none", "log_level_stderr": "info", "hostname": "10.79.218.2", "policy": { "type": "k8s", "k8s_api_root": "http://127.0.0.1:8080/api/v1/" } } } - path: "/etc/kubernetes/manifests/policy-controller.yaml" content: | apiVersion: v1 kind: Pod metadata: name: calico-policy-controller namespace: calico-system spec: hostNetwork: true containers: # The Calico policy controller. - name: k8s-policy-controller image: calico/kube-policy-controller:v0.2.0 env: - name: ETCD_ENDPOINTS value: "https://10.79.218.2:2379,http://10.79.218.3:2379" - name: K8S_API value: "http://127.0.0.1:8080" - name: LEADER_ELECTION value: "true" # Leader election container used by the policy controller. - name: leader-elector image: quay.io/calico/leader-elector:v0.1.0 imagePullPolicy: IfNotPresent args: - "--election=calico-policy-election" - "--election-namespace=calico-system" - "--http=127.0.0.1:4040" - path: "/etc/kubernetes/manifests/kube-scheduler.yaml" content: | apiVersion: v1 kind: Pod metadata: name: kube-scheduler namespace: kube-system spec: hostNetwork: true containers: - name: kube-scheduler image: quay.io/coreos/hyperkube:v1.3.6_coreos.0 command: - /hyperkube - scheduler - --master=http://127.0.0.1:8080 - --leader-elect=true livenessProbe: httpGet: host: 127.0.0.1 path: /healthz port: 10251 initialDelaySeconds: 15 timeoutSeconds: 1 - path: "/etc/kubernetes/manifests/kube-controller-manager.yaml" content: | apiVersion: v1 kind: Pod metadata: name: kube-controller-manager namespace: kube-system spec: hostNetwork: true containers: - name: kube-controller-manager image: quay.io/coreos/hyperkube:v1.3.6_coreos.0 command: - /hyperkube - controller-manager - --master=http://127.0.0.1:8080 - --leader-elect=true - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem - --root-ca-file=/etc/kubernetes/ssl/ca.pem livenessProbe: httpGet: host: 127.0.0.1 path: /healthz port: 10252 initialDelaySeconds: 15 timeoutSeconds: 1 volumeMounts: - mountPath: /etc/kubernetes/ssl name: ssl-certs-kubernetes readOnly: true - mountPath: /etc/ssl/certs name: ssl-certs-host readOnly: true volumes: - hostPath: path: /etc/kubernetes/ssl name: ssl-certs-kubernetes - hostPath: path: /usr/share/ca-certificates name: ssl-certs-host - path: "/etc/kubernetes/manifests/kube-proxy.yaml" content: | apiVersion: v1 kind: Pod metadata: name: kube-proxy namespace: kube-system spec: hostNetwork: true containers: - name: kube-proxy image: quay.io/coreos/hyperkube:v1.3.6_coreos.0 command: - /hyperkube - proxy - --master=http://127.0.0.1:8080 - --proxy-mode=iptables securityContext: privileged: true volumeMounts: - mountPath: /etc/ssl/certs name: ssl-certs-host readOnly: true volumes: - hostPath: path: /usr/share/ca-certificates name: ssl-certs-host - path: "/etc/kubernetes/manifests/kube-apiserver.yaml" content: | apiVersion: v1 kind: Pod metadata: name: kube-apiserver namespace: kube-system spec: hostNetwork: true containers: - name: kube-apiserver image: quay.io/coreos/hyperkube:v1.3.6_coreos.0 command: - /hyperkube - apiserver - --bind-address=0.0.0.0 - --etcd-servers=https://10.79.218.2:2379,https://10.79.218.3:2379 - --allow-privileged=true - --service-cluster-ip-range=10.0.0.0/24 - --secure-port=443 - --advertise-address=10.79.218.2 - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem - --client-ca-file=/etc/kubernetes/ssl/ca.pem - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true ports: - containerPort: 443 hostPort: 443 name: https - containerPort: 8080 hostPort: 8080 name: local volumeMounts: - mountPath: /etc/kubernetes/ssl name: ssl-certs-kubernetes readOnly: true - mountPath: /etc/ssl/certs name: ssl-certs-host readOnly: true volumes: - hostPath: path: /etc/kubernetes/ssl name: ssl-certs-kubernetes - hostPath: path: /usr/share/ca-certificates name: ssl-certs-host - path: "/etc/flannel/options.env" content: | FLANNELD_IFACE=10.79.218.2 FLANNELD_ETCD_ENDPOINTS=https://10.79.218.2:2379,https://10.79.218.3:2379 - path: "/etc/kubernetes/ssl/ca.pem" permissions: "0600" owner: "root:root" content: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - path: "/etc/kubernetes/ssl/apiserver.pem" permissions: "0600" owner: "root:root" content: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - path: "/etc/kubernetes/ssl/apiserver-key.pem" permissions: "0600" owner: "root:root" content: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- ssh_authorized_keys: - "ssh-rsa ... ufk@ufk-osx-music" coreos: update: reboot-strategy: etcd-lock etcd2: discovery: https://discovery.etcd.io/... # multi-region and multi-cloud deployments need to use $public_ipv4 advertise-client-urls: https://10.79.218.2:2379 initial-advertise-peer-urls: https://10.79.218.2:2380 # listen on both the official ports and the legacy ports # legacy ports can be omitted if your application doesn't depend on them listen-client-urls: https://0.0.0.0:2379 listen-peer-urls: https://10.79.218.2:2380 units: - name: 00-enp4s0.network runtime: true content: | [Match] Name=enp4s0 [Network] Address=10.79.218.2/24 Gateway=10.79.218.232 DNS=8.8.8.8 - name: var-lib-rkt.mount enable: true command: start content: | [Mount] What=/dev/disk/by-uuid/23cc3284-e121-4448-b755-b74b5a022251 Where=/var/lib/rkt Type=btrfs Options=loop,discard - name: etcd2.service command: start drop-ins: - name: 30-certs.conf content: | [Service] Environment="ETCD_CERT_FILE=/etc/ssl/etcd/server1.pem" Environment="ETCD_KEY_FILE=/etc/ssl/etcd/server1-key.pem" Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem" Environment="ETCD_CLIENT_CERT_AUTH=true" Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/server1.pem" Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/server1-key.pem" Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem" Environment="ETCD_PEER_CLIENT_CERT_AUTH=true" - name: flanneld.service command: start drop-ins: - name: 50-network-config.conf content: | [Service] ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}' - name: 40-ExecStartPre-symlink.conf content: | [Service] ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env - name: kubelet.service enable: false content: | [Service] ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests Environment=KUBELET_VERSION=v1.3.6_coreos.0 ExecStart=/usr/lib/coreos/kubelet-wrapper \ --api-servers=http://127.0.0.1:8080 \ --network-plugin-dir=/etc/kubernetes/cni/net.d \ --network-plugin=cni \ --register-schedulable=false \ --allow-privileged=true \ --config=/etc/kubernetes/manifests \ --hostname-override=10.79.218.2 \ --cluster-dns=8.8.8.8 \ --cluster-domain=cluster.local Restart=always RestartSec=10 [Install] WantedBy=multi-user.target - name: calico-node.service enable: false content: | [Unit] Description=Calico per-host agent Requires=network-online.target After=network-online.target [Service] Slice=machine.slice Environment=CALICO_DISABLE_FILE_LOGGING=true Environment=HOSTNAME=10.79.218.2 Environment=IP=${ADVERTISE_IP} Environment=FELIX_FELIXHOSTNAME=10.79.218.2 Environment=CALICO_NETWORKING=false Environment=NO_DEFAULT_POOLS=true Environment=ETCD_ENDPOINTS=https://10.79.218.2:2379,https://10.79.218.3:2379 ExecStart=/usr/bin/rkt run --inherit-env --stage1-from-dir=stage1-fly.aci \ --volume=modules,kind=host,source=/lib/modules,readOnly=false \ --mount=volume=modules,target=/lib/modules \ --trust-keys-from-https quay.io/calico/node:v0.19.0 KillMode=mixed Restart=always TimeoutStartSec=0 [Install] WantedBy=multi-user.target
- 我刪除了重要的東西（如鑰匙）並用
...- 我已經配置了 kubelet 和 calico，但現在禁用了，直到我啟動 etcd2 並使用 tls 執行。
所以我試圖
etcdctl從我的 mac pro 筆記型電腦上執行以使用命令列出 etcd2 成員（同時連接到同一個本地網路）./etcdctl --endpoints="https://10.79.218.2:2379,https://10.79.218.3:2379" --key-file="./cfssl/client-key.pem" --cert-file="./cfssl/client.pem" --ca-file="./cfssl/ca.pem" member list我得到回應
Try '--no-sync' if you want to access non-published client endpoints(https://10.79.218.2:2379,https://10.79.218.3:2379). Error: client: no endpoints available執行相同的命令，
--no-sync結果如下：Failed to get leader: client: etcd cluster is unavailable or misconfigured我按照https://github.com/coreos/docs/blob/master/os/generate-self-signed-certificates.md中的描述創建了證書
我錯過了什麼？
謝謝
更新
所以我發現了https://github.com/coreos/etcd/tree/master/hack/tls-setup。我使用此腳本重新創建了證書，並使用這些腳本配置了我的 coreos etcd2 伺服器。
它創建了以下證書： ca-key.pem ca.csr ca.pem etcd1-key.pem etcd1.csr etcd1.pem etcd2-key.pem etcd2.csr etcd2.pem etcd3-key.pem etcd3.csr etcd3.pem proxy1-key.pem proxy1.csr proxy1.pem
問題是我沒有看到任何客戶端證書..所以我不太明白要傳遞哪些證書作為 etcdctl 在我的桌面上工作的參數。有任何想法嗎 ？
更新 2
所以我找到瞭如何將 etcdctl 與 tls 一起使用 .. 我想。但我仍然得到錯誤。我執行了以下命令：
/etcdctl --endpoint=https://10.79.218.2:2379 --ca-file=./certs/ca.pem --cert-file=./certs/etcd1.pem --key-file=./certs/etcd1-key.pem --no-sync ls但我得到了錯誤
Error: client: etcd cluster is unavailable or misconfigured error #0: x509: cannot validate certificate for 10.79.218.2 because it doesn't contain any IP SANs
我終於搞定了。我沒有正確閱讀 hack/tls-setup 的說明。我以為我需要將 req-csr.json 下載到 hack/tls-setup 目錄，而實際上我只需要在發出
make命令之前編輯 hack/tls-setup/config/req-csr.json 文件。這樣做之後，有時我會遇到超時錯誤，所以我只是etcdctl以 10 秒的超時時間執行。我需要執行的完整命令是
etcdctl --endpoint=https://10.79.218.2:2379 --ca-file=./certs/ca.pem --cert-file=./certs/etcd1.pem --key-file=./certs/etcd1-key.pem --timeout=10s member list