Coreos

帶有 etcd2-tls 的 kubelet 服務無法連接到 127.0.0.1:8080 - getsockopt：連接被拒絕

  • September 27, 2016

我安裝了 CoreOS stable v1122.2.0。

我已經用 tls 配置了 etcd2 並且工作正常。我基於https://github.com/coreos/etcd/tree/master/hack/tls-setup使用我為我的伺服器創建的子域而不是特定的 IP 地址創建了證書，以使 calico tls 工作。

etcd2 和 calcio-node 已配置並正常工作。現在我想配置 Kubernetes。我使用了https://coreos.com/kubernetes/docs/latest/deploy-master.html上的說明，現在我只配置了一個 coreos 伺服器。

當我啟動 kubelet 並執行時，journalctl -f -u kubelet我收到以下消息：

Sep 23 23:30:11 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:11.495381    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:11 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:11.889187    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.292061    1473 reflector.go:205] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.307222    1473 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: getsockopt: connection refused' (may retry after sleeping)
Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.495982    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.889756    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.292671    1473 reflector.go:205] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.496732    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.589335    1473 kubelet.go:1938] Failed creating a mirror pod for "kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)": Post http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.890294    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: I0923 23:30:13.979257    1473 docker_manager.go:2289] checking backoff for container "kube-apiserver" in pod "kube-apiserver-coreos-2.tux-in.com"
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: I0923 23:30:13.980071    1473 docker_manager.go:2303] Back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)
Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.980144    1473 pod_workers.go:183] Error syncing pod 9b41319800532574b4c4ac760c920bee, skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)"

這是我的/var/lib/coreos-install/user_data文件：

#cloud-config

hostname: "coreos-2.tux-in.com"
write_files:
 - path: "/etc/ssl/etcd/ca.pem"
   permissions: "0666"
   owner: "etcd:etcd"
   content: |
    ...
 - path: "/etc/ssl/etcd/etcd1.pem"
   permissions: "0666"
   owner: "etcd:etcd"
   content: |
    ...
 - path: "/etc/ssl/etcd/etcd1-key.pem"
   permissions: "0666"
   owner: "etcd:etcd"
   content: |
    ...
 - path: "/etc/kubernetes/ssl/ca.pem"
   permissions: "0600"
   owner: "root:root"
   content: |
    ...
 - path: "/etc/kubernetes/ssl/apiserver.pem"
   permissions: "0600"
   owner: "root:root"
   content: |
    ...
 - path: "/etc/kubernetes/ssl/apiserver-key.pem"
   permissions: "0600"
   owner: "root:root"
   content: |
    ...
 - path: "/etc/kubernetes/cni/net.d/10-calico.conf"
   content: |
     {
         "name": "calico",
         "type": "flannel",
         "delegate": {
             "type": "calico",
             "etcd_endpoints": "https://coreos-2.tux-in.com:2379",
             "log_level": "none",
             "log_level_stderr": "info",
             "hostname": "coreos-2.tux-in.com",
             "policy": {
                 "type": "k8s",
                 "k8s_api_root": "http://127.0.0.1:8080/api/v1/"
             }
         }
     }
 - path: "/etc/kubernetes/manifests/policy-controller.yaml"
   content: |
    apiVersion: v1
     kind: Pod
     metadata:
       name: calico-policy-controller
       namespace: calico-system
     spec:
       hostNetwork: true
       containers:
         # The Calico policy controller.
         - name: k8s-policy-controller
           image: calico/kube-policy-controller:v0.2.0
           env:
             - name: ETCD_ENDPOINTS
               value: "https://coreos-2.tux-in.com:2379"
             - name: K8S_API
               value: "http://127.0.0.1:8080"
             - name: LEADER_ELECTION
               value: "true"
         # Leader election container used by the policy controller.
         - name: leader-elector
           image: quay.io/calico/leader-elector:v0.1.0
           imagePullPolicy: IfNotPresent
           args:
             - "--election=calico-policy-election"
             - "--election-namespace=calico-system"
             - "--http=127.0.0.1:4040"
 - path: "/etc/kubernetes/manifests/kube-scheduler.yaml"
   content: |
     apiVersion: v1
     kind: Pod
     metadata:
       name: kube-scheduler
       namespace: kube-system
     spec:
       hostNetwork: true
       containers:
       - name: kube-scheduler
         image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
         command:
         - /hyperkube
         - scheduler
         - --master=http://127.0.0.1:8080
         - --leader-elect=true
         livenessProbe:
           httpGet:
             host: 127.0.0.1
             path: /healthz
             port: 10251
           initialDelaySeconds: 15
           timeoutSeconds: 1
 - path: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
   content: |
     apiVersion: v1
     kind: Pod
     metadata:
       name: kube-controller-manager
       namespace: kube-system
     spec:
       hostNetwork: true
       containers:
       - name: kube-controller-manager
         image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
         command:
         - /hyperkube
         - controller-manager
         - --master=http://127.0.0.1:8080
         - --leader-elect=true
         - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
         - --root-ca-file=/etc/kubernetes/ssl/ca.pem
         livenessProbe:
           httpGet:
             host: 127.0.0.1
             path: /healthz
             port: 10252
           initialDelaySeconds: 15
           timeoutSeconds: 1
         volumeMounts:
         - mountPath: /etc/kubernetes/ssl
           name: ssl-certs-kubernetes
           readOnly: true
         - mountPath: /etc/ssl/certs
           name: ssl-certs-host
           readOnly: true
       volumes:
       - hostPath:
           path: /etc/kubernetes/ssl
         name: ssl-certs-kubernetes
       - hostPath:
           path: /usr/share/ca-certificates
         name: ssl-certs-host
 - path: "/etc/kubernetes/manifests/kube-proxy.yaml"
   content: |
     apiVersion: v1
     kind: Pod
     metadata:
       name: kube-proxy
       namespace: kube-system
     spec:
       hostNetwork: true
       containers:
       - name: kube-proxy
         image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
         command:
         - /hyperkube
         - proxy
         - --master=http://127.0.0.1:8080
         - --proxy-mode=iptables
         securityContext:
           privileged: true
         volumeMounts:
         - mountPath: /etc/ssl/certs
           name: ssl-certs-host
           readOnly: true
       volumes:
       - hostPath:
           path: /usr/share/ca-certificates
         name: ssl-certs-host
 - path: "/etc/kubernetes/manifests/kube-apiserver.yaml"
   content: |
     apiVersion: v1
     kind: Pod
     metadata:
       name: kube-apiserver
       namespace: kube-system
     spec:
       hostNetwork: true
       containers:
       - name: kube-apiserver
         image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
         command:
         - /hyperkube
         - apiserver
         - --bind-address=0.0.0.0
         - --etcd-servers=https://coreos-2.tux-in.com:2379
         - --allow-privileged=true
         - --service-cluster-ip-range=10.0.0.0/24
         - --secure-port=443
         - --advertise-address=coreos-2.tux-in.com
         - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
         - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
         - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
         - --client-ca-file=/etc/kubernetes/ssl/ca.pem
         - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
         - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true
         ports:
         - containerPort: 443
           hostPort: 443
           name: https
         - containerPort: 8080
           hostPort: 8080
           name: local
         volumeMounts:
         - mountPath: /etc/kubernetes/ssl
           name: ssl-certs-kubernetes
           readOnly: true
         - mountPath: /etc/ssl/certs
           name: ssl-certs-host
           readOnly: true
       volumes:
       - hostPath:
           path: /etc/kubernetes/ssl
         name: ssl-certs-kubernetes
       - hostPath:
           path: /usr/share/ca-certificates
         name: ssl-certs-host
ssh_authorized_keys:
         - ...
coreos:
  etcd2:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
    # specify the initial size of your cluster with ?size=X
    discovery: ...
    advertise-client-urls: https://coreos-2.tux-in.com:2379,https://coreos-2.tux-in.com:4001
    initial-advertise-peer-urls: https://coreos-2.tux-in.com:2380
    # listen on both the official ports and the legacy ports
    # legacy ports can be omitted if your application doesn't depend on them
    listen-client-urls: https://0.0.0.0:2379,https://0.0.0.0:4001
    listen-peer-urls: https://coreos-2.tux-in.com:2380
  flannel:
    etcd_endpoints: "https://coreos-2.tux-in.com:2379"
    etcd_cafile: /etc/ssl/etcd/ca.pem
    etcd_certfile: /etc/ssl/etcd/etcd1.pem
    etcd_keyfile: /etc/ssl/etcd/etcd1-key.pem
  update:
    reboot-strategy: etcd-lock
  units:
    - name: 00-enp4s0.network
      runtime: true
      content: |
       [Match]
       Name=enp4s0

       [Network]
       Address=10.79.218.2/24
       Gateway=10.79.218.232
       DNS=8.8.8.8
    - name: var-lib-rkt.mount
      enable: true
      command: start
      content: |
        [Mount]
        What=/dev/disk/by-uuid/daca9515-5040-4f1d-ac0b-b69de3b91343
        Where=/var/lib/rkt
        Type=btrfs
        Options=loop,discard
    - name: etcd2.service
      command: start
      drop-ins:
       - name: 30-certs.conf
         content: |
          [Service]
          Environment="ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem"
          Environment="ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem"
          Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
          Environment="ETCD_CLIENT_CERT_AUTH=true"
          Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/etcd1.pem"
          Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem"
          Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
          Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: flanneld.service
      command: start
      drop-ins:
       - name: 50-network-config.conf
         content: |
          [Service]
          ExecStartPre=/usr/bin/etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd1.pem --key-file=/etc/ssl/etcd/etcd1-key.pem --endpoint=https://coreos-2.tux-in.com:2379 set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
    - name: calico-node.service
      command: start
      content: |
       [Unit]
       Description=Calico per-host agent
       Requires=network-online.target
       After=network-online.target

       [Service]
       Slice=machine.slice
       Environment=CALICO_DISABLE_FILE_LOGGING=true
       Environment=HOSTNAME=coreos-2.tux-in.com
       Environment=IP=10.79.218.2
       Environment=FELIX_FELIXHOSTNAME=coreos-2.tux-in.com
       Environment=CALICO_NETWORKING=false
       Environment=NO_DEFAULT_POOLS=true
       Environment=ETCD_ENDPOINTS=https://coreos-2.tux-in.com:2379
       Environment=ETCD_AUTHORITY=coreos-2.tux-in.com:2379
       Environment=ETCD_SCHEME=https
       Environment=ETCD_CA_CERT_FILE=/etc/ssl/etcd/ca.pem
       Environment=ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem
       Environment=ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem
       ExecStart=/usr/bin/rkt run --volume=resolv-conf,kind=host,source=/etc/resolv.conf,readOnly=true \
       --volume=etcd-tls-certs,kind=host,source=/etc/ssl/etcd,readOnly=true --inherit-env --stage1-from-dir=stage1-fly.aci \
       --volume=modules,kind=host,source=/lib/modules,readOnly=false \
       --mount=volume=modules,target=/lib/modules \
       --trust-keys-from-https quay.io/calico/node:v0.19.0 \
       --mount=volume=etcd-tls-certs,target=/etc/ssl/etcd \
       --mount=volume=resolv-conf,target=/etc/resolv.conf

       KillMode=mixed
       Restart=always
       TimeoutStartSec=0

       [Install]
       WantedBy=multi-user.target
    - name: kubelet.service
      command: start
      content: |
       [Service]
       ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
       ExecStartPre=/usr/bin/mkdir -p /var/log/containers

       Environment=KUBELET_VERSION=v1.3.7_coreos.0
       Environment="RKT_OPTS=--volume var-log,kind=host,source=/var/log \
         --mount volume=var-log,target=/var/log \
         --volume dns,kind=host,source=/etc/resolv.conf \
         --mount volume=dns,target=/etc/resolv.conf"

       ExecStart=/usr/lib/coreos/kubelet-wrapper \
         --api-servers=http://127.0.0.1:8080 \
         --network-plugin-dir=/etc/kubernetes/cni/net.d \
         --network-plugin=cni \
         --register-schedulable=false \
         --allow-privileged=true \
         --config=/etc/kubernetes/manifests \
         --hostname-override=coreos-2.tux-in.com \
         --cluster-dns=8.8.8.8 \
         --cluster-domain=tux-in.com
       Restart=always
       RestartSec=10
       [Install]
       WantedBy=multi-user.target

127.0.0.1:8080應該由 kubelet-apiserver 打開嗎？我在這裡想念什麼？

謝謝！

在許多情況下，API 伺服器是由 Kubelet 啟動的，導致在 API 端點可用之前出現初始連接錯誤。如果此錯誤在一段時間後仍然存在，您可能希望查看您的 API 伺服器是否正在啟動以及是否啟動。

Kubelet 將自動啟動/etc/kubernetes/manifests您的kube-apiserver.yaml所在位置的服務。

如果您的 API Server 未啟動，您將需要：

1：檢查 Kubelet 命令行選項以確保使用這些--config=/etc/kubernetes/manifests選項啟用清單。這可以檢查 ps aux | grep kubelet

2：檢查 Kubelet 和 API 容器日誌以查看 API 啟動期間發生的問題。這通常是證書不匹配、依賴失敗、etcd 服務未偵聽等。

Kubelet 服務日誌：

  $ journalctl -fu kubelet.service

這個例子我通過 ‘docker logs’ 從 API 伺服器收集日誌並顯示我的 Kubelet 啟動 API 伺服器。在伺服器偵聽和最終啟動之前註意類似的連接問題。

  $ docker ps -l
   543022a70bc6        gcr.io/google_containers/hyperkube:v1.3.7   "/hyperkube apiserver"   3 seconds ago        Exited (1) 3 seconds ago

  $ docker logs 543022a70bc6
   I0920 00:26:33.903861       1 genericapiserver.go:606] Will report 10.0.104.100 as public IP address.
   E0920 00:26:33.937478       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/exists/admission.go:86: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
   E0920 00:26:33.937651       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go:116: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
   E0920 00:26:33.937821       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: Get http://0.0.0.0:8080/api/v1/limitranges?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
   E0920 00:26:33.939508       1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://0.0.0.0:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
   E0920 00:26:33.939741       1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: Get http://0.0.0.0:8080/api/v1/serviceaccounts?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
   E0920 00:26:33.947780       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/resourcequota/controller.go:121: Failed to list *api.ResourceQuota: Get http://0.0.0.0:8080/api/v1/resourcequotas?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
   [restful] 2016/09/20 00:26:34 log.go:30: [restful/swagger] listing is available at https://10.0.104.100:6443/swaggerapi/
   [restful] 2016/09/20 00:26:34 log.go:30: [restful/swagger] https://10.0.104.100:6443/swaggerui/ is mapped to folder /swagger-ui/
   I0920 00:26:34.235914       1 genericapiserver.go:690] Serving securely on 0.0.0.0:6443
   I0920 00:26:34.235941       1 genericapiserver.go:734] Serving insecurely on 0.0.0.0:8080

引用自：https://serverfault.com/questions/805053

相關問答

Domain-Name-System

Kubernetes 子域上的訪問服務

  • February 26, 2021
檢視問答
Coreos

在 CoreOS 上增加熵

  • May 30, 2016
檢視問答
Amazon-Ec2

在 AWS 上部署 Kubernetes 集群

  • January 9, 2016
檢視問答
Coreos

Kubernetes 服務 etcd 參考

  • November 21, 2015
檢視問答
Coreos

無法通過IP連接kubernetes的服務

  • March 27, 2015
檢視問答
Kubernetes

當一個 initContainer 被 OOMKilled 時，Pod 卡在 PodInitializing 狀態

  • August 30, 2022
檢視問答