Ssl

無法從 k8s pod 發出 https 請求

  • September 24, 2020

我在新的 Ubuntu Server 20.04 上使用 Rancher 部署了一個新的 kubernetes 集群(版本 1.18)。我面臨的問題是從任何 pod 發出的每個 https 請求都失敗並顯示以下消息:

curl -v https://pypi.org/simple/pip/
*   Trying 44.227.65.245...
* Connected to pypi.org (44.227.65.245) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Handshake failed
* Closing connection 0
curl: (35) gnutls_handshake() failed: Handshake failed

我已經使用它進行了測試

kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4
kubectl exec -it <pod_id> bash

我在主機系統上通過 ssh 嘗試了同樣的事情,一切正常。我還嘗試使用與 pod ( ) 相同的映像執行獨立的 docker 容器(在同一台機器上)docker run -it --rm k8s.gcr.io/echoserver:1.4 bash,這裡也沒有問題。所以我猜這與我的 kubernetes 設置有關。

它只發生在 https 請求中 - http 請求正常工作。

我試圖刪除集群並從頭開始準備,但這沒有幫助。

更新:

我收到兩種類型的消息,具體取決於我嘗試連接的伺服器:

root@hello-node-7bf657c596-smr2h:/# openssl s_client -connect google.com:443 -debug
CONNECTED(00000003)
write to 0xda15b0 [0xda1630] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 4e c0 cf d9 62   ....,...(..N...b
0010 - 58 6e 88 7b 28 7f e8 c1-62 c6 8f 23 60 86 b0 53   Xn.{(...b..#`..S
0020 - df 43 cd a7 58 52 bc 59-6a ae bf 00 00 aa c0 30   .C..XR.Yj......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1   .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37   ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a   .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0   .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31   ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43   .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c   .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02   .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d   ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04   ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c   ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c   ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01   .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03   ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0130 - 01                                                .
read from 0xda15b0 [0xda6b90] (7 bytes => 7 (0x7))
0000 - 15 00 00 00 02 02 28                              ......(
140080050284184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
   Protocol  : TLSv1.2
   Cipher    : 0000
   Session-ID: 
   Session-ID-ctx: 
   Master-Key: 
   Key-Arg   : None
   PSK identity: None
   PSK identity hint: None
   SRP username: None
   Start Time: 1600154660
   Timeout   : 300 (sec)
   Verify return code: 0 (ok)
---

root@hello-node-7bf657c596-smr2h:/# openssl s_client -connect pypi.org:443 -debug
CONNECTED(00000003)
write to 0x24645b0 [0x2464630] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 0c 2b bf dc 71   ....,...(...+..q
0010 - d2 41 56 f3 40 e4 c5 3f-62 88 cf 2a a7 76 e4 a4   .AV.@..?b..*.v..
0020 - 08 96 13 89 30 13 fa 75-c9 2d 32 00 00 aa c0 30   ....0..u.-2....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1   .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37   ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a   .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0   .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31   ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43   .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c   .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02   .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d   ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04   ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c   ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c   ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01   .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03   ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0130 - 01                                                .
read from 0x24645b0 [0x2469b90] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 28                              ......(
140284912154264:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
   Protocol  : TLSv1.2
   Cipher    : 0000
   Session-ID: 
   Session-ID-ctx: 
   Master-Key: 
   Key-Arg   : None
   PSK identity: None
   PSK identity hint: None
   SRP username: None
   Start Time: 1600154752
   Timeout   : 300 (sec)
   Verify return code: 0 (ok)
---

我發現這兩條消息一起出現在這裡描述:https ://blog.techstacks.com/2010/03/3-common-causes-of-unknown-ssl-protocol-errors-with-curl.html

目標站點不喜歡該密碼 您可能正在嘗試使用站點配置為拒絕的 ssl 密碼連接到該站點。例如,匿名密碼通常在面向客戶的 ssl 加密站點上被禁用。(我們中的許多人在任何 SSL 加密的網站上都設置了全面拒絕策略——不管它的目的是什麼。)以下命令字元串“can”也會導致 curl (35) 錯誤:

curl –ciphers ADH-RC4-MD5 https://some_web_site.some_domain.com/ 不幸的是,您可以從 curl 獲得的錯誤響應類型很大程度上取決於 ssl 伺服器。在某些站點上,您會收到 Unknown SSL Protocol 錯誤,但在我的 techstacks-tools 站點上,我得到:

curl:(35)錯誤:14077410:SSL常式:SSL23_GET_SERVER_HELLO:sslv3警報握手失敗

我發現的另一件事是我在 DigitalOcean 液滴上準備了相同的環境,那裡沒有問題。

我的裸機環境在世界範圍內不可用,它位於內部網路中 - 也許這有某種關聯?

我也嘗試使用 microk8s 設置 k8s,但問題仍然存在,所以這可能與 rancher 無關,而僅與我的機器設置有關。

pod 的 YAML:

apiVersion: v1
kind: Pod
metadata:
 annotations:
   cni.projectcalico.org/podIP: 10.42.0.8/32
   cni.projectcalico.org/podIPs: 10.42.0.8/32
 creationTimestamp: "2020-09-14T19:51:57Z"
 generateName: hello-node-7bf657c596-
 labels:
   app: hello-node
   pod-template-hash: 7bf657c596
 managedFields:
 - apiVersion: v1
   fieldsType: FieldsV1
   fieldsV1:
     f:metadata:
       f:generateName: {}
       f:labels:
         .: {}
         f:app: {}
         f:pod-template-hash: {}
       f:ownerReferences:
         .: {}
         k:{"uid":"dda770d0-3aa3-4fc9-97e1-c929fc3629e9"}:
           .: {}
           f:apiVersion: {}
           f:blockOwnerDeletion: {}
           f:controller: {}
           f:kind: {}
           f:name: {}
           f:uid: {}
     f:spec:
       f:containers:
         k:{"name":"echoserver"}:
           .: {}
           f:image: {}
           f:imagePullPolicy: {}
           f:name: {}
           f:resources: {}
           f:terminationMessagePath: {}
           f:terminationMessagePolicy: {}
       f:dnsPolicy: {}
       f:enableServiceLinks: {}
       f:restartPolicy: {}
       f:schedulerName: {}
       f:securityContext: {}
       f:terminationGracePeriodSeconds: {}
   manager: kube-controller-manager
   operation: Update
   time: "2020-09-14T19:51:57Z"
 - apiVersion: v1
   fieldsType: FieldsV1
   fieldsV1:
     f:status:
       f:conditions:
         .: {}
         k:{"type":"PodScheduled"}:
           .: {}
           f:lastProbeTime: {}
           f:lastTransitionTime: {}
           f:message: {}
           f:reason: {}
           f:status: {}
           f:type: {}
   manager: kube-scheduler
   operation: Update
   time: "2020-09-14T19:51:57Z"
 - apiVersion: v1
   fieldsType: FieldsV1
   fieldsV1:
     f:metadata:
       f:annotations:
         .: {}
         f:cni.projectcalico.org/podIP: {}
         f:cni.projectcalico.org/podIPs: {}
   manager: calico
   operation: Update
   time: "2020-09-14T19:52:01Z"
 - apiVersion: v1
   fieldsType: FieldsV1
   fieldsV1:
     f:status:
       f:conditions:
         k:{"type":"ContainersReady"}:
           .: {}
           f:lastProbeTime: {}
           f:lastTransitionTime: {}
           f:status: {}
           f:type: {}
         k:{"type":"Initialized"}:
           .: {}
           f:lastProbeTime: {}
           f:lastTransitionTime: {}
           f:status: {}
           f:type: {}
         k:{"type":"Ready"}:
           .: {}
           f:lastProbeTime: {}
           f:lastTransitionTime: {}
           f:status: {}
           f:type: {}
       f:containerStatuses: {}
       f:hostIP: {}
       f:phase: {}
       f:podIP: {}
       f:podIPs:
         .: {}
         k:{"ip":"10.42.0.8"}:
           .: {}
           f:ip: {}
       f:startTime: {}
   manager: kubelet
   operation: Update
   time: "2020-09-14T19:52:01Z"
 name: hello-node-7bf657c596-smr2h
 namespace: default
 ownerReferences:
 - apiVersion: apps/v1
   blockOwnerDeletion: true
   controller: true
   kind: ReplicaSet
   name: hello-node-7bf657c596
   uid: dda770d0-3aa3-4fc9-97e1-c929fc3629e9
 resourceVersion: "3468"
 selfLink: /api/v1/namespaces/default/pods/hello-node-7bf657c596-smr2h
 uid: cf37b215-8269-42ce-9e70-750f9f862cac
spec:
 containers:
 - image: k8s.gcr.io/echoserver:1.4
   imagePullPolicy: IfNotPresent
   name: echoserver
   resources: {}
   terminationMessagePath: /dev/termination-log
   terminationMessagePolicy: File
   volumeMounts:
   - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
     name: default-token-9jlms
     readOnly: true
 dnsPolicy: ClusterFirst
 enableServiceLinks: true
 nodeName: gepard
 priority: 0
 restartPolicy: Always
 schedulerName: default-scheduler
 securityContext: {}
 serviceAccount: default
 serviceAccountName: default
 terminationGracePeriodSeconds: 30
 tolerations:
 - effect: NoExecute
   key: node.kubernetes.io/not-ready
   operator: Exists
   tolerationSeconds: 300
 - effect: NoExecute
   key: node.kubernetes.io/unreachable
   operator: Exists
   tolerationSeconds: 300
 volumes:
 - name: default-token-9jlms
   secret:
     defaultMode: 420
     secretName: default-token-9jlms
status:
 conditions:
 - lastProbeTime: null
   lastTransitionTime: "2020-09-14T19:52:00Z"
   status: "True"
   type: Initialized
 - lastProbeTime: null
   lastTransitionTime: "2020-09-14T19:52:01Z"
   status: "True"
   type: Ready
 - lastProbeTime: null
   lastTransitionTime: "2020-09-14T19:52:01Z"
   status: "True"
   type: ContainersReady
 - lastProbeTime: null
   lastTransitionTime: "2020-09-14T19:52:00Z"
   status: "True"
   type: PodScheduled
 containerStatuses:
 - containerID: docker://2d2ddbf42fc4e6634a782c7036cf4a9a1d9f50a3d847bb5444932c701cd8186e
   image: k8s.gcr.io/echoserver:1.4
   imageID: docker-pullable://k8s.gcr.io/echoserver@sha256:5d99aa1120524c801bc8c1a7077e8f5ec122ba16b6dda1a5d3826057f67b9bcb
   lastState: {}
   name: echoserver
   ready: true
   restartCount: 0
   started: true
   state:
     running:
       startedAt: "2020-09-14T19:52:01Z"
 hostIP: 10.0.0.3
 phase: Running
 podIP: 10.42.0.8
 podIPs:
 - ip: 10.42.0.8
 qosClass: BestEffort
 startTime: "2020-09-14T19:52:00Z"

防火牆暫時被禁用。

OP選擇不發布答案,但確實說他們解決了問題:

問題是/etc/resolve.conf在主機的搜尋部分添加了一些奇怪的域。當我嘗試發出 https 請求時,我已被 DNS 重定向到其他以純 http 響應的伺服器。

更正 DNS 設置解決了問題。

引用自:https://serverfault.com/questions/1033455