Nginx

nginx 將一些 POST 請求掛起一段時間

  • January 18, 2020

我在 Ubuntu 18.08 上執行以下堆棧並定義為 docker-compose:

  1. 的實例mariadb:10.3.20
  2. 基於wordpress:5.3.0-php7.2安裝ioncube的自定義wordpress實例
  3. 基於nginx:1.13安裝nginx-amplify-agent的自定義 nginx 實例

nginx的配置:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
   worker_connections  10000;
}

http {
   include       /etc/nginx/mime.types;
   default_type  application/octet-stream;

   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for"';

   log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for" '
                         '"$host" sn="$server_name" '
                         'rt=$request_time '
                         'ua="$upstream_addr" us="$upstream_status" '
                         'ut="$upstream_response_time" ul="$upstream_response_length" '
                         'cs=$upstream_cache_status' ;

   access_log  /var/log/nginx/access.log  main_ext;
   error_log  /var/log/nginx/error.log warn;

   sendfile        on;
   #tcp_nopush     on;

   keepalive_timeout  65;

   gzip  on;

   include /etc/nginx/conf.d/*.conf;
}

站點定義如下:

server {
       listen 80;
       listen [::]:80;

       server_name some.org www.some.org;

       location / {
               rewrite ^ https://$host$request_uri? permanent;
       }
}

server {
       listen 443 ssl http2;
       listen [::]:443 ssl http2;
       server_name some.org www.some.org;

       index index.php index.html index.htm;

       root /var/www/html;

       server_tokens off;

       ssl_certificate /etc/letsencrypt/live/some.org/fullchain.pem;
       ssl_certificate_key /etc/letsencrypt/live/some.org/privkey.pem;

       add_header X-Frame-Options "SAMEORIGIN" always;
       add_header X-XSS-Protection "1; mode=block" always;
       add_header X-Content-Type-Options "nosniff" always;
       add_header Referrer-Policy "no-referrer-when-downgrade" always;
       add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
       # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
       # enable strict transport security only if you understand the implications

       location / {
               proxy_connect_timeout 600;
               proxy_send_timeout    600;
               proxy_read_timeout    600;
               proxy_redirect        off;
               proxy_pass http://wordpress;

               proxy_set_header      X-Real-IP $remote_addr;
               proxy_set_header      X-Forwarded-Proto https;
               proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header      Host $host;
       }

       location = /favicon.ico {
               log_not_found off; access_log off;
       }

       location = /robots.txt {
               log_not_found off; access_log off; allow all;
       }

       location ~* \.(css|js|gif|ico|jpeg|jpg|png)$ {
               expires max;
               log_not_found off;
       }
}

整個堆棧按預期工作,除非有 10-15 個使用者來到網站並嘗試執行操作。在這種情況下,一些請求開始掛起(通常是對某些組件的相同 POST 請求),3-4 分鐘後它正在發布沒有任何錯誤,使用者實際上可以看到它的結果。在掛起網站變得不負責從同一個瀏覽器(但從其他一切都好!)。一旦請求發布 - 站點再次負責。

日誌也很奇怪:

  • 一旦掛起/掛起的請求到達伺服器,它就會顯示在 nginx 訪問日誌中,但不會顯示在 wordpress 日誌中
  • 一旦請求最終被釋放(即處理),它會在 nginx 訪問日誌和 wordpress 日誌中第二次顯示,但時間不同

nginx訪問日誌:

62.96.39.243 - - [17/Jan/2020:10:56:41 +0000] "GET /something

78.43.40.52 - - [17/Jan/2020:10:56:41 +0000] "POST /ajax-bidsform.html?meth=post&yid=d7f9f1a1a0bf HTTP/2.0" 200 208 "https://some.org/xchange_XRP_to_SBERRUB/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" "-"

78.43.40.52 - - [17/Jan/2020:10:56:41 +0000] "GET /something

WordPress:

134.19.130.91 - - [17/Jan/2020:10:56:40 +0000] "GET /something

78.43.40.52 - - [17/Jan/2020:10:50:07 +0000] "POST /ajax-bidsform.html?meth=post&yid=d7f9f1a1a0bf HTTP/1.0" 200 559 "https://some.org/xchange_XRP_to_SBERRUB/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"

62.96.39.243 - - [17/Jan/2020:10:56:41 +0000] "GET /something

如您所見,掛起POST /ajax-bidsform.html...在 nginx 日誌中的時間為 10:56,但在 wordpress 中為 10:50 - 這正是客戶端完成此請求的時間。據我了解,這意味著請求在 nginx 級別的某個地方停留了近 6 分鐘,直到它實際傳遞給 wordpress。如您所見,我沒有 nginx 的 ddos​​ 保護指令。

我這邊還有一些注意事項:在掛起請求期間,沒有任何 CPU 或 RAM 長期峰值,因此它可能與硬體問題無關。我還認為它與掛起腳本(即ajax-bidsform.html)有某種關係,但它只有在我們從虛擬主機遷移到數字海洋的雲實例時才開始發生(以前從未發生過)所以我猜這是一個配置問題。日誌中的請求時間表也可以證明這一點。

到目前為止,我確實嘗試過:

  1. 增加到worker_connections10000
  2. 將 nginx 實例的(不是主機的)net.core.somaxconn增加到 1024

但問題仍然存在。任何想法或想法將不勝感激!

nginx 日誌中的 10:56,但 wordpress 中的 10:50

我會將此解釋為 wordpress 在 10:50 接收請求並在 10:56 將結果返回給 nginx。要確定,您可以upstream_response_timelog_formatnginx 中添加。請參閱使用 NGINX 日誌記錄進行應用程序性能監控

引用自:https://serverfault.com/questions/999456