logstash mutate 從 URL 中刪除 API 密鑰

我將 nginx 日誌放入 logstash，不幸的是，api 資訊是通過 get 發送的。

因此，logstash 中有 2 個部分用於儲存 API creditianals。以下是範例

message: 10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] "PUT /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=2222222222222222222222222 HTTP/1.1" 200 689 "-" "python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64" "10.120.40.105" 0.180 0.180
request: /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=2222222222222222222222222

我正在通過

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time} %{NUMBER:upstream_time}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time}

我的輸入看起來像

   grok {
       match => { "message" => "%{NGINXACCESS}" }
       patterns_dir => ["/etc/logstash/patterns"]
   }
   date {
       match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
   }
   geoip {
       source => "clientip"
       target => "geoip"
       database => "/usr/share/GeoIP/GeoLiteCity.dat"
       add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
       add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
   }
   mutate {
       convert => [ "[geoip][coordinates]", "float"]
       convert => [ "request_time", "float"]
       convert => [ "upstream_time", "float"]
   }

是否有任何變異方法可以將 api_secret= 之後的任何內容替換為“xxxxxxxxxxxx”

謝謝！

這實際上比看起來要難一些，因為gsubfor 欄位mutate實際上並沒有做你想要的。它似乎沒有你想像的那麼聰明。

我不得不修改您正在使用的模式，以擷取request(pre_req和post_req分別) 之前和之後的所有內容，但這似乎是可能的。

不知道它將如何擴展性能，因為這裡有很多過濾，但它確實有效。

我用這個配置測試了它：

input {
 stdin {}
}

filter {
 grok {
   match => [
     "message" , "(?<pre_req>%{IPORHOST:clientip} (?<ident>[a-zA-Z\.\@\-\+_%]+) (?<auth>[a-zA-Z\.\@\-\+_%]+) \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} )%{URIPATHPARAM:request}(?<post_req> HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time} %{NUMBER:upstream_time})",
     "message" , "(?<pre_req>%{IPORHOST:clientip} (?<ident>[a-zA-Z\.\@\-\+_%]+) (?<auth>[a-zA-Z\.\@\-\+_%]+) \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} )%{URIPATHPARAM:request}(?<post_req> HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time})"
     ]
   break_on_match => true
 }
 grok {
   match => { "request" => "(?<request_path>[^?]*)?(?<request_params>.*)"
 }

 }
 mutate {
   gsub => [ "request_params" , "[?]", "" ]
 }
 kv {
   field_split => "&"
   source => "request_params"
   prefix => "request_params_"
 }
 mutate {
   replace => { "request" => "%{request_path}?api_key=%{request_params_api_key}&api_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX" }
   replace => { "message" => "%{pre_req}%{request}%{post_req}" }
   remove_field => [ "request_path", "request_params", "request_params_api_key", "request_params_api_secret", "pre_req", "post_req" ]
 }
}

output {
 stdout { codec => rubydebug }
}

它似乎已經完成了你想要的..

# /opt/logstash/bin/logstash -f config.conf
Logstash startup completed
10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] "PUT /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=2222222222222222222222222 HTTP/1.1" 200 689 "-" "python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64" "10.120.40.105" 0.180 0.180
{
         "message" => "10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] \"PUT /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX HTTP/1.1\" 200 689 \"-\" \"python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64\" \"10.120.40.105\" 0.180 0.180",
        "@version" => "1",
      "@timestamp" => "2015-07-29T19:21:14.678Z",
            "host" => "elk.example.com",
        "clientip" => "10.120.40.105",
           "ident" => "-",
            "auth" => "-",
       "timestamp" => "29/Jul/2015:16:41:09 +0000",
            "verb" => "PUT",
         "request" => "/v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
     "httpversion" => "1.1",
        "response" => "200",
           "bytes" => "689",
           "agent" => "\"python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64\"",
   "xforwardedfor" => "\"10.120.40.105\"",
    "request_time" => "0.180",
   "upstream_time" => "0.180"
}

引用自：https://serverfault.com/questions/709451