メモ > サーバ > 各論: トラブル対応例 > サーバが重い・サーバに繋がらない 20
■サーバが重い・サーバに繋がらない 20
AWSのCloudWatchから、ALBについて「ALB UnHealthy Host Count」の警告が来た
サイトも「504 Gateway Time-out」の表示になった
# ps aux | grep httpd
で確認すると、Apacheのプロセス数が300くらいになっていた。普段は10〜50程度
# systemctl restart php-fpm
# systemctl restart httpd
でApacheを再起動しても変わらず
ネットワークの障害が関係しているのかと思って同じVPC内にEC2を立ててApacheをインストールすると、問題なくアクセスできた
状況としては
・SSHでは問題なく接続できる
・ロードアベレージとCPU使用状況は普段どおり
・メモリは普段より使われているものの、稼働に問題があるとは思えない
・ディスク容量とinodeも普段どおり
・ネットワークの使用状況も普段どおり
・接続中のユーザ数は跳ね上がっている
・プロセスはApacheが大量に立ち上がっている
・HTTPDの接続数は跳ね上がっている
・Apacheのメモリ確保量は跳ね上がっている
・HTTPステータスは、「200 OK」が普段より非常に多いタイミングがある
しばらく原因が判らなかったが、Apacheのアクセスログを確認すると、負荷が上がった時間帯から以下のようなログが大量に現れていた
/var/log/httpd/access_log
148 Safari/604.1" 327 49.98.155.114 https any
::1 - - [24/Dec/2020:16:10:27 +0900] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.39 () (internal dummy connection)" 170 - - any
::1 - - [24/Dec/2020:16:10:28 +0900] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.39 () (internal dummy connection)" 168 - - any
::1 - - [24/Dec/2020:16:10:29 +0900] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.39 () (internal dummy connection)" 183 - - any
::1 - - [24/Dec/2020:16:10:30 +0900] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.39 () (internal dummy connection)" 167 - - any
::1 - - [24/Dec/2020:16:10:31 +0900] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.39 () (internal dummy connection)" 167 - - any
::1 - - [24/Dec/2020:16:10:32 +0900] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.39 () (internal dummy connection)" 163 - - any
216.144.247.78 - - [24/Dec/2020:16:10:34 +0900] "CONNECT www.pawmotorsport.com.au:443 HTTP/1.1" 200 21176 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 1.0" 32694 - - any
136.175.9.211 - - [24/Dec/2020:16:10:34 +0900] "CONNECT www.pawmotorsport.com.au:443 HTTP/1.1" 200 21176 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36" 38492 - - any
136.175.9.57 - - [24/Dec/2020:16:10:47 +0900] "CONNECT www.pawmotorsport.com.au:443 HTTP/1.1" 200 21176 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36" 34702 - - any
136.175.9.209 - - [24/Dec/2020:18:58:32 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "https://www.facebook.com" "Mozilla/5.0 (Linux; Android 4.4.2; GT-N5110 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.84 Safari/537.36" 238096628 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "https://www.yahoo.com" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0" 238164924 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "" "Mozilla/5.0 (iPad; CPU OS 8_0 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12A365 Safari/600.1.4" 238740621 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" 243895845 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:40.0) Gecko/20100101 Firefox/40.0" 246130485 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "https://www.gmail.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:34.0) Gecko/20100101 Firefox/34.0" 246415880 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "" "Mozilla/5.0 (Windows NT 6.0; rv:38.0) Gecko/20100101 Firefox/38.0" 246967450 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "https://www.yahoo.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" 247372227 - - any
136.175.9.209 - - [24/Dec/2020:18:58:33 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36" 247609739 - - any
136.175.9.209 - - [24/Dec/2020:18:58:34 +0900] "CONNECT www.acenursing.org:443 HTTP/1.1" 504 247 "https://www.google.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" 247921879 - - any
また、Apacheのエラーログを確認すると、負荷が挙がった時間帯から以下のようなログが現れていた
/var/log/httpd/error_log
[Thu Dec 24 16:09:57.979458 2020] [proxy_fcgi:error] [pid 31611] (70007)The timeout specified has expired: [client 216.144.247.78:39728] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:10:00.948288 2020] [proxy_fcgi:error] [pid 32350] (70007)The timeout specified has expired: [client 136.175.9.211:41736] AH01075: Error dispatching request to : (polling), referer: https://www.baidu.com
[Thu Dec 24 16:10:09.933879 2020] [proxy_fcgi:error] [pid 32455] (70007)The timeout specified has expired: [client 136.175.9.213:35128] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:10:18.089186 2020] [proxy_fcgi:error] [pid 32464] (70007)The timeout specified has expired: [client 64.31.35.10:40948] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:10:21.053775 2020] [proxy_fcgi:error] [pid 31686] (70007)The timeout specified has expired: [client 136.175.9.211:36960] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:10:58.806091 2020] [proxy_fcgi:error] [pid 32229] (70007)The timeout specified has expired: [client 216.144.247.78:46352] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:10:59.053005 2020] [proxy_fcgi:error] [pid 31618] (70007)The timeout specified has expired: [client 216.144.247.78:47300] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:11:01.567021 2020] [proxy_fcgi:error] [pid 32482] (70007)The timeout specified has expired: [client 216.144.247.78:55810] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:11:06.398091 2020] [proxy_fcgi:error] [pid 31860] (70007)The timeout specified has expired: [client 216.144.247.78:45130] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 16:11:06.830902 2020] [proxy_fcgi:error] [pid 32247] (70007)The timeout specified has expired: [client 216.144.247.78:46422] AH01075: Error dispatching request to : (polling)
[Thu Dec 24 19:02:11.239968 2020] [proxy_fcgi:error] [pid 9096] (70007)The timeout specified has expired: [client 136.175.9.209:40102] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:11.239973 2020] [proxy_fcgi:error] [pid 9143] (70007)The timeout specified has expired: [client 136.175.9.209:41870] AH01075: Error dispatching request to :443: (polling), referer: https://www.baidu.com
[Thu Dec 24 19:02:11.245293 2020] [proxy_fcgi:error] [pid 9149] (70007)The timeout specified has expired: [client 136.175.9.209:42220] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:12.243959 2020] [proxy_fcgi:error] [pid 9184] (70007)The timeout specified has expired: [client 136.175.9.209:42412] AH01075: Error dispatching request to :443: (polling), referer: https://www.google.com
[Thu Dec 24 19:02:14.415915 2020] [proxy_fcgi:error] [pid 9190] (70007)The timeout specified has expired: [client 136.175.9.209:43746] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:15.306974 2020] [proxy_fcgi:error] [pid 9196] (70007)The timeout specified has expired: [client 136.175.9.209:45576] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:23.567014 2020] [proxy_fcgi:error] [pid 9198] (70007)The timeout specified has expired: [client 136.175.9.209:47104] AH01075: Error dispatching request to :443: (polling), referer: https://www.baidu.com
[Thu Dec 24 19:02:27.618868 2020] [proxy_fcgi:error] [pid 9215] (70007)The timeout specified has expired: [client 136.175.9.209:49368] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:27.619120 2020] [proxy_fcgi:error] [pid 9208] (70007)The timeout specified has expired: [client 136.175.9.209:47952] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:29.017232 2020] [proxy_fcgi:error] [pid 9221] (70007)The timeout specified has expired: [client 136.175.9.209:52202] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:31.002278 2020] [proxy_fcgi:error] [pid 9210] (70007)The timeout specified has expired: [client 136.175.9.209:54108] AH01075: Error dispatching request to :443: (polling), referer: https://www.facebook.com
[Thu Dec 24 19:02:32.073475 2020] [proxy_fcgi:error] [pid 9233] (70007)The timeout specified has expired: [client 136.175.9.209:44848] AH01075: Error dispatching request to :443: (polling), referer: https://www.yahoo.com
[Thu Dec 24 19:02:32.650459 2020] [proxy_fcgi:error] [pid 9240] (70007)The timeout specified has expired: [client 136.175.9.209:46818] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:37.806771 2020] [proxy_fcgi:error] [pid 9242] (70007)The timeout specified has expired: [client 136.175.9.209:46950] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:40.042510 2020] [proxy_fcgi:error] [pid 9252] (70007)The timeout specified has expired: [client 136.175.9.209:47766] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:40.329017 2020] [proxy_fcgi:error] [pid 9232] (70007)The timeout specified has expired: [client 136.175.9.209:47984] AH01075: Error dispatching request to :443: (polling), referer: https://www.gmail.com
[Thu Dec 24 19:02:40.880690 2020] [proxy_fcgi:error] [pid 9235] (70007)The timeout specified has expired: [client 136.175.9.209:49442] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:41.286272 2020] [proxy_fcgi:error] [pid 9255] (70007)The timeout specified has expired: [client 136.175.9.209:49864] AH01075: Error dispatching request to :443: (polling), referer: https://www.yahoo.com
[Thu Dec 24 19:02:41.567903 2020] [proxy_fcgi:error] [pid 9247] (70007)The timeout specified has expired: [client 136.175.9.209:50212] AH01075: Error dispatching request to :443: (polling)
[Thu Dec 24 19:02:42.837072 2020] [proxy_fcgi:error] [pid 9280] (70007)The timeout specified has expired: [client 136.175.9.209:51344] AH01075: Error dispatching request to :443: (polling), referer: https://www.google.com
136.175.9.209
の接続元を調べるとアメリカだった
攻撃と思われるので、ネットワークACLでこのIPに対して「すべてのトラフィック」を遮断した
復旧したが、またすぐに重くなった
どこかで「このサイトを攻撃しよう」のようにして晒されている可能性があるかもしれない
いったん、目についたいくつかのIPアドレスをアクセス制限対象にし、EC2を3台とも再起動すると復旧した
それだけだとイタチごっこになる可能性があるので、Webサーバへの直接アクセス自体を遮断した
つまり
・ロードバランサー経由でWebサーバへは、これまでどおりアクセスできる
・IPアドレスを指定してのWebサーバへの直接アクセスは禁止。ただしメンテナンスのため、自社のIPアドレスからなら許可
とした
この状態で各Webサーバを再起動することで復旧した
今回のように「Webサーバへの直接アクセスを遮断する」という対応がしづらいため
そもそも、ロードバランサーとWebサーバのセキュリティグループは別々にしておくべきではあった
■攻撃手法
うちのapacheにCONNECTとかいうリクエストが。 - Blanktar
https://blanktar.jp/blog/2013/10/what-is-http-connect-request
HTTPのCONNECTというのは、代理接続を要求するためのメソッド
今回のサーバを踏み台にした上で、他のサーバへ接続しようとしている
ただし踏み台にしようとしたというより、アクセスできないサイトへの接続欲求を大量に送ることにより、サーバの負荷を上げようとしたものと思われる
詳細は引き続き要勉強