refirio.org

■RAIDについて

■MegaCliでのRAID監視（さくらの専用サーバ）

# wget http://progeny.sakura.ad.jp/scripts/raid_check/setup_raid_chk.sh # sh setup_raid_chk.sh

14 3 * * * root sh /opt/sakura/raid_chk.sh 22 1 5 * * root sh /opt/sakura/watch_chk.sh

# vi /etc/snmp/snmpd.conf

# vi /etc/sysconfig/iptables

#!/bin/sh #### raid status check # # return 0: no problem # return 1: RAID error # return 2: Rebuilding status # return 3: Another problem # check_status (){ MegaCli=`which MegaCli 2> /dev/null` if [ -z "${MegaCli}" ]; then return 3 fi ${MegaCli} -LDInfo -Lall -aALL -NoLog > /dev/null 2>&1 if [ "$?" -ne 0 ]; then return 3 fi stat=`${MegaCli} -PDList -aALL -NoLog | awk '/Rebuild/ {print $3}'` if [ "$stat" = "Rebuild" ]; then return 2 fi stat=`${MegaCli} -LDInfo -Lall -aALL -NoLog | grep ^State` if echo "$stat" | grep -q Optimal 2> /dev/null; then return 0 fi return 1 } check_status echo $?

# ./my_check_raid.sh 0

# ./my_check_raid.sh 1

弊社監視システムにて、お客様ご利用中のサーバに搭載されているストレージに、 何らかの障害が発生している可能性があることを検知いたしました。 ストレージ障害によるデータ消失等を未然に防ぐため、サーバの障害ストレージ 交換対応を実施させていただきたく存じます。 障害を検知したストレージ　Port0側 本作業にサーバの停止は伴いません。しかしながら、ミラーリングの再構築時に DISKI/O のパフォーマンスが低下いたします。 再構築の処理が完了するまでの所要時間は、ストレージ容量とサーバ負荷状況 により変動いたします。 サーバ負荷の少ない時間をご指定いただければ、作業日時の調整の上で作業を 実施させていただきます。 現在正常に稼働しているストレージで万が一障害が発生いたしますと、 データが損失してしまう可能性がございます。 ストレージ交換の前に、データのバックアップを取得いただきますこと を推奨させていただきます。 以下にストレージ交換作業に関する作業依頼申込書を送付させていただきます。

# ./my_check_raid.sh 1

■MegaCliのコマンドの意味（さくらの専用サーバ）

State : Optimal … 正常な状態 State : Degraded … RAID が縮退動作（制限された状態で動作している）にある

Slot Number: … 対象のハードディスクが接続されたスロット番号 Firmware state : Online … 正常な状態 Firmware state : Critical … 対象のハードディスクで故障している可能性がある Firmware state : Rebuild … リビルド中 Firmware state : Hotspare … Disk がホットスペアとして確保されている（ホットスペアは提供しない） Firmware state : 上記以外 … なんらかの障害が起きている可能性がある

Enclosure Device ID: 252 Slot Number: 0

# MegaCli -PDRbld -ShowProg -PhysDrv [252:0] -aALL

■MegaCliの実行結果例（さくらの専用サーバ）

# MegaCli -LDInfo -Lall -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name :array0 RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 930.812 GB Sector Size : 512 Is VD emulated : No Mirror Data : 930.812 GB State : Optimal … 通常状態 Strip Size : 256 KB Number Of Drives : 2 Span Depth : 1 Default Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disabled Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: None Can spin up in 1 minute: Yes LD has drives that support T10 power conditions: No LD's IO profile supports MAX power savings with cached writes: No Bad Blocks Exist: No PI type: No PI Is VD Cached: No Exit Code: 0x00 # MegaCli -PDList -aALL Adapter #0 Enclosure Device ID: 252 … デバイスID（エンクロージャー番号） Slot Number: 0 … スロット0の情報 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: N/A Device Id: 9 WWN: 5000c500791160c2 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 931.322 GB [0x746a5288 Sectors] Non Coerced Size: 930.822 GB [0x745a5288 Sectors] Coerced Size: 930.812 GB [0x745a0000 Sectors] Sector Size: 512 Logical Sector Size: 512 Physical Sector Size: 512 Firmware state: Online, Spun Up … オンライン Device Firmware Level: N007 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x4433221107000000 Connected Port Number: 1(path0) Inquiry Data: 9XG7753VST91000640NS N007 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive: Not Certified Drive Temperature :28C (82.40 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Drive has flagged a S.M.A.R.T alert : No Enclosure Device ID: 252 … デバイスID Slot Number: 1 … スロット1の情報 Drive's position: DiskGroup: 0, Span: 0, Arm: 1 Enclosure position: N/A Device Id: 8 WWN: 5000c50079117423 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 931.322 GB [0x746a5288 Sectors] Non Coerced Size: 930.822 GB [0x745a5288 Sectors] Coerced Size: 930.812 GB [0x745a0000 Sectors] Sector Size: 512 Logical Sector Size: 512 Physical Sector Size: 512 Firmware state: Online, Spun Up … オンライン Device Firmware Level: N007 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x4433221106000000 Connected Port Number: 0(path0) Inquiry Data: 9XG774V8ST91000640NS N007 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive: Not Certified Drive Temperature :28C (82.40 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Drive has flagged a S.M.A.R.T alert : No Exit Code: 0x00

# MegaCli -LDInfo -Lall -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name :array0 RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 930.812 GB Sector Size : 512 Is VD emulated : No Mirror Data : 930.812 GB State : Degraded … 縮退動作 Strip Size : 256 KB Number Of Drives : 2 Span Depth : 1 Default Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disabled Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: None Can spin up in 1 minute: Yes LD has drives that support T10 power conditions: No LD's IO profile supports MAX power savings with cached writes: No Bad Blocks Exist: No PI type: No PI Is VD Cached: No Exit Code: 0x00 # MegaCli -PDList -aALL Adapter #0 Enclosure Device ID: 252 … デバイスID Slot Number: 1 … スロット1の情報は表示されるが、スロット0の情報が表示されない。異常があってアクセスできないからと思われる Drive's position: DiskGroup: 0, Span: 0, Arm: 1 Enclosure position: N/A Device Id: 8 WWN: 5000c50079117423 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 931.322 GB [0x746a5288 Sectors] Non Coerced Size: 930.822 GB [0x745a5288 Sectors] Coerced Size: 930.812 GB [0x745a0000 Sectors] Sector Size: 512 Logical Sector Size: 512 Physical Sector Size: 512 Firmware state: Online, Spun Up … オンライン Device Firmware Level: N007 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x4433221106000000 Connected Port Number: 0(path0) Inquiry Data: 9XG774V8ST91000640NS N007 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive: Not Certified Drive Temperature :29C (84.20 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Drive has flagged a S.M.A.R.T alert : No Exit Code: 0x00

# MegaCli -LDInfo -Lall -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name :array0 RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 930.812 GB Sector Size : 512 Is VD emulated : No Mirror Data : 930.812 GB State : Degraded … 縮退動作 Strip Size : 256 KB Number Of Drives : 2 Span Depth : 1 Default Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disabled Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: None Can spin up in 1 minute: Yes LD has drives that support T10 power conditions: No LD's IO profile supports MAX power savings with cached writes: No Bad Blocks Exist: No PI type: No PI Is VD Cached: No Exit Code: 0x00 # MegaCli -PDList -aALL Adapter #0 Enclosure Device ID: 252 … デバイスID Slot Number: 0 … スロット0の情報 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: N/A Device Id: 9 WWN: 5000c500b91b76ce Sequence Number: 13 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 931.322 GB [0x746a5288 Sectors] Non Coerced Size: 930.822 GB [0x745a5288 Sectors] Coerced Size: 930.812 GB [0x745a0000 Sectors] Sector Size: 512 Logical Sector Size: 512 Physical Sector Size: 512 Firmware state: Rebuild … リビルド Device Firmware Level: N002 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x4433221107000000 Connected Port Number: 1(path0) Inquiry Data: W471KFMTST1000NX0423 N002 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive: Not Certified Drive Temperature :32C (89.60 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Drive has flagged a S.M.A.R.T alert : No Enclosure Device ID: 252 … デバイスID Slot Number: 1 … スロット1の情報 Drive's position: DiskGroup: 0, Span: 0, Arm: 1 Enclosure position: N/A Device Id: 8 WWN: 5000c50079117423 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 931.322 GB [0x746a5288 Sectors] Non Coerced Size: 930.822 GB [0x745a5288 Sectors] Coerced Size: 930.812 GB [0x745a0000 Sectors] Sector Size: 512 Logical Sector Size: 512 Physical Sector Size: 512 Firmware state: Online, Spun Up … オンライン Device Firmware Level: N007 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x4433221106000000 Connected Port Number: 0(path0) Inquiry Data: 9XG774V8ST91000640NS N007 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive: Not Certified Drive Temperature :31C (87.80 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Drive has flagged a S.M.A.R.T alert : No Exit Code: 0x00

# MegaCli -PDRbld -ShowProg -PhysDrv [252:0] -aALL Device(Encl-252 Slot-0) is not in rebuild process … リビルド中でない場合 # MegaCli -PDRbld -ShowProg -PhysDrv [252:0] -aALL Rebuild Progress on Device at Enclosure 252, Slot 0 Completed 79% in 143 Minutes. … 79%完了。143分経過している

■omreportでのRAID監視（WADAX専用サーバ）

$ omreport storage vdisk $ omreport storage pdisk controller=0

$ /opt/dell/srvadmin/bin/omreport storage vdisk $ /opt/dell/srvadmin/bin/omreport storage pdisk controller=0

$ omreport storage vdisk … 仮想ディスクのステータス確認 List of Virtual Disks in the System Controller SAS 6/iR Integrated (Embedded) ID : 0 Status : Ok … 通常状態 Name : Virtual Disk 0 State : Ready … 通常状態 HotSpare Policy violated : Not Assigned Virtual Disk Bad Blocks : Not Applicable Secured : Not Applicable Progress : Not Applicable Layout : RAID-1 Size : 931.00 GB (999653638144 bytes) Device Name : /dev/sda Bus Protocol : SAS Media : HDD Read Policy : Not Applicable Write Policy : Not Applicable Cache Policy : Not Applicable Stripe Element Size : Not Applicable Disk Cache Policy : Disabled $ omreport storage pdisk controller=0 … 物理ディスクのステータス確認 List of Physical Disks on Controller SAS 6/iR Integrated (Embedded) Controller SAS 6/iR Integrated (Embedded) ID : 0:0:0 Status : Ok Name : Physical Disk 0:0:0 State : Online Failure Predicted : No Certified : Not Applicable Encryption Capable : No Secured : Not Applicable Progress : Not Applicable Bus Protocol : SAS Media : HDD Capacity : 931.00 GB (999653638144 bytes) Used RAID Disk Space : 931.00 GB (999653638144 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST31000424SS Revision : KS68 Serial No. : 9WK1QBT7 Part Number : TH0U738K212330AU003DA02 Negotiated Speed : 3.00 Gbps Capable Speed : 3.00 Gbps Manufacture Day : 02 Manufacture Week : 46 Manufacture Year : 2010 SAS Address : 5000C50025A5FC0D ID : 0:0:1 Status : Ok … 問題は発生していない Name : Physical Disk 0:0:1 State : Online … 通常状態 Failure Predicted : No Certified : Not Applicable Encryption Capable : No Secured : Not Applicable Progress : Not Applicable … 通常状態 Bus Protocol : SAS Media : HDD Capacity : 931.00 GB (999653638144 bytes) Used RAID Disk Space : 931.00 GB (999653638144 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST31000424SS Revision : KS68 Serial No. : 9WK2EPRG Part Number : TH0U738K212330B9001GA02 Negotiated Speed : 3.00 Gbps Capable Speed : 3.00 Gbps Manufacture Day : 05 Manufacture Week : 46 Manufacture Year : 2010 SAS Address : 5000C50025B0A515

# omreport storage vdisk … 仮想ディスクのステータス確認 List of Virtual Disks in the System Controller SAS 6/iR Integrated (Embedded) ID : 0 Status : Non-Critical … 致命的ではないが警告が発生している Name : Virtual Disk 0 State : Degraded … 縮退動作 HotSpare Policy violated : Not Assigned Virtual Disk Bad Blocks : Not Applicable Secured : Not Applicable Progress : Not Applicable Layout : RAID-1 Size : 931.00 GB (999653638144 bytes) Device Name : /dev/sda Bus Protocol : SAS Media : HDD Read Policy : Not Applicable Write Policy : Not Applicable Cache Policy : Not Applicable Stripe Element Size : Not Applicable Disk Cache Policy : Disabled # omreport storage pdisk controller=0 … 物理ディスクのステータス確認 List of Physical Disks on Controller SAS 6/iR Integrated (Embedded) Controller SAS 6/iR Integrated (Embedded) ID : 0:0:0 Status : Critical … 問題が発生している Name : Physical Disk 0:0:0 State : Failed … オンラインでは無い Failure Predicted : No Certified : Not Applicable Encryption Capable : No Secured : Not Applicable Progress : Not Applicable Bus Protocol : SAS Media : HDD Capacity : 931.00 GB (999653638144 bytes) Used RAID Disk Space : 931.00 GB (999653638144 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST31000424SS Revision : KS68 Serial No. : 9WK1QBT7 Part Number : TH0U738K212330AU003DA02 Negotiated Speed : 3.00 Gbps Capable Speed : 3.00 Gbps Manufacture Day : 02 Manufacture Week : 46 Manufacture Year : 2010 SAS Address : 5000C50025A5FC0D ID : 0:0:1 Status : Ok … 問題は発生していない Name : Physical Disk 0:0:1 State : Online … オンライン Failure Predicted : No Certified : Not Applicable Encryption Capable : No Secured : Not Applicable Progress : Not Applicable Bus Protocol : SAS Media : HDD Capacity : 931.00 GB (999653638144 bytes) Used RAID Disk Space : 931.00 GB (999653638144 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST31000424SS Revision : KS68 Serial No. : 9WK2EPRG Part Number : TH0U738K212330B9001GA02 Negotiated Speed : 3.00 Gbps Capable Speed : 3.00 Gbps Manufacture Day : 05 Manufacture Week : 46 Manufacture Year : 2010 SAS Address : 5000C50025B0A515 # omreport storage vdisk | grep State State : Degraded # omreport storage pdisk controller=0 | grep State State : Failed State : Online

この度はご不便をおかけし、誠に申し訳ございません。 弊社担当部署にてご利用サーバーの状態を確認し HDD 0の故障を確認致しましたので、至急交換作業を実施致しました。 ※ホットスワップで交換致しましたので、ダウンタイムはございません。 先程、HDD交換が完了いたしましたので、 ご報告申し上げます。 お手数ではございますが、弊社にてサーバ起動の確認はしておりますが、 お客様側でもサーバへの接続・動作等、ご確認くださいますよう お願いいたします。 誠に恐れ入りますが、よろしくお願い申し上げます。

$ omreport storage vdisk … 仮想ディスクのステータス確認 List of Virtual Disks in the System Controller SAS 6/iR Integrated (Embedded) ID : 0 Status : Non-Critical … 致命的ではないが警告が発生している Name : Virtual Disk 0 State : Degraded … 縮退動作 HotSpare Policy violated : Not Assigned Virtual Disk Bad Blocks : Not Applicable Secured : Not Applicable Progress : Not Applicable Layout : RAID-1 Size : 931.00 GB (999653638144 bytes) Device Name : /dev/sda Bus Protocol : SAS Media : HDD Read Policy : Not Applicable Write Policy : Not Applicable Cache Policy : Not Applicable Stripe Element Size : Not Applicable Disk Cache Policy : Disabled $ omreport storage pdisk controller=0 … 物理ディスクのステータス確認 List of Physical Disks on Controller SAS 6/iR Integrated (Embedded) Controller SAS 6/iR Integrated (Embedded) ID : 0:0:0 Status : Ok Name : Physical Disk 0:0:0 State : Online Failure Predicted : No Certified : Not Applicable Encryption Capable : No Secured : Not Applicable Progress : Not Applicable Bus Protocol : SAS Media : HDD Capacity : 931.00 GB (999653638144 bytes) Used RAID Disk Space : 931.00 GB (999653638144 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST31000424SS Revision : KS68 Serial No. : 9WK1QBT7 Part Number : TH0U738K212330AU003DA02 Negotiated Speed : 3.00 Gbps Capable Speed : 3.00 Gbps Manufacture Day : 02 Manufacture Week : 46 Manufacture Year : 2010 SAS Address : 5000C50025A5FC0D ID : 0:0:1 Status : Ok … 問題は発生していない Name : Physical Disk 0:0:1 State : Rebuilding … リビルド中になっている Failure Predicted : No Certified : Not Applicable Encryption Capable : No Secured : Not Applicable Progress : 9% complete … リビルドは9%完了している Bus Protocol : SAS Media : HDD Capacity : 931.00 GB (999653638144 bytes) Used RAID Disk Space : 931.00 GB (999653638144 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST31000424SS Revision : KS68 Serial No. : 9WK2EPRG Part Number : TH0U738K212330B9001GA02 Negotiated Speed : 3.00 Gbps Capable Speed : 3.00 Gbps Manufacture Day : 05 Manufacture Week : 46 Manufacture Year : 2010 SAS Address : 5000C50025B0A515

■ZabbixでのRAID監視

# cat /proc/mdstat Personalities : unused devices: <none>