Kernel error EXT2-fs error, EXT3-fs abort, Disk unavailable | FortiWiFi-60D
Environment
Summary
As Kernel error EXT2-fs error, EXT3-fs abort, Disk unavailable, required us to restore the FortiGate.
Questions
From 2018-Jul-26 to 2018-Aug-10: Fail
Abnormal:
Normal:
On 2018-Jul-01: Normal although the Errors occurred a few time
Solution
Connect the FortiGate via USB cable
Run FortiExplorer. Backup the Config first. Then, Reboot
Run FortiExplorer
Format boot device
Review TFTP parameters
Initiate TFTP firmware transfer
Restore back the Config
Done
References
Using the USB MGMT Console port https://cookbook.fortinet.com/using-the-usb-mgmt-console-port/
Update
- FortiWiFi-60D: Firmware 5.2.12
Summary
As Kernel error EXT2-fs error, EXT3-fs abort, Disk unavailable, required us to restore the FortiGate.
Questions
From 2018-Jul-26 to 2018-Aug-10: Fail
date=2018-07-26 time=09:31:48 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100041001 type=event subtype=system level=critical vd="root" logdesc="FortiGate update failed" status=update msg=" Fortigate push update failed"
[...]
date=2018-08-08...
date=2018-08-09...
[...]
date=2018-08-10 time=10:11:19 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020209 type=event subtype=system level=critical vd="root" logdesc="Disk unavailable" msg="Disk /dev/sdb is unavailable"
[...]
date=2018-08-10 time=10:14:18 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT2-fs error (device sd(8,1)): ext2_read_inode: unable to read inode block - inode=24577, block=98307"
[...]
date=2018-08-08...
date=2018-08-09...
[...]
date=2018-08-10 time=10:11:19 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020209 type=event subtype=system level=critical vd="root" logdesc="Disk unavailable" msg="Disk /dev/sdb is unavailable"
[...]
date=2018-08-10 time=10:14:18 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT2-fs error (device sd(8,1)): ext2_read_inode: unable to read inode block - inode=24577, block=98307"
Abnormal:
ATP Services
AntiVirus Licensed (Expires YYYY-MM-DD)
AV Definitions 1.00000 (Updated 2012-10-17 via Manual Update)
AntiVirus Licensed (Expires YYYY-MM-DD)
AV Definitions 1.00000 (Updated 2012-10-17 via Manual Update)
Normal:
ATP Services
AntiVirus Licensed (Expires YYYY-MM-DD)
AV Definitions 61.00303 (Updated 2018-08-08 via Manual Update)
AntiVirus Licensed (Expires YYYY-MM-DD)
AV Definitions 61.00303 (Updated 2018-08-08 via Manual Update)
On 2018-Jul-01: Normal although the Errors occurred a few time
date=2018-07-01 time=02:14:38 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT2-fs error (device sd(8,1)): ext2_read_inode: unable to read inode block - inode=20530, block=81929"
[...]
date=2018-07-01 time=02:14:41 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020209 type=event subtype=system level=critical vd="root" logdesc="Disk unavailable" msg="Disk /dev/sdb is unavailable"
[...]
date=2018-07-01 time=06:21:50 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT3-fs abort (device sd(8,3)): ext3_journal_start: Detected aborted journal"
[...]
date=2018-07-01 time=02:14:41 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020209 type=event subtype=system level=critical vd="root" logdesc="Disk unavailable" msg="Disk /dev/sdb is unavailable"
[...]
date=2018-07-01 time=06:21:50 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT3-fs abort (device sd(8,3)): ext3_journal_start: Detected aborted journal"
# get hardware status
Model name: FortiWiFi-60D
ASIC version: CP0
ASIC SRAM: 64M
CPU: FortiSOC2
Number of CPUs: 1
RAM: 1835 MB
Compact Flash: 3879 MB /dev/sda
Hard disk: not available
USB Flash: not available
WiFi Chipset: Atheros AR9300
WiFi firmware version: 0.9.17.1
Model name: FortiWiFi-60D
ASIC version: CP0
ASIC SRAM: 64M
CPU: FortiSOC2
Number of CPUs: 1
RAM: 1835 MB
Compact Flash: 3879 MB /dev/sda
Hard disk: not available
USB Flash: not available
WiFi Chipset: Atheros AR9300
WiFi firmware version: 0.9.17.1
# diag hardware deviceinfo disk
Disk Internal(boot) ref: 3.8GB type: USB [FORTINET S02_V140905_004] dev: /dev/sda
partition ref: 247.0MB, 213.0MB free mounted: Y label: dev: /dev/sda1(boot) start: 1
partition ref: 247.0MB, 214.0MB free mounted: N label: dev: /dev/sda2(boot) start: 524289
partition ref: 3 3.2GB, 3.1GB free mounted: Y label: dev: /dev/sda3 start: 1048577
Disk Internal ref: 16 7.5GB type: USB [FORTINET S02_V140905_008] dev: /dev/sdb
Total available disks: 2
Max SSD disks: 0 Available storage disks: 0
Disk Internal(boot) ref: 3.8GB type: USB [FORTINET S02_V140905_004] dev: /dev/sda
partition ref: 247.0MB, 213.0MB free mounted: Y label: dev: /dev/sda1(boot) start: 1
partition ref: 247.0MB, 214.0MB free mounted: N label: dev: /dev/sda2(boot) start: 524289
partition ref: 3 3.2GB, 3.1GB free mounted: Y label: dev: /dev/sda3 start: 1048577
Disk Internal ref: 16 7.5GB type: USB [FORTINET S02_V140905_008] dev: /dev/sdb
Total available disks: 2
Max SSD disks: 0 Available storage disks: 0
# diag disktest device 1
Curent Test Device: /dev/sda
device sd(8,1): /dev/sda1
device sd(8,16): /dev/sdb
# diag disktest block 1
Current Test Block: 1M.
# diag disktest size 10
Current Size Limit: 10 MB
# diag disktest run 3
Round 1 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 5.6 sec
Read Speed: 13.2MB/s
Write Speed: 5.0MB/s
Round 1 Finished!
Round 2 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 4.3 sec
Read Speed: 15.3MB/s
Write Speed: 7.1MB/s
Round 2 Finished!
Round 3 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 4.2 sec
Read Speed: 15.9MB/s
Write Speed: 7.3MB/s
Roud 3 Finished!
Curent Test Device: /dev/sda
device sd(8,1): /dev/sda1
device sd(8,16): /dev/sdb
# diag disktest block 1
Current Test Block: 1M.
# diag disktest size 10
Current Size Limit: 10 MB
# diag disktest run 3
Round 1 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 5.6 sec
Read Speed: 13.2MB/s
Write Speed: 5.0MB/s
Round 1 Finished!
Round 2 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 4.3 sec
Read Speed: 15.3MB/s
Write Speed: 7.1MB/s
Round 2 Finished!
Round 3 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 4.2 sec
Read Speed: 15.9MB/s
Write Speed: 7.3MB/s
Roud 3 Finished!
Solution
Connect the FortiGate via USB cable
Run FortiExplorer. Backup the Config first. Then, Reboot
Devices: Office-FW-01
--> Command-line Interface
--> Command-line Interface
Office-FW-01 login: admin
Password: ***************
No entry for terminal type "vt220";
using dumb terminal settings.
Welcome !
Office-FW-01 # execute reboot
This operation will reboot the system !
Do you want to continue? (y/n)
Password: ***************
No entry for terminal type "vt220";
using dumb terminal settings.
Welcome !
Office-FW-01 # execute reboot
This operation will reboot the system !
Do you want to continue? (y/n)
date=2018-08-11 time=10:16:16 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100032138 type=event subtype=system level=critical vd="root" logdesc="Device rebooted" user="admin" ui="GUI(127.0.0.1)" action=reboot msg="User admin rebooted the device from GUI(127.0.0.1). The reason is 'reboot through JSON monitor API'"
Run FortiExplorer
USB Serial Console
--> COM3
--> COM3
Format boot device
Please wait for OS to boot, or press any key to display
[C]: Configure TFTP parameters.
[R]: Review TFTP parameters.
[T]: Initiate TFTP firmware transfer.
[F]: Format boot device.
[I]: System information.
[B]: Boot with backup firmware and set as default.
[Q]: Quit menu and continue to boot.
[H]: Display this list of options.
Enter C,R,T,F,I,B,Q, or H: F
It will erase data in boot device. Continue? [yes/no]:yes
[C]: Configure TFTP parameters.
[R]: Review TFTP parameters.
[T]: Initiate TFTP firmware transfer.
[F]: Format boot device.
[I]: System information.
[B]: Boot with backup firmware and set as default.
[Q]: Quit menu and continue to boot.
[H]: Display this list of options.
Enter C,R,T,F,I,B,Q, or H: F
It will erase data in boot device. Continue? [yes/no]:yes
Review TFTP parameters
Enter C,R,T,F,I,B,Q, or H: R
Image download port: WAN1
DHCP status: Disabled
Local VLAN ID:
Local IP address: 192.168.1.1
Local subnet mask: 255.255.255.0
Local gateway: 192.168.1.254
TFTP server IP address: 192.168.1.100
Firmware file name: image.out
Image download port: WAN1
DHCP status: Disabled
Local VLAN ID:
Local IP address: 192.168.1.1
Local subnet mask: 255.255.255.0
Local gateway: 192.168.1.254
TFTP server IP address: 192.168.1.100
Firmware file name: image.out
Initiate TFTP firmware transfer
Enter C,R,T,F,I,B,Q, or H: T
Please connect TFTP server to Ethernet port 'WAN1'.
MAC: 00:11:22:AA:BB:CC
Connect to tftp server 192.168.1.100 ...
TFTP error 'File not found' (1)
Re-enter firmware file name [image.out]: image-01.out
...done
Starting again.
MAC: 00:11:22:AA:BB:CC
Connect to tftp server 192.168.1.100 ...
###########################################
Image Received.
Check image... ok
Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]?D
Programming the boot device now.
The system must reformat the boot device to install this firmware.
The default and backup firmware will be lost.
Continue:[Y/N]?Y
...........................................
Booting OS...
Reading boot image... 1952445 by
Please connect TFTP server to Ethernet port 'WAN1'.
MAC: 00:11:22:AA:BB:CC
Connect to tftp server 192.168.1.100 ...
TFTP error 'File not found' (1)
Re-enter firmware file name [image.out]: image-01.out
...done
Starting again.
MAC: 00:11:22:AA:BB:CC
Connect to tftp server 192.168.1.100 ...
###########################################
Image Received.
Check image... ok
Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]?D
Programming the boot device now.
The system must reformat the boot device to install this firmware.
The default and backup firmware will be lost.
Continue:[Y/N]?Y
...........................................
Booting OS...
Reading boot image... 1952445 by
Restore back the Config
Please wait for OS to boot, or press any key to display configuration menu......
Booting OS...
Reading boot image... 1952445 by
Booting OS...
Reading boot image... 1952445 by
Done
References
Troubleshooting Tip : FortiGate or FortiAnalyzer error message "EXT3-fs error (device...)..." http://kb.fortinet.com/kb/viewContent.do?externalId=FD31857
Technical Note: Diagnose command to test flash SSD http://kb.fortinet.com/kb/documentLink.do?externalID=FD37415
Troubleshooting Tip: How to detect affected device for file system error 'EXT3-fs error (device)' http://kb.fortinet.com/kb/viewContent.do?externalId=FD37499&sliceId=1
Using the USB MGMT Console port https://cookbook.fortinet.com/using-the-usb-mgmt-console-port/
Update