Kernel error EXT2-fs error, EXT3-fs abort, Disk unavailable | FortiWiFi-60D

Environment
  • FortiWiFi-60D: Firmware 5.2.12


Summary
As Kernel error EXT2-fs error, EXT3-fs abort, Disk unavailable, required us to restore the FortiGate.


Questions
From 2018-Jul-26 to 2018-Aug-10: Fail
date=2018-07-26 time=09:31:48 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100041001 type=event subtype=system level=critical vd="root" logdesc="FortiGate update failed" status=update msg=" Fortigate push update failed"

[...]

date=2018-08-08...

date=2018-08-09...

[...]

date=2018-08-10 time=10:11:19 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020209 type=event subtype=system level=critical vd="root" logdesc="Disk unavailable" msg="Disk /dev/sdb is unavailable"

[...]

date=2018-08-10 time=10:14:18 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT2-fs error (device sd(8,1)): ext2_read_inode: unable to read inode block - inode=24577, block=98307"


Abnormal:
ATP Services
AntiVirus Licensed (Expires YYYY-MM-DD)
AV Definitions 1.00000 (Updated 2012-10-17 via Manual Update)


Normal:
ATP Services
AntiVirus Licensed (Expires YYYY-MM-DD)
AV Definitions 61.00303 (Updated 2018-08-08 via Manual Update)


On 2018-Jul-01: Normal although the Errors occurred a few time
date=2018-07-01 time=02:14:38 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT2-fs error (device sd(8,1)): ext2_read_inode: unable to read inode block - inode=20530, block=81929"

[...]

date=2018-07-01 time=02:14:41 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020209 type=event subtype=system level=critical vd="root" logdesc="Disk unavailable" msg="Disk /dev/sdb is unavailable"

[...]

date=2018-07-01 time=06:21:50 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT3-fs abort (device sd(8,3)): ext3_journal_start: Detected aborted journal"

# get hardware status
   Model name: FortiWiFi-60D
   ASIC version: CP0
   ASIC SRAM: 64M
   CPU: FortiSOC2
   Number of CPUs: 1
   RAM: 1835 MB
   Compact Flash: 3879 MB /dev/sda
   Hard disk: not available
   USB Flash: not available
   WiFi Chipset: Atheros AR9300
   WiFi firmware version: 0.9.17.1

# diag hardware deviceinfo disk
   Disk Internal(boot)  ref:       3.8GB    type: USB [FORTINET S02_V140905_004] dev: /dev/sda
     partition ref:     247.0MB, 213.0MB free  mounted: Y  label:  dev: /dev/sda1(boot) start: 1
     partition ref:     247.0MB, 214.0MB free  mounted: N  label:  dev: /dev/sda2(boot) start: 524289
     partition ref:   3   3.2GB,   3.1GB free  mounted: Y  label:  dev: /dev/sda3 start: 1048577
   
   Disk Internal        ref:  16   7.5GB    type: USB [FORTINET S02_V140905_008] dev: /dev/sdb
   
   Total available disks: 2
   Max SSD disks: 0  Available storage disks: 0

# diag disktest device 1
Curent Test Device: /dev/sda
device sd(8,1): /dev/sda1
device sd(8,16): /dev/sdb

# diag disktest  block 1
Current Test Block: 1M.

# diag disktest size 10
Current Size Limit: 10 MB

# diag disktest run 3
Round 1 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0   0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 5.6 sec
Read Speed: 13.2MB/s
Write Speed: 5.0MB/s
Round 1 Finished!

Round 2 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0   0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 4.3 sec
Read Speed: 15.3MB/s
Write Speed: 7.1MB/s
Round 2 Finished!

Round 3 started.
Current Test Device: /dev/sdb
Total size: 3864M
Current Test Block: 1M.
Current Time Limit: 10MB
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0   0(0.00%): .........
Test Result: Passed
Tested size: 10MB (0.26% Coverage of whole disk)
Time used: 4.2 sec
Read Speed: 15.9MB/s
Write Speed: 7.3MB/s
Roud 3 Finished!


Solution
Connect the FortiGate via USB cable

Run FortiExplorer. Backup the Config first. Then, Reboot
Devices: Office-FW-01
--> Command-line Interface

Office-FW-01 login: admin
Password: ***************
No entry for terminal type "vt220";
using dumb terminal settings.
Welcome !

Office-FW-01 # execute reboot
This operation will reboot the system !
Do you want to continue? (y/n)

date=2018-08-11 time=10:16:16 devname=Office-FW-01 devid=FWF60DXXXDEVID logid=0100032138 type=event subtype=system level=critical vd="root" logdesc="Device rebooted" user="admin" ui="GUI(127.0.0.1)" action=reboot msg="User admin rebooted the device from GUI(127.0.0.1). The reason is 'reboot through JSON monitor API'"


Run FortiExplorer
USB Serial Console
--> COM3


Format boot device
Please wait for OS to boot, or press any key to display

[C]: Configure TFTP parameters.
[R]: Review TFTP parameters.
[T]: Initiate TFTP firmware transfer.
[F]: Format boot device.
[I]: System information.
[B]: Boot with backup firmware and set as default.
[Q]: Quit menu and continue to boot.
[H]: Display this list of options.

Enter C,R,T,F,I,B,Q, or H: F
It will erase data in boot device. Continue? [yes/no]:yes


Review TFTP parameters
Enter C,R,T,F,I,B,Q, or H: R

Image download port: WAN1
DHCP status: Disabled
Local VLAN ID:
Local IP address: 192.168.1.1
Local subnet mask: 255.255.255.0
Local gateway: 192.168.1.254
TFTP server IP address: 192.168.1.100
Firmware file name: image.out


Initiate TFTP firmware transfer
Enter C,R,T,F,I,B,Q, or H: T

Please connect TFTP server to Ethernet port 'WAN1'.

MAC: 00:11:22:AA:BB:CC

Connect to tftp server 192.168.1.100 ...

TFTP error 'File not found' (1)
Re-enter firmware file name [image.out]: image-01.out

...done
Starting again.

MAC: 00:11:22:AA:BB:CC

Connect to tftp server 192.168.1.100 ...

###########################################
Image Received.
Check image... ok
Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]?D

Programming the boot device now.
The system must reformat the boot device to install this firmware.
The default and backup firmware will be lost.
Continue:[Y/N]?Y
...........................................

Booting OS...
Reading boot image... 1952445 by

Restore back the Config
Please wait for OS to boot, or press any key to display configuration menu......

Booting OS...
Reading boot image... 1952445 by


Done



References
Troubleshooting Tip : FortiGate or FortiAnalyzer error message "EXT3-fs error (device...)..."   http://kb.fortinet.com/kb/viewContent.do?externalId=FD31857

Technical Note: Diagnose command to test flash SSD   http://kb.fortinet.com/kb/documentLink.do?externalID=FD37415


Troubleshooting Tip: How to detect affected device for file system error 'EXT3-fs error (device)'   http://kb.fortinet.com/kb/viewContent.do?externalId=FD37499&sliceId=1

Using the USB MGMT Console port   https://cookbook.fortinet.com/using-the-usb-mgmt-console-port/



Update