boot to black screen, tty not available

liepumartins

This is second time my system has "soft bricked".
Both times it happened during an eopkg update, likely during linux-firmware update, system suddenly reboots, presents LUKS input and then blank screen.

First time Ctrl+Alt+F2 worked and I was able to run sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall

This time tty is not available. Screen is blank and every minute or so text cursor blinks once in upper left corner.
I also tried to SSH to no avail. Happens with both installed kernels 5.10.7-168 and 5.10.9-169
I guess live boot and chroot is the answer.

My main concern is not the fix, but the cause. eopkg up should not be able to crash the system. :sad:

I have intel video, the rig is fairly old i5-4670k on msi z87-g45 board. I updated BIOS to latest (E7821IMS v1.9), after first time this happened.

Ideas?

liepumartins

Apparently it is worse. I managed to chroot, but after running eopkg check | grep Broken | awk '{print $4}' | xargs eopkg it --reinstall and reinstall of 51 packages it crashed system. "syncing filesystems..." was the last message I saw.

EDIT:
Beginning to suspect SSD.
Chrooted again, ran smartctl found no errors or problems.
Currently cherry picking broken packages and reinstalling. Leaving linux-current for last. Did not crash this time.

EDIT2: More fun. Confirmed that eopkg it linux-firmware --reinstall crashes system.
Even more fun. While the system is now bootable, user is no longer in sudoers file 🤦‍♂️

algent

liepumartins Maybe it is better to reinstall the system, since there is a new ISO, no updates needed to do after installation, just install your favorite apps.

DasJott

That really sounds like defective RAM or ssd...

liepumartins

Chrooted again to reset root password.
/etc/sudo.conf and /etc/sudoers where empty, did cp /etc/sudo.conf.newconfig /etc/sudo.conf and the sudoers file.

Yea, I should run memtest tonight. And smartctl -t long already running, will see in 85 minutes.
Anyway here is current output, I do not see anything wrong there:

smartctl -i /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.10.9-169.current] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 860 EVO mSATA 1TB
Serial Number:    S41PNB0K102047E
LU WWN Device Id: 5 002538 e40057176
Firmware Version: RVT41B6Q
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      mSATA
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Feb  4 16:04:50 2021 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl -A /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.10.9-169.current] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       4481
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       1102
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       5
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   061   045   000    Old_age   Always       -       39
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       52
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       6967690010

liepumartins

I have no clue. 🤔 😕
24 hours of memtest did not throw any errors.
smartctl long test also came in negative.

downhill

You might want to check the log files to see if there might be any evidence as to the cause. Of course, you'd need to be able to access the partition in which Solus is installed.