Recovering system after failed update

laurker

A dear friend of mine unplugged the power cord of my battery-less laptop while I was installing updates yesterday.

I am running Solus 4.8 XFCE. As expected, the system no longer boots, with a number of services failing to start and missing dependencies. TTY is not accessible.

I have tried to follow the system recovery guide at help.getsol.us, using a 4.8 live USB, and after a bit of faffing was able to get it to ping google.com from the chroot environment.

However, when subsequently typing the commands sudo eopkg rebuild-db, sudo eopkg up or any other eopkg command, there is no response whatsoever from eopkg. The command completes instantly and I am prompted with root@solus / # on the next line. Other commands in the chroot environment execute as expected, and eopkg in the live environment responds normally.

Have I done something wrong, is this a problem with 4.8, or did eopkg get broken with the update process? Thank you for any suggestions.

laurker

I followed these instructions to install a binary package of eopkg. For some reason wget gave a 404, even though I had internet connection and the URL can be accessed from a browser. So I downloaded the package through the browser, moved it to the chroot mount point with mv, and extracted it using commands pointing to that location.

I was able to then ./usr/bin/eopkg4-bin up ./usr/bin/eopkg4-bin rbd and ./usr/bin/eopkg4-bin check -Y. There were a tonne of broken packages, though I don't know which were really relevant. up only installed rclone, gnupg, libtasn1-32bit and appstream-catalog. However the regular eopkg command still does nothing and the system returns the same errors during boot.

Surprisingly I was able to view full eopkg history using ./usr/bin/eopkg4-bin history. This showed that network-manager-32bit was removed as the last action at the time the system was shut down. However I have been unable to rollback or install this package as both commands result in System error. Program terminated. Repo item network-manager-32bit not found

laurker

I ran ./usr/bin/eopkg4-bin check | grep Broken | awk '{print $4}' | xargs -r ./usr/bin/eopkg4-bin it --reinstall and the system is now installing a substantial number of packages. I will mark this as solved if it fixes the issue.

laurker

laurker After reinstalling said broken packages, eopkg command is now responsive. Following a restart, the system is bootable but lightdm.service fails. TTY is accessible. After sudo usysconf run -f sudo eopkg clean sudo eopkg delete-cache sudo eopkg rdb sudo eopkg up the same behaviour persists on reboot.

edit: This was likely because at some point in the process I tried to boot a different kernel, so it was first in the bootloader menu. Switching back to the LTS kernel that I had been previously using, the system boots successfully. I shall leave this stage out of the write-up.

brent

thoughts--
-a lot of your troubleshooting looked sound to me.
-unplugged/accidental shutdown during updates typically has awful results every distro.
-why you think you think you broke the eopkg package I can't tell..
-sudo eopkg rmo is the only cleanup command you did not run but might be moot now
-sudo eopkg dc should have cleaned the pipes imo
-PS--you won't see any verbose output from eopkg commands so assume you succeeded. or go eopkg --help to -see if there is a --verbose flag although I'm thinking no

-I wonder if a rollback is feasible at this point? you could try
-the failing of lightdm means a lot of things; some of these are solved:
https://discuss.getsol.us/d/10884-failed-to-start-lightdm-service/3
https://discuss.getsol.us/d/10513-failed-to-start-lightdm-service
https://discuss.getsol.us/d/10545-failed-to-start-lightdm-light-display-manager-after-restart
(if you have Budgie its sometimes indicative of having to to do a panel reset)
https://discuss.getsol.us/d/12163-cant-login-to-budgie-failed-to-start-session/5
^^that one shows it's the result of removing the wrong packages
-you also have a curious output comment about 32-bit which seems 'funny' i.e. what is 32bit for except for gaming?
-personally if I felt I exhausted all avenues (a few more hours for you!) then get my needed stuff off that drive and fresh install but you already considered that

hope any of this has value

laurker

brent Hello brent, I figured either the eopkg package, or something it depends on was broken, because the command eopkg produced no output whatsoever via chroot. So I couldn't run any commands normally.

From the eopkg4-bin binary, I did delete-cache to no effect. rmo didn't produce anything useful either. It seems only reinstalling broken packages fixed the issue.

Regarding the 32bit package I don't know, I couldn't find it in the repo so I just forgot about it. I don't know what it was (or wasn't) doing there.

I appreciate your input in my last help thread; that and your various forum posts helped me solve this issue.

However, I am pretty much afraid of updating now, because about ¼ of the time I end up with an unbootable system. Granted in this case it was human error, albeit not mine. However the less frequently you update, the more likely that something goes wrong it seems, which puts me in a quandry. Maybe I'll try Debian and never be troubled by updates, ever.

laurker

This issue was solved by

chroot from a live USB as described here.
Download eopkg4-bin via wget or browser from packages.getsol.us
exit chroot
sudo mv ~/Downloads/eopkg4-bin-4.0.0-17-1-x86_64.eopkg /[chroot mount point] to move the file from live USB download folder, if you used a browser
chroot /[chroot mount point] return to chroot
/usr/bin/unzip eopkg4-bin-4.0.0-17-1-x86_64.eopkg /usr/bin/tar xf install.tar.xz extract and install eopkg4-bin binary
sudo ./usr/bin/eopkg4-bin check | grep Broken | awk '{print $4}' | xargs -r ./usr/bin/eopkg4-bin it --reinstall check for and reinstall broken packages
sudo usysconf run -f if needed.

brent

laurker However, I am pretty much afraid of updating now, because about ¼ of the time I end up with an unbootable system

laurker . However the less frequently you update, the more likely that something goes wrong it seems, which puts me in a quandry.

this is no way to live with any distro. there's some unseen worm in the works (metaphorically, not malware-y) somewhere. a fresh 4.8 install is probably not in the interests of time available either. thanks for getting back, and also the nice words, good sir.
Speaking of Debian, I installed Lilidog with Openbox (yet another light Deb knockoff) on an old laptop today that I don't use much. between dpkg and apt and synaptic and the other two package manager (yes, that's 5) my head is spinning. But I need to choose just one eventually.