Automatic kernel update constantly fails with no errors

nolan

[deleted] Enlarge your EFI partition to be at least 1GB

mine is less than 512M and it works well.

@mck does sudo eopkg it --reinstall linux-current works for you?

mck

nolan Currently I am on the latest linux-current kernel so I don't know how I suppose to test it. Also I do not get any errors from eopkg during update. For me it looks like some post upgrade operations with kernel are simply failing with no errors.

Staudey

mck This command also does NOT work:
sudo clr-boot-manager update

When you say it doesn't work, do you mean you get errors when running it? Warnings? Or does it complete without any of those, but the results are not as expected?

mck

Staudey Sorry, description is not precise. The command sudo clr-boot-manager update executes without any errors and the result is like it does nothing.

TraceyC Thanks for the info! You are right, the requirement of having at least 1GB boot partition appeared later. I remember 512MB as suggested boot partition size at that time in the docs.

Looks like I will gonna have to create partitions again on this SSD. Increasing the boot partition and decreasing the main file system partition probably will take a lot of time or can be error prone unless I am wrong.

Thank you very much for your suggestions.

TraceyC

Due to how we build kernels, we (Solus Staff) were forced to make the decision to not support EFI boot partitions < 1GB. This is because failures can happen when installing new kernels if the EFI partition runs out of space.

The official guidance is to ensure that the system's EFI partition is a minimum of 1GiB as @[deleted] pointed out.
If you run your system with an EFI partition of less than 1GB, you may be OK or you may run into issues.

Reinstalling the kernel package is not going to solve this issue. Per the original post, this has persisted through many kernel updates. Reinstalling the kernel package again would only produce the same result.

@mck ,
I would recommend increasing the size of your boot partition, and see if that solves the problem with the next kernel update. From your original post, the problem goes back before the EFI partition needed to be 1G in size, so I can't guarantee this will be all that's needed. It's the necessary first step, however. You'll need to do this from a live environment, not from the booted system.

Please let us know how things work out for you.

Justin

There are a few ways to do it. As the EFI System Partition is usually at the front of the disc, resizing can be dangerous and is time confusing. You can create a new EFI System Partition at the end of the disk which is a lot faster but is a little involved technically.

It depends on how much data you have to backup for a reinstall.

mck

Justin Thanks for the confirmation of my concerns. I am a technical person so no problem in this area. The only thing I took into account was the time factor. There is not a lot of data to backup so I will just remove all partitions and re-create with the 1GB EFI boot partition and bigger swap since I have more RAM memory now.

I will let you know later about the results of this operation in this topic.

Thanks @Justin for your suggestion.

mck

Hello Guys,

Time for some update from my side.

I created new partition table with the following partition layout:

sudo fdisk -o Device,Size,Type -l /dev/sda
Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 860 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: <ID>

Device       Size Type
/dev/sda1      1G EFI System
/dev/sda2      8G Linux swap
/dev/sda3  406.8G Linux filesystem

I installed Solus 4.5 Budgie Edition successfully, however distro is not booting. Previous installation of Solus 4.2 (Budgie) around 2021 went well back then. My BIOS is UEFI compatible. I also lost SSD boot entry in BIOS which I had with previous Solus installation. From what I understand Solus during installation didn't set correctly UEFI boot entry. Correct me if I am wrong.

Weird thing is that during installation I have got info dialog with message about possibility to create 8MB partition for grub-boot, maybe installer detected my setup as Legacy BIOS? 🤔

Anyway I would like to ask you to send me your output from efibootmgr command from the latest Solus. Thanks to this I will create UEFI boot entry manually and see if it helps.

Here is content of my boot EFI partition, looks a little bit weird to me:

ls -la /target/boot/
total 72868
drwxr-xr-x  3 root root     4096 Jan  1  1970 .
drwxr-xr-x 19 root root     4096 Mar 11 16:39 ..
-rwxr-xr-x  1 root root 10946352 Mar 11 17:39 com.solus-project.current.6.6.9-269
drwxr-xr-x  4 root root     4096 Mar 11 17:39 grub
-rwxr-xr-x  1 root root 63652875 Mar 11 17:39 initrd-com.solus-project.current.6.6.9-269

P.S. Operations from https://help.getsol.us/docs/user/troubleshooting/boot-rescue docs didn't help.

If you have any ideas what happened just let me know.

UPDATE

It turned out the solution was simpler than I thought. After mounting my installed system according to https://help.getsol.us/docs/user/troubleshooting/boot-rescue#mounting-your-system docs and chrooting to installed system according to https://help.getsol.us/docs/user/troubleshooting/boot-rescue#chrooting-to-your-solus-system docs finally the clr-boot-manager list-kernels command started to work.

The final solution was to simply run in chrooted system:

clr-boot-manager update

After that my boot EFI partition content has been changed to:

ls -la /boot/
razem 24
drwxr-xr-x  5 root root 4096 1970-01-01  .
drwxr-xr-x 20 root root 4096 03-12 11:02 ..
drwxr-xr-x  4 root root 4096 03-12 10:56 EFI
drwxr-xr-x  4 root root 4096 03-11 18:39 grub
drwxr-xr-x  3 root root 4096 03-12 11:22 loader
-rwxr-xr-x  1 root root 1321 03-12 11:21 solus-enroll-me.cer

My Solus installation has been correctly registered in efibootmgr and BIOS boot menu entry as Solus Linux Bootloader. Solus booted immediately.

I was able to upgrade all packages to the latest versions without any errors. The linux kernel finally updated automagically as it should.

Few changes I spotted after installation:

The boot EFI partition is now automagically mounted which was not a case in the previous installation from 2021.
As the result of the previous point bootctl list command now correctly lists all boot entries. Previously I had to mount boot filesystem manually by running sudo clr-boot-manager mount-boot to list them.

Conclusions

After performing this installation I have a suggestion for developers/maintainers to update https://help.getsol.us/docs/user/troubleshooting/boot-rescue documentation by adding information:

Running the following command in chrooted system fixes broken UEFI boot entry for Solus after installation:

clr-boot-manager update

The question is why it happened to me (looks like a bug). After some research I found out that other people had similar problems but trying to solve them manually with efibootmgr which maybe not always is the best way. This is something for developers to investigate.

brent

mck this probably explains my 50% success rate in the chroot environment as I've repaired boot.
"sudo eopkg rebuild-db sudo eopkg up sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall If you see output that starts with "Usage", this means the system has no broken packages."

^^ this didn't always work if I wrecked boot, but half the time it did.
It's nice to have clr-boot-manager update this one in my arsenal as well.
Thanks for your diligence.

"Previously I had to mount boot filesystem manually by running sudo clr-boot-manager mount-update to list them."
(I don't remember doing it this way).

TraceyC

@mck Thanks for keeping us updated, and for all the detail you've provided. I've opened a pull request to add your suggestion of regenerating the EFI menu in the chroot environment.

mck

TraceyC That is great news for me! Thank you for taking my suggestion into consideration.

brent Hope it will help also other users to save their time and their Solus 😄

In the last message I made a typo in the command which is already fixed. To mount boot filesystem manually I was using:

sudo clr-boot-manager mount-boot   // this command is not documented, I found it somewhere

The above command is probably equivalent of the following command:

mount /dev/sda1 /boot

where /dev/sda1 is boot EFI partition.

TraceyC

mck

I just wanted to clarify that the command sudo clr-boot-manager mount-boot is indeed documented, and in a few places. It's easy to find by searching the User docs in the Help Center

mck

I apologize for my absence for some time 😄

TraceyC I meant specifically a mount-boot option of this command which I still can't find documented in the link provided by you:

Maybe I am doing something wrong? 🤔

This option was also not documented in the past in the command help accessible from the terminal, but when I did the sync today I actually saw it documented, which is something new for me:

krzysztof@krzysztof-solus ~ $ clr-boot-manager --help
Usage: clr-boot-manager

        version - Print the version and quit
  report-booted - Report the current kernel as successfully booted
  remove-kernel - Remove the kernel from the system
     mount-boot - Mount the boot directory
           help - Show help message
         update - Perform post-update configuration of the system
    set-timeout - Set the timeout to be used by the bootloader
    get-timeout - Get the timeout to be used by the bootloader
     set-kernel - Configure kernel to be used at next boot
   list-kernels - Display currently selectable kernels to boot

Options:
  -p, --path           Set the base path for boot management operations.
  -i, --image          Force clr-boot-manager to run in image mode.
  -n, --no-efi-update  Don't update efi vars when using shim-systemd backend.

Also big thanks @TraceyC for updating docs Regenerate UEFI Boot Entries 👍️ 💓

Anyway I just wanted to return to the main topic which is expected automatic kernel update. Unfortunately after today's sync (week 11) I noticed that the problem still persists:

krzysztof@krzysztof-solus ~ $ sudo clr-boot-manager list-kernels
* com.solus-project.current.6.6.21-280
  com.solus-project.current.6.6.20-279

As visible above the command reports the current kernel as 6.6.21-280, which is not true as visible below:

krzysztof@krzysztof-solus ~ $ uname -a
Linux krzysztof-solus 6.6.20-279.current #1 SMP PREEMPT_DYNAMIC 0 x86_64 GNU/Linux

Running sudo clr-boot-manager update executes with no errors/warnings and does nothing.

List from bootctl command:

krzysztof@krzysztof-solus ~ $ sudo bootctl list 
         type: Boot Loader Specification Type #1 (.conf)
        title: Solus 4.5 Resilience (Solus-current-6.6.21-280.conf)
           id: Solus-current-6.6.21-280.conf
       source: /boot//loader/entries/Solus-current-6.6.21-280.conf
        linux: /boot//EFI/com.solus-project/kernel-com.solus-project.current.6.6.21-280
       initrd: /boot//EFI/com.solus-project/initrd-com.solus-project.current.6.6.21-280
      options: root=PARTUUID=<UUID> quiet splash systemd.show_status=false rw

         type: Boot Loader Specification Type #1 (.conf)
        title: Solus 4.5 Resilience (Solus-current-6.6.20-279.conf) (default) (selected)
           id: Solus-current-6.6.20-279.conf
       source: /boot//loader/entries/Solus-current-6.6.20-279.conf
        linux: /boot//EFI/com.solus-project/kernel-com.solus-project.current.6.6.20-279
       initrd: /boot//EFI/com.solus-project/initrd-com.solus-project.current.6.6.20-279
      options: root=PARTUUID=<UUID> quiet splash systemd.show_status=false rw

         type: Automatic
        title: EFI Default Loader
           id: auto-efi-default
       source: /sys/firmware/efi/efivars/LoaderEntries-<UUID>

         type: Automatic
        title: Reboot Into Firmware Interface
           id: auto-reboot-to-firmware-setup
       source: /sys/firmware/efi/efivars/LoaderEntries-<UUID>

Any ideas?

TraceyC

mck

Sorry to hear that clr-boot-manager is still not switching to the latest kernel after updating the system. At this point, you'll probably get more help from the Clear Linux forums. They develop clr-boot-manager.

I found this thread that describes the exact scenario you're running into. See if the solution there works for you, and keep us updated.

mck

TraceyC The solution suggested in the thread you linked unfortunately is not the same case as mine.

In my case the EFI partition is already mounted by default at the /boot mount point:

After unmounting the EFI partition /boot folder is empty:

According to the mentioned solution people already had some directories/files in the /boot path that prevented the /boot path from being used as a mount point for an EFI partition. I also found another thread that explains the same problem in details here. Unfortunately as mentioned earlier this is not my case.

I will try to spend more time on this to search for a solution but this is very weird that because of some reason clr-boot-manager does not work only for me 🤔

Output for debugging purposes:

krzysztof@krzysztof-solus ~ $ sudo CBM_DEBUG=1 clr-boot-manager update
[DEBUG] cbm (../src/cli/cli.c:L142): No such file: //etc/kernel/update_efi_vars
[INFO] cbm (../src/bootman/bootman.c:L785): Current running kernel: 6.6.22-281.current
[INFO] cbm (../src/bootman/sysconfig.c:L179): Discovered UEFI ESP: /dev/disk/by-partuuid/fd85b8cc-d5e6-4043-bd97-879bd3e8a821
[INFO] cbm (../src/bootman/sysconfig.c:L256): Fully resolved boot device: /dev/sda1
[DEBUG] cbm (../src/bootman/bootman.c:L141): shim-systemd caps: 0x26, wanted: 0x26
[DEBUG] cbm (../src/bootman/bootman.c:L156): UEFI boot now selected (shim-systemd)
[INFO] cbm (../src/bootman/bootman.c:L805): path ///etc/kernel/initrd.d does not exist
[INFO] cbm (../src/bootman/bootman.c:L805): path ///usr/lib/initrd.d does not exist
[INFO] cbm (../src/bootman/bootman.c:L501): Checking for mounted boot dir
[INFO] cbm (../src/bootman/bootman.c:L508): boot_dir is already mounted: /boot
[DEBUG] cbm (../src/bootman/update.c:L164): Now beginning update_native
[DEBUG] cbm (../src/bootman/update.c:L173): update_native: 2 available kernels
[DEBUG] cbm (../src/bootman/update.c:L191): update_native: Running kernel is (current) ///usr/lib/kernel/com.solus-project.current.6.6.22-281
[SUCCESS] cbm (../src/bootman/update.c:L205): update_native: Bootloader updated
[SUCCESS] cbm (../src/bootman/update.c:L220): update_native: Repaired running kernel ///usr/lib/kernel/com.solus-project.current.6.6.22-281
[DEBUG] cbm (../src/bootman/update.c:L230): update_native: Checking kernels for type current
[INFO] cbm (../src/bootman/update.c:L243): update_native: Default kernel for type current is ///usr/lib/kernel/com.solus-project.current.6.6.22-281
[SUCCESS] cbm (../src/bootman/update.c:L255): update_native: Installed tip for current: ///usr/lib/kernel/com.solus-project.current.6.6.22-281
[SUCCESS] cbm (../src/bootman/update.c:L269): update_native: Installed last_good kernel (current) (///usr/lib/kernel/com.solus-project.current.6.6.22-281)
[DEBUG] cbm (../src/bootman/update.c:L280): update_native: Analyzing for type current: ///usr/lib/kernel/com.solus-project.current.6.6.22-281
[DEBUG] cbm (../src/bootman/update.c:L285): update_native: Skipping running kernel
[DEBUG] cbm (../src/bootman/update.c:L280): update_native: Analyzing for type current: ///usr/lib/kernel/com.solus-project.current.6.6.20-279
[INFO] cbm (../src/bootman/update.c:L288): update_native: not-running: ///usr/lib/kernel/com.solus-project.current.6.6.20-279
[INFO] cbm (../src/bootman/update.c:L296): update_native: not-default-current: ///usr/lib/kernel/com.solus-project.current.6.6.20-279
[INFO] cbm (../src/bootman/update.c:L304): update_native: not-last-booted: ///usr/lib/kernel/com.solus-project.current.6.6.20-279
[INFO] cbm (../src/bootman/update.c:L313): update_native: Proposed for deletion from current: ///usr/lib/kernel/com.solus-project.current.6.6.20-279
[INFO] cbm (../src/bootman/bootman.c:L501): Checking for mounted boot dir
[INFO] cbm (../src/bootman/bootman.c:L508): boot_dir is already mounted: /boot
[SUCCESS] cbm (../src/bootman/update.c:L338): update_native: Default kernel for current is ///usr/lib/kernel/com.solus-project.current.6.6.22-281
[INFO] cbm (../src/bootman/update.c:L360): update_native: Garbage collecting current: ///usr/lib/kernel/com.solus-project.current.6.6.20-279

This looks like something is not correct here: 🤔

[DEBUG] cbm (../src/cli/cli.c:L142): No such file: //etc/kernel/update_efi_vars

I found another old thread on our forum that describes the same problem as mine, so I decided to link it here for other users to keep looking in current thread for any updates.

TraceyC

mck I found another old thread on our forum that describes the same problem as mine, so I decided to link it here for other users to keep looking in current thread for any updates.

The other thread describes the same issue, but had a different cause. That system had different contents in /boot when it was mounted vs. unmounted.

In your case, the /boot partition is empty when it is unmounted, which is what's expected. I suggest searching through the Clear Linux forums. Clear Linux develops clr-boot-manager.