Solus packaging for Python programs

palto42

I'm trying to package the python program borgmatic and have a question about the correct way to add the required Python libraries.

In the package.yml I have used the Python Actionable Macros and the package is build correctly, but the library dependencies defined in setup.py are not installed if I install the package with eopkg. Do I need to add this dependencies individually as rundeps or can this be done automatic (extracting from `setup.py')?

My current yml settings:

builddeps  :
    - pkgconfig(python3)
    - python-setuptools
build      : |
    %python3_setup
install    : |
    %python3_install

kyrios

palto42 You have to add them as rundeps

algent

palto42 From what I see upstream there are two run depenencies python-pykwalify and ruamel_yaml.
We don't have python-pykwalify in repo, so you have to package it.

Edit: In fact there are more:

python-colorama
python-docopt
python-pykwalify # not in repo
python-dateutil
pyyaml
python-requests
ruamel_yaml
ruamel-yaml-clib # not in repo

palto42

I started to make a package for ruamel_yaml as a first step since the version in the Solus repository is only Python 2 and quite outdated.
But the make fails during install with below error:

+ pip3 install . --root=/home/build/YPKG/root/python3-ruamel_yaml/install
Processing /home/build/YPKG/root/python3-ruamel_yaml/build/ruamel.yaml-0.16.10
Collecting ruamel.yaml.clib>=0.1.2 (from ruamel.yaml==0.16.10)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84308f56d0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/ruamel-yaml-clib/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84308bcd50>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/ruamel-yaml-clib/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84308bc4d0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/ruamel-yaml-clib/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84308bc690>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/ruamel-yaml-clib/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84308bcc90>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/ruamel-yaml-clib/
ERROR: Could not find a version that satisfies the requirement ruamel.yaml.clib>=0.1.2 (from ruamel.yaml==0.16.10) (from versions: none)
ERROR: No matching distribution found for ruamel.yaml.clib>=0.1.2 (from ruamel.yaml==0.16.10)
Command '['/bin/bash', '--norc', '--noprofile', '/tmp/ypkg-installzTtR63']' returned non-zero exit status 1
[Build] install failed

In package.yml I have install : pip3 install . --root=%installroot% like in current Solus repository for Python 2.7. The macro python3_install throws an error telling me that setup.py has no install section and I hae to use pip install.

In a Python3 virtual environment I could do a pip3 install ruamel.yaml.clibwithout issues, not clear why it doesn't seem to work in solbuild make process. Or do I need to create the ruamel.yaml.clib first and add it as build dependency?

algent

palto42 Yes you need to build first python-ruamel-yaml-clib copy it to local repo and then pacakge python-ruamel-yaml, because this have python-ruamel-yaml-clib as builddep I think. Didn't try this yet.
I suggest you to leave alone ruamel_yaml and make a new package instead as python-ruamel-yaml.

palto42

Tried pip3 in verbose mode install : pip3 install . --root=%installroot% -v, but still not clear why it fails.

Collecting ruamel.yaml.clib>=0.1.2 (from ruamel.yaml==0.16.10)
1 location(s) to search for versions of ruamel.yaml.clib:
* https://pypi.org/simple/ruamel-yaml-clib/
Getting page https://pypi.org/simple/ruamel-yaml-clib/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/ruamel-yaml-clib/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
Incremented Retry for (url='/simple/ruamel-yaml-clib/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8872ddef10>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/ruamel-yaml-clib/

algent

@palto42 Here you have the package.yml for missing packages.

palto42

algent Thanks a lot for your help!

Some more questions:

Are the builddeps: -pkgconfig(python3) -python-setuptools not required anymore for python packages?
Is there a package naming convention to replace dots with hyphen? The original name of the ruamel packages uses dots like ruamel.yaml.clib and you replaced it with ruamel-yaml-clib.

I didn't know the option --single-version-externally-managed but found a good explanation on Python for Formula Authors.

JacekJagosz

palto42 You don't need to list all dependencies if one of your dependencies relies on other one. Because e.g. python-ruamel-yaml needs python3, you don't need to mention it again.
Normally you do that by doing eopkg info <dependency name> and look at dependencies (and sometimes dependencies of dependencies) and you don't need to include those. Also reverse dependencies are helpful, because they rely on that dependency.

algent

palto42 Are the builddeps: -pkgconfig(python3) -python-setuptools not required anymore for python packages?

pkconfig(python3) - not always, if it is needed, solbuild will complain for missing Python.h for example.
python-setuptools - is needed for python2 packages.

palto42 Is there a package naming convention to replace dots with hyphen? The original name of the ruamel packages uses dots like ruamel.yaml.clib and you replaced it with ruamel-yaml-clib.

I prefer using hyphen, and add python- to the python packages. No dots and underlines.

Different distros use different names for example.
Arch - python-ruamel.yaml.clib and python-ruamel-yaml
Fedora - python-ruamel-yaml-clib and python-ruamel-yaml

Check this links and you will get an idea.
python-ruamel.yaml.clib
python-ruamel-yaml-clib

palto42

I have managed to build borgmatic and all missing dependencies locally and the installation via eopkg worked 😀 . Thanks all for your support!

Now I have submitted a package request T9241 to add it to the Solus repository.

algent

palto42 can you share your package.yml for borgmatic? This is mine

palto42

algent This is mine, see some small differences:

You don't have the runtime dependency for ruamel.yaml (because pykwalify has it)
You added installations of the systemd and cron samples

https://gist.github.com/palto42/c4b2440ea8ae0bd8b037dc5d64fd4e58

davidjharder

@palto42 Thanks for taking a run at this. I just found out about Borgmatic and it looks promising

palto42

Updated my above package.yaml:

removed dependency python-ruamel.yaml (indirect dependency via pykwalify)
install README, CHANGELOG, cron example and systemd example to usr/share/doc/borgmatic

kyrios

palto42 install README, CHANGELOG, cron example and systemd example to usr/share/doc/borgmatic

Doesn't make sense to provide the cron example since Solus doesn't provide cron. Same for the changelog, it is useless to add it in the package. Since this info is available in the package history.

palto42

kyrios Agree that cron example is useless for Solus.
But how do I get the changelog of borgmatic into the package history? Currently it only shows "Packaging update" in my build.
One thing I noticed is that during make I get a warning WARN[17:39:08] Failed to obtain package git history error="No usable git history found". Currently this is just a local build project on my local git (Gitea), is this the reason why it ignores the git log?

kyrios

palto42 Because when a package is submitted for review, you have to provide a summary in the git message. This summary is usually a summarized changelog and this is what you see changelog tab of the packages in the software center.

palto42

kyrios Thanks for your help and patience!
Do I correctly assume that the package information is finally coming from the arc diff commit message (is remarkup format?) and not from my local git commit messages?
So I need to add changelog information in the arc diffcommit message, either by copy & paste from the program's changelog and/or by adding a link reference to the changelog (e.g. on source code page).

algent

palto42 No, you don't have to add changelog because this is a new package. Changelogs are required when you submit an update. Your case should be something similar to this package.

kyrios

palto42 Exactly 🙂