Bug #13445
Faulty example in pkgrecv(1)
0%
Description
The existing example in the pkgrecv man page reads like this:
Example 10 Change publisher name
Change the publisher name of the package 'foo' and all its dependencies
into 'extra' during republishing.
$ echo '<transform set name=pkg.fmri -> edit value
(pkg://).*?(/.*) \1extra\2>' | \
pkgrecv -s repo1 -d repo2 --mog-file - foo
Trying this on OI produces this error:
Processing packages for publisher userland ... Retrieving and evaluating 1 package(s)... pkgrecv: 'None' is not a valid publisher name.
There are at least two errors in this example. One is that backreferences in transforms do not work, probably as a result of the upgrade to python3.5 . Another is that a package rebuild is required immediately after the example, for new packages to appear in the repository. With those two problems worked around, the example actually succeeds. The workaround for the backreference error is to avoid using backreferences. This solution is already used in OI. The correct solution is to fix pkgrecv so that backreferences work once again. The interm solution is to change the example in the man page so that it at least works.
Updated by Andy Fiddaman 3 months ago
The following works.
I agree the example in the man page needs updating; perhaps this would do for now?
% pkgrepo -s /tmp/bob create
% echo '<transform set name=pkg.fmri value=(pkg://).*?(/.*) -> set value %<1>extra%<2>>' | \
pkgrecv -s https://pkg.omnios.org/bloody/core -d /tmp/bob --mog-file - screen
Processing packages for publisher omnios ...
Retrieving and evaluating 1 package(s)...
PROCESS ITEMS GET (MB) SEND (MB)
Completed 1/1 0.6/0.6 0.6/0.6
bloody:illumos:ksh% pkgrepo -s /tmp/bob rebuild
Initiating repository rebuild.
bloody:illumos:ksh% pkgrepo -s /tmp/bob list
PUBLISHER NAME O VERSION
extra terminal/screen 4.8.0-151037.0:20210107T163112Z
Updated by Andy Fiddaman 3 months ago
It's something to do with escaping. Add enough \ characters and the example works:
% echo '<transform set name=pkg.fmri -> edit value \
(pkg://).*?(/.*) \\\\1extra\\\\2>' | \
pkgrecv -s https://pkg.omnios.org/bloody/core -d /tmp/bob --mog-file - tmux
Processing packages for publisher omnios ...
Retrieving and evaluating 1 package(s)...
PROCESS ITEMS GET (MB) SEND (MB)
Completed 1/1 0.5/0.5 0.5/0.5
Updated by Gary Mills 3 months ago
There was a change in recent versions of python that affects regular expressions. They split the string type into two types: cooked strings and raw strings. If regular expressions are read into cooked strings, all of the backslash escapes, including backreferences, fail to work. There are many complaints about this new behavior on the web. I assume that's what happened to the pkg commands that use python-style backreferences.
I accidentally discovered another way to make backreferences as shown in the man page example work: Simply enclose the replacement portion of the transform in single quotes. Here's the transform I've been using:
<transform set name=pkg.fmri -> edit value (pkg://).*?(/.*) '\1oi-userland\2'>
I still don't like it because it's too complex and thus too error-prone. Writing the transform without using backreferences is easier and simpler. OI does it that way.
The difficult part, at least for me, is to introduce the man page changes into the OI packages. As far as I can tell, the Makefile obtains all the binaries and man pages from another consolidation and changes the publisher and some other package meta-data. I don't know how a content change can be added.
Updated by Andy Fiddaman 3 months ago
It doesn't seem to be related to the python upgrade, actually.
The IPS code uses the shlex module and the same thing happens with both python 2 and python 3:
Python 3.9.1 (default, Jan 7 2021, 16:20:35) [GCC 10.2.0] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import shlex >>> a = 'value (pkg://).*?(/.*) \\1extra\\2' >>> a 'value (pkg://).*?(/.*) \\1extra\\2' >>> shlex.split(a) ['value', '(pkg://).*?(/.*)', '1extra2']
Python 2.7.18 (default, Jan 7 2021, 16:00:02) [GCC 10.2.0] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import shlex >>> a = 'value (pkg://).*?(/.*) \\1extra\\2' >>> a 'value (pkg://).*?(/.*) \\1extra\\2' >>> shlex.split(a) ['value', '(pkg://).*?(/.*)', '1extra2']
I looked at the history in the repository, and compared against the Solaris IPS sources and cannot see how this can ever have worked without extra escaping with quotes or backslashes.
In terms of updating the man page, it should just be a matter of opening a pull request against these files here:
Updated by Gary Mills 3 months ago
I now have created a pull request for the pkg5 repository. It's PR number 89. The new example in the man page is this:
Example 10 Change publisher name
Change the publisher name of the package 'foo' and all its
dependencies into 'extra' during republishing. Backrefer-
ences can also be used in the replacement string of the
transform but they are complex and error-prone.
$ echo '<transform set name=pkg.fmri -> edit value
pkg://[^/]+/ pkg://extra/>' > /tmp/x.tr
$ pkgrecv -s repo1 -d repo2 --mog-file /tmp/x.tr foo
$ pkgrepo rebuild -s repo2