Project

General

Profile

Bug #13445

Faulty example in pkgrecv(1)

Added by Gary Mills 9 days ago. Updated 3 days ago.

Status:
New
Priority:
Normal
Assignee:
OI Userland
Category:
OI-Userland
Target version:
2020.10
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:

Description

The existing example in the pkgrecv man page reads like this:

       Example 10 Change publisher name

       Change the publisher name of the package 'foo' and all its dependencies
       into 'extra' during republishing.

         $ echo '<transform set name=pkg.fmri -> edit value
           (pkg://).*?(/.*) \1extra\2>' | \
           pkgrecv -s repo1 -d repo2 --mog-file - foo

Trying this on OI produces this error:

Processing packages for publisher userland ...
Retrieving and evaluating 1 package(s)...
pkgrecv: 'None' is not a valid publisher name.

There are at least two errors in this example. One is that backreferences in transforms do not work, probably as a result of the upgrade to python3.5 . Another is that a package rebuild is required immediately after the example, for new packages to appear in the repository. With those two problems worked around, the example actually succeeds. The workaround for the backreference error is to avoid using backreferences. This solution is already used in OI. The correct solution is to fix pkgrecv so that backreferences work once again. The interm solution is to change the example in the man page so that it at least works.

#1

Updated by Andy Fiddaman 9 days ago

The following works.

I agree the example in the man page needs updating; perhaps this would do for now?

% pkgrepo -s /tmp/bob create
% echo '<transform set name=pkg.fmri value=(pkg://).*?(/.*) -> set value %<1>extra%<2>>' | \
           pkgrecv -s https://pkg.omnios.org/bloody/core -d /tmp/bob --mog-file - screen
Processing packages for publisher omnios ...
Retrieving and evaluating 1 package(s)...
PROCESS                                         ITEMS    GET (MB)   SEND (MB)
Completed                                         1/1     0.6/0.6     0.6/0.6

bloody:illumos:ksh% pkgrepo -s /tmp/bob rebuild
Initiating repository rebuild.
bloody:illumos:ksh% pkgrepo -s /tmp/bob list
PUBLISHER NAME                                          O VERSION
extra     terminal/screen                                 4.8.0-151037.0:20210107T163112Z
#2

Updated by Andy Fiddaman 9 days ago

It's something to do with escaping. Add enough \ characters and the example works:

% echo '<transform set name=pkg.fmri -> edit value \
           (pkg://).*?(/.*) \\\\1extra\\\\2>' | \
           pkgrecv -s https://pkg.omnios.org/bloody/core -d /tmp/bob --mog-file - tmux
Processing packages for publisher omnios ...
Retrieving and evaluating 1 package(s)...
PROCESS                                         ITEMS    GET (MB)   SEND (MB)
Completed                                         1/1     0.5/0.5     0.5/0.5
#3

Updated by Gary Mills 7 days ago

There was a change in recent versions of python that affects regular expressions. They split the string type into two types: cooked strings and raw strings. If regular expressions are read into cooked strings, all of the backslash escapes, including backreferences, fail to work. There are many complaints about this new behavior on the web. I assume that's what happened to the pkg commands that use python-style backreferences.

I accidentally discovered another way to make backreferences as shown in the man page example work: Simply enclose the replacement portion of the transform in single quotes. Here's the transform I've been using:

<transform set name=pkg.fmri -> edit value (pkg://).*?(/.*) '\1oi-userland\2'>

I still don't like it because it's too complex and thus too error-prone. Writing the transform without using backreferences is easier and simpler. OI does it that way.

The difficult part, at least for me, is to introduce the man page changes into the OI packages. As far as I can tell, the Makefile obtains all the binaries and man pages from another consolidation and changes the publisher and some other package meta-data. I don't know how a content change can be added.

#4

Updated by Andy Fiddaman 6 days ago

It doesn't seem to be related to the python upgrade, actually.

The IPS code uses the shlex module and the same thing happens with both python 2 and python 3:

Python 3.9.1 (default, Jan  7 2021, 16:20:35)
[GCC 10.2.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import shlex
>>> a = 'value (pkg://).*?(/.*) \\1extra\\2'
>>> a
'value (pkg://).*?(/.*) \\1extra\\2'
>>> shlex.split(a)
['value', '(pkg://).*?(/.*)', '1extra2']

Python 2.7.18 (default, Jan  7 2021, 16:00:02)
[GCC 10.2.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import shlex
>>> a = 'value (pkg://).*?(/.*) \\1extra\\2'
>>> a
'value (pkg://).*?(/.*) \\1extra\\2'
>>> shlex.split(a)
['value', '(pkg://).*?(/.*)', '1extra2']

I looked at the history in the repository, and compared against the Solaris IPS sources and cannot see how this can ever have worked without extra escaping with quotes or backslashes.

In terms of updating the man page, it should just be a matter of opening a pull request against these files here:

https://github.com/OpenIndiana/pkg5/tree/oi/src/man

#5

Updated by Gary Mills 3 days ago

I now have created a pull request for the pkg5 repository. It's PR number 89. The new example in the man page is this:

     Example 10 Change publisher name

     Change the publisher name of the package 'foo' and  all  its
     dependencies  into  'extra' during republishing.  Backrefer-
     ences can also be used in  the  replacement  string  of  the
     transform but they are complex and error-prone.

       $ echo '<transform set name=pkg.fmri -> edit value
         pkg://[^/]+/ pkg://extra/>' > /tmp/x.tr
       $ pkgrecv -s repo1 -d repo2 --mog-file /tmp/x.tr foo
       $ pkgrepo rebuild -s repo2

Also available in: Atom PDF