illumos gate

As a long-but-unique string is too unwieldy, I'm partial to 1.0 and 5.11 (for maximum redundancy).

Result is:
pkg:/consolidation/osnet/osnet-incorporation@0.5.11,5.11-1.0:...
pkg:/consolidation/osnet/osnet-incorporation@0.5.11,5.11-5.11:...

Dropping branch version numbers is another option worth considering going forward. Current pkg5 tools don't have any problems dealing with shorter version numbers. As an experiment I took current oi151 repo and removed -0.151 references from it and upgraded my system. This is how the package list looks like:

$ pkg list -v|head
FMRI                                                                         IFO
pkg://openindiana.org/SUNWcs@0.5.11,5.11:20110523T153305Z                        i--
pkg://openindiana.org/SUNWcsd@0.5.11,5.11:20110702T023648Z                       i--
pkg://openindiana.org/archiver/gnu-tar@1.23,5.11:20110523T143144Z                i--
pkg://openindiana.org/archiver/unrar@3.8.5,5.11:20110523T143150Z                 i--
pkg://openindiana.org/audio/audio-utilities@0.5.11,5.11:20110702T023710Z         i--
pkg://openindiana.org/benchmark/x11perf@1.5.1,5.11:20110523T152813Z              i--
pkg://openindiana.org/codec/flac@0.5.11,5.11:20110523T143155Z                    i--
pkg://openindiana.org/codec/libtheora@0.5.11,5.11:20110523T143153Z               i--
pkg://openindiana.org/codec/ogg-vorbis@0.5.11,5.11:20110523T143156Z              i--

Upgrading existing systems could be tricky, but not impossible.

Dropping the branch versions would make this harder, not easier. (we'd go from having one number to keep >= to having one per package).

I was thinking about relying on timestamps in case of illumos packages when doing upgrades and keeping individual package versions intact (they are set to 0.5.11 in most cases).

I think Andrzej's idea has merit and is worth considering, but post-oi151-stable.

For oi_151-stable, I don't mind which route we take, as long as the route doesn't preclude oi from dropping branch version numbers at a later date.

(The background here is that part of our future development process plan is to streamline things as much as possible, cutting out unnecessary steps especially in release engineering. My understanding is that dropping the branch number would let development become more piecemeal, and unless there is something we've missed, what branch version numbers are used for can probably be done well enough with incorporations - but this should probably be debated on another issue.)

I can't speak as to what OI want to do, but it's very likely that incorporations don't give you the behaviour you wish for in this case. Incorporation at the timestamp level is likely possible, but it would make generating the incorporation rather difficult, and we do have cases where lock-step upgrade of packages not only appropriate, but necessary (system/library and system/kernel, for eg). This also makes "package FOO didn't upgrade" a completely silent failure, and gets us back to people running random untested gloms of software. I would think about it carefully before going that route.

1. Meant to be allowed (I thought it wasn't)
2. Greater or less than a specified one (so it may still screw illumos)

I'm still in favour of giving illumos a large branch version, however, if you're willing to remove branch versions from your OI packages, surely you'd be similarly willing to leave illumos at 0.148 where we put it? The end results are identical as far as install and upgrade are concerned. This also, conveniently, wouldn't screw us (but would mean /dev-il users had nowhere to go)

I was only experimenting with things. I am not sure personally what would be the best approach to versioning. I kind of like flexibility lack of branch version numbers brings though.

Not having branch version number would make life easier for people building their own packages. Especially when using pkgdepend to resolve dependencies. Software would not need to re-republished with every release of the distribution. It is possible that in many cases the resulting packages will install fine even on Solaris 11!

Also, people working on a distribution could choose to not to rebuild/republish every single package when preparing new release. Existing packages could be used instead.

Making incorporations less restrictive is another area to explore. We could make them to be just empty stubs in the development branch of the distribution. We could potentially also remove reverse dependencies on the incorporations from the actual packages. It would make it easier for people to create their own 'backports' of never software without messing with incorporation packages.

To not to make it a one big mess we could have separate repositories for releases: pkg.o.o/release/1.0, pkg.o.o/release/2.0, etc. Separate repos would let us lock versions of available packages. Other benefits of separate repos include ability to archive old unsupported release repos and letting mirrors to mirror only ones that are current. Also no surprises when doing pkg image-update when release 2.0 comes out and you want to stay on version 1.0 and continue receiving security updates/bugfixes.

I don't think technically lack of branch version number will pose any problems. I have not checked the source but people working on pkg5 don't seem to use it all the time. Check the version numbers at the bottom of this message for example:

http://mail.opensolaris.org/pipermail/pkg-discuss/2011-June/027123.html

I am sure pkg5's solver can be modified to handle X-0.148 or X-0.151 to X version transition in a special way. And also I'm sure dropping branch version number from illumos-gate could be coordinated with illumos devs so they don't have to worry about it ever again. It would actually be possible to make illumos generate differently versioned packages in one go and publish then to the same repo without taking too much extra space. Is there a reason you'd like to keep -0.148 branch version specifically for illumos delivered bits by the way (other than letting current pkg5/onu to continue to work on new OI)?

Again, I am not sure this is the way to go forward and I am probably forgetting about something. Idea itself though makes a lot of sense to me at the moment.

Andrzej Szeszo wrote:

Not having branch version number would make life easier for people building their own packages. Especially when using pkgdepend to resolve dependencies. Software would not need to re-republished with every release of the distribution. It is possible that in many cases the resulting packages will install fine even on Solaris 11!

That's an artifact of the incorporations, nothing to do with package versioning. If you want that to be true, alter how you generate the incorporation packages to incorporate less specifically where appropriate.

Also, people working on a distribution could choose to not to rebuild/republish every single package when preparing new release. Existing packages could be used instead.

That's also the result of incorporations, not versioning.

Making incorporations less restrictive is another area to explore. We could make them to be just empty stubs in the development branch of the distribution. We could potentially also remove reverse dependencies on the incorporations from the actual packages. It would make it easier for people to create their own 'backports' of never software without messing with incorporation packages.

That's the right way to achieve all this, though there are packages that do have to be precisely in sync, so just removing them is a bad way to do it. That also makes it difficult to discover bugs.

Again, I am not sure this is the way to go forward and I am probably forgetting about something. Idea itself though makes a lot of sense to me at the moment.

Certainly not something we can use to fix the immediate problem.

Rich Lowe wrote:

Andrzej Szeszo wrote:

Not having branch version number would make life easier for people building their own packages. Especially when using pkgdepend to resolve dependencies. Software would not need to re-republished with every release of the distribution. It is possible that in many cases the resulting packages will install fine even on Solaris 11!

That's an artifact of the incorporations, nothing to do with package versioning. If you want that to be true, alter how you generate the incorporation packages to incorporate less specifically where appropriate.

Richard is right. We've always had software not build-number constrained by incorporations which have no problems installing (for example, the remnants of the opensolaris.org publisher, which has no incorporations at all).

Rich Lowe wrote:

Andrzej Szeszo wrote:

Not having branch version number would make life easier for people building their own packages. Especially when using pkgdepend to resolve dependencies. Software would not need to re-republished with every release of the distribution. It is possible that in many cases the resulting packages will install fine even on Solaris 11!

That's an artifact of the incorporations, nothing to do with package versioning. If you want that to be true, alter how you generate the incorporation packages to incorporate less specifically where appropriate.

Let's say I compile mbuffer on oi148 and use pkgdepend to generate dependencies when packaging it. The package will not install on oi151 because it depends on pkg:/system/library/math@0.5.11-0.148 and pkg:/system/library@0.5.11-0.148. The issue has nothing to do with incorporations. It is more about how pkgdepend works. If pkgdepend had an option to drop branch version numbers when generating/resolving the dependencies the issue could be worked around by using it.

Also, people working on a distribution could choose to not to rebuild/republish every single package when preparing new release. Existing packages could be used instead.

That's also the result of incorporations, not versioning.

Not exactly. Illumos packages built on oi148 will not install 100% on oi151 and vice versa (let's say we keep 0.148 branch version). pkgdepend is being used during the packaging phase. So, for example ipfilter package will depend on pkg:/runtime/perl-584@5.8.4-0.148 when built on oi148 but it will depend on pkg:/runtime/perl-510@5.10.0-0.151 when built on oi151. Some other packages will have branch versioned dependencies (@X.X-0.148 or @X.X-0.151) on expat, glib2, libxml2, libxslt, openssl, zlib and possibly few others which are not part of illumos. You could not really take packages generated on one release and use them on a new one with the current state of things.

Making incorporations less restrictive is another area to explore. We could make them to be just empty stubs in the development branch of the distribution. We could potentially also remove reverse dependencies on the incorporations from the actual packages. It would make it easier for people to create their own 'backports' of never software without messing with incorporation packages.

That's the right way to achieve all this, though there are packages that do have to be precisely in sync, so just removing them is a bad way to do it. That also makes it difficult to discover bugs.

In case of "stable" releases keeping stuff in sync would not be an issue if we had separate repos for them and were pushing only security updates/bugfixes to the repos.

In case of the development branch we could ask people to run pkg image-update before reporting any bugs. This is assuming that all pushes to the dev repo are synchonized and latest packages are always in sync.

Again, I am not sure this is the way to go forward and I am probably forgetting about something. Idea itself though makes a lot of sense to me at the moment.

Certainly not something we can use to fix the immediate problem.

Reverting the number back to 0.148 will not solve the problem completely either due to branch versioned dependencies on external libraries. I am assuming you would like to be able to exchange locally built repos with other people who are not necessarily running the same system as you do. For local onu use adding ONNV_BUILDNUM=151; export ONNV_BUILDNUM to the env file seems to be good enough workaround.

Andrzej Szeszo wrote:

Rich Lowe wrote:

Andrzej Szeszo wrote:

Not having branch version number would make life easier for people building their own packages. Especially when using pkgdepend to resolve dependencies. Software would not need to re-republished with every release of the distribution. It is possible that in many cases the resulting packages will install fine even on Solaris 11!

That's an artifact of the incorporations, nothing to do with package versioning. If you want that to be true, alter how you generate the incorporation packages to incorporate less specifically where appropriate.

Let's say I compile mbuffer on oi148 and use pkgdepend to generate dependencies when packaging it. The package will not install on oi151 because it depends on pkg:/system/library/math@0.5.11-0.148 and pkg:/system/library@0.5.11-0.148. The issue has nothing to do with incorporations. It is more about how pkgdepend works. If pkgdepend had an option to drop branch version numbers when generating/resolving the dependencies the issue could be worked around by using it.

'depend' relationships are >=. A piece of software (SFEllvm, for example) with
depend fmri=system/library@0.5.11,5.11-0.148 type=require@

is installable even though I have system/library@0.5.11,5.11-0.151:20110523T153839Z

Also, people working on a distribution could choose to not to rebuild/republish every single package when preparing new release. Existing packages could be used instead.

That's also the result of incorporations, not versioning.

Not exactly. Illumos packages built on oi148 will not install 100% on oi151 and vice versa (let's say we keep 0.148 branch version). pkgdepend is being used during the packaging phase. So, for example ipfilter package will depend on pkg:/runtime/perl-584@5.8.4-0.148 when built on oi148 but it will depend on pkg:/runtime/perl-510@5.10.0-0.151 when built on oi151. Some other packages will have branch versioned dependencies (@X.X-0.148 or @X.X-0.151) on expat, glib2, libxml2, libxslt, openssl, zlib and possibly few others which are not part of illumos. You could not really take packages generated on one release and use them on a new one with the current state of things.

The case of Perl is a result of the actual software requirements changing, the packaging is merely describing the requirements correctly here. The build version does affect the minimum build number necessary, though.

Albert Lee wrote:

'depend' relationships are >=. A piece of software (SFEllvm, for example) with
depend fmri=system/library@0.5.11,5.11-0.148 type=require@

Oh wow! Kill me. I thought that type=require dependencies were behaving like type=incorporate ones for some reason. Thanks for an explanation Albert! I kind of recall reading about the dependency types long time ago but must have forgotten about it.

Rich, regarding incorporations, branch version numbers and keeping packages in sync. How would you handle illumos update which requires system/kernel and system/library to be upgraded at the same time? Were you thinking about bumping branch version numbers for both packages and updating incorporation? Or maybe bumping the actual package versions from 0.5.11? Just curious.

Andrzej Szeszo wrote:

Rich, regarding incorporations, branch version numbers and keeping packages in sync. How would you handle illumos update which requires system/kernel and system/library to be upgraded at the same time? Were you thinking about bumping branch version numbers for both packages and updating incorporation? Or maybe bumping the actual package versions from 0.5.11? Just curious.

For illumos, I was thinking of just updating the branch numbers and the incorporation to match; seems more reasonable than updating the package versions with the branch version forever stuck at 0.148.

For OpenIndiana /hipster you need to update also PKGVERS_BRANCH.