Project

General

Profile

Bug #12503

pkg runs for hours then crashes the system (memory leak)

Added by Majik Bear 7 months ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:

Description

Freshly installed:

uname -rosv

SunOS 5.11 illumos-87d7b64204 illumos

Installed in virtualbox.

Running pkg update or pkg image-update causes the little spinny wheel to spin forever until the system runs out of memory and crashes.

#1

Updated by giahung 1997tn 5 months ago

Majik Bear wrote:

Freshly installed:

uname -rosv

SunOS 5.11 illumos-87d7b64204 illumos

Installed in virtualbox.

Running pkg update or pkg image-update causes the little spinny wheel to spin forever until the system runs out of memory and crashes.

I used to have the same problem, too. Sometimes it works, sometimes it not. pkg is very unstable.

#2

Updated by Aurélien Larcher 5 months ago

PKG requires a lot of memory to resolve dependencies, if your VM has less than 8GB then you may run into issues for large updates: there is most likely no issue with any memory leak, you are just swapping.

The memory footprint is due to the use of Python dict() to feed the SAT solver.

#3

Updated by giahung 1997tn 5 months ago

Aurélien Larcher wrote:

PKG requires a lot of memory to resolve dependencies, if your VM has less than 8GB then you may run into issues for large updates: there is most likely no issue with any memory leak, you are just swapping.

The memory footprint is due to the use of Python dict() to feed the SAT solver.

My VM only has 4G, and the host has 8G. So this is problem. BTW, I found limiting ZFS ARC MAX cache to 512M really helps. I have not yet have out of memory problem with pkg since then.

#4

Updated by Majik Bear 5 months ago

This is a fundamental problem because a majority of VM users are going to install your distro and then try to do the most basic thing you can do with an OS based on a package manager, which is update their packages, and it's going to fail.

I shouldn't need 8GB of RAM for a package manager.

This is not a problem on other desktop unix-like operating systems. Debian, CentOS and Ubuntu don't have it.

The mission of this distro is to be a desktop-friendly BSD distribution.

If I can't upgrade my software right after a fresh intall without getting into terminal hell that's not very friendly.

#5

Updated by Aurélien Larcher 5 months ago

The difference lies in what the package manager is responsible for...

APT or RPM are basically just fetching tarballs with metadata, that's why they do not require much work... then you pay the cost of pre/post scripts so the install/update process does not scale well with the number of packages.
PKG is like a big GIT for the entire system that tracks files, users, services, and deals with containers (zones).
It is a problem that the SAT solver takes so much memory, partly because of the implementation in Python and the fact that our repository has grown too much, cleanup is needed.
The update of PKG to use Python 3.7/3.8 helps with the memory consumption but in the short term there is not much we can change without reworking the low level implementation...
Remember that this package manager was developed for servers not user desktops so design choices have been made in consequences and we need to deal with them...

Also available in: Atom PDF