Project

General

Profile

Bug #5468

Missing dependencies in lib/Makefile

Added by Gary Mills almost 5 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
lib - userland libraries
Start date:
2014-12-21
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:

Description

I'm doing this build on a T2000, which has 32 virtual CPUs. I've seen dozens of dmake processes running simultaneously. One result of this parallelism is that some libraries are linked with other libraries on the build host because these other libraries have not been built and installed yet in the proto area. Often you can't tell that this has happened when the build host is also an illumos system. I was building on a Solaris 11.2 system, making it obvious because of a new symbol.

I've noticed that many libraries are missing dependancies on other libraries in lib/Makefile . Adding some of them was an improvement, but I never knew that I got all of them. Something that would limit library search to specified paths would be a better way of finding what's missing.


Files

mf.diff (1.82 KB) mf.diff Gary Mills, 2014-12-22 04:47 PM
elf.err (5.07 KB) elf.err Gary Mills, 2015-02-22 08:59 PM

Related issues

Related to illumos gate - Bug #729: wsdiff should ignore .SUNW_dofNew2011-02-16

Actions
Related to illumos gate - Bug #5465: CFLAGS missing from BUILD.SO in lib/Makefile.libClosed2014-12-21

Actions
Related to illumos gate - Bug #5837: libdtrace audit library build missing LDLIBSClosed2015-04-14

Actions

History

#1

Updated by Gary Mills almost 5 years ago

  • File mf.diff mf.diff added
  • Status changed from New to Feedback
  • % Done changed from 0 to 20

I'm attaching a patch file that shows the dependancies I've added to the Makefile. I only fixed some targets, though. Likely I've missed many more.

#2

Updated by Rich Lowe almost 5 years ago

At one point I had patches that made this better, and were likely very similar to yours (I didn't look).
All I did was grep the Makefiles for '-l' and state the dependencies so discovered.

I'd say it's totally worth fixing these piecemeal, if you wanted, as long as the bug made it clear there were still problems remaining

#3

Updated by Andrew Stormont almost 5 years ago

A few that you've missed:

- libdladm depends on libpool
- librestart depends on libpool
- libdtrace depends on libelf, libmapmalloc
- libmapmalloc depends on libc
- libmp depends on libc
- libkvm depends on libc, libelf
- libtnfctl depends on libc, libelf
- libelfsign depends on libelf, libmd

Also in your patch you have two copyright notices.

#4

Updated by Gary Mills almost 5 years ago

I believe that it's not necessary to include libc in dependancy lists of this file; there is an explicit .WAIT target to ensure that libc is built first.

The first Copyright notice is historical, from a previous change I made. Should I have deleted it?

#5

Updated by Andrew Stormont almost 5 years ago

Without the libc dependency it seems to be pulling in the host version of these libraries. I'm also building on a machine with 32 logical cores (16 core Xeon with hyper threading).

Generally we update the copyright notices instead of adding a new one for each year - though there are times when you see a copyright notice span multiple years - e.g "Copyright 2014-2014 Andrew Stormont". I pointed it out because it looked like a mistake.

#6

Updated by Andrew Stormont almost 5 years ago

  • Priority changed from Normal to High

The problem is actually worse than originally thought. Any time a make target uses $(LINK.c) but does not pass $(LDLIBS) we are linking against libraries outside the proto area. I think it's about time illumos started passing -zfatal-warnings and -zassert-deflibs in the default LDFLAGS to catch these and other linker errors.

#7

Updated by Andrew Stormont almost 5 years ago

  • Subject changed from Missing dependancies in lib/Makefile to Missing dependencies in lib/Makefile
#8

Updated by Andrew Stormont almost 5 years ago

Here is a webrev which fixes all instances of the build linking against libraries outside of the proto area: http://cr.illumos.org/~webrev/andy_js/5468/

I have not done a clean build so I don't know if there are any issues, but I think it's a good first step. Please review it when you get a chance.

#9

Updated by Gary Mills over 4 years ago

After a clean build on x86, I tried applying your patch and doing another build. There were no build errors, but I did get a bunch of ELF errors. I've attached elf.err for these.

I also ran wsdiff, comparing the prototype area before and after. Here's how it looked:

<mills@ati:941>$ env PATH=/opt/onbld/bin:$PATH wsdiff -r /tmp/1.res proto-bas>
usr/share/doc/ksh/shell_styleguide.html
usr/sbin/isns
usr/demo/link_audit/Makefile
usr/lib/libc/libc_hwcap3.so.1
usr/lib/libc/libc_hwcap2.so.1
usr/lib/libc/libc_hwcap1.so.1
usr/lib/netsvc/yp/udpublickey
usr/lib/netsvc/yp/stdethers
usr/lib/nfs/mountd
usr/lib/picl/plugins/libpicldevtree.so.1
usr/lib/amd64/libzpool.so.1
usr/lib/dtrace/64/libdtrace_forceload.so
usr/lib/dtrace/libdtrace_forceload.so
usr/lib/libzpool.so.1
etc/motd
lib/amd64/libc.so.1
lib/libc.so.1
<mills@ati:943>$ grep ^NOTE /tmp/1.res
NOTE: HTML difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ASCII difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ELF .text difference detected.
NOTE: ELF .text difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ELF .text difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ELF .text difference detected.
NOTE: ELF .text difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ASCII difference detected.
NOTE: ELF .SUNW_dof difference detected.
NOTE: ELF .SUNW_dof difference detected.

The files that had the .text differences were these:

usr/lib/netsvc/yp/udpublickey
usr/lib/netsvc/yp/stdethers
usr/lib/picl/plugins/libpicldevtree.so.1
usr/lib/dtrace/64/libdtrace_forceload.so
usr/lib/dtrace/libdtrace_forceload.so
#10

Updated by Andrew Stormont over 4 years ago

The .SUNW_dof stuff might be noise if Rich is right in #729.

#11

Updated by Gary Mills over 4 years ago

Yes, only the .text differences are significant here. I suspect they will disappear when the ELF errors are resolved.

#12

Updated by Andrew Stormont over 4 years ago

  • Related to Bug #5837: libdtrace audit library build missing LDLIBS added
#13

Updated by Yuri Pankov almost 4 years ago

  • Status changed from Feedback to In Progress
  • Assignee set to Yuri Pankov
  • % Done changed from 20 to 50
  • Difficulty changed from Medium to Bite-size
  • Tags deleted (needs-triage)

I have what looks to be complete fix for this. Andrew's changes to add missing LDLIBS should (and will) be handled in separate issue.

#14

Updated by Andrew Stormont almost 4 years ago

Makes sense to me.

#15

Updated by Yuri Pankov almost 4 years ago

  • Status changed from In Progress to Pending RTI
  • % Done changed from 50 to 90
#16

Updated by Electric Monk almost 4 years ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 90 to 100

git commit ba9ca9119d43e9c8e21e14f2129dce27d2dc0cdc

commit  ba9ca9119d43e9c8e21e14f2129dce27d2dc0cdc
Author: Yuri Pankov <yuri.pankov@nexenta.com>
Date:   2015-12-05T14:50:53.000Z

    5468 Missing dependencies in lib/Makefile
    Reviewed by: Andy Stormont <astormont@racktopsystems.com>
    Reviewed by: Gary Mills <gary_mills@fastmail.fm>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

#17

Updated by Electric Monk almost 4 years ago

git commit b5254d6ec12a0443b3a431a7e656f38e4ac98b2e

commit  b5254d6ec12a0443b3a431a7e656f38e4ac98b2e
Author: Yuri Pankov <yuri.pankov@nexenta.com>
Date:   2015-12-14T22:04:05.000Z

    5468 Missing dependencies in lib/Makefile (fix scsi)

Also available in: Atom PDF