Bug #3353

/usr/xpg4/bin/fgrep not compliant with /usr/bin/fgrep nor /usr/gnu/bin/fgrep

Added by Richard PALO over 1 year ago. Updated about 1 month ago.

Status:Feedback Start date:2012-11-10
Priority:Normal Due date:
Assignee:- % Done:

0%

Category:- Spent time: -
Target version:-
Difficulty:Medium Tags:needs-triage

Description

naughty bug encountered with /usr/xpg4/bin/fgrep
please see https://github.com/joyent/pkgsrc/issues/21#issuecomment-10252529

test-fgrep.sh (713 Bytes) Richard PALO, 10/06/2013 04:41 pm

History

Updated by Igor Pashev over 1 year ago

rm -rf /usr/xpg4

Updated by Andrew Stormont 7 months ago

  • Status changed from New to Rejected

In the link you posted $i is a variable - it's supposed to be expanded by the shell and not fgrep - that's why it's in double quotes in the first place. Changing from double to single quotes was not the correct thing to do and that it appeared to solve the problem is purely coincidental. The snippet posted from the manpage is irrelevant in this context.

Updated by Richard PALO 7 months ago

Andrew Stormont wrote:

In the link you posted $i is a variable - it's supposed to be expanded by the shell and not fgrep - that's why it's in double quotes in the first place. Changing from double to single quotes was not the correct thing to do and that it appeared to solve the problem is purely coincidental. The snippet posted from the manpage is irrelevant in this context.

That was a bit myopic to reject before testing.

I encourage you to "pkg install libgcrypt" if not installed already, and
try once without /usr/xpg4/bin at the head of your PATH and then with.

You output should be somewhat as follows:

richard@omni1:~$ echo $PATH
/usr/bin:/usr/sbin:/sbin
richard@omni1:~$ libgcrypt-config --cflags
-I/usr/include/amd64
richard@omni1:~$ export PATH=/usr/xpg4/bin:$PATH
richard@omni1:~$ libgcrypt-config --cflags

richard@omni1:~$ 

Updated by Yuri Pankov 7 months ago

Could you please provide a simple testcase for this (something simpler than trying to build libgcrypt)?

Updated by Richard PALO 7 months ago

Yuri Pankov wrote:

Could you please provide a simple testcase for this (something simpler than trying to build libgcrypt)?

??? why build it, just install it and debug with the /usr/bin/i386/libgcrypt-config script
(or /usr/bin/amd64/libgcrypt-config if on x86_64) as indicated!

the "--cflags" as well as the "--libs" target demonstrate the problem.

Updated by Garrett D'Amore 7 months ago

So I've not tried to debug libgrypt's scripts, but clearly the use of "$i" instead of '$i' in that script is busted (unless you mean that value to be expanded by the shell?). And my guess is that the problem is not fgrep but something else (perhaps xpg4/sh is behaving differently?!?)

Notably I tried the following:

gdamore@deylite{87}> cat /tmp/tfile
Line one
This i$ a file
Line three
A bracket [ is here ] and there
A question ? To be or not.
gdamore@deylite{88}> /usr/bin/fgrep -v -- 'i$' /tmp/tfile
Line one
Line three
A bracket [ is here ] and there
A question ? To be or not.
gdamore@deylite{89}> /usr/xpg4/bin/fgrep -v -- 'i$' /tmp/tfile
Line one
Line three
A bracket [ is here ] and there
A question ? To be or not.

Both seem to work properly. So, I'd like a simpler explanation -- trying to dig down into some third party installation script is not a very useful bug report, especially since clearly the use of double quotes in that script is broken.

If $i was meant to be expanded by the shell, what is its value? That would be most helpful.

Updated by Richard PALO 7 months ago

Garrett D'Amore wrote:

So I've not tried to debug libgrypt's scripts, but clearly the use of "$i" instead of '$i' in that script is busted (unless you mean that value to be expanded by the shell?). And my guess is that the problem is not fgrep but something else (perhaps xpg4/sh is behaving differently?!?)

Notably I tried the following:

[...]

Both seem to work properly. So, I'd like a simpler explanation -- trying to dig down into some third party installation script is not a very useful bug report, especially since clearly the use of double quotes in that script is broken.

If $i was meant to be expanded by the shell, what is its value? That would be most helpful.

In any event, I repeat.... this script works to all appearances just fine with /usr/bin/fgrep and /usr/gnu/bin/fgrep but not with /usr/xpg4/bin/fgrep... it is rather widely used on other platforms as well.

using 'sh -x' or 'ksh93 -x' may shed light on how $i takes on its values.

Updated by Richard PALO 7 months ago

I believe this is recursively digressing, filip already answered the question as to why double quotes are necessary:

==========================
Er, that doesn't make sense at all. You need the $i expanded there, because it's part of the Bash 'for' loop. The script needs to fgrep for whatever $i stands for in the loop, not for a literary '$i' string. Whatever the problem is, it's not in the quoting for sure. ==========================

Updated by Richard PALO 7 months ago

Okay, it is useful to recreate, as the problem appears to be not in /usr/xpg4/bin/fgrep but in the path selection process.

I boiled down the test program, and the following exhibits much more clearly the problem:

richard@x3200:~$ echo $PATH
/usr/bin:/usr/sbin:/sbin
richard@x3200:~$ ./test-fgrep.sh 
fgrep: -L/usr/lib/amd64 -lgcrypt -lgpg-error
/usr/xpg4/bin/fgrep: -L/usr/lib/amd64 -lgcrypt -lgpg-error
richard@x3200:~$ PATH=/usr/xpg4/bin:$PATH ./test-fgrep.sh 
fgrep:
/usr/xpg4/bin/fgrep: -L/usr/lib/amd64 -lgcrypt -lgpg-error

BTW... the script does nothing more than making sure the libs (or cflags) returned are unique and only appear once in the list.

Updated by Richard PALO 6 months ago

I happened to come across https://getupdates.oracle.com/readme/README.146054-06
which mentions a number of /usr/xpg4/bin/sh problems.

So I tried:

richard@x3200:~$ PATH=/usr/xpg4/bin:$PATH bash ./test-fgrep.sh 
fgrep: -L/usr/lib/amd64 -lgcrypt -lgpg-error
/usr/xpg4/bin/fgrep: -L/usr/lib/amd64 -lgcrypt -lgpg-error

I guess sheepishly this seems to be more a /usr/xpg4/bin/sh problem instead.

Perhaps the title could be corrected. It is a bug, though.

Updated by Richard PALO 6 months ago

  • Status changed from Rejected to Feedback

Just noticed the status was 'rejected', yet this is very definitely a bug, and potentially quite a perverse one at that since it involves /usr/xpg4/bin/sh itself.

Updated by Andrew Stormont about 1 month ago

Richard PALO wrote:

I believe this is recursively digressing, filip already answered the question as to why double quotes are necessary:

========================== Er, that doesn't make sense at all. You need the $i expanded there, because it's part of the Bash 'for' loop. The script needs to fgrep for whatever $i stands for in the loop, not for a literary '$i' string. Whatever the problem is, it's not in the quoting for sure. ==========================

I agree. "$i" is what it should be otherwise it's going to be interpreting '$i' (literally) as regex - and that doesn't look like useful (or even valid) regex to me as it's going to be trying to match something after the end of the match, which obviously makes no sense at all; you're effectively telling it to not match anything, and by passing the -v flag to fgrep you're telling it to invert the match (i.e match everything). The script is busted, not fgrep.

Updated by Andrew Stormont about 1 month ago

<duplicate post>

Updated by Richard PALO about 1 month ago

Andrew Stormont wrote:

The script is busted, not fgrep.

Andrew, perhaps there could be a better script, but clearly there is an interaction anomaly with sh (ksh) and /usr/xpg4/bin at the head of PATH as test-fgrep.sh indicates.
https://www.illumos.org/issues/3353#note-9
PM: the same incantation with bash operates correctly. https://www.illumos.org/issues/3353#note-10

    for i in $libdirs $libs_final ; do
       if echo "$tmp" | fgrep -v -- "$i" >/dev/null; then
           tmp="$tmp $i" 
       fi
    done

something between the`if echo "$tmp"`, the pipe ('|') and the `fgrep` command is not returning the right exit status whereas surprisingly it does if `/usr/xpg4/bin/fgrep` is used.

Updated by Richard PALO about 1 month ago

Lets simplify this even more :

$ ksh -c 'echo "" | fgrep -v -- "-L/usr/lib/amd64"; echo "exit status $?"'

exit status 0
$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | fgrep -v -- "-L/usr/lib/amd64"; echo "exit status $?"'
exit status 1

The only difference between these two commands is the PATH

And you can see by trying it that if you specify the full path of `fgrep` as
`/usr/xpg4/bin/fgrep`, then the result is the same in both cases.

Updated by Richard PALO about 1 month ago

This is interesting:

$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | fgrep -v -- "-L/usr/lib/amd64"; echo "exit status $?"'
exit status 1
$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo " " | fgrep -v -- "-L/usr/lib/amd64"; echo "exit status $?"'

exit status 0

Here, the only difference is that I added a space in the initial `echo`...

Updated by Marcel Telka about 1 month ago

[removed confusing comment]

Updated by Andrew Stormont about 1 month ago

Richard PALO wrote:

Lets simplify this even more : [...]

The only difference between these two commands is the PATH

And you can see by trying it that if you specify the full path of `fgrep` as `/usr/xpg4/bin/fgrep`, then the result is the same in both cases.

It seems there is a difference in how the fgrep versions treat inverted empty matches. I think we need to look at the standards and then at how the GNU tools behave.

Updated by Richard PALO about 1 month ago

Andrew Stormont wrote:

Richard PALO wrote:

Lets simplify this even more : [...]

The only difference between these two commands is the PATH

And you can see by trying it that if you specify the full path of `fgrep` as `/usr/xpg4/bin/fgrep`, then the result is the same in both cases.

It seems there is a difference in how the fgrep versions treat inverted empty matches. I think we need to look at the standards and then at how the GNU tools behave.

Andy, just out of curiosity, why would there be a difference in the fgrep version if, with PATH=/usr/xpg4/bin:$PATH, `fgrep` is invoked or `/usr/xpg4/bin/fgrep`?
This is what intrigues me, the divergent treatment when the cases should be equivalent..

By the way, using `/usr/bin/echo` instead of `echo` doesn't change anything either...
Seems to in and/or around the `|` (pipe).
Is there some special treatment [ef]grep in some mysterious codepath that might be interfering?

Updated by Marcel Telka about 1 month ago

I think the problem could be easily demonstrated using this:

$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | fgrep -v bb; echo "exit status $?"'

exit status 0
$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | fgrep -v bbb; echo "exit status $?"'
exit status 1
$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | fgrep bb; echo "exit status $?"'
exit status 1
$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | fgrep bbb; echo "exit status $?"'
exit status 1
$

The problem is that the 2nd invocation (fgrep -v bbb) fails with status 1 (it should return an empty line and exist status should be 0).

Updated by Gary Mills about 1 month ago

This is a rather subtle bug. On oi_151a9, the second invocation above does indeed return status 1and no empty line. On Solaris 11.1, the same thing returns status 0 and an empty line. When I replace the echo and the pipe with input from /dev/null, they both return status 1. The difference is that the echo produces one character, a newline, whereas the /dev/null gives an immediate EOF.

Updated by Richard PALO about 1 month ago

Gary Mills wrote:

This is a rather subtle bug. On oi_151a9, the second invocation above does indeed return status 1and no empty line. On Solaris 11.1, the same thing returns status 0 and an empty line. When I replace the echo and the pipe with input from /dev/null, they both return status 1. The difference is that the echo produces one character, a newline, whereas the /dev/null gives an immediate EOF.

PM:

User Commands                                             ECHO(1)

NAME
     echo - echo arguments

SYNOPSIS
     /usr/bin/echo [string]...

DESCRIPTION
     The echo utility writes its arguments, separated  by  BLANKs
     and  terminated  by  a  NEWLINE,  to the standard output. If
     there are no arguments, only the NEWLINE character is  writ-
     ten.

thought I mentioned above the Or*cle update ...

Interesting Marcel's finding.. but note still:

$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | /usr/xpg4/bin/fgrep -v bb; echo "exit status $?"'

exit status 0
$ PATH=/usr/xpg4/bin:$PATH ksh -c 'echo "" | /usr/xpg4/bin/fgrep -v bbb; echo "exit status $?"'

exit status 0

Also available in: Atom PDF