"default" isn't really a group, it's more the absence of one. With
Bundler, this means that a gem should be installed unconditionally,
regardless of which groups are specified. It doesn't really make sense
to allow these gems to be omitted from a bundlerEnv.
This wasn't really an issue until the latest minor release of Bundix
(2.4), because prior to then Bundix didn't emit group attributes, and so
this functionality of bundlerEnv wasn't really used. However, it is now
apparent that a better default for bundlerEnv would be to include all
gem groups by default, not just the default group. This matches the
behavior of Bundler, and makes more sense, because the default group
alone isn't necessarily useful for anything -- consider a Rails app with
production, development, and test groups. It has the additional benefit
of being backwards compatible with how this would have worked before the
Bundix update.
Suppose I have a Gemfile like this:
source "https://rubygems.org"
gem "actioncable"
gem "websocket-driver", group: :test
The gemset.nix generated by Bundix 2.4.1 will set ActionCable's groups
to [ "default" ], and websocket-driver's to [ "test" ]. This means that
the generated bundlerEnv wouldn't include websocket-driver unless the
test group was included, even though it's required by the default group.
This is arguably a bug in Bundix (websocket-driver's groups should
probably be [ "default" "test" ] or just [ "default" ]), but there's no
reason bundlerEnv should omit dependencies even given such an input --
it won't necessarily come from Bundix, and it would be good for
bundlerEnv to do the right thing.
To fix this, filterGemset is now a recursive function, that adds
dependencies of gems in the group to the filtered gemset until it
stabilises on the gems that match the required groups, and all of their
recursive dependencies.
Change introduced in #50672.
The patch does not apply on older protobuf like protobuf3_1
```
$ nix-build -E 'with import ./. { }; python3Packages.protobuf.override { protobuf = protobuf3_1; }'
unpacking sources
unpacking source archive /nix/store/1zdyl0cxaa8ha2v1zp75zzdjd6j99d0m-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/util/python/BUILD
patching sources
applying patch /nix/store/yagx7hvylnnjq7lxbcia0y5lq1r736w3-0a59054c30e4f0ba10f10acfc1d7f3814c63e1a7.patch
patching file google/protobuf/pyext/descriptor.cc
Hunk #1 succeeded at 55 (offset -1 lines).
patching file google/protobuf/pyext/descriptor_containers.cc
patching file google/protobuf/pyext/descriptor_pool.cc
Hunk #1 succeeded at 47 (offset -1 lines).
patching file google/protobuf/pyext/extension_dict.cc
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED -- saving rejects to file google/protobuf/pyext/extension_dict.cc.rej
patching file google/protobuf/pyext/message.cc
Hunk #1 succeeded at 82 (offset 3 lines).
Hunk #2 succeeded at 1425 (offset -104 lines).
```
Since the patch isn't necessary on python versions older than 3.7, let's
only apply it for version 3.7. This means that most things using older
protobuf implementation will now be able to build when using an older
pythonPackage set (as is most probably the case anyway).
This still leaves protobuf 3.1 using packages hanging, but the errors
will be localized to those that would be breaking anyway with the
upgrade to 3.7 as default python.
credstash was only available as a library. Provide it as a standalone
application as well.
In order for this to work, I needed to remove the copy of
the library that's placed in $out/bin and marked executable
during the install phase. Other than the patched shebang and
executable bit, it's identical to the library that's installed to
$out/lib/python3.7/site-packages.
Before the postFixup has run `wrapPythonPrograms`, $out/bin contains
two Python files -- credstash and credstash.py -- where bin/credstash
is the executable you'd expect a user to invoke from the command-line
and bin/credstash.py contains the credstash module, which bin/credstash
imports.
After `wrapPythonPrograms` has run, bin/credstash is a shell
wrapper around the bin/.credstash-wrapped python entrypoint, and
bin/credstash.py is shell wrapper around bin/.credstash.py-wrapped.
Invoking bin/credstash execs bin/.credstash-wrapped, and that python
script attempts to import the credstash module from bin/credstash.py,
the shell wrapper, rather than either bin/.credstash.py-wrapped or
lib/python3.7/site-packages/credstash.py.
This leads to an error:
$ credstash get mykey
Traceback (most recent call last):
File "/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin/.credstash-wrapped", line 8, in <module>
from credstash import main
File "/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin/credstash.py", line 2
export PATH='/nix/store/6lm4gi5iv8fbf1b1mm6g3gfnnv63f1gn-python3-3.7.1/bin:/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin:/nix/store/2n13gf1zdr39ir5dynxlkqndxgy36g08-python3.7-setuptools-40.4.3/bin:/nix/store/mhnqwpa4y1l81zi4cwx989i8h8z9g67l-python3.7-jmespath-0.9.0/bin:/nix/store/qc6q3a2nv4211wyh7q319v6zzd3ab6pc-python3.7-docutils-0.14/bin'${PATH:+':'}$PATH
^
SyntaxError: invalid syntax
If we try using `dontWrapPythonPrograms` to resolve this, runtime
dependency lookups fail:
$ credstash get mykey
Traceback (most recent call last):
File "/run/current-system/sw/bin/credstash", line 7, in <module>
from credstash import main
File "/nix/store/8rmldlvlv1z1xl7w02dy7f5qhkzdrg8z-python3.7-credstash-1.15.0/bin/credstash.py", line 26, in <module>
import boto3
ModuleNotFoundError: No module named 'boto3'
I was able to resolve things by simply removing bin/credstash.py before
the postFixup phase has a chance to wrap any executables. Now the
executable imports the library correctly:
(shell wrapper)
bin/credstash
│ (python executable)
└─> bin/.credstash-wrapped
│ (python library)
└─> lib/python3.7/site-packages/credstash.py
