Commit graph

2987 commits

Author SHA1 Message Date
Austin Seipp 04a543b3a0 defaultCrateOverrides: foundationdb native dependencies
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-17 19:28:48 -06:00
zimbatm 695a3d4254
Merge pull request #50302 from zimbatm/libredirect-misc
libredirect: misc changes
2018-11-14 00:16:56 +01:00
zimbatm d04a1265a1
libredirect: set install_name on Darwin
fixes https://github.com/NixOS/nixpkgs/pull/50246#issuecomment-437975038
2018-11-14 00:05:26 +01:00
zimbatm 91c130e2f5
libredirect: introduce optional setup-hook
This allows to simplify the usage of libredirect inside of nix build
sandboxes. Add "libredirect.hook" to the build inputs to get everything
linked in automaticall. All that's left is to set NIX_REDIRECTS and call
the target program.
2018-11-14 00:05:23 +01:00
Matthew Bauer 4b8c1d23d0
Merge pull request #42794 from telent/make-closure-x
make-closure: needs build system mkdir and jq
2018-11-13 15:02:08 -06:00
zimbatm e62db105c4
libredirect: specify libName
reduces a bit of duplication and can also be used from the outside:

   export LD_PRELOAD=${libredirect}/lib/${libredirect.libName}
2018-11-13 12:26:15 +01:00
Antoine Eiche c12f75649e dockerTools.buildImageWithNixDb: simplifications and switch to closureInfo
Since Nix 2 is now the stable Nix version, we can use closureInfo
which simplifies the Nix database initialisation (size and hash are
included in the "dump").
2018-11-12 18:30:53 +01:00
aszlig a815f53c60
libredirect: Add preload wrapper for stat()
Pull request #50246 was merged a bit too quickly and it was supposed to
fix libredirect on Darwin. However it still failed on Darwin and this
was missed by the person merging the pull request.

The reason this was failing was that there is no __xstat* on Darwin.

So I'm adding a wrapper for stat() as well as it works on Darwin and it
still doesn't hurt on GNU/Linux.

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra, @zimbatm
2018-11-12 13:31:43 +01:00
aszlig 34dd1c68f8
libredirect: Add a small test
This is just a sanity check on whether the library correctly wraps the
syscalls and it's using the "true" executable for posix_spawn() and
execv().

The installCheckPhase is not executed if we are cross-compiling, so this
shouldn't break cross-compilation.

One thing I'm not actually sure is whether ${coreutils}/bin/true is
universally available on all the platforms, nor whether all the
functions we use in the test are available, but we can still fix that
after we've found out about that.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-12 11:02:54 +01:00
aszlig ba1fddb315
libredirect: Use extensions.sharedLibrary
This is to make sure we get the correct shared library suffix of the
target platform. While for example on Darwin it would even work with the
hardcoded .so prefix it's IMHO a bit nicer to have the actual native
extension.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-12 10:08:02 +01:00
zimbatm 9ef52352bd
assume that it works on all unix platforms 2018-11-12 00:09:36 +01:00
zimbatm d76ec523bb
use for cross-compilation 2018-11-12 00:08:18 +01:00
aszlig 753743c37b
libredirect: Add support for Darwin
The library can be used also on Darwin using it like this:

  NIX_REDIRECTS='foo=bar' \
  DYLD_INSERT_LIBRARIES=${libredirect}/lib/libredirect.so \
  DYLD_FORCE_FLAT_NAMESPACE=1 \
  some_program

So let's actually not hardcade gcc and add Darwin to meta.platforms.

No other changes seem to be required.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-11 19:29:12 +01:00
Théo Zimmermann 742bce7793
buildDunePackage: inline dune.installPhase for easier overriding
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2018-11-07 10:08:08 +01:00
Théo Zimmermann 406405d8bd
buildDunePackage: add support for pre and post phase hooks
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2018-11-07 10:08:08 +01:00
Théo Zimmermann 794158fcd5
buildDunePackage: new support function; use it to refactor some OCaml derivations 2018-11-07 10:08:03 +01:00
Yegor Timoshenko 77dad17ab6
Merge pull request #49725 from pbogdan/chrootenv-strip
chrootenv: strip the binary
2018-11-05 18:57:28 +00:00
Yegor Timoshenko cea0e9226f
chrootenv: use meson 2018-11-04 11:33:34 +00:00
Piotr Bogdan ccb76eeb3c chrootenv: strip the binary 2018-11-04 03:43:22 +00:00
aszlig c64624b843
autoPatchelfHook: Correctly detect PIE binaries
I originally thought it would just be enough to just check for an INTERP
section in isExecutable, however this would mean that we don't detect
statically linked ELF files, which would break our recent improvement to
gracefully handle those.

In theory, we are only interested in ELF files that have an INTERP
section, so checking for INTERP would be enough. Unfortunately the
isExecutable function is already used outside of autoPatchelfHook, so we
can't easily get rid of it now, so let's actually strive for more
correctness and make isExecutable actually match ELF files that are
executable.

So what we're doing instead now is to check whether either the ELF type
is EXEC *or* we have an INTERP section and if one of them is true we
should have an ELF executable, even if it's statically linked.

Along the way I also set LANG=C for the invocations of readelf, just to
be sure we don't get locale-dependent output.

Tested this with the following command (which contains almost[1] all the
packages using autoPatchelfHook), checking whether we run into any
library-related errors:

  nix-build -E 'with import ./. { config.allowUnfree = true; };
    runCommand "test-executables" {
      drvs = [
        anydesk cups-kyodialog3 elasticsearch franz gurobi
        masterpdfeditor oracle-instantclient powershell reaper
        sourcetrail teamviewer unixODBCDrivers.msodbcsql17 virtlyst
        vk-messenger wavebox zoom-us
      ];
    } ("for i in $drvs; do for b in $i/bin/*; do " +
       "[ -x \"$b\" ] && timeout 10 \"$b\" || :; done; done")
  '

Apart from testing against library-related errors I also compared the
resulting store paths against the ones prior to this commit. Only
anydesk and virtlyst had the same as they didn't have self-references,
everything else differed only because of self-references, except
elasticsearch, which had the following PIE binaries:

  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autoconfig
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autodetect
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/categorize
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/normalize

These binaries were now patched, which is what this commit is all about.

[1]: I didn't include the "maxx" package (MaXX Interactive Desktop)
     because the upstream URLs are no longer existing and I couldn't
     find them elsewhere on the web.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: https://github.com/NixOS/nixpkgs/issues/48330
Cc: @gnidorah (for MaXX Interactive Desktop)
2018-11-03 08:07:42 +01:00
Kristoffer Søholm 5e5e57c572 buildFHSUserEnv: use runScript in env (#49077)
This makes its behaviour conform to what is implied in the
documentation.
2018-10-30 22:47:08 +01:00
Michał Janiszewski 3f05186984 Compare to None using identity is operator
This is a trivial change that replaces `==` operator with `is` operator, following PEP 8 guideline:

> Comparisons to singletons like None should always be done with is or is not, never the equality operators.

https://legacy.python.org/dev/peps/pep-0008/#programming-recommendations
2018-10-30 21:30:56 +01:00
Matthew Bauer 412093994b gcc: support avr
- respect libc’s incdir and libdir
- make non-unix systems single threaded
- set LIMITS_H_TEST to false for avr
- misc updates to support new libc’s
- use multilib with avr

For threads we want to use:
- posix on unix systems
- win32 on windows
- single on everything else

For avr:
- add library directories for avrlibc
- to disable relro and bind
- avr5 should have precedence over avr3 - otherwise gcc uses the wrong one
2018-10-29 14:34:09 -05:00
Matthew Bauer d59a9ac7cf avr: use new compilation infrastructure
Gets rid of:
  avrbinutils
  avrgcc

to replace with:
  pkgsCross.avr.buildPackages.binutils
  pkgsCross.avr.buildPackages.gcc
2018-10-29 14:34:09 -05:00
Jörg Thalheim 96c627b3f6
defaultCrateOverrides: add serde_derive 2018-10-28 21:59:19 +00:00
Jörg Thalheim e0a5689528
defaultCrateOverrides: order alphabetically 2018-10-28 21:55:26 +00:00
Pierre-Etienne Meunier ae3b4655a4 Carnix: 0.7.2 -> 0.8.10 (#40587)
Carnix: splits input into two parts: creates from creates.io and local ones
2018-10-28 00:06:29 +01:00
Jörg Thalheim f10b935f84
breakpointHook: add for debugging failing builds
Usuage: Add breakpointHook to your `buildInputs` like this:

  stdenv.mkDerivation rec {
    # ...
    buildInputs = [ breakpointHook ];
  });

When the build fails as show in this example:

  pkgs.hello.overrideAttrs (old: {
    buildInputs = [ breakpointHook ];
    postPatch = ''
      false
    '';
  });

It will halt execution printing the following message:

build failed in patchPhase with exit code 1
To attach to this build run the following command as root:

   cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10

Installing cntr and running the command will provide shell access to the
build sandbox of failed build:

sudo cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
WARNING: bad ownership on /nix/var/nix/profiles/per-user/root, should be 1000
[nixbld@localhost:/var/lib/cntr]$

At /var/lib/cntr the sandbox filesystem is mounted. All commands and
files of the system are still accessible within the shell.
To execute commands from the sandbox use the `cntr exec` subcommand.
2018-10-25 10:19:41 +01:00
Frederik Rietdijk 821a3beb10
Merge pull request #48306 from NixOS/staging-next
Merge staging-next into master
2018-10-18 11:23:04 +02:00
Renaud 3583fe7586
Merge pull request #26839 from volth/fetchmavenartifact-do-not-leak-hash
fetchMavenArtifact: prevent leaking nix hash to jar name
2018-10-17 09:34:34 +02:00
Timo Kaufmann 1aff3da14e
Merge pull request #48020 from erictapen/47709-fix-regex
buildRustPackage: fix regex for separating lib and bin
2018-10-10 19:29:09 +02:00
Frederik Rietdijk bc9bd012c4 Merge staging-next into staging 2018-10-09 15:37:52 +02:00
Frederik Rietdijk eeaf3a131f Merge master into staging-next 2018-10-09 15:37:22 +02:00
Linus Heckemann 9cc18fa7f9 debian vm tools: use snapshot.debian.org
snapshot.debian.org actually keeps track of all of the updates as they
come in rather than doing arbitrary (?) snapshots.
2018-10-08 18:05:09 +02:00
Justin Humm 64d0676fe1
buildRustPackage: fix regex for separating lib and bin
E.g. exa was wrongly put into /lib, as it matches

  .*.a

but not

  .*\.a
2018-10-07 22:14:19 +02:00
Sarah Brofeldt 2e38f5fc6e
Merge pull request #47448 from kalbasit/nixpkgs_add-bazel-watcher
bazel-watcher: init at 0.5.0
2018-10-04 00:00:47 +02:00
Edward Tate 6ad43a0bce
buildRustPackage now correctly installs binaries to bin and libraries to lib. 2018-10-03 16:27:10 +02:00
Samuel Leathers 024eb9a5a5 trivial builders: adding usage documentation for functions 2018-10-02 22:09:09 +02:00
Frederik Rietdijk 6ce04af137 Merge master into staging 2018-10-02 18:22:37 +02:00
Daiderd Jordan 1383c08f2c
Merge branch 'master' into staging-next 2018-10-01 19:42:07 +02:00
Sarah Brofeldt b256df4937 dockerTools: Use nix instead of nixUnstable 2018-10-01 09:51:52 +02:00
lewo 56b4db9710
Merge pull request #47411 from graham-at-target/multi-layered-images-crafted
Multi-Layered Docker Images
2018-10-01 09:48:24 +02:00
Wael M. Nasreddine 86a5535b2f
bazel-watcher: init at 0.5.0 2018-09-29 13:33:00 -07:00
Wael M. Nasreddine 90b7b4a509
build-bazel-package: remove any .git, .svn and .hg from external 2018-09-29 13:28:15 -07:00
Wael M. Nasreddine 18aa9b0b65
build-bazel-package: prefix bazel with the USER variable
Bazel computes the default value of output_user_root before parsing the
flag[0]. The computation of the default value involves getting the $USER
from the environment. I don't have that variable when building with
sandbox enabled.

[0]: 9323c57607/src/main/cpp/startup_options.cc (L123-L124)
2018-09-29 13:28:12 -07:00
Will Dietz f7db287960 patch-shebangs.sh: use more robust 'for each file' loop, check for dir
The latter is to avoid warnings printed by find if it doesn't exist.
2018-09-28 11:21:51 -05:00
Will Dietz 286381f072 patch-shebangs: simplify a bit per reviewer suggestion 2018-09-28 11:17:33 -05:00
Will Dietz 830f9fabd4 patch-shebangs: use isScript to safely check for shebang start
Fixes commonly encountered errors about broken pipes or null-bytes in
command-substitution.
2018-09-28 11:15:36 -05:00
Graham Christensen fb2d153dac
dockerTools: test buildLayeredImage 2018-09-27 14:19:43 -04:00
Graham Christensen 4fe9006190 dockerTools.buildLayeredImage: init
Create a many-layered Docker Image.

Implements much less than buildImage:

 - Doesn't support specific uids/gids
 - Doesn't support runninng commands after building
 - Doesn't require qemu
 - Doesn't create mutable copies of the files in the path
 - Doesn't support parent images

If you want those feature, I recommend using buildLayeredImage as an
input to buildImage.

Notably, it does support:

 - Caching low level, common paths based on a graph traversial
   algorithm, see referencesByPopularity in
   0a80233487993256e811f566b1c80a40394c03d6
 - Configurable number of layers. If you're not using AUFS or not
   extending the image, you can specify a larger number of layers at
   build time:

       pkgs.dockerTools.buildLayeredImage {
         name = "hello";
         maxLayers = 128;
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

 - Parallelized creation of the layers, improving build speed.
 - The contents of the image includes the closure of the configuration,
   so you don't have to specify paths in contents and config.

   With buildImage, paths referred to by the config were not included
   automatically in the image. Thus, if you wanted to call Git, you
   had to specify it twice:

       pkgs.dockerTools.buildImage {
         name = "hello";
         contents = [ pkgs.gitFull ];
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

   buildLayeredImage on the other hand includes the runtime closure of
   the config when calculating the contents of the image:

       pkgs.dockerTools.buildImage {
         name = "hello";
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

Minor Problems

 - If any of the store paths change, every layer will be rebuilt in
   the nix-build. However, beacuse the layers are bit-for-bit
   reproducable, when these images are loaded in to Docker they will
   match existing layers and not be imported or uploaded twice.

Common Questions

 - Aren't Docker layers ordered?

   No. People who have used a Dockerfile before assume Docker's
   Layers are inherently ordered. However, this is not true -- Docker
   layers are content-addressable and are not explicitly layered until
   they are composed in to an Image.

 - What happens if I have more than maxLayers of store paths?

   The first (maxLayers-2) most "popular" paths will have their own
   individual layers, then layer #(maxLayers-1) will contain all the
   remaining "unpopular" paths, and finally layer #(maxLayers) will
   contain the Image configuration.
2018-09-26 17:54:14 -04:00