Commit graph

1132 commits

Author SHA1 Message Date
K900 59bccda153 Revert "stdenv: add meta.repository field"
This reverts commit 7e1443abbb.
2024-03-30 17:19:21 +03:00
K900 0e8bcaa661 Revert "stdenv/check-meta: Don't create new environments when computing meta.repository"
This reverts commit f8b091d53f.
2024-03-30 17:18:47 +03:00
K900 ba79149c66 stdenv/check-meta: don't infrec on unsupported platforms 2024-03-30 14:30:20 +03:00
adisbladis bff4c55f3a
Merge pull request #300177 from adisbladis/meta-repository-perf-fixups
stdenv/check-meta: Fix performance regressions introduced in #294347
2024-03-30 20:22:33 +13:00
adisbladis dc49349c84 stdenv/check-meta: Inherit remaining lib access into scope
Hopefully this will result in people not adding new `lib.xxx` to check-meta.nix.
2024-03-30 19:20:59 +13:00
adisbladis f8b091d53f stdenv/check-meta: Don't create new environments when computing meta.repository 2024-03-30 19:19:05 +13:00
adisbladis c605bfd501
Merge pull request #297350 from adisbladis/commonmeta-output-list
stdenv/check-meta: Remove outputsToInstall list concat from common meta
2024-03-30 19:07:32 +13:00
a-n-n-a-l-e-e 2809c84cfb
Merge pull request #294347 from lolbinarycat/meta.repository
stdenv: add meta.repository field
2024-03-29 13:03:04 -07:00
binarycat 7e1443abbb stdenv: add meta.repository field 2024-03-29 14:34:09 -04:00
adisbladis 1e2f8f2a84 stdenv/check-meta: Remove outputsToInstall list concat from common meta
Normally either of "bin" or "out" will hit first so we can avoid dynamic looping altogether.
2024-03-20 16:29:17 +13:00
Robert Hensing 05f4b7b46d make-derivation.nix: Fix checkMetaRecursively
Oddly, I can't reproduce the error, but this change will make it
more robust.
See https://github.com/NixOS/nixpkgs/pull/295378#issuecomment-2002094487
2024-03-17 19:28:47 +01:00
Robert Hensing ba463e70e3 make-derivation.nix: Float out unsafeDerivationToUntrackedOutpath 2024-03-12 20:15:47 +01:00
Robert Hensing 0ab7b23637 make-derivation.nix: Update inline docs 2024-03-12 19:06:07 +01:00
Robert Hensing 797ad5ae87 make-derivation.nix: Float out new constant removedOrReplacedAttrNames 2024-03-12 18:43:11 +01:00
Robert Hensing cac23248b1 make-derivation.nix: Float out knownHardeningFlags 2024-03-12 18:41:57 +01:00
Robert Hensing 464d8c4a6c make-derivation.nix: Float inward darwin-specific derivation attributes 2024-03-12 18:41:18 +01:00
Robert Hensing 2fb7255cc0 make-derivation.nix: Evaluate flip 2024-03-12 18:39:19 +01:00
Robert Hensing 90197b6833 make-derivation.nix: Apply map composition law
... after inlining chooseDevOutputs.
2024-03-12 18:37:53 +01:00
Robert Hensing 954d9ce64b make-derivation.nix: Drop duplicate functionality from previous commit
These two commits make for a cleaner commit history and git blame than
https://github.com/NixOS/nixpkgs/pull/295105, where this refactor
was developed.

See its commit messages for details and design choices, esp. up to
and including 37f76fd4c3b23b324ab7ea06ef3b747338d2080f.
2024-03-12 18:32:03 +01:00
Robert Hensing 8dad51a2e2 make-derivation.nix: Split makeDerivationArgument, mkDerivation with duplicate functionality 2024-03-12 18:16:56 +01:00
Robert Hensing de516f6f13 make-derivation.nix: Return mkDerivation as an attribute 2024-03-11 17:06:02 +01:00
Robert Hensing ab56d3ac71 make-derivation.nix: Move into let binding 2024-03-11 17:02:59 +01:00
Robert Scott 4a91b3e798 cc-wrapper: add trivialautovarinit hardening flag support
this equates to -ftrivial-auto-var-init=pattern

clang has removed support for -ftrivial-auto-var-init=zero and
are unlikely to re-add it, so use -ftrivial-auto-var-init=pattern
on both compilers if only to make behaviour more consistent
between the two.

add to pkgsExtraHardening's defaultHardeningFlags.
2024-02-24 12:00:09 +00:00
Vladimír Čunát eff11adc57
Merge #286721: stdenv: fix rare tar.xz decompression issues
...into staging
2024-02-17 08:45:01 +01:00
Isidor Zeuner 11a19109b6 stdenv: disregard xz exit status in order to fix subtle decompression issues
There is a subtle bug with unpacking `tar.xz` archives which seems to happen only on some setups, and sometimes not in a reproducible manner (https://github.com/NixOS/nixpkgs/issues/278130, https://github.com/NixOS/nixpkgs/issues/20950). On the last occurrence, it could be tracked down to `xz` failing from a `SIGPIPE`, which can happen when it's connected to `tar` through a pipe and `tar` exits earlier (see e.g. https://www.linuxquestions.org/questions/slackware-14/%5Bpatch%5D-tar-issuing-a-sigpipe-in-installpkg-4175637923/ or https://bugs.gentoo.org/573642#c5).

Since `tar` should be able by itself to detect whether the archive is complete, I suggest to disregard the exit code from the `xz` invocation, done in this PR.

Fixes  https://github.com/NixOS/nixpkgs/issues/278130 (script tested here: https://github.com/NixOS/nixpkgs/pull/286579)
Probably also fixes https://github.com/NixOS/nixpkgs/issues/20950 (issue not reproduced here, feedback therefore welcome)
2024-02-13 17:00:48 +01:00
Kait Lam 355ab764b4 stdenv: refactor of --replace-{quiet,warn,fail} logic
This is a small simplification of the control flow surrounding these cases. It should make it more obvious when each case happens, and also explicitly defines the current behaviour of --replace.
2024-02-11 10:48:37 +10:00
Rick van Schijndel 2c98eaf597
Merge pull request #286526 from pbsds/fix-replace-quiet-1707148527
stdenv: fix `substituteStream --replace-quiet` deprecation warning
2024-02-07 21:35:19 +01:00
Peder Bergebakken Sundt cb4c41f93a stdenv: fix substituteStream --replace-quiet deprecation warning 2024-02-05 17:28:02 +01:00
Robert Scott 40868719b0 cc-wrapper: add zerocallusedregs hardening flag
this uses the value `used-gpr` which seems to be a commonly
chosen value for general use
2024-01-20 13:48:33 +00:00
Peder Bergebakken Sundt 4c1b74bac7 stdenv: substituteStream: escape echoed pattern in --replace mismatch warning 2024-01-14 22:08:03 +01:00
Peder Bergebakken Sundt e07a2fab7f stdenv: substituteStream: deprecate --replace in favor of --replace-{fail,warn,quiet} 2024-01-14 22:07:58 +01:00
sternenseemann f734056102 stdenv: fix evaluation if !stdenv.cc.hasCC
stdenv.cc may throw, e.g. in the case of pkgsCross.ghcjs where we must
not force it for the purpose of attribute accessing (`or` doesn't
implicitly tryEval…).

Regression introduced in 1a5bd697ad.
2024-01-13 12:45:36 +01:00
K900 bff9ee8862 Merge remote-tracking branch 'origin/master' into staging-next 2024-01-09 13:03:34 +03:00
Ryan Burns c324705cc3 treewide: simplify exec format conditionals 2024-01-07 17:43:33 -08:00
Weijia Wang eb5334c27d Merge branch 'master' into staging-next 2024-01-03 03:22:40 +01:00
adisbladis 4a5f2bd6e8 stdenv/check-meta: Use bespoke type checking
Aka `checkMeta` goes brrr.

Using the module system type checking works OK & generates good error messages.
The performance of using it however is terrible because of the value merging it does being very allocation heavy.

By implementing a very minimal type checker we can drastically improve the performance when nixpkgs is evaluated with `checkMeta = true`.
2023-12-26 15:14:42 +13:00
Martin Weinelt e8dd1cf066 Merge remote-tracking branch 'origin/staging-next' into staging 2023-12-12 14:13:07 +01:00
adisbladis 1e66093cca
Merge pull request #269546 from adisbladis/stdenv-meta-no-intermediate-alloc
stdenv: Avoid allocating intermediate attrset when checking meta validity
2023-12-12 10:57:44 +13:00
adisbladis 6aa414bcfd
Merge pull request #269782 from adisbladis/stdenv-meta-tolist-license-allocations
stdenv: Avoid some list allocations in check-meta when checking licenses
2023-12-12 10:55:58 +13:00
adisbladis 9a0a097a94 stdenv: Avoid allocating intermediate attrset when checking meta validity
This is a small performance optimization. It should be impercetible to most.

Benchmarks:

- Before
``` json
{
  "cpuTime": 0.2777960002422333,
  "envs": {
    "bytes": 3832648,
    "elements": 189513,
    "number": 144784
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 50229344
  },
  "list": {
    "bytes": 655304,
    "concats": 3249,
    "elements": 81913
  },
  "nrAvoided": 218962,
  "nrFunctionCalls": 127718,
  "nrLookups": 40946,
  "nrOpUpdateValuesCopied": 1563978,
  "nrOpUpdates": 8542,
  "nrPrimOpCalls": 113032,
  "nrThunks": 329605,
  "sets": {
    "bytes": 29774864,
    "elements": 1824537,
    "number": 36392
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 235909,
    "number": 24432
  },
  "values": {
    "bytes": 9691392,
    "number": 403808
  }
}
```

- After
```
{
  "cpuTime": 0.2615779936313629,
  "envs": {
    "bytes": 3833832,
    "elements": 189661,
    "number": 144784
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 50212960
  },
  "list": {
    "bytes": 655304,
    "concats": 3249,
    "elements": 81913
  },
  "nrAvoided": 218814,
  "nrFunctionCalls": 127718,
  "nrLookups": 40798,
  "nrOpUpdateValuesCopied": 1563978,
  "nrOpUpdates": 8542,
  "nrPrimOpCalls": 113032,
  "nrThunks": 329457,
  "sets": {
    "bytes": 29765392,
    "elements": 1824093,
    "number": 36244
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 235909,
    "number": 24432
  },
  "values": {
    "bytes": 9687840,
    "number": 403660
  }
}
```
2023-12-12 00:08:34 +13:00
adisbladis 3b13bd5c84 stdenv: Avoid some list allocations in check-meta when checking licenses 2023-12-12 00:03:20 +13:00
Robert Scott 1a5bd697ad mkDerivation, bintools-wrapper: move defaultHardeningFlags determination to bintools-wrapper
this makes it a lot easier to create a modified stdenv with a
different set of defaultHardeningFlags and as a bonus allows us
to inject the correct defaultHardeningFlags into toolchain wrapper
scripts, reducing repetition.

while most hardening flags are arguably more of a compiler thing,
it works better to put them in bintools-wrapper because cc-wrapper
can easily refer to bintools but not vice-versa.

mkDerivation can still easily refer to either when it is constructed.

this also switches fortran-hook.sh to use the same defaults for
NIX_HARDENING_ENABLE as for C. previously NIX_HARDENING_ENABLE
defaults were apparently used to avoid passing problematic flags
to a fortran compiler, but this falls apart as soon as mkDerivation
sets its own NIX_HARDENING_ENABLE - cc.hardeningUnsupportedFlags
is a more appropriate mechanism for this as it actively filters
out flags from being used by the wrapper, so switch to using that
instead.

this is still an imperfect mechanism because it doesn't handle a
compiler which has both langFortran *and* langC very well - applying
the superset of the two's hardeningUnsupportedFlags to either
compiler's invocation. however this is nothing new - cc-wrapper
already poorly handles a langFortran+langC compiler, applying two
setup hooks that have contradictory options.
2023-12-09 16:30:45 +00:00
Martin Weinelt 570af7d909
Merge pull request #263446 from emilylange/stdenv-xz-multithread
stdenv: enable multithreading for `xz` decompression
2023-12-06 15:41:01 +01:00
Vladimír Čunát 305cd929fd
Merge branch 'staging' into staging-next 2023-12-01 07:33:49 +01:00
Alyssa Ross 0d44b5b773 pkgsStatic.stdenv: fix custom CMake LINKER_LANGUAGE
If a CMake target has a non-default LINKER_LANGUAGE set, CMake will
manually add the libraries it has detected that language's compiler as
linking implicitly.  When it does this, it'll pass -Bstatic and
-Bdynamic options based on the vibes it gets from each such detected
library.  This in itself isn't a problem, because the compiler
toolchain, or our wrapper, or something, seems to be smart enough to
ignore -Bdynamic for those libraries.  But it does create a problem if
the compiler adds extra libraries to the linker command line after
that final -Bdynamic, because those will be linked dynamically.  Since
our compiler is static by default, CMake should reset to -Bstatic
after it's done manually specifying libraries, but CMake didn't
actually know that our compiler is static by default.  The fix for
that is to tell it, like so.

Until recently, this problem was difficult to notice, because it would
result binaries that worked, but that were dynamically linked.  Since
e08ce498f0 ("cc-wrapper: Account for NIX_LDFLAGS and NIX_CFLAGS_LINK
in linkType"), though, -Wl,-dynamic-linker is no longer mistakenly
passed for executables that are supposed to be static, so they end up
created with a /lib interpreter path, and so don't run at all on
NixOS.

This fixes pkgsStatic.graphite2.
2023-11-30 10:46:44 +00:00
github-actions[bot] e019e7cdb1
Merge staging-next into staging 2023-11-29 12:01:23 +00:00
piegames cb47f7e108
Merge pull request #267309: check-meta: Improve performance 2023-11-29 12:46:12 +01:00
Artturi 4b27e14f8c
Merge pull request #265738 from Artturin/patchshebangcon
stdenv: run patchShebangs on the configure script when it's a file
2023-11-26 01:28:41 +02:00
Artturin e68001550d stdenv: run patchShebangs on the configure script when it's a file
if the configure script has a `/usr/bin/env` or some other shebang which
is not in the sandbox then there will be errors such as

`...-stdenv-linux/setup: line 1299: ./configure: cannot execute: required file not found`

There are 250 files which `patchShebangs` `./configure`

https://github.com/search?q=NOT+is%3Afork+lang%3Anix+%2FpatchShebangs+.%5C%2Fconfigure%2F&type=code
2023-11-25 18:16:12 +02:00
Ivan Mincik 3b4b805561 stdenv: consistent phases header
Make phases header consistent for all phases.

`Running phase:` is from an old nix ux doc from 2020

https://github.com/tweag/nix-ux/blob/master/first_steps_with_nix_v2.md

Co-authored-by: Artturin <Artturin@artturin.com>
2023-11-17 03:11:48 +02:00