nixpkgs/pkgs/top-level
Graham Christensen fd045173ce referencesByPopularity: init to sort packages by a cachability heuristic
Using a simple algorithm, convert the references to a path in to a
sorted list of dependent paths based on how often they're referenced
and how deep in the tree they live. Equally-"popular" paths are then
sorted by name.

The existing writeReferencesToFile prints the paths in a simple
ascii-based sorting of the paths.

Sorting the paths by graph improves the chances that the difference
between two builds appear near the end of the list, instead of near
the beginning. This makes a difference for Nix builds which export a
closure for another program to consume, if that program implements its
own level of binary diffing.

For an example, Docker Images. If each store path is a separate layer
then Docker Images can be very efficiently transfered between systems,
and we get very good cache reuse between images built with the same
version of Nixpkgs. However, since Docker only reliably supports a
small number of layers (42) it is important to pick the individual
layers carefully. By storing very popular store paths in the first 40
layers, we improve the chances that the next Docker image will share
many of those layers.*

Given the dependency tree:

    A - B - C - D -\
     \   \   \      \
      \   \   \      \
       \   \ - E ---- F
        \- G

Nodes which have multiple references are duplicated:

    A - B - C - D - F
     \   \   \
      \   \   \- E - F
       \   \
        \   \- E - F
         \
          \- G

Each leaf node is now replaced by a counter defaulted to 1:

    A - B - C - D - (F:1)
     \   \   \
      \   \   \- E - (F:1)
       \   \
        \   \- E - (F:1)
         \
          \- (G:1)

Then each leaf counter is merged with its parent node, replacing the
parent node with a counter of 1, and each existing counter being
incremented by 1. That is to say `- D - (F:1)` becomes `- (D:1, F:2)`:

    A - B - C - (D:1, F:2)
     \   \   \
      \   \   \- (E:1, F:2)
       \   \
        \   \- (E:1, F:2)
         \
          \- (G:1)

Then each leaf counter is merged with its parent node again, merging
any counters, then incrementing each:

    A - B - (C:1, D:2, E:2, F:5)
     \   \
      \   \- (E:1, F:2)
       \
        \- (G:1)

And again:

    A - (B:1, C:2, D:3, E:4, F:8)
     \
      \- (G:1)

And again:

    (A:1, B:2, C:3, D:4, E:5, F:9, G:2)

and then paths have the following "popularity":

    A     1
    B     2
    C     3
    D     4
    E     5
    F     9
    G     2

and the popularity contest would result in the paths being printed as:

    F
    E
    D
    C
    B
    G
    A

* Note: People who have used a Dockerfile before assume Docker's
Layers are inherently ordered. However, this is not true -- Docker
layers are content-addressable and are not explicitly layered until
they are composed in to an Image.
2018-09-26 15:50:10 -04:00
..
aliases.nix goimports: Remove in favor of gotools 2018-09-17 07:58:04 +02:00
all-packages.nix referencesByPopularity: init to sort packages by a cachability heuristic 2018-09-26 15:50:10 -04:00
beam-packages.nix beam-packages: default erlangR19 -> erlangR20 2018-08-07 23:27:41 +02:00
coq-packages.nix coq_8_6: use OCaml 4.05 2018-08-29 12:49:32 +00:00
darwin-packages.nix treewide: Remove usage of remaining redundant platform compatability stuff 2018-08-30 17:20:32 -04:00
default.nix top-level: Move comma for stylistic consistency 2018-03-20 15:37:34 -04:00
dotnet-packages.nix Deedle: 1.2.0 -> 1.2.5 (#46702) 2018-09-15 16:18:29 +02:00
emacs-packages.nix elpy: fix build dep 2018-09-09 17:14:59 +02:00
emscripten-packages.nix Emscripten 1.37.16 to 1.37.36 (#37291) 2018-03-18 13:34:46 +01:00
haskell-packages.nix treewide: Remove usage of remaining redundant platform compatability stuff 2018-08-30 17:20:32 -04:00
haxe-packages.nix pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
impure.nix impure.nix: fix handling of localSystem 2018-06-25 17:13:02 -04:00
java-packages.nix pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
lua-packages.nix luaPackages.luadbi: 0.5 -> 0.6 (#47156) 2018-09-22 22:27:35 +02:00
make-tarball.nix tarball job: more --show-trace 2018-09-03 14:24:42 +02:00
metrics.nix Revert "release/metrics: drop requiredSystemFeatures" 2018-07-30 10:46:20 +02:00
ocaml-packages.nix ocamlPackages.odoc: init at 1.2.0 2018-09-24 05:08:37 +00:00
perl-packages.nix perlPackages.MojoIOLoopForkCall: 0.19 -> 0.20 2018-09-20 15:17:30 -05:00
php-packages.nix phpPackages.xdebug26: 2.6.0 -> 2.6.1 (#46205) 2018-09-07 17:57:37 +02:00
pure-packages.nix
python-packages.nix Merge pull request #47283 from flokli/elasticsearch-dsl-6.2.1 2018-09-24 20:40:21 +01:00
release-cross.nix release-lib: Filter supportedSystems with meta.platforms-style patterns 2018-03-19 21:32:28 -04:00
release-lib.nix lib: Messed up or operator precedence 2018-03-19 22:25:49 -04:00
release-python.nix pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
release-small.nix release-small.nix: Fix evaluation error 2018-03-17 21:57:44 -04:00
release.nix release.nix: fix a typo (breaking tarball job) 2018-08-18 08:45:28 +02:00
splice.nix treewide: Remove usage of remaining redundant platform compatability stuff 2018-08-30 17:20:32 -04:00
stage.nix top-level: system should still come from the host platform 2018-09-23 13:24:44 -04:00
unix-tools.nix treewide: Remove usage of remaining redundant platform compatability stuff 2018-08-30 17:20:32 -04:00
wine-packages.nix wine: build wineWow packages on hydra 2018-07-03 21:29:32 +03:00