Co-authored-by: sterni <sternenseemann@systemli.org>
9.2 KiB
Maintainer Workflow
This is the documentation for periodically merging the haskell-updates
branch
into the master
branch. This workflow is performed by members in the
@NixOS/haskell team.
Each member of the team takes a two week period where they are in charge of
merging the haskell-updates
branch into master
.
The goal of this workflow is to regularly merge the haskell-updates
branch
into the master
branch, while making sure there are no evaluation errors or
build errors that get into master
.
The workflow generally proceeds in three main steps:
- create the initial
haskell-updates
PR, and update Stackage and Hackage snapshots - wait for contributors to fix newly broken Haskell packages
- merge
haskell-updates
intomaster
We describe each of these steps in a separate section.
Initial haskell-updates
PR
In this section we create the PR for merging haskell-updates
into master
.
-
Make sure the
haskell-updates
branch is up-to-date withmaster
. -
Update the Stackage Nightly resolver used by Nixpkgs and create a commit:
$ ./maintainers/scripts/haskell/update-stackage.sh --do-commit
-
Update the Hackage package set used by Nixpkgs and create a commit:
$ maintainers/scripts/haskell/update-hackage.sh --do-commit
-
Regenerate the Haskell package set used in Nixpkgs and create a commit:
$ maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit
-
Push these commits to the Nixpkgs repository.
-
Open a PR on Nixpkgs merging
haskell-updates
intomaster
.
Use the following message body:
### This Merge
This PR is the regular merge of the `haskell-updates` branch into `master`.
This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.
I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @TODO will continue these merges from 2021-TODO-TODO to 2021-TODO-TODO.
### haskellPackages Workflow Summary
Our workflow is currently described at `pkgs/development/haskell-modules/HACKING.md`.
The short version is this:
* We regularly update the Stackage and Hackage pins on `haskell-updates` (normally at the beginning of a merge window).
* The community fixes builds of Haskell packages on that branch.
* We aim at at least one merge of `haskell-updates` into `master` every two weeks.
* We only do the merge if the `mergeable` job is succeeding on hydra.
* If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)
---
This is the follow-up to #TODO.
Make sure to replace all TODO with the actual values.
Notify Maintainers and Fix Broken Packages
After you've done the previous steps, Hydra will start building the new and updated Haskell packages. You can see the progress Hydra is making at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. This Hydra jobset is defined in the file release-haskell.nix.
Notify Maintainers
When Hydra finishes building all the updated packages for the haskell-updates
jobset, you should generate a build report to notify maintainers of their
newly broken packages. You can do that with the following commands:
$ maintainers/scripts/haskell/hydra-report.hs get-report
$ maintainers/scripts/haskell/hydra-report.hs ping-maintainers
The hyda-report.hs ping-maintainers
command generates a Markdown document
that you can paste in a GitHub comment on the PR opened above. This
comment describes which Haskell packages are now failing to build. It also
pings the maintainers so that they know to fix up their packages.
It may be helpful to pipe hydra-report.hs ping-maintainers
into xclip
(XOrg) or wl-copy
(Wayland) in order to post on GitHub.
This build report can be fetched and re-generated for new Hydra evaluations. It may help contributors to try to keep the GitHub comment updated with the most recent build report.
Maintainers should be given at least 7 days to fix up their packages when they
break. If maintainers don't fix up their packages with 7 days, then they
may be marked broken before merging haskell-updates
into master
.
Fix Broken Packages
After getting the build report, you can see which packages and Hydra jobs are
failing to build. The most important jobs are the maintained
and mergeable
jobs. These are both defined in
release-haskell.nix
.
mergeable
is a set of the most important Haskell packages, including things
like Pandoc and XMonad. These packages are widely used. We would like to
always keep these building.
maintained
is a set of Haskell packages that have maintainers in Nixpkgs.
We should be proactive in working with maintainers to keep their packages
building.
Steps to fix Haskell packages that are failing to build is out of scope for this document, but it usually requires fixing up dependencies that are now out-of-bounds.
Mark Broken Packages
Packages that do not get fixed can be marked broken with the following commands. First check which packages are broken:
$ maintainers/scripts/haskell/hydra-report.hs get-report
$ maintainers/scripts/haskell/hydra-report.hs mark-broken-list
This shows a list of packages that reported a build failure on x86_64-linux
on Hydra.
Next, run the following command:
$ maintainers/scripts/haskell/mark-broken.sh --do-commit
This first opens up an editor with the broken package list. Some of these packages may have a maintainer in Nixpkgs. If these maintainers have not been given 7 days to fix up their package, then make sure to remove those packages from the list before continuing. After saving and exiting the editor, the following will happen:
- packages from the list will be added to
configuration-hackage2nix/broken.yaml
hackage-packages.nix
will be regenerated- the
configuration-hackage2nix/transitive-broken.yaml
file will be updated hackage-packages.nix
will be regenerated again- everything will be committed
Merge haskell-updates
into master
Now it is time to merge the haskell-updates
PR you opened above.
Before doing this, make sure of the following:
- All Haskell packages that fail to build are correctly marked broken or transitively broken.
- The
maintained
andmergeable
jobs are passing on Hydra. - The maintainers for any maintained Haskell packages that are newly broken
have been pinged on GitHub and given at least a week to fix their packages.
This is especially important for widely-used packages like
cachix
.
When you've double-checked these points, go ahead and merge the haskell-updates
PR.
After merging, make sure not to delete the haskell-updates
branch, since it
causes all currently open Haskell-related pull-requests to be automatically closed on GitHub.
Additional Info
Here are some additional tips that didn't fit in above.
-
It is possible to start a new Hydra evaluation by logging into Hydra with your GitHub or Google account.
-
You should occasionally merge the
master
branch into thehaskell-updates
branch.In an ideal world, when we merge
haskell-updates
intomaster
, it would cause few Hydra rebuilds onmaster
. Ideally, thenixos-unstable
channel would never be prevented from progressing because of needing to wait for rebuilding Haskell packages.In order to make sure that there are a minimal number of rebuilds after merging
haskell-updates
intomaster
,master
should occasionally be merged into thehaskell-updates
branch.This is especially important after
staging-next
is merged intomaster
, since there is a high chance that this will cause all the Haskell packages to rebuild.However, as we are working on cleaning up
haskell-updates
,master
will continually progress. It may not always be possible to keep thehaskell-updates
branch fully up-to-date withmaster
without causing mass-rebuilds on thehaskell-updates
jobset. -
Make sure never to update the Hackage package hashes in
pkgs/data/misc/hackage/
, or the pinned Stackage Nightly versions on the release branches (likerelease-21.05
).This means that the
update-hackage.sh
andupdate-stackage.sh
scripts should never be used on the release branches.However, changing other files in
./.
and regenerating the package set is encouraged. This can be done withregenerate-hackage-packages.sh
as described above.
Contributor Workflow
(TODO: this section is to describe the type of workflow for non-committers to
contribute to haskell-updates
)