How To Casually Contribute to NixOS/Nixpkgs
2022/10/13
as a nix/NixOS user, you'll inherently find yourself writing code, and some of this code might be valuable to others: be it new packages, improvements upon existing packages, or new services/options. part of what makes nix successful (IMO) is the ease of sharing work and upstreaming, so i'm setting out here to show some easy workflows for doing so.
the number one thing you can do -- if you're not already -- is host your nix config publicly. for example, mine is here, and even without advertising it i occasionally hear from people in my orbit that they've found it useful. as a beginner i had some public configs i found on google or wiby which i routinely consulted, even though many of those authors probably never heard a word from the people like me who found use in their work.
Authoring New Packages
let's say you find some software you want that isn't yet packaged. first, confirm it doesn't exist by searching the package name (and probable variants) on github. remove all filters, so that you're searching both PRs and issues: maybe somebody opened a packaging request and there's already something to work off of. for example.
having confirmed that, let's make a new package. anywhere in your config, add an overlay:
{ ... }:
{
nixpkgs.overlays = [
(final: prev: {
zecwallet-lite = prev.callPackage ./zecwallet-lite { };
})
];
}
i'm just going to show the process i followed when upstreaming that zecwallet-lite package. create the package directory so that callPackage knows what to reference:
$ mkdir zecwallet-lite
$ touch zecwallet-lite/default.nix
when you reference a path in nix like ./zecwallet-lite, it'll resolve to ./zecwallet-lite/default.nix if that's valid. if you're using a flake, the path won't resolve unless it's in git:
$ git add zecwallet-lite/default.nix
now i locate the upstream package distribution. zecwallet-lite is distributed as an appimage, so i search nixpkgs for other appimages to reference:
$ cd ~/
$ git clone https://github.com/NixOS/nixpkgs.git
$ cd nixpkgs
$ rg appimage pkgs/applications -l | xargs wc -l | sort -h
19 pkgs/applications/misc/remnote/default.nix
24 pkgs/applications/video/losslesscut-bin/default.nix
25 pkgs/applications/networking/instant-messengers/caprine-bin/default.nix
26 pkgs/applications/misc/fspy/default.nix
27 pkgs/applications/audio/sonixd/default.nix
...
remnote is a simple derivation, only 19 LOC:
{ lib, fetchurl, appimageTools }:
appimageTools.wrapType2 rec {
pname = "remnote";
version = "1.7.6";
src = fetchurl {
url = "https://download.remnote.io/RemNote-${version}.AppImage";
sha256 = "sha256-yRUpLev/Fr3mOamkFgevArv2UoXgV4e6zlyv7FaQ4RM=";
};
meta = with lib; {
description = "A note-taking application focused on learning and productivity";
homepage = "https://remnote.com/";
maintainers = with maintainers; [ max-niederman ];
license = licenses.unfree;
platforms = platforms.linux;
};
}
so copy that to zecwallet-lite/default.nix and edit the values to match your appimage package. set sha256 = lib.fakeHash, and nix will tell you the actual hash.
then build the package, test it, and tweak it until you're happy. the simplest way to do that is add it to environment.systemPackages and nixos-rebuild switch. normally, if it builds, it'll run (and you might just need to patch up some icon/.desktop files, etc).
Upstreaming New Packages
so the package builds, you've used it long enough to be confident in it: time to upstream it from your silo into the main nixpkgs. even if you're strictly self-interested, upstreaming means less custom code for you to maintain, less burden keeping the version up-to-date since others may help with that, and less time spent rebuilding it when a dependency updates since it'll be present in the official nixos caches. it often takes all of ten minutes once you're familiar.
go back to your nixpkgs repo and check out the master branch. choose a reasonable folder for the package, like pkgs/applications/misc/, and copy your package there:
$ cp -R /etc/nixos/zecwallet-lite ~/nixpkgs/pkgs/applications/misc/
then take the callPackage invocation from your overlay and place it in pkgs/top-level/all-packages.nix instead. git add the file, and make sure the package builds:
~/nixpkgs$ nix build '.#zecwallet-lite'
~/nixpkgs$ ./result/bin/zecwallet-lite # make sure it runs
commit the changes, push to a github account, and open a PR. github should pre-populate the PR description with a checklist for you to complete. near as i can tell, you don't have to check every item: it's just a way for reviewers to know where to focus.
Cleanup and DRY
you could leave the duplicate package definition both in your /etc/nixos repo and in your nixpkgs repo and wait until the next release of nixos/nixpkgs before removing the former; or you can remove the package from your nixos repo now and import it by reference to avoid repetition.
if you're using a flake for your system config, it likely looks something like this right now:
{
inputs = {
nixpkgs.url = "nixpkgs/nixos-22.05";
};
outputs = { self, nixpkgs }: {
nixosConfigurations.mySystem = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./mySystem.nix
];
};
};
}
we want to somehow patch that inputs.nixpkgs so that it includes whatever changes we're trying to upstream (our new package, a package update, etc). the simplest option is to point inputs.nixpkgs.url directly at our open PR (add &rev=<git-commit> to do this), but that doesn't work if we have multiple PRs.
instead, we use a trick shown here:
{
inputs = {
nixpkgs.url = "nixpkgs/nixos-22.05";
};
outputs = { self, nixpkgs }:
let
patchedPkgs = nixpkgs.legacyPackages.x86_64-linux.applyPatches {
name = "nixpkgs-patched";
src = nixpkgs;
patches = [ ./patch1.patch ./patch2.patch ];
};
nixosSystem = import (patchedPkgs + "/nixos/lib/eval-config.nix");
in {
nixosConfigurations.mySystem = nixosSystem {
system = "x86_64-linux";
modules = [
./mySystem.nix
];
};
};
}
that is, we take the base nixpkgs repo/derivation, create a new derivation which patches the original nixpkgs, and then import the result of that derivation so that the rest of our flake refers to this patched nixpkgs.
the version above expects your patches to live in the same repo as your config flake, but if the goal is to avoid duplication we'd rather point these at the PRs hosted on github. you can use fetchpatch for this. if your PR is 180960, append .diff to the URL to get a patch. the let block above becomes this:
let
fetchpatch = nixpkgs.legacyPackages.x86_64-linux.fetchpatch;
patchedPkgs = nixpkgs.legacyPackages.x86_64-linux.applyPatches {
name = "nixpkgs-patched";
src = nixpkgs;
patches = [
(fetchpatch {
url = "https://github.com/NixOS/nixpkgs/pull/180960.diff";
sha256 = "sha256-HVVj/T3yQtjYBoxXpoPiG9Zar/eik9IoDVDhTOehBdY=";
})
];
};
nixosSystem = import (patchedPkgs + "/nixos/lib/eval-config.nix");
and we're set! as before, if you don't know the hash just set it to nixpkgs.lib.fakeHash. you can freely push to your open PR: to have the changes reflect on your system just reset/update the hash and nixos-rebuild switch. once your PR is merged, the next nix flake update might include your patch already, in which case nix will error when building nixpkgs-patched. at that point you can safely remove this entry from patches.
Upstreaming Other Changes
since this approach is ultimately just applying patches onto nixpkgs, it's not limited to only new packages. to update a package version, add a new config option, etc, you can follow effectively the same process: clone nixpkgs, checkout the master branch, make a change and nix build the result, commit & open a PR, and fetchpatch the PR into your flake.
if you're making changes in a hot area of the codebase, the diff from your PR (which targets master) might not apply cleanly to your flake (which pulls from a release or nixos-unstable). in that case you can git log the files you're changing and use the same fetchpatch trick to cherry-pick whatever prerequisite commits you need so that your patch applies.
Closing Thoughts
i described here the workflow i found easiest to get started with. with something as configurable as nix, many people use many different workflows. some prefer maintaining their own long-running fork of nixpkgs wherein they cherry-pick all these PRs and periodically rebase against nixos-unstable (or a release branch), and point their flake directly at their fork of nixpkgs. your workflow will surely evolve as you settle in, but hopefully this gives you enough inspiration to get started :-)
finally, don't be overly intimidated by the nixos-unstable branch. it might not be quite as unstable as the name suggests: i've yet to encounter any show-stopping issues, 90% of the instability is just that packages don't build and need to be pinned or fixed (or you skip the update that day). there's a point where the inconvenience of nixos-unstable is less than the inconvenience of maintaining patches that have already landed upstream. if you're upstreaming enough that you feel that pain point, you'll do fine on the unstable branch.
as always, don't hesitate to contact me or reach out on the forums/chat.