It feels like over the past year, Nix (and NixOS) have gained a significant amount of momentum, helped by introductions like Zero to Nix and nix.dev. With the proliferation of new resources for learning about Nix/NixOS, I decided to migrate my homelab (an old Intel NUC running Proxmox) to NixOS and move to a declarative setup.

Installation

Installation varies; either a full Nix-based operating system (ie NixOS) may be used, or only the Nix package manager.

Nix as a package manager

On non-NixOS platforms (MacOS, Ubuntu, Fedora, etc) installing the Nix package manager is necessary. Now, running the Determinate Systems Nix installer should be enough.

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

NixOS

If using NixOS, install from the ISO.

At this point, the main benefit of NixOS should become apparent; instead of having to (re)configure the entire system, a flake.nix containing a full system configuration may be used. If starting from scratch (ie no flake.nix yet configured), this portion may be skipped.

During the install process, pause once partitions are mounted to /mnt so that secret management can be set up with agenix.

Part of setting up a homelab is secret management - various API keys, passwords, addresses, etc that all need to be stored somewhere. Since all configuration is (ideally) declarative, this means checking in secrets to a Git repo—not ideal! There’s a variety of secret-management tools for Nix, although I’m choosing to use agenix.

Getting secret management set up requires generating an SSH key which is added to Github so that a private repo can be cloned and the hardware_configuration.nix file added/ updated to the repo.

# install agenix to get secrets loaded
nix-channel --add https://github.com/ryantm/agenix/archive/main.tar.gz agenix
nix-channel --update

# Open up a shell to get git:
nix-shell -p git nixFlakes

# copy ssh keys over so that we can authenticate with github
mkdir ~/.ssh
# either create a new key and copy it to github, or use an existing ssh key
vim ~/.ssh/id_ed25519 # enter private key that is used to authenticate with Github (stored in 1password)
chmod 600 ~/.ssh/id_ed25519

# add new keys for agenix
mkdir -p /mnt/etc/secrets/initrd
ssh-keygen -t ed25519 -N "" -f /mnt/etc/secrets/initrd/ssh_host_ed25519_key
cat /mnt/etc/secrets/initrd/ssh_host_ed25519_key.pub # copy the public key
# re-encrypt with this new key, then pull the repo again to get newly encrypted files
cd /mnt/etc/nixos
git pull

# once shell loaded, clone the repo to /mnt/etc/nixos
git clone [email protected]:username/path_to_repo /mnt/etc/nixos
cd /mnt/etc/nixos

# update flake
nix --experimental-features 'flakes nix-command' flake update

# prepare for new hardware-configuration
rm /mnt/etc/nixos/hosts/nixos/nixos-proxmox/hardware-configuration.nix

# generate new config (ignore the generated configuration.nix)
nixos-generate-config --root /mnt
mv /mnt/etc/nixos/hardware-configuration.nix /mnt/etc/nixos/hosts/nixos/nixos-proxmos/
rm /mnt/etc/nixos/configuration.nix

# make sure we're in the right directory
cd /mnt/etc/nixos
git add --all # add the new hardware config
git commit -m "update hardware config" # commit it
git push # and push to github

# needs more space to build (at least 4GB)
mount -o remount,size=4G /run/user/0

# install
nixos-install --flake .#nixos -j 4

Once these commands are finished, the system should be installed. nixos in the final command (nixos-install --flake .#nixos -j 4) should be changed to whatever the host name of the system in flake.nix is. If installing/ migrating to a new system, create a new configuration in flake.nix during the install process (copying an existing one as a template).

Post-Install

If starting from scratch, the benefits of Nix/NixOS won’t be so apparent. It takes a while to configure everything; however, the benefit to spending this time up-front is that recreating the system from scratch takes only minutes.

On NixOS, every time flake.nix is edited, migrating the sysytem to a new configuration is done via nixos-rebuild switch --flake . when in the same directory as a flake.nix containing the system configuration.

On MacOS, system changes can be applied by nix-darwin with darwin-rebuild switch. The first time this command is run, darwin-rebuild may not yet be added to the path. In this case, first build with nix build .#darwinConfigurations.(hostname) and then run darwin-rebuild as ./result/sw/bin/darwin-rebuild switch --flake .

I’m partial to using Tailscale for accessing all my devices, so tailscale up --ssh needs to be run after setup to add to a tailnet.

Homelab Configuration

Here’s a more detailed accounting of my Nix/NixOS configuration as it pertains to homelab setup/ an annotated version of the flake.nix and related files in my repo

# flake.nix
# ...
nixosConfigurations = {
      nixos = nixpkgs.lib.nixosSystem {
        # nixos-rebuild switch --flake .
        system = "x86_64-linux";
        pkgs = legacyPackages.x86_64-linux;
        modules =
          [
            home-manager.nixosModules.home-manager
            agenix.nixosModules.default
            ./hosts/nixos/nixos-proxmox
          ]
          ++ (builtins.attrValues nixosModules);
        specialArgs = {inherit inputs;};
      };
    };
#...

The homelab runs off an old Intel NUC as a VM within Proxmox. NixOS is installed from the defualt ISO; with Proxmox, I can open a web console to the installer and set a root password. Once a root password has been set, the rest of the installer can be run over SSH.

nixosSystem is given three modules: home-manager (for dotfile management and user configuration), agenix (for secret management), and the system configuration in hosts/nixos/nixos-proxmos/default.nix

nixos-proxmox/default.nix

nixos-proxmox/default.nix contains all system-specific configuration necessary to set up users and services for the homelab. As the file is relatively brief, it’s included below.

# nixos-proxmos/default.nix

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).
{
  inputs,
  pkgs,
  config,
  ...
}: {
  imports = [
    # Include the results of the hardware scan.
    ./hardware-configuration.nix

    ## Common
    ../../common # shared between NixOS and Darwin
    ../shared.nix # shared between NixOS

    ## Services
    ./services/acme.nix
    ./services/caddy.nix
    ./services/tailscale.nix
    ## Media
    ./services/sabnzbd
    ./services/plex.nix
    ./services/prowlarr.nix
    ./services/radarr.nix
    ./services/sonarr.nix

    ## Backup
    ./services/rclone.nix
    ./services/restic/healthchecks.nix
    ./services/restic/local.nix
    ./services/restic/b2.nix

    ## Dashboard
    ./services/dashy.nix
  ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking = {
    hostName = "nixos"; # Define your hostname.
    domain = "josephstahl.com";
    firewall.enable = false;
    networkmanager.enable = true; # Easiest to use and most distros use this by default.
  };
  systemd.services.NetworkManager-wait-online.enable = false; # causes problems with tailscale

  # Set your time zone.
  time.timeZone = "America/New_York";

  # Select internationalisation properties.
  i18n.defaultLocale = "en_US.UTF-8";

  age.secrets.smb = {
    file = ../../../secrets/smb.age;
    owner = "root";
    group = "root";
  };

  fileSystems."/mnt/nas" = {
    device = "//192.168.1.10/public";
    fsType = "cifs";
    options = let
      # prevent hanging on network changes
      automount_opts = "x-systemd.automount,noauto,x-systemd.idle-timeout=600,x-systemd.device-timeout=5s,x-systemd.mount-timeout=5s,gid=media,file_mode=0775,dir_mode=0775";
    in ["${automount_opts},credentials=${config.age.secrets.smb.path}"];
  };

  # List services that you want to enable:
  services = {
    qemuGuest.enable = true;
  };

  # Copy the NixOS configuration file and link it from the resulting system
  # (/run/current-system/configuration.nix). This is useful in case you
  # accidentally delete configuration.nix.
  system.copySystemConfiguration = false; # true seems to break usage with flakes

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "22.11"; # Did you read the comment?
}

The file imports both ../../common (which contains configuration relevant to both NixOS and Nix-Darwin hosts) and ../shared.nix (which contains configuration relevant to all NixOS hosts). Additionally, all services are configured here as imports so that service-specific configuration is kept limited to each file.

Here, some agenix configuration can be seen; for example, with getting Samba login credentials for accessing a share on my NAS. Using agenix for secrets is a two-step process; the file containing the (encrypted) secret is declared:

age.secrets.smb = {
  file = ../../../secrets/smb.age;
  owner = "root";
  group = "root";
  };

and then used:

fileSystems."/mnt/nas" = {
  device = "//192.168.1.10/public";
  fsType = "cifs";
  options = let
      # prevent hanging on network changes
      automount_opts = "x-systemd.automount,noauto,x-systemd.idle-timeout=600,x-systemd.device-timeout=5s,x-systemd.mount-timeout=5s,gid=media,file_mode=0775,dir_mode=0775";
  in ["${automount_opts},credentials=${config.age.secrets.smb.path}"];
};

Service configuration

Each service is configured via a service file, such as in services/plex.nix

{
  pkgs,
  config,
  ...
}: let
  inherit (config.networking) domain hostName;
  fqdn = "${hostName}.${domain}";
in {
  services.plex = {
    enable = true;
    group = "media";
    package = pkgs.unstable.plex;
  };

  services.caddy.virtualHosts."plex.${fqdn}" = {
    extraConfig = ''
      reverse_proxy http://localhost:32400
    '';
    useACMEHost = fqdn;
  };

  # Ensure that plex waits for the downloads and media directories to be
  # available.
  systemd.services.plex = {
    wantedBy = ["multi-user.target"];
    after = [
      "network.target"
      "mnt-nas.automount"
    ];
  };
}

Each service file is imported into the main system configuration, so configuration is not isolated to that service. This allows for reverse-proxy configuration for each service to be stored in that service’s file, instead of all having to be configured in a Caddyfile or similar. As services are added/ changed, they automatically are set up with a reverse proxy and HTTPS support (via NixOS’s built-in support for ACME/ Let’s Encrypt).

Agenix

Secrets are managed with Agenix. It allows for secrets to be stored in public Github repos, protected with age/ encrypted with the system’s SSH private key.

If agenix is not yet installed on the system (ie when setting up a new system) and secret management is necessary, running nix develop in the repo will automatically install agenix as part of the setup. Generally, adding/ updating/ re-encrypting secrets is done on an already-set-up system where agenix will already be installed.

The secrets directory stores secrets.nix, an unencrypted file containing configuration for agenix.

let
  joseph = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICxKQtKkR7jkse0KMDvVZvwvNwT0gUkQ7At7Mcs9GEop";
  system = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINmAAEPEulCuQrrU/T2h0pLDdr6BIMycaCa7IEJ24G7X root@nixos";
  allKeys = [joseph system];
in {
  "my_secret.age".publicKeys = allKeys;
  #...
}

Note that this file contains the user’s public SSH key as well as public SSH keys each system that Nix is installed on.

To add a new secret, add a line to secrets.nix describing which keys can be used to read the secret. For example, "secret_2.age".publicKeys = allKeys;. Once the file is saved, run agenix -e my_secret.age which will load the default text editor where the contents of the file can be added to/ updated. When the editor is closed, agenix will automatically encrypt the file with the provided SSH key.

Now, the secret can be used in any Nix file in the repository. Add age.secrets.secret1.file = ../path/to/my_secret.age; to the file configuration. There’s no importing/ loading of agenix to worry about, since it’s declared as a module for nixosConfiguration in flake.nix.

Finally, the secret can be used anywhere Nix expects a file containing a password, such as in users.users.joseph.passwordFile.

{
  users.users.joseph = {
    isNormalUser = true;
    passwordFile = config.age.secrets.my_secret.path;
  };
}

Note that in this case, Nix expects the passwordFile contents to be the password hash that will be copied to /etc/shadow; adding this file to agenix results in a doubly-encrypted password. Once with mkpassword -m sha-512 to create the password hash, and then a second time when using agenix to create the encrypted .age file.

Packages

Occasionally, I have a need for a package that either isn’t in NixOS/ nixpkgs yet, or is out-of-date. Here, being able to create custom packages is useful. flake.nix is already set up to add all packages in ./pkgs to the default package set, so that a package may be installed the same as any other package in the nixpkgs repository.

For example, here’s one for recyclarr (in packages/recyclarr/default.nix):

{
  lib,
  nixosTests,
  stdenv,
  fetchurl,
  pkgs,
}: let
  os =
    if stdenv.isDarwin
    then "osx"
    else "linux";
  arch =
    {
      x86_64-linux = "x64";
      aarch64-linux = "arm64";
      x86_64-darwin = "x64";
      aarch64-darwin = "arm64";
    }
    ."${stdenv.hostPlatform.system}"
    or (throw "Unsupported system: ${stdenv.hostPlatform.system}");
  hash =
    {
      x64-linux_hash = "sha256-96j29Su983CaCVOBHoGduY/0BCWY6cONwub7yCFFIgM=";
      arm64-linux_hash = "sha256-/Xqa2IbTafbYytKG/8jLvNjKAnNcgValDa15nvbzSR8=";
      x64-osx_hash = "sha256-FbDeQd7z5KCIPRBbB/mnnATnSYMaoehBlUljSw87L7M=";
      arm64-osx_hash = "sha256-KTYYEbq2MZaHzxQHO01qeH6PQ7zHy/gW5HaTIDiO0Z8=";
    }
    ."${arch}-${os}_hash";
in
  stdenv.mkDerivation rec {
    pname = "recyclarr";
    version = "v4.3.0";

    src = fetchurl {
      url = "https://github.com/recyclarr/recyclarr/releases/download/${version}/recyclarr-${os}-${arch}.tar.xz";
      hash = hash;
    };

    # Work around the "unpacker appears to have produced no directories"
    # case that happens when the archive doesn't have a subdirectory.
    # setSourceRoot = "sourceRoot=`pwd`";
    sourceRoot = ".";

    installPhase = ''
      runHook preInstall
      mkdir -p $out/bin
      cp -r * $out/bin

      runHook postInstall
    '';

    dontFixup = true; # breaks self-contained .net apps

    meta = with lib; {
      description = "A command-line application that will automatically synchronize recommended settings from the TRaSH guides to your Sonarr/Radarr instances.";
      homepage = "https://github.com/recyclarr/recyclarr";
      license = licenses.mit;
      platforms = ["x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin"];
    };
  }

This allows for recyclarr to be installed on any MacOS or Linux system as if it existed with all the other packages in nixpkgs.

Overlays

Overlays are a natural extension of packages; they are what allow for the packages declared in packages/ to be “overlaid” on the nixpkgs repository so that they all appear as one repository. Overlays can also be used to override package versions or provide additional package versions, optionally using a namespace within nixpkgs (to avoid conflicts).

This is used, for example, by the agenix flake to provide a agenix package that can be installed despite not being listed in nixpkgs.

Sample overlay configuration

# flake.nix
overlays = import ./overlays {inherit inputs;};

# overlays.nix
{inputs, ...}: {
  agenix = inputs.agenix.overlays.default;
  zig = inputs.zig.overlays.default;
  additions = final: prev:
    import ../pkgs {
      pkgs = final;
      inherit inputs;
    };
  unstable = final: prev: {
    unstable = import inputs.nixpkgs-unstable {
      system = final.system;
      config.allowUnfree = true;
    };
  };
  modifications = final: prev: {
    # override lego version (ACME certificates) with newest rev from github
    # which supports google domains
    # TODO: delete this once v4.11 is released to nixos unstable channel
    lego = let
      version = "unstable-2023-04-07";
      pname = "lego";
      src = prev.fetchFromGitHub {
        owner = "go-acme";
        repo = pname;
        rev = "1a16d1ab9b275836ce9fc45ea7871ab4d3811879";
        sha256 = "sha256-ggkeq2ccw0UyxyeMlxuMbEF0dCuyKgirc06m0MmsApw=";
      };
    in (prev.lego.override rec {
      buildGoModule = args:
        prev.buildGoModule (args
          // {
            vendorHash = "sha256-6dfwAsCxEYksZXqSWYurAD44YfH4h5p5P1aYZENjHSs=";
            inherit src version;
          });
    });
  };
}

This overlay illustrates a number of examples:

• agenix and zig: add community-created overlays to our set of overlays
• additions: import the packages we created earlier. Because there is not any assignment to a variable, they won’t be namespaced in the set of nixpkgs
• unstable: include all packages from nixpkgs-unstable in the unstable namespace. These packages will be installable as pkgs.unstable.vim whereas the stable version will be pkgs.vim.
• modifications: this overlay allows for modifying packages in nixpkgs that are out of date. In this case, the lego client (used for getting HTTPS certificates from Let’s Encrypt) hasn’t added support to the HEAD branch for DNS-01 challenges with Google Domains, but there’s not yet a release with this feature. We override the package with our own version, built from the latest Github commit (as of the time of this writing). Ideally, packages in modifications are deleted as changes/ updates are pushed upstream.

Deploy-RS

It’s a pain to have to SSH to my homelab server and run git pull each time I change my configuration. deploy-rs allows for pushing changes to a remote server.

# flake.nix
# ...
deploy.nodes = {
      nixos = {
        hostname = "nixos.josephstahl.com";
        profiles.system = {
          path = deploy-rs.lib.x86_64-linux.activate.nixos self.nixosConfigurations.nixos;
          sshUser = "joseph";
          user = "root";
          sshOpts = ["-t"];
          magicRollback = false; # breaks remote sudo
          remoteBuild = true; # since it may be cross-platform
        };
      };
    };

Since my homelab is accessible from anywhere with Tailscale, deploy-rs can be used to update my NixOS configuration from anywhere.

just deploy is enough to push my NixOS configuration to the homelab server, rebuild the changes (rebuilds done on remote server to avoid cross-compliation with M2 Mac), and deploy the build, with a pause to ask for my sudo password to apply the changes.

MacOS (nix-darwin)

Using Nix as a package manager is not limited to just NixOS or even Linux. It can function alongside or as a replacement for Homebrew on MacOS package management, as well as managing a user’s dotfiles (with Home Manager) and system configuration (with nix-darwin).

Installation

On MacOS, the Determinate Systems nix installer is the easiest and most reliable way to get Nix installed.

Configuration

MacOS configuration is divided into three parts:

1. system configuration
2. package management
3. dotfiles

System configuration in hosts/darwin/shared is shared among all MacOS systems (Nix settings, Mac & Finder options, etc.). All GUI apps are installed via Homebrew with the nix-darwin homebrew module, while CLI apps are installed via environment.systemPackages in darwin/shared.nix and common/default.nix. Finally, dotfiles are managed by Home-Manager via configuration in home/joseph which is also shared with NixOS systems.

To manage MacOS-specific settings, import pkgs.stdenv.hostPlatform isDarwin allows for calling isDarwin as a part of if/then/else statements.

Conclusion

The learning curve to Nix/NixOS is trecherous and the error messages are often inscrutable, but when it works, it’s impressive. I can wipe the NixOS VM on Proxmox and have the entire system reinstalled and configured within minutes.

Installing new packages (and cleanly uninstalling them) is as simple as running nixos-rebuild switch to update a system with the new configuration in flake.nix. If a package does not yet exist on Nixpkgs or is out of date, a new package can be made from a Github release (or the source code on Github, with compliation managed by the package configuration file).

Some software is still not yet properly packaged for NixOS, or needs additional setup—for example, the Sabnzbd module doesn’t allow for setting/ overriding the contents of Sabnzbd’s .ini configuration file.

Still, it’s nice to share so much configuration between my Mac and my homelab server (and to be able to deploy changes to the server via just deploy). The momentum around Nix is reassuring that error messages will improve and it will slowly become more approachable for beginners like myself. In the mean time, be ready to devote hours to the project to play around with things and learn how it all works.