Joseph's Blog

Nix for MacOS and a homelab server

It feels like over the past year, Nix (and NixOS) have gained a significant amount of momentum, helped by introductions like Zero to Nix and nix.dev. With the proliferation of new resources for learning about Nix/NixOS, I decided to migrate my homelab (an old Intel NUC running Proxmox) to NixOS and move to a declarative setup.

Installation

Installation varies; either a full Nix-based operating system (ie NixOS) may be used, or only the Nix package manager.

Nix as a package manager

On non-NixOS platforms (MacOS, Ubuntu, Fedora, etc) installing the Nix package manager is necessary. Now, running the Determinate Systems Nix installer should be enough.

1curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

NixOS

If using NixOS, install from the ISO.

At this point, the main benefit of NixOS should become apparent; instead of having to (re)configure the entire system, a flake.nix containing a full system configuration may be used. If starting from scratch (ie no flake.nix yet configured), this portion may be skipped.

During the install process, pause once partitions are mounted to /mnt so that secret management can be set up with agenix.

Part of setting up a homelab is secret management - various API keys, passwords, addresses, etc that all need to be stored somewhere. Since all configuration is (ideally) declarative, this means checking in secrets to a Git repo—not ideal! There’s a variety of secret-management tools for Nix, although I’m choosing to use agenix.

Getting secret management set up requires generating an SSH key which is added to Github so that a private repo can be cloned and the hardware_configuration.nix file added/ updated to the repo.

 1# install agenix to get secrets loaded
 2nix-channel --add https://github.com/ryantm/agenix/archive/main.tar.gz agenix
 3nix-channel --update
 4
 5# Open up a shell to get git:
 6nix-shell -p git nixFlakes
 7
 8# copy ssh keys over so that we can authenticate with github
 9mkdir ~/.ssh
10# either create a new key and copy it to github, or use an existing ssh key
11vim ~/.ssh/id_ed25519 # enter private key that is used to authenticate with Github (stored in 1password)
12chmod 600 ~/.ssh/id_ed25519
13
14# add new keys for agenix
15mkdir -p /mnt/etc/secrets/initrd
16ssh-keygen -t ed25519 -N "" -f /mnt/etc/secrets/initrd/ssh_host_ed25519_key
17cat /mnt/etc/secrets/initrd/ssh_host_ed25519_key.pub # copy the public key
18# re-encrypt with this new key, then pull the repo again to get newly encrypted files
19cd /mnt/etc/nixos
20git pull
21
22# once shell loaded, clone the repo to /mnt/etc/nixos
23git clone [email protected]:username/path_to_repo /mnt/etc/nixos
24cd /mnt/etc/nixos
25
26# update flake
27nix --experimental-features 'flakes nix-command' flake update
28
29# prepare for new hardware-configuration
30rm /mnt/etc/nixos/hosts/nixos/nixos-proxmox/hardware-configuration.nix
31
32# generate new config (ignore the generated configuration.nix)
33nixos-generate-config --root /mnt
34mv /mnt/etc/nixos/hardware-configuration.nix /mnt/etc/nixos/hosts/nixos/nixos-proxmos/
35rm /mnt/etc/nixos/configuration.nix
36
37# make sure we're in the right directory
38cd /mnt/etc/nixos
39git add --all # add the new hardware config
40git commit -m "update hardware config" # commit it
41git push # and push to github
42
43# needs more space to build (at least 4GB)
44mount -o remount,size=4G /run/user/0
45
46# install
47nixos-install --flake .#nixos -j 4

Once these commands are finished, the system should be installed. nixos in the final command (nixos-install --flake .#nixos -j 4) should be changed to whatever the host name of the system in flake.nix is. If installing/ migrating to a new system, create a new configuration in flake.nix during the install process (copying an existing one as a template).

Post-Install

If starting from scratch, the benefits of Nix/NixOS won’t be so apparent. It takes a while to configure everything; however, the benefit to spending this time up-front is that recreating the system from scratch takes only minutes.

On NixOS, every time flake.nix is edited, migrating the sysytem to a new configuration is done via nixos-rebuild switch --flake . when in the same directory as a flake.nix containing the system configuration.

On MacOS, system changes can be applied by nix-darwin with darwin-rebuild switch. The first time this command is run, darwin-rebuild may not yet be added to the path. In this case, first build with nix build .#darwinConfigurations.(hostname) and then run darwin-rebuild as ./result/sw/bin/darwin-rebuild switch --flake .

I’m partial to using Tailscale for accessing all my devices, so tailscale up --ssh needs to be run after setup to add to a tailnet.

Homelab Configuration

Here’s a more detailed accounting of my Nix/NixOS configuration as it pertains to homelab setup/ an annotated version of the flake.nix and related files in my repo

 1# flake.nix
 2# ...
 3nixosConfigurations = {
 4      nixos = nixpkgs.lib.nixosSystem {
 5        # nixos-rebuild switch --flake .
 6        system = "x86_64-linux";
 7        pkgs = legacyPackages.x86_64-linux;
 8        modules =
 9          [
10            home-manager.nixosModules.home-manager
11            agenix.nixosModules.default
12            ./hosts/nixos/nixos-proxmox
13          ]
14          ++ (builtins.attrValues nixosModules);
15        specialArgs = {inherit inputs;};
16      };
17    };
18#...

The homelab runs off an old Intel NUC as a VM within Proxmox. NixOS is installed from the defualt ISO; with Proxmox, I can open a web console to the installer and set a root password. Once a root password has been set, the rest of the installer can be run over SSH.

nixosSystem is given three modules: home-manager (for dotfile management and user configuration), agenix (for secret management), and the system configuration in hosts/nixos/nixos-proxmos/default.nix

nixos-proxmox/default.nix

nixos-proxmox/default.nix contains all system-specific configuration necessary to set up users and services for the homelab. As the file is relatively brief, it’s included below.

 1# nixos-proxmos/default.nix
 2
 3# Edit this configuration file to define what should be installed on
 4# your system.  Help is available in the configuration.nix(5) man page
 5# and in the NixOS manual (accessible by running ‘nixos-help’).
 6{
 7  inputs,
 8  pkgs,
 9  config,
10  ...
11}: {
12  imports = [
13    # Include the results of the hardware scan.
14    ./hardware-configuration.nix
15
16    ## Common
17    ../../common # shared between NixOS and Darwin
18    ../shared.nix # shared between NixOS
19
20    ## Services
21    ./services/acme.nix
22    ./services/caddy.nix
23    ./services/tailscale.nix
24    ## Media
25    ./services/sabnzbd
26    ./services/plex.nix
27    ./services/prowlarr.nix
28    ./services/radarr.nix
29    ./services/sonarr.nix
30
31    ## Backup
32    ./services/rclone.nix
33    ./services/restic/healthchecks.nix
34    ./services/restic/local.nix
35    ./services/restic/b2.nix
36
37    ## Dashboard
38    ./services/dashy.nix
39  ];
40
41  # Use the systemd-boot EFI boot loader.
42  boot.loader.systemd-boot.enable = true;
43  boot.loader.efi.canTouchEfiVariables = true;
44
45  networking = {
46    hostName = "nixos"; # Define your hostname.
47    domain = "josephstahl.com";
48    firewall.enable = false;
49    networkmanager.enable = true; # Easiest to use and most distros use this by default.
50  };
51  systemd.services.NetworkManager-wait-online.enable = false; # causes problems with tailscale
52
53  # Set your time zone.
54  time.timeZone = "America/New_York";
55
56  # Select internationalisation properties.
57  i18n.defaultLocale = "en_US.UTF-8";
58
59  age.secrets.smb = {
60    file = ../../../secrets/smb.age;
61    owner = "root";
62    group = "root";
63  };
64
65  fileSystems."/mnt/nas" = {
66    device = "//192.168.1.10/public";
67    fsType = "cifs";
68    options = let
69      # prevent hanging on network changes
70      automount_opts = "x-systemd.automount,noauto,x-systemd.idle-timeout=600,x-systemd.device-timeout=5s,x-systemd.mount-timeout=5s,gid=media,file_mode=0775,dir_mode=0775";
71    in ["${automount_opts},credentials=${config.age.secrets.smb.path}"];
72  };
73
74  # List services that you want to enable:
75  services = {
76    qemuGuest.enable = true;
77  };
78
79  # Copy the NixOS configuration file and link it from the resulting system
80  # (/run/current-system/configuration.nix). This is useful in case you
81  # accidentally delete configuration.nix.
82  system.copySystemConfiguration = false; # true seems to break usage with flakes
83
84  # This value determines the NixOS release from which the default
85  # settings for stateful data, like file locations and database versions
86  # on your system were taken. It‘s perfectly fine and recommended to leave
87  # this value at the release version of the first install of this system.
88  # Before changing this value read the documentation for this option
89  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
90  system.stateVersion = "22.11"; # Did you read the comment?
91}

The file imports both ../../common (which contains configuration relevant to both NixOS and Nix-Darwin hosts) and ../shared.nix (which contains configuration relevant to all NixOS hosts). Additionally, all services are configured here as imports so that service-specific configuration is kept limited to each file.

Here, some agenix configuration can be seen; for example, with getting Samba login credentials for accessing a share on my NAS. Using agenix for secrets is a two-step process; the file containing the (encrypted) secret is declared:

1age.secrets.smb = {
2  file = ../../../secrets/smb.age;
3  owner = "root";
4  group = "root";
5  };

and then used:

1fileSystems."/mnt/nas" = {
2  device = "//192.168.1.10/public";
3  fsType = "cifs";
4  options = let
5      # prevent hanging on network changes
6      automount_opts = "x-systemd.automount,noauto,x-systemd.idle-timeout=600,x-systemd.device-timeout=5s,x-systemd.mount-timeout=5s,gid=media,file_mode=0775,dir_mode=0775";
7  in ["${automount_opts},credentials=${config.age.secrets.smb.path}"];
8};

Service configuration

Each service is configured via a service file, such as in services/plex.nix

 1{
 2  pkgs,
 3  config,
 4  ...
 5}: let
 6  inherit (config.networking) domain hostName;
 7  fqdn = "${hostName}.${domain}";
 8in {
 9  services.plex = {
10    enable = true;
11    group = "media";
12    package = pkgs.unstable.plex;
13  };
14
15  services.caddy.virtualHosts."plex.${fqdn}" = {
16    extraConfig = ''
17      reverse_proxy http://localhost:32400
18    '';
19    useACMEHost = fqdn;
20  };
21
22  # Ensure that plex waits for the downloads and media directories to be
23  # available.
24  systemd.services.plex = {
25    wantedBy = ["multi-user.target"];
26    after = [
27      "network.target"
28      "mnt-nas.automount"
29    ];
30  };
31}

Each service file is imported into the main system configuration, so configuration is not isolated to that service. This allows for reverse-proxy configuration for each service to be stored in that service’s file, instead of all having to be configured in a Caddyfile or similar. As services are added/ changed, they automatically are set up with a reverse proxy and HTTPS support (via NixOS’s built-in support for ACME/ Let’s Encrypt).

Agenix

Secrets are managed with Agenix. It allows for secrets to be stored in public Github repos, protected with age/ encrypted with the system’s SSH private key.

If agenix is not yet installed on the system (ie when setting up a new system) and secret management is necessary, running nix develop in the repo will automatically install agenix as part of the setup. Generally, adding/ updating/ re-encrypting secrets is done on an already-set-up system where agenix will already be installed.

The secrets directory stores secrets.nix, an unencrypted file containing configuration for agenix.

1let
2  joseph = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICxKQtKkR7jkse0KMDvVZvwvNwT0gUkQ7At7Mcs9GEop";
3  system = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINmAAEPEulCuQrrU/T2h0pLDdr6BIMycaCa7IEJ24G7X root@nixos";
4  allKeys = [joseph system];
5in {
6  "my_secret.age".publicKeys = allKeys;
7  #...
8}

Note that this file contains the user’s public SSH key as well as public SSH keys each system that Nix is installed on.

To add a new secret, add a line to secrets.nix describing which keys can be used to read the secret. For example, "secret_2.age".publicKeys = allKeys;. Once the file is saved, run agenix -e my_secret.age which will load the default text editor where the contents of the file can be added to/ updated. When the editor is closed, agenix will automatically encrypt the file with the provided SSH key.

Now, the secret can be used in any Nix file in the repository. Add age.secrets.secret1.file = ../path/to/my_secret.age; to the file configuration. There’s no importing/ loading of agenix to worry about, since it’s declared as a module for nixosConfiguration in flake.nix.

Finally, the secret can be used anywhere Nix expects a file containing a password, such as in users.users.joseph.passwordFile.

1{
2  users.users.joseph = {
3    isNormalUser = true;
4    passwordFile = config.age.secrets.my_secret.path;
5  };
6}

Note that in this case, Nix expects the passwordFile contents to be the password hash that will be copied to /etc/shadow; adding this file to agenix results in a doubly-encrypted password. Once with mkpassword -m sha-512 to create the password hash, and then a second time when using agenix to create the encrypted .age file.

Packages

Occasionally, I have a need for a package that either isn’t in NixOS/ nixpkgs yet, or is out-of-date. Here, being able to create custom packages is useful. flake.nix is already set up to add all packages in ./pkgs to the default package set, so that a package may be installed the same as any other package in the nixpkgs repository.

For example, here’s one for recyclarr (in packages/recyclarr/default.nix):

 1{
 2  lib,
 3  nixosTests,
 4  stdenv,
 5  fetchurl,
 6  pkgs,
 7}: let
 8  os =
 9    if stdenv.isDarwin
10    then "osx"
11    else "linux";
12  arch =
13    {
14      x86_64-linux = "x64";
15      aarch64-linux = "arm64";
16      x86_64-darwin = "x64";
17      aarch64-darwin = "arm64";
18    }
19    ."${stdenv.hostPlatform.system}"
20    or (throw "Unsupported system: ${stdenv.hostPlatform.system}");
21  hash =
22    {
23      x64-linux_hash = "sha256-96j29Su983CaCVOBHoGduY/0BCWY6cONwub7yCFFIgM=";
24      arm64-linux_hash = "sha256-/Xqa2IbTafbYytKG/8jLvNjKAnNcgValDa15nvbzSR8=";
25      x64-osx_hash = "sha256-FbDeQd7z5KCIPRBbB/mnnATnSYMaoehBlUljSw87L7M=";
26      arm64-osx_hash = "sha256-KTYYEbq2MZaHzxQHO01qeH6PQ7zHy/gW5HaTIDiO0Z8=";
27    }
28    ."${arch}-${os}_hash";
29in
30  stdenv.mkDerivation rec {
31    pname = "recyclarr";
32    version = "v4.3.0";
33
34    src = fetchurl {
35      url = "https://github.com/recyclarr/recyclarr/releases/download/${version}/recyclarr-${os}-${arch}.tar.xz";
36      hash = hash;
37    };
38
39    # Work around the "unpacker appears to have produced no directories"
40    # case that happens when the archive doesn't have a subdirectory.
41    # setSourceRoot = "sourceRoot=`pwd`";
42    sourceRoot = ".";
43
44    installPhase = ''
45      runHook preInstall
46      mkdir -p $out/bin
47      cp -r * $out/bin
48
49      runHook postInstall
50    '';
51
52    dontFixup = true; # breaks self-contained .net apps
53
54    meta = with lib; {
55      description = "A command-line application that will automatically synchronize recommended settings from the TRaSH guides to your Sonarr/Radarr instances.";
56      homepage = "https://github.com/recyclarr/recyclarr";
57      license = licenses.mit;
58      platforms = ["x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin"];
59    };
60  }

This allows for recyclarr to be installed on any MacOS or Linux system as if it existed with all the other packages in nixpkgs.

Overlays

Overlays are a natural extension of packages; they are what allow for the packages declared in packages/ to be “overlaid” on the nixpkgs repository so that they all appear as one repository. Overlays can also be used to override package versions or provide additional package versions, optionally using a namespace within nixpkgs (to avoid conflicts).

This is used, for example, by the agenix flake to provide a agenix package that can be installed despite not being listed in nixpkgs.

Sample overlay configuration

1# flake.nix
2overlays = import ./overlays {inherit inputs;};

 1# overlays.nix
 2{inputs, ...}: {
 3  agenix = inputs.agenix.overlays.default;
 4  zig = inputs.zig.overlays.default;
 5  additions = final: prev:
 6    import ../pkgs {
 7      pkgs = final;
 8      inherit inputs;
 9    };
10  unstable = final: prev: {
11    unstable = import inputs.nixpkgs-unstable {
12      system = final.system;
13      config.allowUnfree = true;
14    };
15  };
16  modifications = final: prev: {
17    # override lego version (ACME certificates) with newest rev from github
18    # which supports google domains
19    # TODO: delete this once v4.11 is released to nixos unstable channel
20    lego = let
21      version = "unstable-2023-04-07";
22      pname = "lego";
23      src = prev.fetchFromGitHub {
24        owner = "go-acme";
25        repo = pname;
26        rev = "1a16d1ab9b275836ce9fc45ea7871ab4d3811879";
27        sha256 = "sha256-ggkeq2ccw0UyxyeMlxuMbEF0dCuyKgirc06m0MmsApw=";
28      };
29    in (prev.lego.override rec {
30      buildGoModule = args:
31        prev.buildGoModule (args
32          // {
33            vendorHash = "sha256-6dfwAsCxEYksZXqSWYurAD44YfH4h5p5P1aYZENjHSs=";
34            inherit src version;
35          });
36    });
37  };
38}

This overlay illustrates a number of examples:

Deploy-RS

It’s a pain to have to SSH to my homelab server and run git pull each time I change my configuration. deploy-rs allows for pushing changes to a remote server.

 1# flake.nix
 2# ...
 3deploy.nodes = {
 4      nixos = {
 5        hostname = "nixos.josephstahl.com";
 6        profiles.system = {
 7          path = deploy-rs.lib.x86_64-linux.activate.nixos self.nixosConfigurations.nixos;
 8          sshUser = "joseph";
 9          user = "root";
10          sshOpts = ["-t"];
11          magicRollback = false; # breaks remote sudo
12          remoteBuild = true; # since it may be cross-platform
13        };
14      };
15    };

Since my homelab is accessible from anywhere with Tailscale, deploy-rs can be used to update my NixOS configuration from anywhere.

just deploy is enough to push my NixOS configuration to the homelab server, rebuild the changes (rebuilds done on remote server to avoid cross-compliation with M2 Mac), and deploy the build, with a pause to ask for my sudo password to apply the changes.

MacOS (nix-darwin)

Using Nix as a package manager is not limited to just NixOS or even Linux. It can function alongside or as a replacement for Homebrew on MacOS package management, as well as managing a user’s dotfiles (with Home Manager) and system configuration (with nix-darwin).

Installation

On MacOS, the Determinate Systems nix installer is the easiest and most reliable way to get Nix installed.

Configuration

MacOS configuration is divided into three parts:

  1. system configuration
  2. package management
  3. dotfiles

System configuration in hosts/darwin/shared is shared among all MacOS systems (Nix settings, Mac & Finder options, etc.). All GUI apps are installed via Homebrew with the nix-darwin homebrew module, while CLI apps are installed via environment.systemPackages in darwin/shared.nix and common/default.nix. Finally, dotfiles are managed by Home-Manager via configuration in home/joseph which is also shared with NixOS systems.

To manage MacOS-specific settings, import pkgs.stdenv.hostPlatform isDarwin allows for calling isDarwin as a part of if/then/else statements.

Conclusion

The learning curve to Nix/NixOS is trecherous and the error messages are often inscrutable, but when it works, it’s impressive. I can wipe the NixOS VM on Proxmox and have the entire system reinstalled and configured within minutes.

Installing new packages (and cleanly uninstalling them) is as simple as running nixos-rebuild switch to update a system with the new configuration in flake.nix. If a package does not yet exist on Nixpkgs or is out of date, a new package can be made from a Github release (or the source code on Github, with compliation managed by the package configuration file).

Some software is still not yet properly packaged for NixOS, or needs additional setup—for example, the Sabnzbd module doesn’t allow for setting/ overriding the contents of Sabnzbd’s .ini configuration file.

Still, it’s nice to share so much configuration between my Mac and my homelab server (and to be able to deploy changes to the server via just deploy). The momentum around Nix is reassuring that error messages will improve and it will slowly become more approachable for beginners like myself. In the mean time, be ready to devote hours to the project to play around with things and learn how it all works.

#Nix