npins

Simple and convenient dependency pinning for Nix

[![License][license-shield]][license-url] [![Contributors][contributors-shield]][contributors-url] [![Issues][issues-shield]][issues-url] [![PRs][pr-shield]][pr-url] [![Tests][test-shield]][test-url] [![Matrix][matrix-image]][matrix-url]

About

npins is a simple tool for handling different types of dependencies in a Nix project. It is inspired by and comparable to Niv.

Features

• Track git branches
• Track git release tags
• Tags must roughly follow SemVer
• GitHub/GitLab releases are intentionally ignored
• For git repositories hosted on GitHub or GitLab, fetchTarball is used instead of fetchGit
• Track Nix channels
• Unlike tracking a channel from its git branch, this gives you access to the programs.sqlite database
• Can also track Nix channel artifacts like live isos
• Track PyPi packages

Getting Started

Installation

npins should readily be available in all sufficiently new nixpkgs:

nix-shell -p npins

You can easily get a nightly if you want to (requires newstyle Nix commands):

nix shell -f https://github.com/andir/npins/archive/master.tar.gz

You could also install it to your profile using nix-env (not recommended, but might be useful for bootstrapping):

nix-env -f https://github.com/andir/npins/archive/master.tar.gz -i

Quickstart

$ npins init
[INFO ] Welcome to npins!
[INFO ] Writing default.nix
[INFO ] Writing initial sources.json with nixpkgs entry (need to fetch latest commit first)
[INFO ] Successfully written initial files to 'npins'.

$ tree
.
└── npins
    ├── default.nix
    └── sources.json

1 directory, 2 files

$ npins show
nixpkgs: (Nix channel)
    name: nixpkgs-unstable
    url: https://releases.nixos.org/nixpkgs/nixpkgs-22.05pre378171.ff691ed9ba2/nixexprs.tar.xz
    hash: 04xggrc0qz5sq39mxdhqh0d2mljg9wmmn8nbv71x3vblam1wyp9b

$ cat npins/sources.json
{
  "pins": {
    "nixpkgs": {
      "type": "Channel",
      "name": "nixpkgs-unstable",
      "url": "https://releases.nixos.org/nixpkgs/nixpkgs-22.05pre378171.ff691ed9ba2/nixexprs.tar.xz",
      "hash": "04xggrc0qz5sq39mxdhqh0d2mljg9wmmn8nbv71x3vblam1wyp9b"
    }
  },
  "version": 2
}

In Nix, you may then use it like this:

let
  sources = import ./npins;
  pkgs = import sources.nixpkgs {};
in
  …

You may also use attributes from the JSON file, they are exposed 1:1. For example, sources.myPackage.version should work for many pin types (provided that that pin actually tracks some version). Note however that the available attribute may change over time; see npins upgrade below.

Usage

$ npins help
Usage: npins [OPTIONS] <COMMAND>

Commands:
  init          Initializes the npins directory. Running this multiple times will restore/upgrade the `default.nix` and never touch your sources.json
  add           Adds a new pin entry
  show          Lists the current pin entries
  update        Updates all or the given pins to the latest version
  verify        Verifies that all or the given pins still have correct hashes. This is like `update --partial --dry-run` and then checking that the diff is empty
  upgrade       Upgrade the sources.json and default.nix to the latest format version. This may occasionally break Nix evaluation!
  remove        Removes one pin entry
  import-niv    Try to import entries from Niv
  import-flake  Try to import entries from flake.lock
  freeze        Freezes a pin entry, preventing it from being changed during an update
  unfreeze      Thaws a pin entry, allowing it to be changed during an update like a normal pin
  get-path      Evaluates the store path to a pin, fetching it if necessary. Don't forget to add a GC root
  help          Print this message or the help of the given subcommand(s)

Options:
  -d, --directory <FOLDER>     Base folder for sources.json and the boilerplate default.nix [env: NPINS_DIRECTORY=] [default: npins]
      --lock-file <LOCK_FILE>  Specifies the path to the sources.json and activates lockfile mode. In lockfile mode, no default.nix will be generated and --directory will be ignored
  -v, --verbose                Print debug messages
  -h, --help                   Print help
  -V, --version                Print version

Initialization

In order to start using npins to track any dependencies you need to first initialize the project:

npins init

This will create an npins folder with a default.nix and sources.json within. By default, the nixpkgs-unstable channel will be added as pin.

$ npins help init
Initializes the npins directory. Running this multiple times will restore/upgrade the `default.nix` and never touch your sources.json

Usage: npins init [OPTIONS]

Options:
      --bare     Don't add an initial `nixpkgs` entry
  -v, --verbose  Print debug messages
  -h, --help     Print help

Migrate from Niv

You can import your pins from Niv:

npins import-niv nix/sources.json
npins update

In your Nix configuration, simply replace import ./nix/sources.nix with import ./npins — it should be a drop-in replacement.

Note that the import functionality is minimal and only preserves the necessary information to identify the dependency, but not the actual pinned values themselves. Therefore, migrating must always come with an update (unless you do it manually).

$ npins help import-niv
Try to import entries from Niv

Usage: npins import-niv [OPTIONS] [PATH]

Arguments:
  [PATH]  [default: nix/sources.json]

Options:
  -n, --name <NAME>  Only import one entry from Niv
  -v, --verbose      Print debug messages
  -h, --help         Print help

Adding dependencies

Some common usage examples:

npins add channel nixos-21.11
npins add channel nixos-unstable latest-nixos-graphical-x86_64-linux.iso2
# Remove -b to fetch the latest release
npins add git https://gitlab.com/simple-nixos-mailserver/nixos-mailserver.git -b "nixos-21.11"
npins add github ytdl-org youtube-dl
npins add github ytdl-org youtube-dl -b master # Track nightly
npins add github ytdl-org youtube-dl -b master --at c7965b9fc2cae54f244f31f5373cb81a40e822ab # We want *that* commit
npins add gitlab simple-nixos-mailserver nixos-mailserver --at v2.3.0 # We want *that* tag (note: tag, not version)
npins add gitlab my-org my-private-repo --token H_BRqzV3NcaPvXcYs2Xf # Use a token to access a private repository
npins add pypi streamlit # Use latest version
npins add pypi streamlit --at 1.9.0 # We want *that* version
npins add pypi streamlit --upper-bound 2.0.0 # We only want 1.X

Depending on what kind of dependency you are adding, different arguments must be provided. You always have the option to specify a version (or hash, depending on the type) you want to pin to. Otherwise, the latest available version will be fetched for you. Not all features are present on all pin types.

$ npins help add
Adds a new pin entry

Usage: npins add [OPTIONS] <COMMAND>

Commands:
  channel    Track a Nix channel
  github     Track a GitHub repository
  forgejo    Track a Forgejo repository
  gitlab     Track a GitLab repository
  git        Track a git repository
  pypi       Track a package on PyPi
  container  Track an OCI container
  tarball    Track a tarball
  url        Track a URL
  help       Print this message or the help of the given subcommand(s)

Options:
      --name <NAME>  Add the pin with a custom name. If a pin with that name already exists, it will be overwritten
      --frozen       Add the pin as frozen, meaning that it will be ignored by `npins update` by default
  -n, --dry-run      Don't actually apply the changes
  -v, --verbose      Print debug messages
  -h, --help         Print help

There are several options for tracking git branches, releases and tags:

$ npins help add git
Track a git repository

Usage: npins add git [OPTIONS] <URL>

Arguments:
  <URL>
          The git remote URL. For example <https://github.com/andir/ate.git>

Options:
      --forge <FORGE>
          [default: auto]

          Possible values:
          - none:    A generic git pin, with no further information
          - auto:    Try to determine the Forge from the given url, potentially by probing the server
          - gitlab:  A Gitlab forge, e.g. gitlab.com
          - github:  A Github forge, i.e. github.com
          - forgejo: A Forgejo forge, e.g. codeberg.org

      --name <NAME>
          Add the pin with a custom name. If a pin with that name already exists, it will be overwritten

  -b, --branch <BRANCH>
          Track a branch instead of a release

      --frozen
          Add the pin as frozen, meaning that it will be ignored by `npins update` by default

      --at <tag or rev>
          Use a specific commit/release instead of the latest. This may be a tag name, or a git revision when --branch is set

  -v, --verbose
          Print debug messages

      --pre-releases
          Also track pre-releases. Conflicts with the --branch option

      --upper-bound <version>
          Bound the version resolution. For example, setting this to "2" will restrict updates to 1.X versions. Conflicts with the --branch option

      --release-prefix <RELEASE_PREFIX>
          Optional prefix required for each release name / tag. For example, setting this to "release/" will only consider those that start with that string

      --submodules
          Also fetch submodules

  -h, --help
          Print help (see a summary with '-h')

Npins can track plain old links to URL resources. They will never update. Alternatively, you can also add the --mutable flag to make them behave similarly to the Lockable HTTP Tarball Protocol: Npins will follow any redirects and then pin that url as the actual version, while keeping the original url as "update" url.

$ npins help add tarball
Track a tarball

This can be either a static URL that never changes its contents or a "mutable" URL that redirects to an immutable snapshot.

Usage: npins add tarball [OPTIONS] <URL>

Arguments:
  <URL>
          Tarball URL

Options:
      --mutable
          Treat this URL as mutable, and assume it will redirect to an immutable version of the content to be pinned. For example, a HEAD URL redirecting to the currently latest commit

      --name <NAME>
          Add the pin with a custom name. If a pin with that name already exists, it will be overwritten

      --frozen
          Add the pin as frozen, meaning that it will be ignored by `npins update` by default

  -v, --verbose
          Print debug messages

  -h, --help
          Print help (see a summary with '-h')

Removing dependencies

$ npins help remove
Removes one pin entry

Usage: npins remove [OPTIONS] <NAMES>...

Arguments:
  <NAMES>...  

Options:
  -v, --verbose  Print debug messages
  -h, --help     Print help

Show current entries

This will print the currently pinned dependencies in a human readable format. The machine readable sources.json may be accessed directly, but make sure to always check the format version (see below).

$ npins help show
Lists the current pin entries

Usage: npins show [OPTIONS] [NAMES]...

Arguments:
  [NAMES]...  Names of the pins to show

Options:
  -p, --plain    Prints only pin names
  -e, --exclude  Invert [NAMES] to exclude specified pins
  -v, --verbose  Print debug messages
  -h, --help     Print help

Updating dependencies

You can decide to update only selected dependencies, or all at once. For some pin types, we distinguish between "find out the latest version" and "fetch the latest version". These can be controlled with the --full and --partial flags.

$ npins help update
Updates all or the given pins to the latest version

Usage: npins update [OPTIONS] [NAMES]...

Arguments:
  [NAMES]...  Updates only the specified pins

Options:
  -p, --partial
          Don't update versions, only re-fetch hashes
  -f, --full
          Re-fetch hashes even if the version hasn't changed. Useful to make sure the derivations are in the Nix store
  -n, --dry-run
          Print the diff, but don't write back the changes
  -v, --verbose
          Print debug messages
      --frozen
          Allow updating frozen pins, which would otherwise be ignored
      --max-concurrent-downloads <MAX_CONCURRENT_DOWNLOADS>
          Maximum number of simultaneous downloads [default: 5]
  -h, --help
          Print help

Upgrading the pins file

To ensure compatibility across releases, the npins/sources.json and npins/default.nix are versioned. Whenever the format changes (i.e. because new pin types are added), the version number is increased. Use npins upgrade to automatically apply the necessary changes to the sources.json and to replace the default.nix with one for the current version. No stability guarantees are made on the Nix side across versions.

$ npins help upgrade
Upgrade the sources.json and default.nix to the latest format version. This may occasionally break Nix evaluation!

Usage: npins upgrade [OPTIONS]

Options:
  -v, --verbose  Print debug messages
  -h, --help     Print help

Using private GitLab repositories

There are two ways of specifying the access token (not deploy token!), either via an environment variable or via a parameter. The access token needs at least the read_api and read_repository scopes and the Reporter role. The read_api scope is not available for deploy tokens, hence they are not usable for npins.

Specifying the token via environment variable means that npins will use the token for adding/updating the pin but not write it to sources.json. To update the repository in the future, the variable needs to be set again and nix