Nix 2.21.0 release with Jacek Galowicz

Intro
Hello and welcome to Full-time Nix. You’re listening to the release notes for Nix 2.21 with Jacek Galowicz. This podcast is hosted by Shahar “Dawn” Or, it’s processed by the Full-time Nix team; consider becoming our patron. Go to fulltimenix.com.

Shahar "Dawn" Or
Nix 2.21.0 released March 11, 2024. It has security updates, CLI enhancements, REPL improvements, and more. We'll start with a security update, CVE 2024-27297. Fix: a critical sandbox escape was addressed in this release. Previously, cooperating Nix derivations could exchange file descriptors to Nix store files via Unix domain sockets, allowing for output modification post-validation. This loophole is now securely closed, enhancing the reliability and trustworthiness of your Nix environment. Please, Jacek, expand a little bit on what is cooperating Nix derivations and how they could exchange file descriptors. What is this about?

Jacek Galowicz
Generally, software sometimes checks some condition if something is in the state that you want it to. Then some time passes and then when you use it, someone might have changed it in between since the last check. And this is the category that fits on this problem too. So I have seen the pull request that fixes this problem, and it's very clean and tidy: they provide a test that fails with this security hole not being fixed. And then they provide the fix to do this. And in the test, that is actually modeling this scenario of how someone could use it. They say, OK, let's build a fixed output derivation. So they simply create a little derivation that outputs “Hello World!” to an output path on the Nix store. And because they also provide the output hash, this is a fixed-output derivation, right? FOD. Also, like this. And these run in a sandbox that has internet access because for example, every source code download, it of course needs to download from the internet because you cannot generate source code, at least not from a specific commit of a Git repository, right? And because they have the output hash provided already, you can check if the output that they produce by downloading stuff and so on, if it's actually the content that you wanted, because otherwise the hash wouldn't be okay, right? So this kind of derivation can be attacked, so to say, or can be part of an attack. What they did in the pull request in this demo scenario is they wrote an application that runs outside of the sandbox. And it basically listens on an abstract Unix socket. Abstract Unix sockets are a little bit different than the usual Unix sockets. The usual Unix socket is a file in the file system and then two parties can open it and communicate over that. It basically feels like a network socket. But an abstract Unix socket has just a name. It is not in the file system. So two processes that are on the same system who know this name can basically share information with each other. And if you are in a different file system namespace, you can still access it because it doesn't live in the file system. It lives in a different namespace. So what they do is before running a fixed-output derivation, they start this process and it listens on this abstract domain socket and then they run Nix…

Shahar "Dawn" Or
Wait, sorry. What listens exactly?

Jacek Galowicz
So they start one application that they specifically wrote for this purpose. I will explain what it does step by step, but at first it's just running and listening on this Unix domain socket. So then they start a Nix build process, which runs this Hello World thingy in the sandbox. So it basically just prints “Hello World” to the output path. Because the Nix expression in the beginning called the derivation function of Nix with this output hash information. It just runs. Then Nix will check if the output matches. And if it does, the sandbox is closed again, right? But what this little derivation does is it does not only print “Hello World” into the output file. It also starts another little application that takes the $out environment variable from inside the sandbox, which of course points to the output path. And then they send it to this abstract Unix domain socket, because they also know the name of the socket. So then this process that we started before running the Nix sandbox, it receives a file descriptor, like a Linux file descriptor, to this output path. And it does nothing at this point yet, right? So the Nix builder sandbox closes and we have “Hello world” written to the store path. Nix checks it, sees okay, the hash is good, and then the application that was already running before the Nix sandbox, which now has the possession of this file descriptor to the output path, is notified like “Hey, the Nix sandbox is done, please do something now,” and this other application will then write # to the output path. And the demonstration is that this actually changes the Nix store output path because it is writing via a file descriptor that originates from the namespace that has this store path mounted as read-writable. So Nix was already done with checking the output and said okay this derivation is fine, and after that they still had the possession of this old file descriptor that was still able to write to it and then were able to change it.

Shahar "Dawn" Or
File descriptor is the piece of data that was transferred through the abstract Unix domain socket.

Jacek Galowicz
From inside the sandbox to outside the sandbox, yes.

Shahar "Dawn" Or
And that's just an integer, isn't it?

Jacek Galowicz
Yeah, basically, but of course the Linux kernel has its internal data structures that map those integers to the knowledge of what file, who writes and so on, right? And that still exists. Of course, it seems that unless it's closed, they don't delete it. I mean, the thing is when you delete a file, but someone still has a file descriptor on it, the file still exists, right?

Shahar "Dawn" Or
In the sandbox, the out file was opened. I suppose it has been created as well. And this...

Jacek Galowicz
Yeah, but that was already the case after writing “Hello World” to it. So they write “Hello World” to it as a content, and then they open it another time.

Shahar "Dawn" Or
Right. Okay. So, “Hello world” is written to that file and then it is opened via a system call, which returns a file descriptor. And then that file descriptor is transferred via that Unix domain socket to the program outside of the sandbox that is listening. And therefore, the program outside of the sandbox, even after the sandbox is finished, it was able to write to that file. What does it take for that file descriptor to be closed? But let's say theoretically, let's say it was opened by the program in the sandbox and then it was not sent outside through the domain socket. Would it have been somehow closed? Or do they remain open after the program finishes?

Jacek Galowicz
That's a good question. That's of course kind of looking to Linux internals and I'm not completely sure. But I can imagine that there are enough scenarios where it still keeps open for some time.

Shahar "Dawn" Or
Right, I guess the answer is yes, because obviously the sandbox is already closed, and only then the program outside wrote successfully, and possibly reliably, if there's a failing test, to that file. So as you said, we suppose there are some scenarios in which that is the case.

Jacek Galowicz
Yeah. Yeah, however, I mean, they fixed it by copying the output path or file somewhere else and making that the Nix store output path. So that kind of invalidates the file descriptor, so to say. So you can still modify the content of the store, but that's kind of just a temporary copy in between. And the store output that gets reused to the outside world later is just a copy of it, which means that you would need to get a new file descriptor to the target of the copy.

Shahar "Dawn" Or
Sorry, can we take this really slow? Because I'd really like to understand the mitigation here.

Jacek Galowicz
Yeah, so I've seen that they added an additional copy of the Nix output path, kind of. And they will... So they check the Nix store path content after copying it.

Shahar "Dawn" Or
Well, chronologically, this is after the sandbox is finished.

Jacek Galowicz
No, no, no. When the sandbox finishes, they kind of get this path ready, right? And then as you say, when it finishes, they copy the path somewhere else and then do the check there.

Shahar "Dawn" Or
Copy it somewhere else as soon as the sandbox is finished and checksum it there.

Jacek Galowicz
Yeah, exactly. Because if the original path that was created by the Nix builder process, if they still do some kind of weird file handler magic and manipulate it afterward, then this will not happen to the copy that they do. So the copy is later used for the Nix store, for the users and for checking.

Shahar "Dawn" Or
Isn't there some, you know, just like there is a Unix feature that allows this inter-process communication, isn't there a way to launch the sandboxed build process, I don't even… environment, whatever that is, in a way where that kind of thing is not allowed? There's more better… more comprehensive isolation?

Jacek Galowicz
I'm not sure. So these abstract sockets are kind of a specific Linux feature. That doesn't exist on other Unixes.

Shahar "Dawn" Or
Hmm. So this vulnerability was never on Mac.

Jacek Galowicz
Yeah, I guess so. Although Mac is a different story, right? I mean, on Mac you typically run with a disabled sandbox because it produces many problems with a few derivations.

Shahar "Dawn" Or
So maybe it's an entirely different set of existing problems there that perhaps it's a lower expectation from the get-go.

Jacek Galowicz
Yeah, true.

Shahar "Dawn" Or
Okay, interesting. Well, if you want to learn more about it, I suppose you can see the release notes and there should be a link. Alright. There have been Nix REPL improvements. One of them is concise error printing. Errors in lists or attribute sets are now more succinctly reported, making troubleshooting in Nix REPL far less of a headache.

Jacek Galowicz
Yeah, that's true. So I've also just experimented with this with the latest Nix and it's really nice that there's simply less noise. Just inject some kind of error into some list or something and you will get less output which is much clearer. So this is very nice.

Shahar "Dawn" Or
Great, another quality of life improvement that adds up. Pretty printed values: the Nix REPL now pretty prints values, significantly improving the readability and understanding of complex Nix expressions.

Jacek Galowicz
That's also very nice. So let's look into the changelog where they have some nice example. So when you basically just print some convoluted attribute set on the terminal, you will not get it in one line any longer, which is especially very unreadable when you have very big attribute sets. They will simply expand each key = value thing over one line, nicely sorted and yeah, only missing colors right now. So it's very nice to see that, especially when you were debugging NixOS configurations, when you look at what were the values that are set there in the REPL, and it was sometimes really hard to read. And now it's really a breeze. It's really nice to see.

Shahar "Dawn" Or
Yes, I remember that a single line could have ended up easily as multiple lines with wrapping and totally unreasonable, not humanly readable. And I can certainly imagine some have made tools for themselves to pretty print that, to format it better. And now everyone has it. So thank you to whomever contributed that. Certainly a big quality of life improvement and expected of any modern tool. Enhanced debugger features: numerous enhancements such as the ability to access let bindings, simplified cycle detection, and better source position information in the debugger make debugging Nix expressions more intuitive. So I heard three things here. Access to let bindings. What does that mean?

Jacek Galowicz
Yeah, so this is a little bit…

Shahar "Dawn" Or
I mean, how can you not have access to led bindings in a debugger?

Jacek Galowicz
So the thing is that... Let's search over the changelog. We have multiple items in the changelog that are about the debugger, so it's a little bit of a mess here when you look over the changelog to see what really happened. The thing is that the debugger was introduced in one of the earlier versions, so it's not really old or established and it also still has a few bugs that are detected by users here and there. When I played with it, it also feels like, okay, it does most of the things that you would expect it to, but sometimes you have a bad experience because when you step on the breakpoint and inspect stuff, then you cannot access all the values everywhere or it's really slow. And this specific error that you are talking about right now was like in this example here, where they simply have a let binding. So they have let a = something and let b = something. And then they put a breakpoint inside these variable definitions in the sandbox. Sorry, not sandbox, in the let-in expression. And the idea is that when you step on one of the variables, in the let-in expression with a breakpoint, that you get into the debugger session that stops right there. And then you would expect, well, let me see what the values of the other variables are. All the other values in the let-in binding, right?

Shahar "Dawn" Or
Well, at least those that have been evaluated.

Jacek Galowicz
At least know which ones are there and then if you can evaluate them or not, however, print the values of most of them, right? So that wasn't possible in that specific bug scenario. They fixed that. I had, even with that fixed, I still had a few weird scenarios with the debugger where it didn't behave as I would expect it to. So I would generally say that all these changes generally make it more useful, but you have like scattered error situations here and there. Cross-codes (?) can be very complex and the debugger needs to work in all these scenarios. So I guess we need to wait for a few more releases until it gets really stable in all situations. But this is a step into the right direction, right? So maybe let's talk a bit more about the debugger because we are not revisiting it after this bullet point in our plan. The thing is that debugger generally works on any Nix expression. When you nix build, nix develop, or nix repl with the --debugger flag, then the debugger will kick in as soon as you step on some builtins.break expression. builtins.break is also new in that context of debugging, right? And so these break expressions will simply be ignored if you don't use the debugger flag. So you can basically sprinkle it all over your code and normal users will get no impediment from it. But then when you want to debug something, you simply launch any of your Nix commands with the --debugger flag, and then you will be dropped into a REPL where you can simply look around what the variables are, what the backtrace is. You can step and you can continue like with normal debuggers, right? Another thing that I think we don't mention it later in our plan is the debugger-on-trace option. If you, in addition to enabling the debugger flag, set this option, it seems like you can set it in the Nix config globally but also on the command line, like most of the options. And that turns builtins.trace statements or expressions to breakpoints too. So whenever you get a trace, you will be able to handle it like a breakpoint. This is extremely useful in the case, imagine you are using some external Nix library and they added some depreciation warnings or something. And the warning says, please don't use this and that any longer. In the future, it will be like this and that. And then maybe sometimes it's really hard to find out which of my code lines triggers this warning. So where do I need to fix this if I want to fix it right now? There was typically no position indication for that. But when you turn all the trace outputs into breakpoints and break on them, you can actually look at the backtrace and the backtrace will explain to you who triggered this. So this is really useful because this way you can kind of find out which line of code in my project is the one that I need to fix now so that the warning disappears. This will help people a lot. And I think for everyone who doesn't know about the debugger, we'll have more time wasted on finding such warnings. Because you cannot grab it through a source code for this warning, right? Some source code line in your project somewhere triggers this warning somewhere else. And this was kind of really annoying to find out. I very often had that in teams who use a lot of Nix when you introduce some warning somewhere. And the email you would like, hey, you introduced this warning and it triggers all my code, where do I fix this now? And I was like, well, it's your code, right? You trigger the warning. And then we're like, yeah, but we have like thousands of lines of code. It's really annoying to find out. And of course, that's not the mistake of the one who introduced the warning, but the burden of the code writer who has to fix the code triggering this is now kind of smaller because they simply activate the debugger, wait for the warnings to kick in, see where they come from, fix them. Boom, that's done. So that's a huge improvement. And I am really looking forward to the debugger becoming more mature so that you can introduce it to more people. I already started to demo this in my Nix classes. But the problem is that sometimes you run into situations where the debugger is really slow or something. So it doesn't produce a lot of trust on the first sight, so to say. So I really hope that it will get mature quick. And then we have an extremely useful and powerful tool in the tool set of every Nixer, right?

Shahar "Dawn" Or
Definitely looking forward! I never used the debugger. I did not know that was a thing.

Jacek Galowicz
Yeah, try it. It's simple. You just put some builtins.break expression somewhere in whatever Nix expressions you have. And then you run nix build or nix repl with --debugger. And then you can play with it. When you make the :? command to get the help in the REPL, then you will see at the bottom all the debugger commands that are there. And then you can just play with them. It just takes one minute, right?

Shahar "Dawn" Or
On the first opportunity where I feel I should confirm my assumptions, I will use the debugger. I would like to point out that this seems a little bit like an afterthought, because what we want here and what this, the thought behind this trace, what's it called, debug trace flag? The thought behind it is probably not aimed toward trace per se, but aimed towards warnings such as you described. And that brings me to think that Nix should have built-in warnings, or at least the idea should be considered. And maybe it had been considered, because if you had built-in warnings, you could have also said something like, hey, I want to turn warnings into errors. So I make sure I handle them as early as possible, and I have no warnings in my build logs and so on. Um, and I suppose breaking on traces works just well enough because typically you'd have traces either on warnings, because apparently they are the mechanism underneath every warning in Nix code or when you're debugging or when you forgot them in the code after a debugging session and you did not review properly before committing. So it works just well enough, but it would be nice to have proper warnings as a Nix feature.

Jacek Galowicz
Well, you're right that in the case that you want the user experience in the end that seeing where a warning comes from is easy, right? But the implementation in Nix, I wouldn't say that warnings should be a built-in feature. The thing is that we have warnIf functions that come from the nixpkgs library. They internally use kind of, they check for some condition and if that is true or not true, then they use the built-in trace function and so on. So the Nix binary itself, of course, with the Nix code itself, it implements just the primitive functions. Of course, the list of built-in functions should be as small as possible and these implementations should be as fast as possible. But this introduces a lot of complexity into the Nix application code if you put more functions there that are specialized; you generally want to avoid having too many specialized functions. You just want to have the minimal set of as generic as possible standard functionality and then inside Nix libraries built on top of that, which they do and which is completely fine from a design perspective. I would stick to that. But what you could generally do is, of course you could, let's say everyone uses a function that's called like warnIf, and it is implemented on top of builtins.trace and so on. Of course, if you want a behavior like every warning becomes an error, then you could certainly overlay nixpkgs in a way that this warnIf function is substituted by one that blows up in your face very loudly. So you could always have been doing that already. Maybe people actually did that. And then that error message still doesn't tell you who actually did something that led to this warning, because that's basically not how it works. When you have exceptions in other programming languages, you can catch the exception, but you still have to find out who really provoked this exception, right? Then you could still say, OK, show the trace, because on every Nix error message, it tells you how the flag looks like that you could append to get the full trace. And then when you scroll up a bit, you will see the line that actually triggered this. So that was possible already. It was just more work than simply kicking in the debugger flag and break-on-trace option and that's it. So you can have the same experience with less code change in between.

Shahar "Dawn" Or
Another UX thing I would like is in development, locally, for warnings to be allowed, but then use a flag and evaluate differently in CI where there, warnings would not be allowed. So just a piece of feedback. I always say, abusing my position to provide feedback. So what else do we have? Simplified cycle detection. What does that mean?

Jacek Galowicz
Yeah, so generally Nix is very good at finding out if you accidentally created an infinite recursion, for example. So when you have some attribute that references itself somehow, it will not print you infinite output where you see the same thing all over again. It will just tell you that there is an infinite recursion. So let's look at the release notes where they have an example.

Shahar "Dawn" Or
I admit I did trigger infinite recursion in Nix. I did see that error. I believe if I remember correctly, it says “infinite recursion detected” or something similar.

Jacek Galowicz
Although I'm right now not completely sure if infinite recursion as in a bad overlay function or the infinite or the cycles that we have here in the release notes are really the same. But what they did here as an example is they created an attribute set self that inherits itself. So you have to have the self in the self and the self in the self.

Shahar "Dawn" Or
That seems to be the smallest example.

Jacek Galowicz
Yeah, and when you printed it, before the release, you would see the self keyword multiple times until they say that it's repeated and that simply got shorter, right? So this is not a completely new thing. They simply improved something. It was there already. So you get, it's again one of the other things that we talked about earlier, that you simply get less noise on the output that tells you that there is a problem. This seems like an incremental improvement. And if you take all the incremental improvements together, then the whole tool feels nicer and more mature, right?

Shahar "Dawn" Or
Yes, I think in this most minimal example, the value is not apparent, but the data structure could have been much larger. And then maybe it would have been, maybe the output may have been cut by half, which would be significant. Better source position information in the debugger as well.

Jacek Galowicz
Yeah, that speaks for itself, right? So when you step on some breakpoint or you print the trace, what you got before was sometimes you just get numbers. So that would be position indicators in some sort that is not really human readable. And human readable is the file name and the line number, right? And you get that now.

Shahar "Dawn" Or
That is again to be expected and now we have it as soon as we bump Nix. Quality of life improvements. Nested debugger support removed. So it's a quality of life improvement by a removal of a feature.

Jacek Galowicz
Yeah, you could run into situations where you are in the debugger already, and after that it starts in the debugger. And that is, of course, awkward, because you have a new scope of variables and everything. And then when you exit that, it didn't really make sense from a user perspective. You would still be in the debugger when you trip on the next debug point.

Shahar "Dawn" Or
Was it even apparent that a nested debugger session is being created?

Jacek Galowicz
Yeah, yeah, when you trip on the first breakpoint, you get the typical output that you also see when you start a Nix REPL with Nix version information like, hello, you're the REPL, Nix version 1, 2, 3, and so on, right? And then when you get into the nested Nix debugger, you get the greeting again.

Shahar "Dawn" Or
The greeting. Yes.

Jacek Galowicz
And that's gone and fixed now and it feels much better to use this right now.

Shahar "Dawn" Or
I bet. So, the subsequent breakpoint will be handled by the original debugger. Okay. Good. Thank you. Store path changes. Leading periods in store paths are officially supported, resolving previous inconsistencies and user difficulties. When we say leading, we mean leading, but after the hash.

Jacek Galowicz
Yeah, so for example that was I've looked through a few of the bug reports that were involved here it was multiple of them and when someone for example says “I want to copy some local folder like from blah somewhere” and then the last part of the path says .config or dot whatever file right like dotfiles, then the question is, will the dot go into the Nix store output path? Yes or no? And if so, how do we handle this? And somehow it was not handled before. So I don't know what really happened if the dot was part of the Nix store path, but then people thought like, hey, let's handle the situation and simply remove it. So that's not possible. But then the discussion was, wait, so many people are using this already. So it will break existing Nix usage. And then they simply formalized this to get better handling. And now the community decision is basically to officially support it.

Shahar "Dawn" Or
What was ever the problem with store paths that have a dot in the name of the derivation?

Jacek Galowicz
Yeah, I'm not completely sure here what are the situations where this is really problematic. But when you ls into a folder, the dotfiles are hidden by default, right? And then you would kind of specifically address them to look into them too. And I'm not completely sure what's the problem with the Nix store here, but I mean, we can have Nix store paths with dots in them, because when you say the Nix store path is a tarball or something, then you have the -blah-blah-my-file-name.tar, right? So that's not generally a problem. And I'm not sure what would be the problem if the part after the hash and then dash starts with a dot. I really don't know. I didn't have time to invest too much time into this thing here because the problem seems to be relatively abstract to me. However, the decision was to officially support this now and fix whatever problem that might bring with it. It was just interesting that the discussion was, “hey, let's remove the support for this because it's awkward anyway”. Then people come up with, “Sorry, we're using this for quite some time already. You cannot really remove it”.

Shahar "Dawn" Or
All right, there are dots in the beginning and the rest of Nix store paths. We have to deal with it. If ever it was a problem, we have to deal with it. Stack size increase on macOS. A crucial fix for macOS users, where the stack size has been increased to 64 MB to prevent stack overflow segfaults in complex expressions. So in Linux it was already 64 MB, and in Mac it was less?

Jacek Galowicz
I think it was 8 kilobytes by default or something. I'm not completely sure, but it was something much smaller. Of course, that wouldn't be a problem in day-to-day situations. So I didn't... I've been using MacOS again since December now. I bought the Mac because I have more and more customers asking me how Nix works on MacOS to be able to help them. And since then, I didn't ever realize that the stack size is so small there, it was practically never a problem for me. But I read that Mac users had problems, for example, when you use recursive Nix stuff or whatever, and extremely complex expressions in that regard, that sometimes your stack could grow larger. And the situation where this is problematic is, of course, someone finds a complex Nix expression on Linux, which works very well there because you have a huge stack there. And then someone tries to use it on macOS and there it crashes, right? So this feels not really portable. And of course, it's quite a trap when you don't have macOS and you just implement stuff and then only the macOS users trip on this problem just because the default stack size is so small there. Then this is kind of avoidable by setting this stack size there too. And now they finally did it.

Shahar "Dawn" Or
CLI enhancements: new CLI options, --arg-from-file and --arg-from-stdin. They add flexibility to how we can pass arguments to Nix expressions. How is that?

Jacek Galowicz
Yeah, so whenever you have a Nix expression that is basically a function and that takes an attribute set with certain named variables, right? It is important that you have named parameters in this sense. Then you could pattern match an attribute set parameter, right? And the thing is that there are many Nix expressions who do this and then they take default parameters and then you don't have to provide anything. And then you could change stuff. So that has been done when, for example, having a Nix expression, like opening, I don't know, nixpkgs, and nixpkgs takes some parameters and you could provide them on the command line, but also in other cases. And there you can use the --arg and --argstr parameters to...

Shahar "Dawn" Or
Existing flags.

Jacek Galowicz
Yeah, these have been existing for a very long time now. And these helped you to just inject whatever Nix expression or string into your Nix functions and you evaluate them. So you don't have to write or actually change Nix files somewhere to get a different result for different parameters. So this was useful all the time. But if you want to inject some huge file or whatever, you would have to do that on the shell level right so you would write nix build and then whatever expression and then --arg and then have a subshell that kind of opens the file and creates you a string on the command line so this can get very clumsy in different regards especially when you talk about masking quoting stuff and so on and the --arg-from-file and --arg-from-stdin makes your life easier then because it handles this for you so that you don't have to do the bash command line gymnastics.

Shahar "Dawn" Or
Improved function argument order: Function arguments in printed expressions now appear in lexicographic order, offering a more predictable and readable output.

Jacek Galowicz
Exactly. So this is kind of like the pretty printing from before. When you are in a situation where you get the function arguments printed somehow, then they are printed for you in lexicographic order. It's nicer to find a specific entry or something. Another incremental change, right? With the other incremental changes together, it all feels like a huge user experience improvement.

Shahar "Dawn" Or
Yes, and this one may also help if you were ever to parse… for testing or whatever.

Jacek Galowicz
Yeah, maybe, yeah.

Shahar "Dawn" Or
Something with regards to profile management, which is a feature that is quite discouraged in favor of more declarative, Nix-y ways. So new flags have been introduced, --regex and --all, for the experimental commands, nix profile remove and nix profile upgrade. What do they do?

Jacek Galowicz
Yeah, so. First, a bit of context, you said that nix profile is kind of discouraged. That's true. Still a useful command, right? So you generally use it to say, nix profile install, and then some package name. And then you get it installed in your user shell. And then you can, it's like the package manager install commands that other package managers do too, right? And then Nix is very often introduced to people as a package manager. Of course, you can have like, especially when beer is involved on a NixCon and so on, when people discuss what really Nix is. From five people to get seven opinions what it really is. So that might be very confusing to newcomers who are like, “Look, I just installed this package manager and now I want to know how do I install and uninstall and update packages”, right? Can't be so hard. Yeah, yeah, of course.

Shahar "Dawn" Or
No, you don't install.

Jacek Galowicz
Advanced Nix users don't do this any longer. They just, the only thing that you need to have installed on your system is Nix and Git, right?

Shahar "Dawn" Or
Well, if I don't install, what good is a package manager for me?

Jacek Galowicz
Yeah, right. But if you just clone some repository that you need to work in, you run nix develop there. And then you have all the tools. And then when you exit the shell again, they are gone. So you don't want to install anything globally because you can have all the tools on a per-project basis.

Jacek Galowicz
So people who get used to this at some time realize, oh, when was the last time that I installed something globally? I don't really do that any longer, right? So it's kind of, when you have something installed with the nix profile, you can accidentally forget to update it and stuff like that. So it's really annoying if you are accidentally using tools from an install list, so to say, that you are not really maintaining or upgrading while, at the same time, a development shell that is spawned from a project is always the same like your colleagues are using and also curated, right? What's really annoying about nix profile—So in my classes, typically on the first day we go through the standard commands and like, okay, great. We've installed Nix now. What's, what's the basic, most basic things that we can do. And of course you show people the nix profile commands to install and uninstall packages. So because that is something people have the feeling they know from other worlds, and so they want to see how it works here, right? And also the Nix shell. And then you start looking at how to create your own Nix packages and stuff like that. However, when explaining the nix profile commands, you got a very awkward user experience in the past. So you say, nix profile install, and then you say, nixpkgs#hello. And then that would install the hello package. So that's easy. But when you want to update or remove stuff, you had to explain very clumsy syntax to the people. So when you wanted to remove hello again, you would say, nix profile remove. And then instead of just saying hello with the same name as you used during the installation, you would have to say in quotes .*hello because that would by default be a regular expression that matches on the installed whatever, right? So people were like, why do I need the dot asterisk if I just want to reference the package that I just installed? And then you're like, yeah, it's complicated, right?

Shahar "Dawn" Or
Totally. That's a valid question.

Jacek Galowicz
People don't care about the reasons why this is complicated because that should simply be easy. Also, when you wanted to upgrade all the packages, you would have to write nix profile upgrade and then .* as a match all, so to say, right? So that's, I don't know, it felt stupid because what would be the cases that I would need this for where I really am thankful that I have a regular expression there? So what changed now is, you can now say, nix profile install nixpkgs#hello, like before. And the change is that when you delete it, you say, nix profile remove hello. Just the name of the package. And then that works. And when you want to upgrade everything, you would just say, nix profile upgrade --all. And that's it. So this is much nicer to explain. And although nix profile remains a shadow use case for all the newcomers, or the newcomers are the shadow use case, so to say, while no one else really uses Nix Profile, it is now less of an entry hurdle by scaring people away with strange command line syntax. So this is really good. I'm still not using nix profile in my day-to-day life, but when I'm showing Nix to newcomers, it is really less awkward in that regard now.

Shahar "Dawn" Or
And if you're not yet a Nix user and you're wondering, so what do we do if we don't install and remove packages from the system? Well, what I do and what I think many of us do is, as Jacek said, in projects, you typically have all your project dependencies declared via Nix and there is likely a shell declared there that you can step into. And as soon as you step into it, all of the executables that you need for that project should be in your shell's path so that you could execute them. And if there is some program that I would like to have available at all times, independent of any specific project I'm working on, I will typically add it to my home, using Home Manager, which is a project that is in the community, it's not inside nixpkgs. It has its own repository. And yet it is very popular among Nix users.

Jacek Galowicz
Yeah, it's a really nice project. So when I said before, that you basically only need to install Nix and Git to your system, that was of course just half the truth. You still want your browser, your mail program, whatever. And on a NixOS system, you generally don't start with just installing the vanilla system and then Nix profile install this and that. You would basically add the list of applications that you want to have either to your NixOS config or to your home manager config. And then this declarative, right? We use Nix and NixOS because of its declarativity most of the time. So declarativity means you don't need to remember the steps that led you to the system that you have right now. You basically have one file or one set of files that describes all the steps in like one step. And Nix profile is basically not that. So if you install Nix as a package manager, where one of its biggest selling points and killer features is the power to describe everything on your system declaratively or nearly everything, then use these imperative commands is ignoring its biggest selling point, right? So, there are multiple opportunities to use this in a better way, but I totally understand that newcomers first want to see the imperative workflow. Because when you jump on people like, okay, right after installing Nix where a few strange things happened, we throw more strange things on you, namely home manager, and you just need to write this long config to install stuff. Then people will at first see like, okay, why do I have to do so many complex things to just install a package, right? So there's a pedagogic order that you should kind of apply when teaching Nix to people and like one complexity step at a time. And for this, nix profile is really useful. But of course, the pros are using hundreds of lines of configuration to have this luxurious workflow of adding a one step installer for the whole system, right? I mean, everyone who uses Home Manager and NixOS configs and maybe both in combination knows if I take your hard disk away from you now and give you a fresh new one, you will be online here in this session in 20 minutes again, right? With the same setup. It depends on the internet speed, of course.

Shahar "Dawn" Or
One of, actually one way in which I intend to improve on that metric is to, next time I have to set up a system, use this thing I heard about called disko, which is declarative partitioning, which would spare me from the agony of having to read the man pages of all of the partitioning and file system formatting tools over and over again every time, because the distance between those occasions that I reinstall the system are typically measured in months, if not years.

Jacek Galowicz
Yeah, disko is awesome. Of course that has nothing to do with the Nix changelog, but I really like to nerd talk about disko. So the significance of disko is really that you describe not only the partitioning scheme, but also how the partitions should be involved in whatever write scheme you have later, like logical volume management, encryption of partitions, and also, how should they be formatted later? It also works with ZFS pools and whatnot. So you can have randomly complex scenarios where you involve multiple disks and write scenarios with and without encryption and whatever partitions you want on that later. And how they should be mounted. You have one big data structure that describes all the disks, their partitioning layouts, the logical volume management scheme later, which labels they should have, which sizes they have and so on, and how they are mounted where. This can be all described in one big data structure and what disko does is it can generate a script that performs all the partitioning, LVM setup and formatting. And it can also, which then happens at every boot, generates the portion of NixOS configuration that will later be mounted correctly. And I've used this in extremely complex scenarios. So unfortunately, disko is not... The documentation of it is... They simply have a very long list of examples. These are very helpful when you build something, but there is not really a booklet that you can read through it and then you understand all the concepts. You have to play with the examples a lot. However, this works very nicely and they have a lot of them, so this is helpful. And once I had a customer, they wanted a server like on Hetzner, a bare metal server. And they had like two NVMe disks that should be running as a write one (?). And then two other disks that were like multiple terabytes large, and they wanted an OpenStreetMap database on those, and those would be running as another write one (?) too and we had some very special requirements that needed us to actually encrypt those disks too and I was able to, in half an hour, describe a disko configuration file that takes all those four disks, partitions and formats them in the right way with all the boot partitions and whatnot, mount them in the right way and also have encryption. So I was able to provide them with an explanation of how to decrypt the disk at boot and you can have different things here: either you have the normal password entry on a physical server in front of you or you can describe the password unlocking via USB stick, so the USB stick would be the key, right? I also did that with physical servers where you just put a sticker on a special USB slot and say “if you put the stick in here, this thing will boot,” right? But also people are using this, for example, to have the machine connect via SSH somewhere else at boot time, get the key and then decrypt the disk. And you have weird scenarios. For example, I had one customer in Dresden. They told me that they used an encryption scheme in a way that on your home router at home, you would have all the keys for your servers and an SSH server. Then your servers, at boot time, SSH is already part of the RAM (?) disk, and then it connects to the router, gets the password, and decrypts the disk. And in case the police come to you at home and take your computers with you, they cannot boot them because they wouldn't have access to the same infrastructure like you have at home, right?

Shahar "Dawn" Or
Should've taken the router as well.

Jacek Galowicz
Yeah. And then some people even have the keys in the router in a way that when you plug it off, then the key is gone, right? So you can, so however, these were like, I'm not, I'm not saying I want to help criminals or whatever here, but the criminals have the most interesting scenarios, so to say, like technically interesting. And it's really easy to implement weird stuff like this with disko. So you will fight to create one config once, and that might be a lot of work maybe, especially when you do this for the first time. But then when you once have a configuration for your partitioning and formatting scheme in a disko config, you can create the next one and next one and next server like from the shelf. So this is really cool. I really like it. It's one wonderful example of how Nix and NixOS can make server deployment really reproducible, quick and easy.

Shahar "Dawn" Or
Wonderful, I'm looking forward to trying it myself.

Jacek Galowicz
Yeah, if you haven't used it yet, you really should. I'm not partitioning things by hand any longer. I mean, I'm also prone to forgetting things, right? So before I knew disko, I had my laptop configuration with password encryption on the disk, right? And then I would, every time I set up a laptop, I would again Google around how to even do this because there's like 300 commands that you enter until your disk is finally in a state that you can install NixOS. And then next time I'm reinstalling NixOS, I would need to look this up again. But now I have my disko config, which does that and it simply generates the bash script that does all the commands for you. And you cannot do it wrong there any longer.

Shahar "Dawn" Or
That is dope. So I think we're done with this release. This was Nix 2.21.0. Thank you very much, Jacek. I'm looking forward to discussing the release notes of the next release with you.

Jacek Galowicz
Yes, it's always a pleasure to look at the new releases because they always have good stuff. It's really nice to see how Nix is a tool that is being actively worked on. It improves and improves, and especially when teaching Nix to newcomers, all these quality of life things. When you show it to people and then you realize, wow, half a year ago, this error message that you're just seeing and understanding right now would have been much uglier. That's quite a relief to everyone who starts with Nix and also everyone who tries to explain how Nix works to everyone who starts with it. So this is really good. I'm also looking forward to more debugger sessions in the future when it gets a little bit more mature.

Shahar "Dawn" Or
And thanks to all of the Nix contributors.

Jacek Galowicz
Yeah, the Nix source code base is a little bit scary to work with. So I didn't really contribute anything of high value to Nix or of high complexity. I only did a few pull requests with, let's say code hygiene. That might have been annoying to people because it changes so many lines of code but doesn't really do anything. But while doing that, I've seen the code and… Yeah, it's a big C++ code base. Of course, C++ alone is a language that is intimidating to many. But you have so many people who are really motivated to work on this. Some people are paid to work on Nix, but many people are not paid to do it. It seems very altruistic to dive so deeply into such a complex code base and improve things.

Shahar "Dawn" Or
Open source is always an opportunity to make contributions for the betterment of mankind and for self-education and entertainment, if that's your thing. But I do hope that open source developers get compensated in one way or another.

Jacek Galowicz
Yeah, hopefully.

Shahar "Dawn" Or
Alright, until next time, Jacek. Be well.

Jacek Galowicz
See you next time.