The Linux Memory Manager

I have had a fascination with the memory management subsystem for quite some time now. I started contributing to the kernel in 2015, and memory management in 2016. What drew me to it was the fact that memory is such a core thing and yet beneath the surface lie complexity, trade-offs and a great deal of engineering. There is also an ineffable feeling of being 'drawn' to it that I can't quite explain.

The book was an idea that began with efforts I made some time ago to write notes on the mm subsystem - linux-vm-notes and linux-mm-notes. These were intended to help me understand the subsystem myself, but gained some general interest and in the case of the latter project led directly to a couple of patches I had accepted as a result.

I am avoiding the question here, so let me answer it directly - one day I simply woke up with the idea. I can't quite explain why, but I couldn't quite get away from the thought of doing it, and one day I just decided to start. I know that sounds strange, but that is really how it happened.

In terms of motivation, the primary one is to deepen my knowledge of the subsystem. Writing something for other people to understand does force you, if you are humble and care about what you're writing, to really try to understand it and inevitable learn all kinds of things you did not know before.

In addition, I felt that a new book on this subject was long overdue and that other people might find the subject as interesting as I do. I have been overwhelmed and absolutely humbled by the interest so I feel that this point has been proven accurate!

Any author writing about the kernel is aiming at a moving target. The book is likely to end up with thousands of explicit references to specific files and line numbers, hundreds of code listings, dozens of painstakingly drawn diagrams and over a thousand pages of description - updating the links alone for each kernel release cycle would be so time consuming as to be all that I do, let alone the rest that are simply infeasible.

I therefore, practically speaking, must aim at a specific target or simply not write the book at all. In fact the original target was v5.19, the major version roll convinced me to move my existing work to v6.0 for purely human psychology reasons, and porting even that was time-consuming to say the least.

The key concept is that a newer book targeting the mm system will have a smaller delta to the current version than the most up-to-date book on the subject, the excellent Understanding the Virtual Memory Manager by Mel Gorman (amazon link).

This was released in 2004 and targeted kernel v2.4 (with updates targeting v2.6). This renders it significantly out of date compared to current kernels, but encouragingly still contains useful ideas and concepts.

In addition, the book was always intended to be a starting point - to explain concepts that aren't hugely obvious without examining a lot of code and to go deep - but always referencing the source code and to give the reader the background such that they can figure out the delta between v6.0 and the current kernel themselves.

Finally, it is not impossible, should the book be popular enough to warrant it and my time and sanity sufficient for it to be practical, that later editions could be released updating the book to a more recent kernel.

I'd rather release the book as a whole, for a number of reasons:-

I'd like to defer a more in-depth review of the book's contents in order that I not get 'stuck' along the way, which I feel is something that a number of things could so easily bring about, and something that I must constantly fight to avoid. I feel releasing chapters early would jeopardise this.

Additionally, I fear that I might release a draft chapter early and make some kind of a significant error which could mislead people. I am going to great lengths to try to ensure what I write is correct, both checking myself by reading the kernel code (a LOT!), writing and running test code, having the kind support of some kernel maintainers who are checking early chapters, and I feel releasing chapters early would risk me missing something.

I feel the ordering of the chapters is really helpful for somebody who wants to learn this subject deeply - starting with the question - what even is a dynamic allocator at all? Moving on to how the kernel allocates physical memory, then how virtual memory functions, how process memory is managed, and so forth, constantly building on the previous knowledge. It would be a pity to split that up.

Having said all that, I am considering releasing a smaller chapter before I release the book, perhaps the OOM chapter, in order to provide a sample for potential readers and to give a sense of my approach and the contents of the book. That would be a one-off, however.

No I'm afraid not. This is not out of greed, money is not even on my list of motivations for writing it at all, but rather because I both want a physical release of the book (that of course costs money to make) and even for an ebook version I feel that, much like other technical books, it costing a relatively small amount assigns value to it in people's minds.

In any case, it is inevitable that, should it be even slightly popular, that it will be pirated very early on, so those who wish to go down this road will be able to get it for free regardless of my wishes. I'd ask that you don't, however!

I actually started the project as an open-source one, however I then read some ghastly stories about the issues authors face on many platforms with people stealing their work, changing the name and releasing it as their own. It's rife apparently and challenging to get the sites to take such things down. This put me off the idea.

In addition, I really don't want to put out anything that might turn out to be incorrect if I can help it, and doing this would absolutely risk that.

I realise there's an irony in talking about an open source project without the book itself being open source, but it feels like writing a book is a different kind of beast, at least for me. I realise, of course, that there are excellent open source books available and I absolutely respect those authors, however I don't feel that this is the right approach for me.

I am absolutely not against open sourcing it at a later date however, when the above concerns become moot, so it is quite possible I might do so.

I actually wanted to delay talking about it until much later, but the chaos at twitter, a site which has been rather good for small tech communities, has forced me to accelerate things.

Believe me, I'd much rather have at least a thousand pages written before saying a word, but now we are here I feel I ought to try to do things properly!

I am aiming to write a book that I would like to read, and something that I have always been disappointed by are technical books which have been afraid to dive deep into the code and to get into serious levels of detail.

As a result, I have intentionally gone very deep and thus assume some basic skills in doing so (otherwise I'd have to write so much preamble everywhere that it would become unreadable).

This means that you do need some understanding before coming to the book, though I do try to explain everything as clearly as I can. I recommend you get some basic understanding of the Linux kernel from a book such as Robert Love's excellent Linux Kernel Development, and some OS basics from the classic Modern Operating Systems.

I am taking quite a personal approach to the book and work on it part-time, mostly on weekend, and sometimes at odd hours, so it is simply not suited to direct collaboration.

I am, however, keen to get good technical feedback (and later, copy editing) in order to make sure I am as correct as I can be. While I am humbled by the number of offers of help on this front, I am limiting access in order that I not get too distracted early on.

Generally speaking I am limiting review to people who appear on the kernel MAINTAINERS list only. I am extremely grateful to everybody who agrees to help and absolutely do not take it for granted. If you are on there and are interested in helping do drop me a line!

I am putting this further down the list as it is covered in the about section of the site, however generally speaking the target date is late 2024. This may get pushed back as this project is part-time and I have a demanding full-time job which takes precedent, however I feel there is a good prospect of me hitting it.

Similar to my answer regarding the target kernel version, for portions of the code which are architecture-specific, I simply do not have the time to cover all possibilities, and doing so would render the book far less readable.

I was very tempted to target arm64 as the ISA landscape continues to evolve and since mobile devices are so dominant, however x86-64 is still considered by most to be the 'de facto' architecture and helpfully has a simpler memory management architecture than the many arm variants out there.

Equally as to the kernel version question, an astute reader should be able to adapt the architecture-specific stuff to whatever architecture they're targeting. The book is always only a starting point.

As with other such questions, the answer is time. I simply cannot cover everything and I really want to focus on what is core to the subsystem. I would love to cover more but I sometimes have to make some brutal cuts as to what gets focus, what gets a brief mention and what is not covered at all.

The book is written in pure LaTeX, all non-table diagrams are written using the rather excellent TikZ package, and the book is being typed in the ever-eternal emacs (sorry, not-sorry ViM users!)

I also use mupdf to quickly preview PDF output and a bunch of scripts I wrote including one which uses inotify to incrementally build each chapter as I write it.

I am very reliant on the excellent Kinesis Advantage 2 keyboard which lets me type for hours without feeling like my hands are going to fall off.

For diagrams and loose research/planning nothing quite beats pen and paper, except an e-ink tablet, which is the equivalent except you can instantly delete things, move stuff around, export, save and do it all very compactly. For that I currently use a Boox note air 2 plus.

I am so sorry I just realised that I have a dental appointment, I must dash, let's catch up later!