The use of void* had let an overzealous unsafe_ptr() leak a pointer with address
space control to the client (in LocalAllocator::alloc_not_small, specifically).
Correct this to call capptr_chunk_is_alloc() (to capture our intent) and
capptr_to_user_address_control() (to do the bounding) and defer the conversion
to void* until the very periphery of the allocator, using capptr_reveal()
(again, to capture intent).
Motivated by renaming `FreeObject::{Head,Queue,AtomicQueue}Ptr` to
`freelist::...Ptr`, in fact go further, moving `FreeObject` itself to
`freelist::Object` and `FreeListBuilder` to `freelist::Builder` and
`FreeListIter` to `freelist::Iter`
Now that explicit annotations have gotten us through the refactoring, it's time
for the scaffolding to disappear. src/mem/freelist.h is left generic for any
future machinations, but `FreeObject::T<>`, the several `FreeObject::...Ptr<>`s,
`FreeListIter<>`, and `FreeListBuilder<>` are given default parameters and all
uses are shortened to use defaults where possible.
Without this it complains that LocalCache's constructor can't be constexpr
because small_fast_free_lists isn't initialized. It's not clear to me why it
didn't mind before, and nobody else seems to mind now, but this shouldn't break
anyone else, either.
Just an intermediate syntactic step to chase dependencies. All these introduced
"domestication" callbacks are just the identity function, but they will let us
thread the LocalAlloc's handle to the Backend state down to where it's needed.
This is incomplete, yet still more reflective of what's going on: we take the
exported pointers back from userspace and thread them directly into the free
lists.
So: move capptr_to_user_address_control to list construction time rather than
list consumption time.
FreeObject itself is now just a namespace (but `friend`-ly); the actual free
list nodes are FreeObject::T-s that are templatized on the (perceived)
`capptr::bound<>` of the pointer they contain. (These may differ across an
instantiated snmalloc; for example, in the sandboxing design, the in-sandbox
allocators may perceive all remotes to be full of `AllocUser` while the
privileged allocator of sandbox memory should perceive its remote queue as
holding `AllocUserWild` pointers in need of domestication.)
The interfaces to `FreeObject::T`-s now let us distinguish between the base and
inductive cases of the queues:
* in the inductive case, the pointer we hold to a `FreeObject::T` and its
next_object have the same bounds
* in the base case, the pointer we hold has different bounds (typically,
domesticated by contrast to the wild pointers in the queues).
To keep the clutter down a bit, we occasionally use raw pointers when we can be
reasonably certain that domestication is assured. Moreover, we define some type
aliases, `FreeObject::{HeadPtr, QueuePtr, AtomicQueuePtr}`, that are slightly
more convenient labels than, e.g., `CapPtr<FreeObject::T<BQueue>, BView>`.
Because we are using template parameters for the `capptr::bound<>`s themselves,
we cannot use the aliases for `CapPtr<>s` provided within `capptr::`.
The two primary interfaces around free objects (`FreeListIter` AND
`FreeListBuilder`) are adjusted appropriately and their `BView` and `BQueue`
template paramters are plumbed explicitly around the tree. This makes for quite
a bit of noise at the moment, but means that we'll be able to evolve parts of
the tree separately and can consider putting defaults in once that's done.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
David points out that we might not have a static way to get at the pagemap, so
it is potentially useful to pass pointers to state objects down from the
Allocators.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
This commit adds a simple XOR encoding to the next_object pointer in
FreeObjects. This removes the trivial way of getting hold of a physical
address from the system by observing the free list pointers in
deallocated objects.
This extends the freelist protection to the remote message queues. They
effectively perform doubly linked list entries for the message queue
with the enqueue operation first linking in the previous pointer, and
then then atomically setting the next. This ensures that the visible
states always satisfy the invariant that the forward and backward
pointers are correct for any visisble object.
There is a key_global that is used for all remote deallocations. The
remote cache uses the same protection to build the temporary lists
before forwarding to the next allocator.
The mpscq is integrated into the remoteallocator as it is no longer
a reusable datastructure, but a special purpose implementation.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
