Avoid computing bits::next_pow2_bits(1 << n). Even if the compiler can see
through enough of the algebra, it's surely more direct to just use n.
While here, slightly expand documentation about what's going on with the
"sizeclass" encoded into MetaEntry-s.
There is no such thing as "struct Slab" any more.
We use alignof(RemoteAllocator) below, so we already require the complete type
definition at this point.
Motivated by renaming `FreeObject::{Head,Queue,AtomicQueue}Ptr` to
`freelist::...Ptr`, in fact go further, moving `FreeObject` itself to
`freelist::Object` and `FreeListBuilder` to `freelist::Builder` and
`FreeListIter` to `freelist::Iter`
Now that explicit annotations have gotten us through the refactoring, it's time
for the scaffolding to disappear. src/mem/freelist.h is left generic for any
future machinations, but `FreeObject::T<>`, the several `FreeObject::...Ptr<>`s,
`FreeListIter<>`, and `FreeListBuilder<>` are given default parameters and all
uses are shortened to use defaults where possible.
Just an intermediate syntactic step to chase dependencies. All these introduced
"domestication" callbacks are just the identity function, but they will let us
thread the LocalAlloc's handle to the Backend state down to where it's needed.
This is incomplete, yet still more reflective of what's going on: we take the
exported pointers back from userspace and thread them directly into the free
lists.
So: move capptr_to_user_address_control to list construction time rather than
list consumption time.
FreeObject itself is now just a namespace (but `friend`-ly); the actual free
list nodes are FreeObject::T-s that are templatized on the (perceived)
`capptr::bound<>` of the pointer they contain. (These may differ across an
instantiated snmalloc; for example, in the sandboxing design, the in-sandbox
allocators may perceive all remotes to be full of `AllocUser` while the
privileged allocator of sandbox memory should perceive its remote queue as
holding `AllocUserWild` pointers in need of domestication.)
The interfaces to `FreeObject::T`-s now let us distinguish between the base and
inductive cases of the queues:
* in the inductive case, the pointer we hold to a `FreeObject::T` and its
next_object have the same bounds
* in the base case, the pointer we hold has different bounds (typically,
domesticated by contrast to the wild pointers in the queues).
To keep the clutter down a bit, we occasionally use raw pointers when we can be
reasonably certain that domestication is assured. Moreover, we define some type
aliases, `FreeObject::{HeadPtr, QueuePtr, AtomicQueuePtr}`, that are slightly
more convenient labels than, e.g., `CapPtr<FreeObject::T<BQueue>, BView>`.
Because we are using template parameters for the `capptr::bound<>`s themselves,
we cannot use the aliases for `CapPtr<>s` provided within `capptr::`.
The two primary interfaces around free objects (`FreeListIter` AND
`FreeListBuilder`) are adjusted appropriately and their `BView` and `BQueue`
template paramters are plumbed explicitly around the tree. This makes for quite
a bit of noise at the moment, but means that we'll be able to evolve parts of
the tree separately and can consider putting defaults in once that's done.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
# Free List builder track length
This commit makes the free list builder track the length of the lists in
the Random case.
# Refactor free list creation.
Minor refactoring to share code between the new free list and existing
path.
# Randomise slab filling
Knowing when a slab is going to become full makes it easier to by pass
the free list entries as protection for OOB writes. This commit
randomises when a slab will become full.
This commit changes two things
* the free list builder can return some fraction of the deallocations
on a slab.
* when there is a single free slab, we can with some probability
allocate an additional slab.
These two combine to make it difficult to predict when a slab will be
free.
# Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This changes the slab lists to use a sequential queue.
They were previously stored in a stack.
This commit also tidies up some incomplete refactoring from the
initial snmalloc2 work.
Introduce Metaslab::from_link(SlabLink*) to encapsulate the "container of"
dance. Note that Metaslab was not a standard layout type prior to this change
(since both SlabLink and Metaslab defined non-static data members), and so the
reinterpret_cast<>s replaced here with ::from_link() were UB, but everyone lays
out classes as one expects so it was fine in practice.
Most of the uses of ::from_link() are already guarded by checks that the link
pointer is not nullptr, but in src/mem/corealloc.h:/debug_is_empty_impl we shift
to testing the link pointer explicitly before converting to the metaslab.
Despite that Metaslab is now standard layout, we still don't fall back to the
inter-convertibility of a standard layout class and its first[*] data member
since we're going to want to put a common initial sequence across Metaslab and
ChunkRecord and the SlabLink isn't likely to be in it.
With snmalloc2, slabs are linked through the Metaslab structure directly rather
than in-band in a free allocation, so we no longer need to store a SlabLink in
even the smallest allocation classes.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
This commit adds a simple XOR encoding to the next_object pointer in
FreeObjects. This removes the trivial way of getting hold of a physical
address from the system by observing the free list pointers in
deallocated objects.
It is UB to offset from `nullptr` (except perhaps with a 0 offset). Apparently
clang is able to use this to reason, given `void* p`, that comparing
`__builtin_align_down(p, x)` against `handle.fake_large_remote` (i.e., a `static
inline constexpr` `nullptr`) must be the same as comparing `p` itself against
`nullptr`.
In `MetaEntry`'s constructor, converting the provided `RemoteAllocator*` to
`uintptr_t` before offsetting avoids the UB. (While here, don't use
`address_cast()`, as `address_t` will, on CHERI, be `ptraddr_t` and not
`uintptr_t`.)
Doing the alignment in `get_remote` at `uintptr_t` before casting to
`RemoteAllocator*`, rather than converting and then aligning, prevents the
reasoning above from eliminating the alignment.
This extends the freelist protection to the remote message queues. They
effectively perform doubly linked list entries for the message queue
with the enqueue operation first linking in the previous pointer, and
then then atomically setting the next. This ensures that the visible
states always satisfy the invariant that the forward and backward
pointers are correct for any visisble object.
There is a key_global that is used for all remote deallocations. The
remote cache uses the same protection to build the temporary lists
before forwarding to the next allocator.
The mpscq is integrated into the remoteallocator as it is no longer
a reusable datastructure, but a special purpose implementation.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
Continue tightening the screws on pointer bounds.
Notably, pointers in remote queues are bounded to the free objects. While we
believe that something like MTE is required to make in-band metadata safe, this
is a kind of defense in depth for StrictProvenance architectures: UAF for small
and medium objects expose mostly other (free) small or medium objects and not
allocator metadata (modulo some potential aliasing when Superslabs and
Mediumslabs interconvert). This might shift the burdon on an attacker from
simply holding a UAF pointer to having had to farm several heap pointers.
The policy of bounding remote queue pointers may make the allocator's behavior
for small objects unexpected: while initial object construction during
allocation (that is, when the free list is empty) continues to cleave out
exportable pointers from elevated pointers to internal slabs, reuse pulls from
free lists of *already-bounded* objects. These objects are queued by the
deallocation side, of course, but these paths now include "parallel
reconstruction" of a pointer to the free object from the amplified view of the
returned pointer, rather than queueing amplified pointers and leaving
reconstruction to the allocation side.
Medium objects are possibly similarly mysterious with the added twist that
medium slabs do not store pointers but rather always cleave from their
self-reference (but their interface has always operated using pointers).
Nevertheless, pointers to medium objects end up in remote queues, so we continue
to engage in "parallel reconstruction" in the deallocation paths.
Even if we opt not to bound these pointers internally (if they aren't headed out
to the user program or we later derive bounded pointers), they should still be
annotated as something other than CBArena, ensuring that we do not attempt to
use them for general amplification.
Begin turning the screws on bounds: pointers the allocator is about to reveal
must be annotated as CBAllocE. Use the PAL's capptr_export and the AAL's
capptr_bound<> to get them there.
Allocate slab is randomly in all possible permutations. This increases
the entropy of the order considerably. This uses an algorithm to build
a random cycle in a slab, and then use this to build the free list.
We disable the per-slab randomisation in the non-CHECK_CLIENT builds.
This wrapper will allow us to pass `AuthPtr<T,B> p` to zero() without needing to
write `p.unsafe_auth_ptr` to get to a `T*` inside. Moreover, it will give us a
convenient point to assert that `B` is such that the pointer can be used to
manipulate the memory map (i.e. is not exported).
Define various parts of random that can be used to make the layout of
memory more random. Thread this through the allocator.
Expose the concept as part of the Pal. Subsequent commits will expose
that on different platforms.
When a slab has been fully allocated, then we no longer
check it has entries until something returns an allocation to this slab.
However, it is possible that only a single allocation is available, and
then we can end up frequently on the slow path.
This change only considers free lists that cover at least 1/8 of a slab.
This means that we will hit the slow path less frequently. This also
means that the randomisation changes will have more entropy: with a
single element free list there is only one order.
For large small sizes it can still be a single element, as 1/8 is of the
slab capacity is below 1. We max out the trigger at 31 elements to
reduce unneeded wasted space.
The slab allocation pattern is randomised based on the deallocation
pattern. This achieved by using two queues to enqueue free elements
onto. We pick "randomly", which queue to add to, and then when we take
the free_queue to use, we splice the two queues together.
The metaslab contains a field specifying how many elements have been
allocated. As the code has evolved this field has now always become
the maximum capacity of the slab for the sizeclass.
This commit looks up this value based on the sizeclass, and removes the
field from the slab's metadata.
The previous reciprocal division branch on the prime that the sizeclass
was constructed from. All sizeclasses can be represented as
2^n * {1,3,5,7}
This lead to a very small table, but some work to calculate the
appropriate shifts and multiplications to implement reciprocal division.
This commit uses a completely uniform representation for every
sizeclass using a lookup table. Due to the precise ranges that we query
the modulus and rounding on, we can do this much more efficiently.
The func-release-rounding exhaustively tests all the queries we are
interested in.
Free list pointers can be exploited by attackers. This commit implements
a simple encoding scheme to detect corruption of the pointers. This can
be used to detect UAF and double free.
This does not currently address anything for Medium or Large
allocations. It also does not address cross thread deallocations.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
Going forward, this gives us explicit pointers with which to carry bounds
annotations. Otherwise, assuming AuthPtr overloads operator->, a OOP-style call
like
AuthPtr<Slab, Bounds> slab;
slab->foo()
will create a `Slab* this` within the body of `Slab::foo`, leaving it unable to
see or propagate the Bounds annotation. If it invokes callees that expect
`AuthPtr` arguments, it will therefore have to fabricate new `Bounds` unsafely.
The link object was previously stored in a disused object. This is
good for reducing meta-data, but if we want to reduce the meta-data
corruption potential, then this is not a good design choice.
This commit moves it into the Metaslab.
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
For architectures that can't manipulate pointers like integers, don't
try XORing them like this. It's not ideal -- perhaps we should have
"else" branches to these tests.
