Allocate slab is randomly in all possible permutations. This increases
the entropy of the order considerably. This uses an algorithm to build
a random cycle in a slab, and then use this to build the free list.
We disable the per-slab randomisation in the non-CHECK_CLIENT builds.
Like we already do in the size-unknown case. This ensures that in the CHERI
case we'll always bound to the size of the segment we return to the large alloc,
not to a subset. It has the added bonus of ensuring that our bounds are
precisely representable.
This wrapper will allow us to pass `AuthPtr<T,B> p` to zero() without needing to
write `p.unsafe_auth_ptr` to get to a `T*` inside. Moreover, it will give us a
convenient point to assert that `B` is such that the pointer can be used to
manipulate the memory map (i.e. is not exported).
When we get to CHERI, sizes derived from sizeclasses are always precisely
representable, while other sizes may not be.
This should induce no change in behavior without CHERI, except that we might
PAL::zero slightly more memory this way.
Define various parts of random that can be used to make the layout of
memory more random. Thread this through the allocator.
Expose the concept as part of the Pal. Subsequent commits will expose
that on different platforms.
On CHERI, the compiler will always issue a warning for
`reinterpret_cast<T*>(ptraddr_t)` and similar expressions, and of course, if the
compiler can see far enough into the types, the presence of `if constexpr` will
not save us. Therefore, lift the conditional out to two definitions of
`FreeObject::encode` using `std::enable_if_t` to gate which is used.
When a slab has been fully allocated, then we no longer
check it has entries until something returns an allocation to this slab.
However, it is possible that only a single allocation is available, and
then we can end up frequently on the slow path.
This change only considers free lists that cover at least 1/8 of a slab.
This means that we will hit the slow path less frequently. This also
means that the randomisation changes will have more entropy: with a
single element free list there is only one order.
For large small sizes it can still be a single element, as 1/8 is of the
slab capacity is below 1. We max out the trigger at 31 elements to
reduce unneeded wasted space.
The slab allocation pattern is randomised based on the deallocation
pattern. This achieved by using two queues to enqueue free elements
onto. We pick "randomly", which queue to add to, and then when we take
the free_queue to use, we splice the two queues together.
* Replace time measuring macro
The DO_TIME macro was used originally to get performance numbers. The
macro makes tests hard to debug. This commit replaces it with a proper
C++ class with destructor.
* Bug fix
If the superslab meta data is large, then the calculation for the
sizeclasses that could use the short slab was incorrect. This fixes
that calculation.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
The metaslab contains a field specifying how many elements have been
allocated. As the code has evolved this field has now always become
the maximum capacity of the slab for the sizeclass.
This commit looks up this value based on the sizeclass, and removes the
field from the slab's metadata.
The initial performance monitoring for snmalloc used timing of small
operations to guide the design. This feature has not been maintained or
used for several years.
This commit removes the feature.
This was original designed for Project Snowflake to enable a careful
interoperation between an allocator and the thread suspend behaviour in
.NET.
This feature is not being tested or used by any current project. This
form of interop would be better served by designing a special Pal to
interop with the CLR if this is ever needed.
This commit removes the feature.
The previous reciprocal division branch on the prime that the sizeclass
was constructed from. All sizeclasses can be represented as
2^n * {1,3,5,7}
This lead to a very small table, but some work to calculate the
appropriate shifts and multiplications to implement reciprocal division.
This commit uses a completely uniform representation for every
sizeclass using a lookup table. Due to the precise ranges that we query
the modulus and rounding on, we can do this much more efficiently.
The func-release-rounding exhaustively tests all the queries we are
interested in.
Rather than ::get()-ing the `Superslab` and `Slab` for each object we just
created from the bump pointer, recognize that these objects necessarily come
from the same `Slab` (and so the same `Superslab`). In the eventuality of
CHERI, this means we'll amplify once per bump pointer, not once per created
object.
When post()-ing the RemoteCache to message queues, we push an entire bucket
onto a remote allocator's incoming queue (specifically, the allocator owning
the front Remote in the bucket we're moving). In order to do that, we need to
exceed the bounds of the Remote allocation and reference its Allocslab header
(to get the ->message_queue). On StrictProvenance architectures, this will
require that we amplify the head Remote* and then engage in some pointer math.
While Remotes contain the address of the message_queue as the allocator's
identity, this may not be a pointer, just an address, and may have undergone
obfuscation anyway.
Free list pointers can be exploited by attackers. This commit implements
a simple encoding scheme to detect corruption of the pointers. This can
be used to detect UAF and double free.
This does not currently address anything for Medium or Large
allocations. It also does not address cross thread deallocations.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This requires that the caller perform the cast on the output rather than the
input, which is a little closer to the truth. Shuffle some casts into the right
position.
Like alloc_size, this will require amplification internally.
This patch also restores performance to the status quo ante; Clang can once
again see enough to generate the same code as it did before de-static-ing
alloc_size.
We're going to need to amplify the pointer and that's going to require access
to our AddressSpaceManager, which we only get non-statically through our
LargeAlloc.
This patch unto itself makes the world slower, perhaps because Clang can't see
the certainty of aliasing of the static and non-static paths to the same
structure. However, when we also de-static external_pointer, that goes away and
things return to the status quo ante.
Going forward, this gives us explicit pointers with which to carry bounds
annotations. Otherwise, assuming AuthPtr overloads operator->, a OOP-style call
like
AuthPtr<Slab, Bounds> slab;
slab->foo()
will create a `Slab* this` within the body of `Slab::foo`, leaving it unable to
see or propagate the Bounds annotation. If it invokes callees that expect
`AuthPtr` arguments, it will therefore have to fabricate new `Bounds` unsafely.
This will let us use Pagemaps further down the dependency stack (specifically,
we're going to want a Pagemap inside the AddressSpaceManager) by letting us
manually tie the knot rather than rely on GlobalVirtual and
default_memory_provider() being defined by the time we want a Pagemap.
Presently, GlobalPagemap and ExternalPagemap discriminate only by type. If it
ever happened that multiple PageMap consumers instantiated the same type using
these wrapper, they'd be conflated in the symbol table. Therefore, add an
optional Purpose parameter that will be expanded into the symbols (but serves no
other purpose).
- Make GlobalPagemapTemplate and ExternalGlobalPagemap generic in the type of
the pagemap they're encapsulating.
We're going to want to use these for other kinds of pagemaps in the near
future.
- Rename snmalloc_pagemap_global_get to snmalloc_chunkmap_global_get.
- Rename GlobalPagemap to GlobalChunkmap.
The link object was previously stored in a disused object. This is
good for reducing meta-data, but if we want to reduce the meta-data
corruption potential, then this is not a good design choice.
This commit moves it into the Metaslab.
Replace the generic check_size() calls with per-{small,medium,large} code paths,
which follow the progression...
* _unchecked: no validation has been performed; check the chunkmap and move
to...
* _checked_chunkmap: the chunkmap indicates that this is the correct slab-type,
and so we have reason to believe that our supposed Allocslab pointers are of
the correct types. Use those pointers to read the sizeclass and move to...
* _checked_sizeclass: additionally, the Allocslab metadata confirms that our
sizeclass matches the expected value. Check that we are at the start of an
object and move to...
* _start: we now believe we have a pointer to the start of a live object. On
a sufficiently capable architecture, we will eventually atomically check that
our pointer is not (scheduled to be) revoked (and/or not reversioned). If
this test passes, our prior pointer arithmetic and loads are justified
(assuming correctness of the quarantine/revocation/reuse) machinery.
The size-less dealloc(void*) path by necessity reads the chunkmap and sizeclass
data itself and so there's no reason to re-validate; as such, it jumps directly
to the _okcmsc point. Similarly, we assume that remote queues are full of
already validated objects, and so the remote handling paths continue to jump
further along even than the _start methods.
MSVC has strong opinions on implicit conversions as used in CI, while Clang both
locally and in CI has weaker opinions. In an effort to avoid subsequent
roundtrips through CI, make clang more strict. Adding -Wconversion definitely
increases the strength of clang's opinions, apparently to include frowning on
some that even MSVC considers OK, so go make explicit the current implicit
behavior.
* Make LowMemoryNotification object allocated
This makes a separate allocation for the callback object. This makes
it easier for different callbacks to be used.
* Add reserve_with_leftover to address_space
The address_space now supports reserving for non-power of 2 allocations
and the space that is used for rounding up is retained by the
address_space. This means that we can more tightly pack the allocators
internal objects.
Confirm that the client has given dealloc() a start-of-object pointer before
that pointer propagates very far (esp., before it enters the remote queues).
As these tests are not free, structure the code so that future interfaces can
skip over them if there's reason to be sure we're at the start of an object.
(On CHERI, for example, one could imagine using sealed pointers.)
For architectures with SSM/MTE, immediately after this test we can update the
memory granule to guard against double-frees; leave TODOs in place.
Summary of changes:
- Add a new PAL that doesn't allocate memory, which can be used with a
memory provider that is pre-initialised with a range of memory.
- Add a `NoAllocation` PAL property so that the methods on a PAL that
doesn't support dynamically reserving address space will never be
called and therefore don't need to be implemented.
- Slightly refactor the memory provider class so that it has a narrower
interface with LargeAlloc and is easier to proxy.
- Allow the address space manager and the memory provider to be
initialised with a range of memory.
This may eventually also remove the need for (or, at least, simplify)
the Open Enclave PAL.
This commit also ends up with a few other cleanups:
- The `malloc_useable_size` CMake test that checks whether the
parameter is const qualified was failing on FreeBSD where this
function is declared in `malloc_np.h` but where including
`malloc.h` raises an error. This should now be more robust.
- The BSD aligned PAL inherited from the BSD PAL, which does not
expose aligned allocation. This meant that it exposed both the
aligned and non-aligned allocation interfaces and so happily
accepted incorrect `constexpr` if blocks that expected one or
the other but accidentally required both to exist. The unaligned
function is now deleted so the same failures that appear in CI should
appear locally for anyone using this PAL.
