C++20 does away with trivial initializers for std::atomic<T>, which means our
global pagemaps always get zeroed, sometimes after other static ctors have run
(fun fun!). Use the new std::atomic_ref<T> when available. Abstract all this
behind an #ifdef-ful wrapper.
The CapPtr refactoring was largely compiler guided; unfortunately, it turns out
that the static-sized alloc function is not well exercised. As such, some type
errors and unnecessary unsafety lurked behind missing template instantiation.
Correct those and add calls to the test harness to make sure we always generate
at least one instance of each small/medium/large case. While here, it doesn't
hurt to make sure that we call all three possible dealloc() flavors as well.
This will, if nothing else, force instantiation of the static-sized dealloc
template as well.
Continue tightening the screws on pointer bounds.
Notably, pointers in remote queues are bounded to the free objects. While we
believe that something like MTE is required to make in-band metadata safe, this
is a kind of defense in depth for StrictProvenance architectures: UAF for small
and medium objects expose mostly other (free) small or medium objects and not
allocator metadata (modulo some potential aliasing when Superslabs and
Mediumslabs interconvert). This might shift the burdon on an attacker from
simply holding a UAF pointer to having had to farm several heap pointers.
The policy of bounding remote queue pointers may make the allocator's behavior
for small objects unexpected: while initial object construction during
allocation (that is, when the free list is empty) continues to cleave out
exportable pointers from elevated pointers to internal slabs, reuse pulls from
free lists of *already-bounded* objects. These objects are queued by the
deallocation side, of course, but these paths now include "parallel
reconstruction" of a pointer to the free object from the amplified view of the
returned pointer, rather than queueing amplified pointers and leaving
reconstruction to the allocation side.
Medium objects are possibly similarly mysterious with the added twist that
medium slabs do not store pointers but rather always cleave from their
self-reference (but their interface has always operated using pointers).
Nevertheless, pointers to medium objects end up in remote queues, so we continue
to engage in "parallel reconstruction" in the deallocation paths.
Mediumslabs are strung on a dllist and used to feed the allocator there. If we
ensure that these (and the root pointer to the list itself) are already
exported, then our alloc paths can bound these to arrive at exposible pointers.
The dealloc paths, where we might want a non-exported pointer, already have one,
as they have gone through amplification to get an arena-bounded pointer.
The sole wrinkle in this plan is that we might need a pointer without platform
constraints to manipulate the memory map for page-based zeroing. Since we have
ample room in the Mediumslab header (a few kilobytes end up being used for
padding; the curious should see b8b5f305 and 3d3b0487), just cache therein a
copy of the CBChunk-bound pointer used in Mediumslab::init() for ::alloc().
Even if we opt not to bound these pointers internally (if they aren't headed out
to the user program or we later derive bounded pointers), they should still be
annotated as something other than CBArena, ensuring that we do not attempt to
use them for general amplification.
Begin turning the screws on bounds: pointers the allocator is about to reveal
must be annotated as CBAllocE. Use the PAL's capptr_export and the AAL's
capptr_bound<> to get them there.
* The AddressSpaceManager now requests address space in specified granule
sizes and registers those allocations with an external ArenaMap.
* The DefaultArenaMap is a (somewhat erroneously named) Pagemap sparse array /
tree for these provenance roots. Nothing is stored on non-StrictProvenance
architectures.
* In the Sandbox test, give an example of a different ArenaMap structure, which
confines amplification to sandbox memory.
* Adjust some other tests to compile.
The ArenaMap will store our internal provenance roots. This commit simply
introduces the data structure and does not wire it up to the rest of the tree.
Like all the annotations so far, leave these as CBArena-bounded.
However, we can start to do a little better than we were doing before: for
not-large deallocations, we know that the internal superslab pointer is
sufficiently authoritative that we can use it to get to data structures rather
than also passing an amplified pointer to the allocation in question. This,
then, partially reverts some of the earlier changes made while plumbing
CapPtr<>s through the system.
While here, avoid quite as many void*-s in favor of Remote* or SlabNext*.
Work with high-authority pointers internally, but then copy the address onto to
the low-authority pointer originally given to us (using the same primitive as
amplification, just backwards). This ensures that clients cannot behave
non-monotonically even if they have access to external_pointer(). That is, we
are not an amplification oracle, though we might reveal the address of an object
somewhere clients cannot otherwise access. Notably, we might reveal if a
{Super,Medium,}Slab has changed sizeclass (or been repurposed as or from a
Largealloc), forming a covert channel for data but not capabilities. Of course,
if the pointer provided to external_alloc() is still useful, the implied UAF in
these scenarios is still its own problem.
Document and suppress MSVC's confusion with CapPtr<>'s constructor (the
suppression has to happen at the definition, not the use, of the constructor) in
the case of never-needs-initialization Allocator-s.
Allocate slab is randomly in all possible permutations. This increases
the entropy of the order considerably. This uses an algorithm to build
a random cycle in a slab, and then use this to build the free list.
We disable the per-slab randomisation in the non-CHECK_CLIENT builds.
Like we already do in the size-unknown case. This ensures that in the CHERI
case we'll always bound to the size of the segment we return to the large alloc,
not to a subset. It has the added bonus of ensuring that our bounds are
precisely representable.
This wrapper will allow us to pass `AuthPtr<T,B> p` to zero() without needing to
write `p.unsafe_auth_ptr` to get to a `T*` inside. Moreover, it will give us a
convenient point to assert that `B` is such that the pointer can be used to
manipulate the memory map (i.e. is not exported).
When we get to CHERI, sizes derived from sizeclasses are always precisely
representable, while other sizes may not be.
This should induce no change in behavior without CHERI, except that we might
PAL::zero slightly more memory this way.
Define various parts of random that can be used to make the layout of
memory more random. Thread this through the allocator.
Expose the concept as part of the Pal. Subsequent commits will expose
that on different platforms.
On CHERI, the compiler will always issue a warning for
`reinterpret_cast<T*>(ptraddr_t)` and similar expressions, and of course, if the
compiler can see far enough into the types, the presence of `if constexpr` will
not save us. Therefore, lift the conditional out to two definitions of
`FreeObject::encode` using `std::enable_if_t` to gate which is used.
When a slab has been fully allocated, then we no longer
check it has entries until something returns an allocation to this slab.
However, it is possible that only a single allocation is available, and
then we can end up frequently on the slow path.
This change only considers free lists that cover at least 1/8 of a slab.
This means that we will hit the slow path less frequently. This also
means that the randomisation changes will have more entropy: with a
single element free list there is only one order.
For large small sizes it can still be a single element, as 1/8 is of the
slab capacity is below 1. We max out the trigger at 31 elements to
reduce unneeded wasted space.
The slab allocation pattern is randomised based on the deallocation
pattern. This achieved by using two queues to enqueue free elements
onto. We pick "randomly", which queue to add to, and then when we take
the free_queue to use, we splice the two queues together.
* Replace time measuring macro
The DO_TIME macro was used originally to get performance numbers. The
macro makes tests hard to debug. This commit replaces it with a proper
C++ class with destructor.
* Bug fix
If the superslab meta data is large, then the calculation for the
sizeclasses that could use the short slab was incorrect. This fixes
that calculation.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
The metaslab contains a field specifying how many elements have been
allocated. As the code has evolved this field has now always become
the maximum capacity of the slab for the sizeclass.
This commit looks up this value based on the sizeclass, and removes the
field from the slab's metadata.
The initial performance monitoring for snmalloc used timing of small
operations to guide the design. This feature has not been maintained or
used for several years.
This commit removes the feature.
This was original designed for Project Snowflake to enable a careful
interoperation between an allocator and the thread suspend behaviour in
.NET.
This feature is not being tested or used by any current project. This
form of interop would be better served by designing a special Pal to
interop with the CLR if this is ever needed.
This commit removes the feature.
The previous reciprocal division branch on the prime that the sizeclass
was constructed from. All sizeclasses can be represented as
2^n * {1,3,5,7}
This lead to a very small table, but some work to calculate the
appropriate shifts and multiplications to implement reciprocal division.
This commit uses a completely uniform representation for every
sizeclass using a lookup table. Due to the precise ranges that we query
the modulus and rounding on, we can do this much more efficiently.
The func-release-rounding exhaustively tests all the queries we are
interested in.
Rather than ::get()-ing the `Superslab` and `Slab` for each object we just
created from the bump pointer, recognize that these objects necessarily come
from the same `Slab` (and so the same `Superslab`). In the eventuality of
CHERI, this means we'll amplify once per bump pointer, not once per created
object.
When post()-ing the RemoteCache to message queues, we push an entire bucket
onto a remote allocator's incoming queue (specifically, the allocator owning
the front Remote in the bucket we're moving). In order to do that, we need to
exceed the bounds of the Remote allocation and reference its Allocslab header
(to get the ->message_queue). On StrictProvenance architectures, this will
require that we amplify the head Remote* and then engage in some pointer math.
While Remotes contain the address of the message_queue as the allocator's
identity, this may not be a pointer, just an address, and may have undergone
obfuscation anyway.
Free list pointers can be exploited by attackers. This commit implements
a simple encoding scheme to detect corruption of the pointers. This can
be used to detect UAF and double free.
This does not currently address anything for Medium or Large
allocations. It also does not address cross thread deallocations.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This requires that the caller perform the cast on the output rather than the
input, which is a little closer to the truth. Shuffle some casts into the right
position.
Like alloc_size, this will require amplification internally.
This patch also restores performance to the status quo ante; Clang can once
again see enough to generate the same code as it did before de-static-ing
alloc_size.
