* Add default for getting chunk allocator state
Makes the API same between the two configurations.
* Reduce address space usage for Open Enclave
* Fix OE Pal concept
* Add support for Pal not to provide time.
The lazy return of pages to the OS uses a simple time
based heuristic. This enables a PAL to not support time,
and return the memory to a central pool immediately.
* Update src/backend/backend.h
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
The primary aim for this refactor is to use a representation for
sizeclasses that uniformly covers both large and small. This allows
certain operations such as alloc_size and external_pointer to be
uniformly implemented.
The additional types make clear which kind of sizeclass is in use.
This also tidies up the code for sizeclass based divisible by and
modulus.
It fixes a bug in rust_realloc that didn't correctly determine a realloc
was required for large classes.
With the new snmalloc2 changes it seems the larger window is leading to
more fragmentation and harming performance. Reducing size still
provides good batching, improves memory overhead.
This adds a way to periodically pool the PAL to see if any timers have
expired. Timers can be used to periodically provide callbacks to the
rest of snmalloc.
Consuming available slabs in LIFO order makes predicting address reuse harder
but appears to have performance implications. Condition this on CHECK_CLIENT
and instead use FIFO order on !CHECK_CLIENT builds.
On systems with larger than 16KiB page size, we have chunks
that divide a page. This seems a little strange, and if we
want to disable the pages backing a chunk, this is not possible.
This change ensures the chunk is always at least a single page.
This preserves the chunk pointer through the use of a chunk as a slab. It does
grow the structure by one pointer, but on non-CHERI it is still padded to 64
bytes, even with CHECK_CLIENT guards in place:
0: MetaCommon chunk pointer
8: next pointer
16: builder head[0]
24: builder head[1]
32: builder tail[0]
40: builder tail[1]
48: builder length[0] (uint16_t)
50: builder length[1] (uint16_t)
52: padding (4 bytes)
56: needed (uint16_t)
58: sleeping (bool)
(Sadly, on CHERI, even without CHECK_CLIENT guards and with no padding, there
are now four pointers in the structure -- chunk, next, head, tail -- plus five
extra bytes. We will likely wish to explore encoding the head and tail offsets
relative to the chunk pointer.)
This lets us remove the "subversive amplification" in dealloc() in favor of just
preserving the chunk pointer. Speaking of, be sure to assign that in all the
right places, and ASSERT that we've got it right.
The use of void* had let an overzealous unsafe_ptr() leak a pointer with address
space control to the client (in LocalAllocator::alloc_not_small, specifically).
Correct this to call capptr_chunk_is_alloc() (to capture our intent) and
capptr_to_user_address_control() (to do the bounding) and defer the conversion
to void* until the very periphery of the allocator, using capptr_reveal()
(again, to capture intent).
Avoid computing bits::next_pow2_bits(1 << n). Even if the compiler can see
through enough of the algebra, it's surely more direct to just use n.
While here, slightly expand documentation about what's going on with the
"sizeclass" encoded into MetaEntry-s.
capptr_rebound was only ever going to be used for external_pointer, which now
operates entirely using pointer_offset. So instead, just make external_pointer
use capptr::AllocWild<void>, capptr_from_client, and a new capptr_reveal_wild.
There is no such thing as "struct Slab" any more.
We use alignof(RemoteAllocator) below, so we already require the complete type
definition at this point.
The free list builder in a checked build will only validate entries when
they are removed. This commit adds a validate method, so they can be
checked during teardown. This means that programs that leak memory
will still fail if the free list has become corrupt.
This avoids repeated double-tapping domestication of the same pointer in
!QueueHeadsAreTame builds, by keeping the current "front" pointer to the queue
in trusted locations (stack, register) rather than storing it back to possibly
client-accessible memory.
Motivated by renaming `FreeObject::{Head,Queue,AtomicQueue}Ptr` to
`freelist::...Ptr`, in fact go further, moving `FreeObject` itself to
`freelist::Object` and `FreeListBuilder` to `freelist::Builder` and
`FreeListIter` to `freelist::Iter`
Now that explicit annotations have gotten us through the refactoring, it's time
for the scaffolding to disappear. src/mem/freelist.h is left generic for any
future machinations, but `FreeObject::T<>`, the several `FreeObject::...Ptr<>`s,
`FreeListIter<>`, and `FreeListBuilder<>` are given default parameters and all
uses are shortened to use defaults where possible.
Without this it complains that LocalCache's constructor can't be constexpr
because small_fast_free_lists isn't initialized. It's not clear to me why it
didn't mind before, and nobody else seems to mind now, but this shouldn't break
anyone else, either.
Just an intermediate syntactic step to chase dependencies. All these introduced
"domestication" callbacks are just the identity function, but they will let us
thread the LocalAlloc's handle to the Backend state down to where it's needed.
`capptr_domesticate<Backend>(Backend::LocalState*, CapPtr<T, B>)` is the
intended affordance for conversion to covert from a `CapPtr<T, B>` with
`B::wildness` `Wild` to a `CapPtr<>` with `B::with_wildness<Tame>` and thence
plumbed into the rest of the machinery.
David added the SFINAE wrapper so that `Backend`-s now don't need to implement a
domestication callback; instead, if the `Backend` does not provide a
`capptr_domesticate` function, a default, which just does an explicit type cast,
will be used instead.
This is not yet hooked into the rest of the tree.
Co-authored-by: David Chisnall <David.Chisnall@microsoft.com>
This is incomplete, yet still more reflective of what's going on: we take the
exported pointers back from userspace and thread them directly into the free
lists.
So: move capptr_to_user_address_control to list construction time rather than
list consumption time.
Rather than open-code the conversion from &f->next_object to f, add a static
method to FreeObject and take the opportunity to add a static_assert that our
reinterpret_cast is sound.
If we're going to check next's prev in atomic_read_next, we will need to
domesticate the next pointer first. We could push the check up, but that opens
boxes, so it's simpler to plumb domestication this far down. For symmetry, we
also plumb to (non-atomic) read_next.
FreeObject itself is now just a namespace (but `friend`-ly); the actual free
list nodes are FreeObject::T-s that are templatized on the (perceived)
`capptr::bound<>` of the pointer they contain. (These may differ across an
instantiated snmalloc; for example, in the sandboxing design, the in-sandbox
allocators may perceive all remotes to be full of `AllocUser` while the
privileged allocator of sandbox memory should perceive its remote queue as
holding `AllocUserWild` pointers in need of domestication.)
The interfaces to `FreeObject::T`-s now let us distinguish between the base and
inductive cases of the queues:
* in the inductive case, the pointer we hold to a `FreeObject::T` and its
next_object have the same bounds
* in the base case, the pointer we hold has different bounds (typically,
domesticated by contrast to the wild pointers in the queues).
To keep the clutter down a bit, we occasionally use raw pointers when we can be
reasonably certain that domestication is assured. Moreover, we define some type
aliases, `FreeObject::{HeadPtr, QueuePtr, AtomicQueuePtr}`, that are slightly
more convenient labels than, e.g., `CapPtr<FreeObject::T<BQueue>, BView>`.
Because we are using template parameters for the `capptr::bound<>`s themselves,
we cannot use the aliases for `CapPtr<>s` provided within `capptr::`.
The two primary interfaces around free objects (`FreeListIter` AND
`FreeListBuilder`) are adjusted appropriately and their `BView` and `BQueue`
template paramters are plumbed explicitly around the tree. This makes for quite
a bit of noise at the moment, but means that we'll be able to evolve parts of
the tree separately and can consider putting defaults in once that's done.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
# Free List builder track length
This commit makes the free list builder track the length of the lists in
the Random case.
# Refactor free list creation.
Minor refactoring to share code between the new free list and existing
path.
# Randomise slab filling
Knowing when a slab is going to become full makes it easier to by pass
the free list entries as protection for OOB writes. This commit
randomises when a slab will become full.
This commit changes two things
* the free list builder can return some fraction of the deallocations
on a slab.
* when there is a single free slab, we can with some probability
allocate an additional slab.
These two combine to make it difficult to predict when a slab will be
free.
# Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This changes the slab lists to use a sequential queue.
They were previously stored in a stack.
This commit also tidies up some incomplete refactoring from the
initial snmalloc2 work.
* Add extra key to freelist. This follows the encoding Cedric suggested
for a signature of two things. Free list key now has a pair of keys
for encoding previous pointer. This makes it harder to extract the
underlying keys out of the multiplication.
* Apply SFINAE to the extract_segment.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
