This dates back to the much earlier design that required the use of an authority
map. Since the current CHERI design does not, go ahead and ask the platform
whenever underlying allocation requests are sufficiently aligned.
This preserves the chunk pointer through the use of a chunk as a slab. It does
grow the structure by one pointer, but on non-CHERI it is still padded to 64
bytes, even with CHECK_CLIENT guards in place:
0: MetaCommon chunk pointer
8: next pointer
16: builder head[0]
24: builder head[1]
32: builder tail[0]
40: builder tail[1]
48: builder length[0] (uint16_t)
50: builder length[1] (uint16_t)
52: padding (4 bytes)
56: needed (uint16_t)
58: sleeping (bool)
(Sadly, on CHERI, even without CHECK_CLIENT guards and with no padding, there
are now four pointers in the structure -- chunk, next, head, tail -- plus five
extra bytes. We will likely wish to explore encoding the head and tail offsets
relative to the chunk pointer.)
This lets us remove the "subversive amplification" in dealloc() in favor of just
preserving the chunk pointer. Speaking of, be sure to assign that in all the
right places, and ASSERT that we've got it right.
The use of void* had let an overzealous unsafe_ptr() leak a pointer with address
space control to the client (in LocalAllocator::alloc_not_small, specifically).
Correct this to call capptr_chunk_is_alloc() (to capture our intent) and
capptr_to_user_address_control() (to do the bounding) and defer the conversion
to void* until the very periphery of the allocator, using capptr_reveal()
(again, to capture intent).
- Grab a larger second allocation on the first allocator to dodge the sizeclass
of the prior alloc on that allocator *and* any implicit, bootstrapping slabs
that get opened (e.g., for remote queue message stubs).
- De-FAST_PATH the domestication function. No need to always inline it, here.
- Document things a little better
Avoid computing bits::next_pow2_bits(1 << n). Even if the compiler can see
through enough of the algebra, it's surely more direct to just use n.
While here, slightly expand documentation about what's going on with the
"sizeclass" encoded into MetaEntry-s.
capptr_rebound was only ever going to be used for external_pointer, which now
operates entirely using pointer_offset. So instead, just make external_pointer
use capptr::AllocWild<void>, capptr_from_client, and a new capptr_reveal_wild.
There is no such thing as "struct Slab" any more.
We use alignof(RemoteAllocator) below, so we already require the complete type
definition at this point.
We'll want user_address_control_type in some particular PALs, so it can't live
in pal.h.
While here, make the spelling be capptr::is_spatial_refinement.
- `check_result()` `abort()` on `null` and non-`nullptr` result. Otherwise it
just prints and doesn't end the test
- Don't call `realloc(, 0)`; this has never been consistent and the current C2x
draft (see §7.22.3.5 and N2464) finally just declares it to be undefined
behavior. POSIX (2017) tolerates our current behavior of freeing the
passed-in pointer and returning a new object.
The free list builder in a checked build will only validate entries when
they are removed. This commit adds a validate method, so they can be
checked during teardown. This means that programs that leak memory
will still fail if the free list has become corrupt.
This avoids repeated double-tapping domestication of the same pointer in
!QueueHeadsAreTame builds, by keeping the current "front" pointer to the queue
in trusted locations (stack, register) rather than storing it back to possibly
client-accessible memory.
Instantiate two allocators and arrange for a message to get passed between them
by exploiting the existing slow-paths' handling of message queues. Count and
CHECK the number of domestication calls during this message passing. For a
little more excitement, pave over the forward pointer in the freelist::Object::T
that is the message and have the domestication callback patch the original value
back; should we somehow fail to invoke the domestication callback on that
address, this will induce a crash (WHP, on both CHECK_CLIENT and unchecked
builds).
Motivated by renaming `FreeObject::{Head,Queue,AtomicQueue}Ptr` to
`freelist::...Ptr`, in fact go further, moving `FreeObject` itself to
`freelist::Object` and `FreeListBuilder` to `freelist::Builder` and
`FreeListIter` to `freelist::Iter`
Now that explicit annotations have gotten us through the refactoring, it's time
for the scaffolding to disappear. src/mem/freelist.h is left generic for any
future machinations, but `FreeObject::T<>`, the several `FreeObject::...Ptr<>`s,
`FreeListIter<>`, and `FreeListBuilder<>` are given default parameters and all
uses are shortened to use defaults where possible.
Without this it complains that LocalCache's constructor can't be constexpr
because small_fast_free_lists isn't initialized. It's not clear to me why it
didn't mind before, and nobody else seems to mind now, but this shouldn't break
anyone else, either.
Just an intermediate syntactic step to chase dependencies. All these introduced
"domestication" callbacks are just the identity function, but they will let us
thread the LocalAlloc's handle to the Backend state down to where it's needed.
`capptr_domesticate<Backend>(Backend::LocalState*, CapPtr<T, B>)` is the
intended affordance for conversion to covert from a `CapPtr<T, B>` with
`B::wildness` `Wild` to a `CapPtr<>` with `B::with_wildness<Tame>` and thence
plumbed into the rest of the machinery.
David added the SFINAE wrapper so that `Backend`-s now don't need to implement a
domestication callback; instead, if the `Backend` does not provide a
`capptr_domesticate` function, a default, which just does an explicit type cast,
will be used instead.
This is not yet hooked into the rest of the tree.
Co-authored-by: David Chisnall <David.Chisnall@microsoft.com>
This is incomplete, yet still more reflective of what's going on: we take the
exported pointers back from userspace and thread them directly into the free
lists.
So: move capptr_to_user_address_control to list construction time rather than
list consumption time.
Rather than open-code the conversion from &f->next_object to f, add a static
method to FreeObject and take the opportunity to add a static_assert that our
reinterpret_cast is sound.
If we're going to check next's prev in atomic_read_next, we will need to
domesticate the next pointer first. We could push the check up, but that opens
boxes, so it's simpler to plumb domestication this far down. For symmetry, we
also plumb to (non-atomic) read_next.
FreeObject itself is now just a namespace (but `friend`-ly); the actual free
list nodes are FreeObject::T-s that are templatized on the (perceived)
`capptr::bound<>` of the pointer they contain. (These may differ across an
instantiated snmalloc; for example, in the sandboxing design, the in-sandbox
allocators may perceive all remotes to be full of `AllocUser` while the
privileged allocator of sandbox memory should perceive its remote queue as
holding `AllocUserWild` pointers in need of domestication.)
The interfaces to `FreeObject::T`-s now let us distinguish between the base and
inductive cases of the queues:
* in the inductive case, the pointer we hold to a `FreeObject::T` and its
next_object have the same bounds
* in the base case, the pointer we hold has different bounds (typically,
domesticated by contrast to the wild pointers in the queues).
To keep the clutter down a bit, we occasionally use raw pointers when we can be
reasonably certain that domestication is assured. Moreover, we define some type
aliases, `FreeObject::{HeadPtr, QueuePtr, AtomicQueuePtr}`, that are slightly
more convenient labels than, e.g., `CapPtr<FreeObject::T<BQueue>, BView>`.
Because we are using template parameters for the `capptr::bound<>`s themselves,
we cannot use the aliases for `CapPtr<>s` provided within `capptr::`.
The two primary interfaces around free objects (`FreeListIter` AND
`FreeListBuilder`) are adjusted appropriately and their `BView` and `BQueue`
template paramters are plumbed explicitly around the tree. This makes for quite
a bit of noise at the moment, but means that we'll be able to evolve parts of
the tree separately and can consider putting defaults in once that's done.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
# Free List builder track length
This commit makes the free list builder track the length of the lists in
the Random case.
# Refactor free list creation.
Minor refactoring to share code between the new free list and existing
path.
# Randomise slab filling
Knowing when a slab is going to become full makes it easier to by pass
the free list entries as protection for OOB writes. This commit
randomises when a slab will become full.
This commit changes two things
* the free list builder can return some fraction of the deallocations
on a slab.
* when there is a single free slab, we can with some probability
allocate an additional slab.
These two combine to make it difficult to predict when a slab will be
free.
# Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This changes the slab lists to use a sequential queue.
They were previously stored in a stack.
This commit also tidies up some incomplete refactoring from the
initial snmalloc2 work.
The various Pals were given different meanings in CHECK_CLIENT and
non-CHECK_CLIENT builds. This was because it is essential
that in the CHECK_CLIENT builds access is prevented, when not requested.
This PR separates the CHECK_CLIENT concept from how the Pal should be
implemented.
This exposes a readonly notify using, so that the underlying platform
can map the range of pages readonly into the application. This improves
performance of external pointer on platforms that support lazy commit
of pages as it can access anything in the range.
* Add extra key to freelist. This follows the encoding Cedric suggested
for a signature of two things. Free list key now has a pair of keys
for encoding previous pointer. This makes it harder to extract the
underlying keys out of the multiplication.
* Apply SFINAE to the extract_segment.
