The constructor for atomic_flag is challenging to use in a constexpr.
It requires
std::atomic_flag flag = ATOMIC_FLAG_INIT;
which is not constexpr on some compilers in C++20.
Switching to atomic_bool solves this problem.
These encapsulate the wildly powerful reinterpret_cast<> operator where one side
is a uintptr_t and the other is a native pointer. In both cases we require the
pointer type to be explicitly given.
# Small changes before rewrite
* Additional bit in remote allocator to prevent type confusion with the backend.
* Move Chunk allocator to backend.
* Improvements to RedBlack tree
* Expose message from Pal
# Complete backend rewrite
This provides two key changes:
* We use buddy allocators to allow memory to reconsolidated
* The backend is factored into a series of small operations that
allocate and deallocate memory.
The backend now uses "Ranges", there are two ranges that don't require a
parent range:
* EmptyRange - Never returns any memory
* PalRange - Returns memory from the platform.
All other ranges require a parent range to supply memory to them. Some
ranges support both allocation and deallocation, and some just
deallocation. For instance, CommitRange supports both, and maps
requests to the parent range, but will Commit and Decommit the memory.
As the ranges perform only a single task, they are generally small and
easy to follow. The two exceptions to this are the two BuddyRanges
(Large and Small). Large is for CHUNK_SIZE and above blocks, while
Small is for below CHUNK_SIZE blocks. Both are implemented with a buddy
allocator, but the SmallBuddyRange uses in place meta-data, while the
LargeBuddyRange uses the pagemap for its meta-data. This means the
LargeBuddyRange can keep the majority of memory it is managing
decommitted.
The Backend glues together the various ranges to support the appropriate
way to manage memory on the platform.
- Refactor the existing SNMALLOC_ASSERT and SNMALLOC_CHECK. These now
use the FatalErrorBuilder to format the output if a format string is
provided.
- Extend the FatalErrorBuilder to print decimal integers for signed
values.
- Rename FatalErrorBuilder to MessageBuilder.
- Rewrite the macros used in the jemalloc tests to use
FatalErrorBuilder and move them into a header.
- Refactor some of the tests to use the new macros.
This introduces a very limited formatter that can embed strings and hex
representations of pointers / integers in an internal buffer. This is
used to format error strings for passing to `Pal::error`. This is used,
in turn, by a wrapper for reporting bounds checks, which can be used by
external functions to implement bounds checks.
This removes the sprintf_l usage from the bounds checks.
This provides enough of a format implementation that the tests
introduced in #465 can be refactored to use this, instead of their
custom `printf` wrapper and that can be used by SNMALLOC_CHECK. This
will be a follow-on PR.
- Mark the hook that we're exporting for the threading library to call
to clean up per-thread malloc state as 'used'. It was changed to
`inline` to allow duplicate copies of it to be merged but this also
means that it isn't emitted at all in compilation units that don't
use it (and it isn't used internally at all).
- Fix the `__je_bootstrap_*` functions, which are used to bootstrap TLS
allocation, for the changes to `ScopedAllocator`.
The `__je_bootstrap*` functions weren't being built in CI. They now are
for non-PIE targets with a smoke test.
If there is only one slab remaining, then we probabalisticly allocator a
new one. If a slab is barely in use, then this could cause us to
effectively double the number of slabs in use.
This commit checks if the remaining slab has enough remaining elements
to provide randomisation.
This adds a way to periodically pool the PAL to see if any timers have
expired. Timers can be used to periodically provide callbacks to the
rest of snmalloc.
Consuming available slabs in LIFO order makes predicting address reuse harder
but appears to have performance implications. Condition this on CHECK_CLIENT
and instead use FIFO order on !CHECK_CLIENT builds.
This adds a CHERI AAL and expands the FreeBSD PAL to cover CHERI. It updates a
comment in ds/address.h now that there is an example architecture that
differentiates uintptr_t and address_t.
capptr_rebound was only ever going to be used for external_pointer, which now
operates entirely using pointer_offset. So instead, just make external_pointer
use capptr::AllocWild<void>, capptr_from_client, and a new capptr_reveal_wild.
We'll want user_address_control_type in some particular PALs, so it can't live
in pal.h.
While here, make the spelling be capptr::is_spatial_refinement.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
This changes the slab lists to use a sequential queue.
They were previously stored in a stack.
This commit also tidies up some incomplete refactoring from the
initial snmalloc2 work.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
Fortunately, C++ taketh away and C++ giveth, both, so here we are: a way to
detect if we're in the middle of definining a type that uses itself as a
template parameter in a way that flows into a concept check and, if so,
short-circuit out of the need to actually do any checks. Wonders never cease.
- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
Also add a check that the test programs are under about ten megabytes
(they're currently around one on platforms that put inline statics full
of zeroes into BSS and around 270 on ones that don't).
Fixes#339
A few highlights relative to our existing CI:
- Add a FreeBSD 12.2 and 13.0 runner so we have some FreeBSD CI.
- Windows builds use msbuild with the Visual-Studio-provided clang toolchain to test clang
- The matrix builds describe the axes of the matrix, not all points.
- The Arm builds now cross-compile with a native clang and run the tests with qemu, rather than running the compiler, linker, and ctest all with qemu.
This also includes a fix for one of the tests that was doing `static_cast<unsigned int>(1) << 36`, which is undefined behaviour and was sometimes causing qemu to hang. There is now an assert to catch this in the future.
This commit adds a simple XOR encoding to the next_object pointer in
FreeObjects. This removes the trivial way of getting hold of a physical
address from the system by observing the free list pointers in
deallocated objects.
The pagemap allocates it self directly either from
* the original fixed address range it is supplied, and returns the
remaining space after the pagemap is removed; or
* directly allocated from the PAL without using the address space
manager.
This change in layering is required for the next commit, which imposes
the invariant that the pagemap has been committed for all spaced managed
by the address space manager.
This extends the freelist protection to the remote message queues. They
effectively perform doubly linked list entries for the message queue
with the enqueue operation first linking in the previous pointer, and
then then atomically setting the next. This ensures that the visible
states always satisfy the invariant that the forward and backward
pointers are correct for any visisble object.
There is a key_global that is used for all remote deallocations. The
remote cache uses the same protection to build the temporary lists
before forwarding to the next allocator.
The mpscq is integrated into the remoteallocator as it is no longer
a reusable datastructure, but a special purpose implementation.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
