This adds the full set of jemalloc functions that FreeBSD's libc
exposes, including some (the `*allocm` family) that are gone from newer
versions of jemalloc and the `*allocx` family that replaced them. These
are not necessarily efficient implementations but they should allow
snmalloc to replace jemalloc without any ABI breakage (in the loosest
possible sense).
Jemalloc provides a very generic sysctl-like mechanism for setting and
getting some values. These are all implemented to return the
not-supported error code. This may break code that expects that they
will succeed.
In particular, these APIs are used to register custom backing-store
allocators and to manage caches and arenas. These concepts don't map
directly onto snmalloc and attempting to do so would almost certainly
not provide the same performance characteristics and so it's better to
`LD_PRELOAD` jemalloc (or explicitly link to it) for programs that gain
a significant speedup from this.
- Mark the hook that we're exporting for the threading library to call
to clean up per-thread malloc state as 'used'. It was changed to
`inline` to allow duplicate copies of it to be merged but this also
means that it isn't emitted at all in compilation units that don't
use it (and it isn't used internally at all).
- Fix the `__je_bootstrap_*` functions, which are used to bootstrap TLS
allocation, for the changes to `ScopedAllocator`.
The `__je_bootstrap*` functions weren't being built in CI. They now are
for non-PIE targets with a smoke test.
* Improve testing of memcpy including adding perf test.
* Change remaining_bytes to be branch free.
Use reciprocal division followed by multiply to remove a branch.
* Post large deallocations to original thread
This change sets all large allocations to be owned by the originating
thread. This means they will be messaged back to the original thread
before they can be reused.
The following reason for making this change:
* This will improve producer/consumer apps involving large allocations.
* It enables the implementation of a more complex chunk allocator that
reassembles chunks.
* It addresses an issue with compartmentalisation where the handling of
large allocations can result in meta-data ownership changing.
* export netbsd's reallocarr proposal.
acts subtly differently from reallocarray, returns an error code
and first argument as receiver.
* not export by default
* ci tests
* apply suggestions
* doc addition
* Apply suggestions from code review
Co-authored-by: Matthew Parkinson <mjp41@users.noreply.github.com>
On Open Enclave having the `local_alloc` directly in thread-local
storage was causing a crash. This changes the `local_alloc` to be
indirected, and thus puts less pressure on the thread-local storage.
The test also has deals with how to allocate before a thread-local
storage has been established.
The primary aim for this refactor is to use a representation for
sizeclasses that uniformly covers both large and small. This allows
certain operations such as alloc_size and external_pointer to be
uniformly implemented.
The additional types make clear which kind of sizeclass is in use.
This also tidies up the code for sizeclass based divisible by and
modulus.
It fixes a bug in rust_realloc that didn't correctly determine a realloc
was required for large classes.
Errno is not required to be 0 on return from malloc,
so don't bother trying to make it 0. Leads to false test failures where
libc calls have not reset it after a failure.
- Grab a larger second allocation on the first allocator to dodge the sizeclass
of the prior alloc on that allocator *and* any implicit, bootstrapping slabs
that get opened (e.g., for remote queue message stubs).
- De-FAST_PATH the domestication function. No need to always inline it, here.
- Document things a little better
- `check_result()` `abort()` on `null` and non-`nullptr` result. Otherwise it
just prints and doesn't end the test
- Don't call `realloc(, 0)`; this has never been consistent and the current C2x
draft (see §7.22.3.5 and N2464) finally just declares it to be undefined
behavior. POSIX (2017) tolerates our current behavior of freeing the
passed-in pointer and returning a new object.
This avoids repeated double-tapping domestication of the same pointer in
!QueueHeadsAreTame builds, by keeping the current "front" pointer to the queue
in trusted locations (stack, register) rather than storing it back to possibly
client-accessible memory.
Instantiate two allocators and arrange for a message to get passed between them
by exploiting the existing slow-paths' handling of message queues. Count and
CHECK the number of domestication calls during this message passing. For a
little more excitement, pave over the forward pointer in the freelist::Object::T
that is the message and have the domestication callback patch the original value
back; should we somehow fail to invoke the domestication callback on that
address, this will induce a crash (WHP, on both CHECK_CLIENT and unchecked
builds).
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
David points out that we might not have a static way to get at the pagemap, so
it is potentially useful to pass pointers to state objects down from the
Allocators.
This commit splits the sizeclass meta-data to generate better cache
locality for various lookups for checking for size and start of
sizeclasses.
Also, contains some tidying including removing sizeclasses covering
large range. This is left over from an alternative design for large
classes that is no longer in use.
The code was able to use pthread destructors rather than C++ thread
local destructors. This removes the dependence on a C++ .so on linux.
However, this is not stable on other platforms such as Apple. Where the
C++ thread local state can be cleared before the pthread destructor
runs.
Instead, tell the iostream to write out hex. This avoids the CHERI compiler
warning that we're turning a provenance-free value to a pointer.
Co-authored-by: Matthew Parkinson <mattpark@microsoft.com>
This avoids the CHERI compiler warning that we're turning a provenance-free
value to a pointer.
Co-authored-by: Matthew Parkinson <mattpark@microsoft.com>
- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
The Pal was providing policy for overallocating a block of memory to
achieve alignment make that part of the backend.
The backend should be responsible for layout policy.
This PR exposes a pagemap interface to specify ranges that are being
used. The overall invariant is that any memory in the address space
manager has the pagemap committed. This means that individual operations
do not need to commit entries.
This is important for Windows that does not support lazy commit. It is
also important if we want to PROT_NONE most of the pagemap to reduce the
risk of memory safety issues getting access to the pagemap.
There are minor changes to test to pull memory directly from the Pal.
There are also bug fixes in the pagemap tests.
The pagemap allocates it self directly either from
* the original fixed address range it is supplied, and returns the
remaining space after the pagemap is removed; or
* directly allocated from the PAL without using the address space
manager.
This change in layering is required for the next commit, which imposes
the invariant that the pagemap has been committed for all spaced managed
by the address space manager.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
The CapPtr refactoring was largely compiler guided; unfortunately, it turns out
that the static-sized alloc function is not well exercised. As such, some type
errors and unnecessary unsafety lurked behind missing template instantiation.
Correct those and add calls to the test harness to make sure we always generate
at least one instance of each small/medium/large case. While here, it doesn't
hurt to make sure that we call all three possible dealloc() flavors as well.
This will, if nothing else, force instantiation of the static-sized dealloc
template as well.
Even if we opt not to bound these pointers internally (if they aren't headed out
to the user program or we later derive bounded pointers), they should still be
annotated as something other than CBArena, ensuring that we do not attempt to
use them for general amplification.
* The AddressSpaceManager now requests address space in specified granule
sizes and registers those allocations with an external ArenaMap.
* The DefaultArenaMap is a (somewhat erroneously named) Pagemap sparse array /
tree for these provenance roots. Nothing is stored on non-StrictProvenance
architectures.
* In the Sandbox test, give an example of a different ArenaMap structure, which
confines amplification to sandbox memory.
* Adjust some other tests to compile.
