These are almost entirely backend concerns, so move their definitions over
there. Use C++ friend classes to ensure that MetaCommon structures are opaque
to frontend code (at least, at compile time, and neglecting the rest of C++).
(These structures contain high-authority pointers and so should be as closely
guarded as we can make them.)
The bits that leak out are
- the encoding of RemoteAllocator* and sizeclass_t into the uinptr_t within a
MetaEntry. This, however, is almost entirely a frontend concern, so detach
the method definitions from the class and leave those in mem/metaslab.h for
the moment.
- the size of metadata structures pointed to by the MetaEntry meta field.
Rather than use sizeof(Metaslab) (and assert that sizeof(ChunkRecord) is
smaller), instead, define PAGEMAP_METADATA_STRUCT_SIZE once and assert that
all records fit. Additionally, add an assertion that Metaslab is exactly this
size, not for semantic reasons, but because we expect it to be true.
The bits that leak in are
- the need to zero memory corresponding to a chunk. Rather than having an
escape hatch that reveals the MetaCommon.chunk, move the zeroing call into a
small wrapper method within the MetaCommon class itself.
- the need to get the address of a chunk. We want to assert that we've got the
right chunk on occasion (well, at least once so far) and so add a class method
to expose the address_t view of the chunk pointer without exposing the pointer
itself.
The constructor for atomic_flag is challenging to use in a constexpr.
It requires
std::atomic_flag flag = ATOMIC_FLAG_INIT;
which is not constexpr on some compilers in C++20.
Switching to atomic_bool solves this problem.
These encapsulate the wildly powerful reinterpret_cast<> operator where one side
is a uintptr_t and the other is a native pointer. In both cases we require the
pointer type to be explicitly given.
Since Holder is just an alias for uintptr_t and the fields in the MetaEntry are
uintptr_t-s, just return the lvalue-s directly rather than jumping through
*reinterpret_cast<T*>(& ...).
# Small changes before rewrite
* Additional bit in remote allocator to prevent type confusion with the backend.
* Move Chunk allocator to backend.
* Improvements to RedBlack tree
* Expose message from Pal
# Complete backend rewrite
This provides two key changes:
* We use buddy allocators to allow memory to reconsolidated
* The backend is factored into a series of small operations that
allocate and deallocate memory.
The backend now uses "Ranges", there are two ranges that don't require a
parent range:
* EmptyRange - Never returns any memory
* PalRange - Returns memory from the platform.
All other ranges require a parent range to supply memory to them. Some
ranges support both allocation and deallocation, and some just
deallocation. For instance, CommitRange supports both, and maps
requests to the parent range, but will Commit and Decommit the memory.
As the ranges perform only a single task, they are generally small and
easy to follow. The two exceptions to this are the two BuddyRanges
(Large and Small). Large is for CHUNK_SIZE and above blocks, while
Small is for below CHUNK_SIZE blocks. Both are implemented with a buddy
allocator, but the SmallBuddyRange uses in place meta-data, while the
LargeBuddyRange uses the pagemap for its meta-data. This means the
LargeBuddyRange can keep the majority of memory it is managing
decommitted.
The Backend glues together the various ranges to support the appropriate
way to manage memory on the platform.
Expose a memcpy.h that contains all of the bits of memcpy and clean up
the bounds checks header so that versions with both read and write
checks can coexist.
We currently include 256+GiB zeroes in core dumps. This is a QoI issue
because it causes core dumps to take a long time (even if ZFS is able to
compress large runs of zeros down to almost nothing, they still take a
long time to load into debuggers). This patch removes anything marked
as not-using or using-but-read-only (pagemap zero pages) as excluded
from core dumps.
Adding __builtin_trap() to the end of `func-malloc-fast` now gives a 21
MiB core dump rather than a 256 GiB one.
This provides a single place for reporting messages to the user.
While here, be consistent about using stderr for things that should go
to stderr. We were previously using a mix of stderr and stdout.
This now either outperforms, or performs as well within the region of
measurement noise as, FreeBSD's libc memcpy, which is hand-written
assembly.
This uses a jump table for small copies with a sequence of power-of-two
loads and stores for each, a vector-register copy with an overlapping copy
for the last chunk for medium copies and, on x86, rep movsb for large
copies.
The checked version still incurs a noticeable overhead.
- Refactor the existing SNMALLOC_ASSERT and SNMALLOC_CHECK. These now
use the FatalErrorBuilder to format the output if a format string is
provided.
- Extend the FatalErrorBuilder to print decimal integers for signed
values.
- Rename FatalErrorBuilder to MessageBuilder.
- Rewrite the macros used in the jemalloc tests to use
FatalErrorBuilder and move them into a header.
- Refactor some of the tests to use the new macros.
This introduces a very limited formatter that can embed strings and hex
representations of pointers / integers in an internal buffer. This is
used to format error strings for passing to `Pal::error`. This is used,
in turn, by a wrapper for reporting bounds checks, which can be used by
external functions to implement bounds checks.
This removes the sprintf_l usage from the bounds checks.
This provides enough of a format implementation that the tests
introduced in #465 can be refactored to use this, instead of their
custom `printf` wrapper and that can be used by SNMALLOC_CHECK. This
will be a follow-on PR.
Correctly set errno on failure and improve the related test.
Previously the malloc test would emit an error message but not
abort if the errno was not as expected on failure. This
was because the return in the null == true case prevented the
check for failed == true at the end of check_result from
being reached. To resolve this just abort immediately as in the
null case.
Also add tests of allocations that are expected to fail for
calloc and malloc.
To make the tests pass we need to set errno in several places,
making sure to keep this off the fast path.
We must also take care not to attempt to zero nullptr in case
of calloc failure.
See microsoft/snmalloc#461 and microsoft/snmalloc#463.
This is especially important on CHERI to avoid leaking capabilities to
the freelist. In the CHERI case we also zero in clear_slab (see comment).
Also add a check in the malloc functional test that there are no valid
capabilities in the returned allocation.
An annoying amount of real-world code (e.g. mandoc, BSD sort) treats a
NULL return from `realloc` as a failure, even when requesting a size of
0. This code is wrong (the standard explicitly permits a return of NULL
from realloc when given a size 0) but working around it in snmalloc is
easier than fixing it everywhere.
This adds the full set of jemalloc functions that FreeBSD's libc
exposes, including some (the `*allocm` family) that are gone from newer
versions of jemalloc and the `*allocx` family that replaced them. These
are not necessarily efficient implementations but they should allow
snmalloc to replace jemalloc without any ABI breakage (in the loosest
possible sense).
Jemalloc provides a very generic sysctl-like mechanism for setting and
getting some values. These are all implemented to return the
not-supported error code. This may break code that expects that they
will succeed.
In particular, these APIs are used to register custom backing-store
allocators and to manage caches and arenas. These concepts don't map
directly onto snmalloc and attempting to do so would almost certainly
not provide the same performance characteristics and so it's better to
`LD_PRELOAD` jemalloc (or explicitly link to it) for programs that gain
a significant speedup from this.
- Mark the hook that we're exporting for the threading library to call
to clean up per-thread malloc state as 'used'. It was changed to
`inline` to allow duplicate copies of it to be merged but this also
means that it isn't emitted at all in compilation units that don't
use it (and it isn't used internally at all).
- Fix the `__je_bootstrap_*` functions, which are used to bootstrap TLS
allocation, for the changes to `ScopedAllocator`.
The `__je_bootstrap*` functions weren't being built in CI. They now are
for non-PIE targets with a smoke test.
* Improve testing of memcpy including adding perf test.
* Change remaining_bytes to be branch free.
Use reciprocal division followed by multiply to remove a branch.
* Post large deallocations to original thread
This change sets all large allocations to be owned by the originating
thread. This means they will be messaged back to the original thread
before they can be reused.
The following reason for making this change:
* This will improve producer/consumer apps involving large allocations.
* It enables the implementation of a more complex chunk allocator that
reassembles chunks.
* It addresses an issue with compartmentalisation where the handling of
large allocations can result in meta-data ownership changing.
The CHERI-RISC-V `CAndPerm` and `CSetBoundsExact` instructions trap on untagged
inputs, so avoid passing `nullptr` to primitives that become those instructions.
