MetaCommon is now gone. The back end must provide a SlabMetadata,
which must be a subtype of MetaSlab (i.e. MetaSlab or a subclass of
MetaSlab). It may add additional state here.
The MetaEntry is now templated on the concrete subclass of MetaSlab that
the back-end uses. The MetaEntry still stores this as a `uintptr_t` to
allow easier toggling of the boundary bit but the interfaces are all in
terms of stable types now.
Also some tidying of names (SharedStateHandle is now called Backend).
In a follow-on PR, we can then remove the chunk field from the
BackendMetadata in the non-CHERI back end and allow back ends that don't
require extra state to use MetaSlab directly.
Other cleanups:
- Remove backend/metatypes, define the types that the front end expects
in mem/metaslab. The back end may extend them but these types define
part of the contract between the front and back ends.
- Remove FrontendMetaEntry and fold its methods into MetaEntry.
- For example purposes, the default back end now extends MetaEntry.
This also ensures that nothing in the front end depends on the
specific type of MetaEntry.
- Some things now have more sensible names.
The meta entry now operates in one of three modes:
- When owned by the front end, it stores a pointer to a remote, a
pointer to some MetaSlab subclass, and a sizeclass.
- When owned by the back end, it stores two back-end defined values
that must fit in the bits of `uintptr_t` that are not reserved for
the MetaEntry itself.
- When not owned by either, it can be queried as if owned by the front
end.
The red-black tree has been refactored to allow the holder to be a
wrapper type, removing all of the Holder* and Holder& uses and treating
it uniformly as a value type that can be used to access the contents.
The chunk field is fone from the slab medatada.
This will need to be added back in the CHERI back ends, but it's a
back-end policy. The back end can choose to use it or not, depending on
whether it can safely convert between an Alloc-bounded pointer and a
Chunk-bounded pointer.
The term 'metaslab' originated in snmalloc 1 to mean a slab of slabs.
In the snmalloc2 branch it was repurposed to mean metadata about a
slab. To make this clearer, all uses of metaslab are now gone and have
been renamed to slab metadata. The frontend metadata classes are all
prefixed Frontend and some extra invariants are checked with
`static_assert`.
On FreeBSD (possibly elsewhere) the normal versions of these go via an
indirection layer because they are pthread cancellation points. This
indirection layer does not work correctly if pthreads can't allocate
memory and so we can't get debug output until malloc is working, at
least a little bit.
With this version, we can call the __sys_ variants, which skip any libc
/ libthr interposition.
CheriBSD 00d71bd4d11af448871d196f987c2ded474f3039 changes
"CHERI_PERM_CHERIABI_VMMAP" to be spelled "CHERI_PERM_SW_VMEM" and deprecated
the old form. Follow along with fallback so we can use older CheriBSDs.
David points out that the downcasts I had introduced were UB. Instead, go back
to passing MetaEntry-s around and make MetaslabMetaEntry just a namespace of
static methods.
This partially reverts 7940fee00c
Otherwise these won't get updated until the small buddy allocator hands them off
to the large buddy allocator (when they morph into being rbtree nodes) and so
the frontend might get confused in the interim (including risk of UAF on
double-free).
For 32bit external pointer it was performing a divide by size, and for
things not managed by snmalloc this was causing a crash. This checks
for zero, and gives the start of the address range as the start of the
object.
These are almost entirely backend concerns, so move their definitions over
there. Use C++ friend classes to ensure that MetaCommon structures are opaque
to frontend code (at least, at compile time, and neglecting the rest of C++).
(These structures contain high-authority pointers and so should be as closely
guarded as we can make them.)
The bits that leak out are
- the encoding of RemoteAllocator* and sizeclass_t into the uinptr_t within a
MetaEntry. This, however, is almost entirely a frontend concern, so detach
the method definitions from the class and leave those in mem/metaslab.h for
the moment.
- the size of metadata structures pointed to by the MetaEntry meta field.
Rather than use sizeof(Metaslab) (and assert that sizeof(ChunkRecord) is
smaller), instead, define PAGEMAP_METADATA_STRUCT_SIZE once and assert that
all records fit. Additionally, add an assertion that Metaslab is exactly this
size, not for semantic reasons, but because we expect it to be true.
The bits that leak in are
- the need to zero memory corresponding to a chunk. Rather than having an
escape hatch that reveals the MetaCommon.chunk, move the zeroing call into a
small wrapper method within the MetaCommon class itself.
- the need to get the address of a chunk. We want to assert that we've got the
right chunk on occasion (well, at least once so far) and so add a class method
to expose the address_t view of the chunk pointer without exposing the pointer
itself.
The constructor for atomic_flag is challenging to use in a constexpr.
It requires
std::atomic_flag flag = ATOMIC_FLAG_INIT;
which is not constexpr on some compilers in C++20.
Switching to atomic_bool solves this problem.
These encapsulate the wildly powerful reinterpret_cast<> operator where one side
is a uintptr_t and the other is a native pointer. In both cases we require the
pointer type to be explicitly given.
Since Holder is just an alias for uintptr_t and the fields in the MetaEntry are
uintptr_t-s, just return the lvalue-s directly rather than jumping through
*reinterpret_cast<T*>(& ...).
# Small changes before rewrite
* Additional bit in remote allocator to prevent type confusion with the backend.
* Move Chunk allocator to backend.
* Improvements to RedBlack tree
* Expose message from Pal
# Complete backend rewrite
This provides two key changes:
* We use buddy allocators to allow memory to reconsolidated
* The backend is factored into a series of small operations that
allocate and deallocate memory.
The backend now uses "Ranges", there are two ranges that don't require a
parent range:
* EmptyRange - Never returns any memory
* PalRange - Returns memory from the platform.
All other ranges require a parent range to supply memory to them. Some
ranges support both allocation and deallocation, and some just
deallocation. For instance, CommitRange supports both, and maps
requests to the parent range, but will Commit and Decommit the memory.
As the ranges perform only a single task, they are generally small and
easy to follow. The two exceptions to this are the two BuddyRanges
(Large and Small). Large is for CHUNK_SIZE and above blocks, while
Small is for below CHUNK_SIZE blocks. Both are implemented with a buddy
allocator, but the SmallBuddyRange uses in place meta-data, while the
LargeBuddyRange uses the pagemap for its meta-data. This means the
LargeBuddyRange can keep the majority of memory it is managing
decommitted.
The Backend glues together the various ranges to support the appropriate
way to manage memory on the platform.
Expose a memcpy.h that contains all of the bits of memcpy and clean up
the bounds checks header so that versions with both read and write
checks can coexist.
We currently include 256+GiB zeroes in core dumps. This is a QoI issue
because it causes core dumps to take a long time (even if ZFS is able to
compress large runs of zeros down to almost nothing, they still take a
long time to load into debuggers). This patch removes anything marked
as not-using or using-but-read-only (pagemap zero pages) as excluded
from core dumps.
Adding __builtin_trap() to the end of `func-malloc-fast` now gives a 21
MiB core dump rather than a 256 GiB one.
This provides a single place for reporting messages to the user.
While here, be consistent about using stderr for things that should go
to stderr. We were previously using a mix of stderr and stdout.
This now either outperforms, or performs as well within the region of
measurement noise as, FreeBSD's libc memcpy, which is hand-written
assembly.
This uses a jump table for small copies with a sequence of power-of-two
loads and stores for each, a vector-register copy with an overlapping copy
for the last chunk for medium copies and, on x86, rep movsb for large
copies.
The checked version still incurs a noticeable overhead.
- Refactor the existing SNMALLOC_ASSERT and SNMALLOC_CHECK. These now
use the FatalErrorBuilder to format the output if a format string is
provided.
- Extend the FatalErrorBuilder to print decimal integers for signed
values.
- Rename FatalErrorBuilder to MessageBuilder.
- Rewrite the macros used in the jemalloc tests to use
FatalErrorBuilder and move them into a header.
- Refactor some of the tests to use the new macros.
This introduces a very limited formatter that can embed strings and hex
representations of pointers / integers in an internal buffer. This is
used to format error strings for passing to `Pal::error`. This is used,
in turn, by a wrapper for reporting bounds checks, which can be used by
external functions to implement bounds checks.
This removes the sprintf_l usage from the bounds checks.
This provides enough of a format implementation that the tests
introduced in #465 can be refactored to use this, instead of their
custom `printf` wrapper and that can be used by SNMALLOC_CHECK. This
will be a follow-on PR.
Correctly set errno on failure and improve the related test.
Previously the malloc test would emit an error message but not
abort if the errno was not as expected on failure. This
was because the return in the null == true case prevented the
check for failed == true at the end of check_result from
being reached. To resolve this just abort immediately as in the
null case.
Also add tests of allocations that are expected to fail for
calloc and malloc.
To make the tests pass we need to set errno in several places,
making sure to keep this off the fast path.
We must also take care not to attempt to zero nullptr in case
of calloc failure.
See microsoft/snmalloc#461 and microsoft/snmalloc#463.
This is especially important on CHERI to avoid leaking capabilities to
the freelist. In the CHERI case we also zero in clear_slab (see comment).
Also add a check in the malloc functional test that there are no valid
capabilities in the returned allocation.
An annoying amount of real-world code (e.g. mandoc, BSD sort) treats a
NULL return from `realloc` as a failure, even when requesting a size of
0. This code is wrong (the standard explicitly permits a return of NULL
from realloc when given a size 0) but working around it in snmalloc is
easier than fixing it everywhere.
This adds the full set of jemalloc functions that FreeBSD's libc
exposes, including some (the `*allocm` family) that are gone from newer
versions of jemalloc and the `*allocx` family that replaced them. These
are not necessarily efficient implementations but they should allow
snmalloc to replace jemalloc without any ABI breakage (in the loosest
possible sense).
Jemalloc provides a very generic sysctl-like mechanism for setting and
getting some values. These are all implemented to return the
not-supported error code. This may break code that expects that they
will succeed.
In particular, these APIs are used to register custom backing-store
allocators and to manage caches and arenas. These concepts don't map
directly onto snmalloc and attempting to do so would almost certainly
not provide the same performance characteristics and so it's better to
`LD_PRELOAD` jemalloc (or explicitly link to it) for programs that gain
a significant speedup from this.
- Mark the hook that we're exporting for the threading library to call
to clean up per-thread malloc state as 'used'. It was changed to
`inline` to allow duplicate copies of it to be merged but this also
means that it isn't emitted at all in compilation units that don't
use it (and it isn't used internally at all).
- Fix the `__je_bootstrap_*` functions, which are used to bootstrap TLS
allocation, for the changes to `ScopedAllocator`.
The `__je_bootstrap*` functions weren't being built in CI. They now are
for non-PIE targets with a smoke test.
