These are almost entirely backend concerns, so move their definitions over
there. Use C++ friend classes to ensure that MetaCommon structures are opaque
to frontend code (at least, at compile time, and neglecting the rest of C++).
(These structures contain high-authority pointers and so should be as closely
guarded as we can make them.)
The bits that leak out are
- the encoding of RemoteAllocator* and sizeclass_t into the uinptr_t within a
MetaEntry. This, however, is almost entirely a frontend concern, so detach
the method definitions from the class and leave those in mem/metaslab.h for
the moment.
- the size of metadata structures pointed to by the MetaEntry meta field.
Rather than use sizeof(Metaslab) (and assert that sizeof(ChunkRecord) is
smaller), instead, define PAGEMAP_METADATA_STRUCT_SIZE once and assert that
all records fit. Additionally, add an assertion that Metaslab is exactly this
size, not for semantic reasons, but because we expect it to be true.
The bits that leak in are
- the need to zero memory corresponding to a chunk. Rather than having an
escape hatch that reveals the MetaCommon.chunk, move the zeroing call into a
small wrapper method within the MetaCommon class itself.
- the need to get the address of a chunk. We want to assert that we've got the
right chunk on occasion (well, at least once so far) and so add a class method
to expose the address_t view of the chunk pointer without exposing the pointer
itself.
Since Holder is just an alias for uintptr_t and the fields in the MetaEntry are
uintptr_t-s, just return the lvalue-s directly rather than jumping through
*reinterpret_cast<T*>(& ...).
# Small changes before rewrite
* Additional bit in remote allocator to prevent type confusion with the backend.
* Move Chunk allocator to backend.
* Improvements to RedBlack tree
* Expose message from Pal
# Complete backend rewrite
This provides two key changes:
* We use buddy allocators to allow memory to reconsolidated
* The backend is factored into a series of small operations that
allocate and deallocate memory.
The backend now uses "Ranges", there are two ranges that don't require a
parent range:
* EmptyRange - Never returns any memory
* PalRange - Returns memory from the platform.
All other ranges require a parent range to supply memory to them. Some
ranges support both allocation and deallocation, and some just
deallocation. For instance, CommitRange supports both, and maps
requests to the parent range, but will Commit and Decommit the memory.
As the ranges perform only a single task, they are generally small and
easy to follow. The two exceptions to this are the two BuddyRanges
(Large and Small). Large is for CHUNK_SIZE and above blocks, while
Small is for below CHUNK_SIZE blocks. Both are implemented with a buddy
allocator, but the SmallBuddyRange uses in place meta-data, while the
LargeBuddyRange uses the pagemap for its meta-data. This means the
LargeBuddyRange can keep the majority of memory it is managing
decommitted.
The Backend glues together the various ranges to support the appropriate
way to manage memory on the platform.
Correctly set errno on failure and improve the related test.
Previously the malloc test would emit an error message but not
abort if the errno was not as expected on failure. This
was because the return in the null == true case prevented the
check for failed == true at the end of check_result from
being reached. To resolve this just abort immediately as in the
null case.
Also add tests of allocations that are expected to fail for
calloc and malloc.
To make the tests pass we need to set errno in several places,
making sure to keep this off the fast path.
We must also take care not to attempt to zero nullptr in case
of calloc failure.
See microsoft/snmalloc#461 and microsoft/snmalloc#463.
* Add default for getting chunk allocator state
Makes the API same between the two configurations.
* Reduce address space usage for Open Enclave
* Fix OE Pal concept
* Add support for Pal not to provide time.
The lazy return of pages to the OS uses a simple time
based heuristic. This enables a PAL to not support time,
and return the memory to a central pool immediately.
* Update src/backend/backend.h
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
The primary aim for this refactor is to use a representation for
sizeclasses that uniformly covers both large and small. This allows
certain operations such as alloc_size and external_pointer to be
uniformly implemented.
The additional types make clear which kind of sizeclass is in use.
This also tidies up the code for sizeclass based divisible by and
modulus.
It fixes a bug in rust_realloc that didn't correctly determine a realloc
was required for large classes.
This dates back to the much earlier design that required the use of an authority
map. Since the current CHERI design does not, go ahead and ask the platform
whenever underlying allocation requests are sufficiently aligned.
This preserves the chunk pointer through the use of a chunk as a slab. It does
grow the structure by one pointer, but on non-CHERI it is still padded to 64
bytes, even with CHECK_CLIENT guards in place:
0: MetaCommon chunk pointer
8: next pointer
16: builder head[0]
24: builder head[1]
32: builder tail[0]
40: builder tail[1]
48: builder length[0] (uint16_t)
50: builder length[1] (uint16_t)
52: padding (4 bytes)
56: needed (uint16_t)
58: sleeping (bool)
(Sadly, on CHERI, even without CHECK_CLIENT guards and with no padding, there
are now four pointers in the structure -- chunk, next, head, tail -- plus five
extra bytes. We will likely wish to explore encoding the head and tail offsets
relative to the chunk pointer.)
This lets us remove the "subversive amplification" in dealloc() in favor of just
preserving the chunk pointer. Speaking of, be sure to assign that in all the
right places, and ASSERT that we've got it right.
Avoid computing bits::next_pow2_bits(1 << n). Even if the compiler can see
through enough of the algebra, it's surely more direct to just use n.
While here, slightly expand documentation about what's going on with the
"sizeclass" encoded into MetaEntry-s.
`capptr_domesticate<Backend>(Backend::LocalState*, CapPtr<T, B>)` is the
intended affordance for conversion to covert from a `CapPtr<T, B>` with
`B::wildness` `Wild` to a `CapPtr<>` with `B::with_wildness<Tame>` and thence
plumbed into the rest of the machinery.
David added the SFINAE wrapper so that `Backend`-s now don't need to implement a
domestication callback; instead, if the `Backend` does not provide a
`capptr_domesticate` function, a default, which just does an explicit type cast,
will be used instead.
This is not yet hooked into the rest of the tree.
Co-authored-by: David Chisnall <David.Chisnall@microsoft.com>
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
This exposes a readonly notify using, so that the underlying platform
can map the range of pages readonly into the application. This improves
performance of external pointer on platforms that support lazy commit
of pages as it can access anything in the range.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
David points out that we might not have a static way to get at the pagemap, so
it is potentially useful to pass pointers to state objects down from the
Allocators.
And do so by type, rather than by value. While here, introduce a C++20 concept
for this Backend-offered proxy and adjust the template parameters appropriately.
This will be useful for the process sandbox code, which needs to mediate stores
to the pagemap, but can provide a read-only view.
When we are accessing potentially out of range, then we might be
accessing before the pagemap has been initialised. Move the check
into the pagemap for better codegen.
- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
