The constructor for atomic_flag is challenging to use in a constexpr.
It requires
std::atomic_flag flag = ATOMIC_FLAG_INIT;
which is not constexpr on some compilers in C++20.
Switching to atomic_bool solves this problem.
# Small changes before rewrite
* Additional bit in remote allocator to prevent type confusion with the backend.
* Move Chunk allocator to backend.
* Improvements to RedBlack tree
* Expose message from Pal
# Complete backend rewrite
This provides two key changes:
* We use buddy allocators to allow memory to reconsolidated
* The backend is factored into a series of small operations that
allocate and deallocate memory.
The backend now uses "Ranges", there are two ranges that don't require a
parent range:
* EmptyRange - Never returns any memory
* PalRange - Returns memory from the platform.
All other ranges require a parent range to supply memory to them. Some
ranges support both allocation and deallocation, and some just
deallocation. For instance, CommitRange supports both, and maps
requests to the parent range, but will Commit and Decommit the memory.
As the ranges perform only a single task, they are generally small and
easy to follow. The two exceptions to this are the two BuddyRanges
(Large and Small). Large is for CHUNK_SIZE and above blocks, while
Small is for below CHUNK_SIZE blocks. Both are implemented with a buddy
allocator, but the SmallBuddyRange uses in place meta-data, while the
LargeBuddyRange uses the pagemap for its meta-data. This means the
LargeBuddyRange can keep the majority of memory it is managing
decommitted.
The Backend glues together the various ranges to support the appropriate
way to manage memory on the platform.
We currently include 256+GiB zeroes in core dumps. This is a QoI issue
because it causes core dumps to take a long time (even if ZFS is able to
compress large runs of zeros down to almost nothing, they still take a
long time to load into debuggers). This patch removes anything marked
as not-using or using-but-read-only (pagemap zero pages) as excluded
from core dumps.
Adding __builtin_trap() to the end of `func-malloc-fast` now gives a 21
MiB core dump rather than a 256 GiB one.
This provides a single place for reporting messages to the user.
While here, be consistent about using stderr for things that should go
to stderr. We were previously using a mix of stderr and stdout.
- Refactor the existing SNMALLOC_ASSERT and SNMALLOC_CHECK. These now
use the FatalErrorBuilder to format the output if a format string is
provided.
- Extend the FatalErrorBuilder to print decimal integers for signed
values.
- Rename FatalErrorBuilder to MessageBuilder.
- Rewrite the macros used in the jemalloc tests to use
FatalErrorBuilder and move them into a header.
- Refactor some of the tests to use the new macros.
This introduces a very limited formatter that can embed strings and hex
representations of pointers / integers in an internal buffer. This is
used to format error strings for passing to `Pal::error`. This is used,
in turn, by a wrapper for reporting bounds checks, which can be used by
external functions to implement bounds checks.
This removes the sprintf_l usage from the bounds checks.
This provides enough of a format implementation that the tests
introduced in #465 can be refactored to use this, instead of their
custom `printf` wrapper and that can be used by SNMALLOC_CHECK. This
will be a follow-on PR.
Correctly set errno on failure and improve the related test.
Previously the malloc test would emit an error message but not
abort if the errno was not as expected on failure. This
was because the return in the null == true case prevented the
check for failed == true at the end of check_result from
being reached. To resolve this just abort immediately as in the
null case.
Also add tests of allocations that are expected to fail for
calloc and malloc.
To make the tests pass we need to set errno in several places,
making sure to keep this off the fast path.
We must also take care not to attempt to zero nullptr in case
of calloc failure.
See microsoft/snmalloc#461 and microsoft/snmalloc#463.
The CHERI-RISC-V `CAndPerm` and `CSetBoundsExact` instructions trap on untagged
inputs, so avoid passing `nullptr` to primitives that become those instructions.
* Add default for getting chunk allocator state
Makes the API same between the two configurations.
* Reduce address space usage for Open Enclave
* Fix OE Pal concept
* Add support for Pal not to provide time.
The lazy return of pages to the OS uses a simple time
based heuristic. This enables a PAL to not support time,
and return the memory to a central pool immediately.
* Update src/backend/backend.h
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
This adds a way to periodically pool the PAL to see if any timers have
expired. Timers can be used to periodically provide callbacks to the
rest of snmalloc.
The correct thing to do, of course, is to fix these upstream, but that requires
understanding exactly what's wrong, and that's harder than just not tickling the
bugs.
This adds a CHERI AAL and expands the FreeBSD PAL to cover CHERI. It updates a
comment in ds/address.h now that there is an example architecture that
differentiates uintptr_t and address_t.
We'll want user_address_control_type in some particular PALs, so it can't live
in pal.h.
While here, make the spelling be capptr::is_spatial_refinement.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
The various Pals were given different meanings in CHECK_CLIENT and
non-CHECK_CLIENT builds. This was because it is essential
that in the CHECK_CLIENT builds access is prevented, when not requested.
This PR separates the CHECK_CLIENT concept from how the Pal should be
implemented.
This exposes a readonly notify using, so that the underlying platform
can map the range of pages readonly into the application. This improves
performance of external pointer on platforms that support lazy commit
of pages as it can access anything in the range.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
Modernise and tidy the CMake a bit:
- Use generator expressions for a lot of conditionals so that things
are more reliable with multi-config generators (and less verbose).
- Remove C as a needed language. None of the code was C but we were
using C to test if headers worked. This was fragile because a build
with `CMAKE_CXX_COMPILER` set might have checked things compiled with
the system C compiler and then failed when the specified C++ compiler
used different headers.
- Rename the `BACKTRACE_HEADER` macro to `SNMALLOC_BACKTRACE_HEADER`.
This is exposed into code that consumes snmalloc and so should be
'namespaced' (to the degree that's possible with C macros).
- Clean up the options and use dependent options to hide options
that are not always relevant.
- Use functions instead of macros for better variable scoping.
- Factor out some duplicated bits into functions.
- Update to the latest way of telling CMake to use C++17 or C++20.
- Migrate everything that's setting global properties to setting only
per-target properties.
- Link with -nostdlib++ if it's available. If it isn't, fall back to
enabling the C language and linking with the C compiler.
- Make the per-test log messages verbose outputs. These kept scrolling
important messages off the top of the screen for me.
- Make building as a header-only library a public option.
- Add install targets that install all of the headers and provide a
config option. This works with the header-only configuration for
integration with things like vcpkg.
- Fix a missing `#endif` in the `malloc_useable_size` check. This was
failing co compile on all platforms because of the missing `#endif`.
- Bump the minimum version to 3.14 so that we have access to
target_link_options. This is necessary to use generator expressions
for linker flags.
- Make the linker error if the shim libraries depend on symbols that
are not defined in the explicitly-provided libraries.
- Make the old-Ubuntu CI jobs use C++17 explicitly (previously CMake
was silently ignoring the fact that the compiler didn't support C++20)
- Fix errors found by the more aggressive linking mode.
With these changes, it's now possible to install snmalloc and then, in
another project, do something like this:
```cmake
find_package(snmalloc CONFIG REQUIRED)
target_link_libraries(t1 snmalloc::snmalloc)
target_link_libraries(t2 snmalloc::snmallocshim-static)
```
In this example, `t1` gets all of the compile flags necessary to include
snmalloc headers for its build configuration. `t2` is additionally
linked to the snmalloc static shim library.
- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
The Pal was providing policy for overallocating a block of memory to
achieve alignment make that part of the backend.
The backend should be responsible for layout policy.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
This is a rewrite of the Apple PAL that implements the AlignedAllocation, Entropy, and LazyCommit features through native Apple APIs.
It adds a dependency on Security.framework via SecRandomCopyBytes. Apple actively discourages use of getentropy and the symbol is not allowed on the App Store.
These capture the primitive architectural operations we are going to use. At
the moment, since all AALs and PALs are not StrictProvenance, the only
implementations are stubs that just subvert the type system (but give us
something to compile against, going forward).
