This PR provides a templated parameter to the allocation routines. This can be used to add special behaviour in
both the successful allocation behaviour, and in the failing
to allocate cases.
The intent of this is two enable two future features
* set_new_handler - so that the failure case doesn't just set ENOMEM, and return nullptr. But can handle both the Windows and C++ versions of (_)set_new_handler.
* The success handler can be used to add checking, zeroing and in the future storing precise size information in metadata for each allocation.
* Lift checking for init to ThreadAlloc
The check init code was tightly integrated into LocalAllocator. This commit pull that code out into ThreadAlloc, and passes a template parameter into the remaining LocalAllocator to perform the relevant TLS manipulations. This removes some of the awkward layering around register_clean_up.
* Reduce size of test due to failures.
Fully disable lotsofthreads test
Need to investigate if the test is unreliable, or we have actually
regressed perf. A quick mimalloc-bench didn't show any regressions.
* Simplify message queue initialisation
This introduces one additional branch on when processing a batch of messages, but it is likely to only be hit when a lot of messages are processed.
* Patch Domestication test.
* Refactor CoreAlloc/LocalAlloc
This combines the notion of CoreAlloc, LocalAlloc and LocalCache into a single class. Previously, these were separated so that a more complex structure would be stored directly in the TLS. This however, proved to be bad for compatibility if the allocator is part of the libc implementation.
This commit collapses all the stages of the allocator into a single class. This simplifies the sequencing and overall is a nice reduction in complexity.
* Re-enable lots of threads test.
* Reenable concept using alternative lazy checking for concepts.
* Self code review
This changes the shape of check_bounds to take a continuation to call if the bounds check succeeds. This is designed to allow for easily wrapping existing code with a bounds check, e.g.
```
void* memcpy(void* dest, const void* src, size_t n) {
return check_bounds(dest, n, [&] {
return memcpy_impl(dest, src, n);
});
}
```
* Factor out explicit Config type
Instead of using snmalloc::Alloc::Config, expose snmalloc::Config, which is then used to derive the allocator type.
* Move globalalloc to front end.
* Remove unneed template parameter from global snmalloc functions.
* Remove SNMALLOC_PASS_THROUGH
VeronaRT now has an abstraction layer which can easily replace the allocator.
Having such a complex integration still in snmalloc does not make sense.
* Take some global functions off of local alloc.
* Drop comparison overloads on atomic Capptr.
Performing a comparison on two atomic ptr is a complex operation, and should not be implicit. The memory model order and such things needs to be considered by the caller.
* Remove function_ref and use templates
The implementation prefers to use templates over the function_ref. This now only exists in the Pal for a currently unused feature.
* Removing function_ref reduces stl needs.
* Remove use of __is_convertible to support older g++
* Inline function that is only used once.
* Remove unused function
* Restrict ThreadAlloc usage to globalalloc
This commit introduces various inline functions on snmalloc:: that perform allocation/deallocation using the thread local allocator.
They remove all usage from a particular test.
* Move cheri checks to own file.
* Refactor is_owned checks.
* Move alloc_size and check_size to globalalloc.
* Minor simplification of dealloc path
* Fix up is_owned to take a config
* Improve usage of scoped allocator.
* Handle Config_ in globalalloc.
* Add stress test benchmark
Co-authored-by: Alexander Nadeau <wareya@gmail.com>
* Add defensive code against spurious wakeup
This commit checks that wait_on_address has not returned spuriously.
* pal: spurious wake up.
The code in the Pal for wake on address was incorrectly assuming the operation returning success meant it had actually changed. The specification allows for spurious wake ups.
This change makes the Pals recheck for a change.
---------
Co-authored-by: Alexander Nadeau <wareya@gmail.com>
* Removed unneeded headers
This removes some unneeded headers from the headers.
* Remove use of std::string
This stack allocates and copies a c-string to replace the calls to std::string.
When building test/perf/singlethread to use the system allocator, gcc
(Debian 14.2.0-3) correctly sees that we were using the value of a
pointer after it had been passed to the privileged free(), which is UB.
Flip the check and dealloc, so that we query the set of pointers we're
tracking first, using the pointer while the allocation is still live.
In test/perf/startup, gcc (Debian 14.2.0-3) seems to get confused about
the size of the counters vector as the code was written. Rewrite the
code to pass the same value (`std::thread::hardware_concurrency()`, but
in a local) to both `counters.resize()` and the `ParallelTest` ctor.
* msvc: set __cplusplus to the actual value in use
* ds_core/bits: add mask_bits; convert one_at_bit-s
* remotecache: enable reserve_space multiple objects
* nits
* Small changes to tracing
- Trace "Handling remote" once per batch, rather than per element
- Remote queue events also log the associated metaslab; we'll use this
to assess the efficacy of https://github.com/microsoft/snmalloc/issues/634
* freelist builder: allow forcibly tracking length
* Try forward declaring freelist::Builder to appease macos-14
* freelist: tweak intra-slab obfuscation keys by meta address
* NFC: freelist: allow `next` to be arbitrary value
* Switch to a central, tweaked key for all free lists
* allocconfig: introduce some properties of slabs
We'll use these to pack values in message queues.
- Maximum distance between two objects in a single slab
- Maximum number of objects in a slab
* NFC: Templatize LocalCache on Config
* NFC: split dealloc_local_object_slow
We'll use the _slower form when we're just stepping a slab through
multiple rounds of state transition (to come), which can't involve
the actual memory object in question.
* NFC: make freelist::Object::T-s by placement new
* NFC: CoreAlloc: split dealloc_local_object
The pattern of `if (!fast()) { slow() }` occurs in a few places, including in
contexts where we already know the entry and so don't need to look it up.
* NFC: split freelist_queue from remoteallocator
This lets us use freelists as message queues in contexts other than
the remoteallocator. No functional change indended.
* freelist_queue: add and use destroy_and_iterate
* freelist: make backptr obfuscation key "tweakable"
* freelist: tweakable keys in forward direction, too
* test/perf/msgpass: ubench a producer-consumer app
Approximate a message-passing application as a set of producers, a set of
consumers, and a set of proxies that do both. We'll use this for some initial
insight for https://github.com/microsoft/snmalloc/issues/634 but it seems worth
having in general.
This provide a way to configure snmalloc to provide per object meta-data that is out of band. This can be used to provide different mitigations on top of snmalloc, such as storing memory tags in a compressed form, or provide a miracle pointer like feature.
This also includes a couple of TSAN fixes as it wasn't fully on in CI.
* Benchmark for testing startup performance.
* Make pool pass spare space to pooled item
The pool will result in power of 2 allocations as it doesn't have a
local state when it is initially set up.
This commit passes this extra space to the constructor of the pooled
type, so that it can be feed into the freshly created allocator.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
* Rename to use Config, rather than StateHandle/Globals/Backend
* Make Backend a type on Config that contains the address space management implementation
* Make Ranges part of the Backend configuration, so we can reuse code for different ways of managing memory
* Pull the common chains of range definitions into separate files for reuse.
* Move PagemapEntry to CommonConfig
* Expose Pagemap through backend, so frontend doesn't see Pagemap directly
* Remove global Pal and use DefaultPal, where one is not pass explicitly.
Co-authored-by: David Chisnall <davidchisnall@users.noreply.github.com>
Co-authored-by: Nathaniel Filardo <105816689+nwf-msr@users.noreply.github.com>
This commit changes the codegen for error messages for failed memcpys.
This no longer generates a stack frame and correctly tail calls the
error messages generator.
It also turns the error messages on in Release builds. This will lead
to better adoption experience.
If this test fails to allocate memory, that should not cause the test to
fail. The 'abort' was added previously to confirm a infrequent failure
was caused by out-of-memory causing the test to assign to nullptr.
This was confirmed in a CI run, and now the test can be made to ignore
allocation failure.
See src/snmalloc/README.md for an explanation of the layers.
Some other cleanups on the way:
Fine-grained stats support is now gone.
It's been broken for two years, it depends on iostream (which then
causes linker failures with libstdc++) and it's collecting the wrong
stats for the new design. After discussion with @mjp41, it's better to
remove it and introduce new stats support later, rather than keep broken
code in the main branch.
Tracing was controlled with a preprocessor macro, now there's also a
CMake option.
Expose a memcpy.h that contains all of the bits of memcpy and clean up
the bounds checks header so that versions with both read and write
checks can coexist.
* Improve testing of memcpy including adding perf test.
* Change remaining_bytes to be branch free.
Use reciprocal division followed by multiply to remove a branch.
- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
The Pal was providing policy for overallocating a block of memory to
achieve alignment make that part of the backend.
The backend should be responsible for layout policy.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
* Replace time measuring macro
The DO_TIME macro was used originally to get performance numbers. The
macro makes tests hard to debug. This commit replaces it with a proper
C++ class with destructor.
* Bug fix
If the superslab meta data is large, then the calculation for the
sizeclasses that could use the short slab was incorrect. This fixes
that calculation.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
Like alloc_size, this will require amplification internally.
This patch also restores performance to the status quo ante; Clang can once
again see enough to generate the same code as it did before de-static-ing
alloc_size.
We're going to need to amplify the pointer and that's going to require access
to our AddressSpaceManager, which we only get non-statically through our
LargeAlloc.
This patch unto itself makes the world slower, perhaps because Clang can't see
the certainty of aliasing of the static and non-static paths to the same
structure. However, when we also de-static external_pointer, that goes away and
things return to the status quo ante.
* Make LowMemoryNotification object allocated
This makes a separate allocation for the callback object. This makes
it easier for different callbacks to be used.
* Add reserve_with_leftover to address_space
The address_space now supports reserving for non-power of 2 allocations
and the space that is used for rounding up is retained by the
address_space. This means that we can more tightly pack the allocators
internal objects.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
It is important, in test_tasks_f, that the store of the size to the
allocated block be made visible to other processors before the store of
the pointer itself. Otherwise, other cores are justified in reading
junk.
This manifests on PowerPC as tripping the "Deallocating with incorrect
size supplied" assertion in alloc.h:/check_size because the value read
from the allocated block may not be a size but rather an internal queue
pointer, which is implausibly large, as sizes go.
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
On platforms that support low-memory notifications register callbacks
that perform lazy decommit. This allows idle processes to return memory
to the OS. Without incurring the cost of constantly committing and
decommitting memory.
Code review and CI changes
* Fixed test to use a template to make constexpr magic work
* Factored out basic notification mechanism so can be reused on other
platforms.
If the external thread statics are used, then
we don't need to include some C++ runtime
concepts. This refactoring moves some global initialization under
conditional compilation.
Windows is only sending low-memory notifications when the machine
is reaching low-memory. So running a 32bit process on 64bit machine
can easily exhaust address space before machine gets close to
low-memory.
The low-memory notification was getting into an infinite loop. This
fixes the loop termination, and provides a test for platforms which
support low-memory notification.
