The new operator in the snmalloc did not throw exceptions in the case of
allocation failure. Moreover, it was not possible to override the
behaviour of the failure of the new operator using the
std::set_new_handler function.
This PR adds the necessary code to the snmalloc new operator to
throw std::bad_alloc when the allocation fails. It also allows the
std::set_new_handler function to be used to set a custom handler for
allocation failures.
This PR provides a templated parameter to the allocation routines. This can be used to add special behaviour in
both the successful allocation behaviour, and in the failing
to allocate cases.
The intent of this is two enable two future features
* set_new_handler - so that the failure case doesn't just set ENOMEM, and return nullptr. But can handle both the Windows and C++ versions of (_)set_new_handler.
* The success handler can be used to add checking, zeroing and in the future storing precise size information in metadata for each allocation.
Some CI pipelines were occasionally timing out due to the func-memory
test taking too long. This change reduces the number of iterations
from 100 to 50 for each test run, which should help avoid timeouts
while still providing sufficient coverage.
It also adds some debug output to indicate the time taken for each test.
Commit 5680cf6dc6e2 ("jemalloc: don't expose 3.0 compat symbols") in
FreeBSD src removes the default symbols for allocm() and friends, so
this test no longer links. Compatibility is stil provided for programs
linking against FBSD_1.3, but here it seems easier to simply stop
referencing them.
* handle reentrancy during initialization
* use finialization list if possible
* Add test for reentrancy of C++ destructors and allocation
* Add test for reentrancy of setspecific
* Add new mode for directly calling __cxa_thread_atexit_impl
---------
Co-authored-by: Schrodinger ZHU Yifan <yifanzhu@rochester.edu>
The pool sort routine is used by Verona's systematic testing. There was a bug introduced in #612 that could corrupt the list when a sort occured.
This is not used by snmalloc and the test coverage was insufficient to expose the bug.
This PR fixes the bug, and improves test coverage.
* Lift checking for init to ThreadAlloc
The check init code was tightly integrated into LocalAllocator. This commit pull that code out into ThreadAlloc, and passes a template parameter into the remaining LocalAllocator to perform the relevant TLS manipulations. This removes some of the awkward layering around register_clean_up.
* Reduce size of test due to failures.
Fully disable lotsofthreads test
Need to investigate if the test is unreliable, or we have actually
regressed perf. A quick mimalloc-bench didn't show any regressions.
* Simplify message queue initialisation
This introduces one additional branch on when processing a batch of messages, but it is likely to only be hit when a lot of messages are processed.
* Patch Domestication test.
* Refactor CoreAlloc/LocalAlloc
This combines the notion of CoreAlloc, LocalAlloc and LocalCache into a single class. Previously, these were separated so that a more complex structure would be stored directly in the TLS. This however, proved to be bad for compatibility if the allocator is part of the libc implementation.
This commit collapses all the stages of the allocator into a single class. This simplifies the sequencing and overall is a nice reduction in complexity.
* Re-enable lots of threads test.
* Reenable concept using alternative lazy checking for concepts.
* Self code review
This changes the shape of check_bounds to take a continuation to call if the bounds check succeeds. This is designed to allow for easily wrapping existing code with a bounds check, e.g.
```
void* memcpy(void* dest, const void* src, size_t n) {
return check_bounds(dest, n, [&] {
return memcpy_impl(dest, src, n);
});
}
```
If a thread forks, while another thread is holding an snmalloc lock, then the allocator could stop working.
This patch attempts to protect against the cases of this. There is one case that is not covered. If a fork occurs during the very first allocation. This can result in the installation of the fork handler racing with the fork, and all bets are off.
* Factor out small sizeclass check
* Update realloc(p,0) semantics
This commit changes the behaviour of realloc(p,0) to be free(p) if p!=nullptr, and malloc(0) if p== nullptr.
* Fix overflow by alignment
* Bug fix: Ensure bytes_free is the total
The bytes freed was not added to the total, but
overrode it. This meant it never fired. This
commit fixes that.
* Factor out explicit Config type
Instead of using snmalloc::Alloc::Config, expose snmalloc::Config, which is then used to derive the allocator type.
* Move globalalloc to front end.
* Remove unneed template parameter from global snmalloc functions.
* Remove SNMALLOC_PASS_THROUGH
VeronaRT now has an abstraction layer which can easily replace the allocator.
Having such a complex integration still in snmalloc does not make sense.
* Take some global functions off of local alloc.
* Drop comparison overloads on atomic Capptr.
Performing a comparison on two atomic ptr is a complex operation, and should not be implicit. The memory model order and such things needs to be considered by the caller.
* Remove function_ref and use templates
The implementation prefers to use templates over the function_ref. This now only exists in the Pal for a currently unused feature.
* Removing function_ref reduces stl needs.
* Remove use of __is_convertible to support older g++
* Inline function that is only used once.
* Remove unused function
* Restrict ThreadAlloc usage to globalalloc
This commit introduces various inline functions on snmalloc:: that perform allocation/deallocation using the thread local allocator.
They remove all usage from a particular test.
* Move cheri checks to own file.
* Refactor is_owned checks.
* Move alloc_size and check_size to globalalloc.
* Minor simplification of dealloc path
* Fix up is_owned to take a config
* Improve usage of scoped allocator.
* Handle Config_ in globalalloc.
* Add stress test benchmark
Co-authored-by: Alexander Nadeau <wareya@gmail.com>
* Add defensive code against spurious wakeup
This commit checks that wait_on_address has not returned spuriously.
* pal: spurious wake up.
The code in the Pal for wake on address was incorrectly assuming the operation returning success meant it had actually changed. The specification allows for spurious wake ups.
This change makes the Pals recheck for a change.
---------
Co-authored-by: Alexander Nadeau <wareya@gmail.com>
* Removed unneeded headers
This removes some unneeded headers from the headers.
* Remove use of std::string
This stack allocates and copies a c-string to replace the calls to std::string.
When building test/perf/singlethread to use the system allocator, gcc
(Debian 14.2.0-3) correctly sees that we were using the value of a
pointer after it had been passed to the privileged free(), which is UB.
Flip the check and dealloc, so that we query the set of pointers we're
tracking first, using the pointer while the allocation is still live.
In test/perf/startup, gcc (Debian 14.2.0-3) seems to get confused about
the size of the counters vector as the code was written. Rewrite the
code to pass the same value (`std::thread::hardware_concurrency()`, but
in a local) to both `counters.resize()` and the `ParallelTest` ctor.
* msvc: set __cplusplus to the actual value in use
* ds_core/bits: add mask_bits; convert one_at_bit-s
* remotecache: enable reserve_space multiple objects
* nits
* Small changes to tracing
- Trace "Handling remote" once per batch, rather than per element
- Remote queue events also log the associated metaslab; we'll use this
to assess the efficacy of https://github.com/microsoft/snmalloc/issues/634
* freelist builder: allow forcibly tracking length
* Try forward declaring freelist::Builder to appease macos-14
* freelist: tweak intra-slab obfuscation keys by meta address
* NFC: freelist: allow `next` to be arbitrary value
* Switch to a central, tweaked key for all free lists
* allocconfig: introduce some properties of slabs
We'll use these to pack values in message queues.
- Maximum distance between two objects in a single slab
- Maximum number of objects in a slab
* NFC: Templatize LocalCache on Config
* NFC: split dealloc_local_object_slow
We'll use the _slower form when we're just stepping a slab through
multiple rounds of state transition (to come), which can't involve
the actual memory object in question.
* NFC: make freelist::Object::T-s by placement new
* NFC: CoreAlloc: split dealloc_local_object
The pattern of `if (!fast()) { slow() }` occurs in a few places, including in
contexts where we already know the entry and so don't need to look it up.
* NFC: split freelist_queue from remoteallocator
This lets us use freelists as message queues in contexts other than
the remoteallocator. No functional change indended.
* freelist_queue: add and use destroy_and_iterate
* freelist: make backptr obfuscation key "tweakable"
* freelist: tweakable keys in forward direction, too
* test/perf/msgpass: ubench a producer-consumer app
Approximate a message-passing application as a set of producers, a set of
consumers, and a set of proxies that do both. We'll use this for some initial
insight for https://github.com/microsoft/snmalloc/issues/634 but it seems worth
having in general.
This provide a way to configure snmalloc to provide per object meta-data that is out of band. This can be used to provide different mitigations on top of snmalloc, such as storing memory tags in a compressed form, or provide a miracle pointer like feature.
This also includes a couple of TSAN fixes as it wasn't fully on in CI.
* Move sizeclass debugging code to sizeclass test
The sizeclass was already testing most of this, so just add the missing bits.
Forgo some tests whose failure would have implied earlier failures.
This moves the last dynamic call of size_to_sizeclass_const into tests
(and so, too, to_exp_mant_const). sizeclasstable.h still contains a static
call to compute NUM_SMALL_SIZECLASSES from MAX_SMALL_SIZECLASS_SIZE.
* Remove unused to_exp_mant
Only its _const sibling is used, and little at that, now that almost everything
to do with sizes and size classes is table-driven.
* test/memcpy: trap, if we can, before exiting
This just means I don't need to remember to set a breakpoint on exit
* test/memcpy: don't assume sizeclass 0 is allocable
* test/memory: don't assume sizeclass 0 is allocable
* test/sizeclass: handle nonzero minimum sizeclasses
* sizeclass: distinguish min alloc and step size
Add support for a minimum allocation size that isn't the minimum step of
the sizeclass table.
* Expose MIN_ALLOC_{,STEP}_SIZE through cmake
* test/sizeclass: report MIN_ALLOC_{STEP_,}SIZE
* Template construction of Pool elements
The Pool class is used by verona-rt. The recent changes made this
less nice to consume as an API.
This change makes the construction logic a template parameter to the
Pool. This enables standard allocation to be used from Verona.
* Drop parameter from acquire
Pool::acquire took a list of parameters to initialise the object that it
constructed. But if this was serviced from the pool, the parameter
would be ignored. This is not an ideal API.
This PR removes the ability to pass a parameter.
* Benchmark for testing startup performance.
* Make pool pass spare space to pooled item
The pool will result in power of 2 allocations as it doesn't have a
local state when it is initially set up.
This commit passes this extra space to the constructor of the pooled
type, so that it can be feed into the freshly created allocator.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
The current version requires clang-format-9. This now getting hard to get.
This commit moves it to the clang-format-15, which is the latest in 22.04.
Also, updates clang-tidy to 15 as well.
There was a mis-compilation in a Verona configuration that lead to
two instances of key_global existing. This change moves it inside
a struct that seems to fix the issue.
The rest of the changes are limiting the use of key_global as both
RemoteCache and RemoteAllocator must use the same configuration,
so there is no need to take the key_global as a parameter.
All the checks and mitigations have been placed under feature flags.
These can be controlled by defining
SNMALLOC_CHECK_CLIENT_MITIGATIONS
This can take a term that represents the mitigations that should be enabled.
E.g.
-DSNMALLOC_CHECK_CLIENT_MITIGATIONS=nochecks+random_pagemap
The CMake uses this to build numerous versions of the LD_PRELOAD library and
tests to allow individual features to be benchmarked.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
* Implement tracking full slabs and large allocations
This adds an additional SeqSet that is used to track all the fully
used slabs and large allocations. This gives more chances to
detect memory leaks, and additionally catch some more UAF failures
where the object is not recycled.
* Make slabmeta track a slab interior pointer
Use the head of the free list builder to track an interior pointer to
the slab. This is unused unless the list contains something.
Hence, we can use this to represent an interior pointer to the slab and
report more accurate leaks.
* clangformat
* clangtidy
* clangtidy
* Clang tidy again.
* Fixing provenance.
* Clangformat
* Clang tidy.
* Add assert for sanity
* Make reinterpret_cast more descriptive.
Add an operation to get a tag free pointer from an address_t, and use it
* Clangformat
* CR
* Fix calculation of number of allocations.
* Fix calculation of number of allocations.
* Fix test
To date, we've had exactly one kind of Pagemap and it held exactly one
type of thing, a descendant of class MetaEntryBase.
PagemapRegisterRange tacitly assumed that the Pagemap (adapter) it
interacted would therefore store entries that could have .set_boundary()
called on them. But in general there's no requirement that this be
true; Pagemaps are generic data structures.
To enable reuse of the PagemapRegisterRange machinery more generally,
change the type of Pagemap::register_range() to take a pointer (rather
than an address) and move the MetaEntryBase-specific functionality to
the backend_helpers/pagemap adapter.
Instead, take a template parameter for the no-args init() method, so
that randomization can be disabled on StrictProvenance architectures
(CHERI), where we don't expect it to be useful, even when snmalloc is
being built to be otherwise paranoid.
Catch callsites up.
* Extend pagemap test
Check for possible overlap between heap and pagemap, but writing and
reading the heap.
* Return unalign memory from the pagemap
This commit allows the pagemap to return unaligned range of memory. This
means that bump allocation of multiple pagemaps doesn't
waste as much space.
* Fail more abruptly if the bounds are not exact.
* Move bounding from Pool into Backend.
This commit makes the rounding and the bounding occur in the same
function.
* Enable smallbuddyrange to handle larger requests
The smallbuddy can now pass the larger requests up the range chain if
it cannot satisfy it itself.
* Test larger requests for meta-data.
