* Factor out small sizeclass check
* Update realloc(p,0) semantics
This commit changes the behaviour of realloc(p,0) to be free(p) if p!=nullptr, and malloc(0) if p== nullptr.
* Fix overflow by alignment
* Bug fix: Ensure bytes_free is the total
The bytes freed was not added to the total, but
overrode it. This meant it never fired. This
commit fixes that.
* Factor out explicit Config type
Instead of using snmalloc::Alloc::Config, expose snmalloc::Config, which is then used to derive the allocator type.
* Move globalalloc to front end.
* Remove unneed template parameter from global snmalloc functions.
* Remove SNMALLOC_PASS_THROUGH
VeronaRT now has an abstraction layer which can easily replace the allocator.
Having such a complex integration still in snmalloc does not make sense.
* Take some global functions off of local alloc.
* Drop comparison overloads on atomic Capptr.
Performing a comparison on two atomic ptr is a complex operation, and should not be implicit. The memory model order and such things needs to be considered by the caller.
* Remove function_ref and use templates
The implementation prefers to use templates over the function_ref. This now only exists in the Pal for a currently unused feature.
* Removing function_ref reduces stl needs.
* Remove use of __is_convertible to support older g++
* Inline function that is only used once.
* Remove unused function
* Restrict ThreadAlloc usage to globalalloc
This commit introduces various inline functions on snmalloc:: that perform allocation/deallocation using the thread local allocator.
They remove all usage from a particular test.
* Move cheri checks to own file.
* Refactor is_owned checks.
* Move alloc_size and check_size to globalalloc.
* Minor simplification of dealloc path
* Fix up is_owned to take a config
* Improve usage of scoped allocator.
* Handle Config_ in globalalloc.
* Add stress test benchmark
Co-authored-by: Alexander Nadeau <wareya@gmail.com>
* Add defensive code against spurious wakeup
This commit checks that wait_on_address has not returned spuriously.
* pal: spurious wake up.
The code in the Pal for wake on address was incorrectly assuming the operation returning success meant it had actually changed. The specification allows for spurious wake ups.
This change makes the Pals recheck for a change.
---------
Co-authored-by: Alexander Nadeau <wareya@gmail.com>
* Removed unneeded headers
This removes some unneeded headers from the headers.
* Remove use of std::string
This stack allocates and copies a c-string to replace the calls to std::string.
When building test/perf/singlethread to use the system allocator, gcc
(Debian 14.2.0-3) correctly sees that we were using the value of a
pointer after it had been passed to the privileged free(), which is UB.
Flip the check and dealloc, so that we query the set of pointers we're
tracking first, using the pointer while the allocation is still live.
In test/perf/startup, gcc (Debian 14.2.0-3) seems to get confused about
the size of the counters vector as the code was written. Rewrite the
code to pass the same value (`std::thread::hardware_concurrency()`, but
in a local) to both `counters.resize()` and the `ParallelTest` ctor.
* msvc: set __cplusplus to the actual value in use
* ds_core/bits: add mask_bits; convert one_at_bit-s
* remotecache: enable reserve_space multiple objects
* nits
* Small changes to tracing
- Trace "Handling remote" once per batch, rather than per element
- Remote queue events also log the associated metaslab; we'll use this
to assess the efficacy of https://github.com/microsoft/snmalloc/issues/634
* freelist builder: allow forcibly tracking length
* Try forward declaring freelist::Builder to appease macos-14
* freelist: tweak intra-slab obfuscation keys by meta address
* NFC: freelist: allow `next` to be arbitrary value
* Switch to a central, tweaked key for all free lists
* allocconfig: introduce some properties of slabs
We'll use these to pack values in message queues.
- Maximum distance between two objects in a single slab
- Maximum number of objects in a slab
* NFC: Templatize LocalCache on Config
* NFC: split dealloc_local_object_slow
We'll use the _slower form when we're just stepping a slab through
multiple rounds of state transition (to come), which can't involve
the actual memory object in question.
* NFC: make freelist::Object::T-s by placement new
* NFC: CoreAlloc: split dealloc_local_object
The pattern of `if (!fast()) { slow() }` occurs in a few places, including in
contexts where we already know the entry and so don't need to look it up.
* NFC: split freelist_queue from remoteallocator
This lets us use freelists as message queues in contexts other than
the remoteallocator. No functional change indended.
* freelist_queue: add and use destroy_and_iterate
* freelist: make backptr obfuscation key "tweakable"
* freelist: tweakable keys in forward direction, too
* test/perf/msgpass: ubench a producer-consumer app
Approximate a message-passing application as a set of producers, a set of
consumers, and a set of proxies that do both. We'll use this for some initial
insight for https://github.com/microsoft/snmalloc/issues/634 but it seems worth
having in general.
This provide a way to configure snmalloc to provide per object meta-data that is out of band. This can be used to provide different mitigations on top of snmalloc, such as storing memory tags in a compressed form, or provide a miracle pointer like feature.
This also includes a couple of TSAN fixes as it wasn't fully on in CI.
* Move sizeclass debugging code to sizeclass test
The sizeclass was already testing most of this, so just add the missing bits.
Forgo some tests whose failure would have implied earlier failures.
This moves the last dynamic call of size_to_sizeclass_const into tests
(and so, too, to_exp_mant_const). sizeclasstable.h still contains a static
call to compute NUM_SMALL_SIZECLASSES from MAX_SMALL_SIZECLASS_SIZE.
* Remove unused to_exp_mant
Only its _const sibling is used, and little at that, now that almost everything
to do with sizes and size classes is table-driven.
* test/memcpy: trap, if we can, before exiting
This just means I don't need to remember to set a breakpoint on exit
* test/memcpy: don't assume sizeclass 0 is allocable
* test/memory: don't assume sizeclass 0 is allocable
* test/sizeclass: handle nonzero minimum sizeclasses
* sizeclass: distinguish min alloc and step size
Add support for a minimum allocation size that isn't the minimum step of
the sizeclass table.
* Expose MIN_ALLOC_{,STEP}_SIZE through cmake
* test/sizeclass: report MIN_ALLOC_{STEP_,}SIZE
* Template construction of Pool elements
The Pool class is used by verona-rt. The recent changes made this
less nice to consume as an API.
This change makes the construction logic a template parameter to the
Pool. This enables standard allocation to be used from Verona.
* Drop parameter from acquire
Pool::acquire took a list of parameters to initialise the object that it
constructed. But if this was serviced from the pool, the parameter
would be ignored. This is not an ideal API.
This PR removes the ability to pass a parameter.
* Benchmark for testing startup performance.
* Make pool pass spare space to pooled item
The pool will result in power of 2 allocations as it doesn't have a
local state when it is initially set up.
This commit passes this extra space to the constructor of the pooled
type, so that it can be feed into the freshly created allocator.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
The current version requires clang-format-9. This now getting hard to get.
This commit moves it to the clang-format-15, which is the latest in 22.04.
Also, updates clang-tidy to 15 as well.
There was a mis-compilation in a Verona configuration that lead to
two instances of key_global existing. This change moves it inside
a struct that seems to fix the issue.
The rest of the changes are limiting the use of key_global as both
RemoteCache and RemoteAllocator must use the same configuration,
so there is no need to take the key_global as a parameter.
All the checks and mitigations have been placed under feature flags.
These can be controlled by defining
SNMALLOC_CHECK_CLIENT_MITIGATIONS
This can take a term that represents the mitigations that should be enabled.
E.g.
-DSNMALLOC_CHECK_CLIENT_MITIGATIONS=nochecks+random_pagemap
The CMake uses this to build numerous versions of the LD_PRELOAD library and
tests to allow individual features to be benchmarked.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
* Implement tracking full slabs and large allocations
This adds an additional SeqSet that is used to track all the fully
used slabs and large allocations. This gives more chances to
detect memory leaks, and additionally catch some more UAF failures
where the object is not recycled.
* Make slabmeta track a slab interior pointer
Use the head of the free list builder to track an interior pointer to
the slab. This is unused unless the list contains something.
Hence, we can use this to represent an interior pointer to the slab and
report more accurate leaks.
* clangformat
* clangtidy
* clangtidy
* Clang tidy again.
* Fixing provenance.
* Clangformat
* Clang tidy.
* Add assert for sanity
* Make reinterpret_cast more descriptive.
Add an operation to get a tag free pointer from an address_t, and use it
* Clangformat
* CR
* Fix calculation of number of allocations.
* Fix calculation of number of allocations.
* Fix test
To date, we've had exactly one kind of Pagemap and it held exactly one
type of thing, a descendant of class MetaEntryBase.
PagemapRegisterRange tacitly assumed that the Pagemap (adapter) it
interacted would therefore store entries that could have .set_boundary()
called on them. But in general there's no requirement that this be
true; Pagemaps are generic data structures.
To enable reuse of the PagemapRegisterRange machinery more generally,
change the type of Pagemap::register_range() to take a pointer (rather
than an address) and move the MetaEntryBase-specific functionality to
the backend_helpers/pagemap adapter.
Instead, take a template parameter for the no-args init() method, so
that randomization can be disabled on StrictProvenance architectures
(CHERI), where we don't expect it to be useful, even when snmalloc is
being built to be otherwise paranoid.
Catch callsites up.
* Extend pagemap test
Check for possible overlap between heap and pagemap, but writing and
reading the heap.
* Return unalign memory from the pagemap
This commit allows the pagemap to return unaligned range of memory. This
means that bump allocation of multiple pagemaps doesn't
waste as much space.
* Fail more abruptly if the bounds are not exact.
* Move bounding from Pool into Backend.
This commit makes the rounding and the bounding occur in the same
function.
* Enable smallbuddyrange to handle larger requests
The smallbuddy can now pass the larger requests up the range chain if
it cannot satisfy it itself.
* Test larger requests for meta-data.
Make it easier to justify our avoidance of capptr_from_client and
capptr_reveal in external_pointer by performing address_cast earlier.
In particular, with this change, we can see that the pointer (and so its
authority, in CHERI) is not passed to any called function other than
address_cast and pointer_offset, and so authority is merely propagated
and neither exercised nor amplified.
Remove the long-disused capptr_reveal_wild, which was added for earlier
versions of external_pointer.
Expose a static CapPtr<T,B>::unsafe_from() and use that everywhere instead
(though continue to allow implicit and explicit construction of CapPtr from
nullptr).
* Rename to use Config, rather than StateHandle/Globals/Backend
* Make Backend a type on Config that contains the address space management implementation
* Make Ranges part of the Backend configuration, so we can reuse code for different ways of managing memory
* Pull the common chains of range definitions into separate files for reuse.
* Move PagemapEntry to CommonConfig
* Expose Pagemap through backend, so frontend doesn't see Pagemap directly
* Remove global Pal and use DefaultPal, where one is not pass explicitly.
Co-authored-by: David Chisnall <davidchisnall@users.noreply.github.com>
Co-authored-by: Nathaniel Filardo <105816689+nwf-msr@users.noreply.github.com>
This commit changes the codegen for error messages for failed memcpys.
This no longer generates a stack frame and correctly tail calls the
error messages generator.
It also turns the error messages on in Release builds. This will lead
to better adoption experience.
If this test fails to allocate memory, that should not cause the test to
fail. The 'abort' was added previously to confirm a infrequent failure
was caused by out-of-memory causing the test to assign to nullptr.
This was confirmed in a CI run, and now the test can be made to ignore
allocation failure.
This refactoring was provided by David. Previously if a backend
provided a capptr_domesticate function with the wrong type it would be
silently ignored. This change requires backends to explicitly opt in
to domestication via a new Backend::Option and ensures the compiler
will loudly complain if there is a mismatch.
See src/snmalloc/README.md for an explanation of the layers.
Some other cleanups on the way:
Fine-grained stats support is now gone.
It's been broken for two years, it depends on iostream (which then
causes linker failures with libstdc++) and it's collecting the wrong
stats for the new design. After discussion with @mjp41, it's better to
remove it and introduce new stats support later, rather than keep broken
code in the main branch.
Tracing was controlled with a preprocessor macro, now there's also a
CMake option.
