* Factor out explicit Config type
Instead of using snmalloc::Alloc::Config, expose snmalloc::Config, which is then used to derive the allocator type.
* Move globalalloc to front end.
* Remove unneed template parameter from global snmalloc functions.
* Remove SNMALLOC_PASS_THROUGH
VeronaRT now has an abstraction layer which can easily replace the allocator.
Having such a complex integration still in snmalloc does not make sense.
* Take some global functions off of local alloc.
* Drop comparison overloads on atomic Capptr.
Performing a comparison on two atomic ptr is a complex operation, and should not be implicit. The memory model order and such things needs to be considered by the caller.
* Remove function_ref and use templates
The implementation prefers to use templates over the function_ref. This now only exists in the Pal for a currently unused feature.
* Removing function_ref reduces stl needs.
* Remove use of __is_convertible to support older g++
* Inline function that is only used once.
* Remove unused function
* Restrict ThreadAlloc usage to globalalloc
This commit introduces various inline functions on snmalloc:: that perform allocation/deallocation using the thread local allocator.
They remove all usage from a particular test.
* Move cheri checks to own file.
* Refactor is_owned checks.
* Move alloc_size and check_size to globalalloc.
* Minor simplification of dealloc path
* Fix up is_owned to take a config
* Improve usage of scoped allocator.
* Handle Config_ in globalalloc.
* Handle platforms that have `_GLIBCXX_ASSERTIONS`, which require the stdlib++ to be included.
* Stop override of memcpy with FORTIFY_SOURCE enabled
* Add to .gitignore
* Add data for graphs in release notes.
* Minor tidy on CMake.
When processing a remote batch, the system will process every single message that was available at the start of processing.
This can lead to a long pause time if there have been a considerable number of frees to this thread.
This commit introduces a new mechanism to only process messages up to a limit of 1MiB. The limit is configurable using CMake.
Choosing too small a limit can cause freeing to never catch up with the incoming messages.
* Pickup page size from unistd.h
This uses the PAGESIZE constant from the unistd.h on POSIX.
This should make the code more resilient to being compiled on platforms with
different page sizes.
* Allow pagesize to come from cmake.
* Update src/snmalloc/pal/pal_posix.h
Co-authored-by: Nathaniel Filardo <105816689+nwf-msr@users.noreply.github.com>
---------
Co-authored-by: Nathaniel Filardo <105816689+nwf-msr@users.noreply.github.com>
* Rename dealloc_local_object_slower to _meta
Unlike its brethren, `dealloc_local_object` and
`dealloc_local_object_slow`, the `dealloc_local_object_slower` method
does not take a pointer to free space. Make this slightly more apparent
by renaming it and adding some commentary to both definition and call
site.
* corealloc: get meta in dealloc_local_object
Make both _fast() and _slow() arms take the meta as an argument; _meta()
already did.
* Introduce RemoteMessage structure
Plumb its use around remoteallocator and remotecache
* NFC: Plumb metadata to remotecache dealloc
* Initial steps in batched remote messages
This prepares the recipient to process a batched message.
* Initial dealloc-side batching machinery
Exercise recipient machinery by having the senders collect adjacent frees to
the same slab into a batch.
* Match free batch keying to slab freelist keying
* freelist: add append_segment
* SlabMetadata: machinery for returning multiple objects
This might involve multiple (I think at most two, at the moment) transitions in
the slab lifecycle state machine. Towards that end, return indicators to the
caller that the slow path must be taken and how many objects of the original
set have not yet been counted as returned.
* corealloc: operate ring-at-a-time on remote queues
* RemoteCache associative cache of rings
* RemoteCache: N-set caching
* Initial CHERI support for free rings
* Matt's fix for slow-path codegen
* Try: remotecache: don't store allocator IDs
We can, as Matt so kindly reminds me, go get them from the pagemap. Since we
need this value only when closing a ring, the read from over there is probably
not very onerous. (We could also get the slab pointer from an object in the
ring, but we need that whenever inserting into the cache, so it's probably more
sensible to store that locally?)
* Make BatchIt optional
Move ring set bits and associativity knobs to allocconfig and expose them via
CMake. If associtivity is zero, use non-batched implementations of the
`RemoteMessage` and `RemoteDeallocCacheBatching` classes.
By default, kick BatchIt on when we have enough room in the minimum allocation
size to do it. Exactly how much space is enough is a function of which
mitigations we have enabled and whether or not we are compiling with C++20.
This commit reverts the change to `MIN_ALLOC_SIZE` made in "Introduce
RemoteMessage structure" now that we have multiple types, and zies, of
remote messages to choose from.
* RemoteDeallocCacheBatching: store metas as address
There's no need for a full pointer here, it'd just make the structure larger on
CHERI.
* NFC: plumb entropy from LocalAlloc to BatchIt
* BatchIt random eviction
In order not to thwart `mitigations(random_preserve)` too much, if it's on in
combination with BatchIt, roll the dice every time we append to a batch to
decide if we should stochastically evict this batch. By increasing the number
of batches, we allow the recipient allocator increased opportunity to randomly
stripe batches across the two `freelist::Builder` segments associated with each
slab.
---------
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
Co-authored-by: Matthew Parkinson <mattpark@microsoft.com>
* msvc: set __cplusplus to the actual value in use
* ds_core/bits: add mask_bits; convert one_at_bit-s
* remotecache: enable reserve_space multiple objects
* nits
* Small changes to tracing
- Trace "Handling remote" once per batch, rather than per element
- Remote queue events also log the associated metaslab; we'll use this
to assess the efficacy of https://github.com/microsoft/snmalloc/issues/634
* freelist builder: allow forcibly tracking length
* Try forward declaring freelist::Builder to appease macos-14
* freelist: tweak intra-slab obfuscation keys by meta address
* NFC: freelist: allow `next` to be arbitrary value
* Switch to a central, tweaked key for all free lists
* allocconfig: introduce some properties of slabs
We'll use these to pack values in message queues.
- Maximum distance between two objects in a single slab
- Maximum number of objects in a slab
* NFC: Templatize LocalCache on Config
* NFC: split dealloc_local_object_slow
We'll use the _slower form when we're just stepping a slab through
multiple rounds of state transition (to come), which can't involve
the actual memory object in question.
* NFC: make freelist::Object::T-s by placement new
* NFC: CoreAlloc: split dealloc_local_object
The pattern of `if (!fast()) { slow() }` occurs in a few places, including in
contexts where we already know the entry and so don't need to look it up.
This provide a way to configure snmalloc to provide per object meta-data that is out of band. This can be used to provide different mitigations on top of snmalloc, such as storing memory tags in a compressed form, or provide a miracle pointer like feature.
This also includes a couple of TSAN fixes as it wasn't fully on in CI.
* Move sizeclass debugging code to sizeclass test
The sizeclass was already testing most of this, so just add the missing bits.
Forgo some tests whose failure would have implied earlier failures.
This moves the last dynamic call of size_to_sizeclass_const into tests
(and so, too, to_exp_mant_const). sizeclasstable.h still contains a static
call to compute NUM_SMALL_SIZECLASSES from MAX_SMALL_SIZECLASS_SIZE.
* Remove unused to_exp_mant
Only its _const sibling is used, and little at that, now that almost everything
to do with sizes and size classes is table-driven.
* test/memcpy: trap, if we can, before exiting
This just means I don't need to remember to set a breakpoint on exit
* test/memcpy: don't assume sizeclass 0 is allocable
* test/memory: don't assume sizeclass 0 is allocable
* test/sizeclass: handle nonzero minimum sizeclasses
* sizeclass: distinguish min alloc and step size
Add support for a minimum allocation size that isn't the minimum step of
the sizeclass table.
* Expose MIN_ALLOC_{,STEP}_SIZE through cmake
* test/sizeclass: report MIN_ALLOC_{STEP_,}SIZE
* Using exclusive mode prefetch
The prefetching is always used to move the cache line to the current
core for writing. This change makes it use exclusive mode prefetch
and enables it as a feature flag for x64.
* Debug platform for BSDs
* CI fixes
* More CI
* Update ARM prefetch
* Update x64 prefetch default
The current version requires clang-format-9. This now getting hard to get.
This commit moves it to the clang-format-15, which is the latest in 22.04.
Also, updates clang-tidy to 15 as well.
All the checks and mitigations have been placed under feature flags.
These can be controlled by defining
SNMALLOC_CHECK_CLIENT_MITIGATIONS
This can take a term that represents the mitigations that should be enabled.
E.g.
-DSNMALLOC_CHECK_CLIENT_MITIGATIONS=nochecks+random_pagemap
The CMake uses this to build numerous versions of the LD_PRELOAD library and
tests to allow individual features to be benchmarked.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This uses VirtualBox instead of xhyve. It might be slower, but should
be more reliable.
Tests run on FreeBSD, NetBSD, and OpenBSD. Only the FreeBSD ones are
passing at the moment, the others will keep running but aren't added as
dependencies for the action used to guard commits.
As with CTest, but without the full machinery thereof. This facilitates
package builders to use the usual build targets (all, install) without
needing to build the test programs if they're just going to get dropped
on the floor.
* Fix pal_linux.h for older linux systems
Where MADV_FREE is not defined - replaced with MADV_DONTNEED
Where GRND_NONBLOCK is not defined in <sys/random.h> but in <linux/random.h>
* Check for linux/random.h in CMake
as __has_include seems to not be reliable
* Use CMake module CheckIncludeFilesCXX
as C language isn't enabled by default everywhere
* Move madvise flag ifdefs into constexpr for cleaner code
Build three levels of checking
- None
- Checks memcpy only
- Checks (full)
Currently you can build checks without enabling the memcpy protection.
This PR fixes that.
See src/snmalloc/README.md for an explanation of the layers.
Some other cleanups on the way:
Fine-grained stats support is now gone.
It's been broken for two years, it depends on iostream (which then
causes linker failures with libstdc++) and it's collecting the wrong
stats for the new design. After discussion with @mjp41, it's better to
remove it and introduce new stats support later, rather than keep broken
code in the main branch.
Tracing was controlled with a preprocessor macro, now there's also a
CMake option.
* export netbsd's reallocarr proposal.
acts subtly differently from reallocarray, returns an error code
and first argument as receiver.
* not export by default
* ci tests
* apply suggestions
* doc addition
* Apply suggestions from code review
Co-authored-by: Matthew Parkinson <mjp41@users.noreply.github.com>
This adds a way to periodically pool the PAL to see if any timers have
expired. Timers can be used to periodically provide callbacks to the
rest of snmalloc.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
Modernise and tidy the CMake a bit:
- Use generator expressions for a lot of conditionals so that things
are more reliable with multi-config generators (and less verbose).
- Remove C as a needed language. None of the code was C but we were
using C to test if headers worked. This was fragile because a build
with `CMAKE_CXX_COMPILER` set might have checked things compiled with
the system C compiler and then failed when the specified C++ compiler
used different headers.
- Rename the `BACKTRACE_HEADER` macro to `SNMALLOC_BACKTRACE_HEADER`.
This is exposed into code that consumes snmalloc and so should be
'namespaced' (to the degree that's possible with C macros).
- Clean up the options and use dependent options to hide options
that are not always relevant.
- Use functions instead of macros for better variable scoping.
- Factor out some duplicated bits into functions.
- Update to the latest way of telling CMake to use C++17 or C++20.
- Migrate everything that's setting global properties to setting only
per-target properties.
- Link with -nostdlib++ if it's available. If it isn't, fall back to
enabling the C language and linking with the C compiler.
- Make the per-test log messages verbose outputs. These kept scrolling
important messages off the top of the screen for me.
- Make building as a header-only library a public option.
- Add install targets that install all of the headers and provide a
config option. This works with the header-only configuration for
integration with things like vcpkg.
- Fix a missing `#endif` in the `malloc_useable_size` check. This was
failing co compile on all platforms because of the missing `#endif`.
- Bump the minimum version to 3.14 so that we have access to
target_link_options. This is necessary to use generator expressions
for linker flags.
- Make the linker error if the shim libraries depend on symbols that
are not defined in the explicitly-provided libraries.
- Make the old-Ubuntu CI jobs use C++17 explicitly (previously CMake
was silently ignoring the fact that the compiler didn't support C++20)
- Fix errors found by the more aggressive linking mode.
With these changes, it's now possible to install snmalloc and then, in
another project, do something like this:
```cmake
find_package(snmalloc CONFIG REQUIRED)
target_link_libraries(t1 snmalloc::snmalloc)
target_link_libraries(t2 snmalloc::snmallocshim-static)
```
In this example, `t1` gets all of the compile flags necessary to include
snmalloc headers for its build configuration. `t2` is additionally
linked to the snmalloc static shim library.
This passes though to an underlying allocator rather than using
snmalloc. This is required for using ASAN in Verona. Verona takes a
close coupling with snmalloc, but to use with ASAN would require a
more work, so we pass to the system allocator in this case.
