* Removed unneeded headers
This removes some unneeded headers from the headers.
* Remove use of std::string
This stack allocates and copies a c-string to replace the calls to std::string.
* msvc: set __cplusplus to the actual value in use
* ds_core/bits: add mask_bits; convert one_at_bit-s
* remotecache: enable reserve_space multiple objects
* nits
* Small changes to tracing
- Trace "Handling remote" once per batch, rather than per element
- Remote queue events also log the associated metaslab; we'll use this
to assess the efficacy of https://github.com/microsoft/snmalloc/issues/634
* freelist builder: allow forcibly tracking length
* Try forward declaring freelist::Builder to appease macos-14
* freelist: tweak intra-slab obfuscation keys by meta address
* NFC: freelist: allow `next` to be arbitrary value
* Switch to a central, tweaked key for all free lists
* allocconfig: introduce some properties of slabs
We'll use these to pack values in message queues.
- Maximum distance between two objects in a single slab
- Maximum number of objects in a slab
* NFC: Templatize LocalCache on Config
* NFC: split dealloc_local_object_slow
We'll use the _slower form when we're just stepping a slab through
multiple rounds of state transition (to come), which can't involve
the actual memory object in question.
* NFC: make freelist::Object::T-s by placement new
* NFC: CoreAlloc: split dealloc_local_object
The pattern of `if (!fast()) { slow() }` occurs in a few places, including in
contexts where we already know the entry and so don't need to look it up.
This provide a way to configure snmalloc to provide per object meta-data that is out of band. This can be used to provide different mitigations on top of snmalloc, such as storing memory tags in a compressed form, or provide a miracle pointer like feature.
This also includes a couple of TSAN fixes as it wasn't fully on in CI.
* Move sizeclass debugging code to sizeclass test
The sizeclass was already testing most of this, so just add the missing bits.
Forgo some tests whose failure would have implied earlier failures.
This moves the last dynamic call of size_to_sizeclass_const into tests
(and so, too, to_exp_mant_const). sizeclasstable.h still contains a static
call to compute NUM_SMALL_SIZECLASSES from MAX_SMALL_SIZECLASS_SIZE.
* Remove unused to_exp_mant
Only its _const sibling is used, and little at that, now that almost everything
to do with sizes and size classes is table-driven.
* test/memcpy: trap, if we can, before exiting
This just means I don't need to remember to set a breakpoint on exit
* test/memcpy: don't assume sizeclass 0 is allocable
* test/memory: don't assume sizeclass 0 is allocable
* test/sizeclass: handle nonzero minimum sizeclasses
* sizeclass: distinguish min alloc and step size
Add support for a minimum allocation size that isn't the minimum step of
the sizeclass table.
* Expose MIN_ALLOC_{,STEP}_SIZE through cmake
* test/sizeclass: report MIN_ALLOC_{STEP_,}SIZE
* Template construction of Pool elements
The Pool class is used by verona-rt. The recent changes made this
less nice to consume as an API.
This change makes the construction logic a template parameter to the
Pool. This enables standard allocation to be used from Verona.
* Drop parameter from acquire
Pool::acquire took a list of parameters to initialise the object that it
constructed. But if this was serviced from the pool, the parameter
would be ignored. This is not an ideal API.
This PR removes the ability to pass a parameter.
* Benchmark for testing startup performance.
* Make pool pass spare space to pooled item
The pool will result in power of 2 allocations as it doesn't have a
local state when it is initially set up.
This commit passes this extra space to the constructor of the pooled
type, so that it can be feed into the freshly created allocator.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
The current version requires clang-format-9. This now getting hard to get.
This commit moves it to the clang-format-15, which is the latest in 22.04.
Also, updates clang-tidy to 15 as well.
There was a mis-compilation in a Verona configuration that lead to
two instances of key_global existing. This change moves it inside
a struct that seems to fix the issue.
The rest of the changes are limiting the use of key_global as both
RemoteCache and RemoteAllocator must use the same configuration,
so there is no need to take the key_global as a parameter.
All the checks and mitigations have been placed under feature flags.
These can be controlled by defining
SNMALLOC_CHECK_CLIENT_MITIGATIONS
This can take a term that represents the mitigations that should be enabled.
E.g.
-DSNMALLOC_CHECK_CLIENT_MITIGATIONS=nochecks+random_pagemap
The CMake uses this to build numerous versions of the LD_PRELOAD library and
tests to allow individual features to be benchmarked.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
* Implement tracking full slabs and large allocations
This adds an additional SeqSet that is used to track all the fully
used slabs and large allocations. This gives more chances to
detect memory leaks, and additionally catch some more UAF failures
where the object is not recycled.
* Make slabmeta track a slab interior pointer
Use the head of the free list builder to track an interior pointer to
the slab. This is unused unless the list contains something.
Hence, we can use this to represent an interior pointer to the slab and
report more accurate leaks.
* clangformat
* clangtidy
* clangtidy
* Clang tidy again.
* Fixing provenance.
* Clangformat
* Clang tidy.
* Add assert for sanity
* Make reinterpret_cast more descriptive.
Add an operation to get a tag free pointer from an address_t, and use it
* Clangformat
* CR
* Fix calculation of number of allocations.
* Fix calculation of number of allocations.
* Fix test
To date, we've had exactly one kind of Pagemap and it held exactly one
type of thing, a descendant of class MetaEntryBase.
PagemapRegisterRange tacitly assumed that the Pagemap (adapter) it
interacted would therefore store entries that could have .set_boundary()
called on them. But in general there's no requirement that this be
true; Pagemaps are generic data structures.
To enable reuse of the PagemapRegisterRange machinery more generally,
change the type of Pagemap::register_range() to take a pointer (rather
than an address) and move the MetaEntryBase-specific functionality to
the backend_helpers/pagemap adapter.
Instead, take a template parameter for the no-args init() method, so
that randomization can be disabled on StrictProvenance architectures
(CHERI), where we don't expect it to be useful, even when snmalloc is
being built to be otherwise paranoid.
Catch callsites up.
* Extend pagemap test
Check for possible overlap between heap and pagemap, but writing and
reading the heap.
* Return unalign memory from the pagemap
This commit allows the pagemap to return unaligned range of memory. This
means that bump allocation of multiple pagemaps doesn't
waste as much space.
* Fail more abruptly if the bounds are not exact.
* Move bounding from Pool into Backend.
This commit makes the rounding and the bounding occur in the same
function.
* Enable smallbuddyrange to handle larger requests
The smallbuddy can now pass the larger requests up the range chain if
it cannot satisfy it itself.
* Test larger requests for meta-data.
Make it easier to justify our avoidance of capptr_from_client and
capptr_reveal in external_pointer by performing address_cast earlier.
In particular, with this change, we can see that the pointer (and so its
authority, in CHERI) is not passed to any called function other than
address_cast and pointer_offset, and so authority is merely propagated
and neither exercised nor amplified.
Remove the long-disused capptr_reveal_wild, which was added for earlier
versions of external_pointer.
Expose a static CapPtr<T,B>::unsafe_from() and use that everywhere instead
(though continue to allow implicit and explicit construction of CapPtr from
nullptr).
* Rename to use Config, rather than StateHandle/Globals/Backend
* Make Backend a type on Config that contains the address space management implementation
* Make Ranges part of the Backend configuration, so we can reuse code for different ways of managing memory
* Pull the common chains of range definitions into separate files for reuse.
* Move PagemapEntry to CommonConfig
* Expose Pagemap through backend, so frontend doesn't see Pagemap directly
* Remove global Pal and use DefaultPal, where one is not pass explicitly.
Co-authored-by: David Chisnall <davidchisnall@users.noreply.github.com>
Co-authored-by: Nathaniel Filardo <105816689+nwf-msr@users.noreply.github.com>
This refactoring was provided by David. Previously if a backend
provided a capptr_domesticate function with the wrong type it would be
silently ignored. This change requires backends to explicitly opt in
to domestication via a new Backend::Option and ensures the compiler
will loudly complain if there is a mismatch.
See src/snmalloc/README.md for an explanation of the layers.
Some other cleanups on the way:
Fine-grained stats support is now gone.
It's been broken for two years, it depends on iostream (which then
causes linker failures with libstdc++) and it's collecting the wrong
stats for the new design. After discussion with @mjp41, it's better to
remove it and introduce new stats support later, rather than keep broken
code in the main branch.
Tracing was controlled with a preprocessor macro, now there's also a
CMake option.
MetaCommon is now gone. The back end must provide a SlabMetadata,
which must be a subtype of MetaSlab (i.e. MetaSlab or a subclass of
MetaSlab). It may add additional state here.
The MetaEntry is now templated on the concrete subclass of MetaSlab that
the back-end uses. The MetaEntry still stores this as a `uintptr_t` to
allow easier toggling of the boundary bit but the interfaces are all in
terms of stable types now.
Also some tidying of names (SharedStateHandle is now called Backend).
In a follow-on PR, we can then remove the chunk field from the
BackendMetadata in the non-CHERI back end and allow back ends that don't
require extra state to use MetaSlab directly.
Other cleanups:
- Remove backend/metatypes, define the types that the front end expects
in mem/metaslab. The back end may extend them but these types define
part of the contract between the front and back ends.
- Remove FrontendMetaEntry and fold its methods into MetaEntry.
- For example purposes, the default back end now extends MetaEntry.
This also ensures that nothing in the front end depends on the
specific type of MetaEntry.
- Some things now have more sensible names.
The meta entry now operates in one of three modes:
- When owned by the front end, it stores a pointer to a remote, a
pointer to some MetaSlab subclass, and a sizeclass.
- When owned by the back end, it stores two back-end defined values
that must fit in the bits of `uintptr_t` that are not reserved for
the MetaEntry itself.
- When not owned by either, it can be queried as if owned by the front
end.
The red-black tree has been refactored to allow the holder to be a
wrapper type, removing all of the Holder* and Holder& uses and treating
it uniformly as a value type that can be used to access the contents.
The chunk field is fone from the slab medatada.
This will need to be added back in the CHERI back ends, but it's a
back-end policy. The back end can choose to use it or not, depending on
whether it can safely convert between an Alloc-bounded pointer and a
Chunk-bounded pointer.
The term 'metaslab' originated in snmalloc 1 to mean a slab of slabs.
In the snmalloc2 branch it was repurposed to mean metadata about a
slab. To make this clearer, all uses of metaslab are now gone and have
been renamed to slab metadata. The frontend metadata classes are all
prefixed Frontend and some extra invariants are checked with
`static_assert`.
These encapsulate the wildly powerful reinterpret_cast<> operator where one side
is a uintptr_t and the other is a native pointer. In both cases we require the
pointer type to be explicitly given.
# Small changes before rewrite
* Additional bit in remote allocator to prevent type confusion with the backend.
* Move Chunk allocator to backend.
* Improvements to RedBlack tree
* Expose message from Pal
# Complete backend rewrite
This provides two key changes:
* We use buddy allocators to allow memory to reconsolidated
* The backend is factored into a series of small operations that
allocate and deallocate memory.
The backend now uses "Ranges", there are two ranges that don't require a
parent range:
* EmptyRange - Never returns any memory
* PalRange - Returns memory from the platform.
All other ranges require a parent range to supply memory to them. Some
ranges support both allocation and deallocation, and some just
deallocation. For instance, CommitRange supports both, and maps
requests to the parent range, but will Commit and Decommit the memory.
As the ranges perform only a single task, they are generally small and
easy to follow. The two exceptions to this are the two BuddyRanges
(Large and Small). Large is for CHUNK_SIZE and above blocks, while
Small is for below CHUNK_SIZE blocks. Both are implemented with a buddy
allocator, but the SmallBuddyRange uses in place meta-data, while the
LargeBuddyRange uses the pagemap for its meta-data. This means the
LargeBuddyRange can keep the majority of memory it is managing
decommitted.
The Backend glues together the various ranges to support the appropriate
way to manage memory on the platform.
Expose a memcpy.h that contains all of the bits of memcpy and clean up
the bounds checks header so that versions with both read and write
checks can coexist.
- Refactor the existing SNMALLOC_ASSERT and SNMALLOC_CHECK. These now
use the FatalErrorBuilder to format the output if a format string is
provided.
- Extend the FatalErrorBuilder to print decimal integers for signed
values.
- Rename FatalErrorBuilder to MessageBuilder.
- Rewrite the macros used in the jemalloc tests to use
FatalErrorBuilder and move them into a header.
- Refactor some of the tests to use the new macros.
This introduces a very limited formatter that can embed strings and hex
representations of pointers / integers in an internal buffer. This is
used to format error strings for passing to `Pal::error`. This is used,
in turn, by a wrapper for reporting bounds checks, which can be used by
external functions to implement bounds checks.
This removes the sprintf_l usage from the bounds checks.
This provides enough of a format implementation that the tests
introduced in #465 can be refactored to use this, instead of their
custom `printf` wrapper and that can be used by SNMALLOC_CHECK. This
will be a follow-on PR.
Correctly set errno on failure and improve the related test.
Previously the malloc test would emit an error message but not
abort if the errno was not as expected on failure. This
was because the return in the null == true case prevented the
check for failed == true at the end of check_result from
being reached. To resolve this just abort immediately as in the
null case.
Also add tests of allocations that are expected to fail for
calloc and malloc.
To make the tests pass we need to set errno in several places,
making sure to keep this off the fast path.
We must also take care not to attempt to zero nullptr in case
of calloc failure.
See microsoft/snmalloc#461 and microsoft/snmalloc#463.
This is especially important on CHERI to avoid leaking capabilities to
the freelist. In the CHERI case we also zero in clear_slab (see comment).
Also add a check in the malloc functional test that there are no valid
capabilities in the returned allocation.
