If we're going to check next's prev in atomic_read_next, we will need to
domesticate the next pointer first. We could push the check up, but that opens
boxes, so it's simpler to plumb domestication this far down. For symmetry, we
also plumb to (non-atomic) read_next.
FreeObject itself is now just a namespace (but `friend`-ly); the actual free
list nodes are FreeObject::T-s that are templatized on the (perceived)
`capptr::bound<>` of the pointer they contain. (These may differ across an
instantiated snmalloc; for example, in the sandboxing design, the in-sandbox
allocators may perceive all remotes to be full of `AllocUser` while the
privileged allocator of sandbox memory should perceive its remote queue as
holding `AllocUserWild` pointers in need of domestication.)
The interfaces to `FreeObject::T`-s now let us distinguish between the base and
inductive cases of the queues:
* in the inductive case, the pointer we hold to a `FreeObject::T` and its
next_object have the same bounds
* in the base case, the pointer we hold has different bounds (typically,
domesticated by contrast to the wild pointers in the queues).
To keep the clutter down a bit, we occasionally use raw pointers when we can be
reasonably certain that domestication is assured. Moreover, we define some type
aliases, `FreeObject::{HeadPtr, QueuePtr, AtomicQueuePtr}`, that are slightly
more convenient labels than, e.g., `CapPtr<FreeObject::T<BQueue>, BView>`.
Because we are using template parameters for the `capptr::bound<>`s themselves,
we cannot use the aliases for `CapPtr<>s` provided within `capptr::`.
The two primary interfaces around free objects (`FreeListIter` AND
`FreeListBuilder`) are adjusted appropriately and their `BView` and `BQueue`
template paramters are plumbed explicitly around the tree. This makes for quite
a bit of noise at the moment, but means that we'll be able to evolve parts of
the tree separately and can consider putting defaults in once that's done.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
# Free List builder track length
This commit makes the free list builder track the length of the lists in
the Random case.
# Refactor free list creation.
Minor refactoring to share code between the new free list and existing
path.
# Randomise slab filling
Knowing when a slab is going to become full makes it easier to by pass
the free list entries as protection for OOB writes. This commit
randomises when a slab will become full.
This commit changes two things
* the free list builder can return some fraction of the deallocations
on a slab.
* when there is a single free slab, we can with some probability
allocate an additional slab.
These two combine to make it difficult to predict when a slab will be
free.
# Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This changes the slab lists to use a sequential queue.
They were previously stored in a stack.
This commit also tidies up some incomplete refactoring from the
initial snmalloc2 work.
The various Pals were given different meanings in CHECK_CLIENT and
non-CHECK_CLIENT builds. This was because it is essential
that in the CHECK_CLIENT builds access is prevented, when not requested.
This PR separates the CHECK_CLIENT concept from how the Pal should be
implemented.
This exposes a readonly notify using, so that the underlying platform
can map the range of pages readonly into the application. This improves
performance of external pointer on platforms that support lazy commit
of pages as it can access anything in the range.
* Add extra key to freelist. This follows the encoding Cedric suggested
for a signature of two things. Free list key now has a pair of keys
for encoding previous pointer. This makes it harder to extract the
underlying keys out of the multiplication.
* Apply SFINAE to the extract_segment.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
Introduce Metaslab::from_link(SlabLink*) to encapsulate the "container of"
dance. Note that Metaslab was not a standard layout type prior to this change
(since both SlabLink and Metaslab defined non-static data members), and so the
reinterpret_cast<>s replaced here with ::from_link() were UB, but everyone lays
out classes as one expects so it was fine in practice.
Most of the uses of ::from_link() are already guarded by checks that the link
pointer is not nullptr, but in src/mem/corealloc.h:/debug_is_empty_impl we shift
to testing the link pointer explicitly before converting to the metaslab.
Despite that Metaslab is now standard layout, we still don't fall back to the
inter-convertibility of a standard layout class and its first[*] data member
since we're going to want to put a common initial sequence across Metaslab and
ChunkRecord and the SlabLink isn't likely to be in it.
Modernise and tidy the CMake a bit:
- Use generator expressions for a lot of conditionals so that things
are more reliable with multi-config generators (and less verbose).
- Remove C as a needed language. None of the code was C but we were
using C to test if headers worked. This was fragile because a build
with `CMAKE_CXX_COMPILER` set might have checked things compiled with
the system C compiler and then failed when the specified C++ compiler
used different headers.
- Rename the `BACKTRACE_HEADER` macro to `SNMALLOC_BACKTRACE_HEADER`.
This is exposed into code that consumes snmalloc and so should be
'namespaced' (to the degree that's possible with C macros).
- Clean up the options and use dependent options to hide options
that are not always relevant.
- Use functions instead of macros for better variable scoping.
- Factor out some duplicated bits into functions.
- Update to the latest way of telling CMake to use C++17 or C++20.
- Migrate everything that's setting global properties to setting only
per-target properties.
- Link with -nostdlib++ if it's available. If it isn't, fall back to
enabling the C language and linking with the C compiler.
- Make the per-test log messages verbose outputs. These kept scrolling
important messages off the top of the screen for me.
- Make building as a header-only library a public option.
- Add install targets that install all of the headers and provide a
config option. This works with the header-only configuration for
integration with things like vcpkg.
- Fix a missing `#endif` in the `malloc_useable_size` check. This was
failing co compile on all platforms because of the missing `#endif`.
- Bump the minimum version to 3.14 so that we have access to
target_link_options. This is necessary to use generator expressions
for linker flags.
- Make the linker error if the shim libraries depend on symbols that
are not defined in the explicitly-provided libraries.
- Make the old-Ubuntu CI jobs use C++17 explicitly (previously CMake
was silently ignoring the fact that the compiler didn't support C++20)
- Fix errors found by the more aggressive linking mode.
With these changes, it's now possible to install snmalloc and then, in
another project, do something like this:
```cmake
find_package(snmalloc CONFIG REQUIRED)
target_link_libraries(t1 snmalloc::snmalloc)
target_link_libraries(t2 snmalloc::snmallocshim-static)
```
In this example, `t1` gets all of the compile flags necessary to include
snmalloc headers for its build configuration. `t2` is additionally
linked to the snmalloc static shim library.
David points out that we might not have a static way to get at the pagemap, so
it is potentially useful to pass pointers to state objects down from the
Allocators.
Fortunately, C++ taketh away and C++ giveth, both, so here we are: a way to
detect if we're in the middle of definining a type that uses itself as a
template parameter in a way that flows into a concept check and, if so,
short-circuit out of the need to actually do any checks. Wonders never cease.
And do so by type, rather than by value. While here, introduce a C++20 concept
for this Backend-offered proxy and adjust the template parameters appropriately.
This will be useful for the process sandbox code, which needs to mediate stores
to the pagemap, but can provide a read-only view.
When we are accessing potentially out of range, then we might be
accessing before the pagemap has been initialised. Move the check
into the pagemap for better codegen.
This commit splits the sizeclass meta-data to generate better cache
locality for various lookups for checking for size and start of
sizeclasses.
Also, contains some tidying including removing sizeclasses covering
large range. This is left over from an alternative design for large
classes that is no longer in use.
This passes though to an underlying allocator rather than using
snmalloc. This is required for using ASAN in Verona. Verona takes a
close coupling with snmalloc, but to use with ASAN would require a
more work, so we pass to the system allocator in this case.
The code was able to use pthread destructors rather than C++ thread
local destructors. This removes the dependence on a C++ .so on linux.
However, this is not stable on other platforms such as Apple. Where the
C++ thread local state can be cleared before the pthread destructor
runs.
