We'll want user_address_control_type in some particular PALs, so it can't live
in pal.h.
While here, make the spelling be capptr::is_spatial_refinement.
- `check_result()` `abort()` on `null` and non-`nullptr` result. Otherwise it
just prints and doesn't end the test
- Don't call `realloc(, 0)`; this has never been consistent and the current C2x
draft (see §7.22.3.5 and N2464) finally just declares it to be undefined
behavior. POSIX (2017) tolerates our current behavior of freeing the
passed-in pointer and returning a new object.
The free list builder in a checked build will only validate entries when
they are removed. This commit adds a validate method, so they can be
checked during teardown. This means that programs that leak memory
will still fail if the free list has become corrupt.
This avoids repeated double-tapping domestication of the same pointer in
!QueueHeadsAreTame builds, by keeping the current "front" pointer to the queue
in trusted locations (stack, register) rather than storing it back to possibly
client-accessible memory.
Instantiate two allocators and arrange for a message to get passed between them
by exploiting the existing slow-paths' handling of message queues. Count and
CHECK the number of domestication calls during this message passing. For a
little more excitement, pave over the forward pointer in the freelist::Object::T
that is the message and have the domestication callback patch the original value
back; should we somehow fail to invoke the domestication callback on that
address, this will induce a crash (WHP, on both CHECK_CLIENT and unchecked
builds).
Motivated by renaming `FreeObject::{Head,Queue,AtomicQueue}Ptr` to
`freelist::...Ptr`, in fact go further, moving `FreeObject` itself to
`freelist::Object` and `FreeListBuilder` to `freelist::Builder` and
`FreeListIter` to `freelist::Iter`
Now that explicit annotations have gotten us through the refactoring, it's time
for the scaffolding to disappear. src/mem/freelist.h is left generic for any
future machinations, but `FreeObject::T<>`, the several `FreeObject::...Ptr<>`s,
`FreeListIter<>`, and `FreeListBuilder<>` are given default parameters and all
uses are shortened to use defaults where possible.
Without this it complains that LocalCache's constructor can't be constexpr
because small_fast_free_lists isn't initialized. It's not clear to me why it
didn't mind before, and nobody else seems to mind now, but this shouldn't break
anyone else, either.
Just an intermediate syntactic step to chase dependencies. All these introduced
"domestication" callbacks are just the identity function, but they will let us
thread the LocalAlloc's handle to the Backend state down to where it's needed.
`capptr_domesticate<Backend>(Backend::LocalState*, CapPtr<T, B>)` is the
intended affordance for conversion to covert from a `CapPtr<T, B>` with
`B::wildness` `Wild` to a `CapPtr<>` with `B::with_wildness<Tame>` and thence
plumbed into the rest of the machinery.
David added the SFINAE wrapper so that `Backend`-s now don't need to implement a
domestication callback; instead, if the `Backend` does not provide a
`capptr_domesticate` function, a default, which just does an explicit type cast,
will be used instead.
This is not yet hooked into the rest of the tree.
Co-authored-by: David Chisnall <David.Chisnall@microsoft.com>
This is incomplete, yet still more reflective of what's going on: we take the
exported pointers back from userspace and thread them directly into the free
lists.
So: move capptr_to_user_address_control to list construction time rather than
list consumption time.
Rather than open-code the conversion from &f->next_object to f, add a static
method to FreeObject and take the opportunity to add a static_assert that our
reinterpret_cast is sound.
If we're going to check next's prev in atomic_read_next, we will need to
domesticate the next pointer first. We could push the check up, but that opens
boxes, so it's simpler to plumb domestication this far down. For symmetry, we
also plumb to (non-atomic) read_next.
FreeObject itself is now just a namespace (but `friend`-ly); the actual free
list nodes are FreeObject::T-s that are templatized on the (perceived)
`capptr::bound<>` of the pointer they contain. (These may differ across an
instantiated snmalloc; for example, in the sandboxing design, the in-sandbox
allocators may perceive all remotes to be full of `AllocUser` while the
privileged allocator of sandbox memory should perceive its remote queue as
holding `AllocUserWild` pointers in need of domestication.)
The interfaces to `FreeObject::T`-s now let us distinguish between the base and
inductive cases of the queues:
* in the inductive case, the pointer we hold to a `FreeObject::T` and its
next_object have the same bounds
* in the base case, the pointer we hold has different bounds (typically,
domesticated by contrast to the wild pointers in the queues).
To keep the clutter down a bit, we occasionally use raw pointers when we can be
reasonably certain that domestication is assured. Moreover, we define some type
aliases, `FreeObject::{HeadPtr, QueuePtr, AtomicQueuePtr}`, that are slightly
more convenient labels than, e.g., `CapPtr<FreeObject::T<BQueue>, BView>`.
Because we are using template parameters for the `capptr::bound<>`s themselves,
we cannot use the aliases for `CapPtr<>s` provided within `capptr::`.
The two primary interfaces around free objects (`FreeListIter` AND
`FreeListBuilder`) are adjusted appropriately and their `BView` and `BQueue`
template paramters are plumbed explicitly around the tree. This makes for quite
a bit of noise at the moment, but means that we'll be able to evolve parts of
the tree separately and can consider putting defaults in once that's done.
* Switch to a multidimensional taxonomy.
Rather than encoding the abstract bound states in a single enum, move to a
more algebraic treatment. The dimensions themselves are within the
snmalloc::capptr_bounds namespace so that their fairly generic names do not
conflict with consumer code. Aliases for many points in the space are
established outside that namespace for ease of use elsewhere.
* Introduce several new namespaces:
* snmalloc::capptr::dimension holds each of the dimension enums
* snmalloc::capptr holds the bound<> type itself and a ConceptBound
* snmalloc::capptr::bounds gives convenient specializations of bound<>
* snmalloc::capptr also has aliases for CapPtr<> itself
All told, rather than `CapPtr<T, CBChunk>`, we now expect client code to read
`capptr::Chunk<T>` in almost all cases (and this is just an alias for the
appropriate `CapPtr<T, bounds<...>>` type). When the bound<>s themselves are
necessary, as when calling capptr_bound, we expect that they will almost
always be pronounced using an alias (e.g., `capptr::bounds::Alloc`).
* Chase consequences.
* Prune old taxa and aliases that are no longer in use in snmalloc2.
# Free List builder track length
This commit makes the free list builder track the length of the lists in
the Random case.
# Refactor free list creation.
Minor refactoring to share code between the new free list and existing
path.
# Randomise slab filling
Knowing when a slab is going to become full makes it easier to by pass
the free list entries as protection for OOB writes. This commit
randomises when a slab will become full.
This commit changes two things
* the free list builder can return some fraction of the deallocations
on a slab.
* when there is a single free slab, we can with some probability
allocate an additional slab.
These two combine to make it difficult to predict when a slab will be
free.
# Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
This changes the slab lists to use a sequential queue.
They were previously stored in a stack.
This commit also tidies up some incomplete refactoring from the
initial snmalloc2 work.
The various Pals were given different meanings in CHECK_CLIENT and
non-CHECK_CLIENT builds. This was because it is essential
that in the CHECK_CLIENT builds access is prevented, when not requested.
This PR separates the CHECK_CLIENT concept from how the Pal should be
implemented.
This exposes a readonly notify using, so that the underlying platform
can map the range of pages readonly into the application. This improves
performance of external pointer on platforms that support lazy commit
of pages as it can access anything in the range.
* Add extra key to freelist. This follows the encoding Cedric suggested
for a signature of two things. Free list key now has a pair of keys
for encoding previous pointer. This makes it harder to extract the
underlying keys out of the multiplication.
* Apply SFINAE to the extract_segment.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
Introduce Metaslab::from_link(SlabLink*) to encapsulate the "container of"
dance. Note that Metaslab was not a standard layout type prior to this change
(since both SlabLink and Metaslab defined non-static data members), and so the
reinterpret_cast<>s replaced here with ::from_link() were UB, but everyone lays
out classes as one expects so it was fine in practice.
Most of the uses of ::from_link() are already guarded by checks that the link
pointer is not nullptr, but in src/mem/corealloc.h:/debug_is_empty_impl we shift
to testing the link pointer explicitly before converting to the metaslab.
Despite that Metaslab is now standard layout, we still don't fall back to the
inter-convertibility of a standard layout class and its first[*] data member
since we're going to want to put a common initial sequence across Metaslab and
ChunkRecord and the SlabLink isn't likely to be in it.
Modernise and tidy the CMake a bit:
- Use generator expressions for a lot of conditionals so that things
are more reliable with multi-config generators (and less verbose).
- Remove C as a needed language. None of the code was C but we were
using C to test if headers worked. This was fragile because a build
with `CMAKE_CXX_COMPILER` set might have checked things compiled with
the system C compiler and then failed when the specified C++ compiler
used different headers.
- Rename the `BACKTRACE_HEADER` macro to `SNMALLOC_BACKTRACE_HEADER`.
This is exposed into code that consumes snmalloc and so should be
'namespaced' (to the degree that's possible with C macros).
- Clean up the options and use dependent options to hide options
that are not always relevant.
- Use functions instead of macros for better variable scoping.
- Factor out some duplicated bits into functions.
- Update to the latest way of telling CMake to use C++17 or C++20.
- Migrate everything that's setting global properties to setting only
per-target properties.
- Link with -nostdlib++ if it's available. If it isn't, fall back to
enabling the C language and linking with the C compiler.
- Make the per-test log messages verbose outputs. These kept scrolling
important messages off the top of the screen for me.
- Make building as a header-only library a public option.
- Add install targets that install all of the headers and provide a
config option. This works with the header-only configuration for
integration with things like vcpkg.
- Fix a missing `#endif` in the `malloc_useable_size` check. This was
failing co compile on all platforms because of the missing `#endif`.
- Bump the minimum version to 3.14 so that we have access to
target_link_options. This is necessary to use generator expressions
for linker flags.
- Make the linker error if the shim libraries depend on symbols that
are not defined in the explicitly-provided libraries.
- Make the old-Ubuntu CI jobs use C++17 explicitly (previously CMake
was silently ignoring the fact that the compiler didn't support C++20)
- Fix errors found by the more aggressive linking mode.
With these changes, it's now possible to install snmalloc and then, in
another project, do something like this:
```cmake
find_package(snmalloc CONFIG REQUIRED)
target_link_libraries(t1 snmalloc::snmalloc)
target_link_libraries(t2 snmalloc::snmallocshim-static)
```
In this example, `t1` gets all of the compile flags necessary to include
snmalloc headers for its build configuration. `t2` is additionally
linked to the snmalloc static shim library.
David points out that we might not have a static way to get at the pagemap, so
it is potentially useful to pass pointers to state objects down from the
Allocators.
