The primary aim for this refactor is to use a representation for
sizeclasses that uniformly covers both large and small. This allows
certain operations such as alloc_size and external_pointer to be
uniformly implemented.
The additional types make clear which kind of sizeclass is in use.
This also tidies up the code for sizeclass based divisible by and
modulus.
It fixes a bug in rust_realloc that didn't correctly determine a realloc
was required for large classes.
The memcpy implementation is not completely stupid but is almost
certainly not as good as a carefully tuned and optimised one.
Building snmalloc with FreeBSD's libc memcpy + jemalloc and with this,
each 10 times, does not show a statistically significant performance
difference at 95% confidence. The snmalloc version has very slightly
lower median and worst-case times. This is in no way a sensible
benchmark, but it serves as a smoke test for significant performance
regressions.
The CI self-host job now uses the checked memcpy.
This also fixes an off-by-one error in the external bounds. This is
triggered by ninja, so we will see breakage in CI if it is reintroduced.
In debug builds, we provide a verbose error containing the address of
the allocation, the base and bounds of the allocation, and a backtrace.
The backtrace was broken by the CI cleanup moving the BACKTRACE_HEADER
macro into the SNMALLOC_ namespace. This is also fixed.
The test involves hijacking `abort`, which doesn't work everywhere. It
also requires `backtrace` to work in configurations where stack traces
are enabled. This is disabled in QEMU because `backtrace` appears to
crash reliably in QEMU user mode.
For now, in the -checks build configurations, we are hitting a slow path
in the pagemap on accesses so that the pages that are `PROT_NONE` don't
cause crashes. These need to be made read-only, but this requires a PAL
change.
Modernise and tidy the CMake a bit:
- Use generator expressions for a lot of conditionals so that things
are more reliable with multi-config generators (and less verbose).
- Remove C as a needed language. None of the code was C but we were
using C to test if headers worked. This was fragile because a build
with `CMAKE_CXX_COMPILER` set might have checked things compiled with
the system C compiler and then failed when the specified C++ compiler
used different headers.
- Rename the `BACKTRACE_HEADER` macro to `SNMALLOC_BACKTRACE_HEADER`.
This is exposed into code that consumes snmalloc and so should be
'namespaced' (to the degree that's possible with C macros).
- Clean up the options and use dependent options to hide options
that are not always relevant.
- Use functions instead of macros for better variable scoping.
- Factor out some duplicated bits into functions.
- Update to the latest way of telling CMake to use C++17 or C++20.
- Migrate everything that's setting global properties to setting only
per-target properties.
- Link with -nostdlib++ if it's available. If it isn't, fall back to
enabling the C language and linking with the C compiler.
- Make the per-test log messages verbose outputs. These kept scrolling
important messages off the top of the screen for me.
- Make building as a header-only library a public option.
- Add install targets that install all of the headers and provide a
config option. This works with the header-only configuration for
integration with things like vcpkg.
- Fix a missing `#endif` in the `malloc_useable_size` check. This was
failing co compile on all platforms because of the missing `#endif`.
- Bump the minimum version to 3.14 so that we have access to
target_link_options. This is necessary to use generator expressions
for linker flags.
- Make the linker error if the shim libraries depend on symbols that
are not defined in the explicitly-provided libraries.
- Make the old-Ubuntu CI jobs use C++17 explicitly (previously CMake
was silently ignoring the fact that the compiler didn't support C++20)
- Fix errors found by the more aggressive linking mode.
With these changes, it's now possible to install snmalloc and then, in
another project, do something like this:
```cmake
find_package(snmalloc CONFIG REQUIRED)
target_link_libraries(t1 snmalloc::snmalloc)
target_link_libraries(t2 snmalloc::snmallocshim-static)
```
In this example, `t1` gets all of the compile flags necessary to include
snmalloc headers for its build configuration. `t2` is additionally
linked to the snmalloc static shim library.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
Like alloc_size, this will require amplification internally.
This patch also restores performance to the status quo ante; Clang can once
again see enough to generate the same code as it did before de-static-ing
alloc_size.
We're going to need to amplify the pointer and that's going to require access
to our AddressSpaceManager, which we only get non-statically through our
LargeAlloc.
This patch unto itself makes the world slower, perhaps because Clang can't see
the certainty of aliasing of the static and non-static paths to the same
structure. However, when we also de-static external_pointer, that goes away and
things return to the status quo ante.
- Make GlobalPagemapTemplate and ExternalGlobalPagemap generic in the type of
the pagemap they're encapsulating.
We're going to want to use these for other kinds of pagemaps in the near
future.
- Rename snmalloc_pagemap_global_get to snmalloc_chunkmap_global_get.
- Rename GlobalPagemap to GlobalChunkmap.
The existing snmalloc Rust surface only exposes the calls required by the Rust global allocator.
As Rust knows the size of objects it provides those to the allocator, which snmalloc takes advantage of.
This PR, exposes the standard C API for allocation as well with the prefix `sn_`, so that unsafe code can potentially take advantage of using the same allocator.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
These statistics can be maintained with effectively zero cost to
realistic applications. They do not track the precise amount of
memory used, but are an over-approximation.
* Fix sized delete of nullptr
The core snmalloc code assumes if you know the size, then it is not
nullptr. However, the C++ delete operator can be called with nullptr.
This change checks for that case.
* Improved malloc style tests
Added comprehensive testing of realloc, and other minor improvements
to reporting errors.
* Fix realloc resizing for large sizeclasses.
The rounding by sizeclass was incorrect for large allocation. This fixes
that.
* Ensure alloc_size is committed
There is an awkward interaction between alloc_size and
committing only what is requested. If the user assumes
everything up to alloc_size is available, then we need to
either store the more precise size for alloc_size to return
or commit the whole 2^n range, so that alloc_size stays simple.
This changes to just make the whole range committed.
In the future, we might want to store a more precise size, so
that the allocation can be sized more precisely.
* Reduce size of objects.
If the external thread statics are used, then
we don't need to include some C++ runtime
concepts. This refactoring moves some global initialization under
conditional compilation.
* add rust support
* move aligned_size to sizeclass.h
* add static qualifier
* adjust CMakeLists.txt, may broke CI tests
* fix msvc's complaining on c++17
* use SNMALLOC_FAST_PATH as the decorator of aligned_size
* adapt new alignment algorithm and add related test
Co-authored-by: mjp41 <mattpark@microsoft.com>
* fix test cases for msvc
* add extra test for size == 0
* treat memory block of same sizeclass as the same
* fix formatting problem
* remove extra declarations
Co-authored-by: Matthew Parkinson <mjp41@users.noreply.github.com>
Previous implementation of aligned_alloc met the specification, but was
not particularly useful. This uses the same implementation for
alligned_alloc and memalign.
There are two things calling themselves pagemaps:
- the src/mem/pagemap.h objects of that name
- the SuperslabMap object gets called a PageMap inside the Allocator
Rename the latter to chunkmap, with appropriate case and snake,
everywhere, and pull it out to its own file (chunkmap.h).
The default implementation of a chunkmap is a purely static object, but
we nevertheless instantiate it per allocator, so that other
implementations can use stateful instances when interposing on the
mutation methods. Note that the "get" method, however, must remain
static to support the interface required by Allocator objects.
The pagemap global is now an inline static of a template class, so that
we will see different symbols for the different types.
Issue #84 showed that it's possible to compile two compilation units
with different pagemaps, link them together, and have them attempt to
interpret the global pagemap as two different types. This change should
make that impossible.
Also make the `pagemap()` function static so that it can be used from
static functions, avoiding other things that directly reference the
global pagemap.
This change introduces a per small sizeclass free list. That can be
used to access the free objects for that sizeclass with minimal
calculations being required.
It changes to a partial bump ptr. We bump allocate a whole OS
page worth of objects at a go, so we don't switch as frequently
between bump and free list allocation.
The code for the fast paths has been restructured to minimise the
work required on the common case, and also it is all inlined for the
common case.
Allocating a zero sized object is moved off the fast path. Ask for 1
byte if you want to be fast.
This is useful as codegen is nicer if we use size_t, but the semantics
is uint8_t, and is stored as that in many places in the metadata.
Ultimately should introduce a wrapper to check this invariant.
Introduce a `OnePastEnd` option for the pointer immediately after the
end of the allocation. This simplifies some of the logic in callers,
where they wants to say 'is base + length safe to use?'.
Also restructure some of the other logic somewhat.
