Modernise and tidy the CMake a bit:
- Use generator expressions for a lot of conditionals so that things
are more reliable with multi-config generators (and less verbose).
- Remove C as a needed language. None of the code was C but we were
using C to test if headers worked. This was fragile because a build
with `CMAKE_CXX_COMPILER` set might have checked things compiled with
the system C compiler and then failed when the specified C++ compiler
used different headers.
- Rename the `BACKTRACE_HEADER` macro to `SNMALLOC_BACKTRACE_HEADER`.
This is exposed into code that consumes snmalloc and so should be
'namespaced' (to the degree that's possible with C macros).
- Clean up the options and use dependent options to hide options
that are not always relevant.
- Use functions instead of macros for better variable scoping.
- Factor out some duplicated bits into functions.
- Update to the latest way of telling CMake to use C++17 or C++20.
- Migrate everything that's setting global properties to setting only
per-target properties.
- Link with -nostdlib++ if it's available. If it isn't, fall back to
enabling the C language and linking with the C compiler.
- Make the per-test log messages verbose outputs. These kept scrolling
important messages off the top of the screen for me.
- Make building as a header-only library a public option.
- Add install targets that install all of the headers and provide a
config option. This works with the header-only configuration for
integration with things like vcpkg.
- Fix a missing `#endif` in the `malloc_useable_size` check. This was
failing co compile on all platforms because of the missing `#endif`.
- Bump the minimum version to 3.14 so that we have access to
target_link_options. This is necessary to use generator expressions
for linker flags.
- Make the linker error if the shim libraries depend on symbols that
are not defined in the explicitly-provided libraries.
- Make the old-Ubuntu CI jobs use C++17 explicitly (previously CMake
was silently ignoring the fact that the compiler didn't support C++20)
- Fix errors found by the more aggressive linking mode.
With these changes, it's now possible to install snmalloc and then, in
another project, do something like this:
```cmake
find_package(snmalloc CONFIG REQUIRED)
target_link_libraries(t1 snmalloc::snmalloc)
target_link_libraries(t2 snmalloc::snmallocshim-static)
```
In this example, `t1` gets all of the compile flags necessary to include
snmalloc headers for its build configuration. `t2` is additionally
linked to the snmalloc static shim library.
The code was able to use pthread destructors rather than C++ thread
local destructors. This removes the dependence on a C++ .so on linux.
However, this is not stable on other platforms such as Apple. Where the
C++ thread local state can be cleared before the pthread destructor
runs.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
* Defensive code for alloc/dealloc during TLS teardown
If an allocation or deallocation occurs during TLS teardown, then it is
possible for a new allocator to be created and then this is leaked. On
the mimalloc-bench mstressN benchmark this was observed leading to a
large memory leak.
This fix, detects if we are in the TLS teardown phase, and if so,
the calls to alloc or dealloc must return the allocator once they have
perform the specific operation.
Uses a separate variable to represent if a thread_local's destructor has
run already. This is used to detect thread teardown to put the
allocator into a special slow path to avoid leaks.
* Added some printing first operation to track progress
* Improve error messages on posix
Flush errors, print assert details, and present stack traces.
* Detect incorrect use of pool.
* Clang format.
* Replace broken LL/SC implementation
LL/SC implementation was broken, this replaces it with
a locking implementation. Changes the API to support LL/SC
for future implementation on ARM.
* Improve TLS teardown.
* Make std::function fully inlined.
* Factor out PALLinux stack trace.
* Add checks for leaking allocators.
* Add release build of Windows Clang
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
If the external thread statics are used, then
we don't need to include some C++ runtime
concepts. This refactoring moves some global initialization under
conditional compilation.
There are two things calling themselves pagemaps:
- the src/mem/pagemap.h objects of that name
- the SuperslabMap object gets called a PageMap inside the Allocator
Rename the latter to chunkmap, with appropriate case and snake,
everywhere, and pull it out to its own file (chunkmap.h).
The default implementation of a chunkmap is a purely static object, but
we nevertheless instantiate it per allocator, so that other
implementations can use stateful instances when interposing on the
mutation methods. Note that the "get" method, however, must remain
static to support the interface required by Allocator objects.
If we are using USE_MALLOC pass through, then ThreadAlloc::get() can
cause an allocation inside snmalloc still, as it builds the thread
local allocator, which allocs a single stub allocator, using
the underlying allocator. This means that ASAN would detect a
leak in a client, even though the client program has none.
This change stops us ever allocating allocators, if we are passing
through calls to an underlying allocator.
HEADER_GLOBAL was using non-standard attributes to achieve what C++17
now permits with a keyword. Use the standard formulation.
Update the README to note that gcc is still not recommended, but because
of its poor codegen for 128-bit atomic compare and exchange, rather than
because it doesn't support the attribute used for HEADER_GLOBAL.
Made the API so that get always returns an initialised Alloc*. Added
new fast path that doesn't perform checking, but can lead to very slow
behaviour if called and reused.
Copying an idea from mimalloc, initialise the TLS variable to a global
allocator that doesn't own any memory and then lazily check when we hit
a slow path (which we always do when using the global allocator, because
it doesn't own any memory) if we are the global allocator and replace
it.
There is a slight complication compared to mimalloc's version of this
idea. Snmalloc collects outgoing messages and it's possible for the
first operation in a thread to be a free of memory allocated by a
different thread. We address this by initialising the queues with a
size value indicating that they are full and then do the lazy check when
about to insert a message that would make a queue full. This will then
trigger lazy creation of an allocator.
Global initialisation doesn't work for the fake allocator, so skip most
of its constructor.
