Just always work with pointers using the functions defined in
ds/address.h. This more obviously preserves provenance through the
chain of reasoning. Note that there is still risk of malloc() being
used as an amplification oracle on CHERI, but there's no additional risk
from this change.
Rename the external_address into external_pointer.
This change does two things
* correctly passes the template parameters into the callbacks fixing
correct zeroing of memory.
* By making the callbacks more specific it removes the warnings that GCC
was generating.
* Defensive code for alloc/dealloc during TLS teardown
If an allocation or deallocation occurs during TLS teardown, then it is
possible for a new allocator to be created and then this is leaked. On
the mimalloc-bench mstressN benchmark this was observed leading to a
large memory leak.
This fix, detects if we are in the TLS teardown phase, and if so,
the calls to alloc or dealloc must return the allocator once they have
perform the specific operation.
Uses a separate variable to represent if a thread_local's destructor has
run already. This is used to detect thread teardown to put the
allocator into a special slow path to avoid leaks.
* Added some printing first operation to track progress
* Improve error messages on posix
Flush errors, print assert details, and present stack traces.
* Detect incorrect use of pool.
* Clang format.
* Replace broken LL/SC implementation
LL/SC implementation was broken, this replaces it with
a locking implementation. Changes the API to support LL/SC
for future implementation on ARM.
* Improve TLS teardown.
* Make std::function fully inlined.
* Factor out PALLinux stack trace.
* Add checks for leaking allocators.
* Add release build of Windows Clang
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
Improve remote dealloc
- Outline the slow path to improve code gen significantly
- Handle message queue only on slow path for remote dealloc.
- Change remote size to count down 0, so fast path does not need a constant.
- Use signed value so that branch does not depend on addition.
For languages like Verona or Rust, the deallocation calls know the
size of the object originally requested. This change optimises that
code path to create a much better fast path.
There are two things calling themselves pagemaps:
- the src/mem/pagemap.h objects of that name
- the SuperslabMap object gets called a PageMap inside the Allocator
Rename the latter to chunkmap, with appropriate case and snake,
everywhere, and pull it out to its own file (chunkmap.h).
The default implementation of a chunkmap is a purely static object, but
we nevertheless instantiate it per allocator, so that other
implementations can use stateful instances when interposing on the
mutation methods. Note that the "get" method, however, must remain
static to support the interface required by Allocator objects.
We know that these are small objects and that they have already had
their cache friendly offsets applied, so jump in to the dealloc path
futher down, below the stats calls.
Split small_dealloc_offseted into a wrapper around the core to avoid
inlining too much into the debug function.
While here, move the handling of the Remote objects into its own block
so that `p` is out of scope thereafter.
Fixes https://github.com/microsoft/snmalloc/issues/98
Debug_check_empty now empties the free lists, and checks it empties all the
queues in the allocator. This does not require statistic tracking to
work anymore.
This additionally can check internal regression that cause leaks that
are not the clients fault.
If the first call to alloc uses the minimum size class, then we end up
leaking a whole page of allocations. We fill the small_fast_free_list,
in a call to build the message_queue. But this means the
small_fast_free_list[0] is not empty. But the code was staying on the
slow path, and overwriting it.
The pagemap global is now an inline static of a template class, so that
we will see different symbols for the different types.
Issue #84 showed that it's possible to compile two compilation units
with different pagemaps, link them together, and have them attempt to
interpret the global pagemap as two different types. This change should
make that impossible.
Also make the `pagemap()` function static so that it can be used from
static functions, avoiding other things that directly reference the
global pagemap.
If the operating system will allocate private pages on demand for the
pagemap then use the FlatPageMap by default as it generates better code
for deallocation.
This commit changes the strategy for finding a free list from
a stack to a queue. This tends to avoid the slow path considerably more.
It has some memory overheads.
TOOD: We should move the bump allocation data out of the metaslab and
into the allocator. At the moment, the slab contains the bump allocation
data, we should move this into the allocator, as it only ever has one slab
it is bump allocating from per sizeclass.
This is needed because in some configurations the constructor for the
global placeholder is not called before the first allocation (i.e. when
other globals call the allocator in their constructor) and so we ended
up following a null pointer.
Most compilers are happy if you say always-inline but they can't. GCC
will complain. Here, we have two mutually recursive functions that are
marked as always inline. In an optimised build, one is inlined into the
other and then becomes a tail-recursive function that should inline the
tail call. Inlining the tail call can be done by simply jumping to the
start of the function and so everything is fine. In a debug build, the
second transform doesn't happen and so we're left with a call to an
always-inline function.
