Squeeze some bits out of allocator IDs so that we can land the sizeclass in
each Remote object. The intent is that, on StrictProvenance architectures like
CHERI, we will be able to route Remote messages through RemoteCache-s without
needing to amplify back to read the sizeclass metadata field out of the slab
headers.
Presently, our flat pagemap can be configured to take...
32-bit AS 48-bit AS
USE_SMALL_CHUNKS 16 KiB 1 GiB
default 4 KiB 256 MiB
USE_LARGE_CHUNKS 256 B 16 MiB
At 1 GiB, we're already past the 512 MiB threshold imposed when
src/test/func/memory/memory.cc, when configured to TEST_LIMITED, probes the
effect of rlimit.
Instead, restrict flat pagemaps to at most 256 MiB of AS by default (override
by defining SNMALLOC_MAX_FLATPAGEMAP_SIZE), which forces the USE_SMALL_CHUNKS &
48-bit AS configuration to use the tree-based version.
While here, rename USE_FLATPAGEMAP to CHUNKMAP_USE_FLATPAGEMAP.
This had not been observed as an issue prior to
923705e514 because CMakeLists.txt had, until
then, been using EQUAL, not STREQUAL, to test for oe (and to then enable
USE_SMALL_CHUNKS). This test would fail, and so the default SLAB_SIZE was
used. Absent this min operation, the use of a whole page on a 64KiB page
causes a crash when using the largest medium size class, as, ultimately, size
classes are not based on page sizes, and so committing a whole page to the
header leaves too little room for that class.
See also 3d3b048776.
If the Superslab meta-data is larger than an OS page, then the
subsequent pages could be decommited. This removes the skipped
initialisation in that case.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
The pagemap was double incrementing the index level. This did not
exhibit in any use case of the pagemap as it only used 0 or 1 level of
intermediate index for all current use cases.
This commit fixes the bug, and adds a test that uses the pagemap in a
different configuration that has multiple levels of intermediate index
node.
This will not be used unless the C++ standard version is raised to 20. As
concepts and C++20 more generally are quite new, this does not do so.
Nevertheless, the use of concepts can improve the local development experience
as type mismatches are discovered earlier (at template invocation rather than
only during expansion).
"Pal" is a global symbol for the current architecture's platform abstraction
layer class (see src/pal/pal.h); to be less confusing, don't shadow it with a
template parameter on the AddressSpaceManager class, and instead use "PAL" as
is done elsewhere for template arguments.
These statistics can be maintained with effectively zero cost to
realistic applications. They do not track the precise amount of
memory used, but are an over-approximation.
FlatPagemap computes the size of its internal `top` array as if it needed an
entry per byte, rather than an entry per instance of its template type T. In
practice. T has always been uint8_t so far, so this hasn't mattered, but when
we use a larger type, suddenly FlatPagemap balloons to much larger than needed.
This change makes the original 16MiB option not the common option.
It also changes the names of the defines to
SNMALLOC_USE_LARGE_CHUNKS
SNMALLOC_USE_SMALL_CHUNKS
The second should be set for Open Enclave configuration, and results in
256KiB chunk sizes. The first being set builds the original 16MiB chunk
sizes. If neither is set, then we default to 1MiB chunk sizes.
This change brings in a new approach to managing address space.
It wraps the Pal with a power of two reservation system, that
guarantees all returned blocks are naturally aligned to their size. It
either lets the Pal perform aligned requests, or over allocates and
splits into power of two blocks.
With large pages (e.g. the 64K that Debian defaults to for ppc64), this
is a bit much to ask. It's only not true for the bottom few medium size
classes, tho', as all sizes above 256K are multiples of 64K with the
current two mantissa bits size schedule.
Since we anticipate address_t not carrying provenance on CHERI, but
rather being vaddr_t there, it doesn't make sense to offer conversion
back to a provenance-carrying pointer.
Thankfully, there is not much to be done here: the uses were few and
could be replaced with the vocabulary of other pointer operations in
ds/address.h
Just always work with pointers using the functions defined in
ds/address.h. This more obviously preserves provenance through the
chain of reasoning. Note that there is still risk of malloc() being
used as an amplification oracle on CHERI, but there's no additional risk
from this change.
Rename the external_address into external_pointer.
* Improved malloc style tests
Added comprehensive testing of realloc, and other minor improvements
to reporting errors.
* Fix realloc resizing for large sizeclasses.
The rounding by sizeclass was incorrect for large allocation. This fixes
that.
* Ensure alloc_size is committed
There is an awkward interaction between alloc_size and
committing only what is requested. If the user assumes
everything up to alloc_size is available, then we need to
either store the more precise size for alloc_size to return
or commit the whole 2^n range, so that alloc_size stays simple.
This changes to just make the whole range committed.
In the future, we might want to store a more precise size, so
that the allocation can be sized more precisely.
* Reduce size of objects.
This change does two things
* correctly passes the template parameters into the callbacks fixing
correct zeroing of memory.
* By making the callbacks more specific it removes the warnings that GCC
was generating.
* Defensive code for alloc/dealloc during TLS teardown
If an allocation or deallocation occurs during TLS teardown, then it is
possible for a new allocator to be created and then this is leaked. On
the mimalloc-bench mstressN benchmark this was observed leading to a
large memory leak.
This fix, detects if we are in the TLS teardown phase, and if so,
the calls to alloc or dealloc must return the allocator once they have
perform the specific operation.
Uses a separate variable to represent if a thread_local's destructor has
run already. This is used to detect thread teardown to put the
allocator into a special slow path to avoid leaks.
* Added some printing first operation to track progress
* Improve error messages on posix
Flush errors, print assert details, and present stack traces.
* Detect incorrect use of pool.
* Clang format.
* Replace broken LL/SC implementation
LL/SC implementation was broken, this replaces it with
a locking implementation. Changes the API to support LL/SC
for future implementation on ARM.
* Improve TLS teardown.
* Make std::function fully inlined.
* Factor out PALLinux stack trace.
* Add checks for leaking allocators.
* Add release build of Windows Clang
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
* Increase Remote batch size
The remote batch size has not changed since the fast path optimisations.
The optimisations mean we are checking the queue considerably less
often, so the batch should be larger. This has a dramatic improvement
on performance on a few of the mimalloc microbenchmarks.
It is set to 4096 as this should cover the worse case scenario of only
remote deallocation at 16 bytes for the 2^16 slab size.
* Fixes for Clang-10
Clang-10 outputs a warning for calling alignment intrinsic with an
alignment of 1. At add constexpr to handle this case.
Improve remote dealloc
- Outline the slow path to improve code gen significantly
- Handle message queue only on slow path for remote dealloc.
- Change remote size to count down 0, so fast path does not need a constant.
- Use signed value so that branch does not depend on addition.
The bootstrapping allocator needs to perform a memcpy to bypass the
removed move constructors on std::atomic. This is safe as there is no
concurrency at this point, but GCC is unhappy with this.
This commit moves CI to GCC8 and disables this warning for that line.
On platforms that support low-memory notifications register callbacks
that perform lazy decommit. This allows idle processes to return memory
to the OS. Without incurring the cost of constantly committing and
decommitting memory.
Code review and CI changes
* Fixed test to use a template to make constexpr magic work
* Factored out basic notification mechanism so can be reused on other
platforms.
