This requires that the caller perform the cast on the output rather than the
input, which is a little closer to the truth. Shuffle some casts into the right
position.
Like alloc_size, this will require amplification internally.
This patch also restores performance to the status quo ante; Clang can once
again see enough to generate the same code as it did before de-static-ing
alloc_size.
We're going to need to amplify the pointer and that's going to require access
to our AddressSpaceManager, which we only get non-statically through our
LargeAlloc.
This patch unto itself makes the world slower, perhaps because Clang can't see
the certainty of aliasing of the static and non-static paths to the same
structure. However, when we also de-static external_pointer, that goes away and
things return to the status quo ante.
Going forward, this gives us explicit pointers with which to carry bounds
annotations. Otherwise, assuming AuthPtr overloads operator->, a OOP-style call
like
AuthPtr<Slab, Bounds> slab;
slab->foo()
will create a `Slab* this` within the body of `Slab::foo`, leaving it unable to
see or propagate the Bounds annotation. If it invokes callees that expect
`AuthPtr` arguments, it will therefore have to fabricate new `Bounds` unsafely.
This will let us use Pagemaps further down the dependency stack (specifically,
we're going to want a Pagemap inside the AddressSpaceManager) by letting us
manually tie the knot rather than rely on GlobalVirtual and
default_memory_provider() being defined by the time we want a Pagemap.
Presently, GlobalPagemap and ExternalPagemap discriminate only by type. If it
ever happened that multiple PageMap consumers instantiated the same type using
these wrapper, they'd be conflated in the symbol table. Therefore, add an
optional Purpose parameter that will be expanded into the symbols (but serves no
other purpose).
- Make GlobalPagemapTemplate and ExternalGlobalPagemap generic in the type of
the pagemap they're encapsulating.
We're going to want to use these for other kinds of pagemaps in the near
future.
- Rename snmalloc_pagemap_global_get to snmalloc_chunkmap_global_get.
- Rename GlobalPagemap to GlobalChunkmap.
The link object was previously stored in a disused object. This is
good for reducing meta-data, but if we want to reduce the meta-data
corruption potential, then this is not a good design choice.
This commit moves it into the Metaslab.
The dllist was able to call delete during a destructor if a template
flag was set. This flag is never set in snmalloc, and was included
to enable reuse in another project. This was triggering an error on
older mac builds.
This PR calls a templated function when the DLList is destructed. Hence
other projects can specify the `delete` behaviour if required.
The existing snmalloc Rust surface only exposes the calls required by the Rust global allocator.
As Rust knows the size of objects it provides those to the allocator, which snmalloc takes advantage of.
This PR, exposes the standard C API for allocation as well with the prefix `sn_`, so that unsafe code can potentially take advantage of using the same allocator.
Replace the generic check_size() calls with per-{small,medium,large} code paths,
which follow the progression...
* _unchecked: no validation has been performed; check the chunkmap and move
to...
* _checked_chunkmap: the chunkmap indicates that this is the correct slab-type,
and so we have reason to believe that our supposed Allocslab pointers are of
the correct types. Use those pointers to read the sizeclass and move to...
* _checked_sizeclass: additionally, the Allocslab metadata confirms that our
sizeclass matches the expected value. Check that we are at the start of an
object and move to...
* _start: we now believe we have a pointer to the start of a live object. On
a sufficiently capable architecture, we will eventually atomically check that
our pointer is not (scheduled to be) revoked (and/or not reversioned). If
this test passes, our prior pointer arithmetic and loads are justified
(assuming correctness of the quarantine/revocation/reuse) machinery.
The size-less dealloc(void*) path by necessity reads the chunkmap and sizeclass
data itself and so there's no reason to re-validate; as such, it jumps directly
to the _okcmsc point. Similarly, we assume that remote queues are full of
already validated objects, and so the remote handling paths continue to jump
further along even than the _start methods.
MSVC has strong opinions on implicit conversions as used in CI, while Clang both
locally and in CI has weaker opinions. In an effort to avoid subsequent
roundtrips through CI, make clang more strict. Adding -Wconversion definitely
increases the strength of clang's opinions, apparently to include frowning on
some that even MSVC considers OK, so go make explicit the current implicit
behavior.
* Make LowMemoryNotification object allocated
This makes a separate allocation for the callback object. This makes
it easier for different callbacks to be used.
* Add reserve_with_leftover to address_space
The address_space now supports reserving for non-power of 2 allocations
and the space that is used for rounding up is retained by the
address_space. This means that we can more tightly pack the allocators
internal objects.
Confirm that the client has given dealloc() a start-of-object pointer before
that pointer propagates very far (esp., before it enters the remote queues).
As these tests are not free, structure the code so that future interfaces can
skip over them if there's reason to be sure we're at the start of an object.
(On CHERI, for example, one could imagine using sealed pointers.)
For architectures with SSM/MTE, immediately after this test we can update the
memory granule to guard against double-frees; leave TODOs in place.
Summary of changes:
- Add a new PAL that doesn't allocate memory, which can be used with a
memory provider that is pre-initialised with a range of memory.
- Add a `NoAllocation` PAL property so that the methods on a PAL that
doesn't support dynamically reserving address space will never be
called and therefore don't need to be implemented.
- Slightly refactor the memory provider class so that it has a narrower
interface with LargeAlloc and is easier to proxy.
- Allow the address space manager and the memory provider to be
initialised with a range of memory.
This may eventually also remove the need for (or, at least, simplify)
the Open Enclave PAL.
This commit also ends up with a few other cleanups:
- The `malloc_useable_size` CMake test that checks whether the
parameter is const qualified was failing on FreeBSD where this
function is declared in `malloc_np.h` but where including
`malloc.h` raises an error. This should now be more robust.
- The BSD aligned PAL inherited from the BSD PAL, which does not
expose aligned allocation. This meant that it exposed both the
aligned and non-aligned allocation interfaces and so happily
accepted incorrect `constexpr` if blocks that expected one or
the other but accidentally required both to exist. The unaligned
function is now deleted so the same failures that appear in CI should
appear locally for anyone using this PAL.
* Sparc support proposal.
* Tweaks from feedback.
Shift to integer/pointers cast like other archs.
Assembly reworks.
Note apparently reading the register once is sufficient to provoke
a pause in the cpu adding a clobber tough.
Squeeze some bits out of allocator IDs so that we can land the sizeclass in
each Remote object. The intent is that, on StrictProvenance architectures like
CHERI, we will be able to route Remote messages through RemoteCache-s without
needing to amplify back to read the sizeclass metadata field out of the slab
headers.
Presently, our flat pagemap can be configured to take...
32-bit AS 48-bit AS
USE_SMALL_CHUNKS 16 KiB 1 GiB
default 4 KiB 256 MiB
USE_LARGE_CHUNKS 256 B 16 MiB
At 1 GiB, we're already past the 512 MiB threshold imposed when
src/test/func/memory/memory.cc, when configured to TEST_LIMITED, probes the
effect of rlimit.
Instead, restrict flat pagemaps to at most 256 MiB of AS by default (override
by defining SNMALLOC_MAX_FLATPAGEMAP_SIZE), which forces the USE_SMALL_CHUNKS &
48-bit AS configuration to use the tree-based version.
While here, rename USE_FLATPAGEMAP to CHUNKMAP_USE_FLATPAGEMAP.
This had not been observed as an issue prior to
923705e514 because CMakeLists.txt had, until
then, been using EQUAL, not STREQUAL, to test for oe (and to then enable
USE_SMALL_CHUNKS). This test would fail, and so the default SLAB_SIZE was
used. Absent this min operation, the use of a whole page on a 64KiB page
causes a crash when using the largest medium size class, as, ultimately, size
classes are not based on page sizes, and so committing a whole page to the
header leaves too little room for that class.
See also 3d3b048776.
If the Superslab meta-data is larger than an OS page, then the
subsequent pages could be decommited. This removes the skipped
initialisation in that case.
- Close to OpenBSD as there is no malloc*size api nor arbritrary
alignment support.
- Like FreeBSD, MAP_NORESERVE never had been implemented even tough
still present in the header but not mentioned in the man page,
FreeBSD has reserved the value for another later usage seems
DragonFly has just out of sync header.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
The pagemap was double incrementing the index level. This did not
exhibit in any use case of the pagemap as it only used 0 or 1 level of
intermediate index for all current use cases.
This commit fixes the bug, and adds a test that uses the pagemap in a
different configuration that has multiple levels of intermediate index
node.
