Summary of changes:
- Add a new PAL that doesn't allocate memory, which can be used with a
memory provider that is pre-initialised with a range of memory.
- Add a `NoAllocation` PAL property so that the methods on a PAL that
doesn't support dynamically reserving address space will never be
called and therefore don't need to be implemented.
- Slightly refactor the memory provider class so that it has a narrower
interface with LargeAlloc and is easier to proxy.
- Allow the address space manager and the memory provider to be
initialised with a range of memory.
This may eventually also remove the need for (or, at least, simplify)
the Open Enclave PAL.
This commit also ends up with a few other cleanups:
- The `malloc_useable_size` CMake test that checks whether the
parameter is const qualified was failing on FreeBSD where this
function is declared in `malloc_np.h` but where including
`malloc.h` raises an error. This should now be more robust.
- The BSD aligned PAL inherited from the BSD PAL, which does not
expose aligned allocation. This meant that it exposed both the
aligned and non-aligned allocation interfaces and so happily
accepted incorrect `constexpr` if blocks that expected one or
the other but accidentally required both to exist. The unaligned
function is now deleted so the same failures that appear in CI should
appear locally for anyone using this PAL.
* Sparc support proposal.
* Tweaks from feedback.
Shift to integer/pointers cast like other archs.
Assembly reworks.
Note apparently reading the register once is sufficient to provoke
a pause in the cpu adding a clobber tough.
Squeeze some bits out of allocator IDs so that we can land the sizeclass in
each Remote object. The intent is that, on StrictProvenance architectures like
CHERI, we will be able to route Remote messages through RemoteCache-s without
needing to amplify back to read the sizeclass metadata field out of the slab
headers.
Presently, our flat pagemap can be configured to take...
32-bit AS 48-bit AS
USE_SMALL_CHUNKS 16 KiB 1 GiB
default 4 KiB 256 MiB
USE_LARGE_CHUNKS 256 B 16 MiB
At 1 GiB, we're already past the 512 MiB threshold imposed when
src/test/func/memory/memory.cc, when configured to TEST_LIMITED, probes the
effect of rlimit.
Instead, restrict flat pagemaps to at most 256 MiB of AS by default (override
by defining SNMALLOC_MAX_FLATPAGEMAP_SIZE), which forces the USE_SMALL_CHUNKS &
48-bit AS configuration to use the tree-based version.
While here, rename USE_FLATPAGEMAP to CHUNKMAP_USE_FLATPAGEMAP.
This had not been observed as an issue prior to
923705e514 because CMakeLists.txt had, until
then, been using EQUAL, not STREQUAL, to test for oe (and to then enable
USE_SMALL_CHUNKS). This test would fail, and so the default SLAB_SIZE was
used. Absent this min operation, the use of a whole page on a 64KiB page
causes a crash when using the largest medium size class, as, ultimately, size
classes are not based on page sizes, and so committing a whole page to the
header leaves too little room for that class.
See also 3d3b048776.
If the Superslab meta-data is larger than an OS page, then the
subsequent pages could be decommited. This removes the skipped
initialisation in that case.
- Close to OpenBSD as there is no malloc*size api nor arbritrary
alignment support.
- Like FreeBSD, MAP_NORESERVE never had been implemented even tough
still present in the header but not mentioned in the man page,
FreeBSD has reserved the value for another later usage seems
DragonFly has just out of sync header.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
The pagemap was double incrementing the index level. This did not
exhibit in any use case of the pagemap as it only used 0 or 1 level of
intermediate index for all current use cases.
This commit fixes the bug, and adds a test that uses the pagemap in a
different configuration that has multiple levels of intermediate index
node.
This will not be used unless the C++ standard version is raised to 20. As
concepts and C++20 more generally are quite new, this does not do so.
Nevertheless, the use of concepts can improve the local development experience
as type mismatches are discovered earlier (at template invocation rather than
only during expansion).
"Pal" is a global symbol for the current architecture's platform abstraction
layer class (see src/pal/pal.h); to be less confusing, don't shadow it with a
template parameter on the AddressSpaceManager class, and instead use "PAL" as
is done elsewhere for template arguments.
These statistics can be maintained with effectively zero cost to
realistic applications. They do not track the precise amount of
memory used, but are an over-approximation.
FlatPagemap computes the size of its internal `top` array as if it needed an
entry per byte, rather than an entry per instance of its template type T. In
practice. T has always been uint8_t so far, so this hasn't mattered, but when
we use a larger type, suddenly FlatPagemap balloons to much larger than needed.
This change makes the original 16MiB option not the common option.
It also changes the names of the defines to
SNMALLOC_USE_LARGE_CHUNKS
SNMALLOC_USE_SMALL_CHUNKS
The second should be set for Open Enclave configuration, and results in
256KiB chunk sizes. The first being set builds the original 16MiB chunk
sizes. If neither is set, then we default to 1MiB chunk sizes.
The POSIX PAL and in some configurations the Windows PAL overallocate
address space. If address space has become exhausted then this can lead
to issues, where the PAL fails, even though there is enough aligned
address space to satisfy the underlying request.
This commit tries increasingly smaller overallocation sizes in an
attempt to succeed in more cases.
* Reduce code duplication.
The Apple and Haiku PALs both had *almost* identical code to the POSIX
PAL, differing only in some small argument variables. This is fragile
and easy to accidentally get out of sync. For example, the changes to
`reserve_at_least` involved copying identical code into multiple PALs
and it's easy to accidentally miss one.
This change introduces two optional fields on POSIX-derived PALs:
- `default_mmap_flags` allows a PAL to provide additional `MAP_*`
flags to all `mmap` calls.
- `anonymous_memory_fd` allows the PAL to override the default file
descriptor used for memory mappings.
If a PAL does not provide these, default values are used.
Fixes#219
This change brings in a new approach to managing address space.
It wraps the Pal with a power of two reservation system, that
guarantees all returned blocks are naturally aligned to their size. It
either lets the Pal perform aligned requests, or over allocates and
splits into power of two blocks.
This changes the implementation of the open enclacve pal to support
aligned allocations. This reduces the amount of memory required for
the initial reservation as the large allocator doesn't have to
overallocate to get alignment.
PALOpenEnclave object is lazily constructed. I couldn't
figure out a straight-forward way to pass the heap bounds to
the constructor of PALOpenEnclave object.
As an alternative, store the bounds in inline static variables of
the PALOpenEnclave class and set them via static setup_initial_range
function.
- two_alloc_types/alloc1.cc
Define oe_allocator_init to forward base, end values to
PALOpenEnclave::setup_inital_range
- two_alloc_types/main.cc
Use oe_allocator_init function to set up heap range.
- fixed_region/fixed_region.cc
Initialize heap range via call to PALOpenEnclave::setup_inital_range.
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
The Pal should include address.h, this was masked as other Pals included
it, but are only included for simulating OE scenarios, rather than
the actual build for OE.
