- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
The Pal was providing policy for overallocating a block of memory to
achieve alignment make that part of the backend.
The backend should be responsible for layout policy.
This PR exposes a pagemap interface to specify ranges that are being
used. The overall invariant is that any memory in the address space
manager has the pagemap committed. This means that individual operations
do not need to commit entries.
This is important for Windows that does not support lazy commit. It is
also important if we want to PROT_NONE most of the pagemap to reduce the
risk of memory safety issues getting access to the pagemap.
There are minor changes to test to pull memory directly from the Pal.
There are also bug fixes in the pagemap tests.
The pagemap allocates it self directly either from
* the original fixed address range it is supplied, and returns the
remaining space after the pagemap is removed; or
* directly allocated from the PAL without using the address space
manager.
This change in layering is required for the next commit, which imposes
the invariant that the pagemap has been committed for all spaced managed
by the address space manager.
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
The CapPtr refactoring was largely compiler guided; unfortunately, it turns out
that the static-sized alloc function is not well exercised. As such, some type
errors and unnecessary unsafety lurked behind missing template instantiation.
Correct those and add calls to the test harness to make sure we always generate
at least one instance of each small/medium/large case. While here, it doesn't
hurt to make sure that we call all three possible dealloc() flavors as well.
This will, if nothing else, force instantiation of the static-sized dealloc
template as well.
Even if we opt not to bound these pointers internally (if they aren't headed out
to the user program or we later derive bounded pointers), they should still be
annotated as something other than CBArena, ensuring that we do not attempt to
use them for general amplification.
* The AddressSpaceManager now requests address space in specified granule
sizes and registers those allocations with an external ArenaMap.
* The DefaultArenaMap is a (somewhat erroneously named) Pagemap sparse array /
tree for these provenance roots. Nothing is stored on non-StrictProvenance
architectures.
* In the Sandbox test, give an example of a different ArenaMap structure, which
confines amplification to sandbox memory.
* Adjust some other tests to compile.
We're going to try calling (our, out-of-sandbox) ->dealloc() on pointers into
sandbox memory, so, when CHERIfied, we will need amplification authority over
that memory. Rather than asking the PAL for memory directly, ask the
out-of-sandbox snmalloc so that it will, on CHERI, go through its whole dance
with its AuthMap.
The previous reciprocal division branch on the prime that the sizeclass
was constructed from. All sizeclasses can be represented as
2^n * {1,3,5,7}
This lead to a very small table, but some work to calculate the
appropriate shifts and multiplications to implement reciprocal division.
This commit uses a completely uniform representation for every
sizeclass using a lookup table. Due to the precise ranges that we query
the modulus and rounding on, we can do this much more efficiently.
The func-release-rounding exhaustively tests all the queries we are
interested in.
Like alloc_size, this will require amplification internally.
This patch also restores performance to the status quo ante; Clang can once
again see enough to generate the same code as it did before de-static-ing
alloc_size.
We're going to need to amplify the pointer and that's going to require access
to our AddressSpaceManager, which we only get non-statically through our
LargeAlloc.
This patch unto itself makes the world slower, perhaps because Clang can't see
the certainty of aliasing of the static and non-static paths to the same
structure. However, when we also de-static external_pointer, that goes away and
things return to the status quo ante.
This will let us use Pagemaps further down the dependency stack (specifically,
we're going to want a Pagemap inside the AddressSpaceManager) by letting us
manually tie the knot rather than rely on GlobalVirtual and
default_memory_provider() being defined by the time we want a Pagemap.
- Make GlobalPagemapTemplate and ExternalGlobalPagemap generic in the type of
the pagemap they're encapsulating.
We're going to want to use these for other kinds of pagemaps in the near
future.
- Rename snmalloc_pagemap_global_get to snmalloc_chunkmap_global_get.
- Rename GlobalPagemap to GlobalChunkmap.
MSVC has strong opinions on implicit conversions as used in CI, while Clang both
locally and in CI has weaker opinions. In an effort to avoid subsequent
roundtrips through CI, make clang more strict. Adding -Wconversion definitely
increases the strength of clang's opinions, apparently to include frowning on
some that even MSVC considers OK, so go make explicit the current implicit
behavior.
Summary of changes:
- Add a new PAL that doesn't allocate memory, which can be used with a
memory provider that is pre-initialised with a range of memory.
- Add a `NoAllocation` PAL property so that the methods on a PAL that
doesn't support dynamically reserving address space will never be
called and therefore don't need to be implemented.
- Slightly refactor the memory provider class so that it has a narrower
interface with LargeAlloc and is easier to proxy.
- Allow the address space manager and the memory provider to be
initialised with a range of memory.
This may eventually also remove the need for (or, at least, simplify)
the Open Enclave PAL.
This commit also ends up with a few other cleanups:
- The `malloc_useable_size` CMake test that checks whether the
parameter is const qualified was failing on FreeBSD where this
function is declared in `malloc_np.h` but where including
`malloc.h` raises an error. This should now be more robust.
- The BSD aligned PAL inherited from the BSD PAL, which does not
expose aligned allocation. This meant that it exposed both the
aligned and non-aligned allocation interfaces and so happily
accepted incorrect `constexpr` if blocks that expected one or
the other but accidentally required both to exist. The unaligned
function is now deleted so the same failures that appear in CI should
appear locally for anyone using this PAL.
* Add concept of natural alignment to tests.
snmalloc naturally aligns blocks very heavily, so that
the largest power-of-two in the rounded size is the alignment.
This checks that in the test, and provides a method for
finding the natural alignment of a block.
* Improve USE_MALLOC to provide alignment
snmalloc provides a lot of alginment guarantees. This ensures that when
we pass through to the system allocator we still get those alignment
guarantees.
The commit also fixes the tests to work with USE_MALLOC, and builds a
set of unit tests for ctest to check behaviour.
The pagemap was double incrementing the index level. This did not
exhibit in any use case of the pagemap as it only used 0 or 1 level of
intermediate index for all current use cases.
This commit fixes the bug, and adds a test that uses the pagemap in a
different configuration that has multiple levels of intermediate index
node.
These statistics can be maintained with effectively zero cost to
realistic applications. They do not track the precise amount of
memory used, but are an over-approximation.
This change makes the original 16MiB option not the common option.
It also changes the names of the defines to
SNMALLOC_USE_LARGE_CHUNKS
SNMALLOC_USE_SMALL_CHUNKS
The second should be set for Open Enclave configuration, and results in
256KiB chunk sizes. The first being set builds the original 16MiB chunk
sizes. If neither is set, then we default to 1MiB chunk sizes.
PALOpenEnclave object is lazily constructed. I couldn't
figure out a straight-forward way to pass the heap bounds to
the constructor of PALOpenEnclave object.
As an alternative, store the bounds in inline static variables of
the PALOpenEnclave class and set them via static setup_initial_range
function.
- two_alloc_types/alloc1.cc
Define oe_allocator_init to forward base, end values to
PALOpenEnclave::setup_inital_range
- two_alloc_types/main.cc
Use oe_allocator_init function to set up heap range.
- fixed_region/fixed_region.cc
Initialize heap range via call to PALOpenEnclave::setup_inital_range.
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
* Improved malloc style tests
Added comprehensive testing of realloc, and other minor improvements
to reporting errors.
* Fix realloc resizing for large sizeclasses.
The rounding by sizeclass was incorrect for large allocation. This fixes
that.
* Ensure alloc_size is committed
There is an awkward interaction between alloc_size and
committing only what is requested. If the user assumes
everything up to alloc_size is available, then we need to
either store the more precise size for alloc_size to return
or commit the whole 2^n range, so that alloc_size stays simple.
This changes to just make the whole range committed.
In the future, we might want to store a more precise size, so
that the allocation can be sized more precisely.
* Reduce size of objects.
* Defensive code for alloc/dealloc during TLS teardown
If an allocation or deallocation occurs during TLS teardown, then it is
possible for a new allocator to be created and then this is leaked. On
the mimalloc-bench mstressN benchmark this was observed leading to a
large memory leak.
This fix, detects if we are in the TLS teardown phase, and if so,
the calls to alloc or dealloc must return the allocator once they have
perform the specific operation.
Uses a separate variable to represent if a thread_local's destructor has
run already. This is used to detect thread teardown to put the
allocator into a special slow path to avoid leaks.
* Added some printing first operation to track progress
* Improve error messages on posix
Flush errors, print assert details, and present stack traces.
* Detect incorrect use of pool.
* Clang format.
* Replace broken LL/SC implementation
LL/SC implementation was broken, this replaces it with
a locking implementation. Changes the API to support LL/SC
for future implementation on ARM.
* Improve TLS teardown.
* Make std::function fully inlined.
* Factor out PALLinux stack trace.
* Add checks for leaking allocators.
* Add release build of Windows Clang
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
