We mean to be allocating MIN_ALLOC_SIZE (== 2 * sizeof(void*)), not
sizeof(MIN_ALLOC_SIZE) (== sizeof(size_t)). This doesn't matter in practice
since, well, MIN_ALLOC_SIZE is the minimum allocation size, and so requesting
either will have the same effect. Still, best to say what we mean.
With snmalloc2, slabs are linked through the Metaslab structure directly rather
than in-band in a free allocation, so we no longer need to store a SlabLink in
even the smallest allocation classes.
The threshold for waking is used to ensure that we only use a slab when
it has sufficient space that we won't hit the slow path to soon after
using this slab. In the checked version, this is also used to give some
entropy in layout. Changing this to always be a pertcentage in the
checked case increases the effectiveness of the free list to detect OOB
write.
- CI merge issues:
- The malloc shim libraries are renamed.
- CMake gets very unhappy if you don't enable the C language and
tries to link with the C compiler instead of the C++ compiler if
you do enable it.
- The Ubuntu packages for QEMU install a `binfmt_misc` activator for
PowerPC64 little-endian, but set the page size to 4 KiB. We then
tried to run the tests (which expect 64 KiB pages) and became very
confused when `mmap` returned 4 KiB-aligned memory.
- Test failures:
- Fix all of the issues UBsan found.
- Underflow in `pointer_offset` when used to add negative offsets.
- `CoreAlloc`'s `LocalState` accessed on a null `CoreAlloc` pointer.
- Out of bounds access in the sizeclass list on attempts to access
more memory than fits in the VA space.
-
- There was an integer overflow in `AddressSpace` that could cause it
to try to allocate a zero-sized object, get a null pointer, and
then try to do something with 0 - {size of the real allocation}.
- The malloc tests weren't setting `errno` to 0 before doing
calling `malloc`, which should set `errno` on failure, and then
checking that `errno` was 0.
- Don't call `PAL::error` on PAL allocation failure, return `nullptr`.
The PALs were inconsistent about that and the new code expects to be
able to report address-space exhaustion.
- The malloc checks can behave differently with 0-sized allocations
on different platforms but were very fragile about their
expectations.
- The malloc test didn't report failure for all of the ways that it
could fail and so was spuriously passing on some platforms.
- The perf test for external pointer is currently very slow on
Windows. The number of loops have been reduced and a timeout added
for the Windows CI runs.
- The logic to capture `errno` across calls was using
`decltype(errno)`, which on some platforms where `errno` is a macro
evaluated to `int&` and so they captured a reference rather than
the value and failed to reset `errno`.
- The Apple PAL can set `errno` on `notify_using` if it's called with
memory that was not previously passed to `notify_not_using` but was
not adequately protected against this and so would sometimes cause
`malloc` to set `errno` to `EINVAL`.
This is the set of changes required for snmalloc2 to be usable by the
process sandboxing code and incorporates some API changes that reduce
the amount of code required to embed snmalloc. Highlights:
- Merge the config and back-end classes.
- Everything in config is now global (all methods are static)
- The GlobalState class is gone (all global state is managed by global
methods on the config class)
- LocalState is now a member of the config class, all methods are
instance methods.
- Not every configuration needs to use the lazy initialisation hooks.
They now need to be provided only if they are used. If the
configuration does not provide an `ensure_init` method, it is not
called. If it does not provide an `is_initialised` method then the
global initialisation state is not checked.
- There is now an `snmalloc::Options` class that default initialises
itself to the default behaviour. Every configuration must provide a
`constexpr` instance of this class. Each flag can be separately
overridden and new flags can be added without breaking any existing
API consumers.
The config classes are moved into the backend directory.
Use a matrix containing objects rather than a matrix containing keys
that we try to use to define arrays of things.
Fix a typo in ARCH. Too much Arm made it AARCH.
Also add a check that the test programs are under about ten megabytes
(they're currently around one on platforms that put inline statics full
of zeroes into BSS and around 270 on ones that don't).
Fixes#339
The naming is more than a little confusing. AIUI, in Ubuntu's terminology, the
`powerpc` packages are for 32-bit machines, the `ppc64-powerpc` packages are for
Big Endian 64-bit machines, and the `ppc64el` packages are for Little Endian
64-bit machines.
Everyone seems to have agreed that the long-term answer for 64-bit PowerPC is
Little Endian (Debian maintains an unofficial BE port, but Canonical/Ubuntu,
RedHat, and SUSE all seem to have standardized on LE).
For maximal confusion, the appropriate triple's first component is, however,
`powerpc64le`.
The status badge should report the most recent status for the master
branch, not the most recent actions run.
llvm.sh wasn't updated when trunk moved to 14 so was failing to fetch clang-13 (which is now in the llvm-toolchain-focal-13 repo, not the llvm-toolchain-focal repo). Duplicate the correct logic here rather than relying on the external script.
A few highlights relative to our existing CI:
- Add a FreeBSD 12.2 and 13.0 runner so we have some FreeBSD CI.
- Windows builds use msbuild with the Visual-Studio-provided clang toolchain to test clang
- The matrix builds describe the axes of the matrix, not all points.
- The Arm builds now cross-compile with a native clang and run the tests with qemu, rather than running the compiler, linker, and ctest all with qemu.
This also includes a fix for one of the tests that was doing `static_cast<unsigned int>(1) << 36`, which is undefined behaviour and was sometimes causing qemu to hang. There is now an assert to catch this in the future.
This commit adds a simple XOR encoding to the next_object pointer in
FreeObjects. This removes the trivial way of getting hold of a physical
address from the system by observing the free list pointers in
deallocated objects.
* Make address space manager use pagemap for next pointers
The address space manager uses the pagemap entry to form linked lists of
unused address space above MIN_CHUNK_SIZE. It continues to use
references in the block below that threshold.
In the CHECK_CLIENT mode this makes it hard to corrupt the ASM as only
meta-data uses allocations below MIN_CHUNK_SIZE from the ASM. This
allocations will be protected with guard pages by the backend.
* address_space_core: use FreeChunk struct
Purely stylistic, NFCI. This hides some somewhat gnarly reinterpret_cast-s in
favor of more, but hopefully less gnarly, casts elsewhere.
* Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
Take the maximum of...
* CACHELINE_SIZE (for performance)
* next_pow2(NUM_SIZECLASSES + 1) so that, when the pagemap points to a Remote,
the (small) size class stuffed in the bottom bits can be removed by alignment
* next_pow2(NUM_LARGE_CLASSES + 1) so that, when the pagemap isn't pointing to
a Remote, when the associated chunk is (part of) a large allocation, aligning
the Remote* results in 0.
The last of these conditions will almost never be the deciding factor, as there
are generally many more small size classes than large ones, but it shouldn't
hurt to be safe.
The meta-data in the CHECK_CLIENT mode is allocated from its own areas,
where most of the pages have been disabled, and the location within this
range is randomised.
This protects from trivial OOB read/write hitting the meta-data.
More complex controlled offsets are also mitigated due to the sparse
level of pages being turned on (i.e. not PROT_NONE).
The pagemap contains a lot of important data. This commit makes the
checked mode overallocate, and then start the pagemap at a random
offset within this range.
The Pal was providing policy for overallocating a block of memory to
achieve alignment make that part of the backend.
The backend should be responsible for layout policy.
This PR exposes a pagemap interface to specify ranges that are being
used. The overall invariant is that any memory in the address space
manager has the pagemap committed. This means that individual operations
do not need to commit entries.
This is important for Windows that does not support lazy commit. It is
also important if we want to PROT_NONE most of the pagemap to reduce the
risk of memory safety issues getting access to the pagemap.
There are minor changes to test to pull memory directly from the Pal.
There are also bug fixes in the pagemap tests.
The pagemap allocates it self directly either from
* the original fixed address range it is supplied, and returns the
remaining space after the pagemap is removed; or
* directly allocated from the PAL without using the address space
manager.
This change in layering is required for the next commit, which imposes
the invariant that the pagemap has been committed for all spaced managed
by the address space manager.
It is UB to offset from `nullptr` (except perhaps with a 0 offset). Apparently
clang is able to use this to reason, given `void* p`, that comparing
`__builtin_align_down(p, x)` against `handle.fake_large_remote` (i.e., a `static
inline constexpr` `nullptr`) must be the same as comparing `p` itself against
`nullptr`.
In `MetaEntry`'s constructor, converting the provided `RemoteAllocator*` to
`uintptr_t` before offsetting avoids the UB. (While here, don't use
`address_cast()`, as `address_t` will, on CHERI, be `ptraddr_t` and not
`uintptr_t`.)
Doing the alignment in `get_remote` at `uintptr_t` before casting to
`RemoteAllocator*`, rather than converting and then aligning, prevents the
reasoning above from eliminating the alignment.
This extends the freelist protection to the remote message queues. They
effectively perform doubly linked list entries for the message queue
with the enqueue operation first linking in the previous pointer, and
then then atomically setting the next. This ensures that the visible
states always satisfy the invariant that the forward and backward
pointers are correct for any visisble object.
There is a key_global that is used for all remote deallocations. The
remote cache uses the same protection to build the temporary lists
before forwarding to the next allocator.
The mpscq is integrated into the remoteallocator as it is no longer
a reusable datastructure, but a special purpose implementation.
* Cleaner implementation of signed pointers.
This encodes a back pointer in each node. The back pointer is stored
in an encoded form so that it is hard to corrupt and trick the allocator
into following incorrect pointers.
This changes the encoding from previously being a Feistel network on
the next pointer that was using the prev as part of the key, to now
effectively using a doubly linked queue, where the back pointers are
scrambled, so it is hard to forge them.
This has the positive effects of
- Not needing to store previous while building the list, as the append
nows, curr and next at the point of writing into next, and does not
need an additional previous.
- The encoding is not affecting the actual next value, so more
instructions can be executed in parallel by the CPU.
Future extensions, store a changing key in the FreeListBuilder so it
becomes harder to try to forge the previous token.
This approach can also be applied to the remote list, and will in a
subsequent PR. This enables the idea to be tested.
* Remove unused header.
* Apply suggestions from code review
Co-authored-by: Nathaniel Wesley Filardo <VP331RHQ115POU58JFRLKB7OPA0L18E3@cmx.ietfng.org>
Co-authored-by: Nathaniel Wesley Filardo <VP331RHQ115POU58JFRLKB7OPA0L18E3@cmx.ietfng.org>
# Pagemap
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.
This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.
# Backend
There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.
# Thread local state
The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.
# Difference
This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
GCC is a lot more picky about extern "C" definitions in namespaces than
clang and so the `friend` declaration wasn't picked up by the version of
the function declared in the `snmalloc` namespace. Move it out to where
GCC is happy with it.
