This change brings in a new approach to managing address space.
It wraps the Pal with a power of two reservation system, that
guarantees all returned blocks are naturally aligned to their size. It
either lets the Pal perform aligned requests, or over allocates and
splits into power of two blocks.
Since we anticipate address_t not carrying provenance on CHERI, but
rather being vaddr_t there, it doesn't make sense to offer conversion
back to a provenance-carrying pointer.
Thankfully, there is not much to be done here: the uses were few and
could be replaced with the vocabulary of other pointer operations in
ds/address.h
* Improved malloc style tests
Added comprehensive testing of realloc, and other minor improvements
to reporting errors.
* Fix realloc resizing for large sizeclasses.
The rounding by sizeclass was incorrect for large allocation. This fixes
that.
* Ensure alloc_size is committed
There is an awkward interaction between alloc_size and
committing only what is requested. If the user assumes
everything up to alloc_size is available, then we need to
either store the more precise size for alloc_size to return
or commit the whole 2^n range, so that alloc_size stays simple.
This changes to just make the whole range committed.
In the future, we might want to store a more precise size, so
that the allocation can be sized more precisely.
* Reduce size of objects.
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
The bootstrapping allocator needs to perform a memcpy to bypass the
removed move constructors on std::atomic. This is safe as there is no
concurrency at this point, but GCC is unhappy with this.
This commit moves CI to GCC8 and disables this warning for that line.
On platforms that support low-memory notifications register callbacks
that perform lazy decommit. This allows idle processes to return memory
to the OS. Without incurring the cost of constantly committing and
decommitting memory.
Code review and CI changes
* Fixed test to use a template to make constexpr magic work
* Factored out basic notification mechanism so can be reused on other
platforms.
If the external thread statics are used, then
we don't need to include some C++ runtime
concepts. This refactoring moves some global initialization under
conditional compilation.
The low-memory notification was getting into an infinite loop. This
fixes the loop termination, and provides a test for platforms which
support low-memory notification.
On platforms that do not support aligned mmap/VirtualAlloc,
we need to produce heavily aligned blocks to guarantee we can meet
all possible alignment requests.
This commit grabs a block much larger than requested, and then produces
"offcuts" before and after the block of smaller/same "large_classes". This
enables one mmap/virtual alloc request to services many other requests
for aligned memory.
The PAL API previously allowed for returning more memory than asked for.
This was when the PAL performed the alignment work, now this is done in
large alloc, so removing from the PAL.
HEADER_GLOBAL was using non-standard attributes to achieve what C++17
now permits with a keyword. Use the standard formulation.
Update the README to note that gcc is still not recommended, but because
of its poor codegen for 128-bit atomic compare and exchange, rather than
because it doesn't support the attribute used for HEADER_GLOBAL.
This introduces a new `address_t` type and two new casts: `pointer_cast`
and `address_cast` for casting between an `address_t` and a pointer.
These should make it easier to audit the codebase for casts between
pointers and integers. In particular, the remaining `reinterpret_cast`s
and `pointer_cast`s should be the only places where we could perform
invalid pointer arithmetic.
Also adds a `pointer_offset` helper that adds an offset (in bytes) to a
pointer, preserving its original type. This is a sufficiently common
pattern that it seemed worthwhile to centralise it.
