Define various parts of random that can be used to make the layout of
memory more random. Thread this through the allocator.
Expose the concept as part of the Pal. Subsequent commits will expose
that on different platforms.
When a slab has been fully allocated, then we no longer
check it has entries until something returns an allocation to this slab.
However, it is possible that only a single allocation is available, and
then we can end up frequently on the slow path.
This change only considers free lists that cover at least 1/8 of a slab.
This means that we will hit the slow path less frequently. This also
means that the randomisation changes will have more entropy: with a
single element free list there is only one order.
For large small sizes it can still be a single element, as 1/8 is of the
slab capacity is below 1. We max out the trigger at 31 elements to
reduce unneeded wasted space.
The slab allocation pattern is randomised based on the deallocation
pattern. This achieved by using two queues to enqueue free elements
onto. We pick "randomly", which queue to add to, and then when we take
the free_queue to use, we splice the two queues together.
* Replace time measuring macro
The DO_TIME macro was used originally to get performance numbers. The
macro makes tests hard to debug. This commit replaces it with a proper
C++ class with destructor.
* Bug fix
If the superslab meta data is large, then the calculation for the
sizeclasses that could use the short slab was incorrect. This fixes
that calculation.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
The metaslab contains a field specifying how many elements have been
allocated. As the code has evolved this field has now always become
the maximum capacity of the slab for the sizeclass.
This commit looks up this value based on the sizeclass, and removes the
field from the slab's metadata.
Free list pointers can be exploited by attackers. This commit implements
a simple encoding scheme to detect corruption of the pointers. This can
be used to detect UAF and double free.
This does not currently address anything for Medium or Large
allocations. It also does not address cross thread deallocations.
Co-authored-by: Nathaniel Wesley Filardo <nfilardo@microsoft.com>
Going forward, this gives us explicit pointers with which to carry bounds
annotations. Otherwise, assuming AuthPtr overloads operator->, a OOP-style call
like
AuthPtr<Slab, Bounds> slab;
slab->foo()
will create a `Slab* this` within the body of `Slab::foo`, leaving it unable to
see or propagate the Bounds annotation. If it invokes callees that expect
`AuthPtr` arguments, it will therefore have to fabricate new `Bounds` unsafely.
The link object was previously stored in a disused object. This is
good for reducing meta-data, but if we want to reduce the meta-data
corruption potential, then this is not a good design choice.
This commit moves it into the Metaslab.
* Remote dealloc refactor.
* Improve remote dealloc
Change remote to count down to 0, so fast path does not need a constant.
Use signed value so that branch does not depend on addition.
* Inline remote_dealloc
The fast path of remote_dealloc is sufficiently compact that it can be
inlined.
* Improve fast path in Slab::alloc
Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.
* Refactor initialisation to help fast path.
Break lazy initialisation into two functions, so it is easier to codegen
fast paths.
* Minor tidy to statically sized dealloc.
* Refactor semi-slow path for alloc
Make the backup path a bit faster. Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.
* Test initial operation of a thread
The first operation a new thread takes is special. It results in
allocating an allocator, and swinging it into the TLS. This makes
this a very special path, that is rarely tested. This test generates
a lot of threads to cover the first alloc and dealloc operations.
* Correctly handle reusing get_noncachable
* Fix large alloc stats
Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).
* Fix TLS init on large alloc path
* Add Bump ptrs to allocator
Each allocator has a bump ptr for each size class. This is no longer
slab local.
Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.
* Change to a cycle non-empty list
This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.
* Update differences
* Rename first allocation
Use needs initialisation as makes more sense for other scenarios.
* Use a ptrdiff to help with zero init.
* Make GlobalPlaceholder zero init
The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
The performance on Windows was significantly regressed by the
notify_using during the bump allocation. This change removes that.
It appears that the pages are already committed by
the large allocator.
Used is now set to 1, when the slab is full. This means that the test
for
used - 1 == 0
can be used to detect leaving full, and entering empty, reducing fast
path deallocation branchs by 1.
This commit changes the strategy for finding a free list from
a stack to a queue. This tends to avoid the slow path considerably more.
It has some memory overheads.
TOOD: We should move the bump allocation data out of the metaslab and
into the allocator. At the moment, the slab contains the bump allocation
data, we should move this into the allocator, as it only ever has one slab
it is bump allocating from per sizeclass.
This is needed because in some configurations the constructor for the
global placeholder is not called before the first allocation (i.e. when
other globals call the allocator in their constructor) and so we ended
up following a null pointer.
This change introduces a per small sizeclass free list. That can be
used to access the free objects for that sizeclass with minimal
calculations being required.
It changes to a partial bump ptr. We bump allocate a whole OS
page worth of objects at a go, so we don't switch as frequently
between bump and free list allocation.
The code for the fast paths has been restructured to minimise the
work required on the common case, and also it is all inlined for the
common case.
Allocating a zero sized object is moved off the fast path. Ask for 1
byte if you want to be fast.
Will detect corruption caused by either
* Use-after-free
* Double-free
Neither is comprehensive. Full temporal safety is not possible.
This just aids with debugging.
This introduces a new `address_t` type and two new casts: `pointer_cast`
and `address_cast` for casting between an `address_t` and a pointer.
These should make it easier to audit the codebase for casts between
pointers and integers. In particular, the remaining `reinterpret_cast`s
and `pointer_cast`s should be the only places where we could perform
invalid pointer arithmetic.
Also adds a `pointer_offset` helper that adds an offset (in bytes) to a
pointer, preserving its original type. This is a sufficiently common
pattern that it seemed worthwhile to centralise it.
This code assumes that 16bit uint wrapping can be used, rather than
masking by SLAB_SIZE-1. Add explicit mask, so we can change the
SLAB_SIZE to be smaller.
