Expose a memcpy.h that contains all of the bits of memcpy and clean up
the bounds checks header so that versions with both read and write
checks can coexist.
We currently include 256+GiB zeroes in core dumps. This is a QoI issue
because it causes core dumps to take a long time (even if ZFS is able to
compress large runs of zeros down to almost nothing, they still take a
long time to load into debuggers). This patch removes anything marked
as not-using or using-but-read-only (pagemap zero pages) as excluded
from core dumps.
Adding __builtin_trap() to the end of `func-malloc-fast` now gives a 21
MiB core dump rather than a 256 GiB one.
This provides a single place for reporting messages to the user.
While here, be consistent about using stderr for things that should go
to stderr. We were previously using a mix of stderr and stdout.
This now either outperforms, or performs as well within the region of
measurement noise as, FreeBSD's libc memcpy, which is hand-written
assembly.
This uses a jump table for small copies with a sequence of power-of-two
loads and stores for each, a vector-register copy with an overlapping copy
for the last chunk for medium copies and, on x86, rep movsb for large
copies.
The checked version still incurs a noticeable overhead.
- Refactor the existing SNMALLOC_ASSERT and SNMALLOC_CHECK. These now
use the FatalErrorBuilder to format the output if a format string is
provided.
- Extend the FatalErrorBuilder to print decimal integers for signed
values.
- Rename FatalErrorBuilder to MessageBuilder.
- Rewrite the macros used in the jemalloc tests to use
FatalErrorBuilder and move them into a header.
- Refactor some of the tests to use the new macros.
This introduces a very limited formatter that can embed strings and hex
representations of pointers / integers in an internal buffer. This is
used to format error strings for passing to `Pal::error`. This is used,
in turn, by a wrapper for reporting bounds checks, which can be used by
external functions to implement bounds checks.
This removes the sprintf_l usage from the bounds checks.
This provides enough of a format implementation that the tests
introduced in #465 can be refactored to use this, instead of their
custom `printf` wrapper and that can be used by SNMALLOC_CHECK. This
will be a follow-on PR.
Correctly set errno on failure and improve the related test.
Previously the malloc test would emit an error message but not
abort if the errno was not as expected on failure. This
was because the return in the null == true case prevented the
check for failed == true at the end of check_result from
being reached. To resolve this just abort immediately as in the
null case.
Also add tests of allocations that are expected to fail for
calloc and malloc.
To make the tests pass we need to set errno in several places,
making sure to keep this off the fast path.
We must also take care not to attempt to zero nullptr in case
of calloc failure.
See microsoft/snmalloc#461 and microsoft/snmalloc#463.
This is especially important on CHERI to avoid leaking capabilities to
the freelist. In the CHERI case we also zero in clear_slab (see comment).
Also add a check in the malloc functional test that there are no valid
capabilities in the returned allocation.
An annoying amount of real-world code (e.g. mandoc, BSD sort) treats a
NULL return from `realloc` as a failure, even when requesting a size of
0. This code is wrong (the standard explicitly permits a return of NULL
from realloc when given a size 0) but working around it in snmalloc is
easier than fixing it everywhere.
This adds the full set of jemalloc functions that FreeBSD's libc
exposes, including some (the `*allocm` family) that are gone from newer
versions of jemalloc and the `*allocx` family that replaced them. These
are not necessarily efficient implementations but they should allow
snmalloc to replace jemalloc without any ABI breakage (in the loosest
possible sense).
Jemalloc provides a very generic sysctl-like mechanism for setting and
getting some values. These are all implemented to return the
not-supported error code. This may break code that expects that they
will succeed.
In particular, these APIs are used to register custom backing-store
allocators and to manage caches and arenas. These concepts don't map
directly onto snmalloc and attempting to do so would almost certainly
not provide the same performance characteristics and so it's better to
`LD_PRELOAD` jemalloc (or explicitly link to it) for programs that gain
a significant speedup from this.
- Mark the hook that we're exporting for the threading library to call
to clean up per-thread malloc state as 'used'. It was changed to
`inline` to allow duplicate copies of it to be merged but this also
means that it isn't emitted at all in compilation units that don't
use it (and it isn't used internally at all).
- Fix the `__je_bootstrap_*` functions, which are used to bootstrap TLS
allocation, for the changes to `ScopedAllocator`.
The `__je_bootstrap*` functions weren't being built in CI. They now are
for non-PIE targets with a smoke test.
* Improve testing of memcpy including adding perf test.
* Change remaining_bytes to be branch free.
Use reciprocal division followed by multiply to remove a branch.
* Post large deallocations to original thread
This change sets all large allocations to be owned by the originating
thread. This means they will be messaged back to the original thread
before they can be reused.
The following reason for making this change:
* This will improve producer/consumer apps involving large allocations.
* It enables the implementation of a more complex chunk allocator that
reassembles chunks.
* It addresses an issue with compartmentalisation where the handling of
large allocations can result in meta-data ownership changing.
The CHERI-RISC-V `CAndPerm` and `CSetBoundsExact` instructions trap on untagged
inputs, so avoid passing `nullptr` to primitives that become those instructions.
TSAN complained that there was a race that after some thoughts appears
to be due to this exchange needing to be an `acquire`.
I still wonder if the data dependence that is threaded through the
exchange induces enough order.
* export netbsd's reallocarr proposal.
acts subtly differently from reallocarray, returns an error code
and first argument as receiver.
* not export by default
* ci tests
* apply suggestions
* doc addition
* Apply suggestions from code review
Co-authored-by: Matthew Parkinson <mjp41@users.noreply.github.com>
On Open Enclave having the `local_alloc` directly in thread-local
storage was causing a crash. This changes the `local_alloc` to be
indirected, and thus puts less pressure on the thread-local storage.
The test also has deals with how to allocate before a thread-local
storage has been established.
