[NFC][TSan] Move ObtainCurrentStack out of ThreadRegistryLock scope
ObtainCurrentStack only reads the current thread's shadow stack and
allocates a VarSizeStackTrace buffer, which does not require
ThreadRegistryLock (and already runs outside ThreadRegistryLock in
ReportRace, SignalUnsafeCall, ReportErrnoSpoiling, and
ReportDestroyLocked).
Move ObtainCurrentStack (and dummy_pc in ReportDeadlock) before
ScopedReport in ReportMutexHeldWrongContext, ReportMutexMisuse, and
ReportDeadlock so the stack trace buffers also outlive ScopedReport and
OutputReport.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228794
[NFC][TSan] Move RestoreStack before ScopedReport in ReportDestroyLocked
Run RestoreStack in its own lock scope before constructing ScopedReport
in ReportDestroyLocked (matching ReportRace). This avoids acquiring
ScopedErrorReportLock or symbolizing the current stack if RestoreStack
fails, and avoids holding slot_lock and slot_mtx while populating the
ScopedReport.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228643
[NFC][TSan] Merge ScopedReportBase into ScopedReport
ScopedReport is the only subclass of ScopedReportBase and nothing uses
ScopedReportBase directly. Merge ScopedReportBase into ScopedReport.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228640
www/deno: map deno compile payload instead of copying it
Executables produced by `deno compile` read their embedded payload into
memory at startup on FreeBSD, so a 200 MB payload made a trivial app
start with 431 MB RSS instead of 48 MB. Map the payload section from
the executable instead, so it is paged in on demand like on other
platforms.
[CodeGen] Declare command line options in TableGen
Move the cl::opts of TargetPassConfig.cpp and CodeGenPrepare.cpp into
CodeGenOptions.td, private to lib/CodeGen; other files will follow.
-enable-machine-outliner (cl::ValueOptional) and -regalloc
(RegisterPassParser) stay cl::opt.
CodeGenPrepare and its addressing-mode helpers hold
`const CodeGenOptions &Opts`; TargetPassConfig functions read
CodeGenOptions::Global. getCGPassBuilderOption() converts
std::optional<bool> members to the cl::boolOrDefault fields of the
public CGPassBuilderOption. -basic-block-section-match-infer, which was
not cl::Hidden, is now listed by -help-hidden only.
Aided by Opus 5.5
[mlir][tblgen] Warn about the deprecated multi-result fold form
The legacy form `LogicalResult fold(FoldAdaptor,
SmallVectorImpl<OpFoldResult> &)` will be removed. This patch makes
`mlir-tblgen -gen-op-decls` warn when a dialect keeps `useOpFoldResults`
at 0 and has an op with `hasFolder` that does not have exactly one fixed
result. The warning points at the dialect definition. A note names the
first op that uses the legacy form.
The warning comes once for each dialect in one `-gen-op-decls` run. A
dialect with ops in more than one `.td` file can warn once for each file
that contains such an op. `-gen-op-defs` does not warn.
No in-tree dialect warns, because each in-tree dialect with such an op
already sets the bit.
The diagnostic follows the `-on-deprecated` option: `none` silences it,
`warn` (the default) warns, and `error` reports an error and fails the
run. `MlirTblgenMain.h` exposes the option value through
[3 lines not shown]
[mlir] Deprecate the legacy fold APIs with a results vector
The legacy `Operation::fold` overloads with a
`SmallVectorImpl<OpFoldResult> &` parameter drop partial folds. The
overloads that return `OpFoldResults` keep them.
This patch moves every in-tree caller of the legacy overloads to the
overloads that return `OpFoldResults`, except the unit tests of the
legacy APIs. `OpFoldResultsTest.cpp` suppresses the deprecation warnings
for these tests. Then the patch marks these APIs as deprecated: the two
legacy `Operation::fold` overloads, the legacy general `foldTrait` form,
the legacy fold hook overloads of `DynamicOpDefinition`, and the
`DynamicOpDefinition::LegacyFoldHookFn` alias. ODS cannot put an
attribute on an interface method, so only the documentation marks the
legacy `DialectFoldInterface::fold` method as deprecated.
The new code in `cir::CastOp::fold` also fixes two bugs. The fold
crashed when the source of an integral cast was a block argument, and it
read the fold result of result 0 when the source was a different result
[17 lines not shown]
[mlir][CIR] Use OpFoldResults for the cir.scope fold
The CIR dialect now sets the `useOpFoldResults` bit. ODS then declares
`OpFoldResults fold(FoldAdaptor)` for each CIR op that does not have
exactly one fixed result. `cir.scope` is the only such op with a fold.
Its fold now returns the yielded value directly. The behavior does not
change. The existing test `clang/test/CIR/Transforms/canonicalize.cir`
covers this fold.
Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
[mlir] Use OpFoldResults in more upstream dialects
The scf, shape, sparse_tensor, gpu, math, and builtin dialects now set
the `useOpFoldResults` bit. ODS then declares `OpFoldResults
fold(FoldAdaptor)` for each of their ops that does not have exactly one
fixed result. This patch moves the seven folds of such ops to the new
form: `scf.if`, `shape.split_at`, `sparse_tensor.crd_translate`,
`gpu.memcpy`, `gpu.memset`, `math.sincos`, and
`builtin.unrealized_conversion_cast`.
The behavior of the folds does not change, except in the graph-region
case below. Each fold builds the same normalized `OpFoldResults` object
that the legacy adapter built from the old return. The existing tests of
each dialect cover these folds.
In a graph region, an operand that `unrealized_conversion_cast` or
`sparse_tensor.crd_translate` forwards can be another result of the
same op that the same fold replaces. The new form drops such a fold.
For a forward chain, the legacy form gave a correct result by the order
[22 lines not shown]
[mlir][linalg] Use OpFoldResults for linalg folds
The linalg dialect now sets the `useOpFoldResults` bit. ODS then
declares `OpFoldResults fold(FoldAdaptor)` for each linalg op that does
not have exactly one fixed result. This patch moves the eleven folds of
such ops in `LinalgOps.cpp` to the new form. It also changes the fold
template of `mlir-linalg-ods-yaml-gen`, so the generated named ops use
the new form too.
The behavior of the folds does not change. A fold that returned the
result of `memref::foldMemRefCast` now returns the same in-place state
through the `LogicalResult` constructor. The folds of `transpose`,
`pack`, and `unpack` return their replacement value directly.
The yaml-gen test now checks the generated fold definition.
A build directory that uses a native `mlir-linalg-ods-yaml-gen` does
not regenerate `LinalgNamedStructuredOps.yamlgen.cpp.inc` when the tool
changes. Delete that file before the build.
[2 lines not shown]
[mlir][vector] Use OpFoldResults for vector folds
The vector dialect now sets the `useOpFoldResults` bit. ODS then
declares `OpFoldResults fold(FoldAdaptor)` for each vector op that does
not have exactly one fixed result. This patch moves the five folds of
such ops to the new form: `to_elements`, `transfer_write`, `store`,
`masked_store`, and `mask`.
The behavior of the folds does not change, except in the graph-region
case below. A fold that returned success with an empty vector now
returns `success()`, which is an in-place change. A fold that filled the
vector now returns its values.
The all-true fold of `vector.mask` moves the masked op out of the
region, and the terminator then has null operands. So this fold must
replace every result, and the driver erases the op. A mask without
results has nothing to replace, so the fold reports the move as an
in-place change.
[26 lines not shown]
[mlir][memref] Use OpFoldResults for memref folds
The memref dialect now sets `useOpFoldResults`. ODS then declares
`OpFoldResults fold(FoldAdaptor)` for each memref op that does not have
exactly one fixed result. This patch moves the seven folds of such ops
to the new form: `copy`, `dealloc`, `dma_start`, `dma_wait`,
`extract_strided_metadata`, `prefetch`, and `store`. The behavior of
these folds does not change, except for `extract_strided_metadata`.
The `memref.extract_strided_metadata` fold created `arith.constant` ops
with its own builder and replaced the uses of the constant results
itself. No driver saw these changes. The fold now returns a partial
fold: one replacement for each constant result, and an in-place mark
when it removes a `memref.cast` source. This patch removes the helper
`replaceConstantUsesOf`, which has no other user.
DialectConversion folds an op before it applies the patterns. So the
conversion did not see the new constants and left them unconverted.
With pattern rollback on, the conversion now does not apply the partial
[27 lines not shown]
[mlir][affine] Use OpFoldResults for affine folds
The affine dialect now sets `useOpFoldResults`. ODS then declares
`OpFoldResults fold(FoldAdaptor)` for each affine op that does not have
exactly one fixed result. This patch moves the eight folds of such ops
to the new form: `dma_start`, `dma_wait`, `for`, `if`, `store`,
`prefetch`, `parallel`, and `delinearize_index`. The behavior of these
folds does not change, with two exceptions.
The `affine.delinearize_index` fold now replaces each result whose basis
element is 1 with the constant 0, and keeps the other results. In
`Tensor/bubble-up-extract-slice-op.mlir`, these results now fold to a
constant 0. The new test `Affine/fold-partial.mlir` runs
`-test-single-fold` and `-sccp`, because `-canonicalize` also runs
`DropUnitExtentBasis`, which hides a broken fold.
In a graph region, an init of a zero-trip `affine.for` can be another
result of the same loop. The fold also replaces that result, so the new
form drops the fold. For a forward chain such as
[21 lines not shown]
[mlir][arith] Use OpFoldResults for arith folds
The arith dialect now sets `useOpFoldResults`. ODS then declares
`OpFoldResults fold(FoldAdaptor)` for each arith op that does not have
exactly one fixed result. This patch moves the four folds of such ops to
the new form: `addui_extended`, `subui_extended`, `mulsi_extended`, and
`mului_extended`. The behavior of these folds does not change, with two
exceptions.
The `arith.mulsi_extended` fold now replaces the low result of
`mulsi_extended(x, 1)` by `x`, and keeps the high result. This is also
correct for i1, where the constant `true` is -1. The i1 tests in
`Arith/canonicalize.mlir` change their expected output.
In a graph region, the identity fold of `addui_extended`,
`subui_extended`, or `mului_extended` can forward an operand that is the
other result of the same op. The fold also replaces that result, so the
new form drops the fold. The legacy form gave a correct result in this
case, because it first moved the uses to the forwarded result, and then
[7 lines not shown]
[mlir] Use the replaced results of a partial fold in SCCP
`SparseConstantPropagation` now uses the `OpFoldResults` form of
`Operation::fold`. It merges the value of each replaced result into the
lattice of that result, and it sets each kept result to the entry
state. A kept result must not join with its own lattice, because that
leaves the lattice uninitialized. Before this patch, SCCP saw a partial
fold as a failure and set every result to the entry state.
The new test in `sccp.mlir` checks that SCCP uses the replaced results
and that the kept result stays overdefined.
Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
[mlir] Apply partial folds in the greedy pattern rewrite driver
The greedy driver now applies the `OpFoldResults` of a fold:
- A fold that replaces every result erases the op. A result without
uses gets no constant.
- A partial fold replaces the uses of each replaced result and keeps
the op. The driver puts the op on the worklist again.
- The materialization of constants is all-or-nothing. When one
constant fails, the driver erases only the constants of this call
and applies no replacement. An in-place change still counts.
- A fold that keeps every result and has no in-place mark fails.
The driver uses `OpBuilder::materializeFoldResults` for the constants.
When `MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS` is on, the driver
compares the fingerprints of the op before and after the fold. A fold
that changes the op and returns failure is a fatal error. A fold that
changes an op that stays, without an in-place mark, is also a fatal
error.
[8 lines not shown]
[mlir] Apply partial folds in OperationFolder
`OperationFolder::tryToFold` now applies the `OpFoldResults` of a fold.
A fold that replaces every result erases the op. A partial fold replaces
the uses of each replaced result, keeps the op, and sets
`inPlaceUpdate`. The folder materializes constants only for the replaced
results that have uses. If a constant fails to materialize, no result is
replaced, and an in-place change still counts. The private vector
overload of `tryToFold` has no callers after this change, so this patch
removes it.
This patch also fixes `OperationFolder::processFoldResults`. It moved a
reused constant to the front of the block as soon as it used the
constant for a result. If a later result then failed to materialize,
the cleanup erased all ops before the insertion point. These ops
included the moved constant, which still had uses. The folder now moves
the reused constants only after all results materialize.
The bug does not need partial folds. Without this patch, the new test
[16 lines not shown]
[mlir] Deprecate the OpBuilder::tryFold overload with a results vector
The `OpBuilder::tryFold` overload with a `SmallVectorImpl<Value> &`
parameter drops partial folds. The overload that returns
`OpFoldResults`, together with `materializeFoldResults`, keeps them.
No in-tree code calls the old overload anymore, so this patch marks it
as deprecated. The unit test of the old overload suppresses the
deprecation warning.
Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
[flang] Use the OpFoldResults overload of OpBuilder::tryFold
`getAttrIfConstant` in `LLVMInsertChainFolder.cpp` folds the op that
defines a value and reads the constant of the value from the fold. It
used the old `OpBuilder::tryFold` overload, which materializes a
constant for each result. The function only read the value of the new
`llvm.mlir.constant`, so these constants stayed in the IR without uses.
The function now uses the new overload, which returns the fold result
and creates no constant. It accepts the same constants as before. The
fold must replace every result. A `Value` replacement counts if
`llvm.mlir.constant` defines it. An attribute replacement counts if the
op is in the LLVM dialect and `LLVM::ConstantOp::isBuildableWith`
accepts the attribute. Only in this case did the old overload create an
`llvm.mlir.constant`. One difference remains: the other results of the
op no longer have to materialize.
Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
[mlir] Apply partial folds in DialectConversion
`OperationLegalizer::legalizeWithFold` now uses the new
`OpBuilder::tryFold` overload and `materializeFoldResults`, and it
applies a partial fold when pattern rollback is off. The legalizer
cannot roll back the replacement of some results of an op that stays.
So with rollback on, the legalizer applies a fold only if it replaces
every result or changes the op in place, as before. Without rollback,
the legalizer replaces the uses of each replaced result and tries to
legalize the op again.
Without rollback, the legalizer materializes only the replaced results
that have uses, because it must legalize each new op. So it creates no
constant that nothing uses, and a result without uses can hold an
attribute that does not materialize.
In a graph region, the replacement skips the uses that come before the
replacement value. So the legalizer counts progress by the use counts
of the replaced results. A fold that replaces all results still erases
[7 lines not shown]
[mlir] Apply partial folds in OpBuilder::createOrFold
The multi-result `createOrFold` now uses the new `OpBuilder::tryFold`
overload and `materializeFoldResults`. When the fold replaces only some
results, the new op stays, and the results of `createOrFold` mix the
replacement values and the kept results of the op. Before this patch,
such a fold counted as an in-place fold or as a failure. The new op has
no uses yet, so `createOrFold` materializes every replaced result.
The zero-result `createOrFold` also uses the new overload. A fold of a
zero-result op can only change the op in place, so its behavior does
not change.
A new test pattern builds `test.op_partial_fold` with the multi-result
`createOrFold`. The new test in `test-operation-folder.mlir` checks
that the op stays next to the replacement values.
Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
[mlir] Add an OpBuilder::tryFold overload that returns OpFoldResults
`OpBuilder::tryFold(op, results, constants)` applies a fold only if the
fold replaces every result, so it cannot give a partial fold to the
caller. This patch adds two `OpBuilder` functions that let a caller
apply a partial fold:
- `OpFoldResults tryFold(Operation *op)` folds the op and returns the
fold result. Like the old overload, it skips constants and repeats a
fold that only changes the op in place. It creates no constant.
- `materializeFoldResults(op, foldResults, liveOnly)` returns one value
per result: the replacement of a replaced result, or null for a kept
result. It turns each attribute into a new constant at the insertion
point. If `liveOnly` is set, a replaced result without uses gets null
and no constant. If one constant fails to materialize, the function
inserts no constant and fails.
The old overload now uses the two new functions. A fold that replaces
only some results, which only the new fold form can return, counts as
an in-place fold if it changed the op in place, and as a failure
[7 lines not shown]
[TSan] Defer symbolization in AddStack and AddSleep to SymbolizeStackElems
PR #151495 delayed symbolization for memory accesses, locations,
threads, and mutexes until OutputReport (after ThreadRegistryLock is
released), but missed ScopedReport::AddStack and ScopedReport::AddSleep.
As a result, ReportRace (via AddSleep) and ReportMutexMisuse,
ReportDeadlock, and ReportDestroyLocked (via AddStack) still invoked the
symbolizer while holding ThreadRegistryLock.
Store the unsymbolized stack traces and sleep stack ID in ReportDesc and
symbolize them in ScopedReport::SymbolizeStackElems().
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228795
[TSan] Lock ScopedErrorReportLock before slot and thread_registry locks
OutputReport runs while ScopedErrorReportLock is held after slot_mtx and
thread_registry have been unlocked. Because code executed during
OutputReport (symbolizer, callbacks, or signal handlers) can acquire
slot_mtx or thread_registry, ScopedErrorReportLock must precede slot and
thread_registry locks in the lock hierarchy to avoid AB-BA deadlocks
between concurrent reports or fork().
- Move ScopedErrorReportLock::Lock() before slot.mtx, thread_registry,
and slot_mtx in ForkBefore (and unlock in reverse order in ForkAfter).
- Replace ctx->thread_registry.CheckLocked() in ScopedReportBase's
constructor with CheckedMutex::CheckNoLocks(), and add CheckLocked() to
AddThread(const ThreadContext *) and CheckNoLocks() to OutputReport.
- Construct ScopedReport before acquiring ThreadRegistryLock across all
reporting functions, and close the RestoreStack lock scope before
constructing ScopedReport in ReportRace.
Assisted-by: Gemini
[2 lines not shown]
[NFC][TSan] Allocate ScopedReport as a stack variable
Now that ScopedReport is constructed before acquiring ThreadRegistryLock
or slot locks across all reporting functions, it no longer needs to be
constructed inside the lock scope via placement new on __builtin_alloca
storage.
Declare ScopedReport as a normal stack variable before the lock scope
and remove the manual destructor calls.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228637
[NFC][TSan] Move ObtainCurrentStack out of ThreadRegistryLock scope
ObtainCurrentStack only reads the current thread's shadow stack and
allocates a VarSizeStackTrace buffer, which does not require
ThreadRegistryLock (and already runs outside ThreadRegistryLock in
ReportRace, SignalUnsafeCall, ReportErrnoSpoiling, and
ReportDestroyLocked).
Move ObtainCurrentStack (and dummy_pc in ReportDeadlock) before
ScopedReport in ReportMutexHeldWrongContext, ReportMutexMisuse, and
ReportDeadlock so the stack trace buffers also outlive ScopedReport and
OutputReport.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228794
[NFC][TSan] Move RestoreStack before ScopedReport in ReportDestroyLocked
Run RestoreStack in its own lock scope before constructing ScopedReport
in ReportDestroyLocked (matching ReportRace). This avoids acquiring
ScopedErrorReportLock or symbolizing the current stack if RestoreStack
fails, and avoids holding slot_lock and slot_mtx while populating the
ScopedReport.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228643
[NFC][TSan] Merge ScopedReportBase into ScopedReport
ScopedReport is the only subclass of ScopedReportBase and nothing uses
ScopedReportBase directly. Merge ScopedReportBase into ScopedReport.
Assisted-by: Gemini
Pull Request: https://github.com/llvm/llvm-project/pull/228640
