[SelectionDAG] Preserve CTTZ_ELTS lane count during expansion - #217982
[SelectionDAG] Preserve CTTZ_ELTS lane count during expansion#217982Opriego wants to merge 1 commit into
Conversation
CTTZ_ELTS expansion derived VL from its legalized auxiliary step vector. On X86 with AVX512F, the step vector for a semantic v4i1 mask is widened from v4i8 to v16i8, causing an all-zero mask to return 16 instead of 4. Derive VL from the CTTZ_ELTS operand's ElementCount instead. Auxiliary type legalization can still widen the step vector without changing the logical lane domain of the operation.
|
Hello @Opriego 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
I confirm that I've read these policies |
llvmorg-github-actions
Bot
added
backend:X86
llvm:SelectionDAG
|
@llvm/pr-subscribers-backend-x86 @llvm/pr-subscribers-llvm-selectiondag Author: Oscar Priego (Opriego) ChangesFixes #216649.
On X86 with AVX512F, the step vector for a semantic Preserve the Add AVX512F regression coverage for the affected Full diff: https://github.com/llvm/llvm-project/pull/217982.diff 2 Files Affected:
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index cca7cdad0e2c8..081d38dc136c0 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -13859,6 +13859,8 @@ SDValue TargetLowering::expandVECTOR_COMPRESS(SDNode *Node,
SDValue TargetLowering::expandCttzElts(SDNode *Node, SelectionDAG &DAG) const {
SDLoc DL(Node);
EVT VT = Node->getValueType(0);
+ ElementCount EC =
+ Node->getOperand(0).getValueType().getVectorElementCount();
bool ZeroIsPoison = Node->getOpcode() == ISD::CTTZ_ELTS_ZERO_POISON;
auto [Mask, StepVec] =
@@ -13893,8 +13895,7 @@ SDValue TargetLowering::expandCttzElts(SDNode *Node, SelectionDAG &DAG) const {
if (getTypeAction(StepVT.getSimpleVT()) == TypePromoteInteger)
StepVT = getTypeToTransformTo(*DAG.getContext(), StepVT);
- SDValue VL =
- DAG.getElementCount(DL, StepVT, StepVecVT.getVectorElementCount());
+ SDValue VL = DAG.getElementCount(DL, StepVT, EC);
SDValue SplatVL = DAG.getSplat(StepVecVT, DL, VL);
StepVec = DAG.getNode(ISD::SUB, DL, StepVecVT, SplatVL, StepVec);
SDValue Zeroes = DAG.getConstant(0, DL, StepVecVT);
diff --git a/llvm/test/CodeGen/X86/intrinsic-cttz-elts.ll b/llvm/test/CodeGen/X86/intrinsic-cttz-elts.ll
index 65231c484db98..a0b494a6303dc 100644
--- a/llvm/test/CodeGen/X86/intrinsic-cttz-elts.ll
+++ b/llvm/test/CodeGen/X86/intrinsic-cttz-elts.ll
@@ -1,4 +1,5 @@
; RUN: llc -mtriple=x86_64-unknown-unknown < %s | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=+avx512f < %s | FileCheck %s --check-prefix=AVX512
define i8 @ctz_v8i16(<8 x i16> %a) {
; CHECK-LABEL: .LCPI0_0:
@@ -101,5 +102,21 @@ define i8 @ctz_v8i16_poison(<8 x i16> %a) {
ret i8 %res
}
+define i32 @ctz_zero_v4i1() {
+; AVX512-LABEL: ctz_zero_v4i1:
+; AVX512: addb $5, %al
+ %res = call i32 @llvm.experimental.cttz.elts.i32.v4i1(<4 x i1> zeroinitializer, i1 false)
+ ret i32 %res
+}
+
+define i32 @ctz_zero_v8i1() {
+; AVX512-LABEL: ctz_zero_v8i1:
+; AVX512: addb $9, %al
+ %res = call i32 @llvm.experimental.cttz.elts.i32.v8i1(<8 x i1> zeroinitializer, i1 false)
+ ret i32 %res
+}
+
declare i8 @llvm.experimental.cttz.elts.i8.v8i16(<8 x i16>, i1)
declare i16 @llvm.experimental.cttz.elts.i16.v4i32(<4 x i32>, i1)
+declare i32 @llvm.experimental.cttz.elts.i32.v4i1(<4 x i1>, i1)
+declare i32 @llvm.experimental.cttz.elts.i32.v8i1(<8 x i1>, i1)
|
1 participant
Fixes #216649.
expandCttzEltsderivesVLfrom its legalized auxiliary step vector. When that helper vector is widened for target legality, its lane count may differ from the logical lane count of theCTTZ_ELTSoperand.On X86 with AVX512F, the step vector for a semantic
<4 x i1>mask is widened fromv4i8tov16i8. As a result, an all-zerollvm.experimental.cttz.eltsinput returns 16 instead of the required result of 4.Preserve the
ElementCountfrom theCTTZ_ELTSoperand and use it when materializingVL. Auxiliary step-vector legalization can still widen its representation without changing the logical lane domain of the operation.Add AVX512F regression coverage for the affected
v4i1andv8i1cases.