Skip to content

fix(semantic-consistency): keep a usable reason when the model omits one #635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ydflow wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(semantic-consistency): keep a usable reason when the model omits one #635

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion ...t/src/main/java/com/alibaba/cloud/ai/dataagent/workflow/node/SemanticConsistencyNode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import com.alibaba.cloud.ai.graph.streaming.StreamingOutput;
import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.converter.BeanOutputConverter;
import org.springframework.stereotype.Component;
Expand Down Expand Up @@ -109,14 +110,20 @@ private Map<String, Object> buildStructuralValidationFailure(OverAllState state,

/**
* Build validation result
*
* <p>
* The reason is rendered into the {@code sql-error-fixer} prompt on the retry path, so a
* null reason would leave the model with an empty "错误信息" section. Fall back to a
* generic notice when the model reports a failure without explaining it.
*/
private Map<String, Object> buildValidationResult(boolean passed, String validationResult) {
if (passed) {
return Map.of(SEMANTIC_CONSISTENCY_NODE_OUTPUT, true);
}
else {
return Map.of(SEMANTIC_CONSISTENCY_NODE_OUTPUT, false, SQL_REGENERATE_REASON,
SqlRetryDto.semantic(validationResult));
SqlRetryDto.semantic(StringUtils.isNotBlank(validationResult) ? validationResult
: "语义一致性校验未通过，模型未给出具体原因"));
}
}

Expand Down
18 changes: 18 additions & 0 deletions ...c/test/java/com/alibaba/cloud/ai/dataagent/workflow/node/SemanticConsistencyNodeTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,24 @@ void apply_invalidSql_returnsGeneratorWithFailOutput() throws Exception {
assertEquals(new SqlRetryDto("表不存在", true, false), retry);
}

@Test
void apply_failedWithoutReason_doesNotStoreNullReason() throws Exception {
OverAllState state = createTestState();
setupBasicState(state, "SELECT * FROM nonexistent");

// LLM may omit "reason" entirely when it only reports the verdict
when(nl2SqlService.performSemanticConsistency(any(SemanticConsistencyDTO.class)))
.thenReturn(Flux.just(ChatResponseUtil.createPureResponse("{\"passed\":false}")));

NodeExecution execution = execute(semanticConsistencyNode.apply(state), SEMANTIC_CONSISTENCY_NODE_OUTPUT);

assertEquals(false, execution.finalResult().get(SEMANTIC_CONSISTENCY_NODE_OUTPUT));
SqlRetryDto retry = (SqlRetryDto) execution.finalResult().get(SQL_REGENERATE_REASON);
assertNotNull(retry, "retry reason must still be produced so the workflow loops back");
assertTrue(retry.semanticFail());
assertNotNull(retry.reason(), "reason must not be null; it is rendered into the repair prompt");
}

@Test
void apply_missingEvidence_throwsException() {
OverAllState state = createTestState();
Expand Down
Loading