feat: implement intelligence gate for API veracity (de-hallucination)#37
Open
LarytheLord wants to merge 2 commits intomainfrom
Open
feat: implement intelligence gate for API veracity (de-hallucination)#37LarytheLord wants to merge 2 commits intomainfrom
LarytheLord wants to merge 2 commits intomainfrom
Conversation
… support - Track import aliases (import x as y, from x import y) for accurate module resolution during hallucination checks - Expand safe module list from 6 to 35+ stdlib modules - Handle from-import and aliased-import patterns in AST verification - Add 3 new test cases covering import-as, from-import, and from-import-as - Sync docs/SKILL.md with veracity documentation and bump version to 7
📝 WalkthroughWalkthroughAdds a veracity verification campaign to intercept and reject hallucinated API suggestions during import. Introduces a ChangesVeracity Verification Campaign
Sequence DiagramsequenceDiagram
participant CLI as CLI Parser
participant Config as Review Config
participant Holistic as holistic.py
participant Flow as holistic_issue_flow.py
participant Registry as Language Registry
participant Plugin as VeracityPlugin
participant AST as AST Parser
CLI->>Config: --verify-veracity flag
Config->>Holistic: ReviewImportConfig(verify_veracity=True)
Holistic->>Flow: verify_veracity=True
loop For each issue
Flow->>Registry: get_lang_config()
Registry->>Plugin: Load veracity_plugin
alt Has suggestion
Flow->>Plugin: verify_suggestion(suggestion)
Plugin->>AST: parse Python code blocks
AST->>Plugin: AST tree
Plugin->>Plugin: resolve imports & attributes
Plugin->>Plugin: importlib.import_module()
Plugin->>Flow: VeracityIssue[] or []
end
alt Verification errors
Flow->>Flow: Skip issue, add to rejected
else No errors
Flow->>Flow: Import issue normally
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 7 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (7 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@desloppify/app/commands/review/cmd.py`:
- Around line 194-195: The --verify-veracity flag is being set but never
forwarded to the import path; update the import flow to accept and forward this
kwarg: add verify_veracity=opts.verify_veracity when calling
import_holistic_issues in desloppify/app/commands/review/importing/cmd.py (and
the other call around the 211–212 region), and modify the wrapper in
desloppify/intelligence/review/__init__.py to accept a verify_veracity parameter
and pass it through to the underlying import function so the flag reaches
import_holistic_issues.
In `@desloppify/languages/python/veracity.py`:
- Around line 109-147: The two broad "except Exception: pass" blocks in
veracity.py (around the import/attribute resolution logic that checks
module_path, parent_path, and obj attributes) must not swallow unexpected
errors; replace them with specific handlers and surface verifier failures as
veracity issues: keep the existing ImportError handling, explicitly catch
AttributeError or known expected exceptions, and for any other Exception create
and return a veracity issue dict (same shape as the existing hallucination
return: keys "method","module","message","code_block") containing the exception
message and context (e.g., f"Verifier error while resolving {module_path}:
{exc}") so the failure is reported instead of silently accepting the suggestion;
ensure these changes reference the same logic around module import/parent_module
and obj attribute checks in the code paths using module_path, parent_path,
actual_attr, and attr_name.
- Around line 46-48: The verifier only extracts fenced blocks labeled "python"
and only accepts attribute calls whose base is ast.Name, so suggestions like
unlabeled/fenced diff/suggestion blocks and calls like Path("x").missing() are
missed; update _extract_python_blocks to also match unlabeled/fenced blocks and
common suggestion shapes (e.g., ```diff```, ```suggestion```, and plain
```...``` blocks) so Python code in those blocks is captured, and update
_verify_attribute_call to unwrap call/attribute chains: when checking the base
accept ast.Attribute and ast.Call by peeling ast.Call to its func and resolving
nested ast.Attribute/ast.Name (e.g., recognize func names like "Path" or
attribute chains like "pathlib.Path") before validating the attribute access, so
both Path("...").foo and pathlib.Path("...").foo forms pass verification.
In `@ISSUE_DEHALLUCINATION.md`:
- Around line 3-23: The markdown headings in ISSUE_DEHALLUCINATION.md need a
blank line after them to satisfy MD022; insert an empty line immediately after
the headings "## Goal", "## Context", "## Specification", and "## Definition of
Done" so that each heading is followed by a single blank line and the file
passes the linter.
- Line 8: Update the documentation reference so it points to the actual module
path used in this campaign: replace the incorrect
`intelligence/review/importing/holistic.py` with
`desloppify/intelligence/review/importing/holistic.py` wherever the spec
mentions the import-flow implementation (notably the `import_holistic_issues`
implementation), ensuring any follow-up investigations reference the real file
location.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8b137331-fe65-4405-8c03-428b8acc5ea8
📒 Files selected for processing (15)
ISSUE_DEHALLUCINATION.mddesloppify/app/cli_support/parser_groups_admin_review_options_core.pydesloppify/app/commands/review/cmd.pydesloppify/app/commands/review/importing/flags.pydesloppify/app/commands/review/importing/parse.pydesloppify/app/skill_docs.pydesloppify/data/global/SKILL.mddesloppify/intelligence/review/importing/holistic.pydesloppify/intelligence/review/importing/holistic_issue_flow.pydesloppify/intelligence/review/veracity.pydesloppify/languages/_framework/base/types.pydesloppify/languages/python/__init__.pydesloppify/languages/python/tests/test_py_veracity.pydesloppify/languages/python/veracity.pydocs/SKILL.md
Comment on lines
+194
to
195
| verify_veracity=opts.verify_veracity, | ||
| ), |
There was a problem hiding this comment.
--verify-veracity still does not reach the import campaign.
These fields are populated here, but desloppify/app/commands/review/importing/cmd.py:326-330 still calls import_holistic_issues(...) without verify_veracity, and desloppify/intelligence/review/__init__.py:90-109 drops the kwarg in its wrapper. As shipped, desloppify review --import --verify-veracity behaves the same as without the flag.
Also applies to: 211-212
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@desloppify/app/commands/review/cmd.py` around lines 194 - 195, The
--verify-veracity flag is being set but never forwarded to the import path;
update the import flow to accept and forward this kwarg: add
verify_veracity=opts.verify_veracity when calling import_holistic_issues in
desloppify/app/commands/review/importing/cmd.py (and the other call around the
211–212 region), and modify the wrapper in
desloppify/intelligence/review/__init__.py to accept a verify_veracity parameter
and pass it through to the underlying import function so the flag reaches
import_holistic_issues.
Comment on lines
+46
to
+48
| def _extract_python_blocks(self, text: str) -> list[str]: | ||
| """Extract code from ```python ... ``` blocks.""" | ||
| return re.findall(r"```python\s+(.*?)```", text, re.DOTALL) |
There was a problem hiding this comment.
The verifier misses common suggestion shapes, so the gate is easy to bypass.
_extract_python_blocks() only scans fenced python blocks, so diff or unlabeled suggestion blocks are ignored, and _verify_attribute_call() bails unless the base resolves to ast.Name, which misses common forms like pathlib.Path("x").missing() / Path("x").missing(). That leaves real imported fixes outside the campaign’s coverage.
Also applies to: 77-92
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@desloppify/languages/python/veracity.py` around lines 46 - 48, The verifier
only extracts fenced blocks labeled "python" and only accepts attribute calls
whose base is ast.Name, so suggestions like unlabeled/fenced diff/suggestion
blocks and calls like Path("x").missing() are missed; update
_extract_python_blocks to also match unlabeled/fenced blocks and common
suggestion shapes (e.g., ```diff```, ```suggestion```, and plain ```...```
blocks) so Python code in those blocks is captured, and update
_verify_attribute_call to unwrap call/attribute chains: when checking the base
accept ast.Attribute and ast.Call by peeling ast.Call to its func and resolving
nested ast.Attribute/ast.Name (e.g., recognize func names like "Path" or
attribute chains like "pathlib.Path") before validating the attribute access, so
both Path("...").foo and pathlib.Path("...").foo forms pass verification.
Comment on lines
+109
to
+147
| try: | ||
| # We check if it's in our safe list OR if it's already in sys.modules | ||
| # (which means it's already loaded in this environment) | ||
| import sys | ||
| if root_package in self.SAFE_MODULES or root_package in sys.modules: | ||
| # Try to import the specific module path | ||
| try: | ||
| module = importlib.import_module(module_path) | ||
| if not hasattr(module, attr_name): | ||
| return { | ||
| "method": attr_name, | ||
| "module": module_path, | ||
| "message": f"Hallucinated API detected: '{module_path}.{attr_name}' does not exist.", | ||
| "code_block": block | ||
| } | ||
| except ImportError: | ||
| # If we can't import the submodule, it might be a method call | ||
| # on an object, which we don't handle well yet. | ||
| # e.g. os.path.join().exists() | ||
| # In that case, we try to import the parent and see if it has the attribute. | ||
| parent_path = ".".join(module_path.split(".")[:-1]) | ||
| if parent_path: | ||
| try: | ||
| parent_module = importlib.import_module(parent_path) | ||
| actual_attr = module_path.split(".")[-1] | ||
| if hasattr(parent_module, actual_attr): | ||
| # The 'module_path' was actually parent.attr | ||
| obj = getattr(parent_module, actual_attr) | ||
| if not hasattr(obj, attr_name): | ||
| return { | ||
| "method": attr_name, | ||
| "module": module_path, | ||
| "message": f"Hallucinated API detected: '{module_path}.{attr_name}' does not exist.", | ||
| "code_block": block | ||
| } | ||
| except Exception: | ||
| pass | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
Do not fail open on verifier exceptions.
Both except Exception: pass blocks turn verifier failures into silent accepts. For a de-hallucination gate, unexpected resolver errors should surface or produce a veracity issue instead of letting the suggestion through.
As per coding guidelines, "Use specific exception handling, never catch-all blocks; AI suppresses errors and removes safety checks".
🧰 Tools
🪛 Ruff (0.15.12)
[error] 144-145: try-except-pass detected, consider logging the exception
(S110)
[warning] 144-144: Do not catch blind exception: Exception
(BLE001)
[error] 146-147: try-except-pass detected, consider logging the exception
(S110)
[warning] 146-146: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@desloppify/languages/python/veracity.py` around lines 109 - 147, The two
broad "except Exception: pass" blocks in veracity.py (around the
import/attribute resolution logic that checks module_path, parent_path, and obj
attributes) must not swallow unexpected errors; replace them with specific
handlers and surface verifier failures as veracity issues: keep the existing
ImportError handling, explicitly catch AttributeError or known expected
exceptions, and for any other Exception create and return a veracity issue dict
(same shape as the existing hallucination return: keys
"method","module","message","code_block") containing the exception message and
context (e.g., f"Verifier error while resolving {module_path}: {exc}") so the
failure is reported instead of silently accepting the suggestion; ensure these
changes reference the same logic around module import/parent_module and obj
attribute checks in the code paths using module_path, parent_path, actual_attr,
and attr_name.
Comment on lines
+3
to
+23
| ## Goal | ||
| Prevent AI agents from proposing "slop" fixes that utilize hallucinated library methods or deprecated APIs. This is a common failure mode where agents invent methods that "should" exist but do not. | ||
|
|
||
| ## Context | ||
| - **Repository:** `desloppify` | ||
| - **Location of Logic:** `intelligence/review/importing/holistic.py` (specifically `import_holistic_issues`). | ||
| - **Target Language (Phase 1):** Python. | ||
|
|
||
| ## Specification | ||
| 1. **Detection:** Intercept incoming `ReviewIssuePayload` during the import process. | ||
| 2. **Extraction:** Identify code blocks within the `suggestion` field. | ||
| 3. **Verification (Python):** | ||
| * Extract imported modules and method calls from the suggested code. | ||
| * Verify these calls against the local project environment (e.g., `sys.modules`, `pkg_resources`, or by inspecting the AST of installed packages). | ||
| * Reuse logic from `desloppify/languages/python/detectors/deps_resolution.py` if applicable. | ||
| 4. **Feedback:** If a hallucinated API is detected: | ||
| * Reject the specific issue. | ||
| * Return a `VerificationIssue` to the agent with a clear message: `"Hallucinated API detected: [method_name]. Please verify against the actual library structure and refactor."` | ||
| 5. **Configuration:** Allow this check to be toggled via a new flag `--verify-veracity`. | ||
|
|
||
| ## Definition of Done |
There was a problem hiding this comment.
Fix heading spacing (MD022) to keep the spec lint-clean.
Add a blank line after headings at Line 3, Line 6, Line 11, and Line 23.
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 3-3: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 6-6: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 11-11: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 23-23: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@ISSUE_DEHALLUCINATION.md` around lines 3 - 23, The markdown headings in
ISSUE_DEHALLUCINATION.md need a blank line after them to satisfy MD022; insert
an empty line immediately after the headings "## Goal", "## Context", "##
Specification", and "## Definition of Done" so that each heading is followed by
a single blank line and the file passes the linter.
|
|
||
| ## Context | ||
| - **Repository:** `desloppify` | ||
| - **Location of Logic:** `intelligence/review/importing/holistic.py` (specifically `import_holistic_issues`). |
There was a problem hiding this comment.
Correct the spec path for the import-flow implementation.
Line 8 references intelligence/review/importing/holistic.py, but the implemented path in this campaign is desloppify/intelligence/review/importing/holistic.py. This can send follow-up investigation to the wrong location.
Suggested doc fix
-- **Location of Logic:** `intelligence/review/importing/holistic.py` (specifically `import_holistic_issues`).
+- **Location of Logic:** `desloppify/intelligence/review/importing/holistic.py` (specifically `import_holistic_issues`).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - **Location of Logic:** `intelligence/review/importing/holistic.py` (specifically `import_holistic_issues`). | |
| - **Location of Logic:** `desloppify/intelligence/review/importing/holistic.py` (specifically `import_holistic_issues`). |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@ISSUE_DEHALLUCINATION.md` at line 8, Update the documentation reference so it
points to the actual module path used in this campaign: replace the incorrect
`intelligence/review/importing/holistic.py` with
`desloppify/intelligence/review/importing/holistic.py` wherever the spec
mentions the import-flow implementation (notably the `import_holistic_issues`
implementation), ensuring any follow-up investigations reference the real file
location.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--verify-veracityflag todesloppify review --importto toggle the checkimport as,from import, andfrom import aspatternsArchitecture
The feature adds a
VeracityPluginabstract interface (desloppify/intelligence/review/veracity.py) that language configs can implement. The Python plugin (desloppify/languages/python/veracity.py) tracks imports/aliases via AST and verifies attribute accesses against the local environment.Integration points:
desloppify/app/cli_support/parser_groups_admin_review_options_core.py— CLI flagdesloppify/app/commands/review/cmd.py— flag wiring through review optionsdesloppify/app/commands/review/importing/flags.py— import configdesloppify/app/commands/review/importing/parse.py— parse optionsdesloppify/intelligence/review/importing/holistic.py— passes flag to validationdesloppify/intelligence/review/importing/holistic_issue_flow.py— runs veracity check, rejects hallucinated issuesdesloppify/languages/python/__init__.py— wiresPythonVeracityPlugininto Python configdesloppify/languages/_framework/base/types.py— addsveracity_pluginfield toLangConfigdesloppify/data/global/SKILL.md+docs/SKILL.md— version 7, veracity documentationTest Results
5530 passed, 152 skipped
Related
ISSUE_DEHALLUCINATION.mdFUTURE_IDEAS.md(Cognitive Load / De-hallucination)Summary by CodeRabbit
New Features
--verify-veracityCLI flag to validate suggested fixes against actual Python APIs and reject hallucinated methods.Documentation
--verify-veracityfor import operations to catch hallucinated API suggestions.