Skip to content

Make src_paths behave as expected when using --resolve-all-configs and improve performance - #2142

Open
sudowork wants to merge 3 commits into
PyCQA:mainfrom
sudowork:issue/2045
Open

sudowork wants to merge 3 commits into
PyCQA:mainfrom
sudowork:issue/2045

Conversation

@sudowork

@sudowork sudowork commented Jun 6, 2023

Copy link
Copy Markdown

When using --resolve-all-configs, there is unexpected behavior in that src_paths ends up resolving relative to the project root, which defaults to the current working directory. This results in first-party modules being marked as third-party modules in the default case.

Under the previous implementation, one possible workaround would be to specify the relative path to config directory (e.g. relative/path/to/configdir/src). However, assuming that the most common use of --resolve-all-configs is to support multiple sub-projects in the same repository/overall directory, this workaround would now require each sub-project to understand where it lives in the filesystem.

This change proposes a fix that sets directory on the config_data to be the directory containing the used configuration file if not already set. Downstream, this directory is then used to resolve the absolute paths specified by src_paths.

This change also introduces performance improvements to find_all_configs by pruning as we walk the filesystem and other smaller performance enhancements.

Fixes #2045

@sudowork

sudowork commented Jun 6, 2023

Copy link
Copy Markdown
Author

One thing of note: This would be backwards incompatible for anyone depending on the behavior of resolving first-party src_paths from the project root or config root. However, my assumption is that the behavior this change introduces is what users would expect given the types of config files we look for (.isort.cfg, pyproject.toml, setup.cfg, tox.ini, .editorconfig). The only odd one out in my opinion would be .editorconfig, which supports having a non-root configuration; however, given the behavior of the --resolve-all-configs flag to only choose the closest config file, I opted to not handle this case for consistency and to avoid further complicating the code.

This change also makes the behavior consistent with configuring a settings path.

@sudowork sudowork changed the title Make src_paths behave as expected when using --resolve-all-configs Make src_paths behave as expected when using --resolve-all-configs and improve performance Jun 7, 2023
Comment thread isort/settings.py Outdated
potential_config_file.path, CONFIG_SECTIONS[potential_config_file.name]
)
if "directory" not in config_data:
config_data["directory"] = os.path.dirname(potential_config_file.path)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the main logic to fix the src_paths resolution.

@sblask

sblask commented Nov 8, 2024

Copy link
Copy Markdown

I just ran into this problem. I have a mono repo with multiple python projects in sub folders that each have an .isort.cfg. When running isort from the root on a file in directory one, everything works as expected, same on a file in directory two. But when I run on a file from directory one and two at the same time (when using pre-commit), only one of the files is formatted correctly. I thought --resolve-all-configs would solve this, but quite the opposite, when using it, neither works. I'd expect the folder with the config file to be set as root for the files in them. Sounds like this PR would solve this?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes --resolve-all-configs behavior so src_paths resolves relative to the directory containing the config file selected for a given file (instead of the process CWD), addressing misclassification of first-party vs third-party imports (Fixes #2045). It also refactors config discovery to improve performance when scanning large trees.

Changes:

  • Update find_all_configs to scan via os.scandir (with default-skip pruning) and populate config_data["directory"] from the config file’s directory when unset.
  • Extend unit tests for find_all_configs to validate expected src_paths resolution and .venv skipping.
  • Update documentation to clarify src_paths resolution semantics under --resolve-all-configs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
isort/settings.py Refactors config discovery and injects directory into config data for correct src_paths resolution under --resolve-all-configs.
tests/unit/test_settings.py Updates find_all_configs tests to validate src_paths resolution and skip-directory behavior.
docs/configuration/options.md Clarifies --resolve-all-configs behavior for src_paths resolution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread isort/settings.py Outdated
Comment thread isort/settings.py
Comment thread tests/unit/test_settings.py
@DanielNoord

Copy link
Copy Markdown
Member

This needs a rebase unfortunately. Since I can't use Copilot on another repository I can't use it to fix the PR. Unfortunately I'll close it for now. Sorry! We should have gotten to this earlier, but maintenance was a bit low in previous years.

@sudowork

sudowork commented Aug 11, 2026

Copy link
Copy Markdown
Author

@DanielNoord I can go ahead and rebase this if you re-open. Or alternatively, I can just open a new PR.

When using `--resolve-all-configs`, there is unexpected behavior in that
`src_paths` ends up resolving relative to the project root, which
defaults to the current working directory. This results in first-party
modules being marked as third-party modules in the default case.

Under the previous implementation, one possible workaround would be to
specify the relative path to config directory (e.g.
`relative/path/to/configdir/src`). However, assuming that the most
common use of `--resolve-all-configs` is to support multiple
sub-projects in the same repository/overall directory, this workaround
would now require each sub-project to understand where it lives in the
filesystem.

This change proposes a fix that sets `directory` on the `config_data` to
be the directory containing the used configuration file if not already
set. Downstream, this directory is then used to resolve the absolute
paths specified by `src_paths`.

Fixes PyCQA#2045
@DanielNoord DanielNoord reopened this Aug 11, 2026
Avoid recursing into default skip files like .venv. Also avoid doing an
iteration per config file type. Lastly, use scandir to improve file system walk
performance.

The previous implementation would walk the entire tree, and iterate over
each config source type to test if the file exists. Instead, walk over
existing files and do a fast filter to remove candidates.
- Preserve CONFIG_SOURCES precedence when a directory contains multiple
  config files. os.scandir returns entries in an arbitrary order, so the
  scandir-based rewrite could pick a lower-precedence config (e.g. setup.cfg
  over .isort.cfg); restore the deterministic behavior of the previous
  os.walk implementation.
- Do not recurse into symlinked directories in _scanwalk_files, matching
  os.walk's followlinks=False default, to avoid infinite loops on cyclic
  symlinks.
- Skip directories/entries that raise OSError (permissions, broken links)
  instead of crashing.
- Add regression tests for config-source precedence, symlink-loop safety,
  and ignoring config files without a valid isort section.
@sudowork

sudowork commented Aug 11, 2026

Copy link
Copy Markdown
Author

@DanielNoord rebased and also addressed Copilot's feedback + fixed a edge case where config priority wasn't being respected.

Performance

Benchmarked the current implementation vs the previous os.walk version on identical trees (find_all_configs, warmup + timed runs, median):

Tree os.walk (old) os.scandir (new) Speedup
Synthetic, with .venv/node_modules/build/dist (518 dirs / 8.1K files) 31.5 ms 13.0 ms ~2.4×
Synthetic, no skip dirs (500 dirs / 8K files) 42.6 ms 40.8 ms ~1.07×
Large real-world monorepo 8.75 s 0.36 s ~24×

Most of the benefit comes from pruning DEFAULT_SKIP.

@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.17949% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.24%. Comparing base (fad1413) to head (7cd7cfa).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2142      +/-   ##
==========================================
- Coverage   99.39%   99.24%   -0.16%     
==========================================
  Files          41       41              
  Lines        3156     3182      +26     
  Branches      682      688       +6     
==========================================
+ Hits         3137     3158      +21     
- Misses         11       15       +4     
- Partials        8        9       +1     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

isort/settings.py:792

  • This hard-coded pruning drops configs for explicitly passed files. Explicit CLI paths are still processed unless --filter-files is set (isort/main.py:303-308), so isort .venv/foo.py --resolve-all-configs now sorts foo.py with the default/ancestor config instead of .venv/pyproject.toml; the previous os.walk implementation found that config. Pruning needs to be conditional on the actual file-filtering behavior/targets rather than unconditional DEFAULT_SKIP.
    for potential_config_file in _scanwalk_files(
        path, exclude_fn=lambda entry: entry.name in DEFAULT_SKIP
    ):

isort/settings.py:804

  • Because os.scandir order is arbitrary, this can parse a lower-priority config before the higher-priority file is encountered. If that lower-priority file is malformed, isort emits a warning (and pays the parse cost) even though a valid .isort.cfg should make it irrelevant; the previous CONFIG_SOURCES loop never opened lower-priority files after finding a valid higher-priority config. Group candidates per directory and try them in CONFIG_SOURCES order, stopping at the first valid config.
        try:
            config_data = _get_config_data(
                potential_config_file.path, CONFIG_SECTIONS[potential_config_file.name]
            )

@DanielNoord

Copy link
Copy Markdown
Member

@sudowork I think the review from Copilot correctly hints to the fact that the speed optimization might better fit a separate PR as there is a lot of "gotcha's". Can we split this into a PR that does the fix and a PR that does the speed improvement?

Manny7717

This comment was marked as low quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inconsistencies of known_first_party and --resolve-all-configs argument

5 participants