Constant reanalysing of entire project in VSCode with diagnosticMode workspace

[lukaspiatkowski]

I am not sure how much of this falls into Pylance vs Pyright, so sorry if this is the wrong place to ask.

I am working on migrating a relatively large monorepo (Pylance reports 5.5k source files found, with some bash I estimated it to be 1.5M lines; venv is 14.5k files and 4.4M lines) from using mypy to pyright. One important feature for me is being able to see all errors in the workspace, so that I can focus on a single module, check all the errors on it, fix them, mark module as done and go to the next one. For that I've set the "python.analysis.diagnosticMode" to "workspace". Since I also have "python.analysis.userFileIndexingLimit": 8000 (also tried with -1) I have hoped that after initial indexing if I change only one thing in a single file the analysis time won't be very horrible, but it does take noticable amount of time to update inlay hints or show description of a new symbol when hovering. It gets much worse when you want to change several things, then the editor feels like it starts getting clogged on the amount of processing it has to perform.

I've decided to make a test: set log levels to tracing, open a file that I've though wouldn't be imported in too many places, wait for the log lines to finnish logging (entire codebase was analised-parsed-binded-checked), cleared the output and added a space in a comment in that file. The logs exploded again, I've waited until they are done, saved the logs, redacted them and here they are:
https://gist.github.com/lukaspiatkowski/6098003d2c3c4f68a71995528748490e

Some pieces of my configs that might be relevant:

  "python.analysis.userFileIndexingLimit": 8000,
  "python.analysis.inlayHints.functionReturnTypes": true,
  "python.analysis.inlayHints.pytestParameters": true,
  "python.analysis.inlayHints.variableTypes": true,
  "python.analysis.autoImportCompletions": true,
  "python.analysis.nodeExecutable": "/opt/homebrew/bin/node",
  "python.analysis.autoFormatStrings": true,
  "python.analysis.diagnosticMode": "workspace",
  "python.analysis.logLevel": "Trace",

pyrightconfig.json:

{
  "include": ["b92", "ca1.py", "s0e", "uda", "../w2a"],
  "venvPath": "./",
  "venv": ".venv",
  "exclude": [
    ".venv",
    "semgrep",
    "**/__pycache__"
  ],
  "pythonVersion": "3.11",
  "executionEnvironments": [
    {
      "root": "s0e",
      "extraPaths": [
        "../w2a"
      ]
    },
    {
      "root": ".",
      "extraPaths": [
        "s0e",
        "../w2a"
      ]
    },
    {
      "root": "../w2a"
    }
  ],
  "reportPrivateUsage": false,
  "reportPrivateImportUsage": "information",
  "reportUnusedCallResult": false,
  "enableTypeIgnoreComments": true,
  "reportIgnoreCommentWithoutRule": true
}

Things to also point out:

• There is one very bad long file (/Users/lukas/m5a/ma1/s0e/tdd.py) that often causes Long operation warnings which is a ~5k line long list of all available actions and it imports a very substantial portion of the codebase from various places.
• I've tried running pyright -w <submodule>, but after printing Watching for file changes... it never reacts to my changes. - EDIT: I just didn't know that pyright -w outputs all the type errors on every file change and didn't notice the change
• I am using pyright 1.1.394 from command line and pylance 2025.2.101 in VSCode 1.97.2 running M1 MacBook Pro with 16GB RAM.

So my questions are:

• Is it expected that with workspace diagnosticMode the entire codebase is reanalysed on every change?
• Is it better to work with diagnosticMode set to openFilesOnly and open hundreds of files at once (a single module) or some other workaround?
• Is there something I can do to improve the performance of pyright for my repo?

P.S.
While writing this message I've tried my line of testing (opening a single file, adding a space to a comment and watching logs explode) few times and after one of the attempts I've got harrased by MacOS with message: "Your system has run out of application memory" and has asked me to force quit some apps. VSCode was reported to use 42GB, so I've killed it. I have 16GB of RAM and 39GB of Swap. After starting VSCode again I see in htop that Swap is almost completely used, but RAM is around 5GB. I've tried to bleed pyright by changing few things and I see CPU maxed, but memory is still around 5.5GB max... So not sure what was the issue with MacOS or what they mean by "application memory" exactly.

[erictraut]

Based on the symptoms you're reporting, it sounds like pyright is running short on heap space and is discarding all of its internal cache (including cached file contents, cached diagnostics, and cached type information) to avoid crashing and to make forward progress. There will naturally be some project size beyond which a type checker will run out of memory, and it sounds like you've gone past that.

If you're able to keep the entire working set in memory (which is easy for a more typical project), then you shouldn't see all files reanalyzed after a change. However, if the internal memory caches are being purged repeatedly, new edits will force pyright to do much more repeated work.

You can verify my theory by enabling "verboseOutput": true in the pyright config. This will produce voluminous log information, among which you will see memory usage stats if heap exhaustion is close to occurring.

Since your computer has 16 GB, you could try running pyright or pylance with a larger heap size. Then again, based on the last paragraph you appended to your post above, you might not have sufficient memory to do this.

The memory usage for the command-line version of pyright should be much lower than the language server version, especially if you're setting diagnosticMode to "workspace". And the language server version of pyright should also take up somewhat less memory than pylance, since the more advanced features of pylance (such as indexing) consume additional memory.

[lukaspiatkowski]

That does make sense, from logs I see:

(...) Heap stats: total_memory_size=16384MB, total_free_size=91MB, total_heap_size=6267MB, used_heap_size=6178MB, cross_worker_used_heap_size=6178MB, total_physical_size=6267MB, total_available_size=2027MB, heap_size_limit=8240MB
...
(...) Heap stats: total_memory_size=16384MB, total_free_size=79MB, total_heap_size=5696MB, used_heap_size=5284MB, cross_worker_used_heap_size=5284MB, total_physical_size=5696MB, total_available_size=2923MB, heap_size_limit=8240MB

So between those two logs the total_heap_size might have went too close to heap_size_limit. I will try to find a way to increase the memory limit, but IIRC it was doable only on remote vscode server.

In the meantime I've tried the following:

• Setting"python.analysis.indexing": false, but that didn't seem to change much
• Setting "verboseOutput": true in pyrightconfig.json, but the logs from VSCode were the same (maybe the extra logs are stored somewhere else? I still had "python.analysis.logLevel": "Trace" set.)
• I have tried the ms-pyright.pyright extension on its own and it gives a much more snappier response for type changes, even though it still suffers from re-analysing everything on every modification and obsiously means all the Pylance features are off... Some questions for this one:
  • The "python.analysis.nodeExecutable" option is not honored and it runs with --max-old-space-size=3072 according to htop. Is there a way to increase that? I want to see if there is a number that would result in pyright not reanalysing everything and how far are we from that number.
  • More out of curiosity: does Pylance rely on the information provided by Pyright in a way that "Pylance with Pyright" gives you more features than "Pylance without Pyright" and "Pyright" run separately? I see that Pylance turns off Pyright extension, but if they could both work together that could help with memory management for large repositories.
• BTW, pyright -w <submodule> works nicely, I just didn't realize that it outputs all errors after every file change and not only changes for the modified file.

I also wanted to ask, what would be the recommended approach to work with Pyright on a large codebase? I am mostly worried about a use case when an engineer starts to work on a feature branch, pivots for a bit to another and then goes back to his work. How can they see all the type errors that their feature branch introduces if they have to work with "python.analysis.diagnosticMode": "openFilesOnly"?

• Should they just open all the changed files? (I guess I can provide a script that opens all files like that).
• Should they run pyright -w <subdirectory> (assuming their modifications are limited to one or small amount of directories) and open the files that show errors in VSCode? (This seems wasteful as now you have 2 pyrights and both consume memory).
• Maybe I can provide a VSCode task that runs pyright on entire codebase and shows the results in VSCode on top of what LSP provides? But then if you fix the problem that initial pyright output would no disappear. Also running pyright from scrach takes a lot of time for us (3-6min)

Given that pyright strives to be high performant and work on large codebase I would be surprised that the above workflow isn't supported, so maybe I am thinking about this in a wrong way? I would hate to replace mypy with pyright just so that engineers start to be bounced back by CI checks more often, because they cannot see all the type issues before submitting changes. With dmypy the above works OKish, although you have to wait few seconds for the VSCode mypy extension to give you back results after saving the file.

[lukaspiatkowski]

And here is a repro: https://github.com/lukaspiatkowski/repro-pyright-9962

[erictraut]

Thanks for the repro.

This appears to be caused by a change that was introduced by the pylance team about five months ago here. Because it came in as part of a bulk push from the closed-source pylance repo, I can't tell who made the change, which PR it was part of, or which bug it was attempting to fix. It also means that I didn't review the change at the time.

Pyright installs a file system watcher on the directories that comprise the Python environment's installed libraries (typically, site-packages). When this file system watcher event fires, it means that a library file has been modified, which triggers a full reanalysis. The problem is that someone added all of the extraPaths and execution environment root directories to the list of watched directories. The comment indicates that this was done because "the watch list includes extra paths that are not part of user files". This is problematic for monorepos that are using execution environments.

@rchiodo, could you please look into this or make sure it's assigned to someone else on the pylance team? Thanks!

[rchiodo]

Looks like it was this commit:
https://github.com/microsoft/pyrx/commit/5e2ed91374ac345e6ba484eeb715a8c900efb809

[rchiodo]

Created this issue:
#7001

Thanks for investigating Eric :)

[heejaechang]

So, extra path was added for users who use extra path for editable install. The user wanted to see changes in the dependent workspace root connected through editable install (extra path) without reloading VS Code or restarting the language service.

Since we don't have proper project-to-project support for multi-root workspaces like Visual Studio (by "p2p support," I mean live update support between workspace root dependencies—other language servers, such as C#, handle this differently from assembly (static) dependencies, allowing changes to be applied live), our current approach supports it the same way as regular package changes. However, this could be too resource-intensive for certain users.

That said, we might need to either drop live update support for p2p dependencies or improve our handling of them.

[heejaechang]

so, we discussed in the scrum, and we decided to gather some more info before we decide how to address this.