Allow keeping pylance features enabled in Jupyter notebook cells with %% magic

[ian-h-chamberlain]

There are some use cases for keeping Pylance features (type checking, semantic highlight etc.) on for Jupyter notebook cells that use %% magic directives (for example, %%capture). This is sort of counter to a previous issue filed #3327.

For example, if I have a cell like this:

%%capture

def xyz() -> str:
    print("hi")
    return 123

xyz()

The presence of %%capture seems to disable Pylance features (type checking, refactor, semantic highlight etc.) but in this case it would actually be preferable to treat everything but the magic as plain Python.

I'm not sure of a perfect way to solve this generally given the %%sql etc. use cases above, but there are two options that I think could work:

1. Support #!%% magic in notebook cells, which was the approach taken in IPython magic commands need to be runnable from comments so other tools can work with them. vscode-jupyter#3263 but it doesn't seem to work in .ipynb files (maybe it's just for .ipy files?). This would allow Pylance, isort etc. to just do their jobs easily without any special handling, I think.
2. Support %%capture # type:ignore or something along those lines, to indicate that the rest of the file should be treated as Python but ignore the %%. This seems perhaps more compatible with the default Jupyter implementation, but more work on the Pylance side (and other tools that might need to handle it)?

Originally posted by @ian-h-chamberlain in #2582 (reply in thread)

[rchiodo]

If this is the next step for cell magics, we'd need to know which ones aren't expecting another language in the cell.

[rchiodo]

Or maybe if # type : ignore is there, then don't ignore the rest of the cell.

[debonte]

I'm missing something. What would the diagnostic be?

If there's a diagnostic, that means were neither ignoring the cell (current cell magic behavior) nor ignoring the line (current line magic behavior)?

Would this diagnostic only be present for cell magics like %%capture and %%time that are expected to have Python code after them and not for things like %%script? If we're going to special case those, why not treat them as line magics so the user doesn't need to #type: ignore anything?

[rchiodo]

If would show up if we stopped ignoring the current cell.

This would still have to be special cased though. Cause otherwise %%sql wouldn't work.

Logic would be:

if (cellMagicPresent && noTypeIgnore) {
   skipCell()
}

As opposed to right now where we have:

if (cellMagicPresent) {
   skipCell()
}

And because there's a type: ignore on that line, normal processing would just ignore it.

[debonte]

This would still have to be special cased though. Cause otherwise %%sql wouldn't work.

If we're special-casing, couldn't we just treat cell magics that are expected to have Python code after them (ex. %%capture) as line magics? And then no #type: ignore is required?

[rchiodo]

That might be better. I was thinking it was harder to know what cell magics might allow python after than to know which magics definitely didn't, but it's probably just as hard either way.

Like is the list of non python cell magics more 'known'? (Cell magics can be dynamic too but I don't think that happens too often).

[debonte]

Maybe we could put the list of magics in a config setting that users could modify if needed? Here's a horrible name that gets the intent across: python.analysis.cellMagicsThatAreFollowedByNonPythonCode :)

[ian-h-chamberlain]

From this IPython documentation, I'm not sure that's possible in the general case — it looks like the obvious ones would be %%capture, %%python{2,3} but there could be others defined by user packages, which is why I think having a more general way to specify "ignore this line but not the rest of the cell" or "ignore the whole cell" (maybe defaulting to the latter) makes sense.

[rchiodo]

Transferred to a discussion to get up votes.

[Josverl]

✅
I would very much like a way to not be able to VSCode Jupyter with Pylance for use with MicroPython.

such a feature would make the combination very powerful to teach micropython and IoT, and also would make for a simple way to experiments with different and document the results making is easy to share samples in the community,

using the micropython magics
Cells can use cell magics '%%micropython' could could be parsed & checked by pylance ( with the use of micropython type-stubs)

other candidates for an allowlist are:

• %%python
• %%python2
• %%python3
• %%pypy
• regex .*python.*

I would add that cell magics can have additional parameters following the cell magic, for instance:

%% micropython connect COM7 
# blink the led
from machine import Pin
led = Pin(25, Pin.OUT)
import time
for _ in range(5):
    led.value(0)
    time.sleep(0.25)
    led.value(1)
    time.sleep(0.25)

for more details see: https://ipython.readthedocs.io/en/stable/config/custommagics.html#defining-custom-magics

There are other partial solutions implemented that appear to only only in python , and fail to run the magics from Jupyter

• IPython magic commands need to be runnable from comments so other tools can work with them. vscode-jupyter#3263
• Inside of a notebook, magic cells should not cause an error in pylance vscode-python#17058

[rchiodo]

@DonJayamanne suggested we use request forwarding and then map magics to what language they should support.
#4969

The request forwarding should work for python too as we'd just forward it to ourselves.

[heejaechang]

Ya, the issue is assigned to me and I would like to work on it at some point to enable embedded language. starting from notebook magic cells. but once we have infrastructure, we could do it for other things like doc string, regex, string that represents file or folder and etc like how other language such as C# does.