Discuss: File format in the .NET Interactive VS Code extension

[jonsequitur]

The primary reason for the experimental file format supported by .NET Interactive Notebooks is that there are a few design decisions to be worked out to enable the Microsoft Python extension for Visual Studio Code and the .NET Interactive Notebooks extension to work well together. Until a design is agreed upon, we want to avoid introducing potential compatibility issues into .ipynb.

Here are some of the challenges:

• There is not an agreed-upon convention for polyglot notebooks in the .ipynb file format that also takes into account:
  • single-process polyglot kernels, and
  • multiple kernels (whether polyglot or not) within a single notebook.
• There isn't currently a clear way to indicate which VS Code extension should handle a given .ipynb file, and Python clearly takes precedence.

---

I'd like to get people's thoughts on the experimental .dib file format (example below) that the .NET Interactive Visual Studio Code extension supports.

.ipynb is also supported, as well as conversion between the two.

These are the goals we've had in mind that led to the current state of the design:

• It should be trivial to author and read in a simple text editor without special tooling and without concern about escaping, e.g. not JSON, XML, etc.

• It should be copy-pasteable between different contexts, e.g. a bare script file, a notebook cell, or another .NET Interactive host (e.g. https://github.com/CodeConversations/CodeConversations)

• It should be able to contain multiple languages, including languages not yet supported by .NET Interactive.

• It should be amenable to a good tooling experience, including completions and other language services even when multiple languages are present.

• It should be able to be opened as a notebook (as currently implemented by the .NET Interactive VS Code extension).

• It should be executable as an automation script. In the simple case this need not be a polyglot script, but it should be able to take advantage of the full capabilities of .NET Interactive, e.g. magic commands, NuGet support, extensibility, variable sharing.

  • When in automated mode, it should be able to advertise and consume command line arguments in a readable, simple-to-author manner.

  • When in interactive mode (e.g. notebook, REPL), the command line inputs should be able to be collected from the user via a prompt or trivially configured inline the code.

Here's a simple example of the format:

#!powershell

Write-Output "Hello from PowerShell!"

#!fsharp

let hello = "Hello from F#"
hello |> Console.WriteLine

#!csharp

#r "nuget:Humanizer,2.8.1"
using Humanizer;

#!share --from fsharp hello

Console.WriteLine(hello.Replace("F#", "C#").Transform(To.TitleCase));

[zyzhu]

ipynb is the common notebook file format for all languages (Python, R, Julia etc.) on Jupyter.

If dib is the file format supported in VSCode, is it only limited to .Net languages?

If it's intended to be more widely adopted, is this file format extendable and easily convertible so that other language communities can extend their notebook experience in VSCode?

[jonsequitur]

If dib is the file format supported in VSCode, is it only limited to .Net languages?

.dib isn't language-specific and other languages can be added dynamically by adding subkernels.

If it's intended to be more widely adopted, is this file format extendable and easily convertible so that other language communities can extend their notebook experience in VSCode?

The file format can be extended to the degree that .NET Interactive itself can be extended, whether programmatically within the script or via a NuGet extension.

[dsyme]

I think that's a good list of technical goals.

It's fine to contemplate a new file format but it should be 100% optional until it is an established thing - which roughly means online and client tooling for the format is very widespread (including scripting executors for it), stable and trusted, and the format has been emotionally accepted by most potential users especially in team or collaborative settings. Also TBH Microsoft have a reputation when it comes to file formats, we have to be aware of that and be cautious here.

Note that acceptance of new file formats is a social/network/human/emotional/trust thing - these things are not technical things under our control though we can make technical decisions that will affect whether effective acceptance is eventually achieved (and I don't see any particular technical problem with the technical decisions above). However by their nature all new file formats start at "acceptance level of minus 10000" sadly.... I speaking from experience with .fsx as a programming/scripting format - it now has near universal execution tooling via dotnet fsi and fairly widespread tooling but it has taken >10 years to get to that point. And that's not including other forms of tooling.

I think this means loading, saving, editing and executing .ipynb is necessary in very many scenarios (potentially by internally converting to this format on load, but that should not be user-visible).

[jonsequitur]

The file format is optional. The benefits are related to polyglot functionality and automation (e.g. running in production), neither of which are directly supported by Jupyter. Not all users will care about these goals, but many do, and they're becoming more common as notebooks become popular in new areas. For these cases, this format might provide some convenience.

The file format arose organically because it's just the .NET Interactive submission format: magic commands delineating blocks of code in potentially different languages. Since .NET Interactive is submission-centric rather than cell-centric, the following (one cell):

#!pwsh
    < PowerShell code >
#!fsharp 
    < F# code >

...is equivalent to this (two cells):

#!pwsh
    < PowerShell code >

#!fsharp 
    < F# code >

This is why the file format started here. We're simply concatenating all of the cells.

[jonsequitur]

I think this means loading, saving, editing and executing .ipynb is necessary in very many scenarios (potentially by internally converting to this format on load, but that should not be user-visible).

I agree entirely.

Assuming we can sort out the default experience for a given file extension such that we're not in conflict with the VS Code Python extension's Jupyter support, an intuitive workflow might be to allow you to open and work with either format, and when you save, it saves it back to the same file in the same format. You never have to think about the other file format until or unless you want to explicitly convert between them.

[zyzhu]

I apply this snippet in Jupyter config jupyter_notebook_config.py
https://gist.github.com/jbwhit/881bdeeaae3e4128947c#file-post-save-hook-py

Then Jupyter automatically saves Python notebook into .py file and IFSharp notebook into .fs file whenever I run any cells. It happens behind the scene. It also output the .html file. I usually share my result to nontechnical people in html file as you can open it from email directly. Essentially it just concatenates all code cells into one file.

Isn't this functionality already working in Jupyter? Why do we need another file format? I can only imagine the use case for polyglot notebook. It's hard to decide whether it's .cs, .fs or .ps1

[jonsequitur]

This looks like a good workflow and for Jupyter it's nice and simple. You don't need an alternative file format and it's not my intention to give the impression that we're requiring one.

I can only imagine the use case for polyglot notebook. It's hard to decide whether it's .cs, .fs or .ps1

Yes, the alternative file format helps with polyglot scenarios (which not everyone has). But people might also find benefits when editing with plain text editors or using code review tools. These considerations motivated a similar approach in the VS Code Python extension allowing execution of "cells" within a .py file.

But this isn't a language-specific tool, so we wanted to explore a format that doesn't bias toward one language. Conceptually, .NET Interactive's polyglot support includes HTML and JavaScript and might grow to other languages like SQL, bash (#465), and so on. These aren't just magic commands but are implemented as in-process subkernels. These are user-extensible. Here's a sample (.dib | .ipynb) that includes PowerShell, C#, HTML, and JavaScript.

[dsyme]

Assuming we can sort out the default experience for a given file extension such that we're not in conflict with the VS Code Python extension's Jupyter support...

It would be quite nice if VSCode simply prompted in this case. Any idea what actually happens?

Also I guess VSCode can't probe into the file to check if an extension claims ownership?

[brettfo]

@dsyme I just tried it and currently our extension steals the association; there's no prompt and no way to re-associate with the Python extension (short of uninstalling), and since we can really only handle a subset of Jupyter notebooks (i.e., those using the .NET Interactive kernel), we shouldn't take the association outright. There's an open issue at microsoft/vscode#94408 that tracks this. As soon as this is fixed then we can safely associate wtih .ipynb files.

As for probing, for text files an extension can say whether it takes the association outright or if it should simply be added to the consider list, but nothing yet for notebooks.