Retaining some state between Remote Cells targeting the same node

[mbklein]

For the past several months, my team has been using Livebook – especially the Kino.RemoteExecutionCell component – as the primary tool for maintaining a running Elixir application. Being able to connect and run code in the app's environment has been a game changer.

The only major wishlist item I have every time I use the feature is that I wish Remote Execution cells targeting the same node could access local variables, imports, and aliases defined in previous Remote Execution cells the same way local cells can. I started trying to work on the feature myself, thinking I could contribute a PR, but I quickly realized I don't know nearly enough about Elixir's internals and introspection features to do it right.

If any of the project maintainers have even a general idea how this kind of thing might be implemented – or if it's just completely unfeasible and I should give up before I start – I'd love to hear about it.

Thanks!

[jonatanklosko]

Hey @mbklein, it's awesome to hear how useful the remote execution is :)

As for the feature, there are a couple things, but one of primary importance is that for variables we want to transfer as little data between the nodes as possible, especially that copying across processes gets rid of structural sharing, so it would add both to the transfer size and overall runtime memory usage. That's why we allow assigning just the result to a variable, rather than just getting all variables, so that the user explicitly assigns only what they actually need later. Similarly, if you look at the generated code, we use the Kino.RPC.eval_string macro, so that we can analyse which variables are used and send only those.

Also, propagating imports/requires in either direction would be invalid, because it assumes that both the runtime and the remote node have the relevant modules, which may not be true (same for aliases, but those can point to non-existent modules).

Technically, we could say only remote smart cells share the environment, but this means keeping Livebook state in the remote node and I don't think that's something we should do.

For completeness, the attached runtime does what you want (all cells are remote and share environment), but it has its own limitations (not being able to install packages and building livebook apps).

[mbklein]

I was definitely thinking more of aliases than either imports or requires. I know it's a minor and nitpicky thing, but aliasing a module in one cell and not having autocomplete work on it in the next is just “off” enough for me to want to make it work. 😃

As for the variables, your mention of the eval_string macro provoked a couple ideas I can't quite put into words but I might play with a bit and see what comes of it.

Thanks!