Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement an ijar equivalent for Scala 3 #1679

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Implement an ijar equivalent for Scala 3 #1679

wants to merge 7 commits into from

Conversation

jadenPete
Copy link

Description

From my comment in #1657:

From what I understand, though, TASTy markes a major shift in how type information is stored between compiler invocations from Scala 2.13's "Pickle" format in that it includes a completely typed representation of the program's AST. I'm of the opinion that this defeats the purpose of using ijar, since most changes to a program's source code will now induce changes in its TASTy files, thus changing the generated ijar and invalidating reverse dependencies' cached compilation actions.

When you consider that and the fact that "scalac doesn’t have to read their .class files anymore; it can read their .tasty files", it may make more sense to develop an alternative to ijar specially for Scala 3 targets, as opposed to modifying ijar. That way, we can strip out information from TASTy files not needed for compilation (e.g. positional information, documentation, private members, etc.), similar to what @susliko described.

Given these changes in how symbols are read from the compile classpath in Scala 3 vs Scala 2, I don't think ijar is an appropriate solution for producing Scala 3 interface JARs. For that reason, I've developed "dottyijar", a version of ijar that produces an interface JAR by removing all .class files and performing a few modifications to .tasty files:

  • Clearing the UUID, "Positions" section, and "Comments" section
  • Removing all val and def definitions that are private and not inline
  • Replace the values of all val and def definitions and class parent arguments with ???
  • Remove unused names from the name table

For context on the TASTy format, see dotty.tools.tasty.TastyFormat.

Thus, dottyijar produces interface JARs that are far less sensitive to implementation changes than those produced by ijar. Although it's not as fast as ijar (mainly because it's written in Scala and doesn't read and write in a single pass), it's very fast—dottyijar produced an interface JAR for the Scala 3 compiler, which contains 1,011 TASTy files, in 4.369 seconds on my machine, whereas ijar took 0.292 seconds.

Motivation

I'm implementing this change to close #1657.

@jadenPete jadenPete mentioned this pull request Jan 16, 2025
Open
WojciechMazur
WojciechMazur reviewed Jan 20, 2025
View reviewed changes
scala/private/macros/scala_repositories.bzl Outdated
"com_softwaremill_common_tagging",
"dev_zio_izumi_reflect",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these really needed? Yes, these alllow to define more typesafe TASTy format specification, but at the same time these are introducing additional dependencies. Probably it would be best to keep this list as small as possible.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dev_zio_izumi_reflect is needed during testing so we can get information about which TASTy nodes were read if dottyijar fails. Without it, debugging failures becomes near-impossible. com_softwaremill_common_tagging can probably be done without, since it's only used to differentiate SignedInt and UnsignedInt, and SignedLong and UnsignedLong at the type level. This isn't just for type safety—the compiler needs it so it can summon the appropriate TastyFormat. I think I can replicate this with value classes.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@simuons
Copy link
Collaborator

simuons commented Jan 22, 2025

Hi, @jadenPete, thanks for PR. Looks like checks are failing due to strict deps. Could you resolve this? I hadn't had a chance to look at PR yet but I'll try to in upcoming days.

@jadenPete
Copy link
Author

Hi, @jadenPete, thanks for PR. Looks like checks are failing due to strict deps. Could you resolve this? I hadn't had a chance to look at PR yet but I'll try to in upcoming days.

Sure thing! I'll look into the test failures.

Jaden Peterson added 7 commits January 22, 2025 16:42
Added a Scala 3 toolchain for use in rules_scala
c3ab6b7
Vary the specs2 version by the Scala version
3d18e86 
Different versions of specs2 support different Scala versions, so it's
necessary to use a different version, depending on which Scala version
we're on. v4.10.6 is the last to support Scala 2.11, v4.20.9 is the last
to support Scala 2, and the latest version as of the time this commit
was written, v5.5.8, only supports Scala 3.
Wrote a TASTy parser
42c8958
Wrote an early version of dottyijar
4e90f03 
Currently, it just strips private `val`s and `def`s and definition
values, but I plan on making it more aggresive in the future.
Integrated dottyijar with the ruleset and wrote tests for it
8be9015
Don't use com.softwaremill.tagging
4af41de 
We'd like to minimize the number of required dependencies.
Support versions of Scala 3 prior to v3.4.0
4915b4b
@jadenPete
Copy link
Author

All the tests are now passing! I've squashed all of the fixup! commits I added to make the pull request easier to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WojciechMazur WojciechMazur WojciechMazur left review comments

@liucijus liucijus Awaiting requested review from liucijus liucijus is a code owner

@simuons simuons Awaiting requested review from simuons simuons is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Better caching for Scala 3 (via ijar or tasty inspection)
3 participants
@jadenPete @simuons @WojciechMazur