Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Initial Draft of Just Labeling Baseline #89

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from

Conversation

jawache
Copy link
Contributor

@jawache jawache commented Oct 2, 2024

I have not even spell checked this, sorry - it's still a WIP but wanted to share it now to get early feedback (I will spell check it before the next meeting!)

@chrisxie-fw / @seanmcilroy29

@jawache
Copy link
Contributor Author

jawache commented Oct 2, 2024

@chrisxie-fw
Copy link
Contributor

  • line 92: > Note: SCI is a Software Carbon Efficiency Specification which is computed a "Carbon per Functional Unit" of a software product. For example, Carbon per Prompt for a Large Language Model.

Should it be "computed as" or ?

  • line 90: Any computation of a SCI score for the SCER label SHALL adhere to all requirements of the SCI specification.

In this post, I've shared my design considerations for the SCER label. One of the considerations is "ease of use". If SCI score is quite rigid and difficult for people to follow, the requirement of line 90 could also make SCER label difficult to adopt.

Would it be more practical if the wording is such that SCI is a preferred unit of measurement (default), followed by SCC etc. For reference, this was defined in the original SCER spec:

Benchmark Measurement Benchmark Workload Test Environment Test Methodology
  • kWh
  • SCC
  • SCI (default)
  • CO2e, or
  • Other
  • Unit of Software Function (USF)
  • Number of times to execute USF
  • Hardware specifications
  • Software Configuration
  • Setup Instructions
  • Execution Steps
  • Result Collection and Analysis
  • This could be a bigger issue as a whole, but could be deferred to future versions of SCER label, based on user feedback.

    • line 104: Where the common term Carbon Shall be used to represent the more technical term CO2e (Carbon Dioxide Equivalent)

    According to the released SCI spec, Units: this shall be in grams of carbon per kilowatt hours (gCO2eq/kWh).

    When a SCER label is displayed, would the label say "Carbon", or "g Carbon", or "gCO2e"? For example, would it be "27g Carbon/USF"?

    @jawache
    Copy link
    Contributor Author

    jawache commented Oct 8, 2024

    Thanks @chrisxie-fw

    I'll update the first point with your correction.
    Re: the last, yeah I see you're point - it's in the SCI spec so we have to adhere to it and use gCO2e, I'll update the spec.
    RE: The middle point about use with other measurements like SCC.

    • The GSF creating a label for someone else's standard or worse, a measurement which is not a standard, is a minefield of additional research and analysis of risks (legal and brand/trust etc..). Since the aim is to get something through consensus quickly, I'd suggest we push those conversations to the future.
    • Once we get a label for SCI out the door, my gut instinct is that we just copy-paste and create a whole new spec for every standard we want a label for, but again, I think we should push that conversation to the future.

    Updated as per @chrisxie-fw comments
    
    Signed-off-by: Asim Hussain <[email protected]>
    @chrisxie-fw
    Copy link
    Contributor

    Agreed!

    jawache and others added 2 commits October 10, 2024 14:44
    @jawache
    Copy link
    Contributor Author

    jawache commented Oct 10, 2024

    Hey @chrisxie-fw and @seanmcilroy29, so i've:

    • Created a branch called future which is the existing "dev" branch.
    • Removed all the extra files from this branch not related specifically to the labeling.
    • Added the latest design from Jenya with a note it's still WIP (there are a few practical design issues still, needs to handle variable length text)
    • The spec is the new just labelling spec.

    It's not "complete" and ready for initial review, but it's worthwhile sharing around for early feedback.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    3 participants