Correct Way To Create A Package From Protobufs

[inolaole]

About:
We use a repository to generate python files (.py and .pyi) from protobuf definitions. I want to package this repository into a poetry package. I am having trouble finding any relevant documentation outside of this https://github.com/orgs/python-poetry/discussions/10082.

[repo root]
- schemas
   - [org_name]
      - [area (i.e. accounts)] 
         - [sub (v1)]
            - *.proto
         - [sub (v0)]
            - *.proto

which generates .pyi/.py files in the following structure (note that it does not create any init.py files

[repo root]
- generated
   - python
      -  standardproto 
         -  [org_name]
            - [area (i.e. accounts)]
               - [sub (v1)]
                  - *.pyi, *.py
              - [sub (v0)]
                  - *.pyi, *.py

Desire:

1. If possible I would like to learn more about poetry's ability to actually generate the .pyi/.py files using it's protobuf build-backend. I've seen limited documentation on it, and it seems like it might not even exist. If so that's ok, but I'd like to confirm my understanding
2. If poetry can not gen the files that's fine I would just like to know the correct way to define a toml to crawl the generated path, get all of the packages, and then bundle them for distribution via poetry.

[abn]

You can use the include keyword if you want to do this as-is. But unfortunately that won't give you import-able python packages without the __init__.py (you can omit this only for namespace packages). Poetry does not do anything with protobufs.

You can look at https://github.com/python-gnxi/python-gnmi-proto for an example how you could use betterproto to generate python code and then package it with Poetry.

Note that the generation step is done outside of Poetry via the Makefile.

[inolaole]

@abn interesitng thanks for the pointer. I looked over your repository and have some further questions. If I wanted to skip the somewhat involved process you have for that repo and instead simply bundled these generated files with poetry is there a pointer on how to accomplish this? From what I can tell in your repo, you managed to do this by using init.pys genned by betterproto.

[inolaole]

I ask because if I specify the include keyword python complains as there are technically multiple packages defined in that folder.

[abn]

@inolaole I am not sure I fully comprehend what you are trying to achieve. And from what you are describing,. It doesn't sound like a valid python package/module.