Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load data into an integrated OLAP database. Allow downloading a CSV or JSONL data export via API endpoint #314

Open
arov00 opened this issue Jan 20, 2025 · 0 comments · May be fixed by #317
Labels
enhancement New feature or request OLAP Something related to the built-in OLAP module

Comments

@arov00
Copy link
Contributor

arov00 commented Jan 20, 2025

In the interest of simplifying a deployment, we should add an integrated OLAP database (DuckDB) to the aggregator. Emitted events should be ingested into this DB. This makes any kind of downstream system optional for analytics. The aggregator shoud offer an API endpoint like:

GET aggregator:8080/api/export?from=2025-01-01&to=2025-02-01&metrics=bytesin,bytesout,storage

Content-negotiation should be used to determine the exact format of the data export, with CSV and JSON(L) being possible options. This would allow customers to download data for the relevant billing period and generate bills/reports via e.g. Jupyter by talking to the aggregator directly. Operators will no longer need to maintain a dedicated database deployment.

It should be possible to deactivate this OLAP module, in case users prefer collecting aggregated data in an external system.

@arov00 arov00 changed the title Load data into an integrated OLAP database. Allow downloading a CSV or JSONL data report via API endpoint Load data into an integrated OLAP database. Allow downloading a CSV or JSONL data export via API endpoint Jan 20, 2025
@arov00 arov00 added enhancement New feature or request OLAP Something related to the built-in OLAP module labels Jan 20, 2025
@arov00 arov00 linked a pull request Jan 22, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request OLAP Something related to the built-in OLAP module
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant