Integration tests #402
Assignees
Labels
Comments
jhrozek
added a commit
to jhrozek/codegate-open
that referenced
this issue
Jan 9, 2025
Since the copilot provider is a proxy, we add a "requester" module that depending on the provider makes a request either using raw python requests like earlier or by setting a proxy and using a CA cert file. To be able to add more tests, we also add more kinds of checks, in addition to the existing one which makes sure the reply is like the expected one using cosine distance, we also add checks that make sure the LLM reply contains or doesn't contain a string. We use those to add a test that ensures that the copilot provider chat works and that the copilot chat refuses to generate code snippet with a malicious package. To be able to run a subset of tests, we also add the ability to select a subset of tests based on a provider (`codegate_providers`) or the test name (`codegate_test_names`) These serve as the base for further integration tests. To run them, call: ``` CODEGATE_PROVIDERS=copilot \ CA_CERT_FILE=/Users/you/devel/codegate/codegate_volume/certs/ca.crt \ ENV_COPILOT_KEY=your-openapi-key \ python tests/integration/integration_tests.py ``` Related: stacklok#402
jhrozek
added a commit
that referenced
this issue
Jan 9, 2025
Since the copilot provider is a proxy, we add a "requester" module that depending on the provider makes a request either using raw python requests like earlier or by setting a proxy and using a CA cert file. To be able to add more tests, we also add more kinds of checks, in addition to the existing one which makes sure the reply is like the expected one using cosine distance, we also add checks that make sure the LLM reply contains or doesn't contain a string. We use those to add a test that ensures that the copilot provider chat works and that the copilot chat refuses to generate code snippet with a malicious package. To be able to run a subset of tests, we also add the ability to select a subset of tests based on a provider (`codegate_providers`) or the test name (`codegate_test_names`) These serve as the base for further integration tests. To run them, call: ``` CODEGATE_PROVIDERS=copilot \ CA_CERT_FILE=/Users/you/devel/codegate/codegate_volume/certs/ca.crt \ ENV_COPILOT_KEY=your-openapi-key \ python tests/integration/integration_tests.py ``` Related: #402
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have a decent unit test coverage, but we should build some integration tests as well, e.g. try talking to codegate as it passes requests to an LLM and then read the replies.
The text was updated successfully, but these errors were encountered: