We need agentic + a few other ideas

[Aldian1]

From testing theres a few scenarios that would be good to cover:

1. When running tests and my NLP was poor, the Ai struggles to solve and the test fails. Perhaps here it would be cool to pause the test, let me fix it and then re-run... rather than having to restart the entire test file again. Might be my fault but some tests are taking 1764.46s+++ to run. NLP is a little nuanced so needs refining sometimes.

2. Provide the Ai with context upfront by scraping screenshots of the pages it needs to navigate, reducing a thinking step.

3. Solve the need for 20,000 screenshots lololol???

4. A possibility to run multiple agents simultaneously to cover test files. Speeds up entire testing process.

[Aldian1]

7 screenshots for a login

[gladyshcodes]

Hi Aldian,

Thank you for your suggestions!

When running tests and my NLP was poor, the AI struggles to solve, and the test fails.

Could you clarify what you mean by "poor NLP"? Are you referring to the model's inability to interpret specific inputs or something else?

Perhaps here it would be cool to pause the test, let me fix it, and then re-run...

The core idea behind Shortest is to automate and simplify testing. If manual intervention is required, it defeats the purpose of automation 😄. However, if we had our own trained model, this could be more feasible, as the model could improve over time with such feedback

Might be my fault, but some tests are taking 1764.46s+++ to run.

This seems like a significant performance bottleneck. Could you provide an example of such a test and the context in which it was run?

Provide the AI with context upfront by scraping screenshots of the pages it needs to navigate, reducing a thinking step.

The challenge here is that we can’t predict the steps the AI will take until it begins running. Almost every subsequent step depends on the outcome of the previous one

Solve the need for 20,000 screenshots lololol???

There are instances where we take duplicate screenshots unnecessarily. Fixing this would certainly reduce the total count. Additionally, screenshots could be deleted after a certain threshold to manage storage

A possibility to run multiple agents simultaneously to cover test files. Speeds up the entire testing process.

Are you referring to running tests in parallel? This is currently an open topic, and we plan to implement it soon

Looking forward to your thoughts!

[Aldian1]

Morning !

Could you clarify what you mean by "poor NLP"? Are you referring to the model's inability to interpret specific inputs or something else?

The core idea behind Shortest is to automate and simplify testing.

Sometimes my prompt results in a loop that could be fixed by just being a tad more specific. The annoying thing is I then need to re-run the tests again (which take a while to run bc AI, claude and everything else needs running). Would be nice if there was a way to hot reload and just continue from where the test failed. This would allow me to move faster.

This seems like a significant performance bottleneck. Could you provide an example of such a test and the context in which it was run?

Seems to be fixed with some of the new improvements !

Are you referring to running tests in parallel? This is currently an open topic, and we plan to implement it soon

Exactly this !!

here are instances where we take duplicate screenshots unnecessarily. Fixing this would certainly reduce the total count. Additionally, screenshots could be deleted after a certain threshold to manage storage

I wonder if there is a way to cache previous run tests in a way where instead of needing to take a screenshot, it already has the screenshot and just needs to send to claude and get the next Ai command.