llm-arch / src /testing.py

Commit History

Made updates to support automatic reload of the TestGroups after a test run
e35ef72

alfraser commited on

Updated from using random.choices to random.sample throughout where I need a random distinct set as choices does replacement so you can get the same item twice. Discovered in pricing testing.
b897a48

alfraser commited on

Implemented single threaded worker on writing the logs to the json file for controlled access to the resource on the file system now we are multi-threading the tests.
c0a1e47

alfraser commited on

Modified test runner to dispatch requests in parallel to make use of the fact that there is a lot of wait time for the LLM. Defaulting to 16 threads.
bb7db2c

alfraser commited on

Saved file records to DB. Fixed a print to show the correct test-group name.
3a9dec1

alfraser commited on

Updated the testing page to show the request/response pairs
9cec719

alfraser commited on

Updated the offline save to save the actual request and response text
34061f5

alfraser commited on

Added a missing comment
bcc302b

alfraser commited on

Fixed a bug where if the architecture had entirely failed and not generated a response the whole load of TestGroups would crash. Need to fix the root cause of the failure to generate a response, but also should be caught gracefully here in any event.
4332953

alfraser commited on

Refactored loading the TestGroups to make the structure of the json load and the DB load the same and clearer
c76e6f5

alfraser commited on

Added comments throughout
e912278

alfraser commited on

Added loading of test groups from both the DB and the local file and merging these two
1fb12dc

alfraser commited on

Adding the sqlite db where I will archive the test results and added the archiving code
843d9d3

alfraser commited on

Added the test reporting structure
82130cb

alfraser commited on

Added utility to serve up the test questions
a732fe2

alfraser commited on