Spaces:
Running
Running
Commit History
Debugged reference to random_seed in vectorisation and reference to torch in representation_model.py
8216d8c
Importing space package near start of app now to avoid issue with cuda being initialised before
9e84863
Rearranged functions for embeddings creation to be compatible with zero GPU space. Updated packages.
cc495e1
Added example of how to run function from command line. Updated packages. Embedding model default now smaller and at fp16.
34f1e83
Improved initial clean options. Now has option to return embeddings only.
89c4d20
App now retains original index following cleaning to allow for referring back to original data
90553eb
Removed some requirements from Dockerfile for AWS deployment to reduce container size
51ba1cb
Allowed for app running on AWS to use smaller embedding model and not to load representation LLM (due to size restrictions).
22ca76e
Only aggregate topics not 'other', allowed for minimum sentence length, default max_topics now will auto aggregate topics. Added Cognito Auth functionality (boto3 with AWS).
1e2bb3e
Can split passages into sentences. Improved embedding, LLM representation models, improved zero shot capabilities
55f0ce3
Updated packages. Improve hierarchy vis. Better models - mixedbread and phi3. Now option to split texts into sentences before modelling.
04a15c5
Minor cleaning, csv formatting changes
d80c8f5
Sean-Case
commited on
Reduce outliers now more efficient and relabels with correct vectoriser. Default topic labels now tidier. Hiearchical topics outputs more useful for joining to df afterwards. Switched low resource reduction algorithm to UMAP as default is not good.
e1c1f68
Should now parse custom regex correctly. Will now wipe previously created embeddings if 'low resource mode' option switched.
0a543a0
Sean-Case
commited on
Allowed for uploading custom regex for cleaning. Fixed calculate all probabilities, reduce outliers. Added text tree for hierarchical modelling.
381f959
Upgraded to Gradio 4.16.0. Guide for converting to exe added.
0a177ca
Hopefully now LLM download from hub should work
cdcd7af
Added note to say that LLM representation is not currently working on the HF website
3b4333f
Sean-Case
commited on
Trying to download LLM to local_dir instead of cache_dir
539aba9
Sean-Case
commited on
LLM model save is failing in Huggingface - attempting instead to save to base folder
c2bf185
Sean-Case
commited on
Some text changes. Fixed a couple of TF-IDF embeddings issues
87306c7
Sean-Case
commited on
Fixed file load with files including capital letters
9c6425d
Added clean data options, improved re-representation options and visualisation. General format changes
4effac0
Allowed for loading in external topic labels. A few visualisation modifications.
b27bab2
Model save now checks and makes a folder before writing the model
356791c
Lots of general fixes. New visualisations, fixed hierarchical vis for zero shot. Added calc all probabilities.
b4510a6
Changed Phi model to smaller StableLM 2 1.6. Fixed a None type detection error.
1f1a1c7
Switched embeddings model to BGE Small 1.5 as Jina seemed unable to do zero shot topic modelling properly
be094ee
Split off LLM representation, visualisation, and reduce outliers from main function. Added hierarchical visualisation and logs
5d87c3c
More efficient embeddings save and representations load/process. Custom visualisation hover option added, formatting improvements. Version 0.1?
ffe5eb2
App should now check if embeddings are loaded before topic modelling. And will save only once.
9eeba1e
Hopefully fixed install and load of LLM model on systems without a HF_HOME environmental variable
32cf9fb
Sean-Case
commited on
Added option to reduce outliers based on closest topic
e09dd3b
Returned TruncatedSVD components to 100 - higher values don't seem to help
43ac0d8
Sean-Case
commited on
Greatly increased low resource process dimensions for higher quality. Visualisations disabled by default to increase speed.
fac3624
Sean-Case
commited on
Greatly improved low resource mode speed (at cost of potential quality)
aa3df37
Sean-Case
commited on
Changed zero shot min similarity to 0.5
0b7839c
Now should save embeddings by default. Added random seed to representation
e0f53cc
Sean-Case
commited on
Fixed llm_config, environmental variable, zero shot topic model errors with quick embeddings
ff32b4a
Sean-Case
commited on
removed cpu_config as causing problems
82b1ab1
Sean-Case
commited on
Model export changed to safetensors. Improved representational model function. Got zero shot topic modelling working
4cfed8e
Sean-Case
commited on