|
--- |
|
license: mit |
|
--- |
|
## Development Process |
|
This model was created with the intention of supplmenting our mobile application as an ingredient counter. Such that a user could take an image and have AI determine whether they have the necessary ingredients to create a recipe. As a result of the straightforward usecase, we were motivated to use a lightweight model for this classification task. We were recommended Moondream by a mentor from Intel, which we fine-tuned in this project. |
|
|
|
## Model Details |
|
|
|
- **Model Name**: Moondream Fine-tuned Variant |
|
- **Task**: Ingredient Classification and Counting |
|
- **Repository Name**: nyap/cosmo-demo |
|
- **Original Model**: vikhyatk/moondream2 |
|
|
|
## Intended Use |
|
|
|
- **Primary Use Case**: This model counts and classifies ingredients (exclusively avocados, bananas, doritos, and onions during CalHacks) in images. |
|
- **Associated Application**: This model is applied through a cooking assistant mobile application built in React-native. |
|
|
|
## Data |
|
|
|
- **Dataset**: Used a very small dataset of a little less than 100 images collected during the Hackathon. Including images of avocados, bananas, doritos, and onions from various angles and in different combinations. |
|
- **Data Collection**: Images were taken of various tables with the ingredients arranged, before being manually labelled with the count/type of ingredients in each image. |
|
|
|
## Evaluation Metrics |
|
|
|
- **Performance Metrics**: Performance metrics did not provide any valuable feedback due to the size of the dataset (overfit) |
|
|
|
## Training |
|
|
|
- **Training Procedure**: Created a basic dataloader class to read image data and expected output. PIL images were then passed through the VIT provided with Moondream. Moondream was then fine-tuned using the default Adam optimizer with a batch size of 8 for 30 epochs. |
|
- **Hardware**: The model was trained with XPU (Intel(R) Data Center GPU Max 1100) provided by Intel mentors at the Berkeley AI Hackathon. |
|
|
|
## Limitations |
|
|
|
- **Current Limitations**: The model may struggle with images containing overlapping ingredients or items not seen during training. It also only has a limited dataset, so different backgrounds or types of ingredients would not likely not be counted. |
|
- **Potential Improvements**: Incorporate a full-scale dataset with labelled ingredient counts to make the model more knowledgable and applicable within our application. Tune hyperparameters beyond what was done during the Hackathon to optimize performance. |
|
|
|
## Additional Information |
|
|
|
- **License**: Model released under MIT License. |
|
- **Contact Information**: For inquiries, contact [[email protected]](mailto:[email protected]). |
|
- **Acknowledgments**: Acknowledge Intel team for starter code and assistance in fine-tuning. |
|
|
