---
license: mit
---
## Development Process
This model was created with the intention of supplmenting our mobile application as an ingredient counter. Such that a user could take an image and have AI determine whether they have the necessary ingredients to create a recipe. As a result of the straightforward usecase, we were motivated to use a lightweight model for this classification task. We were recommended Moondream by a mentor from Intel, which we fine-tuned in this project.

## Model Details

- **Model Name**: Moondream Fine-tuned Variant
- **Task**: Ingredient Classification and Counting
- **Repository Name**: nyap/cosmo-demo
- **Original Model**: vikhyatk/moondream2

## Intended Use

- **Primary Use Case**: This model counts and classifies ingredients (exclusively avocados, bananas, doritos, and onions during CalHacks) in images.
- **Associated Application**: This model is applied through a cooking assistant mobile application built in React-native.

## Data

- **Dataset**: Used a very small dataset of a little less than 100 images collected during the Hackathon. Including images of avocados, bananas, doritos, and onions from various angles and in different combinations.
- **Data Collection**: Images were taken of various tables with the ingredients arranged, before being manually labelled with the count/type of ingredients in each image.

## Evaluation Metrics

- **Performance Metrics**: Performance metrics did not provide any valuable feedback due to the size of the dataset (overfit)

## Training

- **Training Procedure**: Created a basic dataloader class to read image data and expected output. PIL images were then passed through the VIT provided with Moondream. Moondream was then fine-tuned using the default Adam optimizer with a batch size of 8 for 30 epochs.
- **Hardware**: The model was trained with XPU (Intel(R) Data Center GPU Max 1100) provided by Intel mentors at the Berkeley AI Hackathon.

## Limitations

- **Current Limitations**: The model may struggle with images containing overlapping ingredients or items not seen during training. It also only has a limited dataset, so different backgrounds or types of ingredients would not likely not be counted.
- **Potential Improvements**: Incorporate a full-scale dataset with labelled ingredient counts to make the model more knowledgable and applicable within our application. Tune hyperparameters beyond what was done during the Hackathon to optimize performance.

## Additional Information

- **License**: Model released under MIT License.
- **Contact Information**: For inquiries, contact [nyap@umich.edu](mailto:nyap@umich.edu).
- **Acknowledgments**: Acknowledge Intel team for starter code and assistance in fine-tuning.