pr/2

README.md

@@ -7,40 +7,65 @@ sdk: docker
 pinned: false
 ---
 
-# Classification Model for Climate Disinformation Classification
-
-## Global Informations
-
-The aim of this model is to detect illegal deforestation thanks to audio clips. Our objective is to make this AI system the most frugal possible.
-When you're new to AI for audio processing and you're looking for information on the Internet, the following methodology is often described: transform the audio signal into a spectrogram (2D image) and then have it analyzed by a CNN. It can be necessary when you're working on very precise task such as transcription, but it's too heavy for our task, which is simply to detect chainsaw noises. So, for our baseline, we used an mfcc transform to preprocess the data with, then we transformed the 2D output un a 1D by taking the mean for each feature and finally we applied a basic ml classification algorithm like Random Forest. 
-Then, we tried to optimize the 2 different stages: the data preprocessing (to make it simple, quick and to create the smallest possible preprocessed dataset) and the ml model itself. At this point, we noticed than the preprocessing of our data (with the mfcc transformation) was consuming 12 times more than our model training and inference so we mostly worked on optimizing the data preprocessing part. Here are our main ideas :
-1. The data preprocessing:
-   * Compare different methods of audio feature extraction 
-   * Decrease the size of the data to analyse (i.e. by decreasing the sampling rate, take only a small sample of the initial audio of 3s, since we don’t really need several seconds to identify a sound, remove unnecessary data (as with plots we can see that the characteristic sounds of chainsraw are observed between 200Hz & 1000Hz))
-   * Decrease the number of features extracted 
- 
-2. The ml model: 
-   * Use the most lightweighted model : Avoid neural networks and compare basic ml classification algorithms for example knn is 3 times less energy consuming than the Random Forest with a non-significant loss of accuracy
-
-## Submitted models
-Model 1:
-* Data preprocessing: Resampling to 6000Hz and using librosa mfcc method to calculate 7 MFCC 
-  * Energy consumption for the processing of the training dataset : 0.005349 kWh
-* Model: KNN 
-  * Energy consumption for the training of the model : 0.000002 kWh
-
-Model 2:
-* Data preprocessing: Resampling to 6000Hz and using librosa mfcc method to calculate 10 MFCC  
-    * Energy consumption for the processing of the training dataset : 0.005648 kWh
-* Model: Random Forest
-    * Energy consumption for the training of the model : 0.000326 kWh
-
-The 2nd model has better accuracy but is less energy efficient.
-
-## Other avenues for optimization 
-
-We learned a lot about all the possible optimizations but unfortunately, we didn't keep all of them for the final submission, as they  led to an excessive loss of precision (~85% accuracy), they are described below :
-* Use a less complex method of audio feature extraction (example: spectral centroid is 5 times faster than mfcc)
-* Focus our analysis on the frequencies that really matter for chainsaws (from 0 to 20,000Hz)
-* Use less data (shortest audio), i.e. keep randomly only 0.2s of audio for each audio of the dataset before extracting features with mfcc and then also take only a small sample of each audio of the test dataset to predict the label
 
+# Random Baseline Model for Climate Disinformation Classification
+
+## Model Description
+
+This is a random baseline model for the Frugal AI Challenge 2024, specifically for the text classification task of identifying climate disinformation. The model serves as a performance floor, randomly assigning labels to text inputs without any learning.
+
+### Intended Use
+
+- **Primary intended uses**: Baseline comparison for climate disinformation classification models
+- **Primary intended users**: Researchers and developers participating in the Frugal AI Challenge
+- **Out-of-scope use cases**: Not intended for production use or real-world classification tasks
+
+## Training Data
+
+The model uses the QuotaClimat/frugalaichallenge-text-train dataset:
+- Size: ~6000 examples
+- Split: 80% train, 20% test
+- 8 categories of climate disinformation claims
+
+### Labels
+0. No relevant claim detected
+1. Global warming is not happening
+2. Not caused by humans
+3. Not bad or beneficial
+4. Solutions harmful/unnecessary
+5. Science is unreliable
+6. Proponents are biased
+7. Fossil fuels are needed
+
+## Performance
+
+### Metrics
+- **Accuracy**: ~12.5% (random chance with 8 classes)
+- **Environmental Impact**:
+  - Emissions tracked in gCO2eq
+  - Energy consumption tracked in Wh
+
+### Model Architecture
+The model implements a random choice between the 8 possible labels, serving as the simplest possible baseline.
+
+## Environmental Impact
+
+Environmental impact is tracked using CodeCarbon, measuring:
+- Carbon emissions during inference
+- Energy consumption during inference
+
+This tracking helps establish a baseline for the environmental impact of model deployment and inference.
+
+## Limitations
+- Makes completely random predictions
+- No learning or pattern recognition
+- No consideration of input text
+- Serves only as a baseline reference
+- Not suitable for any real-world applications
+
+## Ethical Considerations
+
+- Dataset contains sensitive topics related to climate disinformation
+- Model makes random predictions and should not be used for actual classification
+- Environmental impact is tracked to promote awareness of AI's carbon footprint
+```

models/{audio_classification_knn.pkl → audio_classification_baseline.pkl}

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a54b7efa8866a57c8b8f45ca1b6b5d04caf83d0e79fb42cfa77ca6e375e1e5d7
-size 4811702
+oid sha256:7a27a9a671a920660995bc08b255e17449427f018402ceec81710a0ae93cb612
+size 36073945

tasks/audio.py

@@ -2,20 +2,21 @@ from fastapi import APIRouter
 from datetime import datetime
 from datasets import load_dataset
 from sklearn.metrics import accuracy_score
+import random
 import os
 import joblib
 import librosa
 import numpy as np
 
-from .utils.evaluation import AudioEvaluationRequest
-from .utils.emissions import tracker, clean_emissions_data, get_space_info
+from utils.evaluation import AudioEvaluationRequest
+from utils.emissions import tracker, clean_emissions_data, get_space_info
 
 from dotenv import load_dotenv
 load_dotenv()
 
 router = APIRouter()
 
-DESCRIPTION = "Model 1 : Knn audio classification"
+DESCRIPTION = "Random Baseline"
 ROUTE = "/audio"
 
 
@@ -42,8 +43,9 @@ async def evaluate_audio(request: AudioEvaluationRequest):
     dataset = load_dataset(request.dataset_name, token=os.getenv("HF_TOKEN"))
 
     # Split dataset
-    train_test = dataset["train"]
-    test_dataset = dataset["test"]
+    train_test = dataset["train"].train_test_split(
+        test_size=request.test_size, seed=request.test_seed)
+    test_dataset = train_test["test"]
 
     # Start tracking emissions
     tracker.start()
@@ -59,12 +61,12 @@ async def evaluate_audio(request: AudioEvaluationRequest):
         features = []
         for row in dataset:
             # Load the audio file and resample it
-            target_sr = 6000
+            target_sr = 25000
             audio = row['audio']['array']
             audio = librosa.resample(audio, orig_sr=12000, target_sr=target_sr)
 
             # Extract MFCC features
-            mfccs = librosa.feature.mfcc(y=audio, sr=target_sr, n_mfcc=7)
+            mfccs = librosa.feature.mfcc(y=audio, sr=target_sr, n_mfcc=40)
             mfccs_scaled = np.mean(mfccs.T, axis=0)
 
             # Append features and labels
@@ -74,7 +76,8 @@ async def evaluate_audio(request: AudioEvaluationRequest):
 
     X_test = preprocess(test_dataset)
 
-    classification_model = joblib.load("./models/audio_classification_knn.pkl")
+    classification_model = joblib.load(
+        "../models/audio_classification_baseline.pkl")
 
     predictions = classification_model.predict(X_test)
     true_labels = test_dataset["label"]