docs: update MiniDoc and README.md for improved clarity and feature descriptions

Docs/MiniDoc.md

@@ -10,32 +10,41 @@ CrawlGPT is a web content crawler with GPT-powered summarization and chat capabi
 crawlgpt/
 ├── src/
 │   └── crawlgpt/
-│       ├── core/
-│       │   ├── DatabaseHandler.py
-│       │   ├── LLMBasedCrawler.py 
-│       │   └── SummaryGenerator.py
-│       ├── ui/
-│       │   ├── chat_app.py
-│       │   └── chat_ui.py
-│       └── utils/
-│           ├── content_validator.py
-│           ├── data_manager.py
-│           ├── helper_functions.py
-│           ├── monitoring.py
-│           └── progress.py
-├── tests/
+│       ├── core/                           # Core functionality
+│       │   ├── database.py                 # SQL database handling
+│       │   ├── LLMBasedCrawler.py         # Main crawler implementation
+│       │   ├── DatabaseHandler.py          # Vector database (FAISS)
+│       │   └── SummaryGenerator.py         # Text summarization
+│       ├── ui/                            # User Interface
+│       │   ├── chat_app.py                # Main Streamlit app
+│       │   ├── chat_ui.py                 # Development UI
+│       │   └── login.py                   # Authentication UI
+│       └── utils/                         # Utilities
+│           ├── content_validator.py        # URL/content validation
+│           ├── data_manager.py            # Import/export handling
+│           ├── helper_functions.py         # General helpers
+│           ├── monitoring.py              # Metrics collection
+│           └── progress.py                # Progress tracking
+├── tests/                                # Test suite
 │   └── test_core/
-│       ├── test_database_handler.py
-│       ├── test_integration.py
-│       ├── test_llm_based_crawler.py
-│       └── test_summary_generator.py
-├── .gitignore
-├── LICENSE
-├── README.md
-├── Docs
-├── pyproject.toml
-├── pytest.ini
-└── setup_env.py
+│       ├── test_database_handler.py       # Vector DB tests
+│       ├── test_integration.py           # Integration tests
+│       ├── test_llm_based_crawler.py     # Crawler tests
+│       └── test_summary_generator.py     # Summarizer tests
+├── .github/                             # CI/CD
+│   └── workflows/
+│       └── Push_to_hf.yaml              # HuggingFace sync
+├── Docs/
+│   └── MiniDoc.md                       # Documentation
+├── .dockerignore                        # Docker exclusions
+├── .gitignore                          # Git exclusions
+├── Dockerfile                          # Container config
+├── LICENSE                             # MIT License
+├── README.md                          # Project documentation
+├── README_hf.md                       # HuggingFace README
+├── pyproject.toml                     # Project metadata
+├── pytest.ini                         # Test configuration
+└── setup_env.py                       # Environment setup
 ```
 
 ## Core Components
@@ -59,6 +68,24 @@ crawlgpt/
 -   Configurable model selection and parameters
 -   Handles empty input validation
 
+### [Database](https://github.com/Jatin-Mehra119/CRAWLGPT/blob/main/src/crawlgpt/core/database.py) (src/crawl/core/database.py)
+
+-   SQLAlchemy-based database handling for user management and chat history
+-   Provides secure user authentication with BCrypt password hashing
+-   Manages persistent storage of chat conversations and context  
+
+- Configuration
+    - Uses SQLite by default (`sqlite:///crawlgpt.db`)
+    - Configurable via DATABASE_URL environment variable
+    - Automatic schema creation on startup
+    - Session management with SQLAlchemy sessionmaker
+- Security Features
+    - BCrypt password hashing with PassLib
+    - Unique username enforcement
+    - Secure session handling
+    - Role-based message tracking
+
+
 ## UI Components
 
 ### [chat_app.py](https://github.com/Jatin-Mehra119/CRAWLGPT/blob/main/src/crawlgpt/ui/chat_app.py) (src/crawlgpt/ui/chat_app.py)

README.md

@@ -1,27 +1,42 @@
-# CRAWLGPT 🤖
+# CrawlGPT 🤖
 
-A powerful web content crawler with LLM-powered summarization and chat capabilities. CRAWLGPT extracts content from URLs, stores it in a vector database (FAISS), and enables natural language querying of the stored content. It combines modern web crawling technology with advanced language models to help you extract, analyze, and interact with web content intelligently.
+A powerful web content crawler with LLM-powered RAG (Retrieval Augmented Generation) capabilities. CrawlGPT extracts content from URLs, processes it through intelligent summarization, and enables natural language interactions using modern LLM technology.
 
-## 🌟 Features
+## 🌟 Key Features
 
--   **Web Crawling**  
-    Async-based crawling powered by [crawl4ai](https://pypi.org/project/crawl4ai/) and Playwright.  
-    Includes configurable rate limiting and content validation.
-    
--   **Content Processing**  
-    Automatically chunks large texts, generates embeddings, and summarizes text via the Groq API.
-    
--   **Chat Interface**  
-    Streamlit-based UI with a user-friendly chat panel.  
-    Supports summarized or full-text retrieval (RAG) for context injection.
-    
--   **Data Management**  
-    Stores content in a local or in-memory vector database (FAISS) for efficient retrieval.  
-    Tracks usage metrics and supports import/export of system state.
-    
--   **Testing**  
-    Comprehensive unit and integration tests using Python’s `unittest` framework.
-    
+### Core Features
+- **Intelligent Web Crawling**
+  - Async web content extraction using Playwright
+  - Smart rate limiting and validation
+  - Configurable crawling strategies
+  
+- **Advanced Content Processing** 
+  - Automatic text chunking and summarization
+  - Vector embeddings via FAISS
+  - Context-aware response generation
+
+- **Streamlit Chat Interface**
+  - Clean, responsive UI
+  - Real-time content processing
+  - Conversation history
+  - User authentication
+
+### Technical Features
+- **Vector Database**
+  - FAISS-powered similarity search
+  - Efficient content retrieval
+  - Persistent storage
+
+- **User Management**
+  - SQLite database backend
+  - Secure password hashing
+  - Chat history tracking
+
+- **Monitoring & Utils**
+  - Request metrics collection
+  - Progress tracking
+  - Data import/export
+  - Content validation
 
 ## 🎥 Demo
 ### [Deployed APP 🚀🤖](https://huggingface.co/spaces/jatinmehra/CRAWL-GPT-CHAT)
@@ -109,35 +124,41 @@ _Example of CRAWLGPT in action!_
 crawlgpt/
 ├── src/
 │   └── crawlgpt/
-│       ├── core/
-│       │   ├── DatabaseHandler.py
-│       │   ├── LLMBasedCrawler.py 
-│       │   └── SummaryGenerator.py
-│       ├── ui/
-│       │   ├── chat_app.py
-│       │   └── chat_ui.py
-│       └── utils/
-│           ├── content_validator.py
-│           ├── data_manager.py
-│           ├── helper_functions.py
-│           ├── monitoring.py
-│           └── progress.py
-├── tests/
+│       ├── core/                           # Core functionality
+│       │   ├── database.py                 # SQL database handling
+│       │   ├── LLMBasedCrawler.py         # Main crawler implementation
+│       │   ├── DatabaseHandler.py          # Vector database (FAISS)
+│       │   └── SummaryGenerator.py         # Text summarization
+│       ├── ui/                            # User Interface
+│       │   ├── chat_app.py                # Main Streamlit app
+│       │   ├── chat_ui.py                 # Development UI
+│       │   └── login.py                   # Authentication UI
+│       └── utils/                         # Utilities
+│           ├── content_validator.py        # URL/content validation
+│           ├── data_manager.py            # Import/export handling
+│           ├── helper_functions.py         # General helpers
+│           ├── monitoring.py              # Metrics collection
+│           └── progress.py                # Progress tracking
+├── tests/                                # Test suite
 │   └── test_core/
-│       ├── test_database_handler.py
-│       ├── test_integration.py
-│       ├── test_llm_based_crawler.py
-│       └── test_summary_generator.py
-├── .github/
+│       ├── test_database_handler.py       # Vector DB tests
+│       ├── test_integration.py           # Integration tests
+│       ├── test_llm_based_crawler.py     # Crawler tests
+│       └── test_summary_generator.py     # Summarizer tests
+├── .github/                             # CI/CD
 │   └── workflows/
-│       └── Push_to_hf.yaml
-├── .gitignore
-├── LICENSE
-├── README.md
-├── Docs
-├── pyproject.toml
-├── pytest.ini
-└── setup_env.py
+│       └── Push_to_hf.yaml              # HuggingFace sync
+├── Docs/
+│   └── MiniDoc.md                       # Documentation
+├── .dockerignore                        # Docker exclusions
+├── .gitignore                          # Git exclusions
+├── Dockerfile                          # Container config
+├── LICENSE                             # MIT License
+├── README.md                          # Project documentation
+├── README_hf.md                       # HuggingFace README
+├── pyproject.toml                     # Project metadata
+├── pytest.ini                         # Test configuration
+└── setup_env.py                       # Environment setup
 ``` 
 
 ## 🧪 Testing

README_hf.md

@@ -8,30 +8,45 @@ colorTo: blue
 pinned: true
 short_description: A powerful web content crawler with LLM-powered RAG.
 ---
-# CRAWLGPT 🤖
+# CrawlGPT 🤖
 
-A powerful web content crawler with LLM-powered summarization and chat capabilities. CRAWLGPT extracts content from URLs, stores it in a vector database (FAISS), and enables natural language querying of the stored content. It combines modern web crawling technology with advanced language models to help you extract, analyze, and interact with web content intelligently.
+A powerful web content crawler with LLM-powered RAG (Retrieval Augmented Generation) capabilities. CrawlGPT extracts content from URLs, processes it through intelligent summarization, and enables natural language interactions using modern LLM technology.
 
-## 🌟 Features
+## 🌟 Key Features
 
--   **Web Crawling**  
-    Async-based crawling powered by [crawl4ai](https://pypi.org/project/crawl4ai/) and Playwright.  
-    Includes configurable rate limiting and content validation.
-    
--   **Content Processing**  
-    Automatically chunks large texts, generates embeddings, and summarizes text via the Groq API.
-    
--   **Chat Interface**  
-    Streamlit-based UI with a user-friendly chat panel.  
-    Supports summarized or full-text retrieval (RAG) for context injection.
-    
--   **Data Management**  
-    Stores content in a local or in-memory vector database (FAISS) for efficient retrieval.  
-    Tracks usage metrics and supports import/export of system state.
-    
--   **Testing**  
-    Comprehensive unit and integration tests using Python’s `unittest` framework.
-    
+### Core Features
+- **Intelligent Web Crawling**
+  - Async web content extraction using Playwright
+  - Smart rate limiting and validation
+  - Configurable crawling strategies
+  
+- **Advanced Content Processing** 
+  - Automatic text chunking and summarization
+  - Vector embeddings via FAISS
+  - Context-aware response generation
+
+- **Streamlit Chat Interface**
+  - Clean, responsive UI
+  - Real-time content processing
+  - Conversation history
+  - User authentication
+
+### Technical Features
+- **Vector Database**
+  - FAISS-powered similarity search
+  - Efficient content retrieval
+  - Persistent storage
+
+- **User Management**
+  - SQLite database backend
+  - Secure password hashing
+  - Chat history tracking
+
+- **Monitoring & Utils**
+  - Request metrics collection
+  - Progress tracking
+  - Data import/export
+  - Content validation
 
 ## 🎥 Demo
 ### [Deployed APP 🚀🤖](https://huggingface.co/spaces/jatinmehra/CRAWL-GPT-CHAT)
@@ -119,35 +134,41 @@ _Example of CRAWLGPT in action!_
 crawlgpt/
 ├── src/
 │   └── crawlgpt/
-│       ├── core/
-│       │   ├── DatabaseHandler.py
-│       │   ├── LLMBasedCrawler.py 
-│       │   └── SummaryGenerator.py
-│       ├── ui/
-│       │   ├── chat_app.py
-│       │   └── chat_ui.py
-│       └── utils/
-│           ├── content_validator.py
-│           ├── data_manager.py
-│           ├── helper_functions.py
-│           ├── monitoring.py
-│           └── progress.py
-├── tests/
+│       ├── core/                           # Core functionality
+│       │   ├── database.py                 # SQL database handling
+│       │   ├── LLMBasedCrawler.py         # Main crawler implementation
+│       │   ├── DatabaseHandler.py          # Vector database (FAISS)
+│       │   └── SummaryGenerator.py         # Text summarization
+│       ├── ui/                            # User Interface
+│       │   ├── chat_app.py                # Main Streamlit app
+│       │   ├── chat_ui.py                 # Development UI
+│       │   └── login.py                   # Authentication UI
+│       └── utils/                         # Utilities
+│           ├── content_validator.py        # URL/content validation
+│           ├── data_manager.py            # Import/export handling
+│           ├── helper_functions.py         # General helpers
+│           ├── monitoring.py              # Metrics collection
+│           └── progress.py                # Progress tracking
+├── tests/                                # Test suite
 │   └── test_core/
-│       ├── test_database_handler.py
-│       ├── test_integration.py
-│       ├── test_llm_based_crawler.py
-│       └── test_summary_generator.py
-├── .github/
+│       ├── test_database_handler.py       # Vector DB tests
+│       ├── test_integration.py           # Integration tests
+│       ├── test_llm_based_crawler.py     # Crawler tests
+│       └── test_summary_generator.py     # Summarizer tests
+├── .github/                             # CI/CD
 │   └── workflows/
-│       └── Push_to_hf.yaml
-├── .gitignore
-├── LICENSE
-├── README.md
-├─��� Docs
-├── pyproject.toml
-├── pytest.ini
-└── setup_env.py
+│       └── Push_to_hf.yaml              # HuggingFace sync
+├── Docs/
+│   └── MiniDoc.md                       # Documentation
+├── .dockerignore                        # Docker exclusions
+├── .gitignore                          # Git exclusions
+├── Dockerfile                          # Container config
+├── LICENSE                             # MIT License
+├── README.md                          # Project documentation
+├── README_hf.md                       # HuggingFace README
+├── pyproject.toml                     # Project metadata
+├── pytest.ini                         # Test configuration
+└── setup_env.py                       # Environment setup
 ``` 
 
 ## 🧪 Testing

src/crawlgpt/core/database.py

@@ -1,4 +1,6 @@
-# crawlgpt/src/crawlgpt/core/database.py
+# This module provides SQLAlchemy models and database utilities for user management 
+# and chat history persistence.
+
 from sqlalchemy import create_engine, Column, Integer, String, Text, DateTime, ForeignKey
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import sessionmaker, relationship
@@ -6,10 +8,23 @@ from datetime import datetime
 from passlib.context import CryptContext
 import os
 
+# SQLAlchemy models
 Base = declarative_base()
+
+# Password hashing
 pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
 
 class User(Base):
+    """User model for authentication and chat history tracking.
+    
+    Attributes:
+        id (int): Primary key
+        username (str): Unique username, max 50 chars
+        password_hash (str): BCrypt hashed password, 60 chars
+        email (str): User email, max 100 chars
+        created_at (datetime): Account creation timestamp
+        chats (relationship): One-to-many relationship to ChatHistory
+    """
     __tablename__ = 'users'
     id = Column(Integer, primary_key=True)
     username = Column(String(50), unique=True)
@@ -19,6 +34,17 @@ class User(Base):
     chats = relationship("ChatHistory", back_populates="user")
 
 class ChatHistory(Base):
+    """ChatHistory model for storing chat messages.
+    
+    Attributes:
+        id (int): Primary key
+        user_id (int): Foreign key to User
+        message (str): Chat message content
+        role (str): Role of the message sender ('user' or 'assistant')
+        context (str): Context of the chat message
+        timestamp (datetime): Timestamp of the message
+        user (relationship): Many-to-one relationship to User
+    """
     __tablename__ = 'chat_history'
     id = Column(Integer, primary_key=True)
     user_id = Column(Integer, ForeignKey('users.id'))
@@ -28,12 +54,22 @@ class ChatHistory(Base):
     timestamp = Column(DateTime, default=datetime.utcnow)
     user = relationship("User", back_populates="chats")
 
+# Database initialization
 engine = create_engine(os.getenv('DATABASE_URL', 'sqlite:///crawlgpt.db'))
 Base.metadata.create_all(bind=engine)
 Session = sessionmaker(bind=engine)
 
 # Database operations
 def create_user(username: str, password: str, email: str):
+    """
+    Creates a new user in the database
+    Args:
+        username (str): Username
+        password (str): Password
+        email (str): Email
+    Returns:
+        bool: True if user is created, False if username is taken
+    """
     with Session() as session:
         if session.query(User).filter(User.username == username).first():
             return False
@@ -44,6 +80,14 @@ def create_user(username: str, password: str, email: str):
     return True
 
 def authenticate_user(username: str, password: str):
+    """
+    Authenticates a user with a username and password
+    Args:
+        username (str): Username
+        password (str): Password
+    Returns:
+        User: User object if authentication is successful, None otherwise
+    """
     with Session() as session:
         user = session.query(User).filter(User.username == username).first()
         if user and pwd_context.verify(password, user.password_hash):
@@ -51,6 +95,17 @@ def authenticate_user(username: str, password: str):
     return None
 
 def save_chat_message(user_id: int, message: str, role: str, context: str):
+    """Saves a chat message to the database
+
+    Args:
+        user_id (int): User ID
+        message (str): Chat message content
+        role (str): Role of the message sender ('user' or 'assistant')
+        context (str): Context of the chat message
+
+    Returns:
+        None
+    """
     with Session() as session:
         chat = ChatHistory(
             user_id=user_id,
@@ -62,12 +117,27 @@ def save_chat_message(user_id: int, message: str, role: str, context: str):
         session.commit()
 
 def get_chat_history(user_id: int):
+    """
+    Retrieves chat history for a user
+    Args:
+        user_id (int): User ID
+        
+    Returns:
+        List[ChatHistory]: List of chat messages
+    """
     with Session() as session:
         return session.query(ChatHistory).filter(
             ChatHistory.user_id == user_id
         ).order_by(ChatHistory.timestamp).all()
         
 def delete_user_chat_history(user_id: int):
+    """Deletes all chat history for a user
+    Args:
+        user_id (int): User ID
+    
+    Returns:
+        None
+    """
     with Session() as session:
         session.query(ChatHistory).filter(
             ChatHistory.user_id == user_id
@@ -75,7 +145,19 @@ def delete_user_chat_history(user_id: int):
         session.commit()
         
 def restore_chat_history(user_id: int):
-    """Restores chat history from database to session state"""
+    """Restores chat history from database to session state
+    Args:
+        user_id (int): User ID
+    
+    Returns:
+        List[Dict]: List of chat messages in the format:
+            {
+                "role": str,
+                "content": str,
+                "context": str,
+                "timestamp": datetime
+            }
+        """
     with Session() as session:
         history = session.query(ChatHistory).filter(
             ChatHistory.user_id == user_id