Feature/add pytest workflow

CHANGELOG_IMPROVEMENTS.md

@@ -0,0 +1,216 @@
+# Changelog - Advanced RAG Improvements
+
+## 2025-11-13 - Production-Ready Enhancements
+
+### 🔧 Fixed Critical Issues
+
+#### 1. **Missing UnstructuredPDFLoader Implementation**
+- **File**: `langchain-crash-course/5_agents_tools/rag_pdf_advanced.py`
+- **Issue**: Function claimed to use advanced PDF parsing but only used basic PyPDFLoader
+- **Fix**: 
+  - Added proper `UnstructuredPDFLoader` import and usage
+  - Configured with `mode="elements"` and `strategy="hi_res"` for optimal parsing
+  - Implemented correct fallback logic: UnstructuredPDFLoader → PyPDFLoader
+  - Added detailed logging for which loader succeeded
+
+**Before:**
+```python
+# Misleading comment, only used PyPDFLoader
+try:
+    loader = PyPDFLoader(file_path=str(file_path))
+    docs = loader.load()
+```
+
+**After:**
+```python
+# Try UnstructuredPDFLoader first for advanced parsing
+try:
+    from langchain_community.document_loaders import UnstructuredPDFLoader
+    loader = UnstructuredPDFLoader(
+        file_path=str(file_path),
+        mode="elements",
+        strategy="hi_res"
+    )
+    docs = loader.load()
+except (ImportError, Exception) as e:
+    # Fallback to PyPDFLoader
+    loader = PyPDFLoader(file_path=str(file_path))
+    docs = loader.load()
+```
+
+#### 2. **Inverted Fallback Logic**
+- **Issue**: Code structure suggested UnstructuredPDFLoader was tried first, but it wasn't
+- **Fix**: Proper try-except nesting with primary loader first, fallback second
+
+#### 3. **Verbose Error Handling**
+- **Issue**: Used `str(object=e)` instead of `str(e)`
+- **Fix**: Simplified to standard Python error string conversion
+
+---
+
+### 🚀 Added Production Features
+
+#### 1. **Model Pre-download Script**
+- **File**: `scripts/setup_models.py`
+- **Purpose**: Pre-download HuggingFace embeddings for production/offline use
+- **Features**:
+  - Downloads `BAAI/bge-large-en-v1.5` (~1.34 GB)
+  - Checks disk space before downloading
+  - Verifies model works after download
+  - Shows progress and cache location
+  - Handles errors gracefully
+
+**Usage:**
+```bash
+python scripts/setup_models.py
+```
+
+#### 2. **Smart Model Cache Detection**
+- **File**: `langchain-crash-course/5_agents_tools/rag_pdf_advanced.py`
+- **Function**: `create_multimodal_embeddings()`
+- **Features**:
+  - Checks if model is already cached
+  - Warns user if download will occur (with time estimate)
+  - Suggests running setup script to avoid delay
+  - Logs whether loading from cache or downloading
+
+**Benefits:**
+- Users know what to expect (no surprise 15-minute waits)
+- Clear guidance on how to avoid delays
+- Better production deployment experience
+
+---
+
+### 📚 Documentation Improvements
+
+#### 1. **Main README.md**
+- Added comprehensive setup instructions
+- Documented model pre-download process
+- Explained why pre-downloading is recommended
+- Added system dependency installation (Tesseract)
+- Included environment variable configuration
+
+#### 2. **Module-Specific README**
+- **File**: `langchain-crash-course/5_agents_tools/README.md`
+- **Content**:
+  - Complete feature documentation
+  - Usage examples for all scripts
+  - Troubleshooting guide
+  - Alternative model suggestions
+  - Directory structure explanation
+  - Step-by-step quick start guide
+
+#### 3. **Setup Guide**
+- **File**: `SETUP_GUIDE.md`
+- **Content**:
+  - Quick 5-minute setup
+  - Testing instructions
+  - Docker setup (optional)
+  - Common troubleshooting
+  - Next steps guidance
+
+---
+
+### 📊 Impact Summary
+
+| Improvement | Before | After |
+|-------------|--------|-------|
+| **PDF Parsing** | Basic text only | Tables, layouts, structure |
+| **First Run Time** | Unknown (surprise wait) | Predictable (with warning) |
+| **Offline Support** | No guidance | Clear setup process |
+| **Error Messages** | Generic | Specific with solutions |
+| **Documentation** | Minimal | Comprehensive |
+| **Production Ready** | ❌ | ✅ |
+
+---
+
+### 🎯 Benefits for Users
+
+#### Developers
+- ✅ Clear setup process
+- ✅ No surprise delays
+- ✅ Better error messages
+- ✅ Comprehensive documentation
+
+#### Production/Deployment
+- ✅ Pre-download models in build process
+- ✅ Offline capability
+- ✅ Predictable behavior
+- ✅ Docker-ready
+
+#### Advanced Users
+- ✅ Proper table extraction
+- ✅ Multi-loader strategy
+- ✅ Fallback mechanisms
+- ✅ Alternative model options
+
+---
+
+### 📝 Files Modified
+
+1. `langchain-crash-course/5_agents_tools/rag_pdf_advanced.py`
+   - Fixed `load_pdf_documents_advanced()` function
+   - Enhanced `create_multimodal_embeddings()` function
+
+2. `README.md`
+   - Added setup instructions
+   - Documented model pre-download
+
+3. `langchain-crash-course/5_agents_tools/README.md`
+   - Complete rewrite with comprehensive documentation
+
+### 📝 Files Created
+
+1. `scripts/setup_models.py`
+   - Model pre-download utility
+
+2. `SETUP_GUIDE.md`
+   - Quick setup reference
+
+3. `CHANGELOG_IMPROVEMENTS.md`
+   - This file
+
+---
+
+### 🔜 Future Enhancements (Suggestions)
+
+1. **Progress Bar for Downloads**
+   - Add `tqdm` progress bar to model downloads
+
+2. **Model Version Pinning**
+   - Pin specific model versions for reproducibility
+
+3. **Batch PDF Processing**
+   - Add parallel processing for multiple PDFs
+
+4. **Custom Model Support**
+   - Allow users to specify alternative embedding models via config
+
+5. **Health Check Script**
+   - Verify all dependencies and models are properly installed
+
+---
+
+### ✅ Testing Checklist
+
+- [x] UnstructuredPDFLoader properly imported and used
+- [x] Fallback to PyPDFLoader works
+- [x] Model cache detection works
+- [x] Setup script downloads model successfully
+- [x] Documentation is clear and comprehensive
+- [x] Error messages are helpful
+- [x] Logging provides useful information
+
+---
+
+## Conclusion
+
+These improvements transform the advanced RAG system from a prototype into a production-ready solution with:
+- **Proper advanced PDF parsing** (tables, layouts)
+- **Predictable deployment** (pre-download models)
+- **Excellent documentation** (setup, usage, troubleshooting)
+- **Graceful degradation** (fallback mechanisms)
+- **Clear user guidance** (warnings, suggestions)
+
+The system is now ready for production use with offline capability and comprehensive documentation.
+

README.md

@@ -16,7 +16,87 @@ This repository contains a crash course on Langchain, a framework for developing
 ### Prerequisites
 
 - Python 3.10 or 3.11
-- Poetry
+- Poetry or uv package manager
+
+## Setup Instructions
+
+### 1. Install Dependencies
+
+```bash
+# Using uv (recommended)
+uv sync
+
+# Or using pip
+pip install -e .
+```
+
+### 2. Pre-download AI Models (Recommended)
+
+For production use or offline environments, pre-download the required AI models:
+
+```bash
+python scripts/setup_models.py
+```
+
+This will download:
+
+- **HuggingFace Embeddings**: `BAAI/bge-large-en-v1.5` (~1.34 GB)
+- Used by advanced RAG features in `5_agents_tools/rag_pdf_advanced.py`
+
+**Why pre-download?**
+
+- ✅ Faster first run (no 5-15 minute wait)
+- ✅ Works offline after initial download
+- ✅ Predictable deployment
+- ✅ Avoids network timeouts in restricted environments
+
+**Skip this step if:**
+
+- You're just experimenting (models auto-download on first use)
+- You have fast, reliable internet
+- You don't mind the initial wait
+
+### 3. Configure Environment Variables
+
+Create a `.env` file in the root directory with your API keys:
+
+```bash
+# OpenAI (optional)
+OPENAI_API_KEY=your_openai_key_here
+
+# Google (optional)
+GOOGLE_API_KEY=your_google_key_here
+
+# Anthropic (optional)
+ANTHROPIC_API_KEY=your_anthropic_key_here
+
+# Tavily (for web search, optional)
+TAVILY_API_KEY=your_tavily_key_here
+```
+
+### 4. Install System Dependencies (for Advanced PDF Processing)
+
+**For OCR and image processing:**
+
+**Windows:**
+
+```bash
+# Install Tesseract OCR
+# Download from: https://github.com/UB-Mannheim/tesseract/wiki
+# Add to PATH: C:\Program Files\Tesseract-OCR
+```
+
+**macOS:**
+
+```bash
+brew install tesseract
+```
+
+**Linux:**
+
+```bash
+sudo apt-get install tesseract-ocr
+```
 
 ### 1_chat_models