Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Permesso di Soggiorno Assistant 🇮🇹

A comprehensive Advanced RAG (Retrieval-Augmented Generation) system to assist users with Italian residence permit procedures. This project implements sophisticated data ingestion, smart chunking, scenario-based retrieval, and legal precision answering.

🚀 Features

Core Capabilities

  • Conversational Assistant: Intuitive chat interface built with Streamlit
  • Knowledge Base: Processed PDF documents, web pages (Crawl4AI), and text files from Italian government sources
  • Local Models: Ollama with Qwen3 14B for the LLM and nomic-embed-text for embeddings
  • Vector Database: ChromaDB for semantic retrieval with persistence

Advanced RAG Features

  • Smart Chunking: Semantic chunking preserving legal document structure (sections, articles, paragraphs)
  • Source Classification: Automatic categorization of URLs into official_gov, legal_texts, case_studies, informative
  • Scenario Detection: Identifies complex cases like mixed marriages, work permits, family reunification, asylum
  • Query Boosting: Enhances retrieval for specific scenarios with Italian legal terminology
  • Precedent Extraction: Extracts and stores documented case examples for quick reference
  • Confidence Scoring: Evaluates answer reliability and suggests official consultation when needed
  • Citation Tracking: Each claim attributed to source documents with numbered references
  • Automated Updates: Daily scheduled crawls of official government sources with change detection
  • Version Control: Tracks document versions with audit trail (keeps last 5 versions)

📁 Project Structure (Advanced RAG Implementation)

permesso-bot/
├── config/                      # Configuration and settings
│   └── settings.py              # Pydantic BaseSettings with all configuration options
├── src/
│   ├── core/                    # Core RAG logic and answer generation
│   │   ├── rag_engine.py        # Main RAG engine with retrieval and response generation
│   │   ├── embeddings.py        # Embedding model configuration (nomic-embed-text)
│   │   ├── llm.py               # LLM interface (Ollama Qwen3 14B)
│   │   ├── scenario_manager.py  # Scenario classification and query boosting
│   │   ├── precedent_extractor.py # Case example extraction from documents
│   │   └── prompt_templates.py  # Legal-specific Italian prompt templates
│   ├── ingestion/               # Data ingestion pipeline (Phase 1-2, 6)
│   │   ├── __init__.py          # Module exports
│   │   ├── web_crawler.py       # Async Crawl4AI-based web crawler with retry logic
│   │   ├── source_classifier.py # URL classification into categories
│   │   ├── pdf_loader.py        # PDF document loading and processing
│   │   ├── text_processor.py    # Text file ingestion
│   │   └── pipelines/           # Data pipeline orchestration
│   │       ├── __init__.py
│   │   ├── processors/          # Document processing modules (Phase 3)
│   │   │   ├── __init__.py
│   │   │   ├── document_processor.py # Smart chunking with Italian legal structure
│   │   │   └── metadata_enricher.py # Scenario tagging and metadata extraction
│   │   └── schedulers/          # Automated update scheduling (Phase 6)
│   │       ├── __init__.py
│   │       └── update_scheduler.py # Daily crawl scheduler with APScheduler
│   ├── database/                # Database management (Phase 3, 6)
│   │   ├── vector_store.py      # ChromaDB operations with advanced queries
│   │   ├── metadata_manager.py  # Metadata storage and retrieval
│   │   └── version_manager.py   # Document version tracking and comparison
│   └── ui/                      # User interface (Streamlit)
│       └── chat_interface.py    # Chat interface with confidence display
├── data/                        # Data directory
│   ├── raw/                     # Raw documents (PDFs, text files)
│   ├── processed/               # Processed and chunked documents
│   ├── chroma_db/               # ChromaDB vector database (persistent)
│   └── update_log.json          # Update activity log
├── scripts/                     # Utility scripts
│   ├── ingest_data.py           # Manual data ingestion script
│   └── check_db.py              # Database statistics checker
├── tests/                       # Test suite (Phase 7)
│   └── test_advanced_rag.py     # Comprehensive pytest tests for all phases
├── .env.example                 # Configuration example with all settings
├── requirements.txt             # Python dependencies
└── main.py                      # Main entry point

## 🛠️ Installation

### 1. Clone the repository

```bash
git clone <repository-url>
cd permesso-bot

2. Create and activate virtual environment

python -m venv env
source env/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Configure Ollama

Ensure Ollama is installed and running:

# Install Ollama (if not already installed)
curl -fsSL https://ollama.com/install.sh | sh

# Pull the required models
ollama pull llama3.2:1b
ollama pull nomic-embed-text

5. Configure environment variables

cp .env.example .env
# Edit .env with your settings

📖 Usage

Data Ingestion

# Run the ingestion script
python scripts/ingest_data.py

Or, for specific sources:

# From PDFs
python -c "from src.ingestion.pdf_loader import load_pdfs; docs = load_pdfs()"

# From text files
python -c "from src.ingestion.text_processor import process_text_files; docs = process_text_files()"

# From URLs
python -c "from src.ingestion.html_crawler import crawl_html; docs = crawl_html()"

Start the Chat Interface

streamlit run main.py

The interface will be available at http://localhost:8501

🔧 Configuration (Advanced Settings)

Edit config/settings.py or .env to customize all RAG parameters:

Model Configuration

OLLAMA_BASE_URL = "http://localhost:11434"
LLM_MODEL = "qwen3:14b"  # or llama3.2:1b for lighter models
EMBEDDING_MODEL = "nomic-embed-text"

RAG Parameters

CHUNK_SIZE = 800           # Maximum tokens per chunk
CHUNK_OVERLAP = 150        # Token overlap between chunks
TOP_K_RESULTS = 10         # Number of documents to retrieve
SIMILARITY_THRESHOLD = 0.7 # Minimum similarity score for retrieval

Web Crawling Settings (Phase 2)

WEB_CRAWLER_ENABLED = True
OFFICIAL_SOURCES_URL = "https://portaleimmigrazione.it/"
MAX_CONCURRENT_REQUESTS = 3
CRAWL_TIMEOUT_SECONDS = 60
DATA_SOURCE_CATEGORIES = ["official_gov", "legal_texts", "case_studies"]
COMPLEX_CASE_PRIORITY = 1.5  # Boost factor for special scenarios
UPDATE_SCHEDULE_HOURS = 24   # Daily refresh interval

Scenario Classification (Phase 4)

SCENARIO_TYPES = [
    "standard",
    "mixed_marriage_italian_citizen",  # Cittadino italiano con coniuge straniero
    "work_permit",                      # Permesso per lavoro
    "family_reunification",             # Ricongiungimento familiare
    "asylum"                            # Asilo politico
]

Version Control (Phase 6)

KEEP_VERSIONS = 5           # Number of versions to retain per document
VERSION_HISTORY_PATH = "./data/document_versions.json"

🏗️ Advanced RAG Implementation Phases (1-7)

Phase 1 - Foundation & Setup

Step 1.1: Configuration Settings (config/settings.py)

  • web_crawler_enabled: Toggle for Crawl4AI web crawling
  • official_sources_url: Default Italian government portal URL
  • max_concurrent_requests: Rate limiting (default: 3)
  • crawl_timeout_seconds: Request timeout (default: 60)
  • data_source_categories: ["official_gov", "legal_texts", "case_studies"]
  • complex_case_priority: Boost factor for special scenarios (1.5)
  • update_schedule_hours: Daily refresh interval (24 hours)

Step 1.2: Ingestion Pipeline Structure (src/ingestion/)

src/ingestion/
├── pipelines/           # Data processing pipeline orchestration
├── processors/          # Document parsing and chunking
│   ├── document_processor.py    # Smart legal text chunking
│   └── metadata_enricher.py     # Scenario tagging & enrichment
├── schedulers/          # Automated update management
│   └── update_scheduler.py      # Daily crawl scheduling
├── web_crawler.py       # Async Crawl4AI implementation
├── source_classifier.py # URL categorization system
├── pdf_loader.py        # PDF document extraction
├── html_crawler.py      # Legacy HTML processing
└── text_processor.py    # Text file handling

Phase 2 - Web Crawling & Data Acquisition

Step 2.1: AsyncWebCrawler (src/ingestion/web_crawler.py)

  • Uses Crawl4AI's AsyncWebCrawler for markdown extraction
  • Supports Italian government domains: .gov.it, portaleimmigrazione.it, ministero.interno.it
  • Exponential backoff retry logic (max 3 retries)
  • Rate limiting: max 1 request/second, respects robots.txt
  • Metadata storage: source_url, crawl_timestamp, page_title, content_type

Step 2.2: SourceClassifier (src/ingestion/source_classifier.py)

class SourceClassifier:
    def classify_url(url: str) -> Dict[category, confidence]:
        # Returns category and confidence score (0.0-1.0)
    
    def is_trusted_source(url: str) -> bool:
        # Validates against whitelist of official sources

Categories: official_gov (.gov.it domains), legal_texts, case_studies, informative

Phase 3 - Data Processing & Database Construction

Step 3.1: DocumentProcessor (src/ingestion/processors/document_processor.py)

  • Chunk size: 800 tokens max, 150 token overlap
  • Preserves Italian legal structure: Articolo X, comma Y, lettera z
  • Semantic chunking based on markdown headings (##, ###)
  • Metadata per chunk: source_url, document_type, section_path

Step 3.2: MetadataEnricher (src/ingestion/processors/metadata_enricher.py)

  • Detects complex case indicators via regex patterns:
    • "cittadino italiano" + "coniuge extracomunitario" → mixed_marriage
    • "lavoro" + "permesso" → work_permit
    • "famiglia" + "ricongiungimento" → family_reunification
  • Extracts dates, article references, official document numbers

Step 3.3: VectorStore Updates (src/database/vector_store.py)

def add_documents_with_metadata(documents, metadatas, collection_name)
def search_with_category_filter(query, category, k=10)
def search_by_scenario(query, scenario_id, k=10)
def get_collection_statistics() -> Dict[category_counts, last_update]

Phase 4 - Complex Case Handling

Step 4.1: ScenarioManager (src/core/scenario_manager.py)

class ScenarioManager:
    SCENARIOS = ["standard", "mixed_marriage_italian_citizen", "work_permit", 
                  "family_reunification", "asylum"]
    
    def identify_scenarios(query) -> List[str]
    def get_scenario_keywords(scenario_id) -> List[str]  # Italian terminology
    def boost_query_for_scenario(query, scenario_ids) -> str

Step 4.2: RAG Engine Updates (src/core/rag_engine.py)

def retrieve_for_complex_scenario(query, scenario_type) -> List[Dict]:
    # Boosts retrieval with scenario keywords if confidence < 0.7
    
def get_case_precedents(scenario_type, limit=5) -> List[Dict]

Step 4.3: PrecedentExtractor (src/core/precedent_extractor.py)

  • Pattern matching: "caso pratico", "esempio", "precedente"
  • Extracts: scenario, key facts, outcome, source reference
  • Stores precedents separately for quick retrieval

Phase 5 - Enhanced Answer Generation

Step 5.1: PromptTemplates (src/core/prompt_templates.py)

SYSTEM_PROMPT_LEGAL = """Strict rules: cite sources, don't hallucinate, 
acknowledge uncertainty. All responses in Italian."""

RESPONSE_FORMAT_STANDARD = """Risposta diretta | Documenti di riferimento | 
Note importanti"""

RESPONSE_FORMAT_COMPLEX = """Prerequisiti | Step-by-step procedure | 
Avvertenze e disclaimer"""

Step 5.2: Citation Tracking in RAG Engine

  • Response format: "Articolo 5 del TUI [1], comma 2 [1]"
  • Tracks number of documents used per response
  • Lists all source URLs at bottom with numbered references

Step 5.3: Confidence Scoring

def calculate_answer_confidence(context, query) -> Dict[score, sources_relevant, coverage]:
    # Returns score (0.0-1.0)
    # If score < 0.6: suggests official portal consultation

Phase 6 - Continuous Updates

Step 6.1: UpdateScheduler (src/ingestion/schedulers/update_scheduler.py)

  • Daily crawls of official sources using APScheduler or schedule library
  • Incremental updates: only new/modified documents
  • Logs to ./data/update_log.json
  • CLI trigger support

Step 6.2: ChangeDetector (src/ingestion/change_detector.py)

  • Content hashing with SHA256 for change detection
  • Version history in ./data/document_versions.json
  • Detects document version changes (new article numbers, date updates)

Step 6.3: VersionManager (src/database/version_manager.py)

def get_document_version_history(doc_id) -> List[Dict[version, date, changes]]
def compare_versions(doc_id, v1, v2) -> Dict[added, removed]
# Keeps last 5 versions per document for audit trail

Phase 7 - Testing & Validation

Step 7: Test Suite (tests/test_advanced_rag.py)

  • Unit tests with pytest fixtures for mock data
  • Integration test: full pipeline on sample query
  • Tests cover all phases from crawling to response generation

🧪 Testing

python -m pytest tests/

🔐 Security

  • Never commit the .env file
  • Use .gitignore to protect sensitive files
  • Keep database paths outside the repository if possible

📝 License

[Your license here]

🤝 Contributing

Contributions are welcome! Please create a PR with your changes.

📞 Support

For issues or questions, open an issue on GitHub.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages