A comprehensive Advanced RAG (Retrieval-Augmented Generation) system to assist users with Italian residence permit procedures. This project implements sophisticated data ingestion, smart chunking, scenario-based retrieval, and legal precision answering.
- Conversational Assistant: Intuitive chat interface built with Streamlit
- Knowledge Base: Processed PDF documents, web pages (Crawl4AI), and text files from Italian government sources
- Local Models: Ollama with Qwen3 14B for the LLM and nomic-embed-text for embeddings
- Vector Database: ChromaDB for semantic retrieval with persistence
- Smart Chunking: Semantic chunking preserving legal document structure (sections, articles, paragraphs)
- Source Classification: Automatic categorization of URLs into official_gov, legal_texts, case_studies, informative
- Scenario Detection: Identifies complex cases like mixed marriages, work permits, family reunification, asylum
- Query Boosting: Enhances retrieval for specific scenarios with Italian legal terminology
- Precedent Extraction: Extracts and stores documented case examples for quick reference
- Confidence Scoring: Evaluates answer reliability and suggests official consultation when needed
- Citation Tracking: Each claim attributed to source documents with numbered references
- Automated Updates: Daily scheduled crawls of official government sources with change detection
- Version Control: Tracks document versions with audit trail (keeps last 5 versions)
permesso-bot/
├── config/ # Configuration and settings
│ └── settings.py # Pydantic BaseSettings with all configuration options
├── src/
│ ├── core/ # Core RAG logic and answer generation
│ │ ├── rag_engine.py # Main RAG engine with retrieval and response generation
│ │ ├── embeddings.py # Embedding model configuration (nomic-embed-text)
│ │ ├── llm.py # LLM interface (Ollama Qwen3 14B)
│ │ ├── scenario_manager.py # Scenario classification and query boosting
│ │ ├── precedent_extractor.py # Case example extraction from documents
│ │ └── prompt_templates.py # Legal-specific Italian prompt templates
│ ├── ingestion/ # Data ingestion pipeline (Phase 1-2, 6)
│ │ ├── __init__.py # Module exports
│ │ ├── web_crawler.py # Async Crawl4AI-based web crawler with retry logic
│ │ ├── source_classifier.py # URL classification into categories
│ │ ├── pdf_loader.py # PDF document loading and processing
│ │ ├── text_processor.py # Text file ingestion
│ │ └── pipelines/ # Data pipeline orchestration
│ │ ├── __init__.py
│ │ ├── processors/ # Document processing modules (Phase 3)
│ │ │ ├── __init__.py
│ │ │ ├── document_processor.py # Smart chunking with Italian legal structure
│ │ │ └── metadata_enricher.py # Scenario tagging and metadata extraction
│ │ └── schedulers/ # Automated update scheduling (Phase 6)
│ │ ├── __init__.py
│ │ └── update_scheduler.py # Daily crawl scheduler with APScheduler
│ ├── database/ # Database management (Phase 3, 6)
│ │ ├── vector_store.py # ChromaDB operations with advanced queries
│ │ ├── metadata_manager.py # Metadata storage and retrieval
│ │ └── version_manager.py # Document version tracking and comparison
│ └── ui/ # User interface (Streamlit)
│ └── chat_interface.py # Chat interface with confidence display
├── data/ # Data directory
│ ├── raw/ # Raw documents (PDFs, text files)
│ ├── processed/ # Processed and chunked documents
│ ├── chroma_db/ # ChromaDB vector database (persistent)
│ └── update_log.json # Update activity log
├── scripts/ # Utility scripts
│ ├── ingest_data.py # Manual data ingestion script
│ └── check_db.py # Database statistics checker
├── tests/ # Test suite (Phase 7)
│ └── test_advanced_rag.py # Comprehensive pytest tests for all phases
├── .env.example # Configuration example with all settings
├── requirements.txt # Python dependencies
└── main.py # Main entry point
## 🛠️ Installation
### 1. Clone the repository
```bash
git clone <repository-url>
cd permesso-bot
python -m venv env
source env/bin/activatepip install -r requirements.txtEnsure Ollama is installed and running:
# Install Ollama (if not already installed)
curl -fsSL https://ollama.com/install.sh | sh
# Pull the required models
ollama pull llama3.2:1b
ollama pull nomic-embed-textcp .env.example .env
# Edit .env with your settings# Run the ingestion script
python scripts/ingest_data.pyOr, for specific sources:
# From PDFs
python -c "from src.ingestion.pdf_loader import load_pdfs; docs = load_pdfs()"
# From text files
python -c "from src.ingestion.text_processor import process_text_files; docs = process_text_files()"
# From URLs
python -c "from src.ingestion.html_crawler import crawl_html; docs = crawl_html()"streamlit run main.pyThe interface will be available at http://localhost:8501
Edit config/settings.py or .env to customize all RAG parameters:
OLLAMA_BASE_URL = "http://localhost:11434"
LLM_MODEL = "qwen3:14b" # or llama3.2:1b for lighter models
EMBEDDING_MODEL = "nomic-embed-text"CHUNK_SIZE = 800 # Maximum tokens per chunk
CHUNK_OVERLAP = 150 # Token overlap between chunks
TOP_K_RESULTS = 10 # Number of documents to retrieve
SIMILARITY_THRESHOLD = 0.7 # Minimum similarity score for retrievalWEB_CRAWLER_ENABLED = True
OFFICIAL_SOURCES_URL = "https://portaleimmigrazione.it/"
MAX_CONCURRENT_REQUESTS = 3
CRAWL_TIMEOUT_SECONDS = 60
DATA_SOURCE_CATEGORIES = ["official_gov", "legal_texts", "case_studies"]
COMPLEX_CASE_PRIORITY = 1.5 # Boost factor for special scenarios
UPDATE_SCHEDULE_HOURS = 24 # Daily refresh intervalSCENARIO_TYPES = [
"standard",
"mixed_marriage_italian_citizen", # Cittadino italiano con coniuge straniero
"work_permit", # Permesso per lavoro
"family_reunification", # Ricongiungimento familiare
"asylum" # Asilo politico
]KEEP_VERSIONS = 5 # Number of versions to retain per document
VERSION_HISTORY_PATH = "./data/document_versions.json"Step 1.1: Configuration Settings (config/settings.py)
web_crawler_enabled: Toggle for Crawl4AI web crawlingofficial_sources_url: Default Italian government portal URLmax_concurrent_requests: Rate limiting (default: 3)crawl_timeout_seconds: Request timeout (default: 60)data_source_categories: ["official_gov", "legal_texts", "case_studies"]complex_case_priority: Boost factor for special scenarios (1.5)update_schedule_hours: Daily refresh interval (24 hours)
Step 1.2: Ingestion Pipeline Structure (src/ingestion/)
src/ingestion/
├── pipelines/ # Data processing pipeline orchestration
├── processors/ # Document parsing and chunking
│ ├── document_processor.py # Smart legal text chunking
│ └── metadata_enricher.py # Scenario tagging & enrichment
├── schedulers/ # Automated update management
│ └── update_scheduler.py # Daily crawl scheduling
├── web_crawler.py # Async Crawl4AI implementation
├── source_classifier.py # URL categorization system
├── pdf_loader.py # PDF document extraction
├── html_crawler.py # Legacy HTML processing
└── text_processor.py # Text file handling
Step 2.1: AsyncWebCrawler (src/ingestion/web_crawler.py)
- Uses Crawl4AI's
AsyncWebCrawlerfor markdown extraction - Supports Italian government domains:
.gov.it,portaleimmigrazione.it,ministero.interno.it - Exponential backoff retry logic (max 3 retries)
- Rate limiting: max 1 request/second, respects robots.txt
- Metadata storage: source_url, crawl_timestamp, page_title, content_type
Step 2.2: SourceClassifier (src/ingestion/source_classifier.py)
class SourceClassifier:
def classify_url(url: str) -> Dict[category, confidence]:
# Returns category and confidence score (0.0-1.0)
def is_trusted_source(url: str) -> bool:
# Validates against whitelist of official sourcesCategories: official_gov (.gov.it domains), legal_texts, case_studies, informative
Step 3.1: DocumentProcessor (src/ingestion/processors/document_processor.py)
- Chunk size: 800 tokens max, 150 token overlap
- Preserves Italian legal structure: Articolo X, comma Y, lettera z
- Semantic chunking based on markdown headings (##, ###)
- Metadata per chunk: source_url, document_type, section_path
Step 3.2: MetadataEnricher (src/ingestion/processors/metadata_enricher.py)
- Detects complex case indicators via regex patterns:
- "cittadino italiano" + "coniuge extracomunitario" → mixed_marriage
- "lavoro" + "permesso" → work_permit
- "famiglia" + "ricongiungimento" → family_reunification
- Extracts dates, article references, official document numbers
Step 3.3: VectorStore Updates (src/database/vector_store.py)
def add_documents_with_metadata(documents, metadatas, collection_name)
def search_with_category_filter(query, category, k=10)
def search_by_scenario(query, scenario_id, k=10)
def get_collection_statistics() -> Dict[category_counts, last_update]Step 4.1: ScenarioManager (src/core/scenario_manager.py)
class ScenarioManager:
SCENARIOS = ["standard", "mixed_marriage_italian_citizen", "work_permit",
"family_reunification", "asylum"]
def identify_scenarios(query) -> List[str]
def get_scenario_keywords(scenario_id) -> List[str] # Italian terminology
def boost_query_for_scenario(query, scenario_ids) -> strStep 4.2: RAG Engine Updates (src/core/rag_engine.py)
def retrieve_for_complex_scenario(query, scenario_type) -> List[Dict]:
# Boosts retrieval with scenario keywords if confidence < 0.7
def get_case_precedents(scenario_type, limit=5) -> List[Dict]Step 4.3: PrecedentExtractor (src/core/precedent_extractor.py)
- Pattern matching: "caso pratico", "esempio", "precedente"
- Extracts: scenario, key facts, outcome, source reference
- Stores precedents separately for quick retrieval
Step 5.1: PromptTemplates (src/core/prompt_templates.py)
SYSTEM_PROMPT_LEGAL = """Strict rules: cite sources, don't hallucinate,
acknowledge uncertainty. All responses in Italian."""
RESPONSE_FORMAT_STANDARD = """Risposta diretta | Documenti di riferimento |
Note importanti"""
RESPONSE_FORMAT_COMPLEX = """Prerequisiti | Step-by-step procedure |
Avvertenze e disclaimer"""Step 5.2: Citation Tracking in RAG Engine
- Response format: "Articolo 5 del TUI [1], comma 2 [1]"
- Tracks number of documents used per response
- Lists all source URLs at bottom with numbered references
Step 5.3: Confidence Scoring
def calculate_answer_confidence(context, query) -> Dict[score, sources_relevant, coverage]:
# Returns score (0.0-1.0)
# If score < 0.6: suggests official portal consultationStep 6.1: UpdateScheduler (src/ingestion/schedulers/update_scheduler.py)
- Daily crawls of official sources using APScheduler or schedule library
- Incremental updates: only new/modified documents
- Logs to
./data/update_log.json - CLI trigger support
Step 6.2: ChangeDetector (src/ingestion/change_detector.py)
- Content hashing with SHA256 for change detection
- Version history in
./data/document_versions.json - Detects document version changes (new article numbers, date updates)
Step 6.3: VersionManager (src/database/version_manager.py)
def get_document_version_history(doc_id) -> List[Dict[version, date, changes]]
def compare_versions(doc_id, v1, v2) -> Dict[added, removed]
# Keeps last 5 versions per document for audit trailStep 7: Test Suite (tests/test_advanced_rag.py)
- Unit tests with pytest fixtures for mock data
- Integration test: full pipeline on sample query
- Tests cover all phases from crawling to response generation
python -m pytest tests/- Never commit the
.envfile - Use
.gitignoreto protect sensitive files - Keep database paths outside the repository if possible
[Your license here]
Contributions are welcome! Please create a PR with your changes.
For issues or questions, open an issue on GitHub.