HybriDFS

High-Performance Distributed File Storage with Hybrid Cloud-Edge Architecture

Tugas Besar UAS - Komputasi Paralel dan Terdistribusi
Kelompok 5A

👥 Anggota Kelompok

No	Nama	NIM
1	Desti Karmini Nudhatilah	237006001
2	Renasya Malkahaq	237006007
3	Rovi Fauzan	237006022
4	Febnawan Fatur Rochman	237006029

📖 Deskripsi

HybriDFS adalah sistem file terdistribusi (Distributed File System) yang mengimplementasikan Hybrid Cloud-Edge Architecture. Sistem ini memanfaatkan:

VPS (Master Node): Server cloud dengan spesifikasi standar untuk manajemen metadata dan koordinasi
Local PC (Worker Node): Komputer lokal berperforma tinggi untuk menyimpan file besar

Dengan arsitektur ini, sistem dapat mengoptimalkan throughput dan biaya penyimpanan dengan melakukan Smart Routing berdasarkan ukuran file.

🏗️ Arsitektur Sistem

Diagram Arsitektur

flowchart TB
    subgraph Client["👤 Client (Browser)"]
        Dashboard["Dashboard Web"]
    end
    
    subgraph VPS["☁️ VPS - Master Node"]
        API["Server API Flask\n:5000"]
        Router["Router Pintar"]
        Registry["Registry Node"]
        TokenMgr["Manajer Token"]
        DB[(SQLite\nDatabase Metadata)]
        Benchmark["Pencatat Benchmark"]
    end
    
    subgraph Network["🔒 Jaringan VPN Tailscale"]
        TS["Koneksi P2P Terenkripsi"]
    end
    
    subgraph Workers["💻 Worker Nodes (PC Lokal)"]
        W1["Worker 1\ndesti-pc\n:6001"]
        W2["Worker 2\nrenasya-pc\n:6001"]
        W3["Worker 3\nrovi-pc\n:6001"]
        W4["Worker 4\nfebnawan-pc\n:6001"]
    end
    
    Dashboard <--> API
    API --> Router
    Router --> Registry
    Router --> TokenMgr
    API --> DB
    API --> Benchmark
    
    Registry <-.->|Heartbeat| TS
    TS <-.-> W1
    TS <-.-> W2
    TS <-.-> W3
    TS <-.-> W4
    
    Dashboard -->|"File > 10MB\n(Transfer Langsung)"| TS

Sequence Diagram - Upload File

sequenceDiagram
    autonumber
    participant C as Client (Browser)
    participant M as Master Node (VPS)
    participant W as Worker Node (PC Lokal)
    
    Note over W,M: Registrasi Worker (Saat Startup)
    W->>M: POST /api/heartbeat
    M-->>W: 201 OK (Terdaftar)
    
    loop Setiap 10 detik
        W->>M: POST /api/heartbeat
        M-->>W: 200 OK
    end
    
    Note over C,M: User Upload File (> 10MB)
    C->>M: POST /api/request_upload {filename, size_mb}
    M->>M: Router Pintar: Cek ukuran > 10MB
    M->>M: Pilih Worker terbaik (space paling besar)
    M->>M: Generate token upload
    M-->>C: {action: "direct_transfer", target_ip, token}
    
    C->>W: POST /api/store_file + File + Token
    W->>M: POST /api/validate_token
    M-->>W: 200 OK (Valid)
    W->>W: Simpan file ke storage
    W->>M: POST /api/upload_confirm
    M->>M: Simpan metadata ke DB
    M->>M: Catat ke benchmark_log.csv
    W-->>C: 201 OK {file_id, upload_time}

Sequence Diagram - Fault Tolerance (Toleransi Kegagalan)

sequenceDiagram
    participant M as Master Node
    participant W as Worker Node
    participant S as Health Scheduler
    
    Note over M,W: Operasi Normal
    W->>M: Heartbeat (setiap 10 detik)
    M->>M: Update timestamp heartbeat terakhir
    
    Note over W: Worker Mati/Terputus
    W-xM: Tidak ada heartbeat
    
    loop Setiap 10 detik
        S->>M: Cek node yang tidak aktif
        M->>M: Bandingkan timestamp
        Note over M: Jika heartbeat terakhir > 15 detik
        M->>M: Set status = OFFLINE
    end
    
    Note over M: Dashboard menampilkan OFFLINE (merah)
    
    Note over W: Worker Hidup Kembali
    W->>M: Heartbeat
    M->>M: Set status = ONLINE
    Note over M: Dashboard menampilkan ONLINE (hijau)

Sequence Diagram - Secure Upload (Otorisasi Token)

sequenceDiagram
    autonumber
    participant C as Client
    participant M as Master Node
    participant W as Worker Node
    
    Note over C,W: Upload Aman dengan Token
    
    C->>M: POST /api/request_upload {filename, size}
    M->>M: Generate token unik (UUID)
    M->>M: Simpan token ke database (kadaluarsa: 5 menit)
    M-->>C: {token: "abc-123", target_ip: "100.x.x.x"}
    
    C->>W: POST /api/store_file + File + Token
    W->>M: POST /api/validate_token {token: "abc-123"}
    
    alt Token Valid
        M-->>W: 200 OK {valid: true}
        W->>W: Simpan file ke storage
        W->>M: POST /api/upload_confirm
        M->>M: Invalidasi token (sudah dipakai)
        W-->>C: 201 OK - Upload berhasil
    else Token Tidak Valid/Kadaluarsa
        M-->>W: 401 Unauthorized
        W-->>C: 401 - Upload ditolak
    end

Logika Smart Routing

flowchart TD
    A[Client Request Upload] --> B{Ukuran File > 10MB?}
    B -->|Ya| C[Pilih Worker Node]
    B -->|Tidak| D[Upload ke VPS Singapore]
    
    C --> E{Ada Worker Online?}
    E -->|Ya| F[Pilih Worker dengan Space Terbesar]
    E -->|Tidak| G[Return Error: Tidak Ada Worker]
    
    F --> H[Generate Token Upload]
    H --> I[Return: direct_transfer + IP Worker]
    
    D --> J[Generate Token Upload]
    J --> K[Return: local_upload]
    
    I --> L[Client upload langsung ke Worker]
    K --> M[Client upload ke Master]

✨ Fitur Utama

1. Smart Routing

File >10MB otomatis diarahkan ke Worker Node untuk throughput tinggi, sedangkan file kecil disimpan di VPS.

2. Multi-Worker Support

Mendukung banyak Worker Node dengan otomatis load balancing berdasarkan free space terbesar.

3. Fault Tolerance (Toleransi Kegagalan)

Kemampuan sistem untuk mendeteksi dan menangani kegagalan node secara otomatis.

Cara Kerja:

Setiap Worker mengirim heartbeat ke Master setiap 10 detik
Master mencatat timestamp heartbeat terakhir
Jika tidak ada heartbeat selama 15 detik, Master menandai node sebagai OFFLINE
Ketika Worker hidup kembali dan mengirim heartbeat, status kembali ONLINE

Manfaat:

Sistem tetap berjalan meskipun ada Worker yang mati
Dashboard menampilkan status real-time (hijau = ONLINE, merah = OFFLINE)
Upload otomatis diarahkan ke Worker yang masih aktif

4. Secure Upload (Token-Based Authorization)

Sistem keamanan untuk memastikan hanya upload yang sah yang diterima oleh Worker.

Masalah yang Diatasi:

Mencegah upload tidak sah ke Worker Node
Melindungi storage Worker dari penyalahgunaan

Cara Kerja:

Client meminta izin upload ke Master
Master generate token unik (UUID) yang berlaku 5 menit
Token disimpan di database Master
Client mengirim file ke Worker beserta token
Worker memvalidasi token ke Master
Jika valid → file disimpan, token di-invalidate (hanya bisa dipakai sekali)
Jika tidak valid/expired → upload ditolak (401 Unauthorized)

Keamanan:

Token bersifat one-time use (sekali pakai)
Token memiliki expiry time (5 menit)
Tanpa token valid, Worker menolak semua upload

5. Web Dashboard

Monitoring real-time untuk:

Status semua Worker Nodes (ONLINE/OFFLINE)
Daftar file yang tersimpan
Benchmark log (throughput upload)
Upload file langsung dari browser

6. Benchmark Logging

Pencatatan otomatis performa setiap upload ke file CSV meliputi:

Timestamp
Ukuran file (MB)
Waktu upload (detik)
Throughput (MB/s)
Target node

7. Tailscale VPN

Koneksi aman antar node menggunakan encrypted tunnel, memungkinkan Worker di jaringan berbeda (rumah masing-masing anggota) terhubung dengan aman.

🛠️ Tech Stack

Komponen	Teknologi
Backend	Python 3.9+, Flask
Database	SQLite
Network	Tailscale VPN
Container	Docker, Docker Compose
Testing	Pytest, Hypothesis (Property-based testing)

📁 Struktur Project

HybriDFS/
├── master/                     # Master Node (VPS)
│   ├── app.py                 # Server API Flask
│   ├── router.py              # Logika routing pintar
│   ├── registry.py            # Registry worker node
│   ├── token_manager.py       # Manajemen token upload
│   ├── benchmark.py           # Pencatatan performa
│   ├── scheduler.py           # Pengecekan kesehatan background
│   ├── database.py            # Operasi SQLite
│   ├── models.py              # Model data
│   └── templates/
│       └── dashboard.html     # UI dashboard web
│
├── worker/                     # Worker Node (PC Lokal)
│   ├── app.py                 # Server API Flask
│   ├── heartbeat.py           # Service heartbeat
│   ├── storage.py             # Manajemen penyimpanan file
│   └── __main__.py            # Entry point
│
├── tests/                      # Test suite
│   ├── property/              # Property-based tests
│   └── integration/           # Integration tests
│
├── docker/
│   └── Dockerfile.worker      # Docker image untuk worker
│
├── config.py                   # Pengaturan konfigurasi
├── docker-compose.yml          # Setup multi-worker
├── requirements.txt            # Dependencies Python
└── benchmark_log.csv           # Log performa upload

🚀 Cara Menjalankan

Prerequisites

Python 3.9+
Akun Tailscale
Docker (opsional, untuk menjalankan worker di container)

1. Setup Jaringan Tailscale

Di VPS (Master Node):

# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh

# Aktifkan dan login
sudo tailscale up
# Buka URL yang muncul di browser untuk login

Di PC Lokal (Worker Node):

# Windows: Download dari https://tailscale.com/download/windows
# Linux:
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# Atau pakai auth key (lebih cepat):
sudo tailscale up --authkey=tskey-auth-xxxxx

Dapatkan Auth Key (untuk invite anggota):

Buka https://login.tailscale.com/admin/settings/keys
Klik "Generate auth key"
Pilih Reusable: Yes, Expiry: 1 day
Share key ke anggota kelompok

2. Jalankan Master Node (di VPS)

# Clone repository
git clone https://github.com/feb027/HybriDFS.git
cd HybriDFS

# Install dependencies
pip install -r requirements.txt

# Jalankan Master
python -m master.app

# Master berjalan di http://<TAILSCALE_IP>:5000
# Dashboard: http://<TAILSCALE_IP>:5000/dashboard

3. Jalankan Worker Node (di PC Lokal)

Windows (PowerShell):

# Clone repository
git clone https://github.com/feb027/HybriDFS.git
cd HybriDFS

# Install dependencies
pip install -r requirements.txt

# Set environment variables
$env:WORKER_NODE_ID="nama-anda-pc"
$env:WORKER_HOST="<IP_TAILSCALE_ANDA>"
$env:WORKER_PORT="6001"
$env:MASTER_URL="http://<IP_TAILSCALE_VPS>:5000"

# Jalankan Worker
python -m worker

Linux/Mac:

# Clone dan install
git clone https://github.com/feb027/HybriDFS.git
cd HybriDFS
pip install -r requirements.txt

# Jalankan Worker
WORKER_NODE_ID="nama-anda-pc" \
WORKER_HOST="<IP_TAILSCALE_ANDA>" \
WORKER_PORT="6001" \
MASTER_URL="http://<IP_TAILSCALE_VPS>:5000" \
python -m worker

4. Akses Dashboard

Buka browser dan akses:

http://<IP_TAILSCALE_VPS>:5000/dashboard

📊 API Endpoints

Master Node API

Endpoint	Method	Deskripsi
`/api/heartbeat`	POST	Terima heartbeat dari worker
`/api/request_upload`	POST	Request routing untuk upload
`/api/upload_confirm`	POST	Konfirmasi upload selesai
`/api/validate_token`	POST	Validasi upload token
`/api/files`	GET	List semua file
`/api/files/<id>`	GET	Detail file
`/api/files/<id>`	DELETE	Hapus file
`/api/nodes`	GET	List semua worker nodes
`/api/download/<id>`	GET	Download file
`/api/benchmark`	GET	Data benchmark
`/dashboard`	GET	Dashboard web

Worker Node API

Endpoint	Method	Deskripsi
`/api/store_file`	POST	Simpan file (dengan token)
`/api/download/<file_id>`	GET	Download file
`/api/health`	GET	Health check
`/api/files`	GET	List file lokal

🧪 Testing

# Jalankan semua test
pytest

# Dengan coverage report
pytest --cov=master --cov=worker

# Test spesifik
pytest tests/property/         # Property-based tests
pytest tests/integration/      # Integration tests

📈 Benchmark

Sistem otomatis mencatat performa upload ke benchmark_log.csv:

Timestamp,FileSize_MB,UploadTime_Sec,Throughput_MBps,Target_Node
2025-12-02T08:29:59,20.0,0.019,1017.28,febnawan-pc
2025-12-02T08:44:41,0.28,0.0006,450.67,vps-singapore

Data benchmark juga dapat dilihat langsung di Dashboard pada bagian "Benchmark Log".

⚙️ Konfigurasi

File config.py:

Parameter	Default	Deskripsi
`LARGE_FILE_THRESHOLD_MB`	10.0	Batas ukuran file untuk routing ke worker
`HEARTBEAT_INTERVAL_SEC`	10	Interval heartbeat worker
`HEARTBEAT_TIMEOUT_SEC`	15	Timeout untuk deteksi offline
`TOKEN_EXPIRY_SEC`	300	Masa berlaku upload token (5 menit)

📝 License

MIT License

👤 Repository

GitHub: https://github.com/feb027/HybriDFS

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
deploy		deploy
docker		docker
master		master
master_storage		master_storage
scripts		scripts
tests		tests
worker		worker
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
config.py		config.py
docker-compose.yml		docker-compose.yml
prd.md		prd.md
pytest.ini		pytest.ini
requirements.txt		requirements.txt
test_benchmark_log.csv		test_benchmark_log.csv

Folders and files

Latest commit

History

Repository files navigation