🧠 Intelligent Workflow Orchestration Platform

This platform is a modular, DAG-based low-code system for building AI-augmented workflows using intelligent agents, vector stores, embeddings, LLMs, and various data/logic helpers. It supports seamless chaining of components using execution_order and connectors, with task execution powered by Celery.

Status on Component Implementation: https://docs.google.com/spreadsheets/d/1S-viV3B6S_sz1yrs9Fx-_nYTFob7XvXcCKdquSndxbs/edit?usp=sharing

---

🚀 Getting Started

🔧 Prerequisites

• Python 3.9+
• Redis
• [Optional] Poetry or virtualenv

⚙️ Install Dependencies

pip install -r requirements.txt

🔌 Start Services

Run these in separate terminals:

# Redis
redis-server

# Celery workers (4 concurrent workers)
celery -A app.services.task_service worker --loglevel=info --concurrency=4

# FastAPI backend
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

---

🧩 Available Components

📥 Inputs

• Text Input

📤 Outputs

• Text Output

💬 Prompts

• Prompt – generate LLM-ready prompts using template + variables

💿 Data

• URL Input
• File Input
• Directory
• API Request
• Webhook

🔁 Processing

• Combine Text
• Combine Data
• Data to Message
• Update Text
• Split Text

🧠 Models & Agents

• Agent – connect to LLMs like Falcon, OpenAI
• AIML – classic AIML rule-based chatbot support

🧠 Embeddings & Memory

• Hugging Face Embeddings
• Redis Chat Memory

📦 Vector Store

• FAISS Vector DB

🧮 Tools

• Calculator
• Yahoo Finance API
• Bing Search API

🔀 Logic

• If Else
• Loop
• Pass (forward values)

🧰 Helpers

• Batch Run
• Current Date
• ID Generator
• Message History
• Message Store

---

🧪 Example CURL Workflows

🔗 1. Multi-file Summarization with Agent

curl -X POST http://localhost:8000/submit-workflow/ \
  -H "Content-Type: multipart/form-data" \
  -F workflow='{
    "workflow_id": "multi_file_with_agent",
    "execution_order": [
      "file_processing_file1.txt",
      "file_processing_file2.pdf",
      "file_processing_file3.pdf",
      "agent"
    ],
    "connectors": [
      {"from_node": "file_processing_file1.txt", "to_node": "file_processing_file2.pdf"},
      {"from_node": "file_processing_file2.pdf", "to_node": "file_processing_file3.pdf"},
      {"from_node": "file_processing_file3.pdf", "to_node": "agent"}
    ],
    "parameters": {
      "agent": {
        "type": "agent",
        "model_name": "tiiuae/falcon-7b-instruct",
        "api_key": "hf_***",
        "system_prompt": "Summarize documents",
        "temperature": 0.7,
        "max_new_tokens": 256,
        "user_input": "Summarize the extracted content."
      }
    }
  }' \
  -F files=@"/path/to/file1.txt" \
  -F files=@"/path/to/file2.pdf" \
  -F files=@"/path/to/file3.pdf"

🌐 2. URL → Agent → Output

curl -X POST http://localhost:8000/submit-workflow/ \
  -H "Content-Type: multipart/form-data" \
  -F 'workflow={
    "workflow_id": "summarize_example_domain",
    "execution_order": ["url_input", "agent", "text_output"],
    "connectors": [
      {"from_node": "url_input", "to_node": "agent"},
      {"from_node": "agent", "to_node": "text_output"}
    ],
    "parameters": {
      "url_input": {
        "type": "url_input",
        "url": "https://example.com",
        "output_format": "raw_text"
      },
      "agent": {
        "type": "agent",
        "model_name": "tiiuae/falcon-7b-instruct",
        "api_key": "hf_***",
        "system_prompt": "Summarize this webpage",
        "user_input": "Summarize this: ${url_input_output}"
      },
      "text_output": {"type": "text_output"}
    }
  };type=application/json'

📊 3. Yahoo Finance + Agent + Output

curl -X POST http://localhost:8000/submit-workflow/ \
  -H "Content-Type: multipart/form-data" \
  -F workflow='{
    "workflow_id": "yahoo_news_summarize",
    "execution_order": ["yahoo_finance_api", "agent", "text_output"],
    "connectors": [
      {"from_node": "yahoo_finance_api", "to_node": "agent"},
      {"from_node": "agent", "to_node": "text_output"}
    ],
    "parameters": {
      "yahoo_finance_api": {
        "type": "yahoo_finance_api",
        "stock_symbol": "AAPL",
        "data_method": "get_news",
        "num_news": 2
      },
      "agent": {
        "type": "agent",
        "model_name": "tiiuae/falcon-7b-instruct",
        "api_key": "hf_***",
        "system_prompt": "You are a financial analyst.",
        "user_input": "Summarize this: ${yahoo_finance_text}"
      },
      "text_output": {"type": "text_output"}
    }
  };type=application/json'

✂️ 4. Split Text

curl -X POST http://localhost:8000/submit-workflow/ \
  -H "Content-Type: multipart/form-data" \
  -F 'workflow={
    "workflow_id": "test_split_text",
    "execution_order": ["split_text"],
    "connectors": [],
    "parameters": {
      "split_text": {
        "type": "split_text",
        "input_documents": "This is a long input that needs to be split into smaller chunks.",
        "chunk_size": 10,
        "chunk_overlap": 2,
        "separator": " "
      }
    }
  }'

---

📌 Notes

• Workflows are constructed using a DAG format:
  • execution_order: List of task names in topological order
  • connectors: Defines directed edges between nodes
• Parameter injection supports ${output_key} syntax for dynamic chaining
• Each task's output is available as taskname_output

---

🧠 Prompt Component

Use Prompt to dynamically fill templates:

{
  "type": "prompt",
  "template": "Hello {name}, your balance is {balance}",
  "data": {
    "name": "Alice",
    "balance": "$20"
  }
}

Outputs:

{"prompt_output": {"prompt": "Hello Alice, your balance is $20"}}

---

🏁 Summary

This system lets you flexibly combine AI models, tools, logic, and I/O into customizable agentic workflows — using a single API call. You can:

• Ingest multiple file types
• Generate prompts dynamically
• Route and process content
• Use embeddings, vector search, and memory

Perfect for building LLM chains, chatbots, research agents, data pipelines, and much more.

---

✨ Built for modularity, powered by Celery, and designed for real-world use cases.

---

Happy building!