AutoPYara

Automated, Cluster-Driven YARA Rule Generation

Automatically discover malware families and generate high-quality, tightly scoped YARA rules using probabilistic clustering and Bloom-filtered n-gram analysis.

Documentation · PyPI · Report a Bug · Changelog · Releasing

---

📌 Overview

AutoPYara is a Python framework for automated YARA rule generation from collections of malware samples. It combines:

• Variational Bayesian Gaussian Mixture Models (VBGMM)
• Augmented DBSCAN with centroid refinement
• Malicious/benign Bloom filter isolation
• Byte-level n-gram feature extraction

The result: cluster-aware, precision-engineered YARA signatures with minimal manual effort.

🧠 How it works

flowchart TD
    A[Malware Samples] --> B[Byte n-gram Extraction]
    B --> C["Bloom Filter Isolation<br/>(benign removal + malicious focus)"]
    C --> D["Clustering Engine<br/>(VBGMM or Augmented DBSCAN)"]
    D --> E[Cluster-Specific Signature Construction]
    E --> F[High-Quality YARA Rules]

Loading

✨ Features

• Automated Clustering — group similar malware samples together automatically to create concise, targeted rules.
• Two Core Presets — the standard AutoYara (VBGMM) approach, or the enhanced AutoPYara (Augmented DBSCAN) pipeline.
• Built-in Bloom Filters — ships with pre-trained EMBER and AutoPYara filters to efficiently filter out benign n-grams.
• Multiple Output Formats — raw strings, compiled yara-python objects, or yaramod parsed objects.
• Custom Training — train your own Bloom filters on proprietary datasets.

🚀 Installation

Requirements

• Python >= 3.9
• A Java Runtime Environment (JRE 11+) on PATH or pointed to by JAVA_HOME. AutoPYara's clustering/rule-generation backend runs inside a JVM. pip install itself doesn't need Java, but AutoPYara() will raise a clear error the first time you construct it without one — install a JRE before you actually use the tool. On Debian/Ubuntu: sudo apt install default-jre.

pip install autopyara

Install from a local build instead

python -m build
pip install dist/autopyara-*.whl

Note on first run: the Bloom filter data (~600MB)

To keep the initial install lightweight, the package needs about 600MB of pre-trained Bloom filter data that isn't bundled in the distribution. You don't need to fetch this manually — the first time you import autopyara and the data is missing, it's downloaded automatically from the data-branch branch of this repository. To trigger it explicitly (e.g. to pre-warm a Docker image):

autopyara-download

⚡ Quick Start

Generating your first YARA rule is as simple as pointing the tool at a directory of malware samples.

from autopyara import AutoPYara

# 1. Initialize the tool
tool = AutoPYara()

# 2. Generate a rule using the AutoPYara preset
results = tool.generate(
    input_files="/path/to/malware/directory",
    preset="AutoPYara",
    rule_name="my_custom_rule",
    output_format="string"
)

# 3. Print the results
print(f"Discovered {results['k_clusters']} distinct malware clusters.")
print("\nGenerated YARA Rule:")
print(results['rule_string'])

⚙️ Core presets

preset="AutoYara" (Standard)

Algorithm: Variational Bayesian Gaussian Mixture Model (VBGMM)

Behavior: Automatically infers the number of clusters ($K$) probabilistically.

Best for: General-purpose rule generation where the structural diversity of the input directory is completely unknown.

preset="AutoPYara" (Enhanced)

Algorithm: Augmented DBSCAN combined with KMeans soft clustering

Behavior: Uses a custom Augmented DBSCAN to calculate $K$ prior to centroid optimization.

Best for: Producing more tightly bound rules for closely related malware families.

🛠 Advanced usage

Defining a custom $K$

If you want to manually force the algorithm to split your samples into a specific number of clusters, you can override the presets:

# Force exactly 4 clusters using the AutoPYara augmented pipeline
results = tool.generate(
    input_files="/path/to/malware",
    preset="AutoPYara",
    augmented_target_k=4  # Forces the optimizer to find 4 clusters
)

Output formats

By default, AutoPYara returns a raw string. You can integrate it directly into existing analysis pipelines by requesting Python objects instead:

# Returns a compiled yara-python object ready for immediate scanning
results = tool.generate(
    input_files="/path/to/malware",
    output_format="yara-python"
)

compiled_rule = results["output"]
matches = compiled_rule.match("/path/to/suspicious/file.exe")

Supported formats: 'string', 'yara-python', and 'yaramod'.

Custom Bloom filters

generate() defaults to the built-in "ember" Bloom filters for both benign and malicious data. You can switch to the "autopyara" defaults, or provide absolute paths to your own retrained filters:

results = tool.generate(
    input_files="/path/to/malware",
    bloom_malicious="/absolute/path/to/custom/malicious_bloom",
    bloom_benign="/absolute/path/to/custom/benign_bloom",
)

Training new Bloom filters

Train custom Bloom filters on your own proprietary benign or malicious datasets with train():

tool = AutoPYara()

# Extract 8-grams from a directory of benign software
tool.train(
    input_dir="/path/to/benign/software",
    output_dir="/path/to/save/new/bloom",
    ngram_size=8
)

📚 Full API reference: generate()

Parameter	Type	Default	Description
input_files	str | list	Required	Path to input directory or list of sample file paths.
preset	str	None	'AutoYara' or 'AutoPYara'. Auto-configures the clustering pipeline.
bloom_malicious	str	'ember'	Built-in flag ('ember', 'autopyara') or path to custom malicious Bloom filters.
bloom_benign	str	'ember'	Built-in flag ('ember', 'autopyara') or path to custom benign Bloom filters.
output_format	str	'string'	'string', 'yara-python', or 'yaramod'. Determines output rule format.
rule_name	str	'autoyara_rule'	Base string used to name the generated rules.
k_cluster	int	0	Hardcode $K$ for VBGMM. Do not use with preset="AutoPYara".
augmented_target_k	int	None	Hardcode target $K$ for the Augmented DBSCAN pipeline.
verbose	bool	False	Enable detailed logging during cluster generation.

📖 Documentation

Full documentation lives at botacin-s-lab.github.io/AutoPYaraPyPI. It's intentionally basic for now — installation, quick start, and the API reference — with more material (including the accompanying paper, once published) landing there over time.

🧪 Development

pip install -e ".[test]"
pytest tests/

tests/test_core_helpers.py and tests/test_augmented_dbscan.py are pure-Python unit tests (no JVM/network needed). tests/test_smoke_generate.py runs the real pipeline end-to-end against small synthetic dummy files (not real malware) using the built-in Bloom filters.

See RELEASING.md for how versioning and PyPI publishing work.

🤝 Contributing

We're accepting contributions — if you run into an issue or have a fix, fork the repo, open a PR against main, and we'll take a look. PRs are automatically built and tested; once checks pass and a maintainer approves, it gets merged. main itself isn't open to direct pushes from anyone (including maintainers) — everything goes through review. See CONTRIBUTING.md for details.

📄 Citation

If you use AutoPYara in academic work, please cite:

Mabon Ninan*, Nhat Minh Nguyen*, Soumyajyoti Dutta, Sidharth Anil, and Marcus Botacin. "AutoPYara: Next-Gen YARA Rule Generator for Malware Family Clustering." Annual Computer Security Applications Conference (ACSAC 2026), to appear. *Equal contribution. Texas A&M University — {ninanmm, nmnguy29, soumyajyoti1998, sid.anil, botacin}@tamu.edu

@inproceedings{autopyara2026,
  title     = {AutoPYara: Next-Gen YARA Rule Generator for Malware Family Clustering},
  author    = {Ninan, Mabon and Nguyen, Nhat Minh and Dutta, Soumyajyoti and Anil, Sidharth and Botacin, Marcus},
  booktitle = {Proceedings of the Annual Computer Security Applications Conference (ACSAC)},
  year      = {2026},
  note      = {To appear}
}

The AutoYara preset builds on the original AutoYara project (Apache 2.0); if you use it, please also cite:

Edward Raff, Richard Zak, Gary Lopez Munoz, William Fleming, Hyrum S. Anderson, Bobby Filar, Charles Nicholas, and James Holt. "Automatic Yara Rule Generation Using Biclustering." 13th ACM Workshop on Artificial Intelligence and Security (AISec '20), 2020. doi:10.1145/3411508.3421372 · arXiv:2009.03779

See the Architecture page for the full attribution.

License

MIT — see LICENSE.

---

Mabon Ninan, Nhat Minh Nguyen, Soumyajyoti Dutta, Sidharth Anil, and Marcus Botacin — Texas A&M University Maintained by Mabon Ninan — ninanmm@tamu.edu