This project documents the design and implementation of an isolated malware analysis sandbox built on Proxmox VE.
The goal of the project was to create a controlled environment where suspicious Windows executables could be analyzed using a repeatable workflow combining automated triage, manual static analysis, dynamic behavioral analysis, simulated network services, and custom detection engineering.
Rather than relying entirely on existing analysis tools, I developed a Python-based analysis script to automate the initial investigation of each sample. The script performs file hashing, file-type identification, string extraction, PE analysis, section entropy analysis, import inspection, YARA scanning, and automated report generation.
Manual analysis was then used to validate and expand on the automated findings using tools such as PE-bear, Sysinternals Strings, Process Monitor, System Informer, Sysmon, and REMnux.
The project followed this workflow:
Automated Triage → Manual Static Analysis → Hypothesis Development → Dynamic Analysis → Detection Engineering
Two Practical Malware Analysis samples were investigated as full case studies, producing contrasting static and dynamic analysis results. The investigations also exposed limitations in the custom automation and ultimately led to the development and validation of original YARA detection rules.
- Build an isolated malware analysis network separated from the production homelab and Internet.
- Configure Windows and REMnux systems for static and dynamic malware analysis.
- Simulate network services without allowing malware Internet access.
- Develop custom Python tooling to automate initial malware triage.
- Perform manual static analysis to validate and expand automated findings.
- Develop testable behavioral hypotheses from static evidence.
- Use dynamic analysis to confirm or refute those hypotheses.
- Create and validate original YARA detection rules from observed sample characteristics.
- Document both positive and negative analysis results without overstating the available evidence.
Malware analysis was performed inside a dedicated virtual network hosted on Proxmox VE. The sandbox was intentionally separated from the primary homelab network to prevent analyzed samples from communicating with production systems or reaching the public Internet.
A dedicated Proxmox Linux bridge, vmbr2, was created specifically for malware analysis. Unlike the primary homelab bridge, vmbr2 has no physical network interface attached, no Proxmox host IP assigned, and no route through pfSense.
The sandbox uses the private 192.168.66.0/24 network:
| System | Role | IP Address |
|---|---|---|
| Windows 10 Analysis VM | Malware execution, static analysis, and endpoint telemetry | 192.168.66.10 |
| REMnux | Network-analysis and simulated-service support | 192.168.66.20 |
Neither analysis VM was configured with a default gateway. More importantly, vmbr2 itself has no physical uplink or external route. This means there is no network path out of the sandbox by design, rather than relying solely on configuration within the analysis VMs.
Proxmox VE
|
+-------------------+
| vmbr2 |
| No physical NIC |
| No host IP |
| No pfSense route |
+---------+---------+
|
192.168.66.0/24
/ \
/ \
+----------------+ +----------------+
| Windows 10 | | REMnux |
| Analysis VM | | Support VM |
| 192.168.66.10 | | 192.168.66.20 |
+----------------+ +----------------+
| Static analysis| |Network analysis|
| Malware testing| |Simulated svc. |
| Procmon/Sysmon | | |
+----------------+ +----------------+
Isolation was verified from both sides of the sandbox before malware analysis began.
From REMnux, the following connectivity tests were performed:
- Public Internet (
8.8.8.8) — unreachable - pfSense gateway (
192.168.1.1) — unreachable - Proxmox management interface (
192.168.1.10) — unreachable - Windows analysis VM (
192.168.66.10) — reachable
The Windows analysis VM was also verified to have no default gateway. Attempts to reach external and production-homelab addresses failed while communication within the 192.168.66.0/24 sandbox remained available.
These tests validated the intended design: the Windows analysis VM and REMnux could communicate with each other across vmbr2, but neither system had a functional route to the primary homelab or public Internet.
This isolated network became the foundation for all subsequent static and dynamic malware analysis performed in the project.
With network isolation established, the Windows 10 and REMnux systems were configured with a combination of static-analysis, behavioral-monitoring, and detection-engineering tools.
The environment was designed around a layered workflow: automated triage provided an initial view of each sample, while independent tools were used to manually validate findings and observe behavior during controlled execution.
The Windows 10 analysis VM served as the primary system for sample inspection and controlled execution.
| Tool | Purpose |
|---|---|
| Custom Python Analyzer | Automated hashing, file identification, string extraction, PE inspection, entropy analysis, import analysis, YARA scanning, and report generation |
| PE-bear | Manual inspection of PE headers, sections, imports, entry points, and other executable structure |
| Sysinternals Strings | Independent review of embedded ASCII and Unicode strings |
| Process Monitor (Procmon) | Captured process, file-system, and registry activity during dynamic analysis |
| Sysmon | Provided persistent Windows process and system-event telemetry |
| System Informer | Monitored running processes and process-level activity during execution |
| YARA / yara-python | Developed and tested custom detection rules against analyzed samples |
| Python / pefile | Provided the foundation for the custom automated-analysis workflow |
Before analyzing malware, the monitoring stack was validated using normal Windows activity.
Sysmon was configured with a baseline configuration and verified to be recording events in:
Applications and Services Logs → Microsoft → Windows → Sysmon → Operational
Process Monitor was also verified to be actively capturing Windows activity, including events generated by processes such as Explorer.EXE, lsass.exe, svchost.exe, MsMpEng.exe, and Sysmon64.exe.
This validation established that endpoint activity could be captured before any malware samples were introduced into the workflow.
PE-bear was validated against a known benign Windows executable before being used on malware samples. Loading notepad.exe confirmed that PE headers, sections, imports, entry-point information, hexadecimal data, and disassembly could be inspected successfully.
This benign baseline provided a known-good reference before the same tooling was applied to suspicious executables.
Because the malware sandbox was intentionally isolated from the Internet, cloud-based scanning and automatic sample-submission features were disabled in System Informer.
Automatic scanning and submission integrations were disabled to keep sample analysis contained within the local environment.
Python was installed inside the isolated Windows VM to support the custom analysis tooling.
Because the analysis VM had no Internet access, dependencies had to be staged and configured offline. Installing pefile through pip was unsuccessful because its dependency chain required additional packages that were not available inside the isolated environment.
Instead of weakening the sandbox's network isolation, the required pefile, peutils, and ordlookup source components were staged offline and placed directly into the local analysis environment. The Python import was then validated before integrating PE parsing into the custom analyzer.
YARA support was configured separately using the yara-python 4.5.4 wheel, which was staged and installed offline. This allowed the custom analysis script to compile and execute local YARA rules without requiring Internet access.
REMnux provided the Linux-based network-analysis side of the sandbox. Its role was to support controlled observation of network-aware samples while keeping the Windows malware-analysis VM isolated from real external infrastructure.
REMnux was later configured with simulated DNS and network services so samples could receive controlled responses without being given direct Internet access. That configuration is covered separately in the Simulated Network Services section.
No single tool was treated as authoritative.
Automated findings were cross-checked with manual tools, and static indicators were treated as hypotheses until runtime evidence supported them. This became particularly important during the case studies, where manual analysis identified evidence that the automation had collected but had not elevated into its summary, and dynamic testing did not always reproduce behavior suggested by static analysis.
This layered approach formed the basis of the project's analysis methodology:
Collect → Validate → Hypothesize → Test → Detect
A fully isolated malware sandbox creates an analysis challenge: network-aware samples may attempt DNS lookups, HTTP requests, or other communications, but allowing those requests to reach the real Internet would weaken the containment model.
To preserve isolation while still observing network behavior, the REMnux VM at 192.168.66.20 was configured to provide simulated network services to the Windows analysis VM.
INetSim was configured on REMnux to emulate common Internet services, including HTTP and other network protocols. Instead of allowing a sample to communicate with an external server, requests could be redirected to REMnux and answered locally.
This allowed network-aware behavior to be observed without introducing a route from the sandbox to the public Internet.
During configuration, INetSim's built-in DNS service failed to bind to port 53.
Troubleshooting identified a compatibility issue between the installed INetSim 1.3.2 DNS module and the newer Net::DNS 1.44 library. Other INetSim services were operational, but the DNS component could not provide the name resolution required for the simulated network environment.
Rather than changing the sandbox's isolation model, the INetSim DNS service was disabled and DNS functionality was separated from the rest of the simulation stack.
A standalone FakeDNS service was configured on REMnux to resolve requested domain names to:
192.168.66.20
INetSim continued to provide the simulated application-layer services.
The resulting network path was:
+-------------------------+
| Windows 10 Analysis VM |
| 192.168.66.10 |
+------------+------------+
|
| DNS / HTTP requests
v
+-------------------------+
| REMnux |
| 192.168.66.20 |
| |
| FakeDNS -> DNS |
| INetSim -> HTTP/etc. |
+------------+------------+
|
X
No Internet
This allowed arbitrary test domains to resolve to REMnux, where INetSim could return controlled responses instead of allowing traffic to leave the sandbox.
The complete simulated-network path was validated from the Windows analysis VM.
A DNS lookup for a test domain returned the REMnux address:
testmalwarequery.net -> 192.168.66.20
HTTP requests were then issued from Windows using both curl and PowerShell Invoke-WebRequest. The requests were redirected to REMnux and returned an HTTP 200 response containing INetSim's simulated webpage.
This validated the complete path:
Windows Analysis VM → FakeDNS → REMnux → INetSim → Simulated HTTP Response
The result was a controlled "fake Internet" environment that preserved the sandbox's network isolation while still allowing network-aware malware behavior to be observed.
FakeDNS and INetSim were intentionally treated as analysis services rather than permanent background infrastructure.
Before network-aware sample execution, the services could be started and validated as part of the analysis pre-flight process. This provided an additional opportunity to confirm that DNS resolution and simulated services were functioning correctly before malware execution.
The services were only required during testing that needed simulated network interaction.
To make the initial analysis process repeatable, I developed a custom Python script, analyze_sample.py, to perform automated static triage before beginning manual investigation.
Rather than attempting to classify a sample as malicious automatically, the script was designed to collect and organize evidence that could guide deeper analysis.
The script performs the following tasks for each submitted sample:
- Validates that the sample exists and can be accessed.
- Calculates MD5, SHA-1, and SHA-256 hashes.
- Collects basic file-system metadata.
- Identifies file type using file signatures rather than relying only on the extension.
- Detects potential mismatches between the claimed extension and observed file type.
- Extracts printable ASCII and UTF-16LE strings.
- Parses Windows PE files using
pefile. - Extracts imported DLLs and API functions.
- Records PE section names, sizes, and entropy values.
- Flags selected imports associated with behaviors such as process manipulation, encryption, keylogging, and downloader activity.
- Compiles and executes local YARA rules.
- Generates both structured and human-readable analysis output.
The automated results provide a starting point for investigation rather than a final verdict. Suspicious imports, strings, entropy values, and YARA matches are treated as indicators requiring analyst interpretation.
Each analyzed sample receives a dedicated directory named using its SHA-256 hash:
C:\MalwareAnalysis\
├── samples\
├── scripts\
│ └── analyze_sample.py
├── yara_rules\
└── analysis\
└── <SHA256>\
├── analysis_results.json
├── extracted_strings.txt
└── report.txt
The three output files serve different purposes:
| Output | Purpose |
|---|---|
analysis_results.json |
Structured analysis data containing hashes, metadata, strings, PE information, imports, entropy values, and YARA results |
extracted_strings.txt |
Complete extracted ASCII and UTF-16LE string set for manual review |
report.txt |
Human-readable summary designed to prioritize notable findings |
The human-readable report begins with a Key Findings section that attempts to elevate potentially important indicators, including:
- File-extension mismatches
- Notable imported APIs
- YARA matches
- High-entropy PE sections
This allows the full analysis data to remain available in JSON while giving the analyst a smaller set of indicators to investigate first.
Before the automation was used against malware samples, the complete workflow was tested against the legitimate Windows notepad.exe binary.
The test confirmed that the script could successfully:
- Calculate file hashes.
- Identify the sample as a Windows PE executable.
- Extract ASCII and UTF-16LE strings.
- Parse PE structure and imported functions.
- Calculate section entropy.
- Execute the YARA scanning stage.
- Generate the per-sample analysis directory and all three output files.
The benign test extracted 2,659 strings, consisting of 2,520 ASCII strings and 139 UTF-16LE strings.
The generated report.txt was also reviewed to confirm that the automated results were converted into a readable analyst-facing report.
The custom analyzer became the first stage of the malware-analysis workflow.
Suspicious Sample
|
v
+-------------------------+
| analyze_sample.py |
+------------+------------+
|
+------------+------------+
| | |
v v v
JSON Strings Report
| | |
+------------+------------+
|
v
Manual Validation
PE-bear / Strings
|
v
Behavioral Hypothesis
|
v
Dynamic Analysis
|
v
YARA Detection
Automation reduced repetitive collection work while preserving the analyst's role in interpreting the results.
This distinction became important during later analysis. The script successfully collected evidence that was not always elevated into the Key Findings summary, demonstrating that automated collection and analyst interpretation serve different purposes within the workflow.
Each sample was analyzed using a repeatable workflow designed to separate observed evidence from analyst interpretation.
The objective was not to label individual indicators as malicious in isolation, but to use multiple sources of evidence to develop hypotheses and then test those hypotheses through controlled execution.
Each sample was first processed with the custom analyze_sample.py script.
The automated triage stage collected:
- MD5, SHA-1, and SHA-256 hashes
- File type and extension consistency
- ASCII and UTF-16LE strings
- PE headers and section information
- Section entropy
- Imported DLLs and API functions
- Notable API indicators
- YARA results
The script also generated structured JSON output, a complete strings file, and a human-readable report for each sample.
Automated findings were then independently reviewed using tools such as PE-bear and Sysinternals Strings.
Manual analysis focused on identifying relationships between:
- Embedded strings
- Imported APIs
- PE structure
- Section names and entropy
- File paths and referenced components
- Potential packing or obfuscation indicators
This stage was used to validate automated findings and identify evidence that might require additional analyst interpretation.
Static indicators were used to develop testable behavioral hypotheses rather than being treated as proof of runtime behavior.
For example, combinations of file-enumeration APIs, file paths, service-related APIs, network-related imports, or packing indicators could suggest behaviors worth investigating dynamically.
The distinction between capability and observed behavior was maintained throughout the project.
Before execution, the sandbox environment and monitoring tools were prepared and a fresh snapshot was taken.
Samples were then executed under controlled conditions while Process Monitor, Sysmon, System Informer, and the simulated network environment were used as appropriate to observe:
- Process activity
- File-system operations
- Registry activity
- Service-related behavior
- Network-related behavior
Dynamic evidence was compared against the hypotheses developed during static analysis.
A hypothesis was considered supported only when the expected behavior was actually observed. When behavior could not be reproduced, the result was documented as unconfirmed rather than inferred from the static evidence alone.
Findings from the investigations were then used to develop original YARA rules.
Rule conditions were selected from combinations of sample characteristics such as distinctive strings, PE imports, and structural indicators rather than relying on a single generic characteristic.
Rules were compiled and tested against the analyzed samples to verify that they matched their intended target. Testing also exposed rule-development issues that required troubleshooting and refinement before successful validation.
The complete methodology can be summarized as:
Sample Intake
|
v
Automated Triage
|
v
Manual Static Analysis
|
v
Behavioral Hypothesis
|
v
Controlled Dynamic Analysis
|
v
Confirm / Refute Findings
|
v
Detection Engineering
This methodology allowed both positive and negative results to contribute to the investigation. A static indicator could identify a capability worth testing, but only observed runtime evidence was used to claim that the corresponding behavior occurred during analysis.
The first full case study analyzed Lab01-01.exe from the Practical Malware Analysis lab collection.
The objective was to apply the complete workflow to an unknown executable: automated triage, manual static analysis, hypothesis development, controlled execution, and evidence review.
| Attribute | Value |
|---|---|
| Sample | Lab01-01.exe |
| SHA-256 | 58898bd42c5bd3bf9b1389f0eeee5b39cd59180e8370eb9ea838a0b327bd6fe47 |
| Source | Practical Malware Analysis — Chapter 1 |
| File Type | Windows PE executable |
No malware binaries are included in this repository. Samples are referenced only by filename, hash, and analysis results.
The sample was first processed using the custom analyze_sample.py workflow.
Automated analysis collected the sample hashes, extracted strings, parsed the PE structure and imports, calculated section entropy, executed the YARA stage, and generated the standard JSON, strings, and human-readable report outputs.
The results were then manually reviewed rather than treating the automated report as a final determination.
PE-bear and Sysinternals Strings were used to examine the executable independently.
Several strings and imports were relevant to further investigation.
Notable strings included:
Lab01-01.dll
C:\*
C:\Windows\System32\kernel32.dll
WARNING_THIS_WILL_DESTROY_YOUR_MACHINE
The warning string was treated as an embedded indicator rather than proof that destructive behavior would occur.
PE-bear showed imports from KERNEL32.dll that included:
CloseHandle
UnmapViewOfFile
IsBadReadPtr
MapViewOfFile
CreateFileMappingA
CreateFileA
FindClose
FindNextFileA
FindFirstFileA
CopyFileA
The combination of FindFirstFileA, FindNextFileA, CopyFileA, file-mapping APIs, the C:\* string, and the reference to Lab01-01.dll suggested that file-system interaction was worth investigating dynamically.
However, these indicators established potential capability only. They did not demonstrate that the executable would perform those actions when executed.
The executable contained three primary sections:
| Section | Raw Size | Virtual Size |
|---|---|---|
.text |
0x1000 |
0x970 |
.rdata |
0x1000 |
0x2B2 |
.data |
0x1000 |
0xFC |
The section layout and entropy results did not provide strong evidence that the sample was packed.
This contrasted with the second case study, where packing indicators were substantially more apparent.
Based on the static evidence, the working hypothesis was:
Lab01-01.exemay enumerate files or directories and perform file-manipulation operations involving copying or mapped-file access.
This hypothesis was intentionally kept broader than labeling the executable a confirmed file infector. The static evidence justified testing file-system behavior, but not claiming that infection or destructive modification would occur.
The sample was executed inside the isolated Windows analysis VM while endpoint activity was monitored.
The suspected file-manipulation behavior was tested under multiple execution conditions:
- Initial execution from the default analysis location.
- Execution after relocating the sample.
- Execution with elevated privileges.
- Execution with the referenced companion file present.
Across these tests, the expected file-manipulation behavior suggested by the static evidence was not observed.
The available runtime evidence did not confirm file enumeration, copying, or mapped-file modification corresponding to the original hypothesis.
The dynamic result was therefore recorded as unconfirmed behavior, rather than interpreting the static indicators as proof.
Static Evidence
|
v
File-related APIs + paths + DLL reference
|
v
Hypothesis:
Potential file-manipulation behavior
|
v
Controlled Execution
|
v
Behavior not reproduced
|
v
Hypothesis not confirmed
under tested conditions
This was an important outcome for the project.
Static analysis successfully identified capabilities and relationships worth investigating, but dynamic analysis demonstrated why those indicators should not automatically be converted into behavioral claims.
The absence of the expected behavior does not establish that the executable is incapable of performing it. The behavior may depend on conditions that were not reproduced in the sandbox. The defensible conclusion is limited to what was observed:
The suspected file-manipulation behavior was not reproduced under the execution conditions tested during this analysis.
Lab01-01.exe demonstrated the importance of separating static capability indicators from observed runtime behavior.
Rather than modifying the conclusion to fit the original hypothesis, the negative dynamic result was preserved as part of the investigation. This established an evidence-driven approach that was then applied to the second case study.
The second case study analyzed Lab01-02.exe from the Practical Malware Analysis lab collection.
This sample provided a useful contrast to the first investigation. Static analysis revealed clear packing indicators along with service- and network-related capabilities, while controlled dynamic testing produced limited but meaningful runtime evidence.
| Attribute | Value |
|---|---|
| Sample | Lab01-02.exe |
| SHA-256 | c876a332d7dd8da331cb8eee7ab7bf32752834d4b2b54eaa362674a2a48f64a6 |
| Source | Practical Malware Analysis — Chapter 1 |
| File Type | Windows PE executable |
No malware binaries are included in this repository. Samples are referenced only by filename, hash, and analysis results.
Lab01-02.exe was first processed using the custom analyze_sample.py workflow.
The automation successfully collected PE section information, imports, strings, entropy values, hashes, and other analysis data.
One result became particularly important during manual review: the script had captured UPX-related section names and a section entropy value of approximately 7.067, but its Key Findings logic did not identify packing as a notable finding.
The underlying evidence had been collected successfully; the limitation was in how the automation prioritized that evidence for the analyst.
This exposed an important distinction between data collection and analysis logic.
Manual review using PE-bear and Sysinternals Strings revealed several strong indicators.
Notable strings included:
UPX0
UPX1
UPX2
UPX!
3.04
MalService
The presence of multiple UPX markers, combined with the section structure and elevated entropy, provided strong evidence that the executable was UPX-packed.
The sample also imported several functions that warranted further investigation:
LoadLibraryA
GetProcAddress
VirtualProtect
VirtualAlloc
VirtualFree
ExitProcess
CreateServiceA
InternetOpenA
The combination of LoadLibraryA and GetProcAddress also indicated that the visible import table might not represent every API used at runtime because additional functions could be resolved dynamically.
VirtualAlloc and VirtualProtect were treated as capability indicators rather than proof of process injection. In the context of the UPX evidence, memory allocation and protection changes were also consistent with packed-code execution or unpacking behavior.
The static evidence supported three primary observations:
- The executable was strongly consistent with UPX packing.
- The
MalServicestring andCreateServiceAimport suggested service-related behavior worth testing. - The
InternetOpenAimport indicated network-related capability.
These findings were treated as hypotheses for dynamic investigation rather than proof that service persistence or network communication would occur.
Based on the static evidence, the dynamic analysis focused on two questions:
- Would the executable demonstrate service-related behavior when executed under appropriate conditions?
- Would the executable attempt observable network communication inside the simulated network environment?
A fresh Proxmox snapshot was taken before execution, and Process Monitor was prepared to capture activity associated with Lab01-02.exe.
Process Monitor recorded runtime activity associated with the sample, including registry and file-system events.
One observed registry write occurred beneath:
HKLM\System\CurrentControlSet\Services\bam\State\UserSettings\<SID>\...
Although the path contains Services, this entry belongs to Windows Background Activity Moderator (BAM) state tracking. It represents operating-system bookkeeping associated with execution and was not treated as evidence that the malware registered itself as a Windows service.
A CreateFile operation targeting:
C:\Windows
was also observed.
This event was treated as directory access rather than evidence that the sample created or copied a file into the Windows directory.
These distinctions prevented generic Windows activity from being incorrectly attributed to malicious behavior.
Because static analysis identified both the MalService string and the CreateServiceA import, service-related execution was investigated further under controlled conditions.
The executable was manually registered as a Windows service for testing and launched through the Windows Service Control Manager.
During the test, the service entered a START_PENDING state and was assigned process ID 724. The process subsequently terminated, and the service returned to the STOPPED state.
The final service status reported:
WIN32_EXIT_CODE: 1067
Windows error 1067 indicates that the process terminated unexpectedly.
This demonstrated that the executable could be registered and launched through the Service Control Manager during controlled testing, but it did not remain operational as a service under the conditions tested.
Because service creation was performed manually as part of the experiment, this result was not treated as evidence that the malware itself successfully established service persistence.
The runtime evidence partially supported the static service-related hypothesis but did not justify a broader persistence claim.
Static Evidence
|
v
UPX + MalService + CreateServiceA
|
v
Service-related hypothesis
|
v
Controlled SCM execution
|
v
START_PENDING / PID 724
|
v
Process terminates
|
v
STOPPED / Error 1067
No confirmed network behavior was established from the retained runtime evidence, so the presence of InternetOpenA was documented as a static capability indicator rather than proof of network communication.
This investigation also revealed a weakness in the custom analyzer.
The script successfully collected the UPX section names and elevated entropy value, but its report-generation logic did not elevate the combination into the Key Findings section as a possible packing indicator.
Manual review identified the significance of the data.
Rather than treating this as a failure of the overall workflow, the result demonstrated why automated triage was intentionally paired with analyst validation:
The automation collected the evidence; analyst interpretation established its significance.
This finding identified a clear future improvement for the script: incorporate section-name and entropy relationships into the prioritization logic so common packing indicators can be surfaced automatically.
Lab01-02.exe demonstrated how static indicators can guide targeted dynamic testing while still requiring careful interpretation of runtime evidence.
The investigation identified strong UPX packing indicators and service- and network-related capabilities, but the final conclusions remained limited to what was actually observed.
Together, the two case studies produced contrasting results:
Lab01-01.exegenerated a file-manipulation hypothesis that could not be reproduced under the tested conditions.Lab01-02.exeproduced strong packing and service-related static indicators, followed by limited service-related runtime evidence during controlled testing.
Both outcomes reinforced the same principle:
Static analysis identifies what is worth testing; dynamic evidence determines what can be claimed about observed behavior.
Following static and dynamic analysis, original YARA rules were developed for both case-study samples.
The objective was not simply to create rules that matched known files. Each rule was built from combinations of characteristics identified during analysis, with an emphasis on avoiding overly broad conditions that could match unrelated software.
Both rules were loaded together and tested against their intended samples to verify that each sample matched the expected rule.
The first rule targeted characteristics identified during the Lab01-01.exe investigation.
Rather than relying on generic Windows API imports, the rule used distinctive embedded strings associated with the sample.
The final detection logic required a sample-specific warning string together with at least one additional corroborating indicator.
Conceptually, the condition followed this structure:
Distinctive warning string
AND
at least one additional indicator:
- referenced DLL
- C:\* path
- kernel32.dll path
This provided more specificity than matching any individual string while avoiding an unnecessarily brittle requirement that every indicator be present.
The initial Lab01-01 rule compiled successfully but failed to match the sample it was designed to detect.
Because successful compilation only confirmed that the rule was syntactically valid, the failed functional test required the individual strings to be compared against the actual extracted sample content.
Two discrepancies were identified.
The rule initially searched for:
LAB01-01.dll
while the extracted sample string was:
Lab01-01.dll
The rule was case-sensitive, so the difference prevented that indicator from matching.
The warning string also differed from the exact value used in the initial rule. The rule expected additional punctuation surrounding the warning, while validation of the extracted sample content showed that the detection string needed to be adjusted to match the verified sample data.
Because the warning string was mandatory in the rule condition, that mismatch prevented the entire rule from matching even when other indicators were present.
The rule was corrected and retested.
Write Rule
|
v
Compilation Successful
|
v
Test Against Source Sample
|
v
0 Matches
|
v
Inspect Individual Strings
|
v
Case / Exact-String Mismatch Identified
|
v
Correct Rule
|
v
Retest
|
v
Successful Match
This demonstrated an important distinction in detection engineering:
A rule that compiles successfully is not necessarily a rule that works correctly.
Functional validation against known evidence was required before the rule could be considered successful.
The second rule targeted the stronger structural and behavioral indicators identified during analysis of Lab01-02.exe.
The sample contained multiple UPX markers:
UPX0
UPX1
UPX2
However, UPX alone was not considered sufficiently specific because legitimate software can also be packed with UPX.
The rule therefore combined multiple categories of evidence:
At least two UPX section indicators
AND
"MalService"
AND
PE import: ADVAPI32.dll -> CreateServiceA
AND
PE import: WININET.dll -> InternetOpenA
The YARA pe module was used to validate the imported functions directly rather than relying only on their appearance as strings.
The resulting condition followed this structure:
2 of ($upx0, $upx1, $upx2)
and $svcname
and pe.imports("ADVAPI32.dll", "CreateServiceA")
and pe.imports("WININET.dll", "InternetOpenA")This combined packing indicators, a sample-specific service string, and PE import characteristics into a more selective rule.
Fragmented URL-related strings observed during static analysis were intentionally excluded because they were less reliable detection anchors.
Both custom rules were loaded simultaneously into the Python analysis environment and tested against the two case-study samples.
The validation produced the expected results:
| Sample | Rules Matched | Matched Rule |
|---|---|---|
Lab01-01.exe |
1 | PMA_Lab01_01_FileManipulation |
Lab01-02.exe |
1 | PMA_Lab01_02_PackedServiceBackdoor |
Each sample matched only its intended rule while both rules were loaded.
This provided an initial specificity check and confirmed that the rules functioned as intended against the analyzed samples.
The validation should not be interpreted as comprehensive false-positive testing across a large benign or malicious dataset. It demonstrates functional correctness within the scope of the samples analyzed in this project.
YARA development extended the project beyond identifying suspicious characteristics and into translating analysis findings into reusable detection logic.
The Lab01-01 rule also demonstrated why detection development requires iterative testing. Syntax validation alone did not reveal the exact-string mismatch; testing against the source sample exposed the problem and allowed the rule to be corrected.
The Lab01-02 rule demonstrated a different principle: common indicators such as UPX packing become more useful when combined with additional sample-specific and structural evidence.
Together, the rules followed the same evidence-driven approach used throughout the project:
Analyze → Select Indicators → Write Rule → Test → Debug → Validate
Building the sandbox and completing the investigations required troubleshooting several issues involving network simulation, offline dependency management, sample validation, automation logic, and detection engineering.
Rather than weakening the sandbox to work around these problems, the goal was to resolve them while preserving the project's isolation and evidence-driven analysis methodology.
During configuration of the simulated network environment, INetSim's built-in DNS service failed to bind to port 53 even though other INetSim services were operational.
Troubleshooting identified a compatibility issue between the installed INetSim 1.3.2 DNS module and Net::DNS 1.44.
Instead of abandoning the simulated-network design, the DNS component was separated from INetSim. The built-in DNS service was disabled and a standalone FakeDNS service was configured to resolve requested domains to the REMnux system at 192.168.66.20.
INetSim continued providing simulated application-layer services.
The solution was validated end-to-end from Windows:
DNS Query
|
v
FakeDNS -> 192.168.66.20
|
v
HTTP Request
|
v
INetSim
|
v
HTTP 200 Simulated Response
This preserved the sandbox's network isolation while restoring the simulated Internet functionality needed for analysis.
The Windows analysis VM intentionally had no Internet access, which complicated installation of Python dependencies.
An attempt to install pefile through pip encountered an unavailable dependency chain that required additional packages such as setuptools.
Rather than temporarily connecting the analysis VM to the Internet, the required pefile, peutils, and ordlookup source components were staged offline and placed directly into the local analysis environment.
The Python imports were then validated before PE parsing was integrated into analyze_sample.py.
This allowed development of the automation to continue without changing the sandbox's network-isolation model.
A standalone prebuilt YARA executable triggered a Microsoft Defender heuristic detection during tooling setup.
Because the binary was not required to complete the project, Defender protections were not weakened simply to force the executable into the environment.
Instead, YARA functionality was implemented through yara-python 4.5.4, which was staged as a compatible Python wheel and installed offline.
The Python implementation was then successfully integrated into the custom analyzer for local rule compilation and scanning.
This provided the required YARA functionality while avoiding dependence on the standalone executable that had triggered the security control.
EICAR was used as an early safe test sample before introducing malware lab binaries.
The initial test file did not produce the expected published hash values.
Rather than assuming the reference hashes were incorrect, the file contents were compared at the byte level. The discrepancy was traced to a single-character transcription error: the letter O had been used where the EICAR test string required the digit 0.
After correcting the byte sequence, the file produced the expected hashes:
MD5
44d88612fea8a8f36de82e1278abb02f
SHA-1
3395856ce81f2b7382dee72602f798b642f14140
SHA-256
275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f
Microsoft Defender then immediately detected the corrected EICAR file.
This demonstrated why hash validation was treated as part of sample integrity rather than simply as metadata collection.
Analysis of Lab01-02.exe exposed a limitation in the custom analyzer.
The script successfully recorded UPX-related section names and a section entropy value of approximately 7.067. However, the report-generation logic did not elevate those combined indicators into the Key Findings section as evidence of possible packing.
Manual analysis using PE-bear and Strings identified their significance.
The problem was therefore not failure to collect the underlying evidence, but a limitation in the script's prioritization logic:
Evidence Collection Successful
|
v
UPX Sections + Entropy 7.067
|
v
Report Prioritization Incomplete
|
v
Manual Review
|
v
Packing Significance Identified
This identified a future improvement for the analyzer: correlate suspicious section names with entropy values and surface potential packing indicators automatically.
More importantly, it reinforced the decision not to treat automated triage as a replacement for analyst review.
The first custom YARA rule introduced another distinction between successful execution and successful detection.
The rule compiled without errors but returned zero matches against the same Lab01-01.exe sample it had been designed to detect.
Individual rule strings were compared against the verified sample content. This exposed exact-string and case-sensitivity differences, including the DLL reference:
Rule:
LAB01-01.dll
Sample:
Lab01-01.dll
The warning-string condition also contained characters that did not match the verified detection string being tested.
Because the warning string was mandatory in the rule condition, the mismatch prevented the rule from matching regardless of the presence of other indicators.
After correcting the detection strings, the rule was retested and successfully matched its intended sample.
The troubleshooting process demonstrated that:
Successful compilation validates YARA syntax; functional testing validates detection logic.
Several recurring principles emerged from these problems:
- Preserve isolation rather than solving dependency problems by introducing unnecessary Internet access.
- Validate assumptions using observable evidence.
- Verify sample integrity before trusting analysis results.
- Distinguish successful data collection from successful prioritization.
- Treat security-tool alerts as information to investigate rather than controls to automatically bypass.
- Test detection rules against real evidence instead of assuming syntactically valid rules are functionally correct.
These troubleshooting experiences became part of the project itself rather than being treated only as setup problems. Each issue resulted in a change to the environment, workflow, automation, or detection logic that could be independently validated.
This project developed a complete malware-analysis workflow rather than focusing on a single tool or analysis technique.
Building the sandbox, developing the automation, investigating the samples, and creating detection rules demonstrated practical experience across several areas of malware analysis and cybersecurity.
- Performed automated and manual static analysis of Windows PE files.
- Examined hashes, strings, PE sections, entropy, imports, and embedded indicators.
- Used static evidence to develop testable behavioral hypotheses.
- Performed controlled dynamic analysis using Process Monitor, Sysmon, and System Informer.
- Distinguished potential capabilities identified statically from behavior actually observed at runtime.
- Documented negative and inconclusive results rather than overstating findings.
- Developed a custom Python triage tool to automate repetitive analysis tasks.
- Implemented MD5, SHA-1, and SHA-256 hashing.
- Identified files using signatures rather than relying solely on extensions.
- Extracted ASCII and UTF-16LE strings.
- Parsed PE structure and imports using
pefile. - Calculated PE section entropy and flagged selected notable APIs.
- Integrated
yara-pythonfor automated rule scanning. - Generated structured JSON results, complete string output, and human-readable reports.
- Identified a limitation in the report-prioritization logic through manual validation.
- Developed original YARA rules from indicators identified during analysis.
- Combined sample-specific strings, structural characteristics, and PE imports to improve rule specificity.
- Used the YARA
pemodule to evaluate imported functions directly. - Tested multiple rules simultaneously against analyzed samples.
- Debugged a syntactically valid rule that failed functional testing because of exact-string and case-sensitivity differences.
- Distinguished functional validation from comprehensive false-positive testing.
- Designed an isolated Proxmox malware-analysis network using a dedicated Linux bridge with no physical uplink or external route.
- Validated isolation from the production homelab and public Internet before sample execution.
- Configured REMnux to provide controlled network-analysis services.
- Implemented FakeDNS and INetSim to create a simulated Internet environment without exposing samples to real external infrastructure.
- Diagnosed and worked around an INetSim DNS compatibility issue while preserving the isolation model.
- Used snapshots to support controlled testing and recovery.
The project also reinforced several broader security-engineering principles:
- Containment should not be weakened for convenience. Offline dependency and tooling problems were solved without introducing unnecessary Internet connectivity into the analysis environment.
- Automation should support analyst judgment, not replace it. The analyzer collected useful evidence, but manual review was still required to understand its significance.
- Static indicators are hypotheses, not runtime proof. Capabilities suggested by imports and strings were tested dynamically before behavioral claims were made.
- Negative results are still evidence. When expected behavior could not be reproduced, the result was documented rather than forced into the original hypothesis.
- Detection logic requires validation. A rule compiling successfully does not guarantee that it detects what it was designed to detect.
The most important lesson from the project was that malware analysis is not simply about identifying suspicious artifacts. It is a process of collecting evidence, interpreting that evidence, testing assumptions, and limiting conclusions to what the investigation can actually support.
Collect → Validate → Hypothesize → Test → Detect
This project was created for educational and defensive cybersecurity research purposes.
All malware analysis was performed inside an isolated lab environment designed to prevent communication with the production homelab and public Internet. Malware samples are not distributed through this repository; only analysis results, hashes, detection rules, screenshots, and supporting documentation are included.
The techniques and tooling documented in this project are intended to demonstrate malware-analysis, detection-engineering, and defensive security skills in a controlled environment.