Sandbox Malware Analysis: How Security Teams Use Sandboxes to Detect Unknown Threats

A phishing email arrives with a PDF attachment. Your antivirus engine checks the file hash against its signature database. No match. It scans for known malicious patterns. Nothing flagged. The file is new, unrecognised, and potentially dangerous. How do you determine whether it's safe without risking your production environment?

You detonate it in a sandbox.

Sandbox malware analysis is the process of executing suspicious files in a controlled, isolated environment that monitors every action the file takes: which processes it creates, which files it drops, which network connections it establishes, which registry keys it modifies, and which system APIs it calls. The sandbox observes behaviour rather than matching signatures, making it effective against zero-day malware, polymorphic threats, and previously unseen attack tools that no signature database contains.

Every enterprise email gateway, most endpoint detection platforms, and every serious SOC operation uses sandbox malware analysis as a core detection capability. Understanding how sandboxes work, what tools are available, what analysts look for in sandbox reports, and where sandboxes have limitations is essential knowledge for security teams defending modern networks.

This guide covers the step-by-step sandbox malware analysis process, the tools security teams use, how to read and interpret sandbox analysis reports, what analysts look for at each stage, and how sandbox analysis connects to threat hunting and incident response. For foundational sandbox concepts, see our sandbox cybersecurity guide. For the technical execution process, see our dynamic malware analysis guide.

How Sandbox Malware Analysis Works: Step by Step

Step 1: Sample Acquisition

Before analysis begins, the suspicious file must be captured and isolated.

Sources of samples: Email gateway quarantine (suspicious attachments flagged by email security). EDR alerts (files flagged as suspicious on endpoints). SOC escalation (analyst investigates an alert and extracts a file for deeper analysis). Threat intelligence feeds (samples shared through ISACs, VirusTotal, malware repositories). Incident response (files collected from compromised systems during investigations). Attack surface monitoring (suspicious files discovered on external-facing systems).

Sample handling: Samples must be handled safely. Transfer via encrypted archives (password-protected ZIP). Never execute on production systems. Store in dedicated malware repositories with access controls. Maintain chain of custody for incident response and legal purposes.

Step 2: Environment Preparation

The sandbox environment must be configured to match the likely target and resist detection by sandbox-aware malware.

Virtual machine configuration: Operating system matching the target environment (Windows 10/11 for most enterprise malware, Windows Server for server-targeting threats). Common applications installed (Microsoft Office for document-based malware, PDF readers, web browsers). Realistic hardware profile: 4+ CPU cores, 8+ GB RAM, 256+ GB disk to defeat hardware fingerprinting evasion. User profile populated with documents, browser history, and desktop files to simulate a used workstation.

Network configuration: Simulated internet services (DNS, HTTP, HTTPS, SMTP) capturing all communication without allowing real internet access. Alternatively, controlled real internet access through a monitored proxy for samples requiring live connectivity (with C2 takedown risk considerations).

Monitoring setup: System monitoring capturing file operations, registry changes, process creation, and API calls. Network capture recording all traffic. Memory dumping capability for post-execution analysis. Screenshot capture at intervals. For understanding how sandboxes differ from plain virtual machines in this setup, see our sandbox vs virtual machine comparison.

Step 3: Sample Execution

The sample is executed within the sandbox using the method appropriate to its file type.

Execution by file type:

Execution triggers: Some malware requires user interaction before executing its payload. Advanced sandboxes simulate: mouse movement and clicks, keyboard input, dialog box interaction ("Enable Macros" clicks), document scrolling, and waiting periods. Interactive sandboxes (like ANY.RUN) allow analysts to provide real human interaction.

Step 4: Behaviour Monitoring

While the sample executes, the sandbox captures every observable action.

File system monitoring captures:

Files created (dropped executables, scripts, configuration files, ransom notes). Files modified (encrypted files in ransomware cases, hosts file modification). Files deleted (self-deletion to remove traces). File system paths reveal intent: files dropped in startup directories indicate persistence; files dropped in temp directories indicate staging.

Process monitoring captures:

Process creation (child processes spawned, including their command-line arguments). Process injection (code injected into legitimate processes like explorer.exe, svchost.exe). Process termination (killing security software). Parent-child process relationships revealing the execution chain.

Registry monitoring captures:

Registry keys created (persistence through Run keys, scheduled tasks, services). Registry keys modified (disabling security features, modifying system configuration). Registry keys queried (environment detection, checking for specific software).

Network monitoring captures:

DNS queries revealing C2 domains. HTTP/HTTPS connections showing communication patterns. Raw TCP/UDP connections to specific IPs and ports. Data volume indicating exfiltration. Protocol analysis revealing custom C2 protocols.

API call monitoring captures:

Cryptographic API calls (suggesting encryption, potentially ransomware). Process manipulation APIs (suggesting injection). Network APIs (suggesting communication). File APIs (suggesting file operations beyond normal application behaviour).

Step 5: Analysis and Classification

After the execution window (typically 5 to 15 minutes, longer for evasion-resistant analysis), the sandbox processes captured data into a structured analysis.

Behavioural classification: The sandbox compares observed behaviour against known malware behaviour profiles.

Severity scoring: The sandbox assigns a threat score based on the combination and severity of observed behaviours. A file that creates persistence, establishes C2, and begins encrypting files scores higher than a file that only creates a temporary file.

Step 6: IOC Extraction

The sandbox extracts indicators of compromise from the analysis for use in detection and response.

Network IOCs: C2 IP addresses (for firewall blocking). C2 domain names (for DNS sinkholing). URLs contacted (for proxy blocking). JA3/JA3S hashes (TLS fingerprints for network detection). User-Agent strings (for proxy log searching).

File IOCs: SHA-256 hashes of the original sample and all dropped files. File names and paths. YARA signatures generated from unique binary patterns.

Host IOCs: Registry keys created (for endpoint detection rules). Scheduled task names (for persistence hunting). Mutex names (for identifying running infections). Service names created.

These IOCs feed directly into threat hunting operations: security teams search their environment for the extracted indicators to determine whether the threat has already compromised other systems.

Step 7: Report Generation

The sandbox produces a comprehensive report consolidating all observations into an actionable document.

What Analysts Look For in Sandbox Reports

The Executive Verdict

Clean, Suspicious, or Malicious. The top-level classification. But experienced analysts never stop at the verdict. A "clean" verdict on a suspicious file may indicate sandbox evasion, not a genuinely benign file.

Process Tree

The process tree shows the execution chain: which processes were created, by whom, and in what sequence.

What to look for:

Parent process unexpected (Word spawning PowerShell, PDF reader spawning cmd.exe). Multi-stage execution chains (document opens, drops a script, script downloads an executable, executable injects into svchost.exe). Process injection (legitimate process suddenly performing malicious activities). Execution of living-off-the-land binaries (LOLBins): certutil, mshta, wscript, rundll32 used for malicious purposes.

Network Activity

What to look for:

DNS queries to newly registered domains or domains with high entropy (suggesting DGA, Domain Generation Algorithm). HTTP POST requests to unknown servers (data exfiltration). Regular-interval connections (beaconing to C2). Connections to known-malicious infrastructure. Encrypted communication on non-standard ports. Large outbound data transfers (exfiltration indicator).

File System Changes

What to look for:

Executables dropped in user temp directories, startup folders, or system directories. Batch files or scripts created for execution. Ransom notes dropped across directories. Large numbers of file modifications (encryption in progress). Deletion of volume shadow copies (ransomware preparation). Modification of the hosts file (blocking security update servers).

Registry Modifications

What to look for:

Run key entries (persistence through auto-start). Scheduled task creation through registry. Service registration. Security feature modification (disabling Windows Defender, disabling UAC). Firewall rule additions (allowing C2 communication).

Evasion Indicators

What to look for:

Environment detection queries (WMI queries for hardware, registry queries for VM artifacts). Extended sleep calls before malicious activity. Process enumeration scanning for analysis tools. Network connectivity tests before payload execution. These indicate sandbox evasion techniques and should be treated as suspicious even if no malicious payload executed.

Sandbox Malware Analysis Tools

Cuckoo Sandbox

What it is: The most widely deployed open-source malware analysis sandbox. Automates execution, monitoring, and reporting of suspicious files.

Capabilities: Supports Windows, Linux, macOS, and Android analysis guests. File system, registry, process, and network monitoring. API call tracking. Memory dump collection. YARA rule matching. Signature-based detection alongside behavioural analysis. Extensible through community modules.

Best for: Organisations building in-house malware analysis capability. Security teams wanting full control over the analysis environment. Integration into automated analysis pipelines. Budget-conscious deployments.

Limitations: Requires significant setup and maintenance. VM detection by sophisticated malware. Community support varies.

ANY.RUN

What it is: Interactive cloud-based malware analysis platform allowing real-time human interaction with samples during execution.

Capabilities: Analysts interact with malware in real time: clicking dialogs, entering data, browsing, scrolling documents. Visual process tree. Network traffic capture. MITRE ATT&CK mapping of observed techniques. Shareable analysis sessions. Growing database of community analyses.

Best for: SOC analysts who need interactive analysis (clicking through macro warnings, navigating multi-stage infection chains). Investigating samples that require human interaction to trigger payloads. Training junior analysts in malware behaviour.

Limitations: Cloud-based (samples are submitted to third-party infrastructure). Subscription pricing for advanced features.

Joe Sandbox

What it is: Commercial deep malware analysis platform with advanced anti-evasion technology.

Capabilities: Supports Windows, macOS, Linux, Android, and iOS. Hypervisor-based monitoring options. Detailed behavioural analysis with classification scoring. Extensive anti-evasion techniques. API for automated submission. Integration with SIEM and SOAR platforms.

Best for: Enterprise SOC teams requiring high-throughput automated analysis. Financial institutions and regulated industries needing commercial support. Environments where anti-evasion capability is critical.

VMRay

What it is: Hypervisor-based malware analysis platform with monitoring at the hypervisor level (below the guest OS), making detection by malware extremely difficult.

Capabilities: Hypervisor-level monitoring (no guest-side agents to detect). Automated classification. Comprehensive IOC extraction. MITRE ATT&CK mapping. Enterprise API integration.

Best for: Analysing sophisticated malware that detects and evades traditional sandboxes. Organisations facing advanced persistent threats. Environments requiring the strongest anti-evasion capability.

Hybrid Analysis (CrowdStrike Falcon Sandbox)

What it is: Free community malware analysis platform backed by CrowdStrike's threat intelligence.

Capabilities: Combined static and dynamic analysis. Community-shared analysis results. CrowdStrike threat intelligence correlation. Free public submissions. API access.

Best for: Quick triage of suspicious files. Checking whether a file has been analysed before. Community threat intelligence sharing. Budget-constrained teams needing sandbox capability.

CAPE Sandbox (Cuckoo-based)

What it is: Fork of Cuckoo Sandbox with enhanced malware configuration extraction and payload dumping capabilities.

Capabilities: Extracts malware configurations (C2 addresses, encryption keys) from running samples. Better unpacking of packed malware. Enhanced memory analysis. YARA integration.

Best for: Malware analysts needing configuration extraction. Threat intelligence teams building IOC feeds. Advanced analysis beyond behavioural classification.

Tool Comparison

Building a Sandbox Malware Analysis Workflow

For SOC Teams

Tier 1 (Automated): Email gateway and EDR automatically submit suspicious files to sandbox. Clean verdicts pass. Malicious verdicts alert SOC. Suspicious verdicts queue for analyst review.

Tier 2 (Analyst-Assisted): SOC analyst reviews sandbox reports for queued samples. Interactive analysis (ANY.RUN) for samples requiring human interaction. Analyst determines final verdict and response action.

Tier 3 (Deep Analysis): Malware analyst investigates complex samples. Memory forensics on sandbox dumps. Configuration extraction. Reverse engineering for novel threats. IOC production for detection engineering.

Integration Points

Email gateway. Attachments automatically detonated before delivery. Malicious verdicts block delivery. Suspicious verdicts hold for analyst review.

EDR. Suspicious binaries submitted from endpoints. Verdicts feed back into EDR policy (block, isolate, alert).

SIEM. Sandbox verdicts and IOCs flow into SIEM. Alert correlation between sandbox findings and endpoint/network telemetry.

Threat intelligence platform. Extracted IOCs feed TIP. Enrichment from TIP feeds sandbox analysis (known C2, known malware families).

Threat hunting. IOCs from sandbox analysis become hunting hypotheses. "This ransomware sample contacts domain X. Does any system in our environment communicate with domain X?"

Workflow Automation

Automate the pipeline: file arrival triggers sandbox submission, analysis runs automatically, verdicts update security controls, IOCs propagate to blocking and detection, and analysts focus on samples requiring human judgment rather than processing every file manually.

Interpreting Sandbox Results: Common Scenarios

Scenario 1: Clear Malicious Verdict

What the report shows: Document drops PowerShell script. Script downloads executable from external URL. Executable creates persistence through registry Run key. Establishes C2 connection. Begins collecting browser credentials.

Analyst action: Block the C2 domain and IP at the network perimeter. Hunt for the downloaded executable hash across all endpoints. Hunt for the registry persistence key. Block the download URL at the proxy. Update email gateway rules for the initial document indicators.

Scenario 2: Clean Verdict on Suspicious File

What the report shows: File executes, checks registry for VMware/VirtualBox keys, queries hardware via WMI, then terminates without further activity.

Analyst assessment: This isn't clean. It's evasive. The file detected the sandbox and suppressed its payload. See our sandbox evasion guide for the techniques involved.

Analyst action: Resubmit to a sandbox with stronger anti-evasion (VMRay or bare-metal). Try interactive analysis (ANY.RUN) with manual interaction. If the file passed to an endpoint, monitor that endpoint for delayed execution. Flag the file hash as "evasive" in threat intelligence.

Scenario 3: Partial Execution

What the report shows: File attempts to download a second-stage payload from a URL. The URL is offline or returns an error. No further malicious activity.

Analyst assessment: The file is a first-stage loader. It's malicious by design but the payload wasn't available during analysis.

Analyst action: Block the download URL. Hunt for the download URL in proxy logs (other systems may have successfully downloaded the payload). Monitor for the URL becoming active. Classify the loader hash as malicious.

Scenario 4: Ransomware Detection

What the report shows: File enumerates drives. Begins encrypting files with unique extensions. Drops ransom note in every directory. Deletes volume shadow copies. Contacts C2 for key exchange.

Analyst action: Emergency response. Block C2 immediately. Hunt for the file hash across all endpoints. Verify no endpoints are actively encrypting. Extract encryption key exchange details if captured. Isolate any affected systems. Feed all IOCs into every detection layer. See our ransomware trends analysis for current threat context.

Sandbox Analysis for Incident Response

During active incidents, sandbox analysis provides critical intelligence.

Collected sample analysis. When incident responders collect malware from compromised systems, sandbox analysis reveals: what the malware does (full capability assessment), where it communicates (C2 infrastructure for blocking), how it persists (for complete removal), and what data it targets (for impact assessment).

Scope determination. IOCs extracted from sandbox analysis enable searching the entire environment for the same threat. If the sandbox reveals the malware contacts domain X and creates registry key Y, searching for these indicators across all systems determines breach scope.

Attribution support. Malware behaviour patterns, C2 infrastructure, and code characteristics can support attribution to known threat actors when correlated with threat intelligence.

Sandbox Analysis and Red Teaming

Red team operations and sandbox malware analysis interact in two directions.

Red teams test sandboxes. During offensive security testing, red teams build custom payloads designed to evade the target organisation's sandbox. If the payload passes sandbox analysis and executes on the target, the sandbox capability has a gap.

Sandboxes test red team tools. Before deploying tools during an engagement, red teams test their payloads in internal sandboxes to verify the tools behave as intended and to understand what their tools look like from the defensive perspective.

This dynamic ensures sandboxes are tested against realistic adversary tools, not just commodity malware. Red teaming engagements provide this validation.

Sandbox Malware Analysis Checklist

Pre-Analysis

• Sample safely acquired and isolated (encrypted archive, dedicated storage)
• Chain of custody documented (for incident response/legal purposes)
• Sandbox environment prepared with target-appropriate OS and applications
• Network monitoring configured (real or simulated)
• Anti-evasion measures active (realistic hardware, user simulation)
• Snapshot taken for clean restore after analysis

During Analysis

• File executed with appropriate method for file type
• Minimum 5-minute execution window (longer for evasion-resistant analysis)
• File system changes captured (created, modified, deleted files)
• Registry modifications captured
• Process tree recorded (parent-child relationships, command lines)
• Network traffic captured (DNS, HTTP, raw connections)
• API calls logged
• Memory dump collected
• Screenshots captured at intervals

Post-Analysis

• Behavioural classification completed
• Threat score assigned
• Network IOCs extracted (IPs, domains, URLs, JA3 hashes)
• File IOCs extracted (hashes of sample and dropped files)
• Host IOCs extracted (registry keys, file paths, mutex names)
• MITRE ATT&CK techniques mapped
• IOCs fed into SIEM, firewall, proxy, and EDR
• Environment searched for extracted IOCs
• Report generated and stored
• Sandbox environment restored to clean snapshot

Evasion Assessment

• Environment detection indicators reviewed
• Timing-based evasion indicators checked
• Clean verdict on suspicious file treated as potentially evasive
• Resubmission to alternative sandbox considered for suspicious-clean verdicts

How AppSecure Validates Sandbox Effectiveness

AppSecure's offensive security testing and red teaming validate whether your sandbox-based detection works against realistic threats.

Sandbox Bypass Testing. Red team engagements test whether your specific sandbox platforms detect custom adversary tooling. Testing reveals which evasion techniques defeat your sandbox and which detection layers catch what the sandbox misses.

Multi-Layer Detection Validation. Testing evaluates the entire detection chain: email sandbox, endpoint detection, network monitoring, and SOC response. Gaps at any layer are identified with specific improvement recommendations.

Comprehensive Security Testing. Web application, API, cloud, network, and mobile penetration testing identifies the vulnerabilities that malware exploits. Fix vulnerabilities through testing. Detect exploitation through sandbox analysis.

Application security assessment provides comprehensive coverage. Continuous testing and PTaaS maintain ongoing validation.

Contact AppSecure:

• Schedule Security Validation
• Red Teaming Services
• Sandbox Cybersecurity Guide

Frequently Asked Questions

1. What is sandbox malware analysis?

Sandbox malware analysis is the process of executing suspicious files in a controlled, isolated environment that monitors every action the file takes: file system changes, registry modifications, network connections, process creation, and API calls. The sandbox observes runtime behaviour rather than matching static signatures, making it effective against zero-day malware and previously unseen threats. It is the primary method for determining whether an unknown file is malicious.

2. How does sandbox analysis detect zero-day malware?

Zero-day malware has no matching signature in antivirus databases, but it still exhibits malicious behaviour when executed: establishing C2 connections, creating persistence, encrypting files, or exfiltrating data. Sandbox analysis detects these behaviours regardless of whether the specific malware variant has been catalogued. The behaviour is the detection signal, not the file's identity. This is why sandbox analysis is the primary defense against novel threats.

3. What sandbox tools do security teams use?

Major platforms include Cuckoo Sandbox (open-source, most widely deployed), ANY.RUN (interactive cloud-based with real-time analyst interaction), Joe Sandbox (commercial with advanced anti-evasion), VMRay (hypervisor-level monitoring for strongest evasion resistance), Hybrid Analysis (free community platform backed by CrowdStrike), and CAPE Sandbox (enhanced Cuckoo fork with configuration extraction). Selection depends on budget, analysis volume, anti-evasion requirements, and whether interactive or automated analysis is needed.

4. What do analysts look for in sandbox reports?

Analysts examine the process tree (unexpected parent-child relationships, LOLBin usage), network activity (C2 communication, DNS queries to suspicious domains, beaconing patterns), file system changes (dropped executables, ransom notes, encrypted files), registry modifications (persistence mechanisms, security feature disabling), and evasion indicators (environment detection queries, extended sleep calls). Experienced analysts treat clean verdicts on suspicious files with scepticism, recognising potential sandbox evasion.

5. Can malware evade sandbox analysis?

Yes. Sophisticated malware detects sandbox environments through VM artifact detection, hardware fingerprinting, timing delays, user interaction requirements, and network environment checks. 30% to 40% of commodity malware and nearly 100% of targeted threat tooling implements evasion. Countermeasures include realistic hardware profiles, hypervisor-level monitoring, bare-metal analysis, and interactive sandboxes with human interaction. See our sandbox evasion guide for the complete evasion landscape.

6. What is the difference between sandbox analysis and antivirus?

Antivirus primarily uses signature matching: comparing files against a database of known malware signatures. Sandbox analysis uses behavioural observation: executing files and watching what they do. Antivirus is fast and scales to every file on every endpoint but misses novel malware. Sandbox analysis catches novel threats through behaviour but is slower and resource-intensive. Modern security stacks use both: antivirus for known threats at endpoint scale, sandboxes for unknown threats requiring deeper analysis.

7. How does sandbox analysis integrate with SOC operations?

Sandbox analysis integrates at multiple points: email gateways automatically submit attachments for detonation, EDR submits suspicious endpoint files, SOC analysts manually submit files during alert investigation, and incident responders analyse collected samples. Verdicts feed back into security controls (blocking decisions). Extracted IOCs flow into SIEM (detection rules) and firewalls (blocking rules). Threat hunters use IOCs as search hypotheses across the environment.

8. How long does sandbox malware analysis take?

Automated analysis typically runs 5 to 15 minutes per sample, with extended windows for evasion-resistant analysis (30+ minutes). Interactive analysis sessions (ANY.RUN) may run 10 to 30 minutes depending on complexity. Post-analysis processing (IOC extraction, report generation) adds 1 to 5 minutes for automated platforms. Enterprise sandboxes processing email attachments typically return verdicts within 10 minutes to minimise email delivery delays.

9. What are indicators of compromise from sandbox analysis?

Sandbox analysis produces network IOCs (C2 IP addresses, domains, URLs, JA3 TLS fingerprints), file IOCs (SHA-256 hashes of the sample and all dropped files), host IOCs (registry keys, file paths, scheduled task names, mutex names, service names), and behavioural IOCs (MITRE ATT&CK technique mappings). These feed into SIEM detection rules, firewall blocking rules, proxy policies, and threat hunting hypotheses.

10. Should I use a cloud or on-premises sandbox?

On-premises sandboxes (Cuckoo, VMRay on-prem) provide full control over samples and data, important for organisations with strict data handling requirements or analysing sensitive incident evidence. Cloud sandboxes (ANY.RUN, Joe Sandbox Cloud, Hybrid Analysis) provide convenience, maintained infrastructure, and community intelligence but require submitting samples to third-party platforms. Many organisations use cloud sandboxes for initial triage and on-premises sandboxes for sensitive analysis.