CrackMeBench: Binary Reverse Engineering for Agents
Mentions AgentRE-Bench as the closest related benchmark for stripped ELF reverse-engineering tasks, with emphasis on malware-like protocol and infrastructure reconstruction.
Read on arXivAgentRE-Bench evaluates LLM agents on compiled reverse-engineering tasks with no source code, no hints, and deterministic scoring.
The public benchmark is the credibility layer: open tasks, reproducible runs, and a leaderboard for AI reverse-engineering agents. Private evaluations and training-environment pilots test whether agents generalize to unreleased binaries instead of producing plausible reports.
Mentions AgentRE-Bench as the closest related benchmark for stripped ELF reverse-engineering tasks, with emphasis on malware-like protocol and infrastructure reconstruction.
Read on arXivCites AgentRE-Bench in the automated vulnerability discovery and AI security benchmark landscape.
Read on IEEE XploreAll 13 Linux ELF levels, 25 tool-call budget, Docker-sandboxed static analysis tools. Six frontier models evaluated end-to-end. Headline finding: a small non-thinking model (Gemini 3.1 Flash Lite) leads the field, beating every frontier reasoning model on Main score. Hallucination calibration — not reasoning depth — is the dominant axis on this bench. Windows PE results coming soon.
Bench leader with the lowest hallucination rate in the field. Calibration, not raw reasoning depth, is the win condition here.
| Metric | Gemini Flash Lite |
|---|---|
| Valid submissions | 13 / 13 |
| Submission rate | 100% |
| Correctness score | 0.667 |
| Hallucinations / task | 1.92 |
| Level | Task | Flash Lite | V4 Pro | Opus 4.7 | Kimi K2.6 | V4 Flash | GPT-5.5 |
|---|---|---|---|---|---|---|---|
| 1 | TCP Reverse Shell | 0.80 | 0.66 | 0.43 | 0.88 | 0.72 | 0.49 |
| 2 | XOR Encoded | 0.59 | 0.65 | 0.75 | 0.83 | 0.30 | 0.58 |
| 3 | Anti-Debug | 0.61 | 0.00 | 0.27 | 0.53 | 0.58 | 0.32 |
| 4 | Polymorphic | 0.05 | 0.40 | 0.00 | 0.40 | 0.14 | 0.00 |
| 5 | Multistage | 0.70 | 0.00 | 0.57 | 0.63 | 0.69 | 0.32 |
| 6 | ICMP Covert | 0.71 | 0.64 | 0.38 | 0.58 | 0.59 | 0.38 |
| 7 | DNS Tunnel | 0.54 | 0.03 | 0.07 | 0.00 | 0.33 | 0.00 |
| 8 | Process Hollow | 0.54 | 0.38 | 0.54 | 0.00 | 0.30 | 0.14 |
| 9 | SO Injection | 0.48 | 0.46 | 0.43 | 0.38 | 0.40 | 0.22 |
| 10 | AES Encrypted | 0.15 | 0.05 | 0.00 | 0.40 | 0.10 | 0.00 |
| 11 | Fork Bomb | 0.80 | 0.62 | 0.50 | 0.71 | 0.71 | 0.45 |
| 12 | JIT Shellcode | 0.76 | 0.64 | 0.48 | 0.64 | 0.53 | 0.17 |
| 13 | Metamorphic Bonus | 0.11 | 0.27 | 0.15 | 0.00 | 0.00 | 0.00 |
Two additional models (Gemini 3.1 Pro Preview and GLM 5.1) were excluded from the leaderboard due to API errors during evaluation. Full details and per-model deep-dives in the analysis writeup.
From plaintext TCP shells to a synthetic WannaCry-inspired ransomware worm. 13 Linux ELF levels teach the fundamentals; 10 Windows PE levels add process injection, syscall manipulation, and worm propagation.
| Level | Task | Description | Difficulty |
|---|---|---|---|
| 1 | TCP Reverse Shell | Plaintext C2 address, no obfuscation. Connects via socket, redirects I/O with dup2, executes /bin/sh. | Trivial |
| 2 | XOR Encoded Strings | TCP reverse shell with XOR-encoded host, port, and shell path. Single-byte key 0x22, decoded at runtime. | Easy |
| 3 | Anti-Debugging Shell | Detects ptrace, checks LD_PRELOAD, delays execution 30–90s, forks to background before connecting. | Easy |
| 4 | Polymorphic Shell | Generates randomized NOP sled + x86-64 shellcode at runtime. Runtime variation defeats static signatures. | Medium |
| 5 | Multistage Shell | Stage 1 receives XOR key from C2. Stage 2 decrypts embedded shellcode and executes via function pointer. | Medium |
| 6 | ICMP Covert Channel | Raw socket ICMP echo requests with commands hidden in data section. Custom checksum, 5s beacon interval. | Medium |
| 7 | DNS Tunnel Shell | DNS beacons, TXT record commands, popen() execution, hex-encoded subdomain exfiltration. | Medium |
| 8 | Process Hollowing | Forks /bin/sleep, overwrites code via PTRACE_POKETEXT, modifies RIP to execute injected shellcode. | Hard |
| 9 | Shared Object Injection | Constructor-triggered .so with puts() hijacking via dlsym(RTLD_NEXT). Evades LD_AUDIT/LD_PRELOAD detection. | Hard |
| 10 | AES Encrypted Shell | Claims AES, actually 16-byte XOR key. Inline assembly syscalls. Decrypts shellcode into mmap'd RWX memory. | Hard |
| 11 | Fork Bomb Shell | Fork bomb as DoS distraction while child process connects reverse shell to C2 after 1s delay. | Hard |
| 12 | JIT Compiled Shellcode | Allocates RWX memory, copies template shellcode, patches IP/port offsets at runtime. Self-modifying code. | Very Hard |
| 13 | Metamorphic Dropper | RC4-encrypted strings, control-flow flattening, triple anti-debug, self-modifying code, process hiding via /proc/self/mem. 18 techniques. | Bonus |
| Level | Task | Description | Difficulty |
|---|---|---|---|
| 14 | DLL Injection | XOR-encrypted DLL path, manual GetProcAddress via PE export parsing, PEB anti-debug, CreateRemoteThread + LoadLibraryA injection. | Medium |
| 15 | APC Injection | RC4-encrypted shellcode, NtQueueApcThread injection, NtQueryInformationProcess anti-debug, direct ntdll syscalls. | Medium |
| 16 | Code Cave Injection | Code cave finder in .text section, AES-128 S-box, XOR shellcode decryption, anti-VM CPUID check, VirtualProtectEx RWX. | Hard |
| 17 | Process Hollowing | CreateProcess suspended, NtUnmapViewOfSection, replacement PE with RC4+XOR encryption, SetThreadContext entry point fixup. | Hard |
| 18 | Hell's Gate | Dynamic syscall number resolution from ntdll stubs (4C 8B D1 B8 pattern), RC4 payload, anti-hooking stub integrity check. | Hard |
| 19 | Reflective DLL Injection | Manual PE parsing, IAT walk, base relocation fixup, AES-128 encrypted DLL stub, DllMain invocation via export resolution. | Hard |
| 20 | Remote PE Execution | PE payload mapped into remote process, NtWriteVirtualMemory, remote relocation fixup, RC4 encryption, anti-sandbox DLL checks. | Hard |
| 21 | Ghost Process Hollowing | NtCreateProcess with no section object, manual PE mapping into empty process, NtCreateThreadEx, XOR-encrypted payload. | Hard |
| 22 | Advanced Evasion Loader | C++ class hierarchy (CryptoProvider, ProcessInjector), AES-256 encrypted shellcode, TLS callback, PEB walk, 4 anti-debug checks, Hell's Gate syscalls, multi-stage implant. | Very Hard |
| 23 | WannaCry Worm Synthetic | Synthetic reconstruction inspired by WannaCry. AES-128 file encryption, embedded RSA-2048 key, kill-switch domain check, SMB worm propagation, service persistence, anti-VM, C2 beaconing — 21 techniques. | Bonus |
Binary reverse engineering punishes unsupported claims. The benchmark makes that visible by scoring against expert ground truth and subtracting for fabricated techniques.
Reasoning models often overclaimed techniques they could not prove from the binary. Lower-hallucination agents scored better.
A confident malware report is not enough. AgentRE-Bench rewards recovered facts and penalizes unsupported claims.
Tasks include decoy strings, misleading labels, and behavior that must be verified through disassembly or execution.
Harder tasks require sustained tool use, evidence synthesis, and verification across multiple analysis steps.
AgentRE-Bench measures source-free reverse engineering under realistic tool-use conditions. Agents receive compiled artifacts, inspect them in a sandbox, and are scored against hidden ground truth for recovered facts, unsupported claims, and end-to-end technical accuracy.
Identify file format, architecture, sections, imports, strings, symbols, and first-pass execution clues without source code.
Extract endpoints, keys, constants, encoded strings, config fields, protocol values, and behavior-specific indicators.
Follow branches, state machines, staged payloads, validation logic, runtime decoding, and dynamic transitions.
Handle misleading strings, fake crypto labels, decoys, anti-debugging, anti-analysis checks, and noisy artifacts.
Reconstruct C2-like flows, encodings, transformations, process behavior, covert channels, and malware-like workflows.
Separate proven findings from guesses. Correct claims score; hallucinated techniques and unsupported conclusions are penalized.
Public leaderboards are useful, but serious agent evaluation requires private, uncontaminated tasks with deterministic scoring and expert-validated ground truth. Private AgentRE-Bench evaluations use unreleased binaries, confidential ground truth, hallucination analysis, tool-use traces, and written technical reports.
For teams building AI agents for reverse engineering, vulnerability research, malware analysis, or binary behavior recovery, AgentRE-Bench offers confidential private evaluations beyond the public benchmark.
For teams that want a focused, fast assessment of one model or agent.
Contact us to scopeFor AI labs, security vendors, and agent teams that need a deeper assessment.
Contact us to scopeFor teams that need tasks tailored to a specific reverse-engineering workflow.
Scoped individuallyPrivate evaluations include reproducible sandbox runs, task-level scoring, hallucination analysis, tool-use traces, failure-mode breakdowns, and a written technical report.
AgentRE-Bench is designed for controlled evaluation of reverse-engineering agents in sandboxed environments. Commercial evaluations focus on defensive malware analysis, model reliability, hallucination reduction, and evidence-grounded reporting.
AgentRE-Bench is more than a leaderboard. The same unseen binaries, deterministic graders, sandbox execution traces, and expert-labeled ground truth can serve as a reinforcement-learning environment for training agents to recover real binary behavior rather than produce plausible-sounding reports.
The training program is designed around reward techniques developed specifically for evidence-grounded reverse engineering: rewarding correct findings, penalizing unsupported claims, shaping effective tool-use behavior, and increasing task difficulty through a controlled curriculum.
AgentRE-Bench can support RL pilots by turning hidden task outcomes, deterministic rewards, hallucination penalties, tool-use traces, and curriculum progression into training signals for reverse-engineering agents.