Debugging Encrypted Files in Closed-Source Tools — A Practical Unix Guide
This guide documents the step-by-step methodology and Unix/Linux tools used to reverse-engineer and decrypt encrypted data embedded in a closed-source EDA tool (innostack 26.10-b006). The techniques apply broadly to any situation where a commercial or proprietary tool stores encrypted content in files and you need to understand or recover that content.Investigation Overview
Table of Contents
- Initial Reconnaissance
- Binary Identification
- Shared Library Analysis
- Symbol Extraction
- String Mining
- Dynamic Tracing — strace / ltrace
- Dynamic Tracing — LD_PRELOAD Interposition
- Disassembly — objdump / readelf
- Runtime Memory Inspection — GDB
- Entropy Analysis — binwalk / ent
- JavaScript / Bundled Script Extraction
- Dynamic Library Loading — dlopen / dlsym Pattern
- Key Reconstruction and Verification
- Writing the Decryptor
- Anti-Tamper Mitigations and Workarounds
- Quick Reference Cheat Sheet
1. Initial Reconnaissance
Before touching any binary tool, map out what you are working with.Tools
Steps
What to look for
- Magic numbers:
7f 45 4c 46= ELF;42 5a 68= bzip2;50 4b 03 04= ZIP/JAR - Base64 padding (
==) in text files → binary data stored as ASCII - XML/HTML-like tags wrapping binary content → custom container format
- Suspiciously large executable → bundled runtime (Bun SEA, PyInstaller, etc.)
2. Binary Identification
Tools
Key Commands
Bundled Runtime Fingerprints
3. Shared Library Analysis
Closed-source tools often do crypto work inside a purpose-built.so to make reverse engineering harder. Identify which library owns the interesting operations.
Tools
Steps
4. Symbol Extraction
When a library exports only a dispatcher function, you need to understand what operations it supports.Tools
Strategy
5. String Mining
Strings embedded in binaries and scripts often reveal op codes, algorithm names, error messages, and hardcoded keys.Tools
Useful Patterns
6. Dynamic Tracing — strace / ltrace
Watch system calls and library calls at runtime to see what files are opened, what syscalls happen during decryption, etc.Tools
Commands
What to look for
openat("...", O_RDONLY)on.mdfiles → confirms which files are readopenaton.licfiles → license check before cryptommap(NULL, large_size, PROT_READ|PROT_EXEC)→ JIT compilation area
strace / ltrace Decision Flow
7. Dynamic Tracing — LD_PRELOAD Interposition
Override specific library functions to capture their arguments before/after the real call.Tools
Example: Intercept HMAC
Limitations
- Anti-tamper (Bun SEA, some Go binaries): check for
LD_PRELOADin environment and sendSIGILL→ exit code 132 RTLD_LOCALdlopen: privately loaded libraries bypass LD_PRELOAD hooks- Direct syscall: some binaries bypass libc entirely for sensitive operations
- Statically linked: LD_PRELOAD has no effect on statically linked binaries
Detecting if LD_PRELOAD is blocked
LD_PRELOAD Decision Tree
8. Disassembly — objdump / readelf
When ltrace/LD_PRELOAD fail, read the assembly directly.Tools
Key Techniques
Reading .data for hardcoded secrets
9. Runtime Memory Inspection — GDB
For cases where static analysis isn’t enough, attach GDB to a running process.Tools
Key Commands
Finding lib_base in C code
10. Entropy Analysis — binwalk / ent
High entropy regions of a binary are likely compressed or encrypted data.Tools
Commands
11. JavaScript / Bundled Script Extraction
Modern CLI tools often ship as Node.js/Bun/Electron bundles with obfuscated JavaScript.Tools
Bun SEA JavaScript Extraction
Reading Obfuscated Bundle Constants
12. Dynamic Library Loading — dlopen / dlsym Pattern
When you want to call a closed-source library function from your own C program without linking against its headers.Why use dlopen/dlsym instead of direct linking
- No headers needed — you define the function pointer type yourself
RTLD_LOCALflag — prevents symbol conflicts when the library and the system have the same symbols (e.g., two OpenSSL versions)- Runtime control — can set environment, load order, and state before the first call
- Isolation — the library’s internal OpenSSL never clashes with
-lcryptoyou linked against
Pattern
Compile
13. Key Reconstruction and Verification
Once you have the algorithm, verify the key before attempting full decryption.Verification Strategies
- Known-plaintext test: encrypt a known string with the derived key; compare ciphertext with known-good output
- Op code probe: if the tool exposes a key-verify op code (like op=122 in innostack), use it
- AES-GCM tag check: AES-GCM includes a 16-byte auth tag — decryption succeeds only if key is correct
- HMAC-SHA256 verification: if a MAC is appended to the ciphertext, verify before decrypting
Example: HMAC-SHA256 key derivation from embedded IV
Why HMAC(secret, IV) and not PBKDF2
PBKDF2is for password-based key derivation (slow, salted, iterated)HMAC(static_secret, IV)is for deterministic key derivation — given the same secret and IV, always produces the same key- This pattern is used when the encrypting side needs to reconstruct the key without storing it: just embed the IV in the ciphertext and re-derive the key at decrypt time
- The embedded IV (random, unique per file) ensures different files get different AES keys even with the same HMAC secret
AES-256-GCM Key Derivation Flow (innostack pattern)
14. Writing the Decryptor
Once the algorithm is confirmed, write a minimal, reproducible decryptor.Shell Script Wrapper Pattern
Tips
- Always use
RTLD_LOCALwhen dlopen-ing a library that ships its own OpenSSL - Use
dl_iterate_phdrto find the library’s load base — never hardcode it - Write to
/tmpfor compiled tools; addtrap 'rm -f ...' EXITto clean up - Test with
xxd output | head -5before claiming success — garbage output looks like success
15. Anti-Tamper Mitigations and Workarounds
Anti-Tamper Bypass Strategy
Common Anti-Tamper Techniques
Detecting Bun SEA Anti-Tamper
16. Quick Reference Cheat Sheet
Identifying what a .so exports
Finding hardcoded keys in .data section
Finding load base of a dlopen’d library at runtime
Extracting base64-encoded binary from tagged text file
Testing entropy of a file
Compile a dlopen-based tool
Full pipeline: identify → trace → disassemble → reconstruct → decrypt
Tool Selection Map
Real-World Case Study: innostack 26.10-b006
The Problem
.md files in /path/to/installation/innostack/26.10-b006/.innostack/ contained: