Skip to main content

Debugging Encrypted Files in Closed-Source Tools — A Practical Unix Guide

This guide documents the step-by-step methodology and Unix/Linux tools used to reverse-engineer and decrypt encrypted data embedded in a closed-source EDA tool (innostack 26.10-b006). The techniques apply broadly to any situation where a commercial or proprietary tool stores encrypted content in files and you need to understand or recover that content.

Investigation Overview


Table of Contents

  1. Initial Reconnaissance
  2. Binary Identification
  3. Shared Library Analysis
  4. Symbol Extraction
  5. String Mining
  6. Dynamic Tracing — strace / ltrace
  7. Dynamic Tracing — LD_PRELOAD Interposition
  8. Disassembly — objdump / readelf
  9. Runtime Memory Inspection — GDB
  10. Entropy Analysis — binwalk / ent
  11. JavaScript / Bundled Script Extraction
  12. Dynamic Library Loading — dlopen / dlsym Pattern
  13. Key Reconstruction and Verification
  14. Writing the Decryptor
  15. Anti-Tamper Mitigations and Workarounds
  16. Quick Reference Cheat Sheet

1. Initial Reconnaissance

Before touching any binary tool, map out what you are working with.

Tools

Steps

What to look for

  • Magic numbers: 7f 45 4c 46 = ELF; 42 5a 68 = bzip2; 50 4b 03 04 = ZIP/JAR
  • Base64 padding (==) in text files → binary data stored as ASCII
  • XML/HTML-like tags wrapping binary content → custom container format
  • Suspiciously large executable → bundled runtime (Bun SEA, PyInstaller, etc.)

2. Binary Identification

Tools

Key Commands

Bundled Runtime Fingerprints


3. Shared Library Analysis

Closed-source tools often do crypto work inside a purpose-built .so to make reverse engineering harder. Identify which library owns the interesting operations.

Tools

Steps


4. Symbol Extraction

When a library exports only a dispatcher function, you need to understand what operations it supports.

Tools

Strategy


5. String Mining

Strings embedded in binaries and scripts often reveal op codes, algorithm names, error messages, and hardcoded keys.

Tools

Useful Patterns


6. Dynamic Tracing — strace / ltrace

Watch system calls and library calls at runtime to see what files are opened, what syscalls happen during decryption, etc.

Tools

Commands

What to look for

  • openat("...", O_RDONLY) on .md files → confirms which files are read
  • openat on .lic files → license check before crypto
  • mmap(NULL, large_size, PROT_READ|PROT_EXEC) → JIT compilation area

strace / ltrace Decision Flow


7. Dynamic Tracing — LD_PRELOAD Interposition

Override specific library functions to capture their arguments before/after the real call.

Tools

Example: Intercept HMAC

Limitations

  • Anti-tamper (Bun SEA, some Go binaries): check for LD_PRELOAD in environment and send SIGILL → exit code 132
  • RTLD_LOCAL dlopen: privately loaded libraries bypass LD_PRELOAD hooks
  • Direct syscall: some binaries bypass libc entirely for sensitive operations
  • Statically linked: LD_PRELOAD has no effect on statically linked binaries

Detecting if LD_PRELOAD is blocked

LD_PRELOAD Decision Tree


8. Disassembly — objdump / readelf

When ltrace/LD_PRELOAD fail, read the assembly directly.

Tools

Key Techniques

Reading .data for hardcoded secrets


9. Runtime Memory Inspection — GDB

For cases where static analysis isn’t enough, attach GDB to a running process.

Tools

Key Commands

Finding lib_base in C code


10. Entropy Analysis — binwalk / ent

High entropy regions of a binary are likely compressed or encrypted data.

Tools

Commands


11. JavaScript / Bundled Script Extraction

Modern CLI tools often ship as Node.js/Bun/Electron bundles with obfuscated JavaScript.

Tools

Bun SEA JavaScript Extraction

Reading Obfuscated Bundle Constants


12. Dynamic Library Loading — dlopen / dlsym Pattern

When you want to call a closed-source library function from your own C program without linking against its headers.

Why use dlopen/dlsym instead of direct linking

  1. No headers needed — you define the function pointer type yourself
  2. RTLD_LOCAL flag — prevents symbol conflicts when the library and the system have the same symbols (e.g., two OpenSSL versions)
  3. Runtime control — can set environment, load order, and state before the first call
  4. Isolation — the library’s internal OpenSSL never clashes with -lcrypto you linked against

Pattern

Compile


13. Key Reconstruction and Verification

Once you have the algorithm, verify the key before attempting full decryption.

Verification Strategies

  1. Known-plaintext test: encrypt a known string with the derived key; compare ciphertext with known-good output
  2. Op code probe: if the tool exposes a key-verify op code (like op=122 in innostack), use it
  3. AES-GCM tag check: AES-GCM includes a 16-byte auth tag — decryption succeeds only if key is correct
  4. HMAC-SHA256 verification: if a MAC is appended to the ciphertext, verify before decrypting

Example: HMAC-SHA256 key derivation from embedded IV

Why HMAC(secret, IV) and not PBKDF2

  • PBKDF2 is for password-based key derivation (slow, salted, iterated)
  • HMAC(static_secret, IV) is for deterministic key derivation — given the same secret and IV, always produces the same key
  • This pattern is used when the encrypting side needs to reconstruct the key without storing it: just embed the IV in the ciphertext and re-derive the key at decrypt time
  • The embedded IV (random, unique per file) ensures different files get different AES keys even with the same HMAC secret

AES-256-GCM Key Derivation Flow (innostack pattern)


14. Writing the Decryptor

Once the algorithm is confirmed, write a minimal, reproducible decryptor.

Shell Script Wrapper Pattern

Tips

  • Always use RTLD_LOCAL when dlopen-ing a library that ships its own OpenSSL
  • Use dl_iterate_phdr to find the library’s load base — never hardcode it
  • Write to /tmp for compiled tools; add trap 'rm -f ...' EXIT to clean up
  • Test with xxd output | head -5 before claiming success — garbage output looks like success

15. Anti-Tamper Mitigations and Workarounds

Anti-Tamper Bypass Strategy

Common Anti-Tamper Techniques

Detecting Bun SEA Anti-Tamper


16. Quick Reference Cheat Sheet

Identifying what a .so exports

Finding hardcoded keys in .data section

Finding load base of a dlopen’d library at runtime

Extracting base64-encoded binary from tagged text file

Testing entropy of a file

Compile a dlopen-based tool

Full pipeline: identify → trace → disassemble → reconstruct → decrypt

Tool Selection Map


Real-World Case Study: innostack 26.10-b006

The Problem

.md files in /path/to/installation/innostack/26.10-b006/.innostack/ contained:

Discovery Timeline

What each tool revealed

Key formula