Systems Architecture & Security Tooling
Inside TraceForge: Building a local-first OSINT & DFIR toolkit.
I built TraceForge because my terminal kept turning into a mess of unsaved shell one-liners, open browser decoders, and unorganized evidence files. Here is an honest look at why I built it, how I split workloads between Python and Go, and the practical systems lessons I learned along the way.
- What it is: A local CLI toolkit that handles the boring, repetitive parts of incident triage — hashing evidence, defanging IOCs, dissecting packet dumps, comparing filesystem snapshots, and exporting STIX 2.1 / MISP bundles.
- Why build it: Existing desktop forensic suites are too heavy for quick CLI work, web decoders risk leaking confidential artifacts, and raw shell one-liners don't maintain cryptographic hash ledgers.
- The Stack: Python for the CLI interface, case state, and reporting; Go for multi-threaded SHA-256 hashing and log streaming; Bash for system health checks and platform detection.
- 100% Offline: Runs entirely on your machine with zero telemetry or cloud dependencies.
The Problem I Was Trying to Solve
Whenever I worked on digital forensics labs or analyzed sample logs from an incident, my desktop would end up covered in throwaway terminal tabs. I would have one tab running tcpdump, another running a half-written Python regex loop to grab IP addresses, and three open browser tabs with online decoders.
It worked, but it had three obvious flaws that kept bothering me:
I wanted a single, cohesive CLI tool that lived directly on my terminal, ran entirely offline, and handled these repetitive tasks with proper forensic hygiene.
Why Build This When Existing Tools Already Exist?
Whenever you build something in the security world, the obvious question is: "Why not just use Autopsy, Maltego, CyberChef, or existing bash commands?" Here is my honest thinking on the trade-offs:
The Learning Benefit: Beyond the practical utility, building TraceForge myself was the fastest way to truly understand the underlying protocols. Writing regex defangers, packet dissectors, hash ledgers, and STIX schemas from scratch forces you to understand every protocol byte and data structure at a fundamental level.
The Architecture: Python + Go + Bash
I didn't want to be dogmatic about using just one language. Each language in TraceForge has a specific job where it shines:
Python for High-Level Workflow Agility
Python is the glue. It handles the Click CLI commands, the SQLite database for case metadata, the Rich terminal formatting, and the export engines for Markdown, HTML, CSV, and STIX 2.1 bundles. It also makes distribution via PyPI very clean.
Go for Raw I/O & Parallel Hashing
Hashing 50,000 files in a filesystem baseline check in pure Python is noticeably slow because of interpreter overhead. I wrote dedicated Go native routines (traceforge-go-fastpaths). Using goroutines and memory-mapped buffers, the Go binary hashes directories in parallel at raw disk throughput.
Silent Fallback: If the compiled Go binary is not available on a particular machine, TraceForge automatically falls back to standard Python routines without throwing an unhandled exception or crashing the CLI.
What an Investigation Actually Looks Like
TraceForge guides the analyst through a clean, reproducible terminal workflow. Click the command tabs below to inspect real lifecycle traces:
$ traceforge tools ioc-extract ./evidence/auth_failures.log --defang --json
[+] Scanned 4,120 lines in 14ms (via Go native engine)
[+] Extracted 14 unique indicators:
[IP] 198.51.100.24 → defanged: 198.51.100[.]24
[IP] 203.0.113.19 → defanged: 203.0.113[.]19
[DOMAIN] malicious-c2.test → defanged: malicious-c2[.]test
[URL] http://bad.test/drop → defanged: hxxp://bad[.]test/drop
[HASH] e3b0c44298fc1c149afb… → indexed SHA-256Every case is stored in its own SQLite database and immutable evidence folder, so you can easily archive, compress, or back up an investigation whenever you need to.
The First-Party Analytical Tools
TraceForge includes purpose-built tools designed for fast, reliable terminal work:
http:// to hxxp:// and bracket-masking dots) to prevent accidental click-throughs.
traceforge tools diff quickly surfaces newly created files, permission changes, and persistence mechanisms.
The 7 Investigation Modules
To keep common investigative tasks organized, TraceForge structures them into 7 modular pipelines:
The 152-Tool Discovery Catalog
You shouldn't have to remember every single obscure CLI flag or search online to see if a utility is installed. TraceForge includes a built-in catalog of 152 external security utilities categorized across 10 disciplines (Network, OSINT, Forensics, Web, Reverse Engineering, etc.).
The tool queries your system package manager — whether you are on Debian/Ubuntu (apt), Arch Linux (pacman), Fedora (dnf), macOS (brew), or Android (pkg in Termux). It checks if a binary is on your $PATH and shows concise documentation and install commands without blindly running unexpected scripts.
Making It Work on Android / Termux
Getting a full forensic CLI to run reliably across macOS, Linux servers, and Android Termux meant dealing with a few real-world systems quirks:
- Dynamic Environment Prefixes: Android Termux places binaries and libraries under
/data/data/com.termux/files/usr/. Hardcoded paths like/tmpor/etc/fail immediately. TraceForge resolves paths dynamically usingos.environ.get('PREFIX')across all file handlers. - Cross-Compiled Architecture Binaries: The native Go subroutines are compiled for
linux/amd64,linux/arm64,darwin/amd64, anddarwin/arm64. TraceForge checks the host architecture at runtime to load the correct binary. - Terminal Rendering: Interactive TTY menus and box-drawing characters adapt automatically on minimal mobile terminal emulators to avoid corrupted escape sequences.
Ethics & Responsible Use
Security tooling carries dual-use responsibility. TraceForge was designed explicitly for authorized digital forensics, defensive incident triage, security research, and educational lab work. The documentation and CLI enforce local processing by default, ensuring tools operate strictly on systems, accounts, networks, and datasets where explicit authorization has been granted.
What I Learned Building It
Building TraceForge gave me direct, hands-on experience solving real software engineering problems:
- Defensive Parsing is Hard: Real-world logs contain broken lines, non-UTF-8 characters, and corrupted archive headers. Building parsers that never crash on bad input requires thorough edge-case handling.
- Schema Conformance Matters: Implementing STIX 2.1 and MISP JSON exports taught me how to make independent tools cleanly interoperable with enterprise SIEM systems.
- Documentation is Core Infrastructure: Writing clear documentation on ReadTheDocs with Sphinx helped me clarify API boundaries and maintain a project I am proud to share publicly.
TraceForge continues to evolve as I advance my cybersecurity studies — with upcoming work focused on expanding test coverage, optimizing memory during large entity graph clustering, and writing more forensic lab guides.