Skip to content
dotfiles

Design Rationale

Design Rationale

Section titled “Design Rationale”

This document explains the key technology choices behind the dotfiles management system — why Go was chosen for the CLI, why shell scripts handle the actual work, and why the system builds from source rather than distributing a pre-built binary.

Why a Hybrid Go + Shell Architecture?

Section titled “Why a Hybrid Go + Shell Architecture?”

Dotfiles managers exist on a spectrum. At one end, tools like GNU Stow are pure symlink managers. At the other, frameworks like Ansible bring full configuration management. This system sits in between: it needs real orchestration (dependency graphs, state tracking, rollback) but the actual work — installing packages, configuring tools — is fundamentally shell scripting.

The hybrid approach plays to each language’s strengths:

Go handles orchestration — the parts that are hard to get right in shell:

CapabilityWhy Go?
Dependency resolutionKahn’s algorithm with cycle detection, deterministic priority ordering, transitive expansion — ~150 lines of type-safe code vs. fragile array manipulation in bash
State trackingJSON state files with per-file SHA256 checksums for idempotence. Structured read/write with proper types vs. requiring jq and brittle parsing
Configurationgopkg.in/yaml.v3 parses config.yml and module.yml into typed structs with compile-time validation. The shell equivalent requires yq or python -c with no type guarantees
RollbackStructured operation log with LIFO reversal, distinguishing reversible operations (file deploys) from informational ones (script runs). Difficult to track reliably in shell
Template renderingFull Go text/template with custom functions (env, default, upper, lower, join). The shell alternatives — envsubst (limited) or sed/awk (fragile with special characters) — don’t scale
Interactive UIThread-safe spinners, multi-select prompts, TTY detection with graceful fallback. Bash spinners conflict with subprocess output and read -p is limited
Secrets abstractionInterface-based provider system (1Password today, extensible to Vault/AWS). Clean abstraction isn’t natural in shell
TestingStandard go test with race detector, integration tests in Docker. Shell testing frameworks exist but are limited

Shell handles system operations — what it does best:

  • Running package managers (apt install, brew install, pacman -S)
  • Git configuration, SSH key management (generate / agent / 1Password / none, selected by modules.ssh.key_source), tool setup
  • OS-specific logic that varies by platform
  • Anything that benefits from being easily readable and modifiable without recompilation

The Alternative: What Pure Shell Would Require

Section titled “The Alternative: What Pure Shell Would Require”

A shell-only version of the orchestration layer would need:

  • External dependencies: jq for JSON state files, yq for YAML config parsing, possibly tsort for dependency ordering — all of which may not be present on a fresh system
  • 500+ lines of state management replacing the state package, with no compile-time guarantees
  • Platform inconsistencies: macOS still ships Bash 3.2 (2007), which lacks associative arrays, mapfile, and other features used in modern bash. The system would need to either target Bash 3.2 or require Bash 4+ as a dependency
  • No structured rollback: tracking operations for undo in shell requires writing to temp files and parsing them back, with no type safety on the metadata

The Go binary is ~2,000 lines across well-separated packages. An equivalent shell implementation would likely be larger, harder to test, and more fragile.

Why Not Python, Ruby, or Another Scripting Language?

Section titled “Why Not Python, Ruby, or Another Scripting Language?”

Go was chosen over other scripting languages for several practical reasons:

  • Single binary, zero runtime: go build produces one static binary with no runtime dependencies. Python needs a Python installation (and often a virtualenv to avoid version conflicts). Ruby needs a Ruby installation. Both add complexity to bootstrapping a fresh system
  • Cross-compilation: Go trivially cross-compiles for any OS/architecture combination, which matters for a tool targeting macOS (Intel and Apple Silicon), Ubuntu, and Arch
  • Startup time: The Go binary starts in milliseconds. Python and Ruby have noticeable interpreter startup overhead, which adds up when the binary is called back from shell scripts (get-secret, render-template)
  • Dependency management: go.mod locks dependencies with checksums. No pip install issues, no gem version conflicts, no virtualenv to manage on a machine that’s still being set up

Why Build from Source?

Section titled “Why Build from Source?”

The bootstrap script (bootstrap.sh) downloads Go, clones the repo, and runs go build. This is deliberate, not a shortcoming.

The Binary Needs the Repository

Section titled “The Binary Needs the Repository”

The Go binary doesn’t embed any assets. At runtime, it reads from the filesystem:

  • config.yml and profiles/*.yml for configuration
  • modules/*/module.yml for module definitions
  • modules/*/install.sh, verify.sh, and os/*.sh for execution
  • lib/helpers.sh, sourced into every shell script

A pre-built binary without the cloned repository can’t do anything. Since git clone is always the first step, the binary alone doesn’t save a meaningful step.

The Cost of Pre-Built Distribution

Section titled “The Cost of Pre-Built Distribution”

Distributing pre-built binaries would require:

  • Release infrastructure: goreleaser configuration, GitHub Actions release workflow, artifact signing
  • Platform matrix: at minimum linux/amd64, linux/arm64, darwin/amd64, darwin/arm64 — four binaries per release
  • Version synchronization: the binary version must match the repository version. A module schema change or new environment variable requires both a new binary and updated repo content. Building from source eliminates this entire class of problems
  • Download logic: the bootstrap script would need platform detection, binary download, checksum verification — essentially the same complexity as downloading Go, but repeated for every release

What Building from Source Provides

Section titled “What Building from Source Provides”
  • Guaranteed consistency: the binary always matches the repository it was built from
  • Zero release overhead: no CI release pipeline, no artifact hosting, no version tags to manage
  • Hackability: modify Go code, run go build, done. No waiting for a release cycle
  • Simplicity: one build step (go build -o bin/dotfiles .) with no configuration

Go Is a Lightweight Bootstrap Dependency

Section titled “Go Is a Lightweight Bootstrap Dependency”

Go installs as a self-contained tarball (~70MB), extracts to a single directory, requires no system libraries, and works identically on all supported platforms. The bootstrap script handles this automatically. Compare this to bootstrapping Python (version management, pip, virtualenv) or Node.js (nvm, npm ecosystem).

The build itself takes 3-5 seconds on modern hardware. For a tool that runs once during system setup, this is negligible.

Why a Generic Engine + Content Overlay?

Section titled “Why a Generic Engine + Content Overlay?”

The repository is a generic engine: it ships conservative, personal-free defaults (secrets.provider: noop, empty user.*, no baked-in vault names) and works out of the box for anyone. Personal identity, secrets choices, and custom profiles/modules live in an optional content overlay directory ($DOTFILES_CONTENT_DIR) — laid out like the repo and deep-merged over it — which the user keeps in their own repo.

This separation is deliberate:

  • Forkability without divergence. You point the engine at your own content repo instead of forking and editing tracked files, so you can pull engine updates without merge conflicts against your personal settings.
  • No secrets in the engine. Identity and secret references never need to be committed to the shared repo; they stay in your overlay (and secrets themselves stay in a provider).
  • Opt-in, zero-cost when unused. With no content dir set, every layer collapses to the engine alone and behavior is identical — the overlay is purely additive.

See Content Overlay for the full model.

Design Tradeoffs

Section titled “Design Tradeoffs”

Every architecture involves tradeoffs. Here are the ones this system makes consciously:

DecisionBenefitCost
Go for orchestrationType safety, testability, single binaryRequires Go toolchain at bootstrap
Shell for executionReadable, modifiable, no recompilation neededPlatform inconsistencies (Bash versions, coreutils differences)
Build from sourceAlways in sync, zero release infra~70MB Go download + ~5s build time on first run
Runtime file loadingModules can be added/modified without rebuildingBinary is useless without the repository
YAML configurationHuman-readable, well-supportedRequires a parsing library (Go) or external tool (shell)
Generic engine + content overlayFork-free personalization, no secrets in the shared repoA second (content) repo to manage for a fully custom setup

These tradeoffs optimize for the primary use case: a developer setting up a new machine, where reliability and correctness matter more than shaving seconds off the initial bootstrap.

References

Section titled “References”