Home/ Documentation

Brain & Muscle
Engineering Spec

Official engineering documentation for the Brain & Muscle distributed task orchestration platform, targeting backend engineers and system administrators.

1. System Architecture

Brain & Muscle is a distributed task orchestration platform designed for high-concurrency workloads. The system enforces a strict decoupling between the Control Plane (the Brain) and the Data Plane (the Muscle fleet).

The Brain (Control Plane)

The Brain orchestrates state transitions, schedules jobs, and proxies external requests. It is a Go-based application that utilizes the Raft consensus algorithm to replicate state across a highly-available cluster. All system communication occurs purely via precise TCP protocols.

To guarantee fault tolerance, the Brain implements strict Write-Ahead Logging (WAL) alongside Raft. In the event of a catastrophic node failure, the Brain instantly reconstructs its exact state by replaying the WAL from the last committed index during the boot sequence. It handles GitHub App authentication, dynamically injects webhooks, queues tasks, and exposes a Next.js administrative dashboard.

The Muscle (Data Plane)

The Muscle is a Go-based worker node operating as a strictly stateless execution environment. Muscles connect to the Brain via persistent outbound TCP connections (typically port 9000). Upon assignment, a Muscle executes the assigned payload inside isolated Docker containers.

Note: The first time a Muscle executes a specific runtime task (such as Go, C++, JS, or Python), it must download the official base image from the registry, which entails a slight initial performance delay.Afterward, it mounts secure volumes, executes the code, and uses temporary SAS tokens to stream logs and results back to the designated cloud storage.

2. Operating the Muscle

Muscle nodes are designed to run persistently in the background. They support two primary paradigms modes of execution.

Network Prerequisites

The Muscle node operates strictly via outbound TCP on port 9000. Ensure that the host network (including Virtual Machine Network Security Groups) permits outbound traffic on this port. Note that standard ICMP (ping) traffic to the Brain is not required.

Foreground Mode

Foreground execution runs the binary directly in the active terminal session. This mode is exclusively used for local development, debugging, and temporary executions. Logs are streamed directly to stdout.

Linux / Ubuntu

bash

chmod +x muscle-linux
./muscle-linux --cpu 4.0 --ram 4096

macOS (Apple Silicon)

bash

chmod +x muscle-macos-arm64
./muscle-macos-arm64 --cpu 4.0 --ram 4096

Windows

powershell

.\bam-muscle.exe --cpu 4.0 --ram 4096

Daemon Mode

For production workloads, Muscle must execute as a background process completely isolated from active user sessions.

Linux (systemd)

Create a unit file /etc/systemd/system/bam-muscle.service, then execute systemctl enable --now bam-muscle.service.

ini

[Unit]
Description=Brain & Muscle Worker Node
After=network.target docker.service

[Service]
ExecStart=/usr/local/bin/muscle-linux --cpu 4.0 --ram 4096
Restart=always
User=muscleuser

[Install]
WantedBy=multi-user.target

macOS (launchd)

Create a .plist file in /Library/LaunchDaemons/com.bam.muscle.plist. Execute sudo launchctl load -w /Library/LaunchDaemons/com.bam.muscle.plist.

xml

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.bam.muscle</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/muscle-macos-arm64</string>
        <string>--cpu</string>
        <string>4.0</string>
    </array>
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>

Windows (Windows Services / Session 0)

It installs itself natively as a Windows Service running in Session 0, ensuring complete isolation from interactive user sessions.

powershell

# Install the binary as a background service (auto-starts instantly)
.\bam-muscle.exe --service install --cpu 4.0 --ram 4096

3. The Execution Lifecycle (GATHER & RUN)

The Data Plane employs a "Thin Host" strategy. Execution occurs in a locked-down Docker runtime (GCC, Go, Python, or Node.js) across a strict three-phase lifecycle:

Phase 1: GATHER

The Muscle validates the assigned task payload. It securely fetches Azure SAS tokens for artifact storage, clones the requisite code shards across branches, and executes any specified pre-flight operations dictated by gather_script_path.

Phase 2: RUN

The Muscle mounts the fetched code and datasets into the running container as isolated volumes. The core engine executes the designated user_script_path. Upon termination, the Muscle captures streams, synchronizes the final output directly to the cloud storage provider via the SAS token, and reports a terminal status back to the Brain.

Phase 3: TEARDOWN

To maintain strict statelessness and prevent host disk saturation, the Muscle forcefully destroys the isolated container and aggressively wipes all cloned repositories, temporary volumes, and local artifacts immediately after the SAS upload transaction completes.

4. Execution Paradigms

Map-Reduce Orchestration

Instead of complex DAGs (Directed Acyclic Graphs), Map-Reduce workloads are managed using a strict chunk index counter system. The Brain splits large processing tasks across the cluster, deploying independent MAP tasks linked by atomic slice indices. Once all chunk counters reach terminal success, the Brain natively executes the REDUCE phase logic to aggregate the distributed outputs securely.

Services (Reverse Proxying)

Long-running web applications (APIs, Dashboards) are operated as Services. The Brain binds the executing code directly inside a continuously running Docker container (your configured codespace) and manages a native reverse proxy tunnel. This securely routes external HTTP requests directly to the bound port explicitly exposed by the Muscle executing the container.

Distributed ML Training (Parameter Server)

For distributed machine learning training sequences, the platform utilizes a Parameter Server (PS) topology. Compute nodes pull neural network weights from the PS, calculate gradients locally against specific dataset shards, and push the gradients back. The Brain enforces a strict staleness bound (typically a staleness threshold of 2-3 batches) to balance rapid asynchronous mathematical computations against overall model convergence stability.

5. Security & Authentication (The GitHub App Model)

The Brain circumvents the limitations of standard OAuth proxies by operating as a fully native GitHub App.

The Brain generates a signed cryptographic JWT using its private key.
It exchanges the JWT with GitHub for an ephemeral Installation Access Token.
With this token, the Brain acts as an independent robot service account to securely drill into and clone private repositories.
It dynamically injects webhook endpoints into the source so that pushes automatically notify the Brain to dispatch pipeline updates to the surrounding Muscle fleet.

6. Execution Tutorial: Batch vs Distributed

Execute your workloads via the Next.js visual dashboard command center.

Batch Task (Single Node)

Navigate to the Deploy Batch page.
Select your isolated coding runtime (e.g., Python 3.11). Verify your logic accepts streams.
Configure your core allocation limit utilizing the sliders (e.g., 2.5 vCores).
Launch. The Brain routes this task to a single capable Muscle which natively provisions the environment.

Distributed Task (Multi-Node)

Navigate to the Distributed Workloads section.
Enter the number of necessary worker shards for your core processing objective.
Specify your Azure storage DATASET_PATH and target your application map scripts/mapper.py.
Deploy the pipeline. The Brain mathematically shards the target job and disperses concurrent segments tightly across multiple Muscle nodes.

7. Storage Maintenance (Cleanup)

To ensure high availability across long-term sustained cluster lifetimes, hosts operating long-living Muscle nodes must strictly undergo routine preventative maintenance sequences.

While Phase 3 TEARDOWN actively manages atomic task artifacts, residual images, log volumes, and abandoned processes will accumulate over months. It is imperative that system administrators schedule crontab / TaskScheduler sweeps enforcing system cleanups. You must periodically erase storage within:

Windows: C:\ProgramData\BrainAndMuscle\tmp
Linux & macOS: /tmp/BrainAndMuscle

Furthermore, ensure active system pruning is utilized via Cron (e.g. docker system prune -af --volumes) to eliminate isolated ghost container layers abandoned over months of heavy data-plane operations.

8. Task Manifests

The Brain synthesizes your user dashboard requests into atomic JSON payloads. Muscles consume these raw manifests over the TCP control channel to allocate resources strictly overriding and enforcing required_cpu constraints and bootstrapping the GATHER and RUN workflows.

json

{
  "job_id": "d256f75a-01a3-461b-8ab1-b2ad06b5e64c",
  "job_type": "MAP",
  "source_type": "github",
  "git_repo": "https://github.com/organization/data-pipeline.git",
  "branch_name": "main",
  "commit_hash": "a4d3f2c98e1b",
  "gather_script_path": "scripts/setup_environment.sh",
  "user_script_path": "src/process_shard.py",
  "run_command": "python3",
  "required_cpu": 4.0,
  "required_ram": 4096,
  "env": {
    "DATASET_PATH": "https://bucket.azure.com/data/shard_42.csv",
    "REPORT_DIR": "/app/reports/output"
  },
  "upload_url": "https://bamstorage.blob.core.windows.net/logs/d256f...?sig=..."
}

Need further assistance?

Explore the source code or report an anomaly directly to the terminal core.

Handshaking

Secure Cluster Sync