0047: Dynamic Workflow Execution via Interpreter

Date: 2025-11-08

Status: Proposed

Context

ADR-0039 and ADR-0040 established a "code-first" approach for workflows using the Temporal engine. While this is excellent for reliability and testability, it presents a significant limitation: every change to a workflow's logic, no matter how small, requires developer intervention and a new service deployment.

This model is too rigid for a platform that aims to be extensible. It prevents business analysts from modifying processes and makes it difficult for third-party developers to introduce their own orchestration logic without forking core Citadel services. We need an abstraction layer that separates the definition of a workflow from its execution.

Decision

We will implement a Workflow Interpreter Pattern on top of the Temporal engine. This involves three core components:

1. A Workflow DSL (The "Port"): We will define a standard, engine-agnostic DSL (Domain-Specific Language) in YAML or JSON format. This DSL will describe the sequence of steps in a business process. It is the abstract "Port" in our Ports and Adapters architecture, defining what should happen.

   name: "Standard Tenant Onboarding"
   steps:
     - name: "Create IAM User"
       activity: "citadel/CreateUserInIAM" # Namespaced activity
       params:
         email: "{{.input.userEmail}}"
   
     - name: "Send Data to Custom CRM"
       activity: "acme-corp/SendToCRM" # Tenant-specific activity
       params:
         # ...

2. A Generic Interpreter Workflow (The "Adapter"): We will develop a single, generic, long-running Temporal workflow named WorkflowInterpreter. Its sole responsibility is to parse a DSL document and execute the defined steps in order. It translates our abstract DSL into concrete ExecuteActivity calls for the Temporal engine. This interpreter is the "Adapter" that makes the underlying engine swappable.

3. A Library of Namespaced Activities (The "Building Blocks"): Developers will create a library of reusable, versioned, and namespaced Activities.

   • Namespacing: Activity names will be namespaced (e.g., namespace/activityName) to prevent collisions and allow for overrides.
     • citadel/ is the reserved namespace for the standard library provided by the core platform.
     • Tenant-specific namespaces (e.g., acme-corp/) will be used for custom activities.
   • Isolation via Task Queues: The WorkflowInterpreter will use the namespace to dispatch the activity to a specific Temporal Task Queue. Citadel's standard activities will run on a citadel-standard-activities queue, while a tenant's custom activities will run on their own isolated worker and queue (e.g., acme-corp-custom-activities).

This model allows non-developers to assemble workflows from pre-approved Activities and enables third parties to securely register and execute their own custom Activities on their own infrastructure.

Consequences

Positive

• Flexibility for Non-Developers: Business analysts can modify or create workflows by editing the DSL document (e.g., via a UI), without requiring developer intervention or a new deployment.
• True Platform Extensibility: Third-party developers can securely add their own custom activities and workflows by deploying their own workers that connect to the central Temporal cluster. They can also provide their own implementation of a standard activity concept (e.g., their own SendToCRM).
• Engine Swappability: The DSL acts as an abstraction layer. If we decide to move from Temporal to another engine like Camunda, we only need to rewrite the WorkflowInterpreter adapter; the DSL documents and the core business logic within the activities remain unchanged.
• Clear Separation of Concerns:
  • Citadel Platform Team: Manages the Temporal cluster and the WorkflowInterpreter.
  • Developers (Core or Third-Party): Build, test, and maintain a library of secure, robust, and reusable Activity "building blocks."
  • Business Users/Analysts: Assemble these blocks into workflows using the DSL.

Negative

• Increased Upfront Complexity: Building the generic WorkflowInterpreter is significantly more complex than building specific, hardcoded workflows. It must handle dynamic activity dispatching, parameter templating, state management between steps, and robust error handling.
• Loss of Static Analysis: The orchestration logic is now dynamic (i.e., data). A typo in an activity name within a DSL document will only be caught at runtime, not at compile time. This necessitates strong validation in any UI or tool that generates the DSL.
• More Complex Debugging: A failed workflow now requires inspecting both the state of the WorkflowInterpreter and the specific DSL document it was executing to understand the full context of the failure.
• Operational Overhead: Requires management and isolation of namespaces and Temporal Task Queues to ensure security and prevent cross-tenant interference.