Orkes Conductor design guidance needed – processing multiple individuals within an existing onboarding workflow
We have an existing Orkes Conductor workflow for corporate/customer fulfillment. Currently, the workflow is designed to process one individual associated with an application.
The current flow roughly works as follows:
applicationId → fetch fulfillment data → check whether individual customer exists → create individual customer if required → Kafka/downstream processing → WAIT_FOR_WEBHOOK → obtain individual customer ID/RID → continue downstream fulfillment processing
The workflow already contains SWITCH, FORK_JOIN, JOIN, INLINE, SIMPLE, and WAIT_FOR_WEBHOOK tasks. There are also corporate customer and CASA-related stages later in the same workflow.
New requirement: One corporate application can now contain multiple individuals, for example one owner plus multiple authorized signatories. The number is dynamic—there could be 1, 5, 10 or more individuals.
Each individual must be processed independently through the required downstream flow and must obtain their own individual customer ID/RID. Each individual may also have their own documents/EDMS references and downstream processing.
For example:
APP100 → P1 → RID101 → downstream processing
APP100 → P2 → RID102 → downstream processing
APP100 → P3 → RID103 → downstream processing
A key complication is that our existing WAIT_FOR_WEBHOOK currently correlates using approximately:
applicationId + callbackType
Since all individuals belong to the same applicationId, we need a safe way to correlate each asynchronous callback to the correct individual/workflow execution, probably using an individual/party/request/correlation ID.
We also need independent failure handling. For example, if P1, P2 and P4 succeed but P3 fails, we should be able to retry/recover P3 without recreating or reprocessing the already successful individuals.
Finally, performance is important. We don’t want an application containing many individuals to suddenly trigger unlimited parallel downstream calls and negatively affect Kafka consumers, workers, databases, FCUBS/core banking APIs, EDMS or other downstream systems.
What would be the recommended Orkes architecture for this? Should we use Dynamic Fork/Fork-Join, a DO_WHILE/iteration approach, a parent workflow dynamically invoking one sub-workflow per individual, or another Orkes pattern? Also, what is the recommended approach for limiting/constraining concurrency, aggregating individual results, webhook correlation, partial failures/retries and idempotency?
We would preferably like to reuse the existing proven workflow/tasks as much as possible rather than duplicate the entire workflow for each individual.