# Bulk API

Running the same workflow operation on hundreds or thousands of process instances one at a time is slow, error-prone, and hard to manage.
Every failure needs manual error handling, and sequential execution wastes time.

The Workflow Engine Bulk API runs runtime operations on multiple process instances in parallel through a single method call. Each operation
is isolated, so one failure never blocks the others.

With the Bulk API, you can create, query, command, or delete hundreds of process instances with one call. For example, approving a queue of
500 leave requests by executing the same command on each instance, with per-item success or failure reporting.

## What it is

Think of it as an express lane for batch operations. Instead of walking each item through the regular counter one at a time - calling a
single-instance method in a `foreach` loop with manual `Task.WhenAll` and per-item error handling - you hand the whole list at once. The
system processes them in parallel up to a configurable limit, tracks each result independently, and reports back which succeeded and which
failed.

**For developers:** The Bulk API is exposed as `runtime.Bulk` on `WorkflowRuntime`. It mirrors single-instance methods with collection-based
inputs and returns a `Dictionary<Guid, BulkTaskResult<TResult>>` where every input key maps to its own result or exception.

```csharp
using OptimaJet.Workflow.Core.Runtime;

// Create 100 process instances from the same scheme
var processIds = Enumerable.Range(1, 100)
    .Select(i => Guid.NewGuid())
    .ToList();

Dictionary<Guid, BulkTaskResult<Empty>> results =
    await runtime.Bulk.CreateInstanceAsync(processIds, "InvoiceApproval");
```

The Bulk API supports the following operations:

Table: Bulk API methods

| Method                                            | Description                                                                                                   |
|---------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| `CreateInstanceAsync`                             | Create process instances from a scheme by `CreateInstanceParams` or by list of process IDs with a scheme code |
| `GetProcessInstanceAndFillProcessParametersAsync` | Fetch multiple instances with their parameters                                                                |
| `GetProcessInstancesTreeAsync`                    | Fetch the process instance trees for multiple root processes                                                  |
| `GetAvailableCommandsAsync`                       | Get available commands for multiple instances, with identity filtering and culture support                    |
| `ExecuteCommandAsync`                             | Execute a command on multiple instances                                                                       |
| `ExecuteCommandWithRestrictionCheckAsync`         | Execute commands with permission verification                                                                 |
| `UpdateSchemeIfObsoleteAsync`                     | Update scheme on multiple instances if a newer version exists                                                 |
| `DeleteInstanceAsync`                             | Delete multiple instances                                                                                     |
| `IsProcessExistsAsync`                            | Check existence of multiple instances                                                                         |

### Per-instance error isolation

When `throwOnError` is `false` (the default), every operation runs independently. The returned dictionary contains a `BulkTaskResult` for
each input identifier, with `State` set to `Completed` or `Failed` and the `Exception` property populated on failures:

```csharp
var results = await runtime.Bulk.CreateInstanceAsync(processIds, "InvoiceApproval");

foreach (var (processId, result) in results)
{
    if (result.State == BulkTaskState.Failed)
    {
        Console.WriteLine($"Process {processId} failed: {result.Exception?.Message}");
    }
}
```

Set `throwOnError` to `true` to throw an `AggregateException` containing all errors when any operation fails. You can also configure
defaults through `runtime.Bulk.DefaultThrowOnError`.

### Concurrency control

The `maxDegreeOfParallelism` parameter controls how many operations run concurrently. The default is set by
`runtime.Bulk.DefaultMaxDegreeOfParallelism`, which defaults to `Environment.ProcessorCount`. Lower it to reduce database connection
pressure in high-contention environments:

```csharp
var results = await runtime.Bulk.CreateInstanceAsync(
    processIds,
    "InvoiceApproval",
    maxDegreeOfParallelism: 4
);
```

### Per-task progress tracking

Pass a `taskFinalizedHandler` to receive a callback as each individual operation completes:

```csharp
var results = await runtime.Bulk.CreateInstanceAsync(
    processIds,
    "InvoiceApproval",
    taskFinalizedHandler: (sender, args) =>
    {
        Console.WriteLine(
            $"Process {args.LastFinalizedTaskId}: {args.LastFinalizedTaskResult.State}"
        );
    }
);
```

## Why it matters

The Bulk API delivers these outcomes:

- **Faster batch processing** - Operations run in parallel up to the configured concurrency limit, not sequentially. Creating 500 process
  instances with one call is significantly faster than a `foreach` loop with 500 individual calls.

- **Resilient to partial failures** - One bad process ID does not derail the entire batch. Each item succeeds or fails independently. Failed
  items are reported with their exception details so you can retry or audit them later.

- **Less integration code** - No manual `Parallel.ForEach` with error handling, no `Task.WhenAll` with result aggregation, no try-catch
  blocks around every single call. The Bulk API handles parallelism, error isolation, and result mapping.

- **Progress visibility** - The optional `taskFinalizedHandler` callback fires after each item completes, enabling progress bars, logging,
  or early warning of failure patterns.

## Who it is for

**Developer:** The Bulk API replaces complex parallel execution patterns with a single method call and handles error isolation
automatically. Use `runtime.Bulk` on `WorkflowRuntime` with collection-based inputs.

**Enterprise architect:** The concurrency limit prevents resource exhaustion. Each operation is isolated, so a corrupted process instance
never blocks other operations.

## When to use it

Use the Bulk API when you need to perform the same workflow operation on many process instances at once:

- **Bulk data migration or import** - Create hundreds or thousands of process instances from the same scheme using `CreateInstanceAsync`
  with a list of process IDs or `CreateInstanceParams` objects.

- **Mass approval or escalation** - Collect available commands from multiple instances, then execute the same command (approve, reject,
  escalate) across all of them.

- **Batch cleanup** - Delete completed, abandoned, or expired process instances by calling `DeleteInstanceAsync` with a list of IDs, instead
  of looping with individual delete calls.

- **Batch scheme update** - Call `UpdateSchemeIfObsoleteAsync` to bring a set of in-flight instances up to the latest scheme version, useful
  after deploying a process change.

## How it compares

The table below shows the tradeoffs between the Workflow Engine approach and the alternative.

Table: Comparison of bulk processing approaches

| Approach                        | What it requires                                                                                          | Result                                                                                                                |
|---------------------------------|-----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| Manual loop with `Task.WhenAll` | Write parallel execution code, manage task limits, handle per-item exceptions manually, aggregate results | Working solution, but more code to write, test, and maintain - each developer may implement it differently            |
| Workflow Engine Bulk API        | Pass a collection of identifiers and optional concurrency limit                                           | Built-in parallelism, error isolation, progress tracking, and result mapping - consistent across every bulk operation |

The Bulk API standardizes a pattern you would otherwise write from scratch every time, with proper error isolation and concurrency control
already built in.

## Frequently asked questions

**What operations does the Bulk API support?**

The Bulk API supports create instance, get process instance with parameters, get process instances tree, get available commands, execute
command, execute command with restriction check, update scheme if obsolete, delete instance, and process existence check.

**Does the Bulk API require a NEO license?**

No. The Bulk API on `WorkflowRuntime` (`runtime.Bulk`) is available in all editions. The HTTP endpoints that wrap bulk operations are part
of the RPC API and subject to its licensing.

**How does the Bulk API handle partial failures?**

By default, every operation runs independently. The returned dictionary contains a `BulkTaskResult` for each input - some complete
successfully, others report a failure. One failed item does not cancel the rest. Set `throwOnError` to `true` if you prefer an exception
that collects all failures after all tasks complete.

**What is the default level of parallelism?**

The default is set by `runtime.Bulk.DefaultMaxDegreeOfParallelism`, which defaults to `Environment.ProcessorCount`. Each operation runs on
its own task, and the Bulk API limits concurrent execution to this value to avoid overwhelming the database.

**Do I need to change my application code to handle bulk results differently?**

The Bulk API returns the same data types as single-instance methods, organized per input identifier. You iterate over the results dictionary
to check success or failure for each item. A single method call replaces what would otherwise be a manual loop with parallel execution and
error handling code.

## See also

- [Clustering](https://docs.workflowengine.io/evaluate/workflow-engine-features/clustering/): Horizontal scale-out across multiple servers for high-throughput scenarios.
- [Persistence Providers](https://docs.workflowengine.io/evaluate/workflow-engine-features/multi-database-support/): Which databases Workflow Engine supports and how to configure them.