Function Re-run Feature Documentation¶

Overview¶

The function re-run feature enables users to re-execute compute functions using the exact state from a previous run. This is particularly useful for:

• Debugging: Reproduce issues with the exact same parameters
• Recovery: Retry failed executions without manual parameter reconstruction
• Testing: Validate fixes by re-running with original data
• Audit: Track lineage and history of function executions

Database Schema Changes¶

New Fields in function_runs Table¶

Re-run Lineage Tracking¶

• originalRunId (uuid, nullable): References the original run if this is a re-run
• isRerun (boolean): Flag indicating if this run was triggered by a re-run
• rerunCount (integer): Number of times this specific run has been re-run
• rerunBy (uuid, nullable): User who triggered the re-run
• rerunReason (text, nullable): User-provided reason for re-running

Function Code Snapshot¶

• functionCodeHash (text, nullable): SHA-256 hash of function code at execution time
• functionVersion (text, nullable): Version/snapshot identifier of function code

Execution Context¶

• executionContext (jsonb, nullable): Additional context (trigger params, environment, etc.)
• executionSource (text): Source of execution (trigger, rerun, manual, scheduled, webhook)

Fixed Metrics (Previously Incorrect Types)¶

• memoryUsageBytes (bigint, nullable): Memory usage in bytes (was incorrectly timestamp)
• cpuUsageSeconds (decimal, nullable): CPU usage in seconds (was incorrectly timestamp)

Migration¶

Run the migration to apply schema changes:

npm run migrate

Migration file: src/migrations/20251010135835_add_rerun_tracking_to_function_runs.ts

API Endpoints¶

1. Re-run a Function¶

Endpoint: POST /workspace/:ws/api/v1/function-runs/:id/rerun

Description: Re-executes a function using the exact state from a previous run.

Request Body:

{
  "useLatestCode": false,           // Optional: Use current function code (default: false)
  "reason": "Retrying after bug fix", // Optional: Reason for re-run
  "overrideParams": {               // Optional: Override specific parameters
    "customParam": "new value"
  }
}

Response:

{
  "jobId": "job-uuid-here",
  "message": "Re-run successfully queued. Original run run-uuid has been re-executed."
}

Example:

curl -X POST \
  'http://localhost:6070/workspace/my-workspace/api/v1/function-runs/abc123/rerun' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
    "reason": "Testing fix for edge case",
    "overrideParams": {
      "debugMode": true
    }
  }'

2. Get Re-run History¶

Endpoint: GET /workspace/:ws/api/v1/function-runs/:id/rerun-history

Description: Get all re-runs that originated from a specific run.

Response:

{
  "originalRunId": "run-uuid",
  "rerunCount": 3,
  "reruns": [
    {
      "id": "rerun-1-uuid",
      "originalRunId": "run-uuid",
      "isRerun": true,
      "rerunBy": "user-uuid",
      "rerunReason": "Testing fix",
      "status": "completed",
      "createdAt": "2025-10-10T14:30:00Z",
      ...
    },
    ...
  ]
}

How It Works¶

1. Initial Function Run¶

When a function executes: 1. Function code is hashed (SHA-256) and stored in functionCodeHash 2. Parameters are encrypted and stored in runData.parameters 3. Execution context is captured in executionContext 4. Metrics (CPU, memory) are properly stored with correct types

2. Re-run Flow¶

User Request: POST /function-runs/:id/rerun
    ↓
1. Fetch original run by ID
    ↓
2. Decrypt stored parameters from runData.parameters
    ↓
3. Apply parameter overrides (if provided)
    ↓
4. Validate original trigger exists
    ↓
5. Branch based on trigger type:
   - On-demand: Use executeOnDemandTrigger() (preserves existing behavior)
   - Other types: Queue execution directly with original parameters
    ↓
6. Increment rerunCount on original run
    ↓
7. Return new job ID

3. Trigger Type Support¶

All trigger types can be re-run: - On-demand: Uses existing executeOnDemandTrigger method - Scheduled: Queues execution with stored parameters (as if the schedule triggered it) - Event-driven: Queues execution with stored event data - Webhook: Queues execution with stored webhook payload

3. Re-run Tracking¶

When the re-run completes: - New run is created with isRerun = true - originalRunId points back to the source run - rerunBy captures who triggered the re-run - rerunReason stores user-provided rationale - Original run's rerunCount is incremented

4. Lineage Example¶

Original Run (run-1)
├─ rerunCount: 2
├─ isRerun: false
└─ originalRunId: null

Re-run 1 (run-2)
├─ isRerun: true
├─ originalRunId: run-1
└─ rerunBy: user-123

Re-run 2 (run-3)
├─ isRerun: true
├─ originalRunId: run-1
└─ rerunBy: user-456

Code Snapshot Strategy¶

Function Code Hash¶

When a run is created, the function code is hashed and stored. This enables:

1. Reproducibility: Know exactly which code version was executed
2. Change Detection: Identify if function code changed between runs
3. Debugging: Understand what code produced specific results

Re-run Behavior Options¶

Option A: Use Original Code Snapshot (Not Yet Implemented)¶

{
  "useLatestCode": false  // Use code from original run
}

Option B: Use Current Code (Current Implementation)¶

{
  "useLatestCode": true   // Use current function code
}

Current Default: Always uses current code (snapshot retrieval not yet implemented)

Validation & Edge Cases¶

Validations Performed¶

1. ✅ Run Exists: Original run ID is valid
2. ✅ Workspace Match: Run belongs to the specified workspace
3. ✅ Parameters Exist: Original run has stored parameters
4. ✅ Trigger Exists: Original trigger still exists in database
5. ✅ All Trigger Types Supported: Can re-run on-demand, scheduled, event-driven, and webhook triggers
6. ✅ Decryption Success: Parameters can be decrypted

Error Scenarios¶

Authorization¶

Re-run requires execute permission on function-triggers resource:

const auth = await authorize(req, res, "execute", workspaceSlug, "function-triggers", "workspace");

This ensures only users with execution rights can trigger re-runs.

Parameter Overrides¶

You can override specific parameters while keeping the rest from the original run:

Original Parameters:

{
  "apiKey": "encrypted-key",
  "mode": "production",
  "timeout": 30
}

Re-run Request:

{
  "overrideParams": {
    "mode": "debug",
    "timeout": 60
  }
}

Effective Parameters (merged):

{
  "apiKey": "encrypted-key",  // ← From original
  "mode": "debug",            // ← Overridden
  "timeout": 60               // ← Overridden
}

Future Enhancements¶

1. Code Snapshot Retrieval¶

Store full function code in runData to enable true "exact re-run":

runData: {
  parameters: encryptedParams,
  outputs: outputs,
  functionCodeSnapshot: functionCode,  // ← New field
  ...
}

2. Bulk Re-runs¶

Re-run multiple failed runs at once:

POST /function-runs/bulk-rerun
{
  "runIds": ["run-1", "run-2", "run-3"],
  "reason": "Retry all failures from outage"
}

3. Conditional Re-runs¶

Re-run only if certain conditions are met:

{
  "condition": {
    "status": "failure",
    "errorType": "timeout"
  }
}

4. Scheduled Re-runs¶

Automatically re-run failed executions:

{
  "autoRerun": {
    "maxAttempts": 3,
    "backoffStrategy": "exponential"
  }
}

5. Re-run Chains¶

When re-running Function A, also re-run dependent Function B:

{
  "cascadeReruns": true
}

Testing¶

Manual Testing¶

1. Execute a function:

   POST /workspace/test/api/v1/function-triggers/:triggerId/execute
   {
     "param1": "value1"
   }
   

2. Capture the run ID from the response or query runs:

   GET /workspace/test/api/v1/function-runs
   

3. Re-run the function:

   POST /workspace/test/api/v1/function-runs/:runId/rerun
   {
     "reason": "Testing re-run feature"
   }
   

4. Check re-run history:

   GET /workspace/test/api/v1/function-runs/:runId/rerun-history
   

Verification¶

After migration, verify schema changes:

SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'function_runs'
AND column_name IN (
  'originalRunId',
  'isRerun',
  'rerunCount',
  'functionCodeHash',
  'memoryUsageBytes',
  'cpuUsageSeconds'
);

Migration Checklist¶

• [x] Create migration file
• [x] Update FunctionRun TypeScript interface
• [x] Add repository methods (incrementRerunCount, getRerunsByOriginalRunId)
• [x] Update persistFunctionRunWorkflow to populate new fields
• [x] Implement rerunFromPreviousRun service method
• [x] Add controller methods (rerun, getRerunHistory)
• [x] Add API routes
• [ ] Run migration: npm run migrate
• [ ] Test re-run API endpoint
• [ ] Update API documentation
• [ ] Add integration tests

Additional Notes¶

Security¶

• Parameters are encrypted in storage using SecretService
• Decryption happens only during re-run execution
• Re-run requires same permissions as original execution

Performance¶

• Indexes added for common query patterns:
• originalRunId for lineage queries
• (workspaceSlug, isRerun) for filtering re-runs
• (triggerId, createdAt) for temporal queries
• (functionId, status) for function-specific analytics

Monitoring¶

Track re-run metrics: - Total re-runs per workspace - Re-run success/failure rate - Most re-run functions (reliability issues) - Average time between original run and re-run