Module 11 — Salesforce Bulk Processing + Mule Batch Jobs
Module 11 — Salesforce Bulk Processing + Mule Batch Jobs
This module is about what changes when the requirement is no longer:
“process 1 customer”
but:
“process 100,000 customers”
At that point, a normal synchronous flow with a For Each and one Salesforce call per record is usually the wrong design.
The two concepts to separate are:
Salesforce Bulk API
vs
Mule Batch Job
They solve related but different problems.
11.1The simplest mental modelThink:11.2Why normal CRUD stops scalingSuppose:11.3Salesforce Bulk APIThe Salesforce Connector exposes Bulk API functionality, including current Bulk API v2 operations for insert, update, upsert, delete, and…11.4Bulk API v2 write exampleRequirement:11.5Why Bulk API is asynchronousIf Salesforce had to synchronously return after processing:11.6Job lifecycleConceptually:11.7Bulk queryBulk isn't only for writes.11.8Bulk query scenarioExample:11.9Pagination and memoryCurrent MuleSoft documentation notes that Bulk API v2 query-result retrieval supports paging and that page size affects both API call count…11.10StreamingWith large data, the word:11.11Repeatable vs non-repeatable streamsHigh-level only:11.12Mule Batch JobNow let's switch from Salesforce to Mule.11.13Batch Job internalsImagine input:11.14Batch StepA Batch Step defines processing applied to records.11.15Record-level processingInside a Batch Step:11.16Batch failures don't necessarily stop everythingThis is one of the best features.11.17`maxFailedRecords`Mule Batch lets you define how many record failures are acceptable before stopping the job.11.18Why this mattersImagine source file:11.19Batch Step accept policiesSuppose Step 1 validates.11.20Example batch pipelineLet's design:11.21Batch AggregatorThis one matters a lot.11.22Why Aggregator is importantBad:11.23Aggregator fixed sizeConceptually:11.24Aggregator streaming modeA Batch Aggregator can also operate in streaming mode instead of waiting for a fixed-size block.11.25Mule Batch vs Salesforce Bulk API againLet's make the distinction crystal clear.11.26You don't always need bothCase:11.27Normal Connector vs Bulk APIA useful interview heuristic:11.28Batch size tuningIf you aggregate:11.29Failure granularity tradeoffImagine batch size:11.30Salesforce partial failuresSuppose a bulk upsert contains:11.31Separate transient vs permanent bulk errorsSame rule as Module 7.11.32Don't replay successful records unnecessarilySuppose:11.33External IDs are even more valuable in bulkImagine bulk job times out from Mule's perspective.11.34Parent-child bulk loadSuppose you need:11.35Don't assume all entities can load in parallelExample:11.36Large synchronization designRequirement:11.37Full reload vs delta loadSimple.11.38Deletes are the awkward partSuppose source record disappears.11.39ReconciliationEven a beautifully designed delta sync can drift.11.40On CompleteMule Batch has an On Complete phase after record processing finishes.11.41Don't use On Complete for every recordOn Complete is job-level reporting.11.42Batch Jobs are asynchronousAn important behavioral point:11.43Batch restartabilityOne reason Mule Batch is attractive is reliability.11.44Parallelism inside Mule BatchBatch steps can process record blocks in parallel.11.45Salesforce record locking at scaleExample:11.46Serial vs parallelHigh level:11.47Don't put an HTTP call inside each Batch record casuallyImagine:11.48N+1 at bulk scale is catastrophicWith normal API work:11.49Memory is an architectural constraintSuppose each customer record averages:11.50Interview scenarioWe have 500,000 records to load into Salesforce overnight. How would you build it?11.51Interview scenarioWhy not Parallel For Each?11.52Interview scenarioOne record in 100,000 is invalid. What happens?11.53Interview scenario15% of the job suddenly fails with INVALID_FIELD.11.54Error-rate thresholdsThis is an excellent production idea.11.55Testing bulk integrationsYou should test more than happy-path count.11.56Test mixed outcomesDataset:11.57Test restart/replaySimulate:11.58Test relationship orderingExample:11.59Test reconciliationAfter job:11.60Batch performance metricsMeasure:11.61Your production dashboardFor a nightly sync, I would want something like:11.62Mule Batch vs Salesforce Bulk interview summaryIf asked:•Module 11 Cheat SheetThe interview sentence to memorize is: