A State Processing Engine

This is my third post about platform events. Last time I shared a few things I've learned about this tool. Now I want to share some great advice I got from my friend Chris Pifer about how to work with platform events. The trick is to allow the platform event to break processing into individual asynchronous bites but not lose track of the state of the work.

Use Case

I'm working with CollegeSpring, who runs college preparation classes for schools all over the country. They need to bring in records from their learning management system (LMS), called Buzz. Once CollegeSpring sets up the LMS for the partner school, the students (and teachers) mainly are tracked in the LMS—that's the system of record for their names, school Ids, and which classes they're in.

The LMS has thousands of records: students (contacts) are enrolled (junction object) in classes (custom object) that are connected to programs (custom object) that are related to schools (accounts). We can get a spreadsheet out of Buzz with a row for every student and teacher, indicating their enrollment. But when we get the spreadsheet again a day or a week later we want to figure out, in Salesforce, whether the student exists (create them if not) and their enrollment is already recorded (create it if not, update it if changed). Particularly at the beginning of the semester, students might come and go or move around from class to class.

Simple in Theory

Thought about singly, a row of the export from the LMS is quite simple:

Student Name (and unique LMS Id) LMS Enrollment Id Enrollment Status Class LMS Id Program LMS Id (tied to school Account)

A flow to check if the student, class, and/or program exists and then create or update as required isn't that hard to conceptualize. The problem is that this flow might hit limits when it runs, particularly if you're inserting a lot of records at a time, as we intend to do.

So from the start I wanted to set this up so that processing would be asynchronous and would run for one record at a time in its own transaction. That's where platform events are my friend.

Platform events also gave me the option of partially processing the import record as it goes through its steps. That has three benefits:

1. If a later step fails the earlier parts that were successful don't get rolled back. In other words, I can save partially processed records in that partial state. Later, I can fix whatever caused them to fail, and then restart processing.

2. When re-running the flow (ie. reprocessing the record) it doesn't have to redo work.

3. In any given processing step I have just two or three SOQL or DML operations (pink flow elements).

A Complicated Flow?

The actual flow, once built, looks a lot more complicated than it really is. But that's because I've put in a bunch of fault paths and logging. I almost don't want to show you a screenshot for fear I make it seem scary. But here you go:

Actually Not Complicated

I promise you, here's all the flow is really doing (illustrated in LucidChart because flow isn't really built for screenshots):

In words instead of image:

1. A platform event is published, which kicks off the flow.

2. The flow looks up the import holding object (called a BuzzDataUpload), where all the data about the import is living. (I didn't put data on the platform event, it all lives on the BuzzDataUpload.)

3. A decision checks if all the related objects have already been found and their lookup fields filled. (It checks the state of the record.)

   1. If so, set the BuzzDataUpload's processing status as "✅ Complete" and we are done.

4. If the Account lookup is empty (meaning the import object hasn't yet been matched to an account)

   1. do a Get Records to find the account,

   2. update the BuzzDataUpload record with the Id of that account and set its processing status to "⏳ Processing,"

   3. publish a new platform event. (Go back to Start.)

   4. [If no account was found, update the import record with a processing status of "⛔️ Processing Error." Do not publish another platform event.]

5. If the Account was filled but the Contact lookup is not filled,

   1. upsert a contact,

   2. update the BuzzDataUpload record with the Id of that contact,

   3. publish a new platform event. (Go back to Start.)

6. If the Account and Contact lookups are filled, but not the Enrollment,

   1. upsert an enrollment,

   2. update the BuzzDataUpload record with the Id of that enrollment,

   3. publish a new platform event. (Go back to Start.)

   Etcetera until the BuzzDataUpload has been fully processed.

All the Bells and Whistles

My final flow has a lot more elements on the canvas in order to include a bunch of refinements, including things like:

• Counting the number of platform events that have fired, as a check on how many times the record has run through processing

• Inserting FlowLog records at various places to help with debugging

  

• Lookups for record types before creating records (like enrollments)

• Decision elements to check if the field(s) we are about to use to look for records are outside of expected values, such as the Role field

  

• Fault Paths on just about every Get Records or Create Records step to log if there is some kind of error

• A Processing Notes field that the flow updates when it has an error (like in step 4d, above)

  

• The real use case has five fields that eventually have to link the BuzzDataUpload to records of other objects, so two more full branches of the flow.

But none of that changes the heart of the flow. It's simple in concept, but in reality covered in frills and decoration.

What to do with ⛔️ Processing Error?

It might seem scary that I've got a branch of that flow that might mark a BuzzDataUpload with "⛔️ Processing Error." But that's the fun of the State Machine. With that field value we know that BuzzDataUpload needs reprocessing. (And the Processing Notes field tells us where processing ended.) That leaves just two things we need:

1. A way to fix whatever problem happened the first time

2. A way to send the record to be reprocessed

Fixing the Problem

Let's take the case of 4d in the example above. If there was no account found based on the values in the import holding object, we don't want to create an account. The account should definitely already exist in Salesforce! (It's one of our partner schools. We have an opportunity that's closed won, details of our contract, several people that are our main points of communication with the school. If we don't have those things, that seems bad...)

The records we're importing from the learning management system include the LMS Id of the school as well as the account name. I want my flow to match on the unique Id, so there is no chance of confusion if school names are similar, mispelled, or whatever. But if we haven't entered the LMS's unique Id for the school into Salesforce, then that Get Records step is going to fail. I want that to happen! Now the flow just marks the import object as follows:

Processing Status: ⛔️ Processing Error

Processing Notes: "No Account found with LMS Id that matches this import record."

When a human sees that processing note it's pretty obvious what needs fixing: We have not taken the LMS's program Id (created when the program was set up for the school) and associated it to the partner school's account record in Salesforce.

[Hopefully we don't get to this point very often. We have an audit report to watch for partner schools that should have a program this year but don't. And we have another report for programs for this year that are active but don't have an LMS Id. But if neither of those audit reports caught the problem or if it wasn't fixed before we imported the LMS data, then the processing of the LMS data for each student and teacher at the school is going to error.]

So once we see a record with the note "No Account found with LMS Id that matches this import record." we just have to add an LMS Id to the right account. Next time that record is processed, it shouldn't have any problem.

Sending it to Reprocess

Reprocessing a BuzzDataUpload record is as easy as checking a box.

Whoever is working with the BuzzDataUpload record can check that box on a single record. Or they can use a list view to bulk inline edit the Reprocess box to checked.

Or they could do it in even larger numbers at a time through Apsona.

Once the box is checked, there's a record-triggered flow that publishes a platform event (and unchecks the box). Publication of that platform event kicks off the flow above. And we're back to the beginning, a very good place to start..

The State's the Thing

Each time the platform event is published it causes that one BuzzDataUpload record's state to be checked, where the state is represented by all the lookup fields to the other records thats should exist when processing is complete. If one or more are not filled, the record needs another turn through the flow. If they're all filled, then we are done:

The Single-Record Workaround

Last post I mentioned that if platform events are published in batches they also get processed in batches, which can result in errors. I promised a workaround. (Credit to Andrew Russo for teaching me this one.)

Originally I had a record-triggered flow that ran on any insert or update of a BuzzDataUpload record and published a platform event to kick off processing of that record. Since the inserts/updates were happening in bulk, processing was in bulk as well and I was finding weird errors. (Like my upsert of the contact erroring because there was already a contact with the unique LMS Id, which clearly was what the upsert was supposed to avoid!)

Andrew's workaround was to put publishing of the platform event onto a scheduled path, where you can set the Batch Size to one.

Adding a scheduled path to my flow also meant that I needed to have two flows, rather than just one.

• On Create

• On Update (when Reprocess is changed to be true.)

(There's no way to combine those two flows because you can't make flow entry criteria that will pick up a new record and pick up a change to Reprocess "Only when a record is updated to meet the condition requirements," which is what you need in order to have a scheduled path.)

But those flows are otherwise identical:

On the immediate path, the BuzzDataUpload record is set to Processing Status "⏳ Processing" and Processing Note "Waiting to publish platform event." After the zero minute pause (which, in practice, seems to take at least one minute) the platform event is published (in a batch size of one).

This does mean that insert or reprocessing of records waits for at least one minute. But the productivity gain over records erroring is worth way more than that!