Batch Processing with BPEL

Reading
the File

The first task is to create a new BPEL project
within JDeveloper.  I decided to create a new BPEL project with an
empty BPEL process.  This was because I wanted to control the partner
links.  I needed control of the partner links because I was not going
to use SOAP over HTTP bindings, which are generated for a default synch
or asynch process.  Rather I wanted to use file and database adapters
for the bindings.  This gave me the blank process as shown
below.

I
added a partner link to this blank process and then used the file
adapter wizard to specify how I wanted it to behave.  To begin with I
specified that I wanted to read from a file.

Having
specified the read operation I then told the adapter where to find the
input files.  I used a physical path, but better practice is to use a
logical name which can be altered via the console, making it easier to
move the process between environments.  in addition to the directory I
also specified that I wanted the file to be copied to an archive
directory after processing.  Finally I specified that I wanted the file
deleted from its original location after processing.  These choices are
reflected in the screen below.

I
now needed to provide some more information about the input files.  I
needed to provide a pattern to identify which files I wanted to
process, I chose a simple wildcard expression rather than a regular
expression.  Note that the file adapter uses the term message rather
than record, this doesn't entirely make sense as a message may contain
many records (as determined by the batch size) - nevertheless I will
use message to be consistent with the screens in the adapter wizard
even though this confuses records and messages.  It is at this point in
the process that the number of messages records in the file is
identified (1 or many) and also the batching of messages is
determined.  To start with I set the number of messages in a batch to
be 1.  This means that for every message (record) in the file a new
BPEL process will be created.  More on this later.

Having
specified batch sizes and file matching expressions I now need to set
the frequency of checking the directory for new files (the Polling
Frequency) and also the minimum file age.  Minimum file age means that
a file must be at least this old before it can be processed.  This
allows for files that take a long time to create, the minimum file age
can be set to a large value to enable the source system to finish
creating the file.

Finally
I needed to specify the format of the records/messages in the file.  To
do this I used the "Define Schema for Native Format"
wizard.

I
said that I wanted to create a new native format, in my case the file
was made up of fixed length fields, but it could have been in delimited
format such as csv (comma seperated values).

I
then provided a sample input file that I could use to identify the
field boundaries.  I then chose which parts of the file I wanted to
read in as a sample. 

My
file had more than one record type, in fact it had a header, an
arbitary number of records and a trailer.  With this in mind I chose
the appropriate type of file organisation.

Before
specifying the individual fields I needed to chose a name space and a
root element name.

I
could now identify the discriminator for each record type - in this
case it was the first character of the record.  The wizard identified
the three record types and I changed the record type names to sensible
names that meant something to me.

I
now used the wizard to identify the individual record field
boundaries.  Each type of record needs specifying seperately.  I found
it best to do this with a mouse as I had some problems when entering
numbers directly into the list.

Having
identified field boundaries I then needed to identify the type of the
fields.  Again this has to be done for each record
type.

Finally
we can review the generated XSchema and select a filename to save it
in.

We
have now defined our file format and mapped it onto an XML record
structure.  We can now return to the file adapter
wizard.

The
file adapter is now set up to read a single record at a time from the
file.
I could now add a Receive to the process and use the new
created FileInputService partner link.  When creating the receive I
marked the "Create Instance" check box as I needed to create a new
process
when a message was received.

Writing
to the
Database

Having created a partner link and a receive activty
to read the input file I now needed to create a partner link to write
it to the database.  To do this I used the database adapter and then
created a simple assign statement to set up the call to the database.
The resultant process shown below was then
deployed.

Running
the Process

After
deploying the process to the BPEL Process Manager server the next step
is to run it.  To do this I dropped a sample file into the specified
input directory.  The file adapter detected the file, read it, and
created a process for each message in the file.  This resulted in a
large number of processes being created.  The process had two major
problems

• Two processes had errors because
  they had no data record (the header and trailer records were passed to
  these processes)
• A single file had created more
  than 200 processes which made it difficult to see what was
  happening

An obvious solution to these
problems would be to batch multiple records into a single
process.

Batching the Data

In order to
receive multiple records into a single process I went back to the File
Adapter wizard and altered the "Publish Messages in Batches of " field
to have a larger number than one, I chose fifty.

If I had deployed and
tested the process without any other changes I would have found only
one record being processed in each batch.  This is because I have yet
to modify the rest of the process to iterate over multiple
records.
The first step is to figure out how many records I
need to process.  To calculate this I created a new integer variable
"NumberOfRecords" to hold the number of records and initialised it with
an assign statement as shown below. 

The assignment calculates the number of Data elements underneath the
document root element Records.
I then created another integer
variable "CurrentRecord", initialised to 0, to act as a for loop
counter.  I then created a while statement to loop over all the Data
records in the message.  The while condition was
"CurrentRecord<NumberOfRecords" as shown below

Within
the while loop I then created an assignment statement to set up the
write to the database.  Unfortunately it is not possible to use the wizard to completely set up the copy operation.  I followed this process

• Create assignment as if there were a single record rather than multiple. This gave me the following xpath expression
• /ns3:Records/ns3:Data
• Manually edit the the assignment statement to add an array index to identify the specific record.  The expression now looked like this.
• /ns3:Records/ns3:Data[bpws:getVariableData('CurrentRecord')]

When incrementing the CurrentRecord variable it is important to remember that, unlike Java and C/C++, XPath indices start at 1, not 0, so the CurrentRecord value must be incremented before using it as an index into the record set.
After deploying this process a 200 record file would now produce just 4 processes, making it much more manageable.

Reasons for Batching

So why would you want to batch messages?

• Easier to track what is happening as fewer processes to manage.
• Slightly more efficient within the BPEL PM - some testing I did indicated it was a little more efficient to batch, presumably because of the reduced process creation overhead.

How do I process a file with multiple records as a single process?

• Treat it as a single message by not checking the "Files contain Multiple Messages" check box in step 5 of the File Adapter Configuration wizard.
• Individual records in the message are indexed in the same way as multiple records in the multiple records example above.

Why would you not batch?

• Really want each individual record to initiate a new process that is to be tracked individually, for example equipment provisioning requests, each of which is part of a seperate customer order.
• Want to simplify process by only handling a single record.
• Too idle to deal with looping over sets of records in a message (oops, that was the same as the last one!)

Key Points to Remember

• Use XPath to index into multiple records
• Use wizard to create copy rules then add indexing afterwards
• Use a while loop to iterate over records
• XPath indices start at 1

So happy batching!