Batch processing is a server-based function that extends EDC and ADC functionality. Elevating an EDC/ADC solution to a batch job means that Corticon Server can take several patientIds from the database's Patient table at once, and pass them in as a set to the Decision Service for processing – a very efficient way to perform processing – where the Decision Service retrieves additional data for each request from the database for rule conditions and to enrich the record in the database with results. You get greater control over queries and insert statements that are used. This is beneficial when you need finer control for performance or need to retrieve large amounts of data.
How batch processing works - Fetching the transaction identifying data from the underlying relational data source that will be injected into the rules engine – seed data retrieval – takes place outside the Decision Service. As such, Decision Service requests that are usually individual transactions are instead fetched in chunks for the rules engine, and then dispersed across multiple processing threads to concurrently process the incoming requests. Batch processing produces no return payload per request -- the result of each rule processing is persisted in the database.
a. Ran the scripts patient and adc that set up the database with the schema and data.
b. Imported the metadata into the Vocabulary.
c. Set the Ruleflow's Get Patient Data query to IndicatedPatients.
2. Packaged and saved the Decision Service file and its Datasource Configuration file where the server will be able to access them.
3. Installed Corticon Server with the server and Web Console options, and then started the server.
To load the batch sample:
In Corticon Studio, choose the menu item Help > Samples. Select the Advanced Sample Batch Rule Processing, and then click Done. Follow the Import dialog to bring the sample into your workspace. The batch sample is just two SQL queries that will be used later in this topic, yet that step completes setup.
4. The Batch Configuration page lists the new configuration:
Run the batch job by clicking on its Batch Configuration name, and then clicking Execute:
5. On the Batch Configuration page detail page you can see the Processed Count each time you run it.
6. Test that the database is getting updates by looking at the Treatment table's Approved column.
Note: Clearing sample writes - To reset the approval values written to the database Treatment table to NULL, run the Batch sample's clear_approved script in your DB management tool.
Parameters in queries
Batch configurations can access queries that are defined to request parameter values. The BATCH_READ query PatientsByRegion runs the query:
SELECT patientId FROM Patient WHERE region IN ({Patient.region})
To run a parameterized batch job:
1. Create batch configuration named MyADC_Batch_Regional. Select the Decision Service you deployed, its Datasource, and the query PatientsByRegion. That adds another field to accept the values of your Query Parameters. Enter the value NE.The settings will look similar to this:
2. Save and then run this batch job. When you inspect the Treatment table you see that only patients from the NE region have been processed.
Creating additional sample records in the database
As the sample set of patient/treatment records is very small, the efficiency of rapid batch processing can be difficult to observe. A SQL utility is provided that will generate PATIENT_COUNT additional records for your tests. It is a good idea to also adjust the FIRST_PATIENT_ID and FIRST_TREATMENT_ID so that they are not overwriting each time you execute the utility
To create large data sets and run large batch tests:
1. Open the script in the Batch sample's generate_patients script in your DB management tool.
2. Change the values for:
SET @PATIENT_COUNT=1000 SET @FIRST_PATIENT_ID=1000 SET @FIRST_TREATMENT_ID=1000
These set the number of patients you want generated, and the starting ids for patients and treatments.
3. Run the batch job with this newly generated data, and then look at the Decision Service page to see the counters and charts.
A Decision Service that has Ruleflow Service Callouts that reference the databases and queries
