skip to main content
Corticon Server: Data Integration Guide : Getting Started with Batch

Try Corticon Now

Getting Started with Batch

Batch processing is a server-based function that extends EDC and ADC functionality. Elevating an EDC/ADC solution to a batch job means that Corticon Server can take several patientIds from the database's Patient table at once, and pass them in as a set to the Decision Service for processing – a very efficient way to perform processing – where the Decision Service retrieves additional data for each request from the database for rule conditions and to enrich the record in the database with results. You get greater control over queries and insert statements that are used. This is beneficial when you need finer control for performance or need to retrieve large amounts of data.
How batch processing works - Fetching the transaction identifying data from the underlying relational data source that will be injected into the rules engine – seed data retrieval – takes place outside the Decision Service. As such, Decision Service requests that are usually individual transactions are instead fetched in chunks for the rules engine, and then dispersed across multiple processing threads to concurrently process the incoming requests. Batch processing produces no return payload per request -- the result of each rule processing is persisted in the database.
Note: For this sample, you must have:
1. Completed the steps in Getting Started with ADC that:
a. Ran the scripts patient and adc that set up the database with the schema and data.
b. Imported the metadata into the Vocabulary.
c. Set the Ruleflow's Get Patient Data query to IndicatedPatients.
2. Packaged and saved the Decision Service file and its Datasource Configuration file where the server will be able to access them.
3. Installed Corticon Server with the server and Web Console options, and then started the server.
To load the batch sample:
In Corticon Studio, choose the menu item Help > Samples. Select the Advanced Sample Batch Rule Processing, and then click Done. Follow the Import dialog to bring the sample into your workspace. The batch sample is just two SQL queries that will be used later in this topic, yet that step completes setup.
Note: For details on defining and running batch processes, see the section "Batch Configurations" in the Web Console Guide
Batch configuration depends on the Server where the Web Console connects to have access to::
*A Decision Service that has Ruleflow Service Callouts that reference the databases and queries
*The Decision Service project's Datasource Configuration
*BATCH_READ queries loaded in the query service database
To run the batch sample:
1. In your browser, connect to the Web Console, typically http://localhost:8850/corticon, using the default credentials Admin/Admin.
2. In the Web Console, choose Decision Services, and then click Add Decision Service for just a single Decision Service. In the dialog box:
a. Name your service. For example, myADC_Sample 1.0.
b. Choose the Decision Service file you created.
c. Choose its server location.
The settings will look similar to this:
d. On the Database tab, choose your Datasource Configuration File.
e. Click Save and Deploy.
3. Choose Batch Configurations, and then click New Batch Configuration. In the dialog box:
a. Name the batch configuration. or example, myADC_Batch.
b. Choose your deployed Decision Service.
c. Choose its Datasource. This is a specific database name within the datasource configuration file.
(See Exporting the Datasource Configuration file)
d. Choose the AllPatients query.
The settings will look similar to this:
e. Save the batch configuration.
4. The Batch Configuration page lists the new configuration:
Run the batch job by clicking on its Batch Configuration name, and then clicking Execute:
5. On the Batch Configuration page detail page you can see the Processed Count each time you run it.
6. Test that the database is getting updates by looking at the Treatment table's Approved column.
Note: Clearing sample writes - To reset the approval values written to the database Treatment table to NULL, run the Batch sample's clear_approved script in your DB management tool.

Parameters in queries

Batch configurations can access queries that are defined to request parameter values. The BATCH_READ query PatientsByRegion runs the query:
SELECT patientId FROM Patient WHERE region IN ({Patient.region})
To run a parameterized batch job:
1. Create batch configuration named MyADC_Batch_Regional. Select the Decision Service you deployed, its Datasource, and the query PatientsByRegion. That adds another field to accept the values of your Query Parameters. Enter the value NE.The settings will look similar to this:
2. Save and then run this batch job. When you inspect the Treatment table you see that only patients from the NE region have been processed.
Note: For more about the format of Corticon's queries, see How Corticon is expressed in SQL.

Creating additional sample records in the database

As the sample set of patient/treatment records is very small, the efficiency of rapid batch processing can be difficult to observe. A SQL utility is provided that will generate PATIENT_COUNT additional records for your tests. It is a good idea to also adjust the FIRST_PATIENT_ID and FIRST_TREATMENT_ID so that they are not overwriting each time you execute the utility
To create large data sets and run large batch tests:
1. Open the script in the Batch sample's generate_patients script in your DB management tool.
2. Change the values for:
These set the number of patients you want generated, and the starting ids for patients and treatments.
3. Run the batch job with this newly generated data, and then look at the Decision Service page to see the counters and charts.