Real Time Imports

Quickly identifying when changes occur in another database can be challenging. Astra Schedule handles this by employing a real-time “differencer”.  The differencer runs as a separate service that constantly watches the SIS database for changes to certain fields.  This process compares field values in changed records to identify key field changes and then notifies the system that an update is required. A real-time import job may be configured and scheduled to run nearly as often as desired to find changes in your SIS and update the corresponding records in Astra Schedule. 

The real-time import process will insert new records when found as well as perform updates to existing records.  Because the real-time process is only watching for changes to a key subset of fields (as defined by the import configuration files), the identification of changes can work extremely quickly, keeping critical elements current during scheduling processes.  Depending on the configuration, a batch import may still be required periodically to update certain fields.

Real Time XML File

When setting up the real-time import process, a real-time SIS job is created. During the configuration of the job, an xml configuration file is selected that specifies which subset of the data will be checked for changes.  By only comparing a few critical fields for changes, the performance of the real-time process can be extremely fast.

The real-time-specific xml import configuration file supplies the system with the information it needs to perform the update, including the SQL query used to get section information from the SIS and the specific fields being compared to identify changes.  The SQL query takes into account the interface job parameters for term and campus.  If applicable in the SIS data, the query can also take into account an activity date, against which it can compare changes in Astra Schedule.  If an activity date is available, this can further increase the speed of real-time updates.

Once the data is returned by the query, the fields specified by the configuration file for the difference check are compared to Astra Schedule.

Default SIS Fields Checked

If a difference is detected in one of these fields, the real-time process informs the updater process that certain section ID's require updating.  The real-time service's only job is to find the change and wake up the update process as needed. The regular update process (the same one used to perform the Batch Import Process) is utilized to perform the updated record import.  The fields included in the difference check may be edited to specify either more or fewer, fields to be checked.  When the update is performed, the fields being updated are dictated by the configuration file being used by the on-demand update process. 

Real-Time Scheduling

The frequency at which the real-time import process runs is dictated by the job's schedule in combination with the processing time required. Job schedule settings can be configured to run as often as desired. However, the processing is limited by the environment (the time it takes to run the required queries against the database). In practice, it is recommended that you do not schedule real-time imports to run more frequently than about 5 minutes.

Multiple schedules can be configured per job to achieve real-time processing at different intervals at different times during the week.  For example, real-time could be scheduled to run every 5 minutes between 6 AM and 6 PM, Monday through Friday, but then only once per hour during the evenings after 6.  Note that creating multiple schedules within a job that happen to overlap will not create a problem because each schedule runs independently on its own communication thread. Real-time import is most commonly used to update with regular frequency during business hours during weekdays.  Real-time import job schedules can be enabled and disabled by changing the "is active" status flag.  To stop a job schedule from running, remove the flag to inactivate the schedule.  Add the flag to start the schedule again.

Before enabling a real-time job schedule, it is necessary to run a batch job for the same term and campus to make sure the existing data is current.  The real-time process will begin checking for changes from the moment it is enabled forward, but will not catch differences that existed when it became live.  Additionally, when a real-time schedule stops and then restarts it doesn't know about changes that may have occurred in between.  Therefore, batch imports may still be required to update the system periodically with any changes that occur while the real-time job schedule is dormant.  The best practice may be to schedule a batch import to occur at a downtime, just before the real-time schedule begins.  For example, a scheduled batch import could run at 6 AM every weekday, and then real-time could begin at 6:30 AM and run until 6 PM every weekday.

The real-time process logs its activity in the Differencer folder in the installation directory.  This log file contains information regarding any errors encountered when attempting to connect to the SIS database as well as any other failures during the difference checking process.  During normal operation, the system creates rows in the log file for each record queried and any changes encountered. 

Setting up a Real-Time Job

To configure the real-time import process:

1. Click the Import/Export Management option on the Settings tab. The Job list page will be displayed.
2. Click the Add Job
3. Select the Real-Time option from the drop-down list and click Add Job.
4. Enter a job name. This should be a name that will serve to remind you of the reason for the job and to help distinguish it from others on the list.
5. Enter a job description if desired.
6. Choose the appropriate "Differencer" xml configuration file from the drop-down list. The database adapter type and data type supported by the configuration file selection is displayed for reference.
7. Select one or more terms that should be included in this job and use the left arrow button to add them to the Selected Items panel.
8. Select one or more campuses that should be included in this job and use the left arrow to add them to the Selected Items panel.
   
   The list of terms and campuses corresponds to the terms and campuses created in Astra Schedule. The SIS names of these items must match in order for the import of data to be successful.
9. Click Add Schedule to create a schedule entry for the job.
10. On the Job Schedule window, verify the active status of the job.
11. Configure a time frequency for the schedule. The "Every" option is pre-selected for real-time import jobs.
12. Select and configure a day pattern for the schedule.
13. Specify the date range for the schedule.
    
    The schedule being created will not start until the next start time in the cycle. In other words, if the schedule has a start time that is before the current time, the job will not start until the next day triggers the service to wake up.
14. Click OK to add the schedule to the Job Schedule list on the job form.
15. Click Save.