New Features

#3547: Rollback (undo) Batch

It is now possible to rollback/undo the last committed batch for a target table. The batch has to be the very last one committed for each of its target tables. It is possible to rollback several batches, but they have to be done in sequence, starting with the very last batch and then proceeding progressively back in time. Both MERGE or REPLACE load strategies are supported.

Current limitations:

  • The temporary staging data must still exist for the batches (~3 days)
  • Model batches are not supported, but may be supported in the future
  • Batches spawned in the PostRun section of a pipeline are not rolled back, but this may be supported in the future.

The rollback button is available on the load, batch and batch history pages. 

During Preview, the status of the batch is “Rollback Preview” and the “To Rollback“ section displays the details. In the original batch of the example below, 1 new record had been inserted and 1 updated (value change from 32 to 30). This appears as 1 record to delete and 1 record to update (rolling back 30 to 32).

Press Commit to commit the rollback. This status of the original batch becomes “Rolled Back”:

Enhancements

#3679: Edit Model Usability Improvement (convenience)

A short-cut link has been added to a table’s model page that directly opens the table for editing (editing its structure, not its data).

When the Edit Model link is clicked, the table is automatically selected on the Edit Model page.

#3847: Re-process in debug mode (convenience)

xMart has a very useful debug feature when working on pipelines which allows the pipeline editor to see the contents of every in-memory pipeline table at each step of the pipeline (how to debug a pipeline).

What is new: on the main load page, for convenience, when the reprocess button is pressed for a batch that was created in debug mode, the new batch will also be created in debug mode. 

#3839: Support/render simple html in mart description

The home page description now supports basic HTML, which is editable on the Admin/Mart page:

The above html rendered on the home page:

Note that both implied (no <a> tag but links beginning with http:// or https://) and explicit hyperlinks (<a> tags) are supported.

HierarchyLookup using LookupInterval performance enhancement

A performance enhancement (60 times faster) has been implemented for pipelines using the HierarchyLookup command with the LookupInterval option. The LookupInterval option supports matching a business date in the loaded data with effective start and end dates in the reference data . See Example 2 of the HierarchyLookup command.

Small "breaking" changes:

  • The <LookupInterval> element can now only be placed first under <Lookups>
  • Lookups elements can only have at most 1 (optional) <LookupInterval> child

#3326: Ability to Group Rows - using CustomScript command

In a pipeline, it is now possible to group/aggregate data using the CustomScript command, as an alternative to doing grouping/aggregation in an SQL view.

It’s a little complicated so an example has been provided in the script guide: see Example #2 of the CustomScript command . We may consider a dedicated pipeline transformation command in the future to simplify its use.