New Features
Centralized business rules
The ability to attach custom business rules directly to tables is a new feature. Custom business rules have been available in pipelines since the first version of xMart 4 (tutorial on pipeline business rules and script guide reference on business rules and they can also be defined in data entry forms, but now it is possible to define them “centrally” by attaching them directly to a table in your model.
When business rules are attached to a table rather than a pipeline or a form, they are applied to all pipelines and forms that target that table: they only need to be defined once and regardless of how data enters the table, the business rules will be respected.
This means that you can define business rules just one time and they will be automatically applied in data entry forms, gridview edits, importing data on the data view page, and when running custom pipelines.
If you are not familiar with custom business rules, one example is “Test Value in Range”, where incoming data can be compared to literal values (between 0 and 100) or to values in other columns in a row (DATE_INVESTIGATION must be greater than DATE_ONSET).
A new custom user interface has been created for the Test Value in Range rule, which is the most common rule.
Please see the new how-to article on centralized business rules
Data driven column renaming - RenameColumnsByLookup
Sometimes there is a need to rename many columns in many source files. One way to do that is with large numbers of the RenameColumn command in a pipeline. This can make a pipeline very long. A new RenameColumnsByLookup command has been added which takes a data-driven approach. The command renames columns based on mappings in a lookup table.
Managing a lookup table like this can be easier to maintain than many RenameColumn commands because you can simply prepare or update the table in Excel or by editing in the UI - without touching the pipeline. The lookup table can also be used by multiple pipelines.
For more information, see the RenameColumnsByLookup command reference documentation.
Enhancements
Mart health home page summary
A detailed mart health page has existing in the Admin area for a long time but now a summary of mart health is displayed on every mart’s home page. Mart health indicates broken pipelines and broken custom views. Click the Explore button to open the detailed mart health page. The summary is visible to mart administrators only (MART_MANAGE permission).
Data URLs are hyperlinks on data view page
On the data view page, any value in a table or view which is a URL is now a clickable hyperlink.
Use left and right arrow keys to edit data in gridview
(Annoyance enhancement) When editing data in gridview mode, the left and right arrow keys now move 1 character at a time within the cell being edited, rather than moving to the next or previous cell.
Open a mart with keyboard only
In the “marts” widget on the top-level home page, you can now open a mart using only the keyboard.
- type some letters to begin filtering the list of marts
- press tab until focused mart is the wanted mart
- press enter to open the mart
Documentation - examples of lookup commands
Previously there were no examples in the script guide for any lookup command, like MartLookup, RenameColumnsByLookup and HierarchyLookup. These have been added so the user may get a better understanding of how lookup commands work.
Admin User page improvements
- Added a search box to make it easier to find a user
- When a user is selected, the users’ roles that appear on the right are now hyperlinks so you can directly open the role
- The users’ roles are always visible, even if the user list is long.
New LAST_BATCH_SUCCESS_TIME pipeline system variable
A new variable called LAST_BATCH_SUCCESS_TIME is now available in pipelines. It’s value is the ISO time stamp of the last successful batch of the origin. It can be used in filter clauses of data extraction commands (GetDb, GetMart, GetOData, etc.) to limit retrieval to records which have been modified since the last successful batch of the origin. It requires of course source data which has a field indicating the last modified date.
Refer to the description of system variables in the How-to Use Variables article.
Home page “xMart in figures” improvements
The home page “mart in Figures” summary has been improved for each mart’s home page, as well as for the xMart system home page:
- Added # connected apps that counts the “users” which are not people but connected applications (ie Client ID values). The connected apps count has also been removed from the “active users” count, so that “active users” only refers to people.
- Added # public API objects that counts the number of objects (tables or custom SQL views) which are available via the public OData API.
PivotColumns performance improvment
The performance of the PivotColumns transformation command (long to wide) has been substantially improved. What used to take 4 minutes now takes 1 second (360 fold improvement)!
Hide already assigned Permissions from Add Permissions Drop Down
Assigning permissions to a role will be a more clear process because the already assigned permissions will not be included in the drop down list.
Column analysis - filter and/or sort
In the analysis of a columns values, we added an auto-complete filter textbox and a sorting icon, so the users may filter and sort the values they see according to their needs. The new feature can be accessed by clicking Analyze/Show more in the columns options popup.
Here is a screenshot of the new feauture.
Batch notification w/wo data changes
Batch notification has been improved with 3 possible options rather than 2:
- all completed batches (new)
- only completed batches with data changes
- only completed batches without data changes
Re-use pipeline file and inputs when testing
When running a pipeline in Test mode, a checkbox has been added to re-use the uploaded file and inputs from the previous test run. So from now on, users will need to upload their source file and enter any batch inputs only once when testing.
Export all issues with message and styling
Exporting issues on the batch preview page now includes 4 new columns: the warning/error message, the failed value, the name of the table and the name of the column. Also, the styling of the export has been improved.
Display user column name instead of “[Sys_RowTitle]” in message to user
When a look up fails, in the displayed message to a user, the code of the field will be displaed instead of the [Sys_RowTitle].
For example before 4.23 the displayed message was like this:
In this scenario the column CODE is the RowTitle for the REF_SAMPLE_TYPES table as you can see below.
Therefore, from 4.23 onwards the message will be like this: “This value ‘MULTIPLE’ was not found in the column(s) ‘CODE’ of table ‘REF_SAMPLE_TYPES’”
Remove LineNumber and LinePosition in pipeline debugger UI
In order to make the pipeline debugger more user friendly, the LineNumber and LinePosition attributes were removed from the displayed pipeline script.
The first image below shows how it used to be and the second shows how it will be from 4.23 onwards.
CustomScript: more helpful validation
From 4.23 the pipeline validation process detects C# syntax errors in the CustomScript command. This allows you to correct custom script errors even before running the pipeline.
Improve custom SQL view error messages
Previously, when a custom SQL view had an error, “Something went wrong” was displayed. Now the exact information of what went wrong is displayed, so the user will have a clearer understanding of any issues in their custom SQL views.
Other changes
4.23.0
- #4192: Error validating pipeline with missing GetWebService.Getter
- #4309: “Drag or browse” origin box should only display file based origins
- #4298: Hangfire error on approval timeout if the batch was cancelled beforehand
- #4279: Table not visible (not allowed) after renaming
- #4384: Extra column in FormatDate command example
- #4401: Forms list on mart home page not responsive to title/code switch
- #4439: Transaction deadlock while inserting input file for data entry batch
- #4269: Maintenance Mode Batch not working with multiple web instances (centralize some web.config in DB)
4.22.14
- #4430: Feedback about messages regarding RLS and batches
- #4428: Why do these batches fail? -> SQLException in async void function
- #4432: Problem with UnpivotColumns (actually with DB constraint when loading same SourceTable twice)
4.22.13
- #4422: RLS not working when user is in 2 roles
- #4420: RLS: User can see data other users uploaded for any country through batch history using the same origin
4.22.11
- #4367: PivotColumn tries to recreate column even if it already exists
- #4381: Use title/code selection in the “Figures” panel title on the home page
- #4389: Regression Home page figures not appearing
- #4379: Re-checking empty columns issue
- #4382: Buttons in Model Page, not visible
- #4385: Batch preview issues lookups resolution + handle overflow in origins dropdown list
4.22.10
- #4370: Data entry form “display all details” not working
- #4366: Health check invalid pipelines spinning
- #3090: Batch preview page not refreshing (auto refresh every 8sec)