New Features

Row-level security (RLS)

Row-level security (RLS) permits user access to some rows in a table or view while denying access to the other rows. “User access” in this case could mean viewing data, loading data using custom pipelines, editing data (in UI or via export-import), form-based data entry and/or consuming the OData API.

RLS was designed to allow multiple countries to store data in the same table, while not being able to see or change data of other countries. But it is generic in nature and can be used for any similar scenario (security by major office, security by org unit or team, etc.).

The row-level security how-to article provides full details on how to set it up and use it.

Enhancements

Data catalog: display objects the user can access

To improve the usefulness of xMart’s data catalogs, by default, users now only see views and tables which the user has access to. Protected (private) tables and views are hidden unless the “Show protected views” checkbox is checked. Checking this box reveals the metadata (titles, descriptions and tags) of all objects for discoverability purposes, but does not mean that the user can access the content of those tables.

Export staged data

It is now possible to download staged data during batch preview. Staged data continues to be downloadable until it is removed from the server (~ 3 days after batch commit).

export staged data

Exported staged data includes all data columns in the format of the target table, plus a number of system columns that indicate the result of the analysis performed by the system for each record.

Please consult the export staged data how-to article for more information and details on the system columns included in the export.

export staged data

Process multiple files in a zip file using GetZip

Not really new behavior, but now the pipeline script guide explains how to process multiple files in a single zip file using GetZip. See Example 3.

GetZip Command

Multiple files with GetZip

As usual, use the file-specific Getter inside GetZip, ie GetText, GetExcel, etc.

Extract data from older EpiInfo Rec files: GetRecfile command

Older EpiInfo Rec files can now be processed using the GetRecFile command.

RecFiles contain both the data as well as the structure of the data and uses a unique fixed-width format.

GetRecFile can be used inside the GetZip command.

If you are curious, a peek at the internal structure of a .REC file:

RecFile structure

Send email upon approval failure

When a batch is rejected during the approval process, an email notification is sent to the person who created the batch.

Very large files can be exported/downloaded in user interface

Previously, exporting very large tables/views (> 500 Mb) might fail (“stream too long” error). This has been improved so that even 2 Gb or larger files can be exported. Larger files will now automatically be downloaded as a zip file while smaller files continue to be downloaded uncompressed so they can immediately be opened by desktop applications without (ie Excel) without the need to unzip first.

Faster REPLACE mechanism when whole table is being replaced

When using the REPLACE load strategy in a pipeline to replace the entire table (a common scenario), the performance has been improved.

Other changes

  • #3995: Hide Edit Model button user has no permission
  • #3988: Data edit: user cannot clear a DATE field
  • #3906: Prevent duplicate row counting
  • #3879: Intra-batch FK fails with implicit sysidlookup and auto column mappings
  • #3986: Batch Preview optim Sort by BPK
  • #3990: Hotfix/uat refmart optim: FillStrictTable by chunk
  • #3809: Improve Data Catalog page load time
  • #3911: Date field is always required even though populated (manually)
  • Disable rollback if table is referenced by others table
  • #3907: Counting rows error even though rows are counted
  • #3890: Catch this special invalid xml error in pipeline
  • #3895: Sys Admin Import error message isn’t very useful.
  • #3896: Mart pipeline import broken because of MART_VARIABLES missing Type (SQL and pipeline already hotfixed)
  • #3882: Update filtering by TABLE_CODE + UI glitch fix
  • #3934: GetWebService can’t handle gzip compression