Enhancements
Data View
Distinct values filtered by existing filters
When clicking on a column in a data view, the “Top 5” distinct values list is now filtered by any filters created by you in other columns. If you filter column A the top 5 distinct values of column B will only show only the values corresponding to the filtered values in column A.
Also, the “Top 5” and “Show more” windows have been merged into a single window.
Admin
New Add User button on Users page
It is now possible to add a new user directly from the Admin/Users page, in addition to doing this in a role. You can choose the user’s roles in one step rather than having to edit each role.
Loading
Latest 50 and batch history performance
The performance of the “Latest 50 batches” and “Batch History” pags should be much faster now.
Pipelines
New maximize editor button!
Press Ctrl+F11 or click the new maximize button in the top right corner of the pipeline editor to enter full screen.
Press Ctrl+F11 again to minimize it or press the minimize button.
Several debug mode improvements
Several improvements have been made to debug mode:
- Significantly faster and more responsive
- Now possible to view all pipeline tables at every step (previously only the tables modified by that step were visible)
- Always start on the first debuggable line
- Ensure that the line being debugged is selected
Prompt user to select multiple worksheets
You can now prompt a user to select multiple worksheets from an Excel file to be uploaded. This is useful if the worksheet names are not known in advance or change.
For example, create 2 input variables of type “userSelectedSource” and create 2 corresponding GetExcel commands which use the variables in TableName:
The data uploader can select the worksheets from dropdowns:
Pipeline Format XML preserves empty lines
The Format button now preserves empty lines rather than deleting them.
New AggregateTable command
A new AggregateTable transform command is available which aggregates data in a table using functions like SUM, COUNT, MAX, MIN, and AVG. The result either overwrites the original table or is output as a new pipeline table.
AggregateTable requires that the data type of the columns be set either by the new SetColumnDataType command (see below) or by setting StrictTypes to true in the GetMart and GetDB commands.
For more information see the AggregateTable command documentation.
New SetColumnDataType command
Strictly sets the datatype of one or more columns so that invalid values are not allowed even during pipeline processing. It also allows you to define a default value for null values (ie replace NULL by 0). By default, the datatype of all columns of in-memory pipeline tables are of type string so that invalid data can still be processed and reported back to the user. But using SetColumnDataType can simplify later operations in the pipeline that would otherwise require complicated casting and null-checking. This can greatly simplify the syntax of AddColumn.FillWithExpression and also allows data aggregation with the AggregateTable command.
For more information see the SetColumnDataType documentation.
New StrictTypes option in datatyped extractors
StrictTypes is a new option on the GetMart and GetDB commands which allows preserving column data types as defined in the data source. Setting StrictType to true can simplify use of AddColumn.FillWithExpression (by eliminating the need to cast) and enables the use of AggregateTable.
HierarchyLookup.FuzzyMatching option
FuzzyMatching is a new option on the HierarchyLookup command which causes the lookup to ignore special characters, accents and whitespace. The benefit of this is management of fewer synonyms.
Optional warning for unmapped columns
A new option is available on the ColumnMappings element to display a warning for every column found in the data source which is unmapped. This option can be useful during creation of new pipelines to know if you missed any columns. This is especially helpful if there are hundred(s) of fields!
SourceFilter in TableLookup and SysIDLookup
A new SourceFilter option makes it possible to define a source-side filter when using the TableLookup and SysIDLookup commands.
Convert date to UNIX timestamp
The FormatDate command now supports the ability to convert a date to a UNIX timestamp. This is possible by setting OutputFormat=”UnixTimeSeconds”.
Other changes
4.26.1 - 4.27.0
- #5277: DECIMAL value removed instead of rounded when too many digits
- #5251: Distinct values filter Url should handle large selections
- #5262: Data view Top values are not RLS filtered
- #5245: Custom SQL View - History diff display issue
- #5264: Custom view model - Cannot copy the SQL View Definition
- #5265: Data edit - changes are lost when user toggles Title/CODE
- #5260: Mart Model Download Include Related tables doesn’t include the related tables
- #2568: Incomprehensible error message for Invalid JSON Path
- #5250: Data view filter not working when Top values not loaded / Top values not loaded for form user
- #5226: User xxxxx does not have RLS permission to access batch ##### without RLS scope (JS_ERROR)
- #5227: Many “Object reference not set to an instance of an object.” errrors reported client-side (JS_ERROR)
- #5228: invalid_grant issues reported client side (JS_ERROR)
- #5256: dev CustomScript and some bugfixes
- #5267: MartLookup exceptionMessage and too many parameters error
- #5108: Hangfire change the delay from 1 minute to 15 seconds?
- #5109: Proxy error: display the actual HTML response so the hyperlink to open the ticket is clickable
- #5128: Can’t see long column names in Validation rules
- #5210: Prevent showing api/doc errors that we know about
- #5213: Batch issues - wrong pagination when ignored issues
- #5206: Load Auto bug when explicitly mapped table have non matching names
- #1562: Trim whitespace of catalog categories and search boxes
- #4791: Table level Validation Rules aren’t uploaded in Excel default model template
- #5207: Can’t click pipeline configure button when code is long
- #5199: Admin Role - Add list of users tweaks
- #5198: Error on Mart Import
- #4858: Support customizable Tag attributes in all lookup commands
- #5049: PostRun - RunPipeline validation does not filter inputs by origin
- #4621: Ability to expand subform table
- #5169: Cleanup optim - store mart DB’s UUID at creation time
- #5183: Batch preview - actions buttons not visible if comment too long
- #4967: Trying to Delete a record which has already been deleted causes the batch to fail.
- #5142: User cannot start a batch of a published pipeline because origin.IsFileBased is null (export - from 4.26 not retrocompatible with 4.25)
- #4780: Can’t easily view far right columns in model
- #5117: Require button press to search batch history
- #5135: Log instead of display any client-side system error messages
- #5141: Validation Error message aren’t very helpful
- #5144: Mart deploy batches shown as created by “xAdmin” instead of real user
- #5120: POLIS Feedback - hide tags not related to filtered pipelines / tables
- #4846: Forms Hidden fields to be flushed at save
- #5148: Upgrade to angular 17
- #5157: Mart Pipelines validation is running too often and blocking some batches
- #5149: No SEC_PRINCIPAL found for account of ‘XSVC_TRAINING_AH
- #5164: Add origin count to home page stats
- #4705: Model - Select particular tables improvement
- #5125: Batch issues resolution bug on MartLookup with multiple columns
- #5122: Forms - ability to add a filter to the underlying data
- #5127: TestValueInRange still done if limit field is blank
- #5137: Data Catalog - Uncheck all, select a few categories not persisted in storage
- #5131: MartLookup fails when matching nulls values against a uniqueidentifier column