The other day, I attended a client’s Spotfire User meeting as Tibco representatives unveiled and highlighted new features in Spotfire X. One specifically caught my eye: the ability to filter as a transformation, before loading data. This can be a huge time saver as the whole table will not need to load at start up. You can only load what you need for that analysis.
Just that morning, I worked with a client, trying to filter data via the Data on Demand feature but it wasn’t behaving. Frankly, I’ve always felt the feature a bit clunky. But, now, there is no need to fight the Data on Demand interface or wrangle SQL filters in the Information Designer. Of course, you can if you feel comfortable in those spaces; however, most users will love this very straightforward and intuitive method of filtering as a transformation.
You no longer need to ask a database expert to limit your view. You can do it yourself. To see how, read on…
First, go to the Data in analysis icon, pick your column, then set your filter. This method was available via the Data Panel in previous Spotfire versions. However, now you can right-click on the filter window and select Create Filter Transformation.
And, you’re done.
Next time you open the analysis, only the data you filtered to will load, reducing your load time to a fraction of the previous duration.
If you want to get creative with your filters, you can also use a custom expression to filter as a transformation. To do so, go to the Data canvas (), select your table from the flow diagram, then click the plus symbol under the table name at the bottom of the page. Then, choose Add transformations.
Once you see the window above, choose Filter rows in the Transformations dropdown, then Insert.
You will see the familiar custom expressions window in which you can type a limiting expression.
This expression serves just as the Limit data using expression feature in the Data tab of a visualization. Limit only to certain categories, remove certain categories, etc.
Except now the limiting applies to the data for the whole analysis and works as a transformation, before data is loaded.
Filter as transformation. So easy. So powerful.
Jason is a Data Scientist at Petro.ai with a master’s degree in Predictive Analytics and Data Science from Northwestern University. He has experience with a multitude of machine learning techniques such as Random Forest, Neural Nets, and Support Vector Machines. With a previous Master’s in Creative Writing, Jason is a fervent believer in the Oxford comma.