Category: Spotfire Tricks

How to Add Lines to a Probit Plot with IronPython

A few weeks ago, I wrote a post detailing how to create a multiple variable probit plot.  This post improved upon an older post on creating a single variable probit plot.  Part of those instructions included adding several “supplemental” lines via Lines and Curves like P10, P90, and the Median.  This is actually the most time-consuming part of the process.  Each line must be added one by one.

Today, while reviewing my instructions, I realized I had to do better. I know this can be done with IronPython! A quick Google search pulled up this TIBCO community post that I was able to use as a guide. I modified that script to work for my probit plot use case. Now, I have a piece of code that will add all of those lines and is easily modifiable and scalable.

The Code

Here is what the code looks like in my DXP.  I made the following modifications from TIBCO’s original:

  1. Changed BarChart to ScatterPlot to suit my visualization
  2. Modified the expressions from an average to the Percentile, P10, P90, and Median.
  3. Added code for vertical lines.

Code for Copy & Paste

from Spotfire.Dxp.Application.Visuals import *

scatterPlot = sp.As[ScatterPlot]()

#Add Horizontal Straight Line
horizontalLine1 = scatterPlot.FittingModels.AddHorizontalLine(‘P90([Y])’)
horizontalLine2 = scatterPlot.FittingModels.AddHorizontalLine(‘P10([Y])’)
horizontalLine3 = scatterPlot.FittingModels.AddHorizontalLine(‘Median([Y])’)
horizontalLine4 = scatterPlot.FittingModels.AddHorizontalLine(‘Percentile([Y],20)’)
horizontalLine5 = scatterPlot.FittingModels.AddHorizontalLine(‘Percentile([Y],30)’)
horizontalLine6 = scatterPlot.FittingModels.AddHorizontalLine(‘Percentile([Y],40)’)
horizontalLine7 = scatterPlot.FittingModels.AddHorizontalLine(‘Percentile([Y],60)’)
horizontalLine8 = scatterPlot.FittingModels.AddHorizontalLine(‘Percentile([Y],70)’)
horizontalLine9 = scatterPlot.FittingModels.AddHorizontalLine(‘Percentile([Y],80)’)

#Add Vertical Straight Line
verticalLine1 = scatterPlot.FittingModels.AddVerticalLine(’10’)
verticalLine2 = scatterPlot.FittingModels.AddVerticalLine(‘100’)
verticalLine3 = scatterPlot.FittingModels.AddVerticalLine(‘1000’)

Detailed Steps

  1. Add a Text Area to the page, right-click, select Edit HTML.
  2. Click the Add Action Control button.
  3. Name the button.
  4. Click the Script button.
  5. Click the New button.
  6. Name the script.
  7. Copy and paste code.  Modify to suit.
  8. Add a parameter called “sp” and connect it to your visualization.
  9. Run script to test. Click OK to close on script window.
  10. Modify the HTML as shown to hide the button.  You don’t want to click it again.

Caveats

  1. Once you run the script, it does not need to be run again.  When you clicked Run Script the first time, 13 lines were created.  Clicking again will create another 13 lines.  I made this mistake when testing.  Then, I had to delete a ton of lines one by one! (Please upvote my Idea to allow users to delete more than one line at a time).
  2. The script creates the lines, but you still have to edit them one by one.  This might also be possible with IronPython, but I haven’t dug that far yet.
  3. If you copy and paste from my code snippet above, you’ll need to replace the quotes.  Spotfire won’t recognize them correctly from copy and paste.

This should make setting up probit plots just a little bit faster.  You can also modify this code any time you want to add multiple lines to a different visualization or another probit plot.

Spotfire Version

Content created with Spotfire 7.12.

Guest Spotfire blogger residing in Whitefish, MT.  Working for SM Energy’s Advanced Analytics and Emerging Technology team!

Spotfire Best Practices

This post is a follow-up to the 7 part series I wrote on Decomposing Spotfire Projects.  The idea first came to me when I documented a large enterprise project. I didn’t build it. It wasn’t my baby, but it was important to the company. Management wanted it documented in case the author moved on. I also think about this subject every time I build a project for a user. What’s the best way to go about helping the user get a solid grasp on the project?  Is there a formulaic way to decompose a Spotfire project or at least a good order of operations.  Decomposing a project you didn’t build is often quite difficult and can seem like trying to assemble a jigsaw puzzle with 1,000 pieces.  Where do you even start?

So, I broke it up into 7 pieces.  As I wrote the series, I realized I was also writing a bit of a best practices guide or a do’s and don’ts guide for project development.  In an effort to help my readers, I thought a summary post would be helpful.  Thus, this post summarizes 3 – 5 pieces of development advice from each part in the series.  Keep in mind, I am not going to elaborate in great detail.  All the info you need is in each post.  Use this to jog your memory.

Data Tables & Data Sources

  1. Use a naming convention.
  2. Delete unused tables, connections, and data sources.
  3. Limit data as much as you can.  Less is more.

Data Functions

  1. Leave comments in your code to explain what the code is doing in case someone has to modify it later.
  2. Provide a general description of what the data function is doing in the description section.
  3. Include code that will install and attach any required packages.

Data Wrangling

  1. Clean up your joins.  Don’t join to the same table over and over again.  Delete duplicates and have one join and only one join from table to table.
  2. Architecture is the single most important component of any project.  Put time into planning it out.  Don’t just start building.
  3. Document your architecture choices.  Include information on not just what you did but why you did it.

Document Properties

  1. Delete unused document properties.
  2. Document what each document property should influence or control. l
  3. Use a naming convention.

Columns & Calculations

  1. Delete unused calculations or columns.
  2. Add a description to the calculation if it gets complex.
  3. Use naming conventions.
  4. Use Exclude columns transformations to exclude any columns that aren’t needed.

Text Areas & Scripts

  1. Don’t copy and paste from Word into a text area.  Just don’t do it.
  2. Include descriptions in scripts and data functions that explain where they are used or what they impact in the project.
  3. Use the text area to explain how the user should move thru a workflow or text area.
  4. Use HTML and CSS.  That’s not really a best practice per se, but learning even a little bit of HTML will make text areas so much better.

Visualizations & Data Limiting

  1. Remove the unnecessary.  Hide column selectors.  Only show what a user needs to see in a legend if a legend is even needed.
  2. Make data limiting as visible as possible with legend items or naming conventions.
  3. Minimize usage of custom expressions in visualization properties.
  4. Don’t put too many visualizations on a page.  Four is usually the limit.
  5. Articulate the business question you are trying to ask and answer with visualizations.

Conclusion

Now, I know some of these may seem super obvious, but I never cease to be amazed at what people create or leave behind.  Please take these to heart.  The developer that comes after you will be thankful.  

Guest Spotfire blogger residing in Whitefish, MT.  Working for SM Energy’s Advanced Analytics and Emerging Technology team!

The Most Important Thing You Don’t Know About How Spotfire Builds Tables

  • After you insert columns, the Spotfire Source Information tab describes the columns ignored rather than the columns added. Why is that?
  • Did Spotfire add columns to your project? Are you seeing columns in your project that you don’t recognize?
  • Do you have duplicate columns in a table, and you have no idea how they got there? Have you tried to delete them, and they keep coming back?

Read More

Guest Spotfire blogger residing in Whitefish, MT.  Working for SM Energy’s Advanced Analytics and Emerging Technology team!

Accessing Data Function Parameters

I had one of those moments recently, where a dialog box I’ve stared at for years presented something new to me.  In this case, it was the Settings button in the Data Table Properties dialog.  Now, this button will be grayed out for all tables except on-demand tables.  At least, that is what I thought.  As I scrolled thru my tables, the box appeared active for a table I knew was not configured on-demand.  Rather, a data function built the table.  Clicking on it then pulled up the Edit Parameters dialog.  When investigating tables, it’s helpful to be able to access data function parameters from Data Table properties.

 

Clicking on Settings pulls up the Edit parameters dialog for any table created by a data function.
This allows the user to access data function parameters without going thru the Data Functions dialogs.

Spotfire Version

Content created with Spotfire 7.12.

Guest Spotfire blogger residing in Whitefish, MT.  Working for SM Energy’s Advanced Analytics and Emerging Technology team!

IronPython to reset all tags

For a recent project, we wanted the user of a Spotfire template to be able to reset all the tags, including both built-in tags and tags the user may have added, with a single click.

One solution is to use an IronPython script to reset each tag collection in the template, keeping in mind that, if we hard-code the names of the tags to reset, we won’t be able to handle cases where users have added custom tags. In order to reset tags that may be added later on, we can use Spotfire’s IronPython API to identify and reset tag columns on the fly.

Here is what the code looks like in Spotfire:

And in more copy/paste-friendly format:

from Spotfire.Dxp.Data import *
from System import Array

for table in Document.Data.Tables:
    for column in table.Columns:
        if str(column.Properties.ColumnType) == “Tags”:
            tagCollectionName = column.Properties.Name
            tagColumn = column.As[TagsColumn]()
            tagNames = Array[str](tagColumn.TagValues)
            currentTable.Columns.Remove(column)
            currentTable.Columns.AddTagsColumn(tagCollectionName,tagNames)

Notes on the code:

This script loops over all the tables in the Spotfire analysis and checks the type of each column. When we locate a tags column, we extract the name of the tag collection, as well as any associated tags, drop the column from the data table, and create the tags collection anew. We can use this approach to reset all tags with one command because, by default, all datapoints are untagged when a “new” tag collection is created. The script will reset all tags in the analysis, including custom tags added by the user, and it will simply exit quietly if there are no tags. There are no input/output parameters for this script, but we could use input parameters to apply the same tag-resetting logic to a specific subset of the data.

Spotfire Version

Content created with Spotfire 7.12

Data Science Intern at Ruths.ai and  Applied Physics PhD candidate at Rice University

How to Pass Marked Data into a Document Property

Consider this scenario, you have a calculation that necessitates the ability to pull a value from marked data. This is a complicated feedback system.

Setup the components

First, let’s setup the moving parts:

  • Create a document property ${sumMarkedRows}. This is going to be the document property that we’ll use in our calculation.
  • Created a text area. We need this so that our JavaScript can pass the data to the document property.
  • In the text area, add a Calculated value that is Sum([Column]).
  • Pick a marking layer that will facilitate this connection.

Connect the parts

Next we’ll need to pass that data from the marked layer to the calculated value to the JavaScript to the document property. What a mouthful!

  1. Create an input field for ${sumMarkedRows}
  2. Create a JavaScript that is called when the calculated value changes so we can pass it to the document property:

    <script>
    $(‘theIDoftheSpotfireCalcValue’).on(‘DOMSubtreeModified’,function(){
    $(‘theIDoftheInputfield’).text($(‘theIDoftheSpotfireCalcValue’).text());
    $(‘theIDoftheInputfield’).blur();
    })
    </script>

  3. That script needs the IDs swapped out then you can put it in the HTML area of that text area.
  4. Use ${sumMarkedRows} in your calculated column as the value that changes
  5. Wrap all of that with <div style=”display:none”>   …all the html…  </div> to hide that trickery

Technical Director at Ruths.ai

A Different Way to Build Drop-Down Property Controls

  • Does adding new columns to a data table “contaminate” your drop down property controls?
  • Would you prefer to not have to create column properties for drop-down controls in addition to the document property?
  • Would you like to learn a property control hack?

Read More

Guest Spotfire blogger residing in Whitefish, MT.  Working for SM Energy’s Advanced Analytics and Emerging Technology team!

How do you set colors in Spotfire without reverting back to defaults?

  • Would you like to “set” colors to be applied to all visualizations in a DXP?
  • Have you set the colors for each unique value in a column but find the colors get reset to the default?
  • Are you using property controls to change the Color By variable?

Read More

Guest Spotfire blogger residing in Whitefish, MT.  Working for SM Energy’s Advanced Analytics and Emerging Technology team!