Sympathy for Data v1.3.0 available!


Sympathy for Data version 1.3.0 offers several new and prominent features such as generic types, higher order functions and much improved support for linked subflows.

Many small improvements were made to the standard node library. Nodes will often cope better with empty input data and deliver informative, but less detailed, feedback.

Nodes from 1.2.x should be compatible with 1.3.0 but there are new, more succinct, ways of writing nodes for 1.3.x that are not backwards compatible with 1.2.x. When writing new nodes, consider which older versions of the platform that will be used.


Download Sympathy for Data built for Windows:


Changes Compared to version 1.2.5

New features:

  • Generic types
  • Higher order functions: Lambda, Map and Apply
  • Official, and much improved, support for linked subflows
  • Official support for locked subflows
  • New library structure using tags

New nodes:

  • New generic versions of all list operations
  • Ensure columns in Table
  • Either with Data Predicate
  • Extract lambdas: builtin node for reading lambda functions from existing workflows

User interface:

  • Right-click on an empty part of the flow to insert higher order functions
  • New command in context menu for inserting a subflow as a link
  • Improved file dialogs in node configurations by using native dialog when asking for an existing directory and starting file dialogs from currently selected file path

API changes:

  • Simpler APIs for writing nodes
  • Improved slicing of (sy)table with slice object with negative or undefined stride
  • Automatically update order, label, and description for parameters when the node’s definition changes
  • NodeContext is no longer a named tuple
  • Added new method: NodeContext.manage_input. A managed input will have its lifetime decided outside of the node
  • New method in ADAF API: Group.number_of_rows()
  • Configuration widgets can expose a method called save_parameters which is called before the gui is closed
  • Added API (parameter helper): List parameter widgets emit valueChanged

Linked/locked subflows:

  • Include subflows relative to path of parent flow, not relative to root flow. This affects where sympathy searches for linked subflows inside linked subflows and should hopefully feel more natural than the old system
  • Allow opening of flows with broken links
  • Import and export nodes can now be used inside locked subflows and lambdas
  • Made it impossible for flows below a locked flow to themselves be locked
  • Improved abort for locked subflows

Node changes:

  • Report framework: histogram2d graph layer is now called heatmap and can handle different reduction functions (count, mean, median, max, min)
  • Improved CSV import. Can now handle empty input, and input with only one row, with or without trailing newline
  • Improved XLS(X) import/export. Especially handling of dates, times, and mixed types. Cells formatted as Time are now imported as timedeltas
  • Renamed Sort Table(s) to Sort rows in Table(s)
  • Calculator: chooses columns case-sensitively on Windows too
  • Calculator: Show number of output rows in preview in calculator gui
  • VSplit Table: Removed constraint that the index should be sorted. The elements will be grouped by the first occurrence of each unique value
  • Convert columns in Table: Added useful conversion path between datetime and float
  • Select columns in ADAF with Table now works as expected when “Remove selected” has been checked
  • Select rows in Table with Table: offers a choice of reduction function between rows in config table. Previously it only read first row of the config table
  • Slice ADAFs/Tables: Basic integer indexing now works as expected
  • Improve interpolation nodes’ handling of one sample signals
  • Report Apply nodes output datasources to created files
  • Improved CSV import: Handles empty csv files and files with errors towards the end. It also features a new option for how to handle errors importing a file. Header row has been made independent of the other input boxes, and no longer affects the data row. When read to end of file is selected, the number of footer rows is ignored. Delimiter detection was improved
  • Fixed issues with nesting of higher order functions (Map, Lambda and Apply)
  • Improvements to reporting: Improved bin placement and x-axis extent of 1d histograms. Automatically set axes labels from data source if they are empty. Added option “Lift pen when x decreases” to line graph layer. Added vline layer in reporting tool.
  • Several nodes are better at forwarding attributes, table names, etc. to output Slice data Table, Select columns in ADAF(s) with Table(s), Unique Table(s), ADAF(s) to Table(s), Select rows in Table(s) with Table, Interpolate ADAF(s), and Rename columns nodes
  • Many nodes are better at handling missing or incomplete input data: Filter Rows in Table, Create index nodes, Search and replace values in Tables, Detrend ADAF(s), Select Report Pages, ADAF(s) to Table(s), Scatter nodes
  • Added ‘calculation’ attribute on all output columns from calculator node
  • Export Tables and Export Datasources create missing folders
  • Fixed Export Texts

Other improvements:

  • Added default workflow environment variables SY_FLOW_FILEPATH, SY_FLOW_DIR and SY_FLOW_AUTHOR. All flows have these and they can’t be set or deleted
  • Subflows can define workflow variables. Each subflow specializes the variables of its parent flow, so that the parent flows vars are accessible in the subflow but not vice versa
  • Improve performance by skipping validation of any nodes that don’t implement verify_parameters
  • Improve performance by changing compression settings for sydata files, compression is faster but compresses slightly less
  • Pretty print workflow xml files, making diffs possible

New requirements:

  • Requiring pandas version 0.15 for the CSV import, for versions before 0.15 down to 0.13 it will still work but may behave slightly differently in edge cases with blank rows