Required fields for specifying data products

[knoepfel]

The product_query data-product specification supports 5 fields:

1. Data product concept
2. Creator name
3. Data layer
4. Suffix
5. Stage name

Of those 5, we have intended 3 of the fields to be required—the data-product concept, the creator name, and the data layer.

We should discuss how many of the fields must be specified by the user.

[knoepfel]

Agreement:

1. Only the data-product concept will be unconditionally required (and that could be calculated by the framework)
2. It is a framework-graph initialization error if the product_query object resolves to multiple data products.
   1. We need to decide under which conditions the graph-formation step should favor the current process before looking into an input file, and when the current process and an input file should be consulted together.
3. Ambiguities can be resolved by specifying additional fields.
   N.B.: this implies that a working Phlex job may fail once a user decides to add another data product whose specification matches an existing product_query.

Further discussion:

• Should there be an option to validate that the input files satisfy a certain schema?
  • Should there be an option to consume a summary file (e.g., database) that describes an entire set of files?
• Will likely need utilities to query what is available in input files
• What algorithm-configuration options should be available whenever a data product is not present in the input?
• Must discuss when input files are opened as part of the graph-formation step
  • Do we want to look at only the first input file to define what the framework expects for the entire job?

[knoepfel]

Comment: It shall be an error for the same stage name to be specified for multiple jobs in the processing history (with the potential exception of recovery jobs).