Final omnibus updates#66
Conversation
(1) Numerous small wordsmithing changes. (2) Split messenger and messenger_pdgid as agreed with Semantics. (3) Moved the extension table summary table to the end of the section that defines the table entries. (4) SIgnificant rewrites in section 6.
bkhelifi
left a comment
bkhelifi
left a comment
There was a problem hiding this comment.
All good for me, thanks a lot @iannevans
|
Thanks for the many modifications and hard work to homogenise the text. What is now section 6 is quite important to the discussion that will come after endorsement, so it should not be discarded. As it is now, it is truncated and limited to DataLink usages, while the concept of "specific tables" is also important to discuss further data discovery in a more generalised way. I am sure we can find a good formulation to express this solution in a paragraph. I however understand that the content of such a specialized table is not yet settled and requires more science cases. The proposition for a specialized response-function table as it is now could thus simply appear as an appendix. All this will then be discussed at the DM WG level. About dataproduct_type = “measurements”, it is already planned to avoid its usage, so this paragraph can be removed. It is indeed written in the product-type vocabulary: "Because of its lack of specificity, this term should generally be avoided, and new, more precise terms should be introduced instead." (see https://www.ivoa.net/rdf/product-type/2026-01-15/product-type.html#measurements). |
The "access section" was written by Mireille and myself. I mainly concentrated on DataLink subsection and Mireille on the specific description tables but we are both responsible of the whole due to extensive cross-reading before the pull request. I apologize that we were not able to propose it before interop, but it was admitted several weeks before interop that such a section will come and was needed. During the interop there was several announces that such a section was close to be posted and this didn't seem to be an issue to wait for it. About the content : there is nothing new there. These ideas have been presented several times in emails, issues, Pull request comment or even in the case of Mireille in previous interop presentation. I remember writing an email in April which had almost the same content as the subsection on DataLink of last Mireille's PR. I also presented some High energy examples in my DAL talk about DataLink explicitly acknowledging the discussions in High Energy Interest Group. A compromise such as "various discovery and access methods are possible and are in the hands of data producers" seems to be reachable. But to do this we have to review these methods somewhere in the document. The data variability and the use cases show so different requirements that we have to browse somewhere what can be done. The whole section including the inputs by Ian and the specific table subsection could become an appendix such as "data access implementation note" as suggested by Mathieu . I still plan to make a couple of comments on Ian's PR. |
Updated PR 66 based on feedback from contributors and editors.
There was a problem hiding this comment.
Here are some comments about the data product types.
I think it is important that the text reflects the semantics strategy we agreed on, which is
to propose 3 vocabularies and allow some extensions for new terms namely in advanced data products .
These data products are also considered in the radio domain, so identifying the need of a new data product vocabulary for advanced data products in this document is important .
see Advanced data product types for radio observatories
| If a data collection consists primarily of individual observations, each with a small set of associated data products such as \glspl{IRF} that are required for further data analysis (this is a very common use case), then either direct access to the {\bf hea-event-bundle}s or access via DataLink is very likely preferable. | ||
|
|
||
| However, if the data collection includes data products such as spectra, light-curves, or advanced data products ({\em e.g.\/}, aperture photometry probability density functions) that are extracted from the primary {\bf hea-event-list} and that can be used independently of the primary dataset, then direct access to these products is likely preferable from the user's perspective. Since \gls{HEA} event lists typically simulateously encode spatial, spectral, and temporal information, a very large amount of data may be associated with a single {\bf hea-event-list}. For example, roughly 4,600 individual detectable X-ray sources are located in a single Chandra observation field of view centered on the Galactic center. Using DataLink to the primary {\bf hea-event-list} to access these spectra, light-curves, and advanced data products would be unwieldy, especially when only a small subset of the products are required. | ||
| However, if the data collection includes data products such as spectra, light-curves, or advanced data products ({\em e.g.\/}, aperture photometry probability density functions) that are extracted from the primary {\bf hea-event-list} and that can be used independently of the primary dataset, then direct access to these products is likely preferable from the user's perspective. Since \gls{HEA} event lists typically simultaneously encode spatial, spectral, and temporal information, a very large amount of data may be associated with a single {\bf hea-event-list}. For example, roughly 4,600 individual detectable X-ray sources are located in a single deep Chandra observation field of view centered on the Galactic center. Using DataLink to the primary {\bf hea-event-list} to access these spectra, light-curves, and advanced data products would be unwieldy, especially when only a small subset of the products are required. |
There was a problem hiding this comment.
The last sentence is strange for me . Spectra and light-curves are available directly by Obscore table already as data products from the existing vocabulary. There might be some misunderstanding about the necessity to discover products from the eventlist.
Any column of the Obscore table can be used to discover a spectrum : obs_id , obs_publisher_id, position, energy band, time interval, etc..
we have use-cases with light-curve discovery in appendix A.
There was a problem hiding this comment.
The last sentence is strange for me . Spectra and light-curves are available directly by Obscore table already as data products from the existing vocabulary. There might be some misunderstanding about the necessity to discover products from the eventlist. Any column of the Obscore table can be used to discover a spectrum : obs_id , obs_publisher_id, position, energy band, time interval, etc.. we have use-cases with light-curve discovery in appendix A.
For HEA the data product that reflects what is seen by the detector is the event list. Unlike optical and similar wavebands, our other observation data products are all extracted from the event list. We don't have images, spectra, light-curves, etc. that are the direct output from the detector. So images, spectra, light curves, and advanced data products such as pdfs, draws, ..., are all the same to us - just different extractions from the original event list. And an event list can contain a huge amount of data - a typical Chandra exposure as a fully pixelated 4-D hypercube would include O(1E14) voxels (very, very sparsely populated). A deep event list might include tens of millions of photons, and since we can detect X-ray sources with as few as ~5 source photons that means an individual event list might include thousands of individual X-ray sources from which we can extract spectra, light-curves, aperture photometry pdfs etc. from the single event list. But using DataLink to attach these data products to the original event list would mean we might have thousands of spectra linked to the event list, thousands of light-curves linked to the event list, thousands of aperture photometry pdfs linked to the event list, thousands of draws linked to the event list, etc. And that's not a good way to access these datasets. Even though the spectra and light-curves can be accessed independently, if the pdfs and draws can't, then there is still a problem using DataLink for the end users who need to access those products. The point here is that the DataLink approach doesn't work effectively in cases such as this.
| \subsubsection{Datalink Access Using a Service Descriptor} | ||
| \label{sec:datalinksd} | ||
| To avoid preventing a direct access to the {\bf hea-event-list} while keeping the explicit access to the various response functions, it is possible to add a ``DataLink'' service descriptor in the ObsTAP result as can be seen in the example below. | ||
| To avoid preventing a direct access to the {\bf hea-event-list} while keeping the explicit access to the various response functions, one can add a ``DataLink'' service descriptor in the ObsTAP result as shown in the example below. |
| \noindent Note that the referenced ID field is not required to be {\em obs\_publisher\_did\/}. It can be {\em obs\_id\/}, or any free identifier attribute, allowing grouping of several rows together. For example, referencing {\em obs\_id\/} will allow all the datasets belonging to the same observation to be bound to the same \blinks\ endpoint response. | ||
|
|
||
| This method is implemented in many archive services such as ESO, CADC, GAVO, HEASARC and even VizieR. | ||
| This method is implemented in many archive services such as CADC, ESO, GAVO, HEASARC, and VizieR. |
| <GROUP name="inputParams"> | ||
| <PARAM name="ID" datatype="char" arraysize="*" value="" | ||
| ref="referred_ID_FIELD"/></GROUP> | ||
| ref="referenced_ID_FIELD"/></GROUP> |
| } | ||
|
|
||
| In this case the service descriptor provides the root URL to the DataLink \blinks\ endpoint. The full \blinks\ endpoint URL for a specific row (or group of rows) is built by adding the content of the referred ID FIELD this way: | ||
| In this case the service descriptor provides the root URL to the DataLink \blinks\ endpoint. The full \blinks\ endpoint URL for a specific row (or group of rows) is built by adding the content of the referenced ID field this way: |
| WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6) | ||
| AND (dataproduct_type = `hea-event-list') | ||
| AND semantics = `#this') | ||
| AND semantics = `#this') |
|
|
||
| Using this approach, {\bf response-function} data products could be described by one or more alternate {\tt ivoa.response$\{$\_xxx$\}$} tables, where {\tt $\{$\_xxx$\}$} is optional and would depend on the type of {\bf response-function} in the case that different sets of queryable attributes are required for different types of {\bf response-function}s. One possible example of such a table is presented in Table~\ref{tab:response_table}. | ||
|
|
||
| In order to handle the various possible cardinality relationships between {\bf response-function} and {\bf hea-event-list} datasets, foreign keys must be defined in the {\tt ivoa.responsee$\{$\_xxx$\}$} tables that will allow {\tt JOIN} operations between those tables and the {\tt ivoa.obscore} table. |
There was a problem hiding this comment.
a typo : should be "ivoa.response" instead of "ivoa.responsee"
There was a problem hiding this comment.
a typo : should be "ivoa.response" instead of "ivoa.responsee"
Thanks, I'll fix it.
|
|
||
| An extensive set of parameter values is often required to construct many types of {\bf response-function} data products, for example depending on the assumed spatial, spectral, and temporal models of the observed astrophysical source as well as multiple instrumental properties. Given the wide range of possibilities, most facilities deliberately choose to limit the set of {\bf response-function} datasets that they make available to the end user, for example by choosing a single spectral model, such as a power-law spectrum, with predefined parameters. Often these {\bf response-function}s will have been used when calibrating the associated {\bf hea-event-list} or will have been created based on some of the assumptions used during the data processing steps ({\em e.g.\/}, based on {\em analysis\_mode\/}, {\em event\_type\/}, and so on). This is evident in the use cases recorded in Appendix~\ref{sec:uc} where typically a single {\bf response-function} of a given type is associated with a single {\bf hea-event-list} and so the selection criteria are simply based on the type of {\bf response-function}, the observation dataset properties, and on the observation properties such as the observing position and field-of-view, the observation date and time, the energy, and time bounds. | ||
|
|
||
| More generally, the cardinality of the relationship between a {\bf response-function} dataset and an {\bf hea-event-list} dataset will vary according to the facility and may be one-to-one, one-to-many, many-to-one, or even many-to-many. Compared to the set of queryable attributes included in {\tt ivoa.obscore} table, typically more but occasionally fewer attributes will be required to select an appropriate {\bf response-function} dataset of a given type. The set of attributes required to identify an appropriate {\bf response-function} dataset will typically depend on the type of {\bf response-function}. |
There was a problem hiding this comment.
I don't follow you on the idea of specializing each table to each kind of response product. My proposal was to allow all responses to be discovered in the table JOIN using either a general term like "response-function" or a specialized one like "psf" or "edisp".
The idea is to define a table template for all responses, and see in practice which columns are suitable to refine the search criteria and add them in the response table. We did not explore yet use cases for fine search in response products.
There was a problem hiding this comment.
The potential issue that I am seeing is that the set of parameters required to define and/or discover an appropriate response function will typically depend on the type of response function, so if different sets of parameters are required then specialized tables may be a more optimal solution. If no specialized parameters are required then specializing the table would be unnecessary. If the relationship between the published event lists and the published response functions of a given type is one-to-one, then specialization would also not be needed. Once you go to one-to-many (one event list, many response functions of a particular type) then the chances that specialized parameters will be needed that differ from response function type to response function type increases.
Here's an example of where you might want to specialize. For some facilities and instruments, PSFs can be quite stable. In such cases, the data provider might choose to create a large library of psf response functions that records the psfs at a set of off-axis and azimuthal angles that sample the instrument field of view, with a set of different spectral models (e.g., absorbed power-law spectra with several power law indices, thermal plasma models with different temperatures, etc.).
| The difference between {\em messenger\/} and {\em messenger\_pdgid} is that ({\em e.g.\/}) a data discovery query that includes {\tt messenger = `neutrino'} would enable the user to identify datasets with a neutrino (any kind) messenger particle, whereas a data discovery query that includes {\tt messenger\_pdgid = `+16' OR messenger\_pdgid = `-16'} would restrict the search to include only $\tau$ neutrinos ($\nu_\tau$) or antineutrinos ($\nu_\tau^{-}$). | ||
|
|
||
| The downside of {\em messenger\_pdgid} is that PDG ID very unlikely to be recognized outside of the particle astrophysics and high energy particle physics communities. We choose to include both {\em messenger\/} and {\em messenger\_pdgid} because the majority of astrophysicists --- even experienced high-energy astrophysicists --- are unlikely to recognize PDG ID\null. While any astrophysicist is likely to be able to write a query such as {\tt messenger = `photon'}, very few would be able to write that query as {\tt messenger\_pdgid = `+22'} from memory. | ||
| The downside of {\em messenger\_pdgid} is that PDG ID very unlikely to be recognized outside of the particle astrophysics and high energy particle physics communities. We choose to include both {\em messenger\/} and {\em messenger\_pdgid} because the majority of astrophysicists --- even experienced high-energy astrophysicists --- are unlikely to recognize PDG ID\null. While any astrophysicist is likely to be able to write a query such as {\tt messenger = `photon'}, very few would be able to cast that query as {\tt messenger\_pdgid = `+22'} from memory. |
There was a problem hiding this comment.
The messenger ( astrophysical thing ) can be described by 'messenger_name' and its type can be encoded as 'messenger_pdgid' when relevant.
I would prefer these column names for more symmetry. But I can live with what is written.
There was a problem hiding this comment.
The messenger ( astrophysical thing ) can be described by 'messenger_name' and its type can be encoded as 'messenger_pdgid' when relevant. I would prefer these column names for more symmetry. But I can live with what is written.
I like the symmetry of messenger_name and messenger_pdgid too. But what I have written down that we agreed to at the Interop is messenger and messenger_pdgid. Unless someone objects, I'm inclined to change it to messenger_name and messenger_pdgid as Mireille suggests.
|
|
||
| The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify more precisely the scientific nature of a data product. Although no vocabulary is defined for {\em dataproduct\_subtype\/}, we recommend that data providers formulate and use a standardized vocabulary for this attribute for data products that are commonly used in \gls{HEA}\null. We have proposed several terms in \S~5 for commonly used \gls{HEA} {\bf response-function} types ({\em e.g.\/}, {\bf aeff}, {\bf edisp}, {\bf psf}), but additional terms could be standardized for other common data products. For example, standardizing using {\bf exposure-map} for an exposure map would enable queries such as ({\em dataproduct\_type\/} = {\bf image}) AND ({\em dataproduct\_subtype\/} = {\bf exposure-map}) to work across multiple facilities. Other possible terms could include (but are not limited to) {\bf significance-map} for a significance map, {\bf probability-map} for a probability map, and {\bf exclusion-map} for an exclusion map ({\em e.g.\/}, as used to adjust TeV background models). | ||
|
|
||
| We note in passing that {\bf response-function}s for some facilities and instruments may not be separable into distinct components such as {\bf aeff}, {\bf edisp}, {\bf psf}. |
There was a problem hiding this comment.
Good to mention it . in that case a new term in the response-type vocabulary could be introduced.
|
Thanks a lot Ian for this hard work. The text reads well and opens the discussion on important topics. I also noticed that the Radio Interest Group recently wrote its roadmap with this item "Shift focus to advanced data products". It will be interesting to discuss possible commonalities with the HEIG. My main comment is that most of what is Appendix B in this PR should really be a section 6. As it is now, this appendix arrives after use cases (Appendix A), but presents access concepts that are also used in Appendix A (i.e. direct DataLink). It would thus naturally fit as section 6, except for the proposed table and associated queries, that would follow naturally Appendix A. The following minor changes in the text would link Appendix B to section 6.3 : "One possible example of such a table is presented in Table 8." Start of Appendix B: I then have a comment on the wording in 6.3/B.3 : "is very likely preferable." Typos :
|
|
Mathieu,
I'm liking this content in the Appendix. These seem like options available to the users for implementing a response, and they have varying degrees of exercise in practice. I agree this is good content for the migration process, presumably more on the DAL side (being more response oriented?). Perhaps after consideration and practice, this can change to recommending one approach or another. |
I also think that the direct access to event-bundle and the datalink strategy should be presented before the Uses cases appendix, in order to understand better use case A1.6 . |
I'm not convinced that this should be part of the HEIG ObsCore Note, as it is not a requirement that we need to impose and we shouldn't constrain the WG discussions beyond our own requirements. I think splitting vocabularies the wrong way can also have unanticipated negative consequences. We have proposed very few data product vocabulary additions that are HEA specific for example (really just a subset of the response-function types: aeff, arf, bkgrate, edisp, rmf). Most of the remaining data product types that we have suggested (draws, pdf, region, response-function, and psf) are panchromatic, in that they are equally applicable for any waveband. event-list was redesignated hea-event-list even though such data products exist in other wavebands. For example, there are some optical detectors that time-tag individual photons where very high time resolution is required (e.g., for millisecond pulsar studies) resulting in data products that are fundamentally photon event lists. I doubt very much they will want to use the term "hea-event-list" for those data products however. We have also suggested a few common terms for dataproduct_subtype that could conceptually become vocabulary entries, although most of those are again panchromatic in nature. I think the only requirement that we have is that the observation-based data products (including e.g. draws, pdf, region) are directly queryable in ObsCore, as we have cases where there is no effective alternative way of querying some of these products. As long as this requirement is met, where the terms fall in the vocabulary is not something we need to constrain. |
While avoiding the usage of "measurements" may already be planned, we have a requirement for there to be a generic tabular data product. According to the data product vocabulary, "measurements" is a "Generic tabular data not fitting any of the other terms. For instance, a list of sources extracted from an image might be a measurement." We have exactly this usage as well as several others. Event lists are simply a specialized form of tabular data, so it should not come as any surprise that HEA uses the tabular data approach for many other types of data product as well. There is no other data product type that describes general tabular data products, so if "measurements" is deprecated there must be another term (tabledata?) that takes it's place. And yes, it does say "Because of its lack of specificity, this term should generally be avoided, and new, more precise terms should be introduced instead." Not the use of "should" and not "must". That is a wise choice. In one sense, "measurements" is at a similar level to "image". The latter is a data product with two spatial axes, but the observable is not specified. It doesn't have to be intensity-related, but could for example be color, exposure, significance, probability, ... we actually suggest trying to standardize several such image dataproduct_subtypes. I also think that the freedom of the data provider to specify dataproduct_subtype is a major asset of the current ObsCore Recommendation. You don't want to have to define all of these possibilities in a vocabulary (even though there may certainly be a meaningful subset for which the commonality that a vocabulary provides would be sensible). Many facilities have facility- or instrument-specific end-user data product types that simply won't be common to other facilities. so having the ability to use the more generic dataproduct_type with a facility-specific dataproduct_subtype allows this case to function effectively. There will always be instruments and data reduction methods that will require some specialized knowledge to get the most out of the data. These are often the state-of-the-art and move astronomical techniques and research forward, so we need to embrace them. |
I'm not convinced this is a strong argument. Prior to the Interop we were happy with just presenting the use cases in Appendix A without this information being present at all. We could also swap the order of Appendices A and B.
I'll incorporate the text updates. |
6 participants
(1) Numerous small wordsmithing changes.
(2) Split messenger and messenger_pdgid as agreed with Semantics.
(3) Moved the extension table summary table to the end of the section that defines the table entries.
(4) SIgnificant rewrites in section 6.