From 06084e921eec5333d10db83bde179b038f7c5a6f Mon Sep 17 00:00:00 2001 From: Mathieu Servillat Date: Wed, 23 Jul 2025 16:43:36 +0200 Subject: [PATCH 01/11] shorten 3.1 --- HighEnergyObsCoreExt.tex | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index de545d4..c4fc435 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -82,7 +82,7 @@ \begin{document} \begin{abstract} -This is a proposed extension to the ObsCore specification for data description, discovery and selection of High Energy Astrophysics (HEA) data, and includes proposed updates to the data product vocabulary, UCDs, and MIME-types to support discovery of HEA data. +This is a proposed extension to the ObsCore specification for data description, discovery and selection of High Energy Astrophysics (HEA) data, and includes proposed updates to the data product vocabulary, UCDs, and MIME-types to support discovery of HEA data. \end{abstract} @@ -121,12 +121,23 @@ \section{ObsCore Attribute Definitions for High Energy Astrophysics Data} \subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} -The attribute {\em dataproduct\_type\/} provides a high level scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for HEA datasets). The current ObsCore Recommendation defines an {\bf event} {\em dataproduct\_type} as: +The attribute {\em dataproduct\_type\/} provides a high level scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for HEA datasets). + +The ObsCore v1.1 recommendation \citep{2017ivoa.spec.0509L} only defines an {\bf event} {\em dataproduct\_type} as: \begin{quote} {\bf event}: an event-counting ({\em e.g.\/}, X-ray or other high energy) dataset of some sort. Typically this is instrumental data, {\em i.e.\/}, ``event data''. An event dataset is often a complex object containing multiple files or other substructures. An event dataset may contain data with spatial, spectral, and time information for each measured event, although the spectral resolution (energy) is sometimes limited. Event data may be used to produce higher level data products such as images or spectra. \end{quote} +An IVOA vocabulary is being developped for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term {\bf event-list} is proposed: +\begin{quote} +{\bf event-list}: a collection of observed events, such as incoming high-energy particles. A row in an event list is typically characterised by a spatial position, a time and an energy. +\end{quote} + +In the context of \gls{HE}, we propose to redefine {\bf event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HE} archives (see section XXX). The concept of an {\bf event-bundle} containing an event list and the relevant associated data products is also important for HE data dissemination. + +%----- from section 3.1, to be moved to section 5 on Vocabularies + We propose to add the following {\em dataproduct\_type}s to better define a HE {\bf event-list} and a bundle that includes the {\bf event-list} and associated ancillary data: \begin{quote} @@ -157,11 +168,17 @@ \subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} {\bf response-function}: A dataset that represents a mapping from a physical quantity to an observable. For HEA this may be the components of the composite Instrument Response Function (IRF)\footnote{We try to avoid using the term IRF in a normative sense since historical usage across the HEA community (and from facility to facility) varies. In some cases IRF has been used to mean specifically the product of the ARF and RMF, whereas in other cases IRF has been used more generally to mean {\em any\/} instrumental response function regardless of type.} such as an Auxiliary Response File (ARF), Redistribution Matrix File (RMF), Effective Area (AEFF), Energy Dispersion (EDISP), and so on. The Point Spread Function (PSF) is a response-function that is generally applicable across multiple wavebands. While these datasets may generally be represented as an {\em N\/}-dimensional data cube, designating them as {\bf response-function}s enhances data discovery for very common types of HEA dataset. \end{quote} -The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, {\bf measurements} associated with sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = {\bf measurements} type in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. +The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, {\bf measurements} associated with sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = {\bf measurements} type in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. + +%----- + + \subsection{{\em dataproduct\_subtype}} -The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify additional information about the nature of the data product. For some datasets this attribute may specify the data level ({\em e.g.\/}, DL3--6), or may be combined with {\em dataproduct\_type\/} to more precisely define the content of the dataset ({\em e.g.\/}, {\em dataproduct\_type\/} = {\bf image}${}+{}${\em dataproduct\_subtype\/} = {\bf exposuremap}, or {\em dataproduct\_type\/} = {\bf response-function}${}+{}${\em dataproduct\_subtype\/} = {\bf rmf}), which is particularly useful and important for discovery of advanced data products. A vocabulary of such data product (sub-)types should be developed to support discovery of advanced data products. +The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify additional information about the nature of the data product. For some datasets this attribute may specify the data level ({\em e.g.\/}, DL3--6, see \citealt{2024ivoa.note.heig}, \S3.1.2), or may be combined with {\em dataproduct\_type\/} to more precisely define the content of the dataset ({\em e.g.\/}, {\em dataproduct\_type\/} = {\bf image}${}+{}${\em dataproduct\_subtype\/} = {\bf exposuremap}, or {\em dataproduct\_type\/} = {\bf response-function}${}+{}${\em dataproduct\_subtype\/} = {\bf rmf}), which is particularly useful and important for discovery of some advanced data products. A vocabulary of such data product (sub-)types should be developed to support discovery of advanced data products. + +% but rmf is proposed in product-type... \subsection{{\em calib\_level}} @@ -185,7 +202,7 @@ \subsection{{\em s\_calib\_status}} We propose that {\em s\_calib\_status} encode the calibration status of an {\bf event-list} dataset's spatial axes. Where multiple spatial axes are included in a dataset ({\em e.g.\/}, physical detector pixel coordinates, virtual detector coordinates corrected for distortions, world coordinates) then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated spatial axes) to define {\em s\_calib\_status\/}. -Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. +Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. For dataset types that do not not encode sky coordinates, we suggest setting this value to NULL\null. @@ -193,7 +210,7 @@ \subsection{{\em t\_calib\_status}} We propose that {\em t\_calib\_status} encode the calibration status of an {\bf event-list} dataset's time axis. Where multiple time axes are included in a dataset ({\em e.g.\/}, instrument counter, absolute time) then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated time axis) to define {\em t\_calib\_status\/}. -Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. +Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. For dataset types that do not not encode time coordinates, we suggest setting this value to NULL\null. @@ -201,7 +218,7 @@ \subsection{{\em em\_calib\_status}} We propose that {\em em\_calib\_status} encode the calibration status of an {\bf event-list} dataset's spectral axis. Where multiple spectral axes are included in a dataset ({\em e.g.\/}, PHA, PI, energy) then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated spectral axis) to define {\em t\_calib\_status\/}. -Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. +Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. For dataset types that do not not encode spectral coordinates, we suggest setting this value to NULL\null. @@ -271,7 +288,7 @@ \subsection{{\em analysis\_mode}} \section{Vocabulary Enhancements} -While the IVOA Data Product Type Vocabulary (\url{http://www.ivoa.net/rdf/product-type}) provides terms, labels, and descriptions for many types of astronomical data products, there are some additions and changes that are appropriate to better support HEA datasets. +While the IVOA Data Product Type Vocabulary (\url{http://www.ivoa.net/rdf/product-type}) provides terms, labels, and descriptions for many types of astronomical data products, there are some additions and changes that are appropriate to better support HEA datasets. We propose to add vocabulary entries for the data product types outlined in \S~\ref{sec:dataproduct_type} ({\em i.e.\/}, {\bf event-bundle}, {\bf draws}, {\bf pdf}, {\bf region}, {\bf response-function}) and also propose to slightly modify the existing definition of {\bf event-list} so that it aligns more accurately with the definition in \S~\ref{sec:dataproduct_type}. Additionally, we propose to add several more specific entries to the data product type vocabulary that specialize these types (especially {\bf response-function}). Any future revision of the ObsCore Recommendation to use the IVOA Data Product Type Vocabulary in preference to the enumerated values for {\em dataproduct\_type\/} would profit from these additional vocabulary entries. From 8e427bc19d3a72058d6f5e50f48e8a3fb25d4d86 Mon Sep 17 00:00:00 2001 From: Mathieu Servillat Date: Wed, 23 Jul 2025 16:46:07 +0200 Subject: [PATCH 02/11] move term definitions to section_5 branch --- HighEnergyObsCoreExt.tex | 37 ------------------------------------- 1 file changed, 37 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index c4fc435..6d10bb4 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -136,43 +136,6 @@ \subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} In the context of \gls{HE}, we propose to redefine {\bf event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HE} archives (see section XXX). The concept of an {\bf event-bundle} containing an event list and the relevant associated data products is also important for HE data dissemination. -%----- from section 3.1, to be moved to section 5 on Vocabularies - -We propose to add the following {\em dataproduct\_type}s to better define a HE {\bf event-list} and a bundle that includes the {\bf event-list} and associated ancillary data: - -\begin{quote} -{\bf event-list}: A collection of observed events, such as incoming HE particles, where an event is typically characterized by a spatial position, a time, and a spectral value ({\em e.g.\/}, an energy, a channel, a pulse height). -\end{quote} - -\begin{quote} -{\bf event-bundle}: An event-bundle dataset is a complex object containing an {\bf event-list} and multiple files or other substructures that are products necessary to analyze the {\bf event-list}. Data in an event-bundle may thus be used to produce higher level data products such as images or spectra. -\end{quote} - -An {\bf event-bundle} might for example consist of an {\bf event-list} and the associated {\bf response-function}s (see below) used to calibrate the dataset; alternatively an {\bf event-bundle} may include the {\bf event-list} and associated ancillary data products necessary for the user to {\em create\/} the {\bf response-function}s (for those cases where detailed knowledge of the scientific use case --- for example, the user's selection of events --- may be required to compute the responses). - -In addition to {\em dataproduct\_type\/}s that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (with {\em calib\_level\/}${}\ge 3$ that may be generated from astronomical observations. The computational complexity of analyzing HEA data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations) means that data providers may choose to provide such analysis products directly to the end user. For example, the {\em Chandra\/} Source Catalog includes 38 types of advanced data products (for a total of $\sim\!90$ million files) and $\sim\!50\%$ of these data product types are not well represented by a {\em dataproduct\_type} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type}s for these advanced data products, and note that these {\em dataproduct\_type}s will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets: - -\begin{quote} -{\bf draws}: A dataset that represents draws computed from a probability distribution, for example the Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable. The draws can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution. -\end{quote} - -\begin{quote} -{\bf pdf}: A dataset that represents the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution. -\end{quote} - -\begin{quote} -{\bf region}: A dataset that includes an encoding of (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary. -\end{quote} - -\begin{quote} -{\bf response-function}: A dataset that represents a mapping from a physical quantity to an observable. For HEA this may be the components of the composite Instrument Response Function (IRF)\footnote{We try to avoid using the term IRF in a normative sense since historical usage across the HEA community (and from facility to facility) varies. In some cases IRF has been used to mean specifically the product of the ARF and RMF, whereas in other cases IRF has been used more generally to mean {\em any\/} instrumental response function regardless of type.} such as an Auxiliary Response File (ARF), Redistribution Matrix File (RMF), Effective Area (AEFF), Energy Dispersion (EDISP), and so on. The Point Spread Function (PSF) is a response-function that is generally applicable across multiple wavebands. While these datasets may generally be represented as an {\em N\/}-dimensional data cube, designating them as {\bf response-function}s enhances data discovery for very common types of HEA dataset. -\end{quote} - -The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, {\bf measurements} associated with sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = {\bf measurements} type in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. - -%----- - - \subsection{{\em dataproduct\_subtype}} From 26ed2e1bb3dbcaaf5bc224a704a9725042f96cea Mon Sep 17 00:00:00 2001 From: Mathieu Servillat Date: Wed, 23 Jul 2025 17:59:36 +0200 Subject: [PATCH 03/11] add access_url and datalink solution --- HighEnergyObsCoreExt.tex | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 6d10bb4..3e15247 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -134,7 +134,7 @@ \subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} {\bf event-list}: a collection of observed events, such as incoming high-energy particles. A row in an event list is typically characterised by a spatial position, a time and an energy. \end{quote} -In the context of \gls{HE}, we propose to redefine {\bf event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HE} archives (see section XXX). The concept of an {\bf event-bundle} containing an event list and the relevant associated data products is also important for HE data dissemination. +In the context of \gls{HE}, we propose to redefine {\bf event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HE} archives (see section \ref{sec:voc_product_type}). The concept of an {\bf event-bundle} containing an event list and the relevant associated data products is also important for HE data dissemination. \subsection{{\em dataproduct\_subtype}} @@ -151,6 +151,11 @@ \subsection{{\em calib\_level}} %\subsection{{\em obs\_collection}} +\subsection{{\em access\_url}} + +Given the complexity and number of HE data products, we propose to point the access\_url either to a file (e.g. an event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). + + \subsection{{\em access\_format}} The {\em access\_format\/} attribute specifies the format of the data product when downloaded as a file from the {\em access\_url\/}. The analysis of HE data often requires use of multiple, related data products, for example an {\bf event-list} combined with associated IRFs or ancillary files that can be employed by the user to create IRFs. These associated products are often bundled together with the {\bf event-list} and we proposed in \S~\ref{sec:dataproduct_type} to assign such bundles {\em dataproduct\_type\/} = {\bf event-bundle}. While these bundles are typically not standardized across different projects, knowledge of the bundle content is useful for client applications to properly handle the bundles (for example to send the data to an appropriate visualization tool). This is readily achieved by encoding an appropriate MIME-type using the {\em access\_format\/} attribute. In Section~\ref{sec:mimetypes} we propose additional MIME-types for some common {\bf event-bundle}s. From 6de4d4d0f17ce7d7b735bebb1c7d585b31b43ff5 Mon Sep 17 00:00:00 2001 From: Mathieu Servillat Date: Thu, 24 Jul 2025 16:59:41 +0200 Subject: [PATCH 04/11] complete access_url section --- HighEnergyObsCoreExt.tex | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 3e15247..91a681c 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -119,7 +119,8 @@ \section{ObsCore Attribute Definitions for High Energy Astrophysics Data} We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. -\subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} +\subsection{{\em dataproduct\_type}} +\label{sec:dataproduct_type} The attribute {\em dataproduct\_type\/} provides a high level scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for HEA datasets). @@ -129,7 +130,7 @@ \subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} {\bf event}: an event-counting ({\em e.g.\/}, X-ray or other high energy) dataset of some sort. Typically this is instrumental data, {\em i.e.\/}, ``event data''. An event dataset is often a complex object containing multiple files or other substructures. An event dataset may contain data with spatial, spectral, and time information for each measured event, although the spectral resolution (energy) is sometimes limited. Event data may be used to produce higher level data products such as images or spectra. \end{quote} -An IVOA vocabulary is being developped for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term {\bf event-list} is proposed: +An IVOA vocabulary is being developped for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term {\bf event-list} is proposed as a replacement: \begin{quote} {\bf event-list}: a collection of observed events, such as incoming high-energy particles. A row in an event list is typically characterised by a spatial position, a time and an energy. \end{quote} @@ -139,13 +140,17 @@ \subsection{{\em dataproduct\_type}}\label{sec:dataproduct_type} \subsection{{\em dataproduct\_subtype}} -The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify additional information about the nature of the data product. For some datasets this attribute may specify the data level ({\em e.g.\/}, DL3--6, see \citealt{2024ivoa.note.heig}, \S3.1.2), or may be combined with {\em dataproduct\_type\/} to more precisely define the content of the dataset ({\em e.g.\/}, {\em dataproduct\_type\/} = {\bf image}${}+{}${\em dataproduct\_subtype\/} = {\bf exposuremap}, or {\em dataproduct\_type\/} = {\bf response-function}${}+{}${\em dataproduct\_subtype\/} = {\bf rmf}), which is particularly useful and important for discovery of some advanced data products. A vocabulary of such data product (sub-)types should be developed to support discovery of advanced data products. +The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify additional information about the nature of the data product. For some datasets this attribute may specify the data level ({\em e.g.\/}, DL3--6, see \citealt{2024ivoa.note.heig}, \S3.1.2), or may be combined with {\em dataproduct\_type\/} to more precisely define the content of the dataset ({\em e.g.\/}, {\em dataproduct\_type\/} = {\bf image}${}+{}${\em dataproduct\_subtype\/} = {\bf exposuremap}). A vocabulary of such data product (sub-)types should be developed to support discovery of advanced data products. % but rmf is proposed in product-type... \subsection{{\em calib\_level}} -ObsCore defines calibration {\bf Level 1} as ``Instrumental data in a standard format (FITS, VOTable, SDFITS, ASDM, etc.) which could be manipulated with standard astronomical packages.'' and {\bf Level 2} as ``Calibrated, science ready data with the instrument signature removed.'' However, many HEA {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. However, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we {\em suggest\/} that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. +ObsCore defines calibration {\bf Level 1} as ``Instrumental data in a standard format (FITS, VOTable, SDFITS, ASDM, etc.) which could be manipulated with standard astronomical packages.'' and {\bf Level 2} as ``Calibrated, science ready data with the instrument signature removed.'' + +However, many HEA {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. + +On the other hand, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. For {\bf event-list}s, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). @@ -153,7 +158,11 @@ \subsection{{\em calib\_level}} \subsection{{\em access\_url}} -Given the complexity and number of HE data products, we propose to point the access\_url either to a file (e.g. an event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). +Given the complexity and number of HE data products, we propose to point the access\_url either to a file directly (e.g. the event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). + +In case Datalinks are provided, it should be indicated that the URL points to a Datalink service (via the access\_format probably). + +If the access\_url points to a bundle, an issue is that the detailed content of the bundle is not directly exposed, we thus see an advantage in using a DataLink service. \subsection{{\em access\_format}} From 6201677f15c7cea78fe9d7e983da747371d9fb87 Mon Sep 17 00:00:00 2001 From: Mathieu Servillat Date: Fri, 25 Jul 2025 17:41:19 +0200 Subject: [PATCH 05/11] typo correction --- HighEnergyObsCoreExt.tex | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 91a681c..3824a67 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -130,7 +130,7 @@ \subsection{{\em dataproduct\_type}} {\bf event}: an event-counting ({\em e.g.\/}, X-ray or other high energy) dataset of some sort. Typically this is instrumental data, {\em i.e.\/}, ``event data''. An event dataset is often a complex object containing multiple files or other substructures. An event dataset may contain data with spatial, spectral, and time information for each measured event, although the spectral resolution (energy) is sometimes limited. Event data may be used to produce higher level data products such as images or spectra. \end{quote} -An IVOA vocabulary is being developped for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term {\bf event-list} is proposed as a replacement: +An IVOA vocabulary is being developed for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term {\bf event-list} is proposed as a replacement: \begin{quote} {\bf event-list}: a collection of observed events, such as incoming high-energy particles. A row in an event list is typically characterised by a spatial position, a time and an energy. \end{quote} @@ -158,11 +158,12 @@ \subsection{{\em calib\_level}} \subsection{{\em access\_url}} -Given the complexity and number of HE data products, we propose to point the access\_url either to a file directly (e.g. the event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). +Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file (e.g. to the event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). -In case Datalinks are provided, it should be indicated that the URL points to a Datalink service (via the access\_format probably). +If DataLink is provided, it should be indicated that the URL points to a Datalink service via the {\em access\_format} = application/x-votable+xml;content=datalink. -If the access\_url points to a bundle, an issue is that the detailed content of the bundle is not directly exposed, we thus see an advantage in using a DataLink service. +If the {\em access\_url} points to a bundle, the detailed content of the bundle is not exposed; therefore using a DataLink service has advantages. +%Otherwise, the description of the content of the bundle could be indicated in dedicated attribute. \subsection{{\em access\_format}} @@ -181,7 +182,7 @@ \subsection{{\em s\_calib\_status}} Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. -For dataset types that do not not encode sky coordinates, we suggest setting this value to NULL\null. +For dataset types that do not encode sky coordinates, we suggest setting this value to NULL\null. \subsection{{\em t\_calib\_status}} From 9eafebaba6767d75e55a497c4d712d428884a2f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20Kh=C3=A9lifi?= Date: Mon, 4 Aug 2025 10:50:24 +0200 Subject: [PATCH 06/11] start review comment --- HighEnergyObsCoreExt.tex | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 63c02cf..6a61e57 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -83,6 +83,8 @@ \begin{abstract} This is a proposed extension to the ObsCore specification for data description, discovery and selection of High Energy Astrophysics (HEA) data, and includes proposed updates to the data product vocabulary, UCDs, and MIME-types to support discovery of HEA data. +This is a proposed extension to the ObsCore specification for data description, discovery and selection of \gls{HEA} data, and includes proposed updates to the data product vocabulary, UCDs, and MIME-types to support discovery of \gls{HEA} data. + \end{abstract} @@ -115,14 +117,14 @@ \section{High Energy Astrophysics Data} \section{ObsCore Attribute Definitions for High Energy Astrophysics Data} -The ObsCore representation of HEA {\bf event-list} data products is described in terms of curation, coverage, and access. However, following the ObsCore Recommendation, several properties, including resolutions, observable axis descriptions, and polarization states are simply set to NULL, and data axis lengths are set to $-1$. Therefore, for these data products and associated IRFs, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. These adjustments will also typically apply to advanced, high-level data products derived from {\bf event-list} data. +The ObsCore representation of \gls{HEA} \textbf{event-list} data products is described in terms of curation, coverage, and access. However, following the ObsCore Recommendation, several properties, including resolutions, observable axis descriptions, and polarization states are simply set to NULL, and data axis lengths are set to $-1$. Therefore, for these data products and associated IRFs, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. These adjustments will also typically apply to advanced, high-level data products derived from \textbf{event-list} data. We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. \subsection{{\em dataproduct\_type}} \label{sec:dataproduct_type} -The attribute {\em dataproduct\_type\/} provides a high level scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for HEA datasets). +The attribute {\em dataproduct\_type\/} provides a high level scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for \gls{HEA} datasets). The ObsCore v1.1 recommendation \citep{2017ivoa.spec.0509L} only defines an {\bf event} {\em dataproduct\_type} as: @@ -130,13 +132,13 @@ \subsection{{\em dataproduct\_type}} {\bf event}: an event-counting ({\em e.g.\/}, X-ray or other high energy) dataset of some sort. Typically this is instrumental data, {\em i.e.\/}, ``event data''. An event dataset is often a complex object containing multiple files or other substructures. An event dataset may contain data with spatial, spectral, and time information for each measured event, although the spectral resolution (energy) is sometimes limited. Event data may be used to produce higher level data products such as images or spectra. \end{quote} -An IVOA vocabulary is being developed for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term {\bf event-list} is proposed as a replacement: +An \gls{IVOA} vocabulary is being developed for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term \textbf{event-list} is proposed as a replacement: + \begin{quote} -{\bf event-list}: a collection of observed events, such as incoming high-energy particles. A row in an event list is typically characterised by a spatial position, a time and an energy. +{\bf event-list}: a collection of observed events, such as incoming high-energy particles. The table of event list is typically characterised by a spatial position, a time and an energy proxy. \end{quote} -In the context of \gls{HE}, we propose to redefine {\bf event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HE} archives (see section \ref{sec:voc_product_type}). The concept of an {\bf event-bundle} containing an event list and the relevant associated data products is also important for HE data dissemination. - +In the context of \gls{HEA}, we propose to redefine \textbf{event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HEA} archives (see section \ref{sec:voc_product_type}). The concept of an \textbf{event-bundle} containing an event list and the relevant associated data products is also important for \gls{HEA} data dissemination. \subsection{{\em dataproduct\_subtype}} From 90418d27e3c14bd4268fc1df1109d4da077be936 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20Kh=C3=A9lifi?= Date: Mon, 4 Aug 2025 12:39:43 +0200 Subject: [PATCH 07/11] Review comments --- HighEnergyObsCoreExt.tex | 50 ++++++++++++++++++++++++++++++++-------- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 6f8a570..54bb5bb 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -121,14 +121,14 @@ \section{High Energy Astrophysics Data} \section{ObsCore Attribute Definitions for High Energy Astrophysics Data} \label{sec:obscore} -The ObsCore representation of \gls{HEA} \textbf{event-list} data products is described in terms of curation, coverage, and access. However, following the ObsCore Recommendation, several properties, including resolutions, observable axis descriptions, and polarization states are simply set to NULL, and data axis lengths are set to $-1$. Therefore, for these data products and associated IRFs, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. These adjustments will also typically apply to advanced, high-level data products derived from \textbf{event-list} data. +The ObsCore representation of any \gls{HEA} \textbf{event-list} data products is described in terms of curation, coverage, and access. However, given the \gls{HEA} data specificities, several properties, including resolutions, observable axis descriptions, and polarization states would be simply set to ``NULL'', and data axis lengths are set to ``$-1$''. Therefore, for these data products and associated \glspl{IRF}, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. These adjustments will also typically apply to advanced, high-level data products derived from \textbf{event-list} data. -We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. +In addition, the hereafter modification proposal faces to the issue that some values of ObsCore attributes ({\em dataproduct\_type} and {\em calib\_level}) are defined both into the Obscore standard document \citep{2017ivoa.spec.0509L} and in the vocabularies documents \citep{2023ivoa.spec.0206D, 2021ivoa.spec.0525D}, which might create some issues for the users. In this context, we have opted to propose modifications of both standards in this document, even if we would have been prefered that everything is uniquely defined in the \gls{IVOA} Vovabulary. Some harmonization should be taken by the Data Model and Semantics working groups in order to avoid duplications. \subsection{{\em dataproduct\_type}} \label{sec:dataproduct_type} -The attribute {\em dataproduct\_type\/} provides a high level scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for \gls{HEA} datasets). +The attribute {\em dataproduct\_type\/} provides a scientific classification of the data product and is of primary importance for data discovery, especially when there may be many different types of data product associated with an observation\footnote{We use the term ``observation'' in the broad sense, as is done in the ObsCore Recommendation. We note that in this context an ``observation'' may not correspond to a single pointed observation defined in the traditional sense.} (as is often the case for \gls{HEA} datasets). The ObsCore v1.1 recommendation \citep{2017ivoa.spec.0509L} only defines an {\bf event} {\em dataproduct\_type} as: @@ -136,24 +136,53 @@ \subsection{{\em dataproduct\_type}} {\bf event}: an event-counting ({\em e.g.\/}, X-ray or other high energy) dataset of some sort. Typically this is instrumental data, {\em i.e.\/}, ``event data''. An event dataset is often a complex object containing multiple files or other substructures. An event dataset may contain data with spatial, spectral, and time information for each measured event, although the spectral resolution (energy) is sometimes limited. Event data may be used to produce higher level data products such as images or spectra. \end{quote} -An \gls{IVOA} vocabulary is being developed for Product Types\footnote{https://www.ivoa.net/rdf/product-type} were the term \textbf{event-list} is proposed as a replacement: +We propose to add the following {\em dataproduct\_type} term in both the Obscore standard and into the \gls{IVOA} vocabulary is of Product Types\footnote{See \url{https://www.ivoa.net/rdf/product-type}.} to better define a \gls{HEA} \textbf{event-list} and a \textbf{event-list} that includes the event-list and its associated data: \begin{quote} -{\bf event-list}: a collection of observed events, such as incoming high-energy particles. The table of event list is typically characterised by a spatial position, a time and an energy proxy. +{\bf event-list}: a collection of observed events, such as incoming high-energy particles. The table of event list is typically characterised by a spatial position, a time and an energy proxy. + +{\bf event-bundle}: compounded dataset containing an {\bf event-list} and multiple files or other substructures that are products necessary to analyze the event-list. Data in an event-bundle may thus be used to produce higher level data products such as images or spectra when containing \glspl{IRF}. \end{quote} -In the context of \gls{HEA}, we propose to redefine \textbf{event-list}, and add new terms for response functions, as well as for several advanced data products found in \gls{HEA} archives (see section \ref{sec:voc_product_type}). The concept of an \textbf{event-bundle} containing an event list and the relevant associated data products is also important for \gls{HEA} data dissemination. +It may be worth mentioning that the term ``event'' caused confusion in the past, as it also is used for astrophysical events like supernova explosions (e.g. VOEvent), and that is not the type of event that is being described here, which are particle detection events. Using "event-list" was meant to help to resolve this ambiguity. + +An {\bf event-bundle} might for example consist of an {\bf event-list} and the associated {\bf response-functions} (see below) used to calibrate the dataset; alternatively an {\bf event-bundle} may include the {\bf event-list} and associated data products necessary for the user to create the {\bf response-functions} (for those X-ray cases where detailed knowledge of the scientific use case — for example, the user’s selection of events — may be required to compute the responses).\\ + +In addition to {\em dataproduct\_type} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (with {\em calib\_level} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime (e.g., Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and ob- +servations) means that data providers may choose to provide such analysis products directly to the end user. For example, the Chandra Source Catalog includes 38 types of advanced data products (for a total of $\sim$90 million files) and $\sim$50\% of these data product types are not well represented by a {\em dataproduct\_type} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional +{\em dataproduct\_type} (or {\em dataproduct\_subtype}) terms for these advanced data products, and note that these terms will certainly be useful independent of waveband (i.e., they can be equally applicable to UV/optical, IR, and radio datasets): + +\begin{quote} +{\bf draws}: a dataset that represents draws computed from a probability distribution, for example the Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable. The draws +can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution. + +{\bf pdf}: a dataset that represents the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution. + +{\bf region}: a dataset that includes an encoding of (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary. + +{\bf response-function}: a dataset that represents a mapping from a physical quantity to an observable. For \gls{HEA}, this may be the components of the composite \gls{IRF}\footnote{We try to avoid using the term \gls{IRF} in a normative sense since historical usage across the broad \gls{HEA} community (and from facility to facility) varies. In some cases, \gls{IRF} has been used to mean specifically the X-ray product of the ``ARF'' and ``RMF'', whereas in other cases \gls{IRF} +has been used more generally to mean any instrumental response function regardless of type.} such as an Auxiliary Response File ({\bf ARF}), Redistribution Matrix File ({\bf RMF}), Effective Area ({\bf AEFF}), Energy Dispersion ({\bf EDISP}), the Background Rate ({\bf BKGRATE}). The Point Spread Function ({\bf PSF}) is a response function that is generally applicable across multiple wavebands. While these datasets may generally be represented as an N-dimensional data cube, designating them as {\bf response-functions} enhances data discovery for very common types of \gls{HEA} dataset (see the use cases in appendix \ref{sec:uc}). + +\end{quote} -repetition + "It may be worth mentioning, maybe just in a footnote, that the term "event" caused confusion in the past, as it also is used for astrophysical events like supernova explosions (e.g. VOEvent), and that is not the type of event that is being described here, which are particle detection events. Using "event-list" was meant to help to resolve this ambiguity." + +The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors (e.g., {\bf measurements associated with sources detected in a single observation field}). We propose to delete the caveat associated with {\bf dataproduct\_type} = ``measurements'' in the ObsCore IVOA Recommendation (\S4.1.1) that requires the derived data products be exposed ``together with the progenitor observation dataset''.\\ + + +Note that these terms will be repeated in the section \ref{sec:voc}, as mentioned in the introduction of this sub-section. \subsection{{\em dataproduct\_subtype}} -The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify additional information about the nature of the data product. For some datasets this attribute may specify the data level ({\em e.g.\/}, DL3--6, see \citealt{2024ivoa.note.heig}, \S3.1.2), or may be combined with {\em dataproduct\_type\/} to more precisely define the content of the dataset ({\em e.g.\/}, {\em dataproduct\_type\/} = {\bf image}${}+{}${\em dataproduct\_subtype\/} = {\bf exposuremap}). A vocabulary of such data product (sub-)types should be developed to support discovery of advanced data products. +The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify additional information about the nature of the data product. For some datasets this attribute may specify the data level ({\em e.g.\/}, DL3--6, see \citealt{2024ivoa.note.heig}, \S3.1.2), or may be combined with {\em dataproduct\_type\/} to more precisely define the content of the dataset ({\em e.g.\/}, {\em dataproduct\_type\/} = {\bf image}${}+{}${\em dataproduct\_subtype\/} = {\bf exposuremap}). A vocabulary of such data product (sub-)types is proposed in section \ref{sec:voc} to support discovery of advanced data products. -% but rmf is proposed in product-type... \subsection{{\em calib\_level}} + +KK: For {\bf event-list}s, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). + + + ObsCore defines calibration {\bf Level 1} as ``Instrumental data in a standard format (FITS, VOTable, SDFITS, ASDM, etc.) which could be manipulated with standard astronomical packages.'' and {\bf Level 2} as ``Calibrated, science ready data with the instrument signature removed.'' However, many HEA {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. @@ -238,7 +267,6 @@ \subsection{{\em proposal\_id}} \section{Extensions to ObsCore Specific to High Energy Astrophysics Data} \label{sec:obscoreext} -\label{sec:voc} \subsection{{\em ev\_xel}} @@ -279,6 +307,7 @@ \subsection{{\em analysis\_mode}} % Need more input/justification from facilities that support these capabilities \section{Vocabulary Enhancements} +\label{sec:voc} While the IVOA Data Product Type Vocabulary (\url{http://www.ivoa.net/rdf/product-type}) provides terms, labels, and descriptions for many types of astronomical data products, there are some additions and changes that are appropriate to better support HEA datasets. @@ -347,6 +376,7 @@ \section{Datalink}\label{sec:datalink} \section{Detailed Science Use Cases for ObsCore} +\label{sec:uc} \input{UseCases.tex} From e4fbc0f8666b31e851548a98c60e8b2ec2fb9efe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20Kh=C3=A9lifi?= Date: Mon, 4 Aug 2025 12:44:24 +0200 Subject: [PATCH 08/11] Editing --- HighEnergyObsCoreExt.tex | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 54bb5bb..9384dc9 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -178,33 +178,23 @@ \subsection{{\em dataproduct\_subtype}} \subsection{{\em calib\_level}} - KK: For {\bf event-list}s, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). ObsCore defines calibration {\bf Level 1} as ``Instrumental data in a standard format (FITS, VOTable, SDFITS, ASDM, etc.) which could be manipulated with standard astronomical packages.'' and {\bf Level 2} as ``Calibrated, science ready data with the instrument signature removed.'' -However, many HEA {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. +However, many \gls{HEA} {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. On the other hand, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. -Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file (e.g. to the event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). - -If DataLink is provided, it should be indicated that the URL points to a Datalink service via the {\em access\_format} = application/x-votable+xml;content=datalink. - -If the {\em access\_url} points to a bundle, the detailed content of the bundle is not exposed; therefore using a DataLink service has advantages. -%Otherwise, the description of the content of the bundle could be indicated in dedicated attribute. - - \subsection{{\em access\_url}} Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file (e.g. to the event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). -If DataLink is provided, it should be indicated that the URL points to a Datalink service via the {\em access\_format} = application/x-votable+xml;content=datalink. +If DataLink is provided, it should be indicated that the URL points to a Datalink service via the {\em access\_format} = application/x-votable+xml;content\\=datalink. If the {\em access\_url} points to a bundle, the detailed content of the bundle is not exposed; therefore using a DataLink service has advantages. -%Otherwise, the description of the content of the bundle could be indicated in dedicated attribute. \subsection{{\em access\_format}} From cab48142757bd0eeab3ad94736ca192deb0e2ec4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20Kh=C3=A9lifi?= Date: Mon, 4 Aug 2025 16:34:21 +0200 Subject: [PATCH 09/11] Add review comments --- HighEnergyObsCoreExt.tex | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 9384dc9..d0279f6 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -123,7 +123,7 @@ \section{ObsCore Attribute Definitions for High Energy Astrophysics Data} The ObsCore representation of any \gls{HEA} \textbf{event-list} data products is described in terms of curation, coverage, and access. However, given the \gls{HEA} data specificities, several properties, including resolutions, observable axis descriptions, and polarization states would be simply set to ``NULL'', and data axis lengths are set to ``$-1$''. Therefore, for these data products and associated \glspl{IRF}, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. These adjustments will also typically apply to advanced, high-level data products derived from \textbf{event-list} data. -In addition, the hereafter modification proposal faces to the issue that some values of ObsCore attributes ({\em dataproduct\_type} and {\em calib\_level}) are defined both into the Obscore standard document \citep{2017ivoa.spec.0509L} and in the vocabularies documents \citep{2023ivoa.spec.0206D, 2021ivoa.spec.0525D}, which might create some issues for the users. In this context, we have opted to propose modifications of both standards in this document, even if we would have been prefered that everything is uniquely defined in the \gls{IVOA} Vovabulary. Some harmonization should be taken by the Data Model and Semantics working groups in order to avoid duplications. +In addition, the hereafter modification proposal faces to the issue that some values of ObsCore attributes ({\em dataproduct\_type} and {\em calib\_level}) are defined both into the Obscore standard document \citep{2017ivoa.spec.0509L} and in the vocabularies documents \citep{2023ivoa.spec.0206D, 2021ivoa.spec.0525D}, which might create some issues for the users. In this context, we have opted to propose in this document some modifications of both standards, even if we would have prefered that everything is uniquely defined in the \gls{IVOA} Vocabulary. Some harmonization should be taken by the Data Model and Semantics working groups in order to avoid duplications. But until such work is achieved, we require modifications in ObsCore and Vocabulary. \subsection{{\em dataproduct\_type}} \label{sec:dataproduct_type} @@ -178,25 +178,20 @@ \subsection{{\em dataproduct\_subtype}} \subsection{{\em calib\_level}} -KK: For {\bf event-list}s, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). - - - ObsCore defines calibration {\bf Level 1} as ``Instrumental data in a standard format (FITS, VOTable, SDFITS, ASDM, etc.) which could be manipulated with standard astronomical packages.'' and {\bf Level 2} as ``Calibrated, science ready data with the instrument signature removed.'' -However, many \gls{HEA} {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. +However, some \gls{HEA} {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. In X-ray, this is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. -On the other hand, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. +On the other hand, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. Also, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis ObsCore {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). \subsection{{\em access\_url}} -Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file (e.g. to the event-list or an event-bundle), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). +Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file (e.g. to the {\bf event-list} or an {\bf event-bundle}), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). If DataLink is provided, it should be indicated that the URL points to a Datalink service via the {\em access\_format} = application/x-votable+xml;content\\=datalink. If the {\em access\_url} points to a bundle, the detailed content of the bundle is not exposed; therefore using a DataLink service has advantages. - \subsection{{\em access\_format}} The {\em access\_format\/} attribute specifies the format of the data product when downloaded as a file from the {\em access\_url\/}. The analysis of HE data often requires use of multiple, related data products, for example an {\bf event-list} combined with associated IRFs or ancillary files that can be employed by the user to create IRFs. These associated products are often bundled together with the {\bf event-list} and we proposed in \S~\ref{sec:dataproduct_type} to assign such bundles {\em dataproduct\_type\/} = {\bf event-bundle}. While these bundles are typically not standardized across different projects, knowledge of the bundle content is useful for client applications to properly handle the bundles (for example to send the data to an appropriate visualization tool). This is readily achieved by encoding an appropriate MIME-type using the {\em access\_format\/} attribute. In Section~\ref{sec:mimetypes} we propose additional MIME-types for some common {\bf event-bundle}s. From 4c3fe2e582a3d91deb5fd5e3d1985aebcbb22391 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20Kh=C3=A9lifi?= Date: Mon, 4 Aug 2025 17:03:33 +0200 Subject: [PATCH 10/11] Add glossary files --- HighEnergyObsCoreExt.glg | 7 +++++++ HighEnergyObsCoreExt.gls | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 HighEnergyObsCoreExt.glg create mode 100644 HighEnergyObsCoreExt.gls diff --git a/HighEnergyObsCoreExt.glg b/HighEnergyObsCoreExt.glg new file mode 100644 index 0000000..35e0aeb --- /dev/null +++ b/HighEnergyObsCoreExt.glg @@ -0,0 +1,7 @@ +This is makeindex, version 2.15 [TeX Live 2022/dev] (kpathsea + Thai support). +Scanning style file ./HighEnergyObsCoreExt.ist.............................done (29 attributes redefined, 0 ignored). +Scanning input file HighEnergyObsCoreExt.glo....done (62 entries accepted, 0 rejected). +Sorting entries....done (398 comparisons). +Generating output file HighEnergyObsCoreExt.gls....done (36 lines written, 0 warnings). +Output written in HighEnergyObsCoreExt.gls. +Transcript written in HighEnergyObsCoreExt.glg. diff --git a/HighEnergyObsCoreExt.gls b/HighEnergyObsCoreExt.gls new file mode 100644 index 0000000..64e1f8e --- /dev/null +++ b/HighEnergyObsCoreExt.gls @@ -0,0 +1,36 @@ +\glossarysection[\glossarytoctitle]{\glossarytitle}\glossarypreamble +\begin{theglossary}\glossaryheader +\glsgroupheading{G}\relax \glsresetentrylist % +\glossentry{GTI}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{14}}}\glsgroupskip +\glsgroupheading{H}\relax \glsresetentrylist % +\glossentry{HE}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{4\delimN 5}\delimN + \setentrycounter[]{page}\glsnumberformat{26}}}% +\glossentry{HEA}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{1}\delimN + \setentrycounter[]{page}\glsnumberformat{4}\delimN + \setentrycounter[]{page}\glsnumberformat{6\delimR 10}\delimN + \setentrycounter[]{page}\glsnumberformat{12\delimN 13}}}% +\glossentry{HEIG}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{4}}}\glsgroupskip +\glsgroupheading{I}\relax \glsresetentrylist % +\glossentry{IRF}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{5\delimR 7}\delimN + \setentrycounter[]{page}\glsnumberformat{9\delimN 10}}}% +\glossentry{IVOA}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{4}\delimN + \setentrycounter[]{page}\glsnumberformat{6\delimN 7}\delimN + \setentrycounter[]{page}\glsnumberformat{14}\delimN + \setentrycounter[]{page}\glsnumberformat{26}}}\glsgroupskip +\glsgroupheading{M}\relax \glsresetentrylist % +\glossentry{MOC}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{14}}}\glsgroupskip +\glsgroupheading{S}\relax \glsresetentrylist % +\glossentry{STI}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{14}}}\glsgroupskip +\glsgroupheading{V}\relax \glsresetentrylist % +\glossentry{VO}{\glossaryentrynumbers{\relax + \setentrycounter[]{page}\glsnumberformat{4}\delimN + \setentrycounter[]{page}\glsnumberformat{6}}}% +\end{theglossary}\glossarypostamble From 36999c394bcdfc9c6dfef6a6a172c9386b598726 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20Kh=C3=A9lifi?= Date: Mon, 4 Aug 2025 17:04:29 +0200 Subject: [PATCH 11/11] Editing --- HighEnergyObsCoreExt.tex | 62 +++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 32 deletions(-) diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index d0279f6..63e472d 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -55,8 +55,8 @@ \newacronym{ANTARES}{ANTARES}{Astronomy with a Neutrino Telescope and Abyss Environmental Research} \newacronym{GW}{GW}{Gravitational wave} \newacronym{WCD}{WCD}{Water Cherenkov Detector} -\newacronym{STI}{STI}{stable time interval} -\newacronym{GTI}{GTI}{good time interval} +\newacronym[plural=STIs]{STI}{STI}{stable time interval} +\newacronym[plural=GTIs]{GTI}{GTI}{good time interval} \newacronym{FITS}{FITS}{Flexible Image Transport System} \newacronym{ACIS}{ACIS}{Advanced CCD Imaging Spectrometer} \newacronym{HRC}{HRC}{High Resolution Camera} @@ -144,11 +144,11 @@ \subsection{{\em dataproduct\_type}} {\bf event-bundle}: compounded dataset containing an {\bf event-list} and multiple files or other substructures that are products necessary to analyze the event-list. Data in an event-bundle may thus be used to produce higher level data products such as images or spectra when containing \glspl{IRF}. \end{quote} -It may be worth mentioning that the term ``event'' caused confusion in the past, as it also is used for astrophysical events like supernova explosions (e.g. VOEvent), and that is not the type of event that is being described here, which are particle detection events. Using "event-list" was meant to help to resolve this ambiguity. +It may be worth mentioning that the term ``event'' caused confusion in the past, as it also is used for astrophysical events like supernova explosions ({\em e.g.\/} VOEvent), and that is not the type of event that is being described here, which are particle detection events. Using "event-list" was meant to help to resolve this ambiguity. An {\bf event-bundle} might for example consist of an {\bf event-list} and the associated {\bf response-functions} (see below) used to calibrate the dataset; alternatively an {\bf event-bundle} may include the {\bf event-list} and associated data products necessary for the user to create the {\bf response-functions} (for those X-ray cases where detailed knowledge of the scientific use case — for example, the user’s selection of events — may be required to compute the responses).\\ -In addition to {\em dataproduct\_type} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (with {\em calib\_level} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime (e.g., Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and ob- +In addition to {\em dataproduct\_type} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (with {\em calib\_level} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and ob- servations) means that data providers may choose to provide such analysis products directly to the end user. For example, the Chandra Source Catalog includes 38 types of advanced data products (for a total of $\sim$90 million files) and $\sim$50\% of these data product types are not well represented by a {\em dataproduct\_type} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type} (or {\em dataproduct\_subtype}) terms for these advanced data products, and note that these terms will certainly be useful independent of waveband (i.e., they can be equally applicable to UV/optical, IR, and radio datasets): @@ -166,7 +166,7 @@ \subsection{{\em dataproduct\_type}} \end{quote} -The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors (e.g., {\bf measurements associated with sources detected in a single observation field}). We propose to delete the caveat associated with {\bf dataproduct\_type} = ``measurements'' in the ObsCore IVOA Recommendation (\S4.1.1) that requires the derived data products be exposed ``together with the progenitor observation dataset''.\\ +The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, {\bf measurements associated with sources detected in a single observation field}). We propose to delete the caveat associated with {\bf dataproduct\_type} = ``measurements'' in the ObsCore IVOA Recommendation (\S4.1.1) that requires the derived data products be exposed ``together with the progenitor observation dataset''.\\ Note that these terms will be repeated in the section \ref{sec:voc}, as mentioned in the introduction of this sub-section. @@ -182,11 +182,13 @@ \subsection{{\em calib\_level}} However, some \gls{HEA} {\bf event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. In X-ray, this is typically done because the {\bf response-funtion}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. -On the other hand, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. Also, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis ObsCore {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). +On the other hand, other {\bf event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf event-list} is considered to be ``uncalibrated'' according to normal usage for their data products and set {\em calib\_level\/} = 2 if an {\bf event-list} is considered to be ``calibrated'' according to normal usage for their data products. + +Also, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis ObsCore {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). \subsection{{\em access\_url}} -Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file (e.g. to the {\bf event-list} or an {\bf event-bundle}), or to a DataLink service that will provide links to the data and to associated data (e.g. response functions). +Given the complexity and number of HE data products, the {\em access\_url} may point either directly to a file ({\em e.g.\/} to the {\bf event-list} or an {\bf event-bundle}), or to a DataLink service that will provide links to the data and to associated data ({\em e.g.\/} response functions). If DataLink is provided, it should be indicated that the URL points to a Datalink service via the {\em access\_format} = application/x-votable+xml;content\\=datalink. @@ -194,56 +196,52 @@ \subsection{{\em access\_url}} \subsection{{\em access\_format}} -The {\em access\_format\/} attribute specifies the format of the data product when downloaded as a file from the {\em access\_url\/}. The analysis of HE data often requires use of multiple, related data products, for example an {\bf event-list} combined with associated IRFs or ancillary files that can be employed by the user to create IRFs. These associated products are often bundled together with the {\bf event-list} and we proposed in \S~\ref{sec:dataproduct_type} to assign such bundles {\em dataproduct\_type\/} = {\bf event-bundle}. While these bundles are typically not standardized across different projects, knowledge of the bundle content is useful for client applications to properly handle the bundles (for example to send the data to an appropriate visualization tool). This is readily achieved by encoding an appropriate MIME-type using the {\em access\_format\/} attribute. In Section~\ref{sec:mimetypes} we propose additional MIME-types for some common {\bf event-bundle}s. +The {\em access\_format\/} attribute specifies the format of the data product when downloaded as a file from the {\em access\_url\/}. The analysis of \gls{HEA} data often requires use of multiple, related data products, for example an {\bf event-list} combined with associated \glspl{IRF} or ancillary files that can be employed by the user to create \glspl{IRF}. These associated products are often bundled together with the {\bf event-list} and we proposed in \S~\ref{sec:dataproduct_type} to assign such bundles {\em dataproduct\_type\/} = {\bf event-bundle}. While these bundles are typically not standardized across different projects, knowledge of the bundle content is useful for client applications to properly handle the bundles (for example to send the data to an appropriate visualization tool). This is readily achieved by encoding an appropriate MIME-type using the {\em access\_format\/} attribute. In Section~\ref{sec:mimetypes} we propose additional MIME-types for some common {\bf event-bundle}s. \subsection{{\em s\_ra\/}/{\em s\_dec}} -We propose that the attributes {\em s\_ra\/}/{\em s\_dec} be redefined to be the ICRS right ascension and ICRS declination of ``a reference position (typically the center)'' of an observation on the sky, rather than the ICRS right ascension and ICRS declination of ``the center'' of the observation. The center of an observation often is not useful for advanced data products that may be extracted from a cut-out from the progenitor observation, and many facilities allow an instrument to be displaced from the optical axis of the telescope, which means that the definition of ``the center'' of an observation may be unclear (especially for facilities for which the PSF varies strongly across the telescope field of view). +We propose that the attributes {\em s\_ra\/}/{\em s\_dec} be redefined to be the ICRS right ascension and ICRS declination of ``a reference position (typically the center)'' of an observation on the sky, rather than the ICRS right ascension and ICRS declination of ``the center'' of the observation. The center of an observation often is not useful for advanced data products that may be extracted from a cut-out from the progenitor observation, and many facilities allow an instrument to be displaced from the optical axis of the telescope, which means that the definition of ``the center'' of an observation may be unclear (especially when the tracking is not fixed in the ICRS system). -For non-pointing instruments (which may include all-sky instruments such as KM3NeT or HAWC) these fields are poorly defined (as is the case, generally for observations that are drift scans). For the time duration of the observation, one can compute an effective center position of the exposure skymap and the maximum radius of the covered area ({\em i.e.\/}, for an all-sky instrument this would be $2\pi\,\rm Sr$ solid angle in Alt/Az, which can be converted into a rotated area in RA/Dec). However, the utility of such a characterization depends on both the duration of the observation and the use case. +For non-pointing instruments (which may include all-sky instruments such as KM3NeT or HAWC), these fields are poorly defined (as is the case, generally for observations that are drift scans). For the time duration of the observation, one can compute an effective center position of the exposure skymap and the maximum radius of the covered area ({\em i.e.\/}, for an all-sky instrument this would be $2\pi\,\rm Sr$ solid angle in Alt/Az, which can be converted into a rotated area in RA/Dec). However, the utility of such a characterization depends on both the duration of the observation and the use case. \subsection{{\em s\_calib\_status}} -We propose that {\em s\_calib\_status} encode the calibration status of an {\bf event-list} dataset's spatial axes. Where multiple spatial axes are included in a dataset ({\em e.g.\/}, physical detector pixel coordinates, virtual detector coordinates corrected for distortions, world coordinates) then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated spatial axes) to define {\em s\_calib\_status\/}. +We propose that {\em s\_calib\_status} encode the calibration status of an {\bf event-list} dataset's spatial axes. Where multiple spatial axes are included in a dataset ({\em e.g.\/}, physical detector pixel coordinates, virtual detector coordinates corrected for distortions, world coordinates), then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated spatial axes) to define {\em s\_calib\_status\/}. -Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. +Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list}, we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. -For dataset types that do not encode sky coordinates, we suggest setting this value to NULL\null. +For dataset types that do not encode sky coordinates, we suggest setting this value to ``NULL''. \subsection{{\em t\_calib\_status}} -We propose that {\em t\_calib\_status} encode the calibration status of an {\bf event-list} dataset's time axis. Where multiple time axes are included in a dataset ({\em e.g.\/}, instrument counter, absolute time) then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated time axis) to define {\em t\_calib\_status\/}. +We propose that {\em t\_calib\_status} encode the calibration status of an {\bf event-list} dataset's time axis. Where multiple time axes are included in a dataset ({\em e.g.\/}, instrument counter, absolute time), then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated time axis) to define {\em t\_calib\_status\/}. -Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. +Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list}, we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. -For dataset types that do not not encode time coordinates, we suggest setting this value to NULL\null. +For dataset types that do not not encode time coordinates, we suggest setting this value to ``NULL''. \subsection{{\em em\_calib\_status}} -We propose that {\em em\_calib\_status} encode the calibration status of an {\bf event-list} dataset's spectral axis. Where multiple spectral axes are included in a dataset ({\em e.g.\/}, PHA, PI, energy) then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated spectral axis) to define {\em t\_calib\_status\/}. +We propose that {\em em\_calib\_status} encode the calibration status of an {\bf event-list} dataset's spectral axis. Where multiple spectral axes are included in a dataset ({\em e.g.\/}, PHA, PI, energy), then we recommend that the data provider use the coordinate system that is most likely to be preferred by the end user (typically the most fully calibrated spectral axis) to define {\em t\_calib\_status\/}. -Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list} we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. +Under the (reasonable) assumption that an end-user searching for {\bf event-bundle} datasets is typically querying based on the properties of the primary {\bf event-list}, we suggest that those values also be used for the {\bf event-bundle}. However, the data provider should ultimately decide which value best describes their {\bf event-bundle} dataset. -For dataset types that do not encode spectral coordinates, we suggest setting this value to NULL\null. +For dataset types that do not encode spectral coordinates, we suggest setting this value to ``NULL''. \subsection{{\em o\_ucd}} -For an {\bf event-list} we can consider that all measures stored in column values are observables. This is {\em the\/} fundamental difference between HEA {\bf event-list}s and typical pixelated datasets. The current ObsCore Recommendation suggests that {\em o\_ucd\/} be set to NULL for event lists; however this significantly hampers data discovery for HEA datasets: since the data content of {\bf event-list}s may vary significantly from facility to facility, meaningful discovery of HEA datasets {\em requires\/} the user be able to query the UCDs of the set of observables included in an {\bf event-list}. +For an {\bf event-list}, we can consider that all measures stored in column values are observables. This is {\em the\/} fundamental difference between \gls{HEA} {\bf event-list}s and typical pixelated datasets. The current ObsCore Recommendation suggests that {\em o\_ucd\/} be set to ``NULL'' for event lists. However this significantly hampers data discovery for \gls{HEA} datasets. Since the data content of {\bf event-list}s may vary significantly from facility to facility, meaningful discovery of \gls{HEA} datasets {\em requires\/} the user be able to query the UCDs of the set of observables included in an {\bf event-list}. A natural way of doing this that is consistent with current usage would be to extend {\em o\_ucd\/} to allow specification of {\em multiple\/} observables for {\bf event-list}s (and {\bf event-bundle}s, for example, {\em o\_ucd\/} = {\em pos.eq,time,phys.pulseHeight\/}. -%%% Can somebody please check that the UCDs in the line above are sensible? -Note that real {\bf event-list}s may include an extensive set of columns ({\em e.g.\/}, a {\em Chandra\/} ACIS Level~1 {\bf event-list} includes $\sim\!20$ columns, depending on observing mode) and several columns may represent similar (but not identical) observables ({\em e.g.\/}, event position in detector pixel coordinates, projected onto the focal surface, corrected for geometric distortions, corrected for spacecraft dither motion, mapped to world coordinates). Currently defined UCDs are not sufficiently fine-grained to be able to differentiate between these various cases but that is likely not be necessary since for data discovery purposes the user is typically interested in the ``most calibrated'' properties in each of the spatial/spectral/time(/polarization) axes ({\em e.g.\/}, world coordinates in the above example). +Note that real {\bf event-list}s may include an extensive set of columns ({\em e.g.\/}, a {\em Chandra\/} ACIS Level~1 {\bf event-list} includes $\sim\!20$ columns, depending on observing mode) and several columns may represent similar (but not identical) observables ({\em e.g.\/}, event position in detector pixel coordinates, projected onto the focal surface, corrected for geometric distortions, corrected for spacecraft dither motion, mapped to world coordinates). Currently defined UCDs are not sufficiently fine-grained to be able to differentiate between these various cases. But that is likely not be necessary, since for data discovery purposes the user is typically interested in the ``most calibrated'' properties in each of the spatial/spectral/time(/polarization) axes ({\em e.g.\/}, world coordinates in the above example). -In the example {\em o\_ucd\/} above, the example UCD {\em phys.pulseHeight\/} is used to represent the detector Pulse Height Amplitude (PHA). There is currently no UCD defined for a raw measure like PHA, but we propose the addition of {\em phys.pulseHeight\/} to the UCDList vocabulary, together with other UCDs that are relevant for HEA data, in Section~\ref{sec:UCDs}. +In the example {\em o\_ucd\/} above, the example UCD {\em phys.pulseHeight\/} is used to represent the detector Pulse Height Amplitude (PHA). There is currently no UCD defined for a raw measure like PHA, but we propose the addition of {\em phys.pulseHeight\/} to the UCDList vocabulary, together with other UCDs that are relevant for \gls{HEA} data, in Section~\ref{sec:UCDs}. -Advanced data products may similarly record multiple observables that can only be differentiated through their UCDs. For example, a {\em Chandra\/} Source Catalog {\bf pdf} dataset for a detection may include multiple marginalized probability density functions computed using a Bayesian X-ray aperture photometry algorithm in units of net counts, net count rates, photon fluxes, and energy fluxes in multiple apertures. The observables recorded in the different MPDFs may be distinguished by their UCDs which then become relevant for data discovery when a user is searching for specific aperture photometry datasets. +Advanced data products may similarly record multiple observables that can only be differentiated through their UCDs. For example, a {\em Chandra\/} Source Catalog {\bf pdf} dataset for a detection may include multiple marginalized probability density functions computed using a Bayesian X-ray aperture photometry algorithm in units of net counts, net count rates, photon fluxes, and energy fluxes in multiple apertures. The observables recorded in the different MPDFs may be distinguished by their UCDs which then become relevant for data discovery when a user is searching for specific aperture photometry datasets. Finally, we note that extending {\em o\_ucd\/} to allow specification of multiple observables would require similar adjustments to the other observable axis attributes {\em o\_unit}, {\em o\_calib\_status}, and {\em o\_stat\_err}. -%\subsection{{\em facility\_name}} - -%\subsection{{\em instrument\_name}} \subsection{{\em proposal\_id}} @@ -265,11 +263,11 @@ \subsection{{\em s\_ref\_energy\/}/{\em em\_ref\_energy\/}/{\em s\_ref\_oaa\/}/{ \subsection{{\em t\_intervals}} -The global time bounds described by {\em t\_min\/}/{\em t\_max} in general are not sufficiently flexible when representing HEA datasets and advanced data products from any waveband. The former are typically composed of many \gls{STIs}/\gls{GTIs}, where data are only valid during the stable or good intervals, while advanced data products may be constructed from multiple progenitor observations that can span decades from the start time of the first observations to the stop time of the last observation (albeit very sparsely). For both cases, data queries using only {\em t\_min\/}/{\em t\_max} may not be adequate to determine whether useful scientific data coincide with a transient cosmic phenomenon. In such cases, a more detailed knowledge of the observation time coverage is necessary. We propose to add a new optional attribute {\em t\_intervals} that would contain the list of observation intervals or STIs/GTIs as a TMOC description following the \gls{MOC} IVOA standard \citep{2022ivoa.spec.0727F}. This element could then be compared across data collections to make the data set selection via simple intersection or union operations in TMOC representation. +The global time bounds described by {\em t\_min\/}/{\em t\_max} in general are not sufficiently flexible when representing HEA datasets and advanced data products from any waveband. The former are typically composed of many \glspl{STI}/\glspl{GTI}, where data are only valid during the stable or good intervals, while advanced data products may be constructed from multiple progenitor observations that can span decades from the start time of the first observations to the stop time of the last observation (albeit very sparsely). For both cases, data queries using only {\em t\_min\/}/{\em t\_max} may not be adequate to determine whether useful scientific data coincide with a transient cosmic phenomenon. In such cases, a more detailed knowledge of the observation time coverage is necessary. We propose to add a new optional attribute {\em t\_intervals} that would contain the list of observation intervals or STIs/GTIs as a TMOC description following the \gls{MOC} \gls{IVOA} standard \citep{2022ivoa.spec.0727F}. This element could then be compared across data collections to make the data set selection via simple intersection or union operations in TMOC representation. \subsection{{\em energy\_min\/}/{\em energy\_max\/}} -The existing attributes {\em em\_min\/} and {\em em\_max\/} that define the coverage of the spectral axis (defined as wavelength expressed in units of m) are not user friendly for HEA where datasets are generally selected according to an energy range ({\em i.e.\/}, inverse wavelength) in units of eV (or scaled units of eV, for example keV, MeV, GeV, TeV, PeV). Unlike the radio domain where $\lambda = c/\nu$, where $c$ is an almost universally remembered physical constant, the conversion $\lambda = hc/E$ is not simple for the user to express. As the spectral range covered by HE data is many decades larger than for other wavebands, the accurate numerical representations of typical HE spectral ranges as {\em em\_min\/}/{\em em\_max\/} requires quantities with many digits of precision and exponents ranging from $\sim\!10^{-5}$--$10^{-22}$. Since specification of the spectral range is largely fundamental to data discovery in the HE regime, we propose to add attributes {\em energy\_min\/} and {\em energy\_max\/} that specify the minimum and maximum spectral range values in units of eV\null. Note that the sense of these attributes is {\em opposite\/} that of {\em em\_min\/} and {\em em\_max\/} because of the inverse wavelength relationship between energy and wavelength, so numerical comparisons must be transposed ({\em e.g.\/}, $E>E_{\rm thresh}$ becomes $\lambdaE_{\rm thresh}$ becomes $\lambda