diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index fcbfe79..86eeed1 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -163,7 +163,6 @@ \subsection{{\em dataproduct\_type}} \end{quote} - The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, {\bf measurements associated with sources detected in a single observation field}). We propose to delete the caveat associated with {\bf dataproduct\_type} = ``measurements'' in the ObsCore IVOA Recommendation (\S4.1.1) that requires the derived data products be exposed ``together with the progenitor observation dataset''.\\ @@ -300,18 +299,107 @@ \subsection{{\em event\_type}} \section{Vocabulary Enhancements} \label{sec:voc} +\subsection{Evolution of the Data Product Type vocabulary} +\label{sec:voc_product_type} + While the IVOA Data Product Type Vocabulary (\url{http://www.ivoa.net/rdf/product-type}) provides terms, labels, and descriptions for many types of astronomical data products, there are some additions and changes that are appropriate to better support HEA datasets. -We propose to add vocabulary entries for the data product types outlined in \S~\ref{sec:dataproduct_type} ({\em i.e.\/}, {\bf event-bundle}, {\bf draws}, {\bf pdf}, {\bf region}, {\bf response-function}) and also propose to slightly modify the existing definition of {\bf event-list} so that it aligns more accurately with the definition in \S~\ref{sec:dataproduct_type}. Additionally, we propose to add several more specific entries to the data product type vocabulary that specialize these types (especially {\bf response-function}). Any future revision of the ObsCore Recommendation to use the IVOA Data Product Type Vocabulary in preference to the enumerated values for {\em dataproduct\_type\/} would profit from these additional vocabulary entries. +The IVOA Data Product Type Vocabulary (\url{http://www.ivoa.net/rdf/product-type}) provides terms, labels, and descriptions for many types of astronomical data products. However, there are some additions and changes that are appropriate to better support \gls{HE} datasets. + +%----- commented as redundant with moved text from section 3.1 +%We propose to add vocabulary entries for the data product types outlined in \S~\ref{sec:dataproduct_type} ({\em i.e.\/}, {\bf event-bundle}, {\bf draws}, {\bf pdf}, {\bf region}, {\bf response-function}) and also propose to slightly modify the existing definition of {\bf event-list} so that it aligns more accurately with the definition in \S~\ref{sec:dataproduct_type}. Additionally, we propose to add several more specific entries to the data product type vocabulary that specialize these types (especially {\bf response-function}). Any future revision of the ObsCore Recommendation to use the IVOA Data Product Type Vocabulary in preference to the enumerated values for {\em dataproduct\_type\/} would profit from these additional vocabulary entries. -The proposed vocabulary entries are listed in Table~\ref{tab:dp_vocabulary}. +%----- from section 3.1, moved to section 5 on Vocabularies -Finally, we propose to clarify the terms that use the word ``flux'' in the description (currently {\bf light-curve}, {\bf polarization-resolved-dataset}, {\bf polarized-spectrum}, and {\bf spectrum}) to determine whether they are applicable to HEA data. The issue is that the term ``flux'' is not defined, and the standard astronomical definition of ``flux'' for at least the last century is an energy flux (with SI units $\rm W\,m^{-2}$). This interpretation is bolstered by the statements ``flux or magnitude'' applied to several of the descriptions since optical/IR magnitude and energy flux density are tightly related. However, with this definition many HEA data products (that typically have units of counts) would not satisfy the descriptions of a light-curve or a spectrum (for example), event though common usage in the HEA community would term the corresponding products ``light-curve'' or ``spectrum''. Restating the descriptions to explicitly state ``Particle or energy flux or magnitude'' where appropriate would resolve this ambiguity. +\subsubsection{Event list} -\begin{landscape} -\begin{longtable}{p{0.17\linewidth}p{0.17\linewidth}p{0.50\linewidth}p{0.16\linewidth}} +We first propose to better define an {\bf event-list}: + +\begin{quote} +{\bf event-list}: A dataset containing a collection of observed events, such as incoming \gls{HE} particles, where an event is typically characterized by a spatial position, a time, and a spectral value ({\em e.g.\/}, an energy, a channel, a pulse height) +\end{quote} + +\subsubsection{Response functions} + +We then propose to add the following product types to define the different response functions (e.g. \glspl{IRF}) generally used in \gls{HE}: + +\begin{quote} +{\bf response-function}: A dataset that maps a physical quantity to an observable. Narrower terms can be used to indictate more precisely the response function. +%This term is mainly intended for retrieval. To annotate datasets, use a narrower term. +\end{quote} + +% \begin{quote} +% {\bf response-function}: +% A dataset that represents a mapping from a physical quantity to an observable. For HEA this may be the components of the composite Instrument Response Function (IRF)\footnote{We try to avoid using the term IRF in a normative sense since historical usage across the HEA community (and from facility to facility) varies. In some cases IRF has been used to mean specifically the product of the ARF and RMF, whereas in other cases IRF has been used more generally to mean {\em any\/} instrumental response function regardless of type.} such as an Auxiliary Response File (ARF), Redistribution Matrix File (RMF), Effective Area (AEFF), Energy Dispersion (EDISP), and so on. The Point Spread Function (PSF) is a response-function that is generally applicable across multiple wavebands. While these datasets may generally be represented as an {\em N\/}-dimensional data cube, designating them as {\bf response-function}s enhances data discovery for very common types of HEA dataset. +% \end{quote} + +\begin{quote} +{\bf aeff}: A dataset that records the ``effective area'' of a telescope and/or instrument. The effective area is the geometric area of the telescope and/or instrument reduced by efficiency factors such as reflectivity and vignetting, among other effects.\footnote{\label{fn:dfgamma}Data Formats for Gamma-ray Astronomy, v0.3, 2018 (\url{https://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/irf_components/index.html}).} +\end{quote} + +\begin{quote} +{\bf edisp}: A dataset that records the probability density function for the energy migration as a function of true energy and spatial position.\footref{fn:dfgamma} +\end{quote} + +\begin{quote} +{\bf bkgrate}: A dataset that records the rate of residual atmospheric cosmic-ray events.\footref{fn:dfgamma} +\end{quote} + +\begin{quote} +{\bf psf}: A dataset that records the probability density function of spatial/angular spreading of incident photons from a point source caused by the instrument (detector and/or mirror and/or analysis).\footref{fn:dfgamma}$^{,}$\footnote{I.M. George and R. Yusaf. The OGIP Format For 2-D (image) Point Spread Function Datasets. Technical Report OGIP/92-027, NASA/GSFC, Nov 2011. (\url{https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_027/cal_gen_92_027.pdf}).} +\end{quote} + +\begin{quote} +{\bf arf}: A dataset that records the combined telescope/instrument effective area and detector quantum efficiency as a function of energy.\footnote{\label{fn:ogip92002}I.M. George, K.A. Arnaud and A.F. Tennant. The Calibration Requirements for Spectral Analysis. Technical Report OGIP/92-002, NASA/GSFC, Dec 1998. (\url{https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_002/cal_gen_92_002.pdf}).} +\end{quote} + +\begin{quote} +{\bf rmf}: A dataset that records the probability density function mapping from energy space into detector pulse height (or position) space.\footref{fn:ogip92002}. +\end{quote} + +\subsubsection{Event bundle} + +Some use cases require access to an {\bf event-bundle}, a bundle of datasets that includes the {\bf event-list} and associated data: + +\begin{quote} +{\bf event-bundle}: An event-bundle dataset is a complex object containing an {\bf event-list} and multiple files or other substructures that are products necessary to analyze the {\bf event-list}. Data in an event-bundle may thus be used to produce calibrated, higher level data products such as images or spectra. +\end{quote} + +An {\bf event-bundle} might for example consist of an {\bf event-list} and the associated {\bf response-function}s used to calibrate the dataset. +%; alternatively an {\bf event-bundle} may include the {\bf event-list} and associated ancillary data products necessary for the user to {\em create\/} the {\bf response-function}s (for those cases where detailed knowledge of the scientific use case --- for example, the user's selection of events --- may be required to compute the responses). +It may also contain provenance information, and preview images or plots. + +\subsubsection{Advanced data products} + +In addition to product types that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (with {\em calib\_level\/}${}\ge 3$ that may be generated from astronomical observations. The computational complexity of analyzing \gls{HE} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations) means that data providers may choose to provide such analysis products directly to the end user. + +% should go in use cases maybe? +For example, the {\em Chandra\/} Source Catalog includes 38 types of advanced data products (for a total of $\sim\!90$ million files) and $\sim\!50\%$ of these data product types are not well represented by a {\em dataproduct\_type} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type}s for these advanced data products, and note that these {\em dataproduct\_type}s will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets: + +\begin{quote} +{\bf draws}: A dataset that represents draws computed from a probability distribution, for example the Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable. The draws can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution. +\end{quote} + +\begin{quote} +{\bf pdf}: A dataset that represents the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution. +\end{quote} + +\begin{quote} +{\bf region}: A dataset that includes an encoding of (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary. +\end{quote} + +The {\bf measurements} data product type is quite useful for many different types of advanced data products (that may be derived from multiple observations) but users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, {\bf measurements} associated with sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = {\bf measurements} type in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. + +%----- + +\subsubsection{Summary table} + +The proposed vocabulary entries are listed in Table~\ref{tab:dp_vocabulary} with their label and parents. + +%\begin{landscape} +\begin{longtable}{p{0.3\linewidth}p{0.3\linewidth}p{0.3\linewidth}} \sptablerule -\textbf{Term} & \textbf{Label} & \textbf{Description} & \textbf{Parent}\cr +\textbf{Term} & \textbf{Label} & \textbf{Parent}\cr \sptablerule {\bf aeff} & Effective Area & A dataset that records the ``effective area'' of a telescope and/or instrument. The effective area is the geometric area of the telescope and/or instrument reduced by efficiency factors such as reflectivity and vignetting, among other effects\footnote{\label{fn:dfgamma}Data Formats for Gamma-ray Astronomy, v0.3, 2018 (\url{https://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/irf_components/index.html}).} & \#response-function \cr {\bf arf} &\raggedright Ancillary Response File & A dataset that records the combined telescope/instrument effective area and detector quantum efficiency as a function of energy\footnote{\label{fn:ogip92002}I.M. George, K.A. Arnaud and A.F. Tennant. The Calibration Requirements for Spectral Analysis. Technical Report OGIP/92-002, NASA/GSFC, Dec 1998. (\url{https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_002/cal_gen_92_002.pdf}).} & \#response-function \cr @@ -326,37 +414,97 @@ \section{Vocabulary Enhancements} {\bf response-function} & Response Function & A dataset that maps a physical quantity to an observable. This term is mainly intended for retrieval. To annotate datasets, use a narrower term. & \cr {\bf rmf} &\raggedright Redistribution Matrix File & A dataset that records the probability density function mapping from energy space into detector pulse height (or position) space.\footref{fn:ogip92002} & \#response-function, \#pdf \cr \sptablerule -\caption{IVOA Data Product Type Vocabulary Additions} +\caption{IVOA Data Product Type Vocabulary extension} \label{tab:dp_vocabulary} \end{longtable} -\end{landscape} +%\end{landscape} + + +\subsubsection{Clarification of ``flux'' in some term definitions} + +We propose to clarify the terms that use the word ``flux'' in the description of some terms, currently {\bf light-curve}, {\bf polarization-resolved-dataset}, {\bf polarized-spectrum}, and {\bf spectrum}, to determine whether they are applicable to \gls{HE} data. + +The issue is that the term ``flux'' is not defined, and the standard astronomical definition of ``flux'' +%for at least the last century +is an energy flux (with SI units $\rm W\,m^{-2}$). This interpretation is bolstered by the statements ``flux or magnitude'' applied to several of the descriptions since optical/IR magnitude and energy flux density are tightly related. However, with this definition many \gls{HE} data products (that typically have units of counts) would not satisfy the descriptions of a light-curve or a spectrum, even though those terms are commonly used by the \gls{HE} community. + +Restating the descriptions to explicitly state ``Particle or energy flux or magnitude'' where appropriate would resolve this ambiguity. + + +\subsection{DataLink vocabularies}\label{sec:UCDs} + +For some use cases, we proposed to show the different associated datasets via Datalink. Each Datalink is described by several attributes, including the mandatory {\em semantics} attribute and a {\em content\_qualifier}. +The terms defined for response functions may thus be used to fill the {\em content\_qualifier} attributes, with {\em semantics} = {\bf #calibration}. -\section{UCD Enhancements}\label{sec:UCDs} -\subsection{Pulse Height} +\subsection{UCD Enhancements}\label{sec:UCDs} + +\subsubsection{Pulse Height} For many X-ray and gamma-ray instruments the signal observed in a given detector spectral channel is the result of event counting and would typically be recorded as a Pulse Height Amplitude (PHA), or perhaps a Pulse Invariant (PI) value that is calculated from PHA by applying an appropriate gain calibration. The PHA (or PI) can be related to the incident particle energy by applying the appropriate {\bf response-function}, and higher data calibration level products may replace or augment these values with quantities such as energy, or perhaps particle or energy flux. -The is currently no UCD defined for a raw pulse height amplitude measure like PHA (or PI). PHA is such an important quantity to HEA datasets that we propose adding a new UCD {\em phys.pulseHeight\/} for these raw data values. We note that the background signal (both of instrumental and cosmological origin) may be significant for many HE detectors and so the detected events are unrelated to any observed source on the sky. A current proposed solution suggests using {\em src.var.amplitude;src.var.pulse;stat.uncalib\/} for PHA, but this is not really appropriate since the connection to {\em src\/} (``observed source viewed on the sky'') is misleading and {\em src.var.amplitude\/} is defined as the ``amplitude of variation'' of the source which is a completely separate concept from an astronomical perspective. +There is currently no UCD defined for a raw pulse height amplitude measure like PHA (or PI). PHA is such an important quantity to \gls{HEA} datasets that we propose adding a new UCD {\em phys.pulseHeight\/} for these raw data values. We note that the background signal (both of instrumental and cosmological origin) may be significant for many \gls{HE} detectors and so the detected events are unrelated to any observed source on the sky. A current proposed solution suggests using {\em src.var.amplitude;src.var.pulse;stat.uncalib\/} for PHA, but this is not really appropriate since the connection to {\em src\/} (``observed source viewed on the sky'') is misleading and {\em src.var.amplitude\/} is defined as the ``amplitude of variation'' of the source which is a completely separate concept from an astronomical perspective. -\subsection{Event Type} +A proposed solution suggests using {\em src.var.amplitude;src.var.pulse;stat.uncalib\/} for PHA, but this is not really appropriate since the connection to {\em src\/} (``observed source viewed on the sky'') is misleading and {\em src.var.amplitude\/} is defined as the ``amplitude of variation'' of the source which is a completely separate concept from an astronomical perspective. -For VHE (and GeV) data there is the notion of event type that can be mandatory for some data releases. We propose to add a new UCD {\em instr.evt-type\/} that identifies these data values. -% Needs input from relevant source +\subsubsection{Electromagnetic spectrum description in UCD} +The current definitions for the gamma-ray domains should be corrected and extended to include the \gls{HE}, \gls{VHE} and \gls{UHE} gamma-ray domains. -\section{MIME-type Enhancements}\label{sec:mimetypes} +\subsubsection{Event Type} -\subsection{x-fits-gadf} +For \gls{VHE} (and GeV) data there is the notion of event type that can be mandatory for some data releases. We propose to add a new UCD {\em instr.evt-type\/} that identifies these data values. % Needs input from relevant source -\subsection{x-fits-vodf} -% Needs input from relevant source +\subsubsection{Particles} + +Observations may concern other particles than the one currently described in the UCD list. The following particles could be added: electron/positron, cosmic rays. + +\subsubsection{Statistical UCDs} + +We suggest restricting {\em stat.max}/{\em stat.min} to mean the maximum/minimum statistic and adding new terms {\em stat.upperlimit}/{\em stat.lowerlimit} for upper/lower limits. + +For upper/lower limits, one expects a confidence level to be provided, which could be described by a UCD {\em stat.confidenceLevel}. + +\subsubsection{Evolution of UCD list} + +The proposed UCD entries are listed in Table~\ref{tab:he_ucds} with their descriptions. + +\begin{longtable}{p{0.1\linewidth}p{0.3\linewidth}p{0.6\linewidth}} +\sptablerule +\textbf{Label} & \textbf{UCD word} & \textbf{Description}\cr +\sptablerule +S & em.gamma.hard & Hard gamma ray (500 keV - 100 MeV) \cr +S & em.gamma.he & High-Energy gamma ray (100 MeV - 10 GeV) \cr +S & em.gamma.vhe & Very-High-Energy gamma ray (10 GeV - 100 TeV) \cr +S & em.gamma.uhe & Ultra-High-Energy gamma ray (100 TeV - 10 PeV) \cr +S & phys.pulseHeight & Pulse height amplitude measure \cr +S & phys.particle.electron & Related to electron/positron \cr +S & phys.particle.cosmicray & Related to cosmic rays particles \cr +P & stats.error.negative & Negative statistical error \cr +P & stats.error.positive & Positive statistical error \cr +P & stat.upperlimit & Upper limit \cr +P & stat.lowerlimit & Lower limit \cr +P & stat.confidenceLevel & Level of confidence for a upper/lower limit computation \cr +\sptablerule +\caption{UCD words proposed extension} +\label{tab:he_ucds} +\end{longtable} + -\section{Datalink}\label{sec:datalink} +\subsection{MIME-types Enhancements}\label{sec:mimetypes} +Data files used in the \gls{HE} domain should have appropriate MIME-types, so that they can be shown in ObsCore tables or elsewhere. +Formats based on FITS could thus be declared as: + +\begin{itemize} +\item {\bf x-fits-gadf}: for FITS files following the GADF specification. +% Needs input from relevant source +\item {\bf x-fits-vodf}: for FITS files following the VODF specification. +% Needs input from relevant source +\end{itemize} \pagebreak \printglossaries