Querry with loading ICON data

[nchawang]

Hi ESMValTool community!

I am fairly new to the ESMValTool and was trying to test loading some of my ICON output runs and use ESMValTool for analysis. I went through the tutorial and got a idea of some of the capabilities of the ESMValTool but still learning bit by bit. To load the ICON model output runs (hourly) I have done the following:

1. Changes in the config-user.yml file
   rootpath:
default: /path/to/the/outputs/
drs: ICON: default
2. Placed the ICON_grid.nc file in the auxiliary directory
3. Changes in the recipe file following here
   datasets:
- {project: ICON, dataset: ICON, exp: icon-AR_CH_RAD, short_name: ta, mip: E1hr, var_type: orig-temp2, raw_name: temp, raw_units: K, latitude: clat, longitude: clon, timerange: 20131107/20131108, horizontal_grid: icon_grid_0014_R02B05_G.nc}
   any help is appreciated!
   thanks!

(I saw in the config-developer.yml file ICON is configures as cmor_type 'CMIP6' so i set the mip as E1hr for hourly input)

when i run the recipe i get the following error:
INFO Found input files for Dataset: ta, E1hr, ICON, ICON, icon-AR_CH_RAD
INFO Starting task map/ta in process [376732]
ERROR Failed to run preprocessor function 'fix_metadata' on the data
... (and finally)
iris.exceptions.CoordinateNotFoundError: "Expected to find exactly 1 'clat' coordinate, but found none."
when i ncdump -h the grid file there is a clat and a clon variable in the grid file. I'm a bit confused if i missed a step of did anything wrong?

any help is deeply appreciated! thanks!

[schlunma]

Hi @nchawang, welcome to the ESMValTool community! Great to have you here!

Whenever you have a specific question on a run, it would be very helpful if you could attach the recipe and (debug) log file.

My guess here is that you specified the wrong values for latitude and longitude. As mentioned in the extra facets table here, these need to point to the standard name of the coordinate (which is identified by the standard_name attribute), which is usually latitude (not clat) for the latitude coordinate and longitude (not clon) for the longitude coordinate. If this is the case, you can omit the facets latitude and longitude altogether, since these are the default values.

In addition, if your grid file is named ICON_grid.nc and is located in the auxiliary data directory, then you need to set horizontal_grid: ICON_grid.nc in your recipe.

[nchawang]

Hi @schlunma ,

Thankyou for your reply and exited to be part of the community!
Ah I see, I have made the corrections and just tried that you suggested and it has moved ahead (thank you!) but has unfortunately struck another error

[839362] INFO     diagnostic,92 Processing variable ta
[839362] INFO     diagnostic,94 Processing dataset ICON
iris.exceptions.MergeError: failed to merge into a single cube.
  cube.standard_name differs: 'air_temperature' != None
  cube.long_name differs: 'Air Temperature' != None
  cube.var_name differs: 'ta' != 'mesh2d_face'
  cube.units differs: Unit('K') != Unit('unknown')
  cube.attributes keys differ: 'CDI_grid_type', 'cf_role', 'location', 'mesh', 'number_of_grid_in_reference', 'param', 'start_index'
  cube.shape differs: (24, 81920) != (81920, 3)
  cube data dtype differs: float32 != int32

Also, i just thought I'd put what I had put in the next steps of the recipe file after loading the data, perhaps I made an error in it somewhere?

preprocessors:
  extract_500hPa_level_from_icon:
    extract_levels:
      levels: 50000
      scheme: linear
      coordinate: air_pressure
diagnostics:
  map:
    description: Global map of temperature
    themes:
      - phys
    realms:
      - atmos
    variables:
      ta:
        mip: E1hr
        preprocessor: extract_500hPa_level_from_icon
        timerange: 2013110703/P1H
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: pcolormesh
          cmap: Reds

To get an understanding of the ESMValTool the Idea I was trying out was, from a multiple days (5 days) hourly global output run. I wanted the ESMValtool to take only 1 day subset (24files) and try to play with the capabilities.

so to start, I took the hint at the input dataset of ICON to add the preprocessors to extract the 500hPA temperature values from the 24h and make a spatial plot any time step (in this case : 20131107 03h).

I dont know if this will help, but I just thought to give a bit of context.

Thanks!

[schlunma]

Great to hear that the initial loading worked. As I already said, it would be very helpful if you could send the recipe and the debug log file. Without that it's very tricky to debug.

It looks like the tool is trying to concatenate the ICON grid file with the actual data, but I can only speculate without the actual log file.

[nchawang]

Oh sorry! I thought i had attached the log and recipe files, but i missed it.

recipe_icon.txt
main_log.txt
main_log_debug.txt
map_script1_log.txt (log from /run/map/scipt1/log.txt)

[schlunma]

Ahh, I get it now! The preprocessing worked (Successfully completed task map/ta (priority 1) in 0:00:13.915621), so you should have the preprocessed file in your preproc output directory.

The thing that fails now is the diagnostic. Those are not very standardized across the tool and might be very sensitive to the input data. One problem here is that the diagnostic does not work with an unstructured grid. An example that works with that is the Psyplot diagnostic (make sure to set ugrid: false in the recipe for that for the ICON data). Alternatively, you can use the regrid preprocessor to regrid the data to a regular grid.

What surprises me is that the data seems to contain more than 1hr, even though you asked for P1H in the recipe. This looks like a bug to me.. I noticed that you work on Levante. Would you be okay if I run some tests with your data?

[nchawang]

Ah I see, Sure!
I shall try both the Psyplot and the regrid preprocessor to plot while loading the specified hour or is there a way to subset one time step in the preprocessor/diagnostic since timerange: 2013110703/P1H is not sub setting for the time being?

[schlunma]

I just google this, looks like a T is missing. Can you try timerange: 20131107T03/PT1H? Or alternatively, timerange: 20131107T03/20131107T04? In addition, if you want to average over a specific time period, you can use the climate_statistics preprocessor. Here is a simple example that we currently use to plot map plots of ICON data:

preprocessors:
  create_map_pr:
    custom_order: true
    regrid:
      target_grid: 2x2
      scheme:
        reference: esmf_regrid.schemes:ESMFAreaWeighted
    climate_statistics:
      operator: mean
      period: full
    convert_units:
      units: mm day-1

diagnostics:
  maps_pr:
    description: Map plots of precipitation.
    variables:
      pr:
        mip: Amon
        preprocessor: create_map_pr
    scripts:
      plot:
          plot_folder: "{plot_dir}"
          plot_filename: "{plot_type}_{exp}_{real_name}_{dataset}_{mip}"
          script: monitor/multi_datasets.py
          facet_used_for_labels: alias
          group_variables_by: variable_group
        plots:
          map:
            common_cbar: true
            plot_kwargs:
              default:
                cmap: Blues
                extend: max
                levels: 12

[nchawang]

I see, So first i added the the regrid preprocessor in the original recipe (with the old diagnostic)

  extract_500hPa_level_from_icon:
    extract_levels:
      levels: 50000
      scheme: linear
      coordinate: air_pressure
    regrid:
      target_grid: 2x2
      scheme:
        reference: esmf_regrid.schemes:ESMFAreaWeighted

and tried with both timerange: 20131107T03/PT1H and 20131107T03/20131107T04. After which i got the following error:
ValueError: Cube must be 2-dimensional. Got 3 dimensions.

After this added the climate_statistics in the preprocessing:

    climate_statistics:
      operator: mean
      period: full

After which the tool ran and I got the 500hPa 24h averaged temp plot (yay!), so the singular hourly time step is not being separated out? I've attached the recipe and log file before I averaged it, if it'll help.

recipe_icon.txt
main_log.txt
main_log_debug.txt
log.txt

[schlunma]

Nice! Yes, currently, you calculate the two-day mean over the period timerange: 20131107/20131108. In your recipe, you define timerange twice. Once in the datasets section and one in the diagnostics section. Here, the first takes priority. Thus, either remove the entry in the datasets section or adapt it.

To check if you only got a 1-hour slice, you could remove the climate_statistics preprocessor again and check the output file in the preproc directory, which should only contain 1 time point (right now it contains 24). One you've checked this, you need to re-add the climate_statistics preprocessor since the diagnostics needs 2D data (a 3D dataset where one dimension has length 1 is not fine).