Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LIDAR ppi converation in gridded format. #1720

Open
priya1809 opened this issue Jan 18, 2025 · 13 comments
Open

LIDAR ppi converation in gridded format. #1720

priya1809 opened this issue Jan 18, 2025 · 13 comments

Comments

@priya1809
Copy link

priya1809 commented Jan 18, 2025

I have LiDAR data that I'd like to convert to radar format (because I want to use pyart package) and then grid it. However, after gridding, I noticed that the ds.rsw values are NaN. Could you please advise on how to retrieve valid values for ds.rsw?

import pyart
import numpy as np
from datetime import datetime
from netCDF4 import Dataset
import warnings
warnings.filterwarnings('ignore')

Load data lidar data

file_path = "D:/lidar/ncfiles/WCS000248_2023-09-23_09-50-36_ppi_40_100m.nc"
data = Dataset(file_path)
sweep_group = data.groups["Sweep_860123"]
time = sweep_group.variables["time"][:]
latitude = data.variables["latitude"][:]
longitude = data.variables["longitude"][:]
altitude = data.variables["altitude"][:]
azimuth = sweep_group.variables["azimuth"][:]
elevation = sweep_group.variables["elevation"][:]
range_ = sweep_group.variables["range"][:]
radial_wind_speed = sweep_group.variables["radial_wind_speed"][:]
rsw = np.array(radial_wind_speed)

Create radar object using pyart

radar = pyart.testing.make_empty_ppi_radar(rsw.shape[1], len(azimuth), 1)

radar.latitude['data'] = np.array([latitude])
radar.longitude['data'] = np.array([longitude])
radar.altitude['data'] = np.array([altitude])
#radar.time['data'] = np.array([(t - time_converted[0]).total_seconds() for t in time_converted])

radar.time = {
'standard_name': 'time',
'long_name': 'time in seconds since volume start',
'calendar': 'gregorian',
'units': 'seconds since 2023-09-23T04:35:05Z',
'comment': 'times are relative to the volume start_time',
'data': np.array([(t - time_converted[0]).total_seconds() for t in time_converted]),
'FillValue':1e+20
}
radar.azimuth['data'] = azimuth
radar.elevation['data'] = elevation
radar.range['data'] = range

radial_wind_speed_dict = {
'long_name': 'radial_wind_speed',
'standard_name': 'radial_wind_speed_of_scatterers_away_from_instrument',
'units': 'm/s',
'sampling_ratio': 1.0,
'_FillValue': -9999 ,
'grid_mapping': 'grid_mapping',
'coordinates': 'time range',
'data': np.ma.masked_invalid(rsw) # Mask invalid data
}
radar.fields = { 'rws': radial_wind_speed_dict}

#success plot this:
import matplotlib.pyplot as plt
from pyart.graph import RadarDisplay
display = RadarDisplay(radar)
fig, ax = plt.subplots(figsize=(10, 8))
display.plot_ppi("rws", sweep=0, ax=ax, cmap="coolwarm")
plt.show()

Now grid

grid_limits = ((10., 4000.), (-4500., 4500.), (-4500., 4500.))
grid_shape = (20, 50, 50)
grid = pyart.map.grid_from_radars([radar], grid_limits=grid_limits, grid_shape=grid_shape)
ds = grid_dv.to_xarray()

print(ds)

print(ds)
<xarray.Dataset> Size: 641kB
Dimensions: (time: 1, z: 20, y: 50, x: 50, nradar: 1)
Coordinates: (12/16)

  • time (time) object 8B 2023-09-23 04:35:05
  • z (z) float64 160B 10.0 220.0 ... 3.79e+03 4e+03
    lat (y, x) float64 20kB 45.02 45.02 ... 45.1 45.1
    lon (y, x) float64 20kB 7.603 7.605 ... 7.715 7.717
  • y (y) float64 400B -4.5e+03 -4.316e+03 ... 4.5e+03
  • x (x) float64 400B -4.5e+03 -4.316e+03 ... 4.5e+03
    ...
    origin_altitude (time) float64 8B nan
    radar_altitude (nradar) float64 8B nan
    radar_latitude (nradar) float64 8B 45.06
    radar_longitude (nradar) float64 8B 7.66
    radar_time (nradar) int64 8B 0
    radar_name (nradar) <U10 40B 'fake_radar'
    Dimensions without coordinates: nradar
    Data variables:
    rws (time, z, y, x) float64 400kB nan nan ... nan
    ROI (time, z, y, x) float32 200kB 500.0 ... 500.0
    Attributes:
    radar_name: fake_radar
    nradar: 1
    instrument_name: fake_radar

np.nanmax(ds.rws)
Out[18]: nan

np.nanmin(ds.rws)
Out[19]: nan

@zssherman
Copy link
Collaborator

@priya1809 Hello, are there values in RWS when you have it in the radar object and when you plot? I see you have #success so double checking? If so, you might need to add a gatefilter and masked the invalid before gridding.

@priya1809
Copy link
Author

priya1809 commented Jan 21, 2025

@priya1809 Hello, are there values in RWS when you have it in the radar object and when you plot? I see you have #success so double checking? If so, you might need to add a gatefilter and masked the invalid before gridding.

data is there, after gridding I am unable to get that data
radar.fields["rws"]
(Out [76]):
{'long_name': 'radial_wind_speed', 'standard_name': 'radial_wind_speed_of_scatterers_away_from_instrument', 'units': 'm/s', 'sampling_ratio': 1.0, '_FillValue': -9999, 'grid_mapping': 'grid_mapping', 'coordinates': 'time range', 'data': masked_array(
data=[[3.78, 3.79, 4.09, ..., -19.07, -9.59, 23.2],
[4.01, 3.57, 3.79, ..., -18.26, 23.44, -17.75],
[3.78, 3.41, 3.88, ..., 6.17, 6.38, -4.49],
...,
[5.25, 5.31, 4.94, ..., -5.03, 19.7, 4.53],
[4.48, 4.95, 4.88, ..., 0.51, -27.85, 8.48],
[4.63, 4.94, 4.84, ..., 5.61, 6.37, 6.23]],
mask=[[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
...,
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False]],
fill_value=1e+20)}

I'm not sure how to apply gatefilter here if I have one field "rws" it. Is there an alternative approach or any guidance on applying Gatefilter in this specific case?

@zssherman
Copy link
Collaborator

A gatefilter within the gridding call?

@priya1809
Copy link
Author

I apologize for the confusion in my previous response. Upon re-examining my process, I realized that I didn't apply the Gatefilter in this case. I re-edited my reply. please check.

@zssherman
Copy link
Collaborator

Try using:

gatefilter = pyart.filters.GateFilter(radar)
gatefilter.exclude_invalid("rws")

grid = pyart.map.grid_from_radars([radar], grid_limits=grid_limits, grid_shape=grid_shape, gatefilters=gatefilter)
ds = grid_dv.to_xarray()

@priya1809
Copy link
Author

Try using:

gatefilter = pyart.filters.GateFilter(radar)
gatefilter.exclude_invalid("rws")

grid = pyart.map.grid_from_radars([radar], grid_limits=grid_limits, grid_shape=grid_shape, gatefilters=gatefilter)
ds = grid_dv.to_xarray()

It still same:
grid_limits = ((10., 4000.), (-4500., 4500.), (-4500., 4500.))
grid_shape = (20, 50, 50)
gatefilter = pyart.filters.GateFilter(radar)
gatefilter.exclude_invalid("rws")

grid = pyart.map.grid_from_radars([radar], grid_limits=grid_limits, grid_shape=grid_shape, gatefilters=gatefilter)
ds = grid.to_xarray()

radar.fields['rws']
Out[79]:
{'long_name': 'radial_wind_speed',
'standard_name': 'radial_wind_speed_of_scatterers_away_from_instrument',
'units': 'm/s',
'sampling_ratio': 1.0,
'_FillValue': -9999,
'grid_mapping': 'grid_mapping',
'coordinates': 'time range',
'data': array([[ 3.78, 3.79, 4.09, ..., -19.07, -9.59, 23.2 ],
[ 4.01, 3.57, 3.79, ..., -18.26, 23.44, -17.75],
[ 3.78, 3.41, 3.88, ..., 6.17, 6.38, -4.49],
...,
[ 5.25, 5.31, 4.94, ..., -5.03, 19.7 , 4.53],
[ 4.48, 4.95, 4.88, ..., 0.51, -27.85, 8.48],
[ 4.63, 4.94, 4.84, ..., 5.61, 6.37, 6.23]])}

np.nanmin(ds['rws'])
Out[80]: nan

np.nanmax(ds['rws'])
Out[81]: nan

@zssherman
Copy link
Collaborator

zssherman commented Jan 22, 2025

What does the grid.fields['rws']['data'] look like? Trying to pinpoint if its the xarray dataset or the grid object.

@priya1809
Copy link
Author

What does the grid.fields['rws']['data'] look like? Trying to pinpoint if its the xarray dataset or the grid object.
grid.fields['rws']['data']
Out[83]:
masked_array(
data=[[[--, --, --, ..., --, --, --],
[--, --, --, ..., --, --, --],
[--, --, --, ..., --, --, --],
...,
[--, --, --, ..., --, --, --],
[--, --, --, ..., --, --, --],
[--, --, --, ..., --, --, --]],

    [[--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     ...,
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --]],

    [[--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     ...,
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --]],

    ...,

    [[--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     ...,
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --]],

    [[--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     ...,
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --]],

    [[--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     ...,
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --],
     [--, --, --, ..., --, --, --]]],

mask=[[[ True, True, True, ..., True, True, True],
[ True, True, True, ..., True, True, True],
[ True, True, True, ..., True, True, True],
...,
[ True, True, True, ..., True, True, True],
[ True, True, True, ..., True, True, True],
[ True, True, True, ..., True, True, True]],

    [[ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     ...,
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True]],

    [[ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     ...,
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True]],

    ...,

    [[ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     ...,
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True]],

    [[ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     ...,
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True]],

    [[ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     ...,
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True],
     [ True,  True,  True, ...,  True,  True,  True]]],

fill_value=1e+20,
dtype=float64)

Please download the file from here.

https://drive.google.com/drive/u/0/folders/1-N323zsaSfNy7C60kqE780j1kf2m-UPQ

@zssherman
Copy link
Collaborator

@priya1809 This could also be a radius of influence issue as well. I'm wondering if when gridding there isn't enough data points when its searching around to interpolate. What does upping the ROI do for you?

@priya1809
Copy link
Author

@priya1809 This could also be a radius of influence issue as well. I'm wondering if when gridding there isn't enough data points when its searching around to interpolate. What does upping the ROI do for you?

grid.fields['ROI']['data']
Out[86]:
array([[[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
...,
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

   [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    ...,
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

   [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    ...,
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

   ...,

   [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    ...,
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

   [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    ...,
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

   [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    ...,
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
    [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]]],
  dtype=float32)

ds['ROI']
Out[87]:
<xarray.DataArray 'ROI' (time: 1, z: 20, y: 50, x: 50)> Size: 200kB
array([[[[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
...,
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

    [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     ...,
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

    [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     ...,

...
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

    [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     ...,
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]],

    [[1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     ...,
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09],
     [1.e+09, 1.e+09, 1.e+09, ..., 1.e+09, 1.e+09, 1.e+09]]]],
  dtype=float32)

Coordinates:

  • time (time) object 8B 2023-09-23 04:35:05
  • z (z) float64 160B 10.0 220.0 ... 3.79e+03 4e+03
    lat (y, x) float64 20kB 45.05 45.05 ... 45.08 45.08
    lon (y, x) float64 20kB 7.641 7.642 ... 7.678 7.679
  • y (y) float64 400B -1.5e+03 -1.439e+03 ... 1.5e+03
  • x (x) float64 400B -1.5e+03 -1.439e+03 ... 1.5e+03
    ProjectionCoordinateSystem int32 4B 1
    projection int32 4B 1
    origin_latitude (time) float64 8B 45.06
    origin_longitude (time) float64 8B 7.66
    origin_altitude (time) float64 8B nan
    Attributes:
    standard_name: radius_of_influence
    long_name: Radius of influence for mapping
    units: m
    least_significant_digit: 1
    _FillValue: -9999.0

@priya1809
Copy link
Author

@priya1809 This could also be a radius of influence issue as well. I'm wondering if when gridding there isn't enough data points when its searching around to interpolate. What does upping the ROI do for you?

@zssherman I've shared the data with you. Could you kindly try processing it? I've provided the exact data that I'm working with now. Your assistance would be greatly appreciated.

@zssherman
Copy link
Collaborator

@priya1809 Talked to some folks, so lidar is outside my expertise, but can I suggest opening a forum here as its more science related:
https://openradar.discourse.group/

There might be some folks who can guide the right parameters for gridding lidar data. I still think this is a ROI issue, but in terms of the right values I'm not 100% sure.

@priya1809
Copy link
Author

@zssherman Thank you so much for your suggestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants