Problem on NuScense Offical Evaluation

[Marcelll2]

Hi, recently I made some modifications to the MTR model and wanted to test it using the official NuScenes evaluation tool.
At first, I used the v1.0-trainval dataset (which was not converted to the unitraj format), but the test failed. Then, I tried using the v1.0-test dataset, but it failed again.
Here is my configuration:

I have two dataset file: v1.0-trainval and v1.0-test.

The following error occurred again. At first, I believed that the scenarionet-devkit was working fine, but I am still confused about the error.
I would like to ask, how do you perform the official evaluation on NuScenes?

Traceback (most recent call last): File "/home/woody/iwnt/iwnt113h/UniTraj/UniTraj_llm/unitraj/evaluation.py", line 21, in evaluation model = build_model(cfg) File "/home/woody/iwnt/iwnt113h/UniTraj/UniTraj_llm/unitraj/models/__init__.py", line 14, in build_model model = __all__[config.method.model_name]( File "/home/woody/iwnt/iwnt113h/UniTraj/UniTraj_llm/unitraj/models/mtr/MTR.py", line 31, in __init__ super(MotionTransformer, self).__init__(config) File "/home/woody/iwnt/iwnt113h/UniTraj/UniTraj_llm/unitraj/models/base_model/base_model.py", line 21, in __init__ self.init_nuscenes() File "/home/woody/iwnt/iwnt113h/UniTraj/UniTraj_llm/unitraj/models/base_model/base_model.py", line 37, in init_nuscenes self.pred_config5 = PredictionConfig.deserialize(pred_config, self.helper) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/eval/prediction/config.py", line 37, in deserialize return cls([deserialize_metric(metric, helper) for metric in content['metrics']], File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/eval/prediction/config.py", line 37, in <listcomp> return cls([deserialize_metric(metric, helper) for metric in content['metrics']], File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/eval/prediction/metrics.py", line 440, in deserialize_metric return OffRoadRate(helper, [deserialize_aggregator(agg) for agg in config['aggregators']]) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/eval/prediction/metrics.py", line 325, in __init__ self.drivable_area_polygons = self.load_drivable_area_masks(helper) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/eval/prediction/metrics.py", line 342, in load_drivable_area_masks masks[map_name] = map_api.get_map_mask(patch_box=None, patch_angle=0, layer_names=['drivable_area'], File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 392, in get_map_mask return self.explorer.get_map_mask(patch_box, patch_angle, layer_names=layer_names, canvas_size=canvas_size) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 870, in get_map_mask map_mask = self.map_geom_to_mask(map_geom, local_box, canvas_size) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 819, in map_geom_to_mask layer_mask = self._layer_geom_to_mask(layer_name, layer_geom, local_box, canvas_size) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1803, in _layer_geom_to_mask return self._polygon_geom_to_mask(layer_geom, local_box, layer_name, canvas_size) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1890, in _polygon_geom_to_mask map_mask = self.mask_for_polygons(new_polygon, map_mask) File "/home/woody/iwnt/iwnt113h/software/private/conda/envs/unitraj/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1823, in mask_for_polygons exteriors = [int_coords(poly.exterior.coords) for poly in polygons] TypeError: 'MultiPolygon' object is not iterable

Thank you in advance for your help.

[Alan-LanFeng]

Hi,

you need to convert the test data into ScenarioNet format. Please let me know if this doesn't work.