citiesapps_com: Update to citiesapp v2 api

[lbloder]

Fixes #3460

Updates citiesapp related code to use v2 api.
Also updates wizard to be compatible.

@mampfes Thank you and all the contributors for your work on this integration. Its much appreciated. Also note that I am not a python developer, so any suggestions are welcome :)

[5ila5]

Looks like the Excption call is wrong (from before you touched it) it should be

        raise SourceArgumentNotFoundWithSuggestions(
            "city", search, [city["name"] for city in cities]
        )

in get_specific_citiy

and

        raise SourceArgumentNotFoundWithSuggestions(
            "calendar", search, [calendar["street"] for calendar in calendars]
        )

here.

[5ila5]

Some of the test cases seem to fail

➜ python ./custom_components/waste_collection_schedule/waste_collection_schedule/test/test_sources.py -s 

citiesapps_com -t
Testing source citiesapps_com ...
  Fürstenfeld Haushalt Altenmarkt failed: We could not find values for the argument 'calendar' with the value 'Haushalt Altenmarkt', you may want to use one of the following: ['A.-Deutsch-Straße', 'A.-Haubitzer-Straße', 'Aindlingerstraße', 'Altessestraße', 'Amtmannweg', 'Angerstraße', 'Anton-Rath-Weg', 'Auenbruggerstraße', 'Augasse', 'Augustinerplatz']
    Traceback (most recent call last):
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/./custom_components/waste_collection_schedule/waste_collection_schedule/test/test_sources.py", line 135, in test_fetch
        result = source.fetch()
                 ^^^^^^^^^^^^^^
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/custom_components/waste_collection_schedule/waste_collection_schedule/source/citiesapps_com.py", line 73, in fetch
        garbage_plans = cities_apps.fetch_garbage_plans(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/custom_components/waste_collection_schedule/waste_collection_schedule/service/CitiesAppsCom.py", line 1493, in fetch_garbage_plans
        specific_calendar = self.get_specific_calendar(city_id, calendar)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/custom_components/waste_collection_schedule/waste_collection_schedule/service/CitiesAppsCom.py", line 1477, in get_specific_calendar
        raise SourceArgumentNotFoundWithSuggestions(
    waste_collection_schedule.exceptions.SourceArgumentNotFoundWithSuggestions: We could not find values for the argument 'calendar' with the value 'Haushalt Altenmarkt', you may want to use one of the following: ['A.-Deutsch-Straße', 'A.-Haubitzer-Straße', 'Aindlingerstraße', 'Altessestraße', 'Amtmannweg', 'Angerstraße', 'Anton-Rath-Weg', 'Auenbruggerstraße', 'Augasse', 'Augustinerplatz']
    
  found 126 entries for Buch - St. Magdalena Buch - St. Magdalena
  Rudersdorf Rudersdorf 3 failed: We could not find values for the argument 'calendar' with the value 'Rudersdorf 3', We could not find any suggestions. Please also check other arguments.
    Traceback (most recent call last):
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/./custom_components/waste_collection_schedule/waste_collection_schedule/test/test_sources.py", line 135, in test_fetch
        result = source.fetch()
                 ^^^^^^^^^^^^^^
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/custom_components/waste_collection_schedule/waste_collection_schedule/source/citiesapps_com.py", line 73, in fetch
        garbage_plans = cities_apps.fetch_garbage_plans(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/custom_components/waste_collection_schedule/waste_collection_schedule/service/CitiesAppsCom.py", line 1493, in fetch_garbage_plans
        specific_calendar = self.get_specific_calendar(city_id, calendar)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/silas/Documents/coding/python/hacs_waste_collection_schedule/custom_components/waste_collection_schedule/waste_collection_schedule/service/CitiesAppsCom.py", line 1477, in get_specific_calendar
        raise SourceArgumentNotFoundWithSuggestions(
    waste_collection_schedule.exceptions.SourceArgumentNotFoundWithSuggestions: We could not find values for the argument 'calendar' with the value 'Rudersdorf 3', We could not find any suggestions. Please also check other arguments.
    
  found 262 entries for Lieboch

I don't think we can fix existing configurations with old calendar names, but could you fix the test cases?

[lbloder]

Thanks for the feedback @5ila5, will look into it and update the PR.

[overide6]

I made the changed, but after reconfiguration, I still don't get the waste type in the calendar.
what do I need to do to get the waste types into the calendar and then setup the sensors again?
City: Str. Ruprecht an der Raab
Calendar: Unterfladnitz

thx

[lbloder]

@overide6 Thank you for testing this. This seems to be the same issue as for Fürstenfeld, which I already have a fix for. In my initial code I did not handle the pagination, so Unterfladnitz is not returned from the API.

I will update the PR here once I'm done testing.

[lbloder]

Short update from my side:
Looks like Rudersdorf is still using the v1 api. There is an api call which we can use to figure out which garbage collection version is used. I'm currently trying to figure out how to implement both v1 and v2 in a nice manner.

I'm not fully done yet, due to time constraints. I hope to have a working version by Monday.

@overide6 As a quick workaround you can replace:

https://api.v2.citiesapps.com/waste-management/by-city/{city_id}/areas

with

https://api.v2.citiesapps.com/waste-management/by-city/{city_id}/areas?pagination=limit:100

in CitiesAppsCom.py

[overide6]

@lbloder
thank you, but I don't have this line in my CitiesAppsCom.py file located in the "service" folder
https://api.v2.citiesapps.com/waste-management/by-city/{city_id}/areas

could you tell me where to find it?
Thank you

[lbloder]

@5ila5
I reworked the citiesapps_com source and corresponding service to handle both v1 and v2 cities. It would be great if you could have another look. I tried to keep the separation between source and service, if you have suggestions for improvements, I'd be happy to adapt the code.

@overide6 maybe you could try the new implementation to see if that works for you.

[overide6]

@lbloder
I now see the waste types in the calendar again, so it seems to work. thank you.

[5ila5]

Type hinted as returning list, but returns dict[str, bool|list]

[lbloder]

Ah, thank you, will update.

[5ila5]

The Rudersdorf Rudersdorf 3 test case seems to fail. If this test case is not compatible with the api_v2, and it cannot be made to work, it's best to remove it and replace it with a working one

[lbloder]

The Rudersdorf Rudersdorf 3 test case seems to fail. If this test case is not compatible with the api_v2, and it cannot be made to work, it's best to remove it and replace it with a working one

Looks like they migrated to the calendar v2 api in the meantime 🤦 . This was basically the reason I built it so we can handle both v1 and v2 :).
I'll keep the v1 compatibility in the code, not sure when they will migrate all cities to v2. Maybe I can find another city that still uses v1.
I'll update the test case.

[lbloder]

Updated the Rudersdorf test case and added a suggestion to re-check the App if no calendar could be found.

Also, looks like they are migrating all cities to v2, as of this writing, there are only 7 cities left on v1. Did some spot checking and it seems like they don't return any up-to-date data. Guessing they will also be migrated to v2 when new data is available.

Once all cities are on v2 we can probably remove all the v1 code.

[5ila5]

Sounds good. I'll merge this, and we can do this in a new PR when they moved over.