(IKEA E1743) - Migrate to zigbee2mqtt MQTT device triggers (mqtt)

[yarafie]

Thank you for taking the time to work on a Pull Request. Your contribution is really appreciated! 🎉
Please don't delete any part of the template, since keeping the provided structure will help maintainers to review your work more rapidly.

Sections marked as * are required and need to be filled in.

Breaking change

Automations using this blueprint may break if you re-download the blueprint.

Proposed change*

With zigbee2mqtt version 2.x, legacy features are planned to be removed and are disabled by default if you have updated to zigbee2mqtt version 2.x.

Further information can be found in in the zigbee2mqtt discussion link below:
Zigbee2MQTT 2.0.0 breaking changes
Koenkk/zigbee2mqtt#24198

This Blueprint has been updated to make use of MQTT device triggers but also allow for the use of legacy entity action sensors (sensor.*_action) if you have legacy enabled (i.e. set to true) in your zigbee2mqtt configuration.

Checklist*

• [Y] I followed sections of the Contribution Guidelines relevant to changes I'm proposing.
• [Y/N] I properly tested proposed changes on my system and confirm that they are working as expected.
• [As Much As Possible] I formatted files with Prettier using the command npm run format before submitting my Pull Request.

[github-actions]

Hey @yarafie, thank you so much for your contribution! 🚀

🔄 We're currently running a few checks to make sure that everything is great with your contribution.
If further actions need to be performed before your contribution can be reviewed, additional guidance will be provided to you in the next comment.

Results are coming soon, stay tuned!

[github-actions]

Hey @yarafie,

❌ It looks like there are some issues with your contribution. Don't worry, here's some additional information and guidance on how to solve them.

• Your files are not properly formatted. Did you remember to run npm run format before submitting your changes?

Please fix reported issues, then submit your updates here. If you have any questions or doubts, you can always contact a project maintainer. :)

Thanks!

[github-actions]

Hey @EPMatt,

✅ Your contribution passed all the checks, awesome!
A maintainer will soon review your submission and provide additional feedback regarding your changes.

Thanks again for dedicating your precious time to this project. 🔥

📝 Updated blueprints included in this PR can be tested by importing them in Home Assistant via the following links.

https://github.com/yarafie/awesome-ha-blueprints/blob/ikea-e1743-device-triggers/blueprints/controllers/ikea_e1743/ikea_e1743.yaml

[EPMatt]

Thank you so much for your contribution @yarafie! Finally your changes made to the original repo 🔥

For anyone following this thread:

• Changes included in this repo and submitted here are confirmed to match 1:1 what included in Update to support mqtt based triggers lsismeiro/awesome-ha-blueprints#33 for the e1743.yaml blueprint.
• @Nicolai- tested the blueprint successfully and reported results here.

Changes approved!

[EPMatt]

@yarafie I've taken care of updating docs, cleaning up comments, improving the description of inputs and including a couple of style improvements to the code that do not impact on the overall functionality or control logic.

Most importantly, I changed the description of this input in such a way that new users are highly discouraged on using it - while still giving time to current ones to migrate to device triggers with no disruptions.

Why such a "harsh" deprecation notice? It looks like Device Triggers were supported by Z2M for quite a long time , so we can safely assume that almost no-one is still running a version that does not support them. The backward compatibility problem here is with our users - which have the controller_entity input configured, and want to update. Hence, here's why we still keep the input as is, but mark it as deprecated.

[yarafie]

@EPMatt Regarding the docs. I noticed each <controller>.yaml has a corresponding <controller>.mdx.
Are these auto-generated or do they need to be updated manually?

[EPMatt]

@yarafie unfortunately they are not auto generated, so I had to update them manually. Auto-generation is on my radar as an improvement 👍🏻

[EPMatt]

Breaking change

Automations using this blueprint may break if you re-download the blueprint.

@yarafie do we have anyone that confirmed that automations break after updating the blueprint code? As far as I can tell, I don't see Breaking Changes - users that now rely on the controller_entity input should still be able to run their old automations with no issues.

[tivoo]

Just came here to say you're doing some legendary work guys! Thank you so much for updating the blueprints I use everyday. Back to this repo for my E1743 and E2001/2002 blueprints and can't wait for the updates to come! Much appreciated.

[yarafie]

Breaking change

Automations using this blueprint may break if you re-download the blueprint.

@yarafie do we have anyone that confirmed that automations break after updating the blueprint code? As far as I can tell, I don't see Breaking Changes - users that now rely on the controller_entity input should still be able to run their old automations with no issues.

@EPMatt Not Really, No one has said anything.
In theory, if a user is using controller_entity even if the blueprint is redownloaded should not affect existing automations since no changes in variables. But if using controller_device since I put a selector not sure if it will affect.

[github-actions]

Hey @EPMatt,

✅ Your contribution passed all the checks, awesome!
A maintainer will soon review your submission and provide additional feedback regarding your changes.

Thanks again for dedicating your precious time to this project. 🔥

📝 Updated blueprints included in this PR can be tested by importing them in Home Assistant via the following links.

https://github.com/yarafie/awesome-ha-blueprints/blob/ikea-e1743-device-triggers/blueprints/controllers/ikea_e1743/ikea_e1743.yaml

[EPMatt]

@EPMatt Not Really, No one has said anything. In theory, if a user is using controller_entity even if the blueprint is redownloaded should not affect existing automations since no changes in variables. But if using controller_device since I put a selector not sure if it will affect.

@yarafie I've just tried and I can confirm that this is a Breaking Change. Moreover, it looks the situation is worse than what I expected:

1. Users that update the blueprint but do not update the automation config will have controller_device set to "".
2. In such situations, the automation will failed to load with error Triggers: Unknown device ''
3. The error seems to be due to a check performed in Home Assistant Core. Device IDs supplied to Device Triggers are checked against the Device Registry
   https://github.com/home-assistant/core/blob/619dee5d9340ba82dbfeaf84c13494066d8da080/homeassistant/components/device_automation/helpers.py#L60-L66
4. Apparently, there's no way to "fool" the check by supplying e.g. an empty string or a formatted string in such a way that the check is passed
5. Device triggers only accept an hard-coded string or !input directive for the device_id attribute. A template is not rendered and interpreted just as a string. Hence, we cannot dynamically supply a value to the Device Trigger, except with a blueprint input.
6. There's no way to dynamically add / remove triggers to the automation based on the blueprint configuration (although it would be a nice to have feature! 👀 )
7. All of this means that the only way to allow an optional device selector in this blueprint is to supply a valid device ID through the default: attribute of the device selector.
8. What Device is always available in every Home Assistant installation? Home Assistant Core! If we provide the hard-coded ID of such device as a default value for the attribute, it's game over.

However, we don't know whether the device ID for Home Assistant Core is somewhat hard-coded, or if it's generated on the fly for each running instance.

@yarafie could you please copy-paste the following template in the Developer Tools -> Template tab and check what's the result?

{{device_id("Home Assistant Core")}}

In this way, we can check whether your HA instance and mine have different device IDs for the "Home Assistant Core" device.

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change.
If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

[yarafie]

@EPMatt Not Really, No one has said anything. In theory, if a user is using controller_entity even if the blueprint is redownloaded should not affect existing automations since no changes in variables. But if using controller_device since I put a selector not sure if it will affect.

@yarafie I've just tried and I can confirm that this is a Breaking Change. Moreover, it looks the situation is worse than what I expected:

1. Users that update the blueprint but do not update the automation config will have controller_device set to "".
2. In such situations, the automation will failed to load with error Triggers: Unknown device ''
3. The error seems to be due to a check performed in Home Assistant Core. Device IDs supplied to Device Triggers are checked against the Device Registry
   https://github.com/home-assistant/core/blob/619dee5d9340ba82dbfeaf84c13494066d8da080/homeassistant/components/device_automation/helpers.py#L60-L66
4. Apparently, there's no way to "fool" the check by supplying e.g. an empty string or a formatted string in such a way that the check is passed
5. Device triggers only accept an hard-coded string or !input directive for the device_id attribute. A template is not rendered and interpreted just as a string. Hence, we cannot dynamically supply a value to the Device Trigger, except with a blueprint input.
6. There's no way to dynamically add / remove triggers to the automation based on the blueprint configuration (although it would be a nice to have feature! 👀 )
7. All of this means that the only way to allow an optional device selector in this blueprint is to supply a valid device ID through the default: attribute of the device selector.
8. What Device is always available in every Home Assistant installation? Home Assistant Core! If we provide the hard-coded ID of such device as a default value for the attribute, it's game over.

However, we don't know whether the device ID for Home Assistant Core is somewhat hard-coded, or if it's generated on the fly for each running instance.

@yarafie could you please copy-paste the following template in the Developer Tools -> Template tab and check what's the result?

{{device_id("Home Assistant Core")}}

Result

null

BTW: I'm running home assistant in a docker

{{device_id("Home Assistant")}}

Result

bf536b8e91584191ce5131ffd063ebcf

In this way, we can check whether your HA instance and mine have different device IDs for the "Home Assistant Core" device.

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change. If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

Maybe split the blueprints into 2 for each controller.
1 for device and one for entity (legacy). Yes overhead but at least things will still work.
ikea_e1743.yaml ikea_e1743_legacy.yaml.

[Nicolai-]

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change.
If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

There is another option, which is MQTT Triggers which is another thing. It could be implemented as non breaking.

But that require an text input, which the user has to fill in with a MQTT topic to listen for.
Ex. "zigbee2mqtt/Hue Dimmer Office"

And then the trigger would look like this.

triggers:
  # trigger for zigbee2mqtt topic
  - trigger: mqtt
    topic: !input mqtt_topic

Since this new field would be a user typed text field, selecting a controller_device is necessary to be able to add the controller_id to the event data.

[yarafie]

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change.
If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

There is another option, which is MQTT Triggers which is another thing. It could be implemented as non breaking.

But that require an text input, which the user has to fill in with a MQTT topic to listen for. Ex. "zigbee2mqtt/Hue Dimmer Office"

And then the trigger would look like this.

triggers:
  # trigger for zigbee2mqtt topic
  - trigger: mqtt
    topic: !input mqtt_topic

Since this new field would be a user typed text field, selecting a controller_device is necessary to be able to add the controller_id to the event data.

Yes. You are correct that can also be done but user will need to correctly type the friendly name or it won't work.

[BlueFyre]

First, thanks for all the effort being put into this. Just wanted to comment that I'm not even sure a typical workflow for a user would be to update their blueprints (unless there's something broken or perhaps they were looking for a feature). So it might make sense to either fork or a breaking change for the sake of ease of use with the latest Z2M. Selecting items from a list would be far easier in my mind

I don't think there's any provision for even knowing that a blueprint was updated or why from the source (only that you can blindly re-import it) so it's been a bit of a gap in HA functionality

[auanasgheps]

I agree with @BlueFyre, it's a manual process and a feature that was recently implemented and I'd bet many people don't even know it exists.

Additionally, HA is warning that automations could actually break:

[yarafie]

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change.
If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

There is another option, which is MQTT Triggers which is another thing. It could be implemented as non breaking.

But that require an text input, which the user has to fill in with a MQTT topic to listen for. Ex. "zigbee2mqtt/Hue Dimmer Office"

And then the trigger would look like this.

triggers:
  # trigger for zigbee2mqtt topic
  - trigger: mqtt
    topic: !input mqtt_topic

Since this new field would be a user typed text field, selecting a controller_device is necessary to be able to add the controller_id to the event data.

@EPMatt @Nicolai-

I may have found a way around this, check the code snippet

The thing is with the code snippet using MQTT messages is that the automation triggers on ANY ‘action’ message and only afterwards filters it via a condition.

..
..
input:
    zigbee_device:
      name: Zigbee Devices
      description: Select the correct Zigbee devices to use.
      selector:
        device:
          integration: mqtt
          multiple: false
..
..
trigger:
  - platform: mqtt
    topic: zigbee2mqtt/+/action
variables:
  device_names: !input zigbee_device
  expected_topics: >
    {% for device_name in device_names %}
      zigbee2mqtt/{{ device_attr(device_name, 'name') }}/action |
    {% endfor %}null
condition:
  - condition: template
    value_template: "{{ trigger.topic in expected_topics }}"
action:
  - variables:
      command: "{{ trigger.payload }}"
  - choose:
      - conditions:
          - "{{ command == '1_single' }}"
        sequence: !input button_one_short_press
..
..

[Nicolai-]

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change.
If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

There is another option, which is MQTT Triggers which is another thing. It could be implemented as non breaking.
But that require an text input, which the user has to fill in with a MQTT topic to listen for. Ex. "zigbee2mqtt/Hue Dimmer Office"
And then the trigger would look like this.

triggers:
  # trigger for zigbee2mqtt topic
  - trigger: mqtt
    topic: !input mqtt_topic

Since this new field would be a user typed text field, selecting a controller_device is necessary to be able to add the controller_id to the event data.

@EPMatt @Nicolai-

I may have found a way around this, check the code snippet

The thing is with the code snippet using MQTT messages is that the automation triggers on ANY ‘action’ message and only afterwards filters it via a condition.

..
..
input:
    zigbee_device:
      name: Zigbee Devices
      description: Select the correct Zigbee devices to use.
      selector:
        device:
          integration: mqtt
          multiple: false
..
..
trigger:
  - platform: mqtt
    topic: zigbee2mqtt/+/action
variables:
  device_names: !input zigbee_device
  expected_topics: >
    {% for device_name in device_names %}
      zigbee2mqtt/{{ device_attr(device_name, 'name') }}/action |
    {% endfor %}null
condition:
  - condition: template
    value_template: "{{ trigger.topic in expected_topics }}"
action:
  - variables:
      command: "{{ trigger.payload }}"
  - choose:
      - conditions:
          - "{{ command == '1_single' }}"
        sequence: !input button_one_short_press
..
..

@yarafie I think there is multiple things talking me way from that aproach, I'm pretty sure it could work tho.

1. What if the base topic is not zigbee2mqtt
2. Now all controller automations would trigger on all "MQTT Action" messages. (Overhead)
3. Debugging traces will get harder, as there will be alot of noise (traces not relating to that automation)

The aproach with MQTT Trigger and the manual user input, I'm not a fan of either, but it has none of the above drawbacks.
The drawback for that one, is as you mentioned above, it relies on the user input to be correct, else it wont work.

Since Z2M delayed removing the legacy triggers, maybe we should wait and see how the changes to "Events" will look in the future.

[yarafie]

If that's not the case, we might need to drop support for legacy controller_entity and consider this as a breaking change.
If anyone else have ideas on how to prevent this breaking change, feel free to post your ideas and suggestions here! Hopefully the context above will help.

There is another option, which is MQTT Triggers which is another thing. It could be implemented as non breaking.
But that require an text input, which the user has to fill in with a MQTT topic to listen for. Ex. "zigbee2mqtt/Hue Dimmer Office"
And then the trigger would look like this.

triggers:
  # trigger for zigbee2mqtt topic
  - trigger: mqtt
    topic: !input mqtt_topic

Since this new field would be a user typed text field, selecting a controller_device is necessary to be able to add the controller_id to the event data.

@EPMatt @Nicolai-
I may have found a way around this, check the code snippet
The thing is with the code snippet using MQTT messages is that the automation triggers on ANY ‘action’ message and only afterwards filters it via a condition.

..
..
input:
    zigbee_device:
      name: Zigbee Devices
      description: Select the correct Zigbee devices to use.
      selector:
        device:
          integration: mqtt
          multiple: false
..
..
trigger:
  - platform: mqtt
    topic: zigbee2mqtt/+/action
variables:
  device_names: !input zigbee_device
  expected_topics: >
    {% for device_name in device_names %}
      zigbee2mqtt/{{ device_attr(device_name, 'name') }}/action |
    {% endfor %}null
condition:
  - condition: template
    value_template: "{{ trigger.topic in expected_topics }}"
action:
  - variables:
      command: "{{ trigger.payload }}"
  - choose:
      - conditions:
          - "{{ command == '1_single' }}"
        sequence: !input button_one_short_press
..
..

@yarafie I think there is multiple things talking me way from that aproach, I'm pretty sure it could work tho.

Just for the sake of discussion comments below

1. What if the base topic is not zigbee2mqtt

can be an input text with default as zigbee2mqtt which is default anyways.
if user changed it they should already kmow what this means. (Advanced User)

2. Now all controller automations would trigger on all "MQTT Action" messages. (Overhead)
3. Debugging traces will get harder, as there will be alot of noise (traces not relating to that automation)

Absolutly correct. No disagreement here

4. The aproach with MQTT Trigger and the manual user input, I'm not a fan of either, but it has none of the above drawbacks. The drawback for that one, is as you mentioned above, it relies on the user input to be correct, else it wont work.

Yes that is why I put the enhanced selector for each for user easy use.
No room for mistakes.

5. Since Z2M delayed removing the legacy triggers, maybe we should wait and see how the changes to "Events" will look in the future.

I agree with all what you said.
I'm going to look at the expetimental events since it can be enabled.
I haven't upgraded yet to z2m 2.0.0 plan to do it this weekend.
Then I can test all scenerios.

[EPMatt]

Hey there,

thanks all for sharing these updates and your thoughts. I took some time to think about this, and I'm more and more leaning towards the "Breaking Change" option.

1. As @BlueFyre and @auanasgheps pointed out, we can safely assume most users do not update blueprints - unless there's some serious bug fix / new feature coming out. Plus, if they do, they are aware that updating a blueprint might break automations. With our online docs we provide a full changelog and information about breaking change and how to migrate, + the community support in this repo is amazing.
2. It looks like MQTT Device Triggers were implemented in Z2M a very long time ago (3-4 years? I can't find the exact release but I've seen them in docs for quite a while). We're not looking at a breaking change that would make anyone not on Z2M 2.0.0 not able to run the blueprint anymore: people on Z2M <2.0.0 would still be able to run the automations by just switching from controller_entity to controller_device input.
3. The approach of maintaining 2 separate blueprints is a good idea for compatibility, but I don't believe the nature of the breaking change motivates the extra effort required to maintain both versions for all the current controllers.
4. Regarding alternative solutions (@yarafie @Nicolai- thank you so much for looking more into those!) I'm concerned about debugging nightmares, and especially overhead of every controller automation being triggered by every action event on the system - for context, we have received reports of people running 20+ controller automations on their HA.

So, at this point, I'd proceed as follows:

• Remove the controller_entity input, mark controller_device input as Required.
• Update the Changelog indicating that this update is a breaking change, and suggesting steps to migrate away from the controller_entity input.
• Add an explicit link to the Changelog online docs in the blueprint description - as an extra reference for people having issues during the upgrade

I'm sure not everyone will be happy with this decision, but hopefully the great discussion we are having here will provide the motivation for the changes, and demonstrate how much we'd like an upgrade process as smooth as possible.

Before proceeding, I'd love to hear back from you guys!

[BlueFyre]

Point 3 sounds reasonable to me, in order to use the latest version an upgrade to the blueprint is necessary and there's no way around that anyhow. Plus if anyone needs the old blueprint they can look to the repo for it (perhaps a git tag would be a good idea)

[yarafie]

@EPMatt

Comments below each point

Hey there,

thanks all for sharing these updates and your thoughts. I took some time to think about this, and I'm more and more leaning towards the "Breaking Change" option.

1. As @BlueFyre and @auanasgheps pointed out, we can safely assume most users do not update blueprints - unless there's some serious bug fix / new feature coming out. Plus, if they do, they are aware that updating a blueprint might break automations. With our online docs we provide a full changelog and information about breaking change and how to migrate, + the community support in this repo is amazing.

I do agree

2. It looks like MQTT Device Triggers were implemented in Z2M a very long time ago (3-4 years? I can't find the exact release but I've seen them in docs for quite a while). We're not looking at a breaking change that would make anyone not on Z2M 2.0.0 not able to run the blueprint anymore: people on Z2M <2.0.0 would still be able to run the automations by just switching from controller_entity to controller_device input.

Yes. It will be a matter of switch the legacy off again.

3. The approach of maintaining 2 separate blueprints is a good idea for compatibility, but I don't believe the nature of the breaking change motivates the extra effort required to maintain both versions for all the current controllers.

I agree.

4. Regarding alternative solutions (@yarafie @Nicolai- thank you so much for looking more into those!) I'm concerned about debugging nightmares, and especially overhead of every controller automation being triggered by every action event on the system - for context, we have received reports of people running 20+ controller automations on their HA.

I agree also.

So, at this point, I'd proceed as follows:

I think we just make the decision announce it and move forward.

• Remove the controller_entity input, mark controller_device input as Required.

I'll update ikea_e1743.yaml sometime today with the changes mentioned above.

• Update the Changelog indicating that this update is a breaking change, and suggesting steps to migrate away from the controller_entity input.

Send me the text of the warning message you want me to put, this I'll put in the changelog at the bottom of ikea_e1743.mdx

• Add an explicit link to the Changelog online docs in the blueprint description - as an extra reference for people having issues during the upgrade

Let me know what and where you want me to put in the ikea_e1743.yaml

I'm sure not everyone will be happy with this decision, but hopefully the great discussion we are having here will provide the motivation for the changes, and demonstrate how much we'd like an upgrade process as smooth as possible.

You can't please everyone :)

Before proceeding, I'd love to hear back from you guys!

[github-actions]

Hey @yarafie, thank you so much for your contribution! 🚀

🔄 We're currently running a few checks to make sure that everything is great with your contribution.
If further actions need to be performed before your contribution can be reviewed, additional guidance will be provided to you in the next comment.

Results are coming soon, stay tuned!

[EPMatt]

@yarafie thanks for your feedback! I've updated blueprint version, added the changelog link and the breaking change notice in the changelog. I leave the removal of controller_entity to you! :)

Don't worry about the failing CI check Analyze changes - apparently dependabot updates broke part of the pipeline. What's relevant for us here (linters and formatters) still run and pass.

[yarafie]

@yarafie thanks for your feedback! I've updated blueprint version, added the changelog link and the breaking change notice in the changelog. I leave the removal of controller_entity to you! :)

Don't worry about the failing CI check Analyze changes - apparently dependabot updates broke part of the pipeline. What's relevant for us here (linters and formatters) still run and pass.

@EPMatt
In the ikea_e1743.mdx
You left the below section.
Shouldn't it be removed since it will no longer be as an input since I will remove controller_entity completly from the ikea_e1743.yaml.

<Input
  name='Controller Entity'
  description='The action sensor of the controller to use for the automation. Choose a value only if the remote is integrated with Zigbe
e2MQTT and Legacy Action Sensors are enabled. This input is deprecated in favor of the Controller Device input, and will be removed in a
 future release.'
  selector='entity'
  deprecated
/>

also in hooks light, cover, media_player there is controller_entity shouldn't it be removed from there also?

[esolitos]

Hi! First of all thanks for the great work on the original blueprints as well as this port to Z2M v2.0!
With that said, what is left to merge this PR? It seems to be a bit stale, so I wonder if I can help move this forward in any way?
I can do a review, test or whatever you need me to get this across the finish line. :)

[yarafie]

Hi! First of all thanks for the great work on the original blueprints as well as this port to Z2M v2.0! With that said, what is left to merge this PR? It seems to be a bit stale, so I wonder if I can help move this forward in any way? I can do a review, test or whatever you need me to get this across the finish line. :)

Waiting for @EPMatt owner of repo and author of the blueprints.
He wiil have to do the merge.

I posted something in the discussions check it out.

[EPMatt]

@yarafie thanks for your patience, and for wrapping up the last changes.
Will quickly review this later today! Hopefully we should be good to go here. 🔥