Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add additional docs for importing code with asyncio #2338

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 1 addition & 9 deletions docs/asyncio_blocking_operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,7 @@ When an `open` call running in the event loop is fixed, all the blocking reads a

#### import_module

When importing a module, the import machinery has to read the module from disk which does blocking I/O. Importing modules is both CPU-intensive and involves blocking I/O, so it is crucial to ensure these operations are executed in the executor.

Importing code in [cpython is not thread-safe](https://github.com/python/cpython/issues/83065). If the module will only ever be imported in a single place, the standard executor calls can be used. If there's a possibility of the same module being imported concurrently in different parts of the application, use the thread-safe `homeassistant.helpers.importlib.import_module` helper.

Example:

```python
platform = await async_import_module(hass, f"homeassistant.components.homeassistant.triggers.{platform_name}")
```
See [Importing code with asyncio](asyncio_imports.md)

#### sleep

Expand Down
38 changes: 38 additions & 0 deletions docs/asyncio_imports.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "Importing code with asyncio"
---

Determining when it is safe to import code when using asyncio can be tricky because two constraints need to be considered:

- Importing code can do blocking I/O to load the files from the disk
- Importing code in [cpython is not thread-safe](https://github.com/python/cpython/issues/83065)

## Top-level imports

Suppose your imports happen at the top-level (nearly all code at indentation level 0). Home Assistant will import your code before the event loop starts or import it in the import executor when your integration is loaded. In this case, you likely do not need to consider whether your imports are safe.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the correct term is "module level" since it mentions indentation level 0?
Should we explain there needs to a chain of imports starting from a package's __init__.py?


## Imports outside of top-level

If your imports are not happening at top-level, you must carefully consider each import, as the import machinery has to read the module from disk which does blocking I/O. If possible, it's usually best to change to a top-level import, as it avoids much complexity and the risk of mistakes. Importing modules is both CPU-intensive and involves blocking I/O, so it is crucial to ensure these operations are executed in the executor.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are two reasons for doing non module-level imports:

  • Breaking circular import chains
    • Here, we could maybe mention imports which are only there for the benefit of the type check can be hidden behind an if TYPE_CHECKING: guard
    • We should probably also explain that if the import is imported also via an import chain starting at module-level of __init__.py it's safe to do a local import
  • Avoiding importing a large set of modules which will mostly be unused
    • I think this is the case where it's relevant to manually do imports in the executor?


If you can be sure that the modules have already been imported, using a bare [`import`](https://docs.python.org/3/reference/simple_stmts.html#import) statement is safe since Python will not load the modules again.

If the module will only ever be imported in a single place, the standard executor calls can be used:

- For imports inside of Home Assistant `hass.async_add_executor_job(_function_that_does_late_import)`
- For imports outside of Home Assistant: [`loop.run_in_executor(None, _function_that_does_late_import)`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor)
If the same module may be imported concurrently in different parts of the application, use the thread-safe `homeassistant.helpers.importlib.import_module` helper.

If its possible the module may be imported from multiple different paths, use `async_import_module`:

Example:

```python
from homeassistant.helpers.importlib import async_import_module

platform = await async_import_module(hass, f"homeassistant.components.homeassistant.triggers.{platform_name}")
```

## Determining if a module is already loaded

If you are unsure if a module is already loaded, you can check if the module is already in [`sys.modules`](https://docs.python.org/3/library/sys.html#sys.modules). You should know that the module will appear in `sys.modules` as soon as it begins loading, and [cpython imports are not thread-safe](https://github.com/python/cpython/issues/83065). For this reason, it's important to consider race conditions when code may be imported from multiple paths.
1 change: 1 addition & 0 deletions sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,7 @@ module.exports = {
"asyncio_working_with_async",
"asyncio_thread_safety",
"asyncio_blocking_operations",
"asyncio_imports",
],
},
],
Expand Down