quick summary of windows

[fernando-jose-silva]

Is your feature request related to a problem? Please describe.

After some time teaching computer classes to users, I noticed that one of the difficulties is understanding how the items are organized in the window. This makes it difficult to understand how some programs work. I think that nvda knows how the programs are organized on the screen, if not in their entirety, but it has valuable information that could be provided to the user when requested.

Describe the solution you'd like

When pressing a command on demand, nvda would inform the structure of the window:
(note that my explanation is from someone who has never seen the window and must be wrong).
Let's take the nvda settings window, the reader could inform something like:
window title at the top.
list of categories on the left side.
controls in the center.
buttons at the bottom (I don't know where the ok and cancel buttons are, lol).

This information would only be given when the user requests it through a command key.

Describe alternatives you've considered

Another suggestion would be:
When moving through the category list with the arrows, many users have no idea that the controls displayed in the center of the window have been updated.
The suggestion would be that if the user chooses this announcement, it would be given to him automatically by nvda.
For example, when an item in the category list, nvda would say something like controls in the center of the screen updated. This also applies when tabs are changed or menu items are displayed.
Once again, this option is explicitly enabled by the user, perhaps even in a super detailed usage mode.
Other useful places are in Excel to inform the user that there is a table organized in a grid pattern.
A Word user to inform the user of the existence of the status bar or that the menu bar is at the top of the screen, or even that there is a giant text field where he can type and navigate.
Also an example is in the desktop, where there are icons arranged without much grid organization, where there are other elements such as the toolbar, among several other areas.

Additional context

[michaelDCurran]

Simply using accessibility APIs or win32 APIs, it would be rather costly for NVDA to actually produce this summary. However, these days, AI is becoming very good at producing this information from screenshots.
Today, a blind user can already use the AI feature in Be My Eyes on Windows to have the screen described, providing this kind of detail.
However, I guess there could be scope to integrate this kind of feature more tightly into NVDA for ease of use.

[fernando-jose-silva]

thank you very much

[Emil-18]

@fernando-jose-silva You can download the cloud vision addon at https://nvda-addons.org/addon.php?id=261
It supports be my AI. Move the navigator object to the area you want to be described, for example to the NVDA dialog, and press NVDA+control+i.