Use prompty to store prompts #2178

pamelafox · 2024-11-19T23:14:34Z

Purpose

This PR uses Prompty (https://prompty.ai/) to store prompts.

Now, instead of storing parts of the prompt in variables, everything is inside prompty files. You can even use the VS Code Prompty extension to play with the prompts, and you can even upload the prompty files to the Chat playground in Azure AI Foundry.

To abstract on top of the prompty interface for greater flexibility and testing, this PR adds a PromptManager class with load_prompt, load_tools, and render_prompt. The render_prompt function returns a structure that contains the system message, past messages, few shots, and new user_query, which can then be used by either our token-truncating function or the chat completion call.

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[ ] Yes
[X] No

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[X] No

Type of change

[ ] Bugfix
[X] Feature
[ ] Code style update (formatting, local variables)
[X] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

The current tests all pass (python -m pytest).
I added tests that prove my fix is effective or that my feature works
I ran python -m pytest --cov to verify 100% coverage of added lines
I ran python -m mypy to check for type errors
I either used the pre-commit hooks or ran ruff and black manually on my code.

jeannotdamoiseaux · 2024-11-22T09:44:22Z

@pamelafox - The absence of tutorials or documentation at prompty.ai or in their GitHub repo has made it challenging to implement the desired prompt management structure using Prompty at this time.

pamelafox · 2024-11-22T16:53:30Z

@jeannotdamoiseaux Agreed, they need additional documentation about the format of the prompt itself (the Jinja2 part, under the YAML). I've passed the feedback onto the Prompty creators.

…ai-demo

pamelafox · 2025-01-13T16:55:54Z

app/backend/approaches/approach.py

@@ -205,6 +207,10 @@ async def search(
    def get_sources_content(
        self, results: List[Document], use_semantic_captions: bool, use_image_citation: bool
    ) -> list[str]:
+


I moved this function out of text.py since it was a 2-line file, and the function is only used in this one method.

pamelafox · 2025-01-13T16:57:42Z

app/backend/approaches/chatreadretrieveread.py

        extra_info = {
-            "data_points": data_points,
+            "data_points": {"text": text_sources},


I moved data_points into the dict itself, as we weren't using that variable separately anyway.

pamelafox · 2025-01-13T16:59:20Z

app/backend/approaches/promptmanager.py

+    new_user_content: str
+
+
+class PromptManager:


I made an abstraction for a PromptManager, since some developers weren't as keen on Prompty as others, but I'm not sure if this abstraction would work for other ways of managing prompts anyway, so it might be a premature abstraction? I could remove it and only have PromptyManager for now.

If I do keep only PromptyManager, then I could also just construct it in the init of each approach, instead of constructing it in app.py.

wrapping prompty is all good IMO. Also helps for tests

pamelafox · 2025-01-13T17:01:15Z

app/backend/approaches/promptmanager.py

+    def load_tools(self, path: str):
+        return json.loads(open(self.PROMPTS_DIRECTORY / path).read())
+
+    def render_prompt(self, prompt, data) -> RenderedPrompt:


This was my solution to my conundrum about needing to retrieve the messages back from the rendered prompt using indexes, so that they can then get passed to the token counter.
Now, what I do is that I mark each example with (EXAMPLE), so that I can distinguish examples from actual past messages, and then I can extract them all back out in this function.

So this function extracts:

system

(EXAMPLE) pairs

past messages

new user message

This is also good feedback for prompty as well, thanks for breaking this out

pamelafox · 2025-01-13T17:04:03Z

app/backend/approaches/retrievethenread.py

-        user_content = q + "\n" + f"Sources:\n {content}"
-
-        response_token_limit = 1024
-        updated_messages = build_messages(


I realized that we do not need to call build_messages in the ask approaches, since it only truncates past messages, and we don't pass in chat history to these. So we remove the call and assume that the /ask questions always fit in the context window.

pamelafox · 2025-01-13T17:07:50Z

app/backend/approaches/retrievethenreadvision.py


-        response_token_limit = 1024
-        updated_messages = build_messages(


Same comment here, I removed the build_messages call since it isn't useful when there's no history being passed in.

pamelafox · 2025-01-13T17:08:37Z

app/backend/core/imageshelper.py

    if result.sourcepage:
        img = await download_blob_as_base64(blob_container_client, result.sourcepage)
-        if img:


The default is "auto", so Prompty doesn't even output detail, so we only need to return the data URI. If developers do need to customize the detail in the future, that'd require a Prompty change.

pamelafox · 2025-01-13T17:08:51Z

pyproject.toml

@@ -33,5 +33,6 @@ module = [
    "azure.cognitiveservices.*",
    "azure.cognitiveservices.speech.*",
    "pymupdf.*",
+    "prompty.*",


I've filed an issue requesting types for Prompty

pamelafox · 2025-01-13T17:09:15Z

tests/conftest.py

@@ -319,7 +319,7 @@ def mock_env(monkeypatch, request):
            yield


-@pytest_asyncio.fixture()
+@pytest_asyncio.fixture(scope="function")


There was a recent change to the asyncio plugin that requires explicit scopes for async fixtures

pamelafox · 2025-01-13T17:10:10Z

tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json

@@ -47,19 +47,19 @@
            {
                "description": [
                    {
-                        "content": "You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer",
+                        "content": "You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions.\nUse 'you' to refer to the individual asking the questions even if they ask with 'I'.\nAnswer the following question using only the data provided in the sources below.\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response.\nIf you cannot answer using the sources below, say you don't know. Use below example to answer",


When we use Prompty, we get new lines in the prompt whereveer there are new lines in the template, which makes sense. I would assume the model is unaffected by newlines, but I could make one long line in the prompty files if there's concern about that.

pamelafox · 2025-01-13T17:36:44Z

tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json

                        "role": "system"
                    },
                    {
-                        "content": "\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\n\nSources:\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\ninfo2.pdf: Overlake is in-network for the employee plan.\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\n",
+                        "content": "What is the deductible for the employee plan for a visit to Overlake in Bellevue?\n\nSources:\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\ninfo2.pdf: Overlake is in-network for the employee plan.\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region.",


This is an improvement from before - no more newline at the start, and no unnecessary quotes around the user question.

tests/snapshots/test_app/test_chat_with_history/client1/result.json

tests/snapshots/test_app/test_chat_with_history/client0/result.json

pamelafox · 2025-01-13T22:22:05Z

Per the discussion on the related PR (#2164), many developers seem interested in using Prompty for prompt management. I have made changes to my original PR to add an abstraction layer to make the Prompty rendering more robust, and also potentially make it flexible for other forms of prompt management, since not everyone is as interested in Prompty.

@mattgotteiner
This cannot yet be merged due to Prompty's lack of 3.9 support, but I am working with the Prompty maintainer on getting that fixed ASAP (just type annotation changes), so all the code changes on our side are done, so it's ready for your review.
There are a lot of changes in the tests snapshots, but they are all whitespace/ordering, and I added inline comments for each instance of a change.

app/backend/app.py

app/backend/approaches/chatapproach.py

app/backend/approaches/chatreadretrieveread.py

app/backend/approaches/prompts/ask/answer_question.prompty

mattgotteiner · 2025-01-14T17:35:06Z

app/frontend/src/components/SupportingContent/SupportingContent.tsx

@@ -3,7 +3,7 @@ import { parseSupportingContentItem } from "./SupportingContentParser";
 import styles from "./SupportingContent.module.css";

 interface Props {
-    supportingContent: string[] | { text: string[]; images?: { url: string }[] };
+    supportingContent: string[] | { text: string[]; images?: string[] };


Why did this change?

We would previously construct and pass down [{url: data:bla..., detail: "auto"}] because that was the format that the OpenAI chat completion request wanted as well. However, the Python now only constructs [url, url] because that's all Prompty needs (and OpenAI assumes "default"), so I propagated that change to the frontend.

github-actions · 2025-01-14T19:16:07Z

Check Broken URLs

We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.

Check the file paths and associated broken URLs inside them. For more details, check our Contributing Guide.

File Full Path Issues

docs/customization.md

#	Link	Line Number
1	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_query_rewrite.prompty`	`39`
2	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question.prompty`	`41`
3	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_query_rewrite.prompty`	`43`
4	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question.prompty`	`43`
5	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question_vision.prompty`	`53`
6	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question.prompty`	`60`
7	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question.prompty`	`62`
8	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question_vision.prompty`	`71`

github-actions · 2025-01-14T19:37:06Z

Check Broken URLs

We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.

Check the file paths and associated broken URLs inside them. For more details, check our Contributing Guide.

File Full Path Issues

docs/customization.md

#	Link	Line Number
1	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_query_rewrite.prompty`	`39`
2	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question.prompty`	`41`
3	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_query_rewrite.prompty`	`43`
4	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question.prompty`	`43`
5	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question_vision.prompty`	`53`
6	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question.prompty`	`60`
7	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question.prompty`	`62`
8	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question_vision.prompty`	`71`

github-actions · 2025-01-14T20:13:53Z

Check Broken URLs

We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.

Check the file paths and associated broken URLs inside them. For more details, check our Contributing Guide.

File Full Path Issues

docs/customization.md

#	Link	Line Number
1	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_query_rewrite.prompty`	`39`
2	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question.prompty`	`41`
3	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_query_rewrite.prompty`	`43`
4	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question.prompty`	`43`
5	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/chat_answer_question_vision.prompty`	`53`
6	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question.prompty`	`60`
7	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question.prompty`	`62`
8	`https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/prompts/ask_answer_question_vision.prompty`	`71`

pamelafox · 2025-01-14T20:25:15Z

Broken URLs should work once this is merged.

jeannotdamoiseaux and others added 5 commits November 18, 2024 16:19

move-prompts-to-jinja-templates

f69861d

refactor: convert few_shots to JSON format and clean up comments

eda6292

Clean retreivethenread.py

e25468c

Merge branch 'main' into refactor/move-prompts-to-jinja-templates

4063767

Port to prompty

c78ffed

pamelafox mentioned this pull request Nov 19, 2024

refactor/move-prompts-to-jinja-templates #2164

Closed

5 tasks

pamelafox added 11 commits January 7, 2025 12:47

Configure Azure Developer Pipeline

6fac970

Merge branch 'main' of https://github.com/pamelafox/azure-search-open…

adf0353

…ai-demo

Merge branch 'main' into prompty

ff40caa

Refactor to use a PromptManager

04b2254

Inject followup at the end

3a6152b

Make mypy so happy, remove conversation truncation for ask approaches

3b3f163

Refactor text.py since it doesnt need to be its own very short file

e42d0d0

Fix Chat approach tests

b4ccc08

More prompty updates, test updates

4cfd7ca

Fix type annotations

fa958df

Update more snapshots

4ef717c

pamelafox commented Jan 13, 2025

View reviewed changes

tests/snapshots/test_app/test_chat_with_history/client1/result.json Outdated Show resolved Hide resolved

pamelafox commented Jan 13, 2025

View reviewed changes

tests/snapshots/test_app/test_chat_with_history/client0/result.json Outdated Show resolved Hide resolved

pamelafox added 5 commits January 13, 2025 09:53

Add prompty metadata, revert some unneeded changes

37d9346

Fix thought process UI keys and data expectations

a507438

Resolve issue with injected prompt, update test

e413978

Pass in past messages to query rewrite prompt

890b8ad

Update snapshots

794730d

pamelafox requested a review from mattgotteiner January 13, 2025 22:18

pamelafox added 2 commits January 14, 2025 09:10

Updated prompty

962791a

Removing prompty from types now that it has them

9492378