| RELEVANCY_PROMPT = """\ | |
| Your goal is to determine if the provided LLM response is relevant to the user's query.\ | |
| You should disregard whether the response is factually accurate or not and only be concerned with relevance. | |
| In this task, I will provide you with the following: | |
| - User Query: the question asked by the user | |
| - LLM Response: a response to the user derived from an input context | |
| You should return a score of 0 if the response is not relevant and a score of 1 if the response is relevant. | |
| # User Query | |
| {query} | |
| # LLM response | |
| {llm_response} | |
| Return your response if the format of "Response: score" where the score is your estimation on relevancy. Return no other text | |
| """ | |
| HALLUCINATION_PROMPT = """\ | |
| Your goal is to determine if the provided LLM response is hallucinating given the provided RAG context. \ | |
| In this task, I will provide you with the following: | |
| - RAG Context: the provided context which will act as your source of truth | |
| - LLM Response: a series of claims derived from the RAG context | |
| You should return a score between 0 and 1 based on how accurate you perceive the claims to be. \ | |
| - If the response does not hallucinate at all then return a value of 1 | |
| - If the entire response is hallucinated then return a value of 0 | |
| - if half of the response is hallucinated then return a score of 0.5 | |
| # RAG Context | |
| {rag_context} | |
| # LLM Response | |
| {llm_response} | |
| Return your response if the format of "Response: score" where the score is your estimation on relevancy. Return no other text | |
| """ | |
| HALLUCINATION_MISTAKES_PROMPT = """\ | |
| Your goal is to extract the hallucinations from the provided respnse given the underlying RAG context if there are any. \ | |
| Essentially, you must compare the response to the RAG context, determine if any of the claims in the response are false, and \ | |
| return back any false claims you identify. | |
| In this task, I will provide you with the following: | |
| - RAG Context: the provided context which will act as your source of truth | |
| - LLM Response: a series of claims derived from the RAG context | |
| Requirements: | |
| - You must return false claims verbatim as they appear in the LLM response | |
| - You must return each false claim separated by a newline character | |
| - Do not return any other text unless you consider it to be false based on the provided RAG context | |
| # RAG Context | |
| {rag_context} | |
| # LLM Response | |
| {llm_response} | |
| Do not return any other text beside the false claims separated by a newline character. | |
| """ | |
| ATTRIBUTION_PROMPT = """\ | |
| Your goal is to determine if the provided LLM response is mis-attributing action items to the wrong person given the provided RAG context. \ | |
| In this task, I will provide you with the following: | |
| - RAG Context: the provided context which will act as your source of truth | |
| - LLM Response: a series of summarized action items attributed to a participant derived from the RAG context | |
| For example, if the RAG context says that Person A must achieve Task 1, but the LLM response incorrectly says that Person B must achieve Task 1 \ | |
| then this would be a misattribution | |
| You should return a score between 0 and 1 based on how accurate you perceive the attributions to be. \ | |
| - If the response attributes action items with complete accuracy then return a value of 1 | |
| - If the entire response is misattributed then return a value of 0 | |
| - if half of the response is misattributed then return a score of 0.5 | |
| # RAG Context | |
| {rag_context} | |
| # LLM Response | |
| {llm_response} | |
| Return your response if the format of "Response: score" where the score is your estimation on relevancy. Return no other text | |
| """ | |
| ATTRIBUTION_MISTAKES_PROMPT = """\ | |
| Your goal is to extract the misattributed action items from the provided respnse given the underlying RAG context if there are any. \ | |
| Essentially, you must compare the LLM response to the RAG context, determine if any of the action items in the response are misattributed, and \ | |
| return back any misattributed action items you identify. | |
| For example, if the RAG context says that Person A must achieve Task 1, but the LLM response incorrectly says that Person B must achieve Task 1 \ | |
| then this would be a misattribution. You must return this action item if it was misattributed. | |
| In this task, I will provide you with the following: | |
| - RAG Context: the provided context which will act as your source of truth | |
| - LLM Response: a series of summarized action items attributed to a participant derived from the RAG context | |
| Requirements: | |
| - You must return misattributed action items verbatim as they appear in the LLM response | |
| - You must return each misattributed action items separated by a newline character | |
| - Do not return any other text unless you consider it to be misattributed based on the provided RAG context | |
| # RAG Context | |
| {rag_context} | |
| # LLM Response | |
| {llm_response} | |
| Do not return any other text beside the misattributed action items separated by a newline character. | |
| """ | |
| SUMMARY_COMPLETENESS_PROMPT = """\ | |
| Your goal is to determine if the provided LLM response is a complete summary given the provided RAG context. \ | |
| In this task, I will provide you with the following: | |
| - RAG Context: the provided context which will act as your source of truth | |
| - LLM Response: a series of claims derived from the RAG context | |
| For example, if the RAG context contains important information that should be summarized then this would be considered an incomplete summary. | |
| You should return a score between 0 and 1 based on how accurate you perceive the claims to be. \ | |
| - If the response is a perfect summary of the RAG context then return a value of 1 | |
| - If the response is missing all important information from RAG context then return a value of 0 | |
| - if the response is missing half of the important information from the RAG context then return a score of 0.5 | |
| # RAG Context | |
| {rag_context} | |
| # LLM Response | |
| {llm_response} | |
| Return your response if the format of "Response: score" where the score is your estimation on relevancy. Return no other text | |
| """ | |
| SUMMARY_MISTAKES_PROMPT = """\ | |
| Your goal is to identify any important topics in the provided RAG context that is not included in the LLM response. \ | |
| Essentially, there is a possibility that the provided summary provided in the LLM response is missing key information, \ | |
| and it is your job to identify this missing information and then summarize it. You must return a summarized version of any \ | |
| missing key information that you identified. | |
| For example, if the RAG context contains important information that should be summarized then this would be considered an incomplete summary. | |
| In this task, I will provide you with the following: | |
| - RAG Context: the provided context which will act as your source of truth | |
| - LLM Response: a series of claims derived from the RAG context | |
| Requirements: | |
| - You must return a summary of any key topics or important information from the RAG context that is not already present in LLM response | |
| - Do not summarize any information that is already included in the LLM response. This would be considered a failure. | |
| - You must return each summary separated by a newline character | |
| - Do not return any other text unless you consider it to be a summary of missing information from RAG context. | |
| # RAG Context | |
| {rag_context} | |
| # LLM Response | |
| {llm_response} | |
| Do not return any other text beside the summaries separated by a newline character. | |
| """ | |