Here's the link to our paper on arxiv: 📜 paper on arxiv
Update LLM endpoint credentials in ./configs/llm_config.yaml. We have used Azure Open AI endpoints to access Gpt-4. If you are planning to use the Azure Open AI service, update values for api_key
, api_version
, azure_endpoint
, deployment_name_in_azure
. Provide your LLM endpoint a unique id/name by updating value for unique_model_id
.
If you are planning to use custom LLM, then you can inherit llama-index's CustomLLM class and GlueLLM class. Here's a script for demo where we use custom LLM from Azure ML's model catalogue, which is hosted as web-service in Azure ML ./scripts/custom_llms/aml_model.py. In this case update parameter custom_models/path_to_py_file in ./configs/llm_config.yaml.
Hyperparameters related to prompt optimization can be edited in ./configs/promptopt_config.yaml.
unique_model_id
that you specify in promptopt_config.yaml should be defined in ./configs/llm_config.yamltask_description
, base_instruction
, answer_format
. Refer to comments provided for each of these parameters in promptopt_config.yaml to understand their meaning/usage.Experiment management related configurations can be set in ./configs/setup_config.yaml.
unique_model_id
that you specify in setup_config.yaml should be defined in ./configs/llm_config.yamlcontent_moderation
. We use Azure AI Content Safety to prevent use of our prompt-optimization framework for malicious tasks. It scans the inputs and outputs and aborts the run if malicious task is detected. You need to update subscription_id
, resource_group
. You can go with defaults provided for name
, location
, sku_name
. It'll automatically create a AACS service in your azure account. You also need to install Azure CLI for authenticating to Azure services.content_moderation
field from setup_config.yaml to avoid system errors.You can run this framework on any dataset. You need to specify dataset related information e.g. question-format, answer-format etc. to the framework. Scripts to run our prompt optimization framework over some the standard datasets are present in directory ./src/glue-promptopt/scripts. These scripts can be used as reference for creating data handling code for your custom-dataset.
For this one needs to implement DatasetSpecificProcessing
class and define its abstract methods.
dataset_to_jsonl
method on how to load the data from any source and put it in .jsonl files. This method can be called twice, once for creating .jsonl file for training dataset and once for test dataset.extract_final_answer
method. It takes as input the raw output of LLM for your question. You need to write the code on how to parse the LLM's output and extract your final answer.access_answer
method takes as input LLM's raw output and ground truth answer. It calls extract_final_answer
method over raw-output of LLM to extract final answer and compares it with ground truth answer and returns True/False based on whether predicted answer matches with ground truth or not. This method can be overridden based on your scenario.All the prompt templates used in this framework are defined in ./src/glue-promptopt/glue/promptopt/techniques/critique_n_refine/prompt_pool.yaml. To understand the use of each prompt template one might have to go through our paper & understand the flow of code.
thinking_styles
defined in PromptBreeder by Google's DeepMind. Copy the thinking styles and paste it under thinking_styles
field in prompt_pool.yamlcd src/glue-promptopt/scripts
bash python run_gsm8k.py
This code repository is licensed under the MIT License.
If you find this repository helpful, please consider citing our paper:
@article{PromptWizardFramework,
title={PromptWizard: Task-Aware Agent-driven Prompt Optimization Framework},
author={Eshaan Agarwal∗, Vivek Dani∗, Tanuja Ganu, Akshay Nambi},
journal={arXiv preprint arXiv:2405.18369},
year={2024}
}
∗Equal Contributions
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
Resources:
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。