To create an effective prompt, we combine aspects of LMP (Language Modeling Pipeline), critic-based refinement, and ToT. The primary goal is to design a prompt that can elicit coherent and relevant responses from language models. We begin with a decorator that initiates the prompt, followed by an increase in subject and degree coverage, reduction of duplicate inputs, and minor improvements.
Step 2: Expert Evaluation
In this step, we engage experts to evaluate the likelihood of success for each step in the ToT process. Each expert provides a rating for the likelihood of success, and the scores are discussed until an agreement is reached. This process enables us to identify areas that require improvement and refine our prompts accordingly.
Step 3: Score Aggregation
Once the experts have provided their ratings, we calculate the overall founder suitability score using a formula that considers both the founder’s and idea’s merits. This score aggregates the results from multiple experts to provide a comprehensive evaluation of the idea’s potential for success.
Self-Play and Black Box Optimization
When designing prompts, we are essentially optimizing a black box process. Techniques such as self-play [Fu et al., 2023] and critic-based refinement [Gou et al., 2023] can help refine our approach by learning from generated data and iteratively improving capabilities until convergence is reached.
Conclusion
In conclusion, Tree of Thoughts (ToT) offers a systematic approach to language modeling that streamlines the process of generating coherent and relevant content. By combining aspects of LMP, critic-based refinement, and ToT, we can create effective prompts that elicit meaningful responses from language models. The expert evaluation process provides a comprehensive assessment of an idea’s potential for success, while the overall founder suitability score aggregates results from multiple experts to provide a definitive evaluation. By leveraging techniques such as self-play and critic-based refinement, we can continue to improve our approach and optimize the language modeling process.
