top of page
Writer's pictureTim Robinson

AI for delphi analysis



The Delphi process is a structured communication technique, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymised summary of the experts' forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel. It is believed that during this process the range of the answers will decrease and the group will converge towards the "correct" answer. Finally, the process is stopped after a pre-defined stop criterion (e.g., number of rounds, achievement of consensus, stability of results), and the mean or median scores of the final rounds determine the results.

The Delphi process has several key features:


- Anonymity of Respondents: Experts participate anonymously to prevent the dominance of one individual's opinion, reduce the effect of group pressure, and encourage open and honest feedback.


- Iteration and Controlled Feedback: Through multiple rounds of questioning, participants can adjust their views, fostering a group convergence towards a consensus.


- Statistical Aggregation of Group Response: The use of statistical methods to aggregate the opinions of experts ensures that the consensus reflects the group's combined expertise rather than the loudest voice.


- Expertise of Participants: The method relies on the knowledge and expertise of participants, who are selected because they have relevant knowledge about the topic in question.


The Delphi method is used in various fields such as technology forecasting, policy-making, and needs assessment. It is particularly useful in situations where individual judgments must be combined to address a lack of definite knowledge or where a problem does not lend itself to precise analytical solutions.


Application Steps:

1. Problem Definition: Clearly define the problem and identify the information needed.

2. Expert Selection: Choose experts with diverse knowledge and experience related to the problem.

3. First Round Questionnaire: Experts provide responses to a set of questions designed to elicit their opinions or forecasts.

4. Feedback and Summary Report: Summarise the responses, including the rationale for decisions, and provide this summary to the participants.

5. Subsequent Rounds: Refine questions and ask participants to reconsider their earlier answers considering the group's feedback. This may involve ranking or rating processes.

6. Final Evaluation: After a predetermined number of rounds or once consensus is reached, the final judgments are aggregated to form the group's conclusion.


involving AI in the process

lets explore if AI can take a complex question

create a reasonable question set

generate a diverse set of personas

and run a delphi process.


why?

AI is NOT a human expert, but we can ask it to pretend, this is a simple experiment and a novel use case which might drive insights into how we can use AI agents to explore difficult problems with reflection, this agentic approach aligns with research on MoE (mixture of experts) and tree of thought principles which are driving significant innovation in AI models, reducing bias, hallucination and errors.



a sample output from a rough PoC (currently framing the personas and question set only):


To create a Delphi question set for the topic "What is the future of agility with the incorporation of AI into business delivery," we must first define the objective of the study. The objective here is to gather expert insights on how AI will influence and shape the agility of business processes and delivery in the future.


### Objective Specification:

- To understand the potential impact of AI on business agility.

- To identify the opportunities and challenges AI presents to agile business practices.

- To forecast the evolution of agile methodologies with the integration of AI technologies.

### Personas:

1. AI Research Scientist: Specializes in AI development and its practical applications in various industries.

2. Agile Methodology Expert: Has extensive experience in implementing agile practices within organizations.

3. Business Strategist: Focuses on long-term business planning and strategy, incorporating emerging technologies.

4. Technology Ethicist: Specializes in the ethical considerations of technology use in business.

5. Operations Manager: Has hands-on experience with the day-to-day implementation of business processes and agility.

6. Customer Experience Specialist: Focuses on how business delivery affects customer satisfaction and engagement.

7. IT Systems Architect: Designs and manages the implementation of technology systems within organizations.

### Question Set:

#### Round 1: Establishing a Baseline Understanding

1. Objective Specification: How do you currently define business agility, and what role does AI play in it?

2. Question Clarity: What are the key components of an agile business that could be most enhanced by AI?

3. Focused Questions: Can you provide an example of AI's current impact on business agility within your field of expertise?

4. Structured Questions: On a scale of 1 to 10, how would you rate the importance of AI in future business agility?

#### Round 2: Exploring Opportunities and Challenges

5. Objective Specification: What opportunities do you foresee for AI to enhance business agility in the next 5-10 years?

6. Question Clarity: What do you believe are the biggest challenges in integrating AI with agile business practices?

7. Focused Questions: How might AI affect the decision-making processes in agile business environments?

8. Structured Questions: Please rank the following areas (e.g., process optimization, decision-making, customer interaction) in order of potential transformation through AI.

#### Round 3: Forecasting and Ethical Considerations

9. Objective Specification: How do you envision the evolution of agile methodologies as AI becomes more integrated into business processes?

10. Question Clarity: What ethical considerations should businesses be aware of when incorporating AI into their agile practices?

11. Focused Questions: Could AI lead to a new paradigm in business agility, and if so, what might that look like?

12. Structured Questions: What policies or frameworks do you recommend for integrating AI into agile business practices responsibly?

#### Round 4: Strategic and Operational Implications

13. Objective Specification: From a strategic standpoint, how should businesses prepare for the integration of AI into their agile practices?

14. Question Clarity: What changes to operational management do you anticipate with the adoption of AI in agile environments?

15. Focused Questions: How can businesses balance the need for AI-driven efficiency with the human aspects of agile teams?

16. Structured Questions: What skill sets will become more important as AI becomes more prevalent in agile business delivery?

#### Round 5: Long-Term Perspectives and Consensus Building

17. Objective Specification: Looking 10-20 years ahead, how do you predict AI will have reshaped the concept of business agility?

18. Question Clarity: What long-term strategies should businesses adopt now to stay ahead in an AI-augmented agile future?

19. Focused Questions: What are the potential risks if businesses fail to effectively integrate AI into their agility practices?

20. Structured Questions: Please provide your level of agreement with the following statement: "AI will become indispensable for maintaining business agility by 2040."

The questions are designed to progressively delve deeper into the topic, starting with a baseline understanding and moving towards strategic and long-term implications. This structured approach facilitates a comprehensive exploration of the subject matter, ensuring that the final consensus incorporates a broad range of expert insights.

32 views0 comments

Recent Posts

See All

Comments


bottom of page