Hybrid Delphi: A methodology to facilitate contribution from experts in professional contexts

Hybrid Delphi: A methodology to facilitate contribution from experts in professional contexts

Technological Forecasting & Social Change 78 (2011) 1629–1641 Contents lists available at ScienceDirect Technological Forecasting & Social Change H...

274KB Sizes 0 Downloads 1 Views

Technological Forecasting & Social Change 78 (2011) 1629–1641

Contents lists available at ScienceDirect

Technological Forecasting & Social Change

Hybrid Delphi: A methodology to facilitate contribution from experts in professional contexts Jon Landeta a,⁎, Jon Barrutia a, Aitziber Lertxundi b a b

Institute of Applied Business Economics, University of the Basque Country, Lehendakari Aguirre 83, 48015 Bilbao, Spain Department of Financial Economics, University College of Business Studies of the University of Basque Country, Plaza de Oñati 2, 20009 San Sebastian, Spain

a r t i c l e

i n f o

Article history: Received 28 October 2010 Received in revised form 2 March 2011 Accepted 3 March 2011 Available online 1 April 2011 Keywords: Nominal Group Technique Focus Group Delphi Qualitative technique Decision-making

a b s t r a c t This work presents a new methodology based on three well-known qualitative techniques (Focus Group, Nominal Group Technique and Delphi method), with a view to harmonising their potentialities and reducing their limitations, through application in real contexts with experts who are professionals in their respective activities. The main contribution of this methodology is its joint consideration of the needs of the investigators and also of experts who act in professional contexts, in order to improve the effectiveness of preceding techniques in achieving the scientific and social objectives of the study. We have tested this methodological approach in three real cases, with experts holding different responsibilities in different companies and public organisations, and the results secured are highly satisfactory, due to both the quantity and quality of the proposals obtained, and the satisfaction exhibited by the experts taking part, with regard to the research methodology. © 2011 Elsevier Inc. All rights reserved.

1. Introduction What is the best way of extracting and processing the information possessed by a set of professionals concerning a problem or phenomenon? Since Helmer and Rescher [1,2] laid the foundations for the scientific use of expert opinion in the areas of decision and prediction, different techniques have been developed over the years aimed at responding to the initial question, in different contexts and with different objectives. All these techniques have, in general, sought to contribute improvements in one or some of the fields of action originally delimited by Helmer [3]: 1- improvement in the selection of the most suitable information sources for each concrete case (relying on the appropriate experts), 2- assisting in the effective development of experts' activities, facilitating transmission of the information required, and 3- development of methodologies of action that make the processing of that information possible, achieving outcomes of greater quality. This methodological development is still of relevance, because knowledge advances and because needs change with time. In this regard, one sphere of great scientific and social interest is the obtaining of ranked lists of problems, practices, solutions… regarding a specific phenomenon using the tacit knowledge of professional experts who are undergoing problems, carrying out practices or applying solutions in their respective contexts of action. By professionals we refer to experts who voluntarily collaborate in a forecasting or group decision activity, freely contributing their knowledge, which is normally a product of their usual working activity. From this category we exclude, therefore, students, scholarship holders and other “captive experts”. For this purpose different group techniques may be employed, some of which already have a broad academic history of application. However, those of us who apply them feel that there is much more knowledge that we have been unable to access. A good deal of the possible inefficiency stems from the fact that it is difficult to find an active route to the professionals who possess the knowledge we are eager for and hard to get them to make that knowledge explicit: it is not easy to locate them, they tend to have ⁎ Corresponding author at: Institute of Applied Business Economics, University of Basque Country, Lehendakari Aguire 83, 48015 Bilbao, Spain. Tel.: +34 46013705; fax: +34 46013710. E-mail address: [email protected] (J. Landeta). 0040-1625/$ – see front matter © 2011 Elsevier Inc. All rights reserved. doi:10.1016/j.techfore.2011.03.009


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641

little time at their disposal, it is complicated to bring them together in face-to-face sessions, they have needs and interests different from those of researchers, they tend to be hard to pin down in their answers, they may have fears, personal characteristics and assumed roles that limit their responses, and so forth. Consequently, the right methodologies need to be developed for these kinds of experts and research objects. With this view in mind, we make a methodology proposal in this work that is a hybrid of three techniques that are widely acknowledged and have a long track record: directed face-to-face discussion groups or Focus Groups, the Delphi Method and the Nominal Group Technique. Each of the three has its own characteristics that are highly suitable for obtaining the required results from this kind of expert, but they also have their disadvantages. A careful combination of these characteristics can reduce their limitations whilst retaining what they contribute. In the article we set out the characteristics and limitations of the Focus Group, the Delphi and the NGT, the purpose, configuration and development of the proposed hybrid methodology, which we have called Hybrid Delphi, and the practical application of this new methodology in three different professional contexts. 2. Focus Group Focus Groups are carefully planned discussions or interviews, designed to obtain information within a defined area of interest, within a permissive and undirected atmosphere [4–8]. They fall within the category of structured face-to-face discussion groups [9], which means, therefore, that a variable group of experts (anyone who can, a priori, contribute input relevant for the study [10]) is brought together within the same physical space in order to interact with one another around a predefined subject of discussion, under the guidance of a moderator. From the perspective of use with professionals, the principal virtues are that they are relatively simple to set up and run (they require absolutely no statistical treatment), they help to satisfy certain social needs (power, status, recognition, affiliation) [9], they are very flexible and open in the way they unfold, they are low cost, the group members themselves can perform the task of validating individual contributions in real time [11], the results are fast [12], they encourage learning through sharing information and opinions [11] and they tend to have high subjective validity, in the sense that the participants tend to have a highly positive acceptance and assessment of the results obtained [5,13–17]. Nonetheless, they also present some important limitations: analysis and presentation of data is very complex due to group interaction; difficulties in calling the group together; risk of high degree of dispersion in the answers; a limited number of participants in each group; possibility of pressure to conform due to haste [18], to groupthink [19] or from a desire to be accepted by the group [20]; and, above all, a risk of participants being distracted or inhibited due to their own personality [21], to personal communication problems [18], to the influence of others (fear of social marginalisation [19]), to contagion from the presence of other inhibited individuals or to status incongruence and dominant personalities [18,22–24]. 3. Delphi method The Delphi method [25–30], is a social research technique which seeks to obtain a reliable group opinion from a set of experts. This is a method of structuring communication between a group of individuals who can provide valuable aid for solving a complex problem. It has been used since the sixties in academic and business spheres and has been employed principally as a technique for planning and consensus in uncertainty situations in which it is not possible to use other techniques based on objective information. Its flexibility and simplicity have led to its successful application in different geographical and thematic contexts [31]. Its main characteristics are that (1) it is an iterative process; (2) it keeps the anonymity of the participants, or at least that of their replies, as these go directly to the coordinating group; (3) there is controlled feedback; and (4) statistical group response: all the opinions form part of the final reply. The methodology of this method provided valuable solutions for problems inherent in the traditional group opinion based on direct interaction: reduction of the influence of some undesired psychological effects amongst participants (inhibition, distraction, dominant personalities…), selective feedback of relevant information, deeper reflection due to iteration, statistical democratic results and a flexible methodology that is simple to manage. Nonetheless, it is not free from important methodological weaknesses, which have been highlighted in numerous works [27,30,32–39] and negatively affect its use with professionals: limitation of the interaction that accompanies controlled feedback, restriction of the possibility of social reward for individual contribution to the group, the impunity that anonymity lends to irresponsible actions, the time required for its development, the effort demanded of participants, the fact that the methodology makes it easy for the study leader or coordinator to manipulate out of self-interest and that it is hard to correctly define the problem so that the experts also understand it in the same way, when there is no personal contact and dialogue and explanation do not easily flow, which may produce an artificial divergence in the answers. Other additional limitations are, for instance, that it is difficult to verify the precision of the method, that possible interrelations between predicted outcomes are not taken into consideration, that consensus is used as a means of approaching the truth and that, a priori, it is difficult to know what constitutes a real expert. 4. Nominal Group Technique (NGT) The NGT is a structured method for capturing and aggregating opinions emerging from a group of experts who physically coincide in terms of place and time [40–42]. In this regard, it might be deemed to be a particular case of FG. The NGT was conceived

J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641


after Delphi, the purpose being to overcome some of its limitations, making use of most of its virtues, and orienting it towards creativity (generation and ordering of ideas). In a nutshell, the process runs as follows: 1- the study leader presents a question or subject for discussion; 2- the experts, in written form, in silence and individually, produce a list of ideas, proposals or answers; 3the experts expound and clarify the individual ideas one by one, taking turns and without any comments from any of those present to judge the proposals; they may also make contributions that occur to them during this process, even though these have not been previously written down; 4- there is individual and anonymous assessment and ranking of all the ideas that appear in the dynamic; and 5- integration and exposition of the results by the study leader [43]. It is particularly useful for the generation of ideas or for the identification of the factors of a problem, as it was devised for that purpose, avoiding factors that exert a negative impact on direct interaction groups [42], although it is less to be recommended for problems that require a synthesis of information or group consensus [41]. In use with professionals it has its virtues and drawbacks, when compared with other techniques: it is more creativity-oriented [13,41] and facilitates the appearance of a “snowball” effect, driven by contributions from the rest of the group members; it permits interaction and the possibility of greater social reward than that offered by Delphi but less than FGs do; it is faster than Delphi [12], there is less variability in the group of experts, the elimination of anonymity in the answers cuts out the danger of there being a lack of responsibility and seriousness when responding and there is less risk of the process being manipulated than in Delphi, since it takes place in full view of the participants [12,41]. This tends to generate a greater sense of acceptance of results [13,17]. It also produces a greater sensation of fulfilment and closure of activity: the NGT ends with clear results that are fed back at once to the experts [13]. However, it seems to be less reliable than Delphi [44] and, in addition, shares certain disadvantages with groups of direct interaction in that there is a need for physical presence (although applications in a web-based environment also exist [45]) and for a limit being set for the number of participants (ideally, between 5 and 9). Another possible drawback is that in groups with a strong stress factor [46] or where there are highly dominant individuals [43], its face-to-face character may set off social problems, if some group members do not accept the highly structured nature of their interventions or the assessment results. 5. Devising a hybrid technique: Hybrid Delphi The three techniques set out earlier exhibit virtues and deficiencies when employed with groups of professionals (Table 1). The table indicates the potential of each technique for tackling certain problems that increase when used with professionals. It can be observed that all the problems recorded in dealing with professional experts are resolved by at least one of these approaches, so a hybrid presentation of these three techniques could, in principle, contribute to eliminating or mitigating them all. Other techniques, such as brainstorming [47], might also be employed in this context. Nevertheless, we have not developed brainstorming in this article because we understand that when working with professional groups whose knowledge and experience the method seeks to obtain, the NGT covers and goes beyond their area of contribution: the NGT shares their principal capacity (generation of ideas) [41], but its more structured reflective nature makes it a superior technique for freeing up knowledge stored by experts and for reducing risk of inhibition and distraction as it unfolds [48].

Table 1 Problems of professional groups. Capacity of the techniques to overcome difficulties of configuration and work in groups of professional experts

Previous ignorance of the technique for consultation and interaction Hard to make up a group with the right number of desired experts Hard to coincide in time and space Little availability of time Needs for relation and social recognition Need for learning and improvement Need for immediate feedback and perceived sense of closure Difficulties in adopting the viewpoint of the study and the investigator. Different backgrounds: risk of lack of focus in the answers Tendency to devote little time to reflecting on the questions (it is a task that does not fall within their professional activity: it is time-consuming) Risk of inhibition due to dominant members or to other causes Risk of behaviours that seek social approval Conformity pressure in decision-making Risk of distraction Difficulties in producing and contributing new ideas in presence of other group members Source: Adapted from Van de Ven and Delbecq [13].

Groups of direct interaction (FG)

Nominal Groups (NGT)

Delphi technique (DT)

High Low Low Medium High Medium Low High

Low Low Low Medium Medium Medium High Medium

Low High High Medium Low Medium Medium Low




Low Low Low Low Low

Medium Medium High High High

High High Medium High Medium


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641

Face-to-face stage


Non face-to-face stage




Other proposals Information Theoretical approach Approach to the problem Information Theoretical approach Approach to the problem


5-10 experts

Focused and involved experts


List of proposals

Modified NGT

Focused and involved experts

1st Delphi questionnaire

Delphi 1st round

Information processing

Delphi Answers Arguments New proposals

Focused and involved experts

2nd Delphi questionnaire

Delphi 2nd round

Lists of ordered proposals and arguments


Satisfied experts


Fig. 1. Hybrid Delphi process.

There are previous methodological works that have already indicated it would be of interest to develop hybrid techniques. Hutchings et al. [44], for example, propose developing a hybrid NGT–Delphi, and Moore [43] classifies the NGT as a starting method that needs to be employed jointly with some other method for the development of ideas. Then there are a good number of works that have totally or partially combined some of these techniques, many of them within the health field: FG and NGT [49,50], Delphi and NGT [51–53], NGT and Delphi [54,55], FG and Delphi [56] and Delphi, FG and NGT [57–59]. We made a proposal deriving from this analysis, the Hybrid Delphi, which comprises two stages: the first is face-to-face, organised in the shape of a seminar, workshop or working breakfast that incorporates both a Focus Group and a modified NGT dynamic; the second is non-face to face, associated with a classical Delphi exercise (Fig. 1).

5.1. Face-to-face stage 5.1.1. Focus group Meeting of the group of selected experts who are able to coincide in time and place, within a Focus Group format, under the guidance of a moderator who presents the problem and who shares his/her theoretical approach with the experts. Just as in FGs and NGTs, a desirable number would be between 5 and 10 experts. Following presentation of the problem, a round of free talk opens in which the experts raise doubts, seek explanations or transmit experiences, ideas or reflections. This helps to satisfy needs for relation and social recognition, information of value for learning and improvement is transmitted, and there is less risk of contributions lacking focus. Suggested duration: one hour.

5.1.2. Modified Nominal Group Following the Focus Group, the moderator sets the question or questions around which ideas, practices or experiences will be solicited and a dynamic similar to the NGT is set in motion. When the experts, over successive rounds, have finished expounding all the proposals they have written or that have occurred to them during the process, the moderator reads (and hopefully projects, if all this has been captured on computer) the list of different ideas that have cropped up in the session. The experts may then offer a last contribution or correction. Unlike the traditional NGT, assessment of the proposals is not made in the face-to-face session. The moderator informs the participants that they will shortly be receiving (generally by email) the list of the proposals that have appeared in the session, along with others drawn from the literature reviewed or deduced from the theoretical approach employed, if such is the case, and asks them to then assess the proposals and proceed with this process of reflection. This exercise manages to get the experts as a group to explore, generate and contribute a great number of ideas, in a short time, with little risk of becoming distracted, so that they leave with a feeling that they have performed a job (although it is not completely over) and learnt new things or tested out what they knew. Moreover, any sensation of there being “winners and losers” is avoided by the elimination of voting, and the structured process cuts out any possible negative influences of the group on its members, while retaining the positive effects. Suggested duration: one hour.

J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641


5.2. Non-face to face stage 5.2.1. Classical Delphi exercise The study object question or questions are referred to the experts, along with the ideas or proposals collected in the NGT, completed with other contributions gathered from other sources. There is no need for a first round with open questions, as this task has already been carried out via the NGT. The experts are those who took part in the face-to-face session, plus other interested experts who could not or did not wish to attend the session. The Delphi questionnaire also has an introductory part to help the new experts concentrate on the objective and focus of the study, and to refresh the memory of those who participated in the first stage. In addition to assessing the proposals contained in the initial questionnaire, the experts can make new proposals that will be appraised in successive rounds. At the end of the exercise a list is obtained, and ordered by the experts, in line with the criterion chosen (importance, feasibility, cost…) for the proposals presented. What this second stage achieves is to facilitate the generation of new contributions, assessments and comments by all the experts who wish to take part, wherever they are, and with flexibility in terms of time. In the voting process the risk of inhibition caused by dominant personalities and of behaviours seeking social approval is avoided, although it is true that the traditional methodology introduces some pressure towards conformity (or consensus). The experts have enough time to think out their answers, and the structured nature of the exercise obviates distractions and the appearance of information that is not relevant. The final result democratically draws together contributions from all the experts, and its quantitative nature facilitates communication and decision-making. It is not easy to offer guidance as to how long this should last. It depends on the number of rounds, (which with professionals rarely exceeds two), and on the speed of response, although it is advisable for this to be as fast as possible, as too much time is negative in any Delphi, especially when it is performed with experts for whom participation is collateral to their daily activity. Suggested duration: two weeks. The order proposed for the application of the techniques is determined by the chronological sequence that is desirable in order to overcome the difficulties or problems of professional groups: a) Face-to-face meetings (whether or not they be FGs) are familiar dynamics for professionals. They are well-acquainted with these atmospheres, need no a priori explanation and are unlikely to feel any rejection towards them, which encourages their voluntary involvement. Interaction with the Focus Group's researcher-moderator facilitates explanation of the study objective and acts against dispersion and lack of focus in the experts' contributions. The professionals can express themselves freely, in the absence of fears and internal pressures, and connect with the other participants, which encourages their emotional commitment to the study and to the whole group dynamic. Due to the confluence of all these factors it is recommendable to start with the FG. b) The NGT is a more structured technique, in which experts can take part in favourable conditions once they are focussed on the subject to be dealt with and are no longer constrained by uncertainties and by their need to freely express opinions and experiences, thanks to the prior FG. In addition, given that it is hard to get professional experts to meet up, implementation of this technique after the FG makes the most of their physical presence to obtain ideas and proposals in conditions conducive to creativity. c) Delphi is also a highly structured technique, the absence of direct interaction makes it a colder experience and its process tends to take longer. Consequently, our understanding is that it may produce an optimum outcome once the experts are focussed on the subject to be studied and emotionally committed to it. The assessments mainly concern proposals they have themselves come up with, in the NGT, meaning that there is a greater degree of involvement. This does not demand a physical presence, so the prior face-to-face stage can be limited to two hours, which does not lay excessive commitments on the professionals' working day, and also makes it possible to bring in other experts to complete the group, with the right number of members and the desired profile. The final outcome is quantified, reflects the participation of all the experts and is easy to present and reason out with third parties. All of which justifies leaving this technique until last. The Hybrid Delphi that we present differs from all the other applications included so far in the literature that utilise the three techniques on which it is based. Robotin et al. [57] employ the FG, NGT and Delphi independently and on different experts, with the aim of increasing the quality of their results through the application of different techniques with the same objective. The approach is not one of complementarity, as ours is. Mc Dougal et al. [58] use the Delphi first to obtain a list of competencies and outcome measures, and then employ the NGT and FGs to refine and prioritise the competencies and outcome measures. Gibson et al. [59] also used FGs, Delphi, and NGT, in addition to semi-structured interviews, as complementary techniques utilised with different people, with different objectives. There are also examples of NGT applications, completed afterwards with Delphi exercises involving a broader panel of experts [54,55], but they lack the initial contribution from FGs to focus the experts on the subject in hand and to satisfy some of their own needs. It also differs from the hybrid technique combining the Delphi Policy [28] and Concept Mapping [64], recently proposed by Klenk and Hickey [65]. It shares with them some concerns (encouraging creativity, eliminating pressure to conform and influences from dominant members), but it is differentiated in the methodology employed (more complex), in the organisation of the results obtained (cluster rating maps) and in its preferred field of application (integrating and clarifying scattered information within contexts of forecasting, planning and decision-making). Lastly, the Hybrid Delphi is also a proposition of a different kind than the relatively frequent proposals in academic literature that fall under the general name of “Modified Delphi”, especially within the field of Life and Health Sciences. Fundamentally, three


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641

kinds of Delphi variants are covered by this designation: (1) those that carry out a Delphi exercise in which the first round contains closed questions but no open ones. In our opinion, this does not justify the use of the term Modified Delphi, but Murry and Hammons [66] did consider this to be so, and, taking their work as a reference, other authors have also followed suit [67–70], (2) in works that are based on the RAND-UCLA Appropriateness Method (RAM), developed by Fitch et al. [71–73] to determine and identify when there is overuse or underuse of health care, within the area of Health Sciences. The RAM basically incorporates a modified Delphi, in which the second round is face to face, in a panel meeting, and (3) all the modifications of the classic Delphi that build in various changes in order to bring it closer to the needs or possibilities of the investigation. These include, for instance, meetings or interviews before, after, or between Delphi rounds [74–79], Delphis with different but related questions in each round [80], Delphis that incorporate scenario writing and seminars [81], applications with direct expert to expert feedback [82], or even cases in which the authors designate applications as modified Delphis when they do not significantly differ from a classic Delphi technique [83–87]. Gupta and Clarke, in their well-known work [36], noted that “it appears that practitioners are often willing, and sometimes even eager, to modify Delphi”. 6. Applications of the Hybrid Delphi Later, by way of example and in an initial testing of our methodological proposal, we present three real applications of the Hybrid Delphi, which we carried out with experts in professional contexts. In the three cases the experts who took part did so voluntarily, contributing knowledge to an academic initiative from their professional activity. They had to fit this collaboration into their tight work schedules, with no compensation other than the information and knowledge that they might obtain from their participation, along with the possible satisfaction of particular social needs and of a sense of recognition. In the first case they were Human Resources managers from important Basque companies, in the second they were mainly small business people from different family firms, and in the third they were Human Resources and Nursing managers from hospitals within the Basque Health System. What differentiated the latter case was that all the experts were from different work centres (hospitals) but worked, however, within the same organisation and were consequently colleagues, which might have a bearing on how freely or not they might express their opinions. In addition, they were answering in a dual capacity: as experts, because they were perfectly acquainted with the subject they were being asked about, and as possibly affected parties, in that their organisation is considering implementing a competence assessment system, which was the object of this third study. 6.1. Proposals for improving continuous management training, from the company perspective We invited the training managers of 40 Basque companies which are characterised by higher investment in continuous management training to a face-to-face session, in the shape of a Working Breakfast, to present and debate the results of a study with which they had collaborated [60,61].1 Nine experts from these companies attended, along with an expert from the management association that endorsed and sponsored the study. We presented them with the theoretical model that in our view best explained company behaviour in relation to continuous management training and we pointed out some of the practices stemming from that theoretical approach that ought to serve to improve the effectiveness of this kind of training. The group of experts then proceeded to debate around the explanatory capacity of the model, in an open round of interventions, with freedom to contribute examples, reflections and experiences (Focus Group: 60 min), and there was, in general, a high degree of agreement with the theoretical proposal received. We went on to invite them to draw up, in silence, in written form and individually, the practices or suggestions, from their own experience or from others' experiences they were aware of, that they felt worthy of mention for achieving improvements in continuous management training. They shared them, following the NGT methodology. As the participants formulated their contributions we wrote them up and projected them. Finally, we collected 28 different proposals or good practices. No assessment was made and no marks were given. They were informed that we would send on these practices in written form for later assessment, along with others deriving from the theoretical framework employed or from the literature review we had performed. The whole face-to-face session lasted roughly two and a half hours, breakfast included. The Delphi questionnaire which we sent them took in 40 proposals, 28 from FG and 12 from other sources, classified within three categories: proposals and good practices for companies, proposals for external suppliers of continuous training and proposals for business associations and public agents. The experts had to assess, on a 7-point Likert scale, the importance they awarded each of the action proposals they were presented with, in terms of their capacity to contribute effectively to increasing investment by their companies in continuous management training. In an open-ended format, they were also asked for any information or comments that they deemed to be of relevance to the proposals given, as well as for new proposals, if they considered that these were not included within those they had been provided with. It was sent to the 40 companies initially considered, and was answered by 13 experts (the 10 who participated in the FG, as well as three others, who had not been able to attend the face-to-face session). 1 Project Company behaviour vis-à-vis continuous management training. University-Company research project (UE03/A029), financed by the Basque Government in collaboration with the Asociación para el Progreso de la Dirección/Association for Progress in Management (APD). The classification of companies in terms of continuous training investment is also a consequence of this project.

J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641


We performed two rounds (10 experts answered in the second). The experts were notified that there would be a second round, in which they would have an opportunity to know the statistical opinion of the group with regard to each proposal and the additional explanatory comments expressed by the experts (altogether 18 different comments were fed back), before reflecting again and responding once more to the same questions, formulated in the same terms. We obtained the ranking, by categories, of the proposals and good practices presented, accompanied by valuable statistical and qualitative information collected in the three dynamics carried out. Via electronic mail the participants received a report with this information and, some time afterwards, when we sent the survey assessing their degree of satisfaction with their participation and with the methodology followed, this was accompanied by the book that we published containing the global results for the project [60]. The final assessment of the participating experts regarding the methodology utilised and the results achieved was highly satisfactory. 6.2. Characteristics of family enterprises that produce competitive advantage and good practices to secure them This study was part of a PhD work, supported by the University of the Basque Country's Family Business Professorship and directed by one of the authors [62]. Participating in the face-to-face stage of this dynamic were 10 experts in family businesses: 5 family businesspeople who manage their own companies, 2 academics with wide experience in family business research, 2 independent experts who lead public institutions that work with family businesses, and 1 legal expert specialising in the family business. In the first part of the session we informed them of the objectives of the investigation, the results obtained up to that point, the theoretical proposals and our working hypotheses. In an open round of interventions they expressed their ideas about the advantages and disadvantages of family enterprises and about the theoretical proposals that we had formulated for them. We then put two questions to them, regarding: 1- characteristics proper to family businesses that may be a source of competitive advantage (positive characteristics of family enterprises) and 2- good practices, associated with their family business status, that had helped them and are helping them to be more competitive as an enterprise, which were treated following the modified NGT methodology described earlier. The experts identified 25 characteristics and 21 good practices, which already included those, set out in the theoretical introduction. The two lists of characteristics and good practices were read and approved in the face-to-face session, after which they were encouraged to continue their reflections through a Delphi exercise. In the Delphi application 10 experts who had formed the NGT took part, plus 8 additional experts who had not been able to attend the face-to-face session. The entire panel was constituted by 11 family businesspeople who manage their businesses, 3 academics with wide experience in family business research, 3 independent experts who lead public institutions that work with family businesses, and 1 legal expert with much experience in the latter. They replied to a total of two rounds, in 32 days, with noone dropping out. In the first round they assessed the importance they assigned to the characteristics of family enterprises in terms of constituting a source of competitive advantage, and likewise where the effectiveness of good family business practices for improving their competitiveness was concerned (Likert scale of 1 to 5). In addition to this, the experts had to indicate which of the characteristics and which of the related practices were in their judgement the most important, giving their reasons, and to say which of these were the most irrelevant, providing reasons for their opinions. In the second round, they were transmitted, along with their own previous reply, the group statistical opinion and explanatory arguments contributed by the experts in the first round with regard to the most and to the least influential characteristics, as well as to the most and to the least useful practices presented in the first round (a total of 18 different arguments), after which the experts repeated their assessments concerning the same items. Finally, we obtained the ranked list of characteristics and good practices that made for the most competitive family businesses, accompanied by 28 different arguments, with a high level of consensus. Four months after the study came to completion the experts who took part were invited to a new seminar, organised by the Family Business Professorship, where the results were distributed, set out and discussed in a Focus Group format and enhanced with some new good practices. In the survey they completed, the experts expressed a high degree of satisfaction at having participated in the experience. 6.3. Assessment of application strategies for an evaluation model of competencies for community nurses This study is part of a doctoral thesis from the University of the Basque Country, which will be read at the beginning of the year 2011, directed by one of the authors [63]. The PhD student had prepared a competence evaluation model, upon a solid theoretical foundation, which she had tested on a representative sample of nurses from the Basque health system. The only thing remaining to be done was to design the strategy for implementing the model, and it was considered appropriate to resort to the knowledge of professionals in the management of Basque hospitals. Thanks to collaboration from the Human Resources Manager of the Basque Health System, it was possible to hold a face-to-face session with 6 experts who were human resources (3) and nursing managers (3) from the main Basque public hospitals. These experts were explained the theoretical foundations of the evaluation tool that had been designed, the purpose of it and the competencies it evaluated. An open round of interventions was held concerning the difficulties of setting up a model of this kind, the misgivings it might arouse, previous experiences, personal opinions, etc. The modified NGT dynamic then took place, in which, in writing, the experts made a list of initiatives for making implementation of the model viable, classified as: (1) communication strategies, (2) training strategies, (3) start and follow-up strategies and (4) general implementation strategies. Four lists of initiatives were obtained with 15, 13, 10 and 10 initiatives, respectively. We read the collection of initiatives to the experts and


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641

they accepted the transcription made. Afterwards they were invited to take part in a Delphi exercise in order to assess and rank the importance of these initiatives (Likert scale of 1 to 10), and to contribute new ideas that had not appeared in the first session. In the first round of the non-face to face stage, 9 new experts joined in, and 1 expert decided to drop out. In the second round, the 15 experts who participated in the first round received the same questionnaire, with their previous response, the group opinion with central tendency and quartile statistics, and a request that they justify their new response if it lay outside the central quartiles. In addition, in this second questionnaire six new questions were included to capture their satisfaction with the methodology employed in the research. In this second round a single dropout was registered. When the study had been carried out, we sent a report by email to all the experts who had taken part in some of the stages of the investigation or who had supported it. The report contained the methodology and results of the study: four lists with the communication, training, monitoring and general implantation initiatives, each of them ranked in accordance with their importance in terms of making viable the implementation of an evaluation model for nursing competencies.

7. Assessment of Hybrid Delphi applications In order to assess the validity of the methodology and the confidence and satisfaction of the participants in the Hybrid Delphi Applications executed, we sent a questionnaire to all the experts who took part in the two stages of each of the three applications presented (Appendix A), asking them to assess from 0 to 10 the degree of interest and relevance to the study objective of each of the stages of the methodology applied: (1) presentation of the objectives of the study and of its theoretical focus, (2) Focus Group, (3) NGT, (4) Delphi, 1st round and (5) Delphi, 2nd round. Additionally, they had to assess how confident they felt about the quality of the results obtained and their degree of satisfaction with their participation, with regard to the relationship with colleagues, testing out their own ideas, learning and capture of new ideas, the opportunity of doing something different from their usual jobs and contribution of their knowledge to academic social research. Lastly, they had to indicate whether they would take part again in a study employing this methodology concerning a subject of their interest. 23 of the 26 experts who participated in the three dynamics answered the questionnaire. As can be observed from the results in (Table 2), the mean (median) assessment of the interest and relevance of the 5 stages was equal to or higher than 7 points for each of the applications. Of the three applications the first stage was the most highly valued: presentation of the study objectives and focus (median 9 and mean 8.65). The remaining stages obtained very similar global results (median equal to 8). As for confidence in the results the global assessment was also high (median 8, mean 7.73). The degree of satisfaction expressed by the experts regarding their participation in the Hybrid Delphi was high. In all the aspects about which they were consulted satisfaction scored higher than 7 points, the highest marks being awarded to the opportunity for learning and for capturing new ideas, as well as for relating to colleagues (means of 8.23 and 8.19, respectively), and the lowest relative score for degree of satisfaction to “the opportunity to do something different from their usual jobs” (mean 7.14). The 23 experts declared their willingness, if they were able, to take part once more in a study employing this methodology. As additional information, it must be pointed out that only one of the 26 experts who participated in the face-to-face stage did not continue at the Delphi stage.

Table 2 Assesment by experts.

Interest and relevance of the following actions 1. Presentation of the objectives and foundatios of the investigation 2. Focus Group 3. NGT 4. Delphi 1st round 5. Delphi 2st round Confidence in the results Satisfaction in terms of the opportunity it offered you of: Being with and associating with other colleagues Being able to test out your own ideas with other colleagues Acquiring new ideas and learning something new Doing something different from your regular activities Contributing your knowledge as part of a social academic investigation Would you take part again in another experience of this kind? No Probably not Probably If I am able, yes

Training managers

Family managers

Health managers














8.3 7.2 6.6 7.1 7.5 7.4

8 7 7 8 7.5 7

8.7 8.3 8.3 7.9 7.9 7.8

9 8 8 8 8 8

9.2 9.2 9.2 8.8 8.4 8.2

10 9 9 9 9 8

8.65 8.09 7.83 7.78 7.86 7.73

9 8 8 8 8 8

8.4 8.1 8.3 6.8 7.8

8 8 8 6.5 8

8.6 8.3 8.6 7.2 8.1

8.5 8 8 7 8

7.2 7.6 7.6 7.6 8

8 8 8 8 8

8.19 8.09 8.23 7.14 7.95

8 8 8 7 8 Sum 0 0 6 17

J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641


The assessment of the quality of the inputs, outputs, and of the process followed in each of these studies is, in our opinion, positive. We base our evaluation on the following elements of judgement: a) Hybrid Delphi 1. Proposals for improving continuous management training: A sufficient number of genuine experts, Human Resources managers from the companies with the highest training input in the Basque Country, took part. They involved themselves in the study (all the experts who participated in the face to face stage carried out the Delphi) and contributed important qualitative information. In most of the items (78%) the consensus achieved in the second round of the Delphi was higher than in the first. Seven proposals were obtained for public agents and business associations, 2 for training providers and 31 for companies. These proposals were favourably assessed by the participants and by the association promoting the study, they were presented in public and via the press, and a book was published detailing their content [60]. b) Hybrid Delphi 2. Characteristics of family enterprises that produce competitive advantage and good practices to secure them: The quality and relevance of the experts were high, their involvement in the study (with no dropout registered, they all attended the seminar at which the results were presented four months afterwards) was exceptional, the quality and wealth of their contributions was outstanding, reflected in the two ranked lists of characteristics (25) and good practices (21), plus the explanatory arguments (28), the results were actually accepted favourably by the family businesspeople, the process was fast (45 days from the FG to the closure of the second round), a higher degree of consensus was achieved in the second round (in 85% of the items) and the stability of the second round was marked (the experts maintained an average of 80% of their proposals). c) Hybrid Delphi 3. Application strategies for an evaluation model of competencies for community nurses: A sufficient number of qualified experts took part, all of whom were HR and Nursing managers (in the first round Delphi 15 replied out of a possible total of 21), their level of involvement and participation was high (only one of the people who took part in the FG dropped out of the study and two did not answer the second round, plentiful qualitative information was obtained in the face to face session, and in the Delphi twelve new contributions or argumentations were made), in all the items a greater level of consensus was achieved in the second round, the results were sent to all the participants and to the HR general management of Basque Health System, and their assessment was very positive, both where the results and the methodology employed were concerned. 8. Discussion and conclusions Hybrid Delphi is a technique that differs in its structure and objectives from other techniques based on the judgement of experts. It is characterised by the fact that it offers its results as a consequence of the application of a methodology that structures its three basic factors (FG, NGT and Delphi) in a particular way. The order established is fixed and reasoned. Modification of it would alter the result. Not only is it chronological, but it also seeks to feed and channel a flow of information and knowledge, in a process where experts are involved and their contributions improved, towards a result that is higher than that which would be obtained by independently employing the techniques on which it is based. FG output is input for NGT, and NGT output is the main input for Delphi. It is not, therefore, a “modified Delphi”, as the Delphi, in itself, is not modified. The Delphi is fed through the FG-NGT process. The final output is that of the Delphi, which has the capacity to reorder and improve the outputs of the other two techniques, which it has received in the form of input. This is why it is fundamentally a Delphi, but not just a Delphi. The final result must be understood and interpreted within the combination of the three techniques that make up a holistic whole. For this reason we proposed the name Hybrid Delphi. It is a general defined structure, which lends itself to being adapted and applied to different contexts. Another element that characterises and distinguishes the Hybrid Delphi is its joint consideration of the needs of the investigator, and also of experts who act in professional contexts, in order to improve the effectiveness of preceding techniques in achieving the scientific and social objectives of the study. The investigator needs to gather the knowledge, opinion and creativity that are possessed principally by these professional experts, so s/he needs to secure collaboration from a sufficient number of qualified experts and establish the conditions for the extraction, transmission and efficient treatment of such knowledge. Then again, the main activity of these experts is not that of collaborating with the investigator; they collaborate voluntarily in the degree to which their participation serves to satisfy their own needs for relation, learning, recognition, etc., and does not ask too much of them in terms of time or risk. The methodology proposed takes care of both aspects. It influences all the factors that might affect output in line with the consensus development methods identified by Black et al. [88] in their exhaustive bibliographical review: the way the task is set, selection of participants, the selection and presentation of scientific information, the way any interaction is structured and the method for synthesising individual judgments. Starting out from the strengths of three preceding tested techniques (Focus Group, NGT and Delphi), it overcomes their weaknesses. To do so, it configures a process of synthesis, which sequentially orders the application of these techniques and tailors the characteristics and dynamic of each of them to the objective and context of application of the Hybrid Delphi. Thereby, it tackles the problems involved in working with experts in professional contexts, more effectively than do the techniques from which it has sprung. It is suitable, therefore, for attracting, collecting and combining the knowledge of professional experts who voluntarily collaborate in an activity that for them is secondary. The experts, meanwhile, find it an interesting activity that binds them to no commitments, through which they learn, mix with others and test out their ideas, whilst


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641

contributing to investigations of social interest. This encourages them to be effective in their collaboration and to repeat the experience on other occasions, as it proves to be engaging and useful. This is corroborated by the assessments of the experts who took part in the three applications carried out to date with this methodology. The best token of the satisfaction produced by their participation in an investigation employing this methodology is that they would all be willing to participate again in a similar dynamic. This methodology, therefore, facilitates the participation, involvement and interaction of real experts, in real contexts, and, deriving from such interaction (quantitative and, above all, qualitative feedback), changes of opinion that lead to superior group results. This constitutes the foundations for improvement in the quality of the outcome of any Delphi process [89]. But certainly, a future line of research that emerges from this proposal is the performance of more tests whose purpose would be to unequivocally validate (or refute) its effectiveness. It is necessary, although not easy in professional contexts, to compare, in homogeneous conditions, the quality and quantity of this methodology's contributions (ideas, opinions, proposals) with those obtained from other preceding techniques, especially with those captured separately using each of the techniques analysed (FG, NGT and Delphi), or with paired combinations of techniques. Likewise, the outcomes of the Hybrid Delphi and those of other techniques will have to be compared in terms of experts’ perception of satisfaction with their participation, their acceptance of the outcomes and willingness to repeat their participation in similar investigations. If a greater number of controlled experiences were carried out, this would also equip us to propose and test contingent modifications when applying the different stages of the Hybrid Delphi, depending on the different professional contexts in which it takes place (experts belonging to a single organisation, pool of experts from diverse backgrounds, professional bureaucracies, etc.), on the needs of the experts participating (predominance of a desire for learning, for social relations,…) and on the objectives of each investigation (obtaining and ranking ideas, ordering alternatives, making decisions, making forecasts, giving impetus to learning and participation….). Consequently, an interesting line of research opens up, the aim of which will be to deliver to researchers and practitioners a Hybrid Delphi that is tested and adapted to their specific needs. Acknowledgements Our thanks go to our colleagues who participated with us in the three cases presented: to Jon Hoyos and Andrés Araujo, in the study on Continuous Management Training, Jon Sánchez, in the work on Family Business, and Laura Alonso, in the Assessment of application strategies for an evaluation model of competencies for community nurses, as well as to all the experts who disinterestedly took part in these Delphi exercises. We would, of course, also like to thank the referees and editors for their valuable suggestions and contributions. Appendix A. Assessment of the methodology employed in the consultation of experts in CONTINUOUS MANAGEMENT TRAINING (in which you took part at the end of 2005) in order to obtain and assess PRACTICAL PROPOSALS FOR IMPROVING THE INTENSITY AND EFFECTIVENESS OF CONTINUOUS MANAGEMENT TRAINING Rate from 0 to 10 the interest and relevance of the following actions (0 of no interest, of no relevance–10 of great interest, of great relevance)


Presentation by researchers at a meeting that took place at the APD (Association for Progress in Management) headquarters, of the objectives of the investigation, of the theoretical foundations that justify a company's attitude towards the continuous training of its managers and of some logical standards of action that stem from the theory, as a step prior to gathering the proposals and experiences of the experts (of yourselves, that is) 2 Open round of interventions from the experts present, with comments, misgivings and reflections concerning the customary behaviour of companies and managers, and with regard to the theoretical model presented to explain such behaviour, as a prior step to the collection of proposals 3 Written individual enumeration, followed by collective ordering, of action proposals for training suppliers and for business associations, as well as good business practices for increasing the intensity and effectiveness of continuous management training 4 Assessment of action proposals for companies, external training agents and associations to make continuous management training more intense and more efficient, and of the good practices collected in the face-to-face session, in addition to others collected from the literature, anonymously and via email, providing an option for the contribution of new proposals by experts who participated in this 4th stage but who did not attend the first 3 stage (Delphi 1st round) 5 New round of assessment of proposals from the first questionnaire, along with others contributed by other experts in the first round, participants being informed this time of the majority opinion and of the comments made by the group of experts in the first round (Delphi 2st round) Rate from 0 to 10 the degree of confidence you feel that through this process or the methodology pursued the most relevant action proposals have been captured in order to make continuous management training more intense and more efficient, and good management training practices more interesting (0 no confidence−10 absolute confidence)


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641


Can you think of any alternative procedure that would be more effective and rigorous in achieving the objectives of the study? Could you specify any suggestion that would improve the procedure adopted? And lastly, with regard to your participation in this investigation: Did you find your participation as an expert in this research around continuous management training a satisfactory experience (0 highly unsatisfactory−10 highly satisfactory) in terms of the opportunity it offered you of:

Being with and associating with other colleagues Being able to test out your own ideas with other colleagues Acquiring new ideas and learning something new Doing something different from your regular activities Contributing your knowledge as part of a social academic investigation Others (indicate which)

Would you take part again in another experience of this kind, organised under similar conditions, concerning another subject that interests you? (Mark your preference with an X)

No Probably not Probably If I am able, yes

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42]

H. Helmer, N. Rescher, On the epistemology of inexact sciences, Manag. Sci. 7 (1959) 179–192. P.D. Aligica, R. Herritt, Epistemology, social technology, and expert judgment: Olaf Helmer's contribution to futures research, Futures 41 (2009) 253–259. O. Helmer, Social Technology, in: O. Helmer (Ed.), Looking Forward, Sage, CA, 1983, pp. 58–59. A.E. Goldman, The group depth interview, J. Mark. 61 (July, 1962) 68. R.A. Krueger, Focus Groups—a Practical Guide for Applied Research, 2nd ed. Sage, London, 1994. R.K. Merton, M. Fiske, P.L. Kendall, The Focused Interview: a Manual of Problems and Procedures, Free Press, New York, 1990. N. Robinson, The use of focus group methodology—with selected examples from sexual health research, J. Adv. Nurs. 29 (4) (1999) 905–913. R. Blackburn, Breaking down the barriers: using focus groups to research small and medium-sized enterprises, Int. Small Bus. J. 19 (1) (2000) 44–63. J.S. Armstrong, Long-range Forecasting: from Crystal Ball to Computer, Wiley, NY, 1978. J. Pill, The Delphi method: substance, content, a critique, and annotated bibliography, Socio-Econ. Plan. Sci. 1 (1971). A. Hyde, E. Howlett, D. Brady, J. Drennan, The focus group method: insights from focus group interviews on sexual health with adolescents, Soc. Sci. Med. 61 (2005) 2588–2599. P.L. Williams, N. White, R. Klem, S.E. Wilson, P. Bartholomew, Clinical education and training: using the nominal group technique in research radiographers to identify factors affecting quality and capacity, Radiography 12 (3) (2006) 215–224. A.H. Van de Ven, A.L. Delbecq, The effectiveness of Nominal, Delphi, and interacting group decision making processes, Acad. Manag. J 17 (4) (1974) 605–621. R.C. Erffmeyer, I.M. Lane, Quality and acceptance of evaluative task: the effects of four group decision-making formats, Group Organ. Stud. 9 (1984) 509–529. J.A. Sniezek, An examination of group process in judgmental forecasting, Int. J. Forecast. 5 (1989) 171–178. J.S. Hornsby, B.N. Smith, J.N.D. Gupta, The impact of decision-making methodology on job evaluation outcomes. A look at three consensus approaches, Group Organ. Manag. 19 (1) (1994) 112–128. A. Graefe, J.S. Armstrong, Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task, Int. J. Forecast. 27 (2010) 183–195. B.E. Collins, H. Guetzkow, A Social Psychology of Group Processes for Decision-making, Wiley, NY, 1964. I.L. Janis, Groupthink, 2nd ed. Houghton-Mifflin, Boston, 1982. J.E. Dittes, H.H. Kelley, Effects of different conditions of acceptance upon conformity to group norms, J. Abnorm. Soc. Psychol. 53 (1956) 100–107. A. Lock, Integrating group judgment in subjective forecast, in: G. Wright, P. Ayton (Eds.), Judgmental Forecasting, Wiley, NY, 1987. M. Watts, D. Ebbutt, More than the sum of the parts: research methods in group interviewing, Br. Educ. Res. J. 13 (1987) 25–34. M. Descombe, Exploration in group interviews: an evaluation of a reflexive and partisan approach, Br. Educ. Res. J. 21 (1995) 131–148. A. Gibbs, Focus Groups, Soci. Res. 19 (1997) 1–78 Update. N. Dalkey, O. Helmer, An experimental application of the Delphi Method to the use of experts, Manag. Sci. 9 (1963) 458–467. T.J. Gordon, O. Helmer, Report on a Long-range Forecasting Study, The Rand Corporation P-2982, Santa Monica, CA, 1964. H.A. Linstone, M. Turoff (Eds.), The Delphi Method, Techniques and Applications, Addison–Wesley, Reading, MA, 1975. M. Turoff, The design of a Policy Dephi, Technol. Forecast. Soc. Change 2 (1970) 149–171. R.J. Parentee, J.K. Anderson, P. Myers, T. O'Brien, An examination of the factors contributing to Delphi accuracy, J. Forecast. 3 (1984) 173–182. G. Rowe, G. Wright, The Delphi technique as a forecasting tool: issues and analysis, Int. J. Forecast. 15 (1999) 353–375. J. Landeta, El método Delphi, Ariel, Barcelona, 1999. R.E. Overbury, Technological forecasting: a criticism of Delphi techniques, Long Range Plan. 1 (4) (1969) 76–77. H. Sackman, Delphi Assessment: Expert Opinion, Forecasting, and Group Process, The Rand Corporation R-1283-PR, Santa Monica, CA, 1974. T.J. Murray, Delphi methodology: a review and critique, Int. J. Urban Syst. 4 (1979) 153–158. T.R. Stewart, The Delphi technique and judgmental forecasting, in: K.C. Land, S.H. Schneider (Eds.), Forecasting in the Social and Natural Sciences, D. Reidel, Dortrech, Holland, 1987. U.G. Gupta, R.E. Clarke, Theory and applications of the Delphi technique: a bibliography (1975–1994), Technol. Forecast. Soc. Change 53 (2) (1996) 183–211. S. Keeney, F. Hasson, H.P. McKenna, A critical review of the Delphi technique as a research methodology for nursing, Int. J. Nurs. Stud. 38 (2001) 195–200. P. Tapio, Disaggregative policy Delphi: using cluster analysis as a tool for systematic scenario formation, Technol. Forecast. Soc. Change 70 (1) (2002) 83–101. J. Landeta, Current validity of the Delphi method in social sciences, Technol. Forecast. Soc. Change 73 (2006) 467–482. A.L. Delbecq, A.H. Van de Ven, A Group process model for problem identification and program planning, J. Appl. Behav. Sci. 7 (1971) 466–492. A.H. Van de Ven, A.L. Delbecq, Nominal versus interacting group processes for committee decision making effectiveness, Acad. Manag. J. 14 (1971) 203–212. A.L. Delbecq, A.H. Van de Ven, E.H. Gustafson, Group Techniques for Program Planning, Scott, Foresman and Company, Glenview, IL, 1975.


J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641

[43] C.M. Moore, Group Techniques for Idea Building, Sage, London, 1987. [44] A. Hutchings, R. Raine, C. Sanderson, N. Black, A comparison of formal consensus methods used for developing clinical guidelines, J. Health Serv. Res. Policy 11 (4) (2006) 218–224. [45] P. Pazos, M.G. Beruvides, J. Jian, A.M. Canto, A. Sandoval, R. Taraban, Structuring group decision making in a web-based environment by using the nominal group technique, Comput. Ind. Eng. 52 (2007) 277–295. [46] P.C. Nutt, Planning Methods: for Health and Related Organizations, Wiley, Toronto, 1984. [47] A.F. Osborn, Applied Imagination: Principles and Procedures of Creative Thinking, Scribner, New York, 1953. [48] A. MacPhail, Nominal Group technique: a useful method for working with young people, Br. Educ. Res. J. 27 (2) (2001) 161–170. [49] S. Gordon, M. Tarafdar, R. Cook, R. Maksimoski, B. Rogowitz, Improving the front end of innovation with information technology, Res. Technol. Manag. 50 (2008) 588 May-June. [50] E.W. Duggan, C.S. Thachenkary, Integrating nominal group technique and joint application development for improved systems requirements determination, Inform. Manag. Amster. 41 (4) (2004) 399–411. [51] S.P. Cook, M. Birrell, Defining an occupational therapy intervention for people with psychosis, Br. J. of Occup. Ther. 70 (3) (2007) 96–106. [52] B.L. Neiger, M.D. Barnes, R. Thackeray, N. Lindman, Use of the Delphi method and Nominal Group technique in front-end market segmentation, Am. J. Health Stud. 17 (3) (2001) 111–119. [53] M. Loeb, D. Bentley, S. Bradley, K. Crossley, R. Garibaldi, N. Gantz, A. McGeer, R. Muder, J. Mylotte, L.E. Nicolle, B. Nurse, S. Paton, A.E. Simor, P. Smith, L. Strausbaugh, Development of minimum criteria for the initiation of antibiotics in residents of long-term-care facilities: results of a consensus conference, Infect. Control Hosp. Epidemiol.: Topics in Long-Term Care 22 (2) (2001) 120–124. [54] F.J. Garjón, M. Aza, M. Elfau, A.I. Aller, M.J. Buisán, C. Celaya, Indicadores de calidad de prescripción seleccionados mediante una técnica de consenso, An. Pediatr. 69 (4) (2008) 329–334. [55] M.L. Barry, H. Steyn, A. Brent, Determining the most important factors for sustainable energy technology selection in Africa: application of the Focus Group Technique, PICMET 2008 Proceedings, Cape Town, South Africa (July 2008) 27–31. [56] K. Tan, G. Baxter, S. Newell, S. Smye, P. Dear, K. Brownlee, J. Darling, Knowledge elicitation for validation of a neonatal ventilation expert system utilizing modified Delphi and focus group techniques, Int. J. Hum. Comput. Stud. 68 (6) (2010) 344–354. [57] M.C. Robotin, S.C. Jones, A.V. Biankin, L. Waters, D. Iverson, H. Gooden, B. Barraclough, A. Penman, Defining research priorities for pancreatic cancer in Australia: results of a consensus development process, Cancer Causes Control 21 (2010) 729–736. [58] J.A. McDougal, C.M. Brooks, M. Albanese, Achieving consensus on leadership competencies and outcome measures: the Pediatric Pulmonary Centers' Experience, Eval. Health Prof. 28 (4) (2005) 428–446. [59] F. Gibson, M. Fletcher, A. Casey, Classifying general and specialist children's nursing competencies, J. Adv. Nurs. 44 (6) (2003) 591–602. [60] J. Landeta, J. Barrutia, A. Araujo, J. Hoyos, Claves del comportamiento de la empresa respecto a la Formación Continua de sus directivos, Thomson-Civitas, Cizur Menor, Navarra, Spain, 2007. [61] J. Landeta, J. Barrutia, J. Hoyos, Management turnover expectations: a variable to explain company readiness to engage in continuous management training? Int. J. Hum. Resour. Manag. 20 (1) (2009) 185–194. [62] J. Sánchez, La interacción sistema familia y sistema empresa como ventaja competitive. PhD dissertation. Facultad de Ciencias Económicas y Empresariales. University of the Basque Country 2010. http://www.ieae.net/file.php/1/Estudios/Hybrid_Delphi/Family_Business._Jon_Sanchez.pdf. Accessed on 27 October 2010. [63] L. Alonso, Evaluación de las competencias clave de la enfermera en el ámbito asistencial: elaboración de un sistema de evaluación, análisis de los factores organizativos que intervienen en el nivel de competencias y recomendaciones para su implantación, PhD unread, Facultad de Ciencias Económicas y Empresariales, University of the Basque Country, unpublished results. [64] M. Kane, W.M.K. Trochim, Concept Mapping for Planning and Evaluation, SAGE, Thousand Oaks, 2007. [65] N.L. Klenk, G.M. Hickey, A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: the concept mapping policy Delphi, Int. J. Forecast. 27 (2007) 152–165. [66] J.W. Murry, J.O. Hammons, Delphi: a versatile methodology for conducting qualitative research, Rev. High. Educ. 18 (4) (1995) 423–436. [67] K. Uegaki, M.C. de Bruijne, J.R. Anema, A.J. van der Beek, M.W. van Tulder, W. van Mechelen, Consensus-based finding and recommendations for estimating the costs of health-related productivity loss from a company's perspective, Scand J Work Environ Health 33 (2) (2007) 122–130. [68] Pi-Fang Hsu, Bi-Yu Chen, Integrated analytic hierarchy process and entropy to develop a durable goods chain store franchisee selection model, Asia Pacific Journal of Marketing and Logistics 20 (1) (2008) 44–54. [69] C. Wu, C. Lin, H. Chen, Optimal selection of location for Taiwanese hospitals to ensure a competitive advantage by using the analytic hierarchy process and sensitivity analysis, Building and Environment 42 (3) (2007) 1431–1444. [70] Hsu Pi-Fang, Selection model based on ANP and GRA for independent media agencies, Qual Quant 2 (April, 2010). [71] K. Fitch, S.J. Bernstein, M.S. Aguilar, B. Burnand, J.R. LaCalle, P. Lazaro, M. van het Loo, J. McDonnell, J. Vader, J.P. Kahan, The RAND/UCLA Appropriateness Method User's Manual, RAND Corporation Monograph Reports, RAND, Santa Monica, CA, 1981. [72] K. Kobayashi, F. Ueno, S. Bito, Y. Iwao, T. Fukushima, N. Hiwatashi, M. Igarashi, B. Iizuka, T. Matsuda, T. Matsui, T. Matsumoto, A. Sugita, M. Takeno, T. Hibi, Development of consensus statements for the diagnosis and management of intestinal Behçet's disease using a modified Delphi approach, J. Gastroenterol 42 (2007) 737–745. [73] K.W. Wyrwich, W.M. Tierney, A.N. Babu, K. Kroenke, F.D. Wolinsky, A comparison of clinically important differences in health-related quality of life for patients with chronic lung disease, asthma, or heart disease, Health Ser Res 40 (2) (2005) 577–591. [74] M. Sweidan, M. Williamson, J.F. Reeve, K. Harvey, J.A. O'Neill, P. Schattner, T. Snowdon, Identification of features of electronic prescribing systems to support quality and safety in primary care using a modified Delphi process, BMC Med. Inf. Decis. Making 10 (2010) 21. [75] C.F. Sori, D.H. Sprenkle, Training family therapists to work with children and families: a modified Delphi study, J Marital Fam Ther 30 (4) (2004) 479–495. [76] A.R. Gagliardi, M. Fung, Kee Fung, B. Langer, H. Stern, A.D. Brown, Development of ovarian cancer surgery quality indicators using a modified Delphi approach, Gynecol. Oncol. 97 (2005) 446–456. [77] A.J. Blow, D.H. Sprenkle, Common factors across theories of marriage and family therapy: a modified Delphi study, J Marital Fam Ther 27 (3) (2001) 479–495. [78] H.M. Esmaily, C. Savage, R. Vahidi, A. Amini, M. Zarrintan, R. Wahlstrom, Identifying outcome-based indicators and developing a curriculum for a continuing medical education programme on rational prescribing using a modified Delphi process, BMC Med. Educ. 8 (33) (2008) 1–8. [79] S.C. Hayne, C.E. Pollard, A comparative analysis of critical issues facing canadian information systems personnel: a national and global perspective, Inform Manage-Amster 38 (2) (2000) 73–86. [80] J. McLeod, S. Childs, Consulting records management oracles—a Delphi in practice, Arch Sci 7 (2007) 147–166. [81] K. Chakravarti, B. Vasanta, A.S.A. Krishnan, R.K. Dubash Modified, Delphi methodology for technology forecasting case study of electronics and information technology in India, Technol Forecast Soc 58 (1998) 155–165. [82] C.L. Paul, A modified Delphi approach to a new card sorting methodology, Journal of Usability Studies 4 (1) (2008) 7–30. [83] H.C. Choi, E. Sirakaya, Sustainability indicators for managing community tourism, Tourist Manage 27 (6) (2006) 1274–1289. [84] G.H. Turner, D.K. Weiner, Essential components of a medical student curriculum on chronic pain management in older adults: results of a modified Delphi process, Pain Med. 3 (3) (2002) 240–252. [85] P.M. Boendermaker, M.H. Conradi, J. Schuling, B. Meyboom-De Jong, R.P. Zwierstra, J.C.M. Metz, Core characteristics of the competent general practice trainer, a Delphi study, Adv Health Sci Educ 8 (2003) 111–116. [86] T.R. Stanton, J. Latimer, C.G. Maher, M.J. Hancock, A modified Delphi approach to standardize low back pain recurrence terminology, Eur Spine J 31 (December, 2010) 1–9. [87] C. Medrano Lopez, L. Garcıa-Guereta, L. Fernandez Pineda, P. Malo Maroto, J. Santos de Soto, J. Lirio Casero, P. Suarez Cabrera, F. Caballero Martınez, Consenso clínico sobre laprofilaxis de la infeccion por virus respiratorio sincitial y el uso del palivizumab en cardiologıa pediátrica, An Pediatr (Barc) 72 (6) (2010) 432. e1–432.e13.

J. Landeta et al. / Technological Forecasting & Social Change 78 (2011) 1629–1641


[88] N. Black, M. Murphy, D. Lamping, M. McKee, C. Sanderson, J. Askham, T. Marteau, Consensus development methods: a review of best practice in creating clinical guidelines, J Health Serv Res Policy 4 (4) (1999) 236–248. [89] G. Rowe, G. Wright, F. Bolger, Delphi a reevaluation of research and theory, Technol Forecast Soc 39 (1991) 235–251.

Jon Landeta is Director of the Institute of Applied Business Economics of the University of the Basque Country. He has carried out several Delphi studies, both academic and professional, and has published the book El Método Delphi. Una técnica de previsión para la incertidumbre (The Delphi Method. A forecasting technique for uncertainty), published in 1999 by Ariel, Barcelona, in addition to numerous articles on Delphi and human resources in journals as Technological Forecasting and Social Change, International Journal of Forecasting, International Journal of Human Resources Management, The European Journal of Innovation Management or Journal of Intellectual Capital. Jon Barrutia (Ph.D degree at the University of the Basque Country, in 1988) is a Senior Research Fellow of the Institute of Applied Business Economics of the University of the Basque Country, and Professor and Director of the Department of Finance Economics II of the University of Basque Country. He was previously Deputy Minister for Universities and Research of the Basque Government. His research focuses on the following topics: Public Policy, Innovation, Governance and Human Resource Management. He has published en International Journal of Human Resources Management, International Journal of Forecasting and Nova Science Publishers Aitziber Lertxundi is PhD from the University of the Basque Country (2008) and Master in Finance. She is Professor in the Department of Finance Economics II in this university. Author of several research articles and other publications, her current research interests include: High Performance Work Systems; Human Resource Management in multinational enterprises and cultural approach in this last field.