Comparison of the degree of creativity in the design outcomes using different design methods

This work analyses the influence of several design methods on the degree of creativity of the design outcome. A design experiment has been carried out in which the participants were divided into four teams of three members, and each team was asked to work applying different design methods. The selected methods were Brainstorming, Functional Analysis, and SCAMPER method. The ‘degree of creativity’ of each design outcome is assessed by means of a questionnaire offered to a number of experts and by means of three different metrics: the metric of Moss, the metric of Sarkar and Chakrabarti, and the evaluation of innovative potential. The three metrics share the property of measuring the creativity as a combination of the degree of novelty and the degree of usefulness. The results show that Brainstorming provides more creative outcomes than when no method is applied, while this is not proved for SCAMPER and Functional Analysis.


Background
Design methods are a series of procedures, techniques, aids, or tools for designing, which represent a series of activities that the designer uses and combines in order to advance in the design process. There are many well-known design methods for stimulating creativity in the earlier phases of the design process (Brainstorming, SCAMPER, Six Hats, Lateral Thinking, Analogies, Functional Analysis, etc.), as can be seen in the collections of methods by Jones (1970), VanGundy (1988), Higgins (1994), de Bono (1970), and others. Several empirical and experimental studies have shown the advantages of applying these methods, usually in combination with sketches and other stimuli, in order to encourage creative thinking (Mulet and Vidal 2001, Van der Lugt 2001, Schütze et al. 2003, Bilda and Gero 2005 among others. According to Shah and Vargas-Hernandez (2003) the methods for idea-generation can be classified into two main groups: intuitive and structured or logical ones.
In a previous study (López-Mesa et al. 2011), ideation effectiveness of SCAMPER method in comparison with that for Visual Brainstorming method has been evaluated using Shah's metric (Shah and Vargas-Hernandez 2003). SCAMPER -which stands for Substitute-Combine-Adapt-Modify-Put to other uses-Eliminate-Rearrange (Vehar et al. 1999) -is a method where the group seeks stimulation from an idea-prompting checklist. In this previous study, ideas feasibility was measured in terms of the time dedicated to a solution and as the rate of attended reflections, using reflection as suggested in Valkenburg (2000). The novelty of a solution generated was evaluated by identifying the similarities of each alternative solution of a team with every alternative solution of the other teams at the level of action function, conceptual structure, and detailed structure. The novelty level indicates how dissimilar the solution is from all other solutions. This study concluded that using SCAMPER method leads to generation of more potentially feasible ideas than does using visual stimuli in a Brainstorming method.
However, the precise influence that design methods have on the degree of creativity of the designed product is not clear, due, among others, to the complexity involved in assessing or measuring creativity. This complexity is even higher in the earlier phases of the design process, when usually several ambiguous design ideas are proposed, and the designers have to decide which one(s) to select for proceeding further.
A deeper understanding of the relative influence that design methods have on creativity would help select the most suitable one depending on the particular goal.
The aim of this work is to determine the influence of several design methods in the degree of creativity of the design outcome. The objectives of this paper are: • To evaluate the relative creativity of different design methods and • To assess how far the academic methods for measuring creativity by means of the degree of novelty and usefulness match with the intuitive notion of experts.
The present work shows an experiment developed for this purpose, where four multidisciplinary design teams are required to solve three different design problems using different methodologies. The methodologies chosen for the experiment are Brainstorming (Osborn 1953), which is focused in idea-generation, Functional Analysis (Jones 1970), which is focused in problem analysis, and SCAMPER (Eberle 1996), which represents a middle point between the other two methodologies. According to Shah and Vargas-Hernandez (2003) classification, the Brainstorming is a strong intuitive method, also sub-classified as a germinal method, while Functional Analysis is classified as a logical and structured one, sub-classified in the group of analytical methods. So these two methods have been chosen for exemplifying two opposite kinds of methodologies in terms of intuitiveness and structureness. The third chosen method, the SCAMPER, despite the fact that it is classified as intuitive by Shah et al., does not represent the extreme of intuitiveness, as Brainstorming does. SCAMPER is sub-classified as transformational, and it represents a moderate intuitiveness, so it has been chosen as a middle point method between Brainstorming and Function Analysis. This paper is organised as follows. Section 2 describes the methods used for measuring the degree of creativity of the design outcomes. Section 3 describes the research experiment and the analysis carried out with the experimental data. Section 4 presents the results, Section 5 shows the discussion of the results, and Section 6 provides some concluding remarks.

Creativity assessment
Literature provides over a hundred definitions of creativity, many with overlapping elements, see Sternberg and Lubart (1999), Shah and Vargas-Hernández (2003), Sarkar andChakrabarti (2007, 2008). Sarkar and Chakrabarti (2007) proposed a common definition of engineering design creativity by analysing a comprehensive list of such definitions. Here, the elements constituting creativity from various definitions are found to occupy a hierarchy of influences on creativity, with novelty and usefulness of the outcomes occupying the most direct links with creativity. Similar definitions are proposed by numerous authors, like Moss (1966) or Besemer and O'Quin (1989). The proposed common definition of creativity is, 'Creativity occurs through a process by which an agent uses its ability to generate ideas, solutions and products that are novel and useful'. Because of its comprehensiveness, this definition is used in this work. The core elements of creativity are 'novelty' and 'usefulness', and a direct measure of creativity should be formed in terms of these two (Chulvi et al. in press). Note that this is an outcome-based definition of creativity, where creativity is judged in terms of the characteristics of the outcomes generated as a result of the creative process or person. This raise is opposed to a person or process-based definition of creativity, in which the focus is, respectively, on the characteristics of the person or the process involved. The argument for this choice is that outcome-based definitions are the most direct forms, since whether a person or a process is creative is ultimately evaluated by the quality of their outcomes.
An outcome is 'new' if it has been recently created (Cambridge 2007). 'Novel' outcomes are those that are new to the entire human race: 'novelty' encompasses both new and original (Cambridge 2007). Novelty is 'not resembling something formerly known' (Sternberg and Lubart 1999). The assessment of novelty of an outcome, therefore, requires comparison with previously known ideas (Sarkar and Chakrabarti 2007).
Dictionaries define 'Usefulness' as the quality of having utility value or practical benefits to the society (Merriam 2007, Oxford 2007. Usefulness of a product therefore should be assessed by its actual use, and when this information is not available, using estimation of its potential use (Sarkar and Chakrabarti 2007).

Methodologies for creativity assessment
2.1.1. Metric of Moss Moss (1966) developed a methodology to identify and estimate the creativity level of a product through a combination of two different parameters: usefulness and unusualness. Usefulness is determined by comparing the level of fulfilment of the product requirements with a standard solution considered correct, which is termed as 'teacher's solution'. This parameter is measured in a 0-3 scale, where 0 corresponds to a design that does not perform the demanded requirements, and 3 corresponds to a solution considered better than 'teacher's' one.
Unusualness is determined through the reverse probability of that idea appearing within a homogeneous group of solutions. It is assessed by comparing the product with similar products that solve the same problem. So, this is a comparative method for which evaluator must be familiar with possible solutions that can appear and the frequency in which these solutions may appear, and he must be able to deduce which probability appearance is considered as normal for average solutions and translate the probability deviations to unusualness rate. This rate has also a value in a 0-3 scale, where zero means that the solution is very common and 3 means that it is exceptionally original.
Finally, creativity degree is calculated by multiplying the two values, usefulness and unusualness. So, creativity is presented in a scale from 0 to 9.  Sarkar and Chakrabarti (2008) present the creativity level of a design as the product of its novelty and usefulness. Similar to the previous metric, creativity is defined as the combination of two parameters. Similarity of meanings among these parameters is perceptible, since they both use the term usefulness, and unusualness may lead to novelty.

Metric of Sarkar and Chakrabarti
The authors use the SAPPhIRE model (standing for State-Action-Part-Phenomenon-Input-oRgan-Effect) model of causality by Chakrabarti et al. (2005) to assess the relative degree of novelty of products (see Figure 1). It has seven elementary constructs. 'Action' is an abstract description or a high-level interpretation of a change of state, a changed state, or creation of an input. 'State' refers to the attributes and their values that define the properties of a given system at a given instant of time during its operation. 'Physical phenomena' are a set of potential changes associated with a given physical effect for a given organ and inputs. 'Physical effects' are the laws of nature governing change. 'Organs' are the structural contexts needed for activation of a physical effect. 'Inputs' are energy, information or material requirements for a physical effect to be activated. 'Parts' are the physical components and interfaces constituting the system and its environment of interaction. Parts are necessary for creating organs, which with inputs activate physical effects, which are needed for creating physical phenomena and state change. State changes are interpreted as actions or inputs, and create or activate parts. Activation, creation, and interpretation are the relationships among the constructs.
For detection of relative degree of novelty in products that are not 'very highly novel' (i.e. those that do not satisfy a function for the first time), state change and input constitute the next level of novelty ('high' novelty), physical phenomena and physical effect the next level ('medium' novelty), and organs and parts constitute the lowest level ('low' novelty) at which a product can be different from other products. By combining these measures, a method for novelty detection has been developed which employs function-behaviour-structure (FBS) (Gero 1990, Umeda et al. 1990) model initially (to find if a product is very highly novel or not) and the SAPPhIRE model later to assess the relative degree of novelty with respect to other products. The method has been formally evaluated in terms of the degree to which its output (i.e. the degree of novelty of products as determined using the method) matched with the output of experienced designers (i.e. the degree of novelty of the same products as perceived by these designers) (Sarkar 2007, Sarkar andChakrabarti 2007).
The usefulness of a product is measured in terms of the degree of usage a product has or is likely to have in the society. The scale is provided by a combination of several elements to assess the degree of usage: the importance of the product function, the number of users, and how long they use it or benefit from it. Together, these give a measure of how extensive the usefulness of the product is to the society. This is explained below.
As to how important the use of a product is depends on its impact on its users' lives. Some products are indispensable; products that are more important to the society should have a higher value for their usefulness. Five levels of importance of products are used in this measure (Sarkar 2007, Sarkar andChakrabarti 2007): extreme importance (e.g. life-saving drugs), very high importance (e.g. compulsory daily activities), high importance (e.g. shelter), medium importance (e.g. machines for daily needs), and low importance (e.g. Entertainment systems). All other parameters being the same, the products that are used by a larger number of people should be more useful to the society. Products that are used more frequently or have longer duration of benefit are likely to be more useful to the society; assuming that their 'level of importance' and 'rate of popularity' is the same, the rate of their usage increases their usefulness. These parameters are combined using Equation (1) below to assess degree of usefulness. (1) L stands for level of importance; F for frequency of usage (how often people use it); D for duration of benefit per usage; R for rate of popularity of use (how many people use it). Formal evaluation of the measure by comparing the ranking of usefulness among various sets of products by experienced designers and that found using the measure showed a high degree of correlation. Lastly, creativity is measured here as the product of the degree of novelty and usefulness. So, in order to assess relative degree of creativity of a product in a given set, the following steps are carried out: (1) Assess novelty of each product on a qualitative scale: 'Very high novelty', 'High novelty', 'Medium novelty' and 'Low novelty' using the novelty assessment method. (2) Convert the qualitative novelty value of each product into quantitative values: Very high novelty = 4 points, High novelty = 3 points, Medium novelty = 2 points, and Low novelty = 1 point. Convert these into relative ranks. (3) Assess the usefulness of each product using the method described before.
(4) Convert the usefulness value into relative ranking using the following scale: if there are five products that are ranked 1-5, give them 1 5 , 2 5 , 3 5 , 4 5 , 5 5 points, respectively. (5) Calculate creativity of a product as a product of its degree of novelty and usefulness.

Evaluation of innovative potential (EPI)
The EPI method (Justel 2008) proposes a measure for determining a product's creativity on the one hand, and on the other hand it proposes a method for predicting the exit of the product from the market, in order to determine the measure of its innovation. As the aspect related to the present work is that which measures creativity, only that part of the EPI will be used here.
In this part of the methodology, design requirements are established in the first place, and a weight is assigned to each to represent their relative importance. As the original method is focused on costumer likes in order to evaluate the market success, the higher weight is assigned to requirements of the user's over-satisfaction, while one-dimensional requirements have a lower weight (3), and a value of zero is assigned if the requirement is supposed in the design (basic).
The proposed adaptation for the present work in order to evaluate the creativity in agreement with the other metrics is to invert the weights of the design requirements, since it is considered that these can indicate the utility better when they are more related to the functionality. So, the higher weight (9) is assigned to basic functions, while over-satisfaction ones will be rated with zero. One-dimensional requirements will have the intermediate value (3). A coefficient corresponding to accomplishment grade will be assigned to each requirement in a design. If the requirement is totally and satisfactorily achieved, the coefficient value is set to 9; if the requirement is only partially accomplished the value is 3; and a value of 1 is assigned if it is weakly accomplished.
Novelty value assigned to each design is determined by the innovation criteria proposed by García and Calantone (2002). A design whose innovation degree is defined as incremental will have a value of 1; a value of 2 is given to innovations defined as moderate, and 3 for radical innovations. The value is 0 if the design is not innovative.
From these weights, coefficients, and novelty values, a creativity rate, also called 'absolute potential' (AP) in the EPI, for each design is calculated from the following equation: ( 2 ) Justel (2008) also recommends to normalise the AP value to a relative value in percentage, where 100 corresponds to the maximum potential level that can be achieved within the group of analysed items.

Design experiment
In order to fulfil these objectives, a design experiment has been carried out in which 12 persons participated. All participants are designers and engineers in a design PhD programme or are working as professional designers. The participants were divided into four multidisciplinary teams of three members each, since it is defended that multidisciplinary teams reinforce creative competencies and allow for rich combinations of otherwise disconnected pools of ideas (Alves et al. 2007), as it has been demonstrated in several experimental works (Peeters et al. 2007, González-Cruz et al. 2008. Each team was asked to work in three one-hour sessions, applying different design methods to solve different design problems. Each of the four teams was composed of one designer, one mechanical or industrial engineer, and another designer or engineer.
The selected design methods were: (i) Brainstorming, which encourages free generation of ideas, being applied in two different versions (Version 1 and Version 2); (ii) Functional Analysis, for involving an abstract problem analysis; (iii) the SCAMPER method, in which the current paradigm is broken by means of series of questions, generating in this way new design ideas; and finally (iv) no method, in which design teams have no instructions or guidelines to follow, and they are free to discuss their ideas in their own way. These methods are better explained above, in Table 2. The reason to consider two different Brainstorming methods was to compare the results of these studies with those from previous design experiments in which these versions had been applied (Chakrabarti 2003, López-Mesa 2004.
Four design problems were selected to solve in the experiment: design of a drawing table occupying as little space as possible when it is not in use (Problem 1), a tubular map case for one by one extraction and introduction of maps (Problem 2), a system to bring together and to hide the wires in a table (Problem 3) and a table for alternating stand up and sit down position (Problem 4). These four problems have been chosen due to their similar level of difficulty according to the disposable time for the experiment, their need of multidisciplinary knowledge in order to achieve good results, and because they present similar potential on achieving creative results. The objective of these choices is to attempt that the design problem do not interfere in the level of creativity of the outcomes. Also, problem 2 has been chosen because it has been used in previous research experiments (López-Mesa 2004, López-Mesa et al. 2011, and studying them again would provide additional data for comparison. Each team got assigned different design problems and different design methods in each session to minimise the noise factors. These factors can potentially arise from difference in participants, problems and methods used across the experiments, since it is impossible to keep these variables constant across the experiments. Table 1 summarises the experiment arrangement, and Figure 2 shows a picture of each design team during the experiment. The experiment was carried out simultaneously for the four teams in a single day. Each team was located in a separate room with a facilitator. The tasks and timing for each design session were organised into four steps: Step 1. Preparatory meeting (15 min) with the participants to explain and apply the design method to a short exercise unrelated to the actual problems.
Step 2. Solving the actual problem (30 min) applying the design method prescribed.
Step 3. Evaluating and selecting one solution (10 min). Neither instructions nor any prescribed method were provided to do this.
Step 4. Documentation. (10 min). During the last 10 min, the participants were asked to prepare the following information: a detailed sketch with major dimensions and materials, describing how it works, explaining how it solves the problems, who the beneficiaries are, and why they should buy and use it.
Each design session was recorded for further analysis. In each room, a set of information and tools were available for the participants: • A computer display showing pictures of current products which solve the design problems proposed ( Figure 3). • A panel summarising the design method steps to apply.
• For Brainstorming methods: small cards.
As indicated in Table 1, every team started solving the first problem in which no method was prescribed to be applied, and they then solved two additional problems each time applying a different design method. This allowed us to obtain appropriate data with which to analyse the relative influence of a design method in comparison with the one in which no method is asked to be applied. The steps for each design method are described in Table 2.
Tables 3-6 describe the design outcomes for each experiment run. Next, the relative degree of creativity of each design outcome from the experiment is assessed in two ways: by means of experts' evaluation and by applying the measurement methods explained in Section 2.

Experts' evaluation
A questionnaire was designed to evaluate the degree of novelty, usefulness, and creativity. For each design problem, the questionnaire asks for ranking each one of the three solutions from 1 to 3, with 1 being the most novel, useful or creative of the three, and 3 being the least. For each solution, a sketch and an explanation are provided.
The questionnaire was then offered for survey to experts. Here, an expert is taken as a person with at least eight years of professional experience in the design of furniture and similar products. Seventeen experts answered the questionnaire.
The responses from these questionnaires are analysed and the median, mode, and standard deviation were obtained. The experts' responses present a significant dispersion, and therefore one expert had to be excluded from the study because the deviation of his answers was higher than 3 sigma. The 'mode' was used instead of the 'mean' to assess the relative degree of novelty, usefulness, and creativity, because the mean has proven to be not really useful when analysing ordinal ratings.

Evaluation using Moss' metric
In order to assess creativity in Problem 1, unusualness has been analysed in the first step. Here, it has been considered that the solutions developed using no method and using SCAMPER have several similarities between them as well as with market solutions, so their unusualness value has been set as 1. On the other hand, the solution achieved with BR1 shows a less conventional solution, so a value of 2 has been assigned to it.
In terms of usefulness, the SCAMPER solution is very bulky and complicated to assemble and disassemble each time it has to be used, so it has been assigned a value of 1. The solution developed using no method makes manipulation easier compared with the last solution, so it has been given a value of 2. Lastly, the solution developed using BR1 fulfils with all the main objectives in an efficient way, so this value has been set as 3.
Creativity values are calculated as a product of the novelty and usefulness, as seen in Table 7.
In order to evaluate the creativity value of solutions for Problem 2, it has been considered in the first place that both the solutions developed using no method and using SCAMPER present low originality from reality, so their value of unusualness is 1. The solution developed using BR1 is considered slightly more original than the previous two, so it is evaluated with 2. Concerning utility, considering the capacity, and the ease of use, both the solutions developed using no method and SCAMPER are equally valid, so they each have a value of 2, while the solution using BR1 presents an improvement of existing solutions since it improves its capacity. The final values of creativity assessment are shown in Table 8. Table 2. Steps followed for the design methods in the experiment.
Brainstorming Version 1 Use these steps as a part of your idea-generation process.
1. Think about ideas for the problem, and say every idea aloud to the team. 2. Write down the idea on a card immediately. (It is not necessary that you write down a very word.) 3. Let the card available on the middle of the table. 4. Every time that someone is saying his/her idea, the others must listen to him/her and they will try to add something new or improve what has been told. It is important that you work as a team. 5. Put together those cards with similar ideas. 6. You are encouraged to take any card to read it again in order to find out more ideas. Follow these 4 rules: -Criticism is ruled out: try not to think of utility, importance or feasibility, and certainly no critical remarks during the Brainstorming. -Freewheeling is welcomed: the wilder the ideas, the better.
-Combination and improvement of ideas sought: add to or build upon ideas of others.
-Quantity is wanted: think of as many associations as possible.
Functional Analysis 1. Express the overall function for the design in terms of the conversion of inputs into outputs • Concentrate on what has to be achieved by the design, rather than how.
• Create a black box of inputs and outputs which defines the overall function as broadly as possible widening the system boundary. • Ensure that all relevant inputs and outputs are included, which can be classified as flows of materials, energy or information. 2. Break down the overall function into a set of essential sub-functions.
• Each function statement should be expressed in the same way (sentence with a verb and a noun).
• Each sub-function has its own inputs and outputs, and compatibility between these should be checked.
• There may be auxiliary sub-functions that has to be added but which do not contribute directly to the overall function. 3. Draw a block diagram showing the interactions between the sub-functions.
• A block diagram consists of all the sub-functions separately identified by enclosing them in boxes and linked together by their inputs and outputs so as to satisfy the overall function of the design. • It is useful to use different kinds of lines for different kinds of inputs. 4. Draw the system boundary.
• Decide which part of the block diagram will be satisfied by the system designed so as to define a feasible product. • This may be required in order to narrow down the scope of the product. 5. Search for appropriate components for performing the sub-functions and their interactions. Many alternative components may be capable of performing the identified functions.
Brainstorming Version 2 1. Now carry out a Brainstorming exercise on the actual problem. Steps for a Brainstorming exercise are: • Write, at the beginning of the session, the statement of the problem clearly visible to everyone, as well as the four rules (see at the end of this page). • Ask the participants to raise their hand if they want to communicate an idea. Have the facilitator write down the idea. Carry this on until the team's ideas have been exhausted. 2. Have the participants themselves delete silly ideas which have only served as stepping stones and have them cluster the remaining ideas according to similarity. The four rules are: 1. Criticism is ruled out: Try not to think of utility, importance or feasibility, and certainly no critical remarks during the Brainstorming step (step 2). 2. Freewheeling is welcomed: The wilder the ideas, the better. 3. Combination and improvement of ideas sought: Add to or build upon ideas of others. 4. Quantity is wanted: Think of as many associations as possible.

Scamper
This is a series of questions that are intended to help you changing an existing solution into a new solution. You have all these existing solutions and you have to generate new ideas applying these questions. Use these questions as a part of your idea-generation process, and you must select the most appropriate idea to solve the problem.

Questions
What can be blended, mixed, or included? What if you reverse the assemble? What are other ways to use it? In the case of Problem 3, the solutions developed using BR2 and that using no method are similar to those present in the market, so they have a low unusualness value (1), while the solution developed using Functional Analysis presents a higher unusualness level (2). Regarding utility, the solutions resulting form the application of Functional Analysis and BR2 are as valid as the available solution (value of 2), but the solution achieved with no method barely fulfils the basic requirements (value of 1). The resulting values of creativity are shown in Table 9.
Lastly, solutions provided by Functional Analysis and no method for problem 4 show a low unusualness level (1), since they are considered very common, while solution found with the application of BR2 is considered as very unusual, so in this case the value assigned is 3. Oppositely, this last solution presents a very low utility compared with a standard solution (1), while the other two solutions, those provided by Functional Analysis and no method, present a usefulness level as good as a standard one, so their value is set as 2. Table 10 shows the final values of creativity assessment of problem 4.

Evaluation using the metric by Sarkar and Chakrabarti
The degree of novelty was measured applying the SAPPhIRE model, thus comparing the differences in action, state change, physical phenomenon (PP), physical effects (PE), organ, parts, and input between the obtained solutions and the previously known solutions for each problem. For Problem 1, Table 11 shows the FBS and SAPPhIRE models for some existing solutions. Table 12 summarises the analysis of differences between the obtained solutions and the existing products for problems 1, 2, 3, and 4, and indicates the degree of novelty and the relative ranks.
Using the method as explained above, it was found that some of the factors that influence the degree of usefulness have the same value for all the three solutions obtained for every design problem. Then, only those having a different score are mentioned in this sub-section.
For Problem 1, the rate of popularity of use (R) and the rate of usage (FD) are the same for the three solutions. The level of importance falls in the medium category, but intermediate points can be assigned considering that the solution obtained with BR1 has a slightly higher level of importance since it occupies the least volume, and the solution obtained with no method occupies less volume than that by the solution obtained using the SCAMPER method. The height and the angle of tilt can be controlled Easy to fold up and store. It can also be used as a blackboard The digital pencil is placed on a standard pencil and it transforms the pencil movements into infra-red signals that are sent to a computer To be open out from the wall two belts are used

Wheels (to move)
It can be manipulated as a holographic system. Hydraulic system (to stand up the board, to change its position) You could write, erase, etc., in a virtual way.
Two legs (to support) The table could be used in all the positions: to work siting down, standing up, in different angles of tilt, etc.
Rack (to keep pencils and objects in the board, avoiding these to fall down) For Problem 2, again the rate of popularity of use (R) and the rate of usage (FD) are the same for the three solutions. The level of importance also falls in the medium category, but the solution obtained with BR1 has a slightly higher level of importance since it provides the best separation between the maps, and the solution obtained with no method separates the maps a little better than the solution obtained using the SCAMPER method.
For Problem 3, four issues concerning the level of importance and the rate of usage have been identified to be different for the three solutions obtained in the experiment: ease of change and expected lifespan (FA > BR > NM), flexibility to connect in many different points (FA and NM > BR2) and capability to hide the wires (BR2 > NM > FA). Finally, equal weighting for the four factors has been applied and the final usefulness rank is: FA > BR2 > NM.
For Problem 4, the rate of usage and the popularity of use (R) are considered to be higher for the solutions obtained by applying Functional Analysis and with no method, since these tables can be used for more functions, such as supporting objects. A slightly higher level of importance is assigned to the solution obtained with no method, because it has more additional uses. Table 13 summarises the relative degree of usefulness for each design outcome. Creativity values are calculated as the product of novelty and usefulness.

Evaluation using EPI metric
The basic requirements considered in the evaluation of creativity level of solutions provided for Problem 1 are that it must be comfortable for drawing at home, which means to be easy to manipulate every time, and that it must allow the professional drawing, which means providing enough working surface. Another requirement is that it must not to take up too much room, which has been considered a secondary or one-dimensional requirement. Solutions developed using SCAMPER and without method are very similar to those existing in the market, so they are considered as incremental innovations and their value is 1. The solution developed using BR1 is considered as a radical innovation. Regarding the accomplishment rate, for taking up little room the following are considered: • The solution using no method: it fulfils the requirement totally.
• The solution using BR1: it fulfils the requirement totally.
• The solution using SCAMPER: it allows disassembling the structure of the surface, but only just, so the accomplishment is moderate.
Regarding the requirement of being comfortable for drawing at home: • The solution obtained using no method: it makes manipulation easier, but requires to be attached to the wall, so it is valued as moderate. • The solution obtained using BR1: it makes manipulation easier, but requires to be attached to the wall as well, so it is valued as moderate. • The solution obtained using SCAMPER: is very bulky and complicated to assemble and disassemble each time it is to be used, so the accomplishment rate is minimum.
Lastly, the requirement of allowing professional drawing is totally achieved by all three solutions. Table 14 shows the result of the assessment. The basic requirements in Problem 2 are to transport maps and to store them, while the two one-dimensional requirements are: to be easy to extract one single map each time, and the map's capacity. For the novelty level, the SCAMPER's solution is considered to be as good as no method's solution with both at an incremental innovation level, while the solution using BR1 presents a higher novelty level, considered as moderated.
Regarding the fulfilment of the requirements, it has been considered that all three solutions fulfil the requirements of transporting and storing maps.
The requirement of map's capacity presents the following evaluations: • The solution obtained with no method: as it includes divider elements inside the tube the total inner space is limited, fulfilling the requirement only moderately. • The solution obtained with BR1: the design makes maximum utilisation of its capacity, so it fulfils the requirement totally. • The solution obtained with SCAMPER: similar to the first solution, the inclusion of divider elements inside the tube limits its capacity, fulfilling the requirement only moderately.
Regarding the ease of extracting a single map: • The solution using no method: extracting one single map is easy, so it fulfils the requirement completely. • The solution using BR1: extracting one single map is easy, so it fulfils the requirement completely. • The solution using SCAMPER: the extraction requires handling with care in order to fit each map in each slot; it also requires time to fix the clip, so it is decided that the requirement is achieved only moderately.
The complete results of assessment using EPI for Problem 2 are shown in Table 15.
For Problem 3, the basic design requirements are the ease of extracting and inserting cables, and that cables must be tied up. There is also a one-dimensional requirement for cable hiding. The novelty of all three solutions is considered incremental, since none of them differs noticeably from existing products.
Evaluation of the accomplishment rate for the requirement of hiding the cables is as follows: • The solution using no method: low accomplishment rate, the cables are hardly hidden. • The solution using BR2: it is moderately accomplished since cables are hidden in the leg of the table, but a part of them will always be over the table until they connect to the plug. • The solution using Functional Analysis: it is moderately accomplished since it allows inserting the cables in any side of the table.
The evaluation of the accomplishment rate of the ease of extracting and inserting cables requirement is as follows: • The solution using no method: it is completely accomplished since employing Velcro requires expending little time or effort. • The solution using BR2: it shows a moderate accomplishment rate, since cables are hidden inside the table's leg, so it is easy to extract a cable but not so easy to insert it. • The solution using Functional Analysis: it is completely accomplished since no difficult manoeuvres are needed to extract or insert a cable.
For the requirement of tying up the cables, the evaluation is as follows: • The solution obtained using no method: low degree of accomplishment, as it takes up a large amount of space over and around the table. • The solution obtained using BR2: moderate accomplishment, as there are cables remaining over the table. • The solution obtained using Functional Analysis: moderate accomplishment, since the cables can be inserted from any side of the table.
The results of complete creativity assessment using EPI for Problem 3 are given in Table 16.
The main basic requirement in the case of Problem 4 is that it must be useful in offices. This requirement takes into account the usual tasks in an office, and dimensions and distributions of workspaces. As one-dimensional requirements, there must be enough surface area for working, and it must be easy for a worker to change his position to be able to work sitting or standing. In order to calculate the novelty level of the solutions, those achieved by means of Functional Analysis or no method are considered to be very similar to the existing ones. So, they present an incremental novelty, while the solution achieved by using BR2 presents a radical innovation, and has the maximum value.
Regarding requirement accomplishment, the ease of changing to sit-stand positions presents is considered below: • The solution obtained with no method: it presents a low degree of accomplishment, since the rail system needs time and effort. • The solution obtained with BR2: it is totally fulfilled; since there is no weight, no effort is needed. • The solution obtained with Functional Analysis: the availability of a hydraulic system helps to carry out the manoeuvre, but it needs considerable time to change the position, so it has been assigned a moderate level of fulfilment.
Concerning the need of enough surface area to perform office work, the evaluations are as follows: • The solution obtained with no method: it fulfils the requirement completely.
• The solution obtained with BR2: it has a low fulfilment; although digital work is possible, the solution does not allow other uses like taking handwritten notes, freehand drawing, leaning objects on it, etc. • The solution obtained with Functional Analysis: it fulfils the requirement completely.
Lastly, for the requirement of being useful to offices, the evaluation is as follows: • The solution obtained with no method: it shows a moderate accomplishment rate; since the structure must be attached to a wall, so central spaces of the office rooms are wasted. • The solution obtained with BR2: it fulfils completely, since the structure can be positioned anywhere inside office premises. • The solution obtained with Functional Analysis: it fulfils completely, since the structure can be positioned in different configurations inside office premises. Table 17 shows the results of the complete assessment using EPI for Problem 4.

Results
In this section we present the data analysis for evaluating the values of novelty, usefulness, and creativity, made with the data collected from the experts' evaluation and the assessment with the different methods proposed by the research community. The first part of this analysis shows a comparison between the experts' evaluation and researchers' metrics, in order to validate the results of the measurements. The second part of the analysis consists of a comparison of the data obtained from the different design methods used to solve the same problem. The aim of this is to determine which design method produces better design outcomes.
In order to facilitate the comparison among the different scales and to treat the data statistically, values of novelty, usefulness, and creativity have been normalised in a 0-1 scale. The normalised values are calculated using the next equation: where 'min' is the minimum value of the selected metric and 'max' is the maximum value. Table 18 illustrates the normalised values. These values are organised according to the solved problem and to the design method applied to solve the problem. The columns show the normalised values from the questionnaires completed by the experts and those obtained using the different metrics.
As it can be seen from Table 18, most of the metrics tend to point to the same conclusions. For Problems 1 and 2, all metrics mark BR1 as the methodology that provides better results both in novelty and usefulness, and consequently in creativity. The only one deviation is presented by the experts' evaluation, who consider the usefulness' value of BR1 as the lowest one for Problem 1, and the novelty and creativity's values of the same method as the lowest ones in Problem 2, opposite to the results from the rest of the metrics. In both these cases, they consider the no metric's solutions as the better ones in these parameters. In Problem 3, all metrics and experts' evaluation coincide in pointing the Functional Analysis as the one that produces the best results in novelty, usefulness, and creativity. Nonetheless, EPI shows a draw among the novelty using the three methods, and Moss considers as equally useful the results achieved with Functional Analysis and BR2. Finally, for Problem 4, all assessments agree in pointing BR2 as the method that produces the most novel results. Experts and Sarkar and Chakrabarti consider the most useful results as those produced using no method, while EPI points to Functional Analysis, and Moss makes a draw between Functional Analysis and no method. Regarding creativity, Moss, EPI, and experts consider the BR2 results as the most creative ones, while Sarkar and Chakrabarti point to no method's results but closely followed by BR2 results.
Although in some cases the normalised absolute values presented by the different metrics do not exactly match, relative values between different methods for a same problem do tend to coincide (Table 19). That is, for the same problem, all assessments tend to point to the same method as the one which presents better outcomes with respect to a particular parameter. For this proposal, the rate of coincidence has been calculated by dividing the number of cases that the metrics compared rate similarly by the total number of cases. This tendency happens with a higher frequency in case of Moss and Sarkar and Chakrabarti's metrics, which coincide 89% of the times. The opposite is the comparison between experts' opinion and EPI, which has the lowest rate (56%). However, in the case of experts' evaluation, the lower coincidence ratio is observed when evaluating the solutions achieved by using no method (50-58%) and in the case of BR1 (50%). Here, solutions have been undervalued in half of the cases if we compare with the other metrics, while the coincidence rate between experts when evaluating the outcomes provided by BR2, SCAMPER, and Functional Analysis remains high: 83% coincidence with Sarkar and Chakrabarti and with Moss, and with respect to EPI it declines when evaluating outcomes of BR2 (33%). This low coincidence rate when evaluating BR2 outcomes is shown by EPI also when compared with Sarkar and Chakrabarti (17%) and Moss (33%). EPI presents not very high coincidence rates also when evaluating no method outcomes (50%), but it does present high coincidence rates when comparing the assessment of the other three methods (between 67% and 100%).
Since Problems 1 and 2 have been solved using the methodologies SCAMPER, BR1, and with no method, and Problems 3 and 4 using Function Analysis, BR2, and no method, the design method comparison is made within these two different groups, referred to as Group a and Group b, respectively. Figure 4(a) and (b) shows the normalised values assigned to novelty in both groups. As can be seen, BR1 in Group a, and BR2 in Group b have the most novel results of the groups. Within Group a, the results show that SCAMPER and no method present similar novelty except in the case of experts evaluation, who have considered the no method solution to be the most novel. For Group b, the results show that BR2 has also the most novelty results while no method produces the lowest novelty results.
The results for usefulness are shown in Figure 5(a) and (b). Here, the design outcomes from Brainstorming 1 is rated as the best within Group a, followed by BR1. Nonetheless, the experts   Global  67  67  56  89  64  69  68  NM  58  58  50  92  50  50  60  BR1  50  50  50  100  100  100  71  BR2  83  83  33  67  17  33  57  SC  83  83  67  100  83  83  83  FA  83  83  83  67  consider again the no method results as the best ones, contrary to the rest of the metrics. On the other hand, all of them agree that the SCAMPER method provides results with the lowest usefulness. Regarding Group b, Function Analysis presents the best results and Brainstorming 2 seems to show the worst ones. In the same way, Figure 6(a) and (b) presents the normalised values assigned to creativity. Here BR1 is the best considered within Group a, followed by no method, while SCAMPER is situated in the last place. As it happened with novelty and usefulness, the experts' opinion gives better rating to the no method results than the scientific metrics do. In Group b, the most creative results are achieved with BR2, closely followed by Function Analysis. Both experts and Sarkar and Chakrabarti's metric consider the two methods to provide the same creativity level, while Moss and EPI point to Function Analysis and BR2, respectively.

Discussion
Different creativity assessment methods have been applied to the design outcomes obtained in the experiment in order to compare the measuring methods and for giving more consistency to the assessment. The assessments coincide in 68% of the cases, but it has been seen that this percentage varies depending on the metrics compared and the methods evaluated. So, the matching is considerably lower when measuring the results achieved with the use of no method, while they are higher when evaluating the outcomes of closer and more elaborated methodologies such as SCAMPER and Function Analysis (see Table 19). This variability can be caused because when no method is applied, the final objectives for designers are not so clear and that makes results more difficult to evaluate, while the more elaborated methodologies make final objectives clearer for designers, so improvements in results will be more easily recognised and evaluated.
The fact that different metrics built independently provides similar results gives more confidence both to the metrics and the assessment. The comparison between the values provided by Sarkar and Chakrabarti and Moss' metrics presents the best results. The results are also good when comparing the experts' assessments with the metrics of Sarkar and Chakrabarti and Moss, when evaluating results provided by SCAMPER, Function Analysis, and BR2, but not as good in the cases of no method or BR1. The deviations in this case can be due to the questionnaire and the kind of experts that answer it. Nonetheless, it is also remarkable that the experts' responses present a significant dispersion, especially in the degree of usefulness. This may indicate that in the absence of available data, even experts find it difficult to assess and compare the potential usefulness of a product. It is also noticeable that experts found difficulties in differentiating between novelty and creativity. In fact, from Table 18 we can notice that in all cases except for one, the experts have assigned the same value to both novelty and creativity.
EPI has a good match with the other two metrics when evaluating SCAMPER, Function Analysis, and BR1, but it drops when evaluating BR2. This can be due to the adaptation made by the authors over the original EPI methodology in order to measure the usefulness of the solutions according to the other metrics. Also, EPI has the worst match with the experts' evaluation, but this can be caused due both to the EPI's adaptation and the mentioned characteristics of experts' evaluation.
The results obtained provided information about the degree to which design methods influence the relative degree of novelty, usefulness, and creativity of the outcomes. From Figure 4, it is found that the Brainstorming method, both in Version 1 and 2, encourages the generation of results with more novelty, while SCAMPER's outcomes present the lowest degree of novelty. This can be due to Brainstorming methods being centred on idea-generation, so it is more likely to help find novel ideas. The level of novelty achieved by using Function Analysis is slightly higher that when no method is used.
Analysing the relative degree of usefulness ( Figure 5), Functional Analysis presents clearly more useful solutions than the other methodologies within Group b. This result can be due to Functional Analysis being centred in functionality, so the more functional a solution is, the more useful it is supposed to be. Regarding Group a, Brainstorming 1 is found to be related to more useful outcomes than when no method is applied. Also, the available data indicate that neither SCAMPER nor Brainstorming 2 seems to encourage generation of more useful outcomes than when using no method.
Finally, regarding creativity assessment (Figure 6), Brainstorming 1 provides more creative outcomes in Group a than when no method is applied, since it presents the best results in both novelty and usefulness within its group. Opposite of this seems to be the case of SCAMPER, which shows the worst results. In Group b, the outcomes of Brainstorming 2 are similar to those achieved with the use of Function Analysis, the first one because of its better novelty results and the second one due to its better usefulness results. Also, both of them are clearly more creative than when no method has been used.
Since the SCAMPER method does not provide with high usefulness outcomes, but in previous studies using SCAMPER a higher number of feasible ideas are found to have been generated in comparison with Visual Brainstorming (López-Mesa et al. 2011). Further research should also analyse the feasibility of the outcomes and its relationships with usefulness.

Conclusions
This study presents a comparison of the degree of creativity of the design outcomes using different design methods with the aim of ascertaining which design method will help to produce more creative design outcomes. In the study the methods Brainstorming, SCAMPER, and Functional Analysis were compared, where Brainstorming is the most intuitive one and Functional Analysis the most structured one. Despite the limitation of the number of methods used in the present study, they reflect the two extremes of methods in terms of intuitiveness-structureness, and the middle point between extremes, so they are considered enough to defend the results presented in this paper. Different creativity assessment methods have been applied to the design outcomes obtained in the experiment. On the one hand, in order to compare the measuring methods with the aim of helping designers in the process of selecting methods for measuring design creativity outcomes, and on the other hand for giving more consistency to the assessment of the design methods' creativity.
The results of the assessment show the advantage of using structured design methodologies in order to facilitate the evaluation of the design outcomes, since structured methodologies make final objectives clearer for designers. So, they allow improvements in results to be more easily recognised in the design outcomes' evaluation process. In this case, more elaborated metrics, such as Moss or Sarkar and Chakrabarti's ones, are strongly recommended for making the evaluation, since they present the best concordance results compared with experts' evaluation and avoid the dispersion present in the experts' opinions. This dispersion has shown to be higher in the evaluation of usefulness, while they obtain better accuracy when evaluating novelty. Further research should analyse the particular experts' notion of usefulness, and also explore whether a higher number of experts' evaluation is needed and whether the experts profile should be modified.
The main conclusion of the present work is that methods based on idea-generation, also known as intuitive methods, provide more novel outcomes, while the most useful outcomes are achieved with the use of more structured methods. As it has been seen in the results section, the Brainstorming, which is the most intuitive method, provides the best rated outcomes in terms of novelty. On the other hand, the Functional Analysis, which is the most structured one, provides with the best rated outcomes in terms of usefulness. In any case, results when using any methodology have been better than when no method is used.
Nonetheless, some considerations and limitations must be considered when assessing the validity of the results of the research, as the number of experiment runs and the involvement of the teams, which can vary across the problem and method combinations, which could be the factors that may be responsible for the outcomes. A further limitation is that it is a macroscopic study, looking at the gross parameters of creativity. The study could be extended to include design process parameters -such as the quality of problem understanding and solution generation, and solution evaluation -along the lines of the work initiated in Chakrabarti's work (2003). Also, personal parameters such as experience, creative style, etc. could be varied and their influence analysed along with the method's parameters.