MOBILE-APPS questionnaire: Developing and validating a scale to measure the attitudes and perceptions of undergraduate students on mobile information literacy

This paper aims at reflecting on the process of developing and validating a scale for measuring the students’ attitudes and perceptions regarding the use of mobile technologies in the teaching-learning of information competencies (MOBILE-APPS). Validation was carried out by administering the questionnaire to a pilot group of students, selected from Education degree, with a rubric to analyse the quality/coherence, clarity and usefulness of the content. The questionnaire was then piloted with a larger sample of students. To analyse the tool’s reliability and internal validity, scale validation techniques and exploratory factorial analysis were used. The resulting questionnaire, MOBILE-APPS, is a simple yet effective scale for collecting information. It can be applied in a number of university settings and degrees to ascertain student attitudes and perceptions of mobile information literacy.


Introduction
In today's higher education setting during the information and technology era, teaching the competencies for managing information -in other words, finding, evaluating, processing and communicating information -requires increasing levels of attention. The information literacy (IL) paradigm plays a key role in developing education initiatives and programmes that establish the foundation for the information competencies that must accompany students throughout their education. The aim is to teach critical thinking, which is essential for any professional profile after graduation. Information literacy is the basis for lifelong learning because it empowers students -future professionals -in any scenario, to find, evaluate, use and create information whenever needed. In this sense, the revised definition of CILIP Information Literacy Group (2018) is highly clarifying: literacy. It is not a stand-alone concept, and is aligned with other areas of knowledge and understanding. This comprehensive definition puts forward the metaliteracy view on IL that is also emphasized by the Framework for Information Literacy in Higher Education (ACRL, 2015). This is much more pressing and necessary today because students are millennials or digital natives (Rossing et al., 2012), members of the Google generation (Kiviluoto, 2015: 308), who use social media and mobile applications, which are integrated in their everyday life, without clearly differentiating academic and personal spaces, and constantly using information technology, sometimes without critical thinking or training. For this generation, technology is something natural and a constant presence. A recent report in Spain (Fundación Telefónica, 2017) found a very revealing fact: young people between the ages of 15 and 24 years define themselves as mobile first or even mobile only, meaning that they only manage and consume information on their smartphones.
Within the framework of the research project 'Innovation and training in the information competencies of university teachers and students in Social Sciences. Model for the development of programs in the mobile environment', and with regard to the phase for diagnosing the perceptions of IL levels in teachers and students, this project focuses on describing the process of validating the MOBILE-APPS questionnaire for measuring the Students' perception regarding the importance of using mobile technologies in the teaching-learning of information competencies (Mobile -Information -Literacy -Education -Attitudes -Perceptions -Prospectings -Students).
The specific objectives of this study are: 1. To describe the design of the MOBILE-APPS online questionnaire with which to measure students' attitudes and perceptions regarding the use of mobile technologies in teaching-learning information competencies. 2. To analyse the validity and consistency of the questionnaire. 3. To establish the tool's applicability based on its originality and usefulness.

Literature review
To answer this question, we used 2009 as the reference point to begin the literature review since no prior studies were found on tools aligned to the use of mobile technologies in the process of teaching-learning information competencies. Similarly, it must be taken into consideration that this last decade is when the use of smartphones has become commonplace in teaching-learning settings, particularly in higher education.
In relation to prior studies and tools, we must first consider the proposal by Kallaya et al. (2009). Its innovative nature was based primarily on accepting that smartphones form part of the teaching process and the various opinions about this. To do so, they presented a tool's validation that may be considered a starting point within the early stages of m-learning (mobile learning). The study by Wang et al. (2009) was positioned similarly, with a specific analysis of the situation within higher education. The authors took into account the integration of smartphones in the teaching and learning process, along with the acceptance of those devices. Additionally, a quantitative analysis considered other potential factors for differences, such as gender and age. The tool developed by Al-Fahad (2009) was positioned along similar lines, although it was in the very early stages of mobile integration in higher education. Also, this tool was made available to a very limited sample that only consisted of women because the aim was to analyse the gender variable in a setting where there are major differences between men and women in terms of their access to ICTs and education. In any case, both tools are some of the first to analyse the use of mobile devices in university classrooms and student perceptions of this purpose.
Another leading tool for analysing information competencies is IL-HUMASS (Pinto, 2009). This questionnaire was developed to analyse the following four macro-competencies in Humanities and Social Sciences university students: searching for information, evaluating, processing and communicating-distributing. Although IL-HUMASS does not directly address mobile devices, it falls within digital competency and technology in general.
In the location-specific setting of university students in Jordan, Khwaileh and AlJarrah (2010) piloted a questionnaire on an undergraduate and graduate level to analyse attitudes about the use of mobile devices in terms of their degree of acceptance in classrooms. The authors confirmed how, in the early stages of integrating mobile devices in teaching and learning processes, students showed growing interest and viewed devices as tools that were increasingly inherent to higher education settings. However, this tool is not centred on IL and omits student perceptions of teachers and how they implement the use of mobile devices in education.
From a qualitative perspective, Woodcock et al. (2012) carried out a study on implementing the use of mobile devices, installing and using mobile applications for educational purposes in higher education, and the benefits they offer. It confirmed that the gradual growth of their use is proportional to their acceptance, and highlighted certain limitations and the fact that a student segment is opposed to its use for academic purposes. The tool used in this study is based on a qualitative methodology and in-depth interviews, so it can be viewed as one of the few questionnaires of this type to be applied to m-learning.
Subsequently, Yang (2013) performed a quantitative analysis on the use and social influence of mobile devices with relation to the teaching and learning processes in universities. The model set out by the author assumes, through complexity, that when students use smartphones, their intent is based more on hedonism (personal use that entails satisfaction, such as communication between equals) than on academic purposes. However, this does not decrease the importance and growing potential of mobile learning. It also analyses the drawbacks resulting from distractions or the lack of critical attitudes towards information.
Vázquez-Cano (2014) also focuses on online learning and the use of mobile devices in university settings. His main objective is to analyse student attitudes of m-learning through the use of a quantitative scale. Su and Cheng (2015) go one step further with their questionnaire, which takes into consideration other factors such as motivation, its relationship with gamification, and achievements as well as results, in addition to the growing use and exponential growth of apps, and the possibilities they offer to ubiquitous learning. Pinto and Fernández-Pascual (2014) and Pinto et al. (2016) used the EVAL-CI questionnaire to analyse the beliefs of Humanities and Social Sciences students in Spain regarding the relevance and self-efficacy in information competencies and skills, highlighting the existence of a low self-perception in terms of efficacy, the growing use of mobile devices in academic settings and the need to delve further in this direction.
Lastly, it is worth noting that the design of the MOBILE-APPS questionnaire took into account the Framework for Information Literacy for Higher Education (ACRL, 2015), which is an undisputed example for reflecting on and rethinking IL education today, in addition to the tool piloted by Sharma et al. (2017), which measures the acceptance and generalization in the use of smartphones as an increasingly essential medium for accessing information, interacting, creating new resources and sharing knowledge. To do so, the authors considered a model made up of six factors: flexibility, adaptability, enjoyment, efficacy, economy and sociability.

Phases
Based on the existing tools that gave growing importance to the general use of mobile devices in the teaching-learning processes of higher education, several factors were considered when creating and piloting the MOBILE-APPS questionnaire. Firstly, the aim was to build an original tool, in a simple and applicable manner, which combined these three concepts: higher education, smartphones and acquiring information competencies. Secondly, the goal was to design an attainable tool since the complexity or length of certain tools often discourages students from participating in studies spontaneously and willingly. As a result, efforts were made to create a simple yet effective questionnaire for collecting information that could be applied in an array of university settings and degrees. Thirdly, it was taken into account that the tool should adjust to two interrelated aspects: the undisputed presence of smartphones in today's society (current situation) and the fast evolution of devices (changing adaptability situation).
To create the tool, the following aspects were also considered and directly formed part of the decision-making process: 1. In the first phase, tools that had already been validated were reviewed, including some of our own. Specifically, the IL-HUMASS and EVAL-CI questionnaires were reviewed, along with those proposed by Wang et al. (2009) and Yang (2013). IL-HUMASS (Pinto, 2009) is a critical survey developed for a group of diverse degrees in Social Sciences and Humanities. It tackles four informational categories (information search, assessment, processing and communication/dissemination) and three self-perceived dimensions (motivation, self-efficacy and preferred source of learning). EVAL-CI (Pinto and Fernández-Pascual, 2014) was developed to assess students' objective knowledge on information competencies. The work by Wang et al. (2009) offers a scale that can be used to analyse the decisive aspects in the acceptance and use of m-learning, and possible differential features, such as gender or age. The questionnaire provided by Yang (2013) deals with expectations on use and social influence of mobile teaching and its positive aftermath in the contexts of higher education. 2. A way was found to create an original and valid questionnaire that addressed the current needs of students as well as the aspects they identify with the most. 3. To validate the questionnaire, a rubric (Annex I) was created that considered four key areas (level of quality, clarity of items, usefulness and perceptions or opinions). It was shared with the volunteer students who had been chosen for this first phase of the test. The volunteers selected were fourth-year undergraduates majoring in Education at the University of Granada, the headquarters of the research project of which this study forms part. These students were chosen because they were nearing the end of their university education and, in light of the degree to which the required competencies had been acquired, they had a more critical attitude, were more mature and therefore had a greater ability to contemplate their teaching-learning process in higher education. 4. To validate the coherence of the questionnaire and its level of efficacy and clarity, it was piloted with a representative sample of Education undergraduates whose involvement has enabled us to quantitatively validate the tool and provide its final version. Second-, third-and fourth-year students were selected for the pilot to ascertain whether there were any differences between their perceptions. 5. Two focus groups were then created with volunteer students who were fourth-year undergraduates in Primary Education (n=30), belonging to two different academic specialities. During a two-hour period, certain conclusions were reached about the tool's quality, clarity and usefulness, as well as other aspects regarding student availability, the time needed to complete the questionnaire and opinions about the methodologies used in classrooms (Pinto et al., 2019).

Questionnaire structure
The first version of the scale consisted of 22 items clustered in four dimensions using a five-point Likert scale: from 'Totally disagree = 1' to 'Totally agree = 5'.
The first dimension focuses on information competencies. Since this basic skill for university students is being compromised by certain endogenous and exogenous factors, it is an increasingly necessary competency that requires greater instructional effort, so the first section asks students about their perception of IL on a personal level as well as in the university setting they form part of. Therefore, the aim is to perform a descriptive analysis of students' thoughts on critical attitudes, accessibility and the degree to which new methods and mediums have provided access to information, as well as the increase in teaching-learning processes and their challenges, and the university's role in this area.
2. An informationally literate person is one who. . .
The tool's second dimension is based on the information competencies that someone classified as informationally literate should learn. It is believed that students should develop these skills during their undergraduate studies, so during this educational stage, it is important to analyse the perceptions of IL skills, which refer to the needs, analytical and critical skills, use and distribution of information.

Rate the following statements
The third dimension of the questionnaire is based on the Framework for Information Literacy for Higher Education (ACRL, 2015) and focuses on the key interconnected concepts regarding information and its use, as opposed to standards or a prescriptive list of skills. As a result, it covers aspects linked to the authority and credibility of information, the value of information, research as questioning, knowledge creation as a form of dialogue, and the strategies needed to search for and select information.

ICT and students
The tool's last section refers specifically to the use of mobile technologies by students within the learning process. In addition to covering how devices are used in the classroom for everyday tasks (such as finding information), it analyses a key question that arises with the inevitable emergence of technology: how has technology influenced analytical-critical skills, and therefore its potential for creating distractions in the classroom.

Methodology
The questionnaire was validated in two stages. First, an evaluation rubric was designed and applied to a sample of 44 Education undergraduate students at the University of Granada. The rubric's qualitative and quantitative analysis provided useful information about the strengths and weaknesses of the questionnaire. A number of suggested improvements were proposed and the work team, upon reaching consensus, designed a second version of the tool that was applied to another sample of 105 Education undergraduate students. Throughout their university studies, the sample must complete two courses about technological resources applied to education. It should be noted that most of the materials covered regularly use group presentations in various formats as part of the methodology, so students are accustomed to using a number of apps and computer programs. To analyse the tool's reliability and internal validity, scale validation techniques and exploratory factorial analysis were used.

Results
The results are offered in two main sections. Firstly, the study on the quality and the coherence of the questionnaire is carried out by analysing the data from the evaluation heading in the first sample. Secondly, the answers collected through the second sample of students are analysed, evaluating the internal consistency, the main gender and course differences, and the latent structures in order to validate the multidimensionality of the proposed scale.

Rubric evaluation
Sample. The sample is made up of 44 undergraduate (fourth year) Education students from the University of Granada, between the ages of 21 and 36 years (average age Design. The evaluation rubric (see Table 1) is suitable and useful in order to evaluate both the quality and the coherence of the questionnaire from the view of the expertsteachers and to rate the items individually regarding their clarity (easy to understand, with simple statements) and relevance (the items should be clearly related to the subject of study). It uses the same 1-5 Likert scale that is applied to the questionnaire itself. On the scale referring to the rubric, 1 means that there is no match between the item of the questionnaire and its subject, meanwhile 5 would indicate an excellent match between the two: 1. Very low, 2. Low, 3. Medium, 4. High, 5. Excellent.
Analysis. The overall results of the questionnaire are broken down into the three dimensions addressed by the rubric (quality/coherence, clarity and usefulness), as follows: 1. Clarity/Quality/Coherence.
According to the results, students believe that the instructions are suitable (clear and legible), giving an overall score of 3.94 points. They also rate the tool's overall coherence and quality with a score of 4.07 (on a 1-5 Likert scale).

Clarity and usefulness of the items.
We provide a two-dimensional analysis of the clarity and usefulness of the 22 proposed items as follows. The descriptive results show which items offer the greatest clarity usefulness, according to the students' opinion and those that should be re-examined (see Table 2 for categories). It can be observed that items 5 (Assumes that teaching will be of an increasingly ubiquitous or blended nature) Maximum in bold, minimum in italics and 12 (Authority is constructed and contextual) did not reach the minimum clarity levels (mean -0.5std) required by the work team, they were reformulated for improved comprehension.

Validating the pilot questionnaire
The internal consistency and reliability of the MOBILE-APPS questionnaire is evaluated from the main descriptive measures of the responses, and Cronbach's alpha values are analysed. Finally, the proposed categories are validated by applying an exploratory factor analysis to identify the structure and layout of the latent categories. Consistency. To examine the reliability, we calculate the Cronbach's alpha for each category. Alfa values exceed the recommended minimum of 0.7 (Nunnally, 1978), so the scales show a good level of reliability, except for the category 4 -ICT and students (Table 3). The overall Cronbach's alpha of the MOBILE-APPS questionnaire was 0.851, which indicates a high level of reliability and internal consistency (Gliem and Gliem, 2003;Novick and Lewis, 1967;Osburn, 2000).
Analysis. The descriptive analysis reveals that the central values of student perceptions are between 3.8 and 4.03, with homogeneous criteria due to the low dispersion. There are no gender differences (U Mann-Whitney, p>0.05) in the mean scores for each category.
We found significant differences in the mean scores by year of study in category 4 -ICT and students (Kruskal-Wallis, p<0.05). This was due to second-year students giving lower scores, which means that they use ICTs less than third-and fourth-year students do, and they have a lower perception of their academic use.
The analysis by items reveals mean scores between 3.48 and 4.23. The most common score is 4, and only item 21 -Immediate access to information diminishes students' capacity for critical thinking, shows a dispersion that is higher than one, followed by item 18 -How often my students usually take notes or search for information using mobile devices (laptops, tablets or smartphones) ( Table 4).

Latent dimensions: Factorial analysis.
With the aim to discover the interrelationships, the latent structures and to analyse the multidimensionality of the scale, an exploratory factor analysis is applied. The Kaiser-Meyer-Olkin (KMO) measure of adequacy reaches 0.774, indicating appropriateness of using factorial analysis (exceeds 0.5, the minimum value required (Kaiser, 1974;Tabachnick and Fedell, 2007)).
Five factors were found (Preacher et al., 2013) explaining the 60.08% of the variance (Table 5). Details regarding the loading factors are displayed in Table 1A-Annex II. As a result, categories 1, 2 and 4 are consolidated (1. The informationally literate university. . .; 2. An informationally literate person is one who. . .; 4. ICT and students), which are shown in factors 1, 2 and 3. To strengthen the consistency of category 3, an explanation was added between brackets to item 12, in accordance with the philosophy of the ACRL Framework, and it was agreed to leave item 5 unchanged. Lastly, the research team took into consideration several suggested improvements to the text and, upon reaching a consensus, the final version of the MOBILE-APPS questionnaire was created (Annex III).

Discussion and conclusions
Firstly, it is worth noting that the MOBILE-APPS questionnaire was processed and created separately from the MOBILE-APP scale, which was designed for teachers . In other words, these are two independent questionnaires. However, they work together and have been designed to measure the perceptions of students (MOBILE-APPS questionnaire) and teachers (MOBILE-APP questionnaire) in the area of Social Sciences and with regard to the use of mobile technologies in the process of teaching-learning information competencies.
The MOBILE-APPS questionnaire was designed in collaboration with renowned experts in this field, and the Framework for Information Literacy (ACRL, 2015) was reviewed and taken into account. This framework stresses that IL is key in ensuring that students are informed and critical citizens -from a perspective of responsible lifelong learning -with a willingness to evolve in light of contextual changes and technological advances, always with a critical focus. The results obtained from our two-stage sequential validation (rubric distributed among volunteer students followed by the pilot and focus groups) confirm the scale's reliability and consistency.
First, the content validity that refers to the comprehensiveness and representativeness of the items in the scale was verified. To do so, a distributed evaluation rubric was used on the initial sample of 44 students, confirming high clarity and relevance of the proposed items, with mean values above 4 points and a low dispersion of less than one on a 1-5 Likert scale.
The pilot questionnaire was then distributed to a second sample of 105 students. The results showed favourable psychometric properties in terms of high reliability and a factorial structure in which the five sections or factors found are in line with the tool's structure, providing validity evidence of the proposed categories (Gadermann et al., 2010). Additionally, there were two focus groups that helped define the qualitative validation of the questionnaire in terms of its quality/relevance, clarity and usefulness. The scores given by students revealed interesting scenarios in the field of mobile IL. Specifically, as expected, students with higher levels of education, contact and experience are expected to have higher perceived self-efficacy and also include mobile technologies in their teachinglearning processes to a greater degree. In other words, the perceptions detected differ according to year of study, resulting in higher scores among those who are further along in their education.
On the other hand, perceptions among genders are similar to the importance of these information competencies and the use of mobile technologies in the learning process. Some studies have already revealed that the gender variable is insignificant; gender is not a pivotal factor that influences self-efficacy and attitudes toward m-learning or m-entertainment (Leong et al., 2013;Yang, 2012). This result suggests and welcomes a more in-depth analysis using a larger student sample. We leave this possibility open for a future study, already in progress, in which we distribute the MOBILE-APPS questionnaire to a representative sample of Social Sciences students in Spain. This will provide a global picture of the attitudes and perceptions on the use of mobile technologies for teaching-learning information competencies.
If other tools used in similar studies are taken into consideration, the one by Hanbidge et al. (2015) is worth noting. They used a longitudinal study to measure the information competencies of Social Sciences university students (Education, Social Work, Psychology, Social Education and Development studies). The results, which showed the importance of m-learning in acquiring information competencies, were obtained using a tool that combined qualitative and quantitative methods. However, it differs from this scale (MOBILE-APPS) because it is not a validated questionnaire and the pilot was used on a small group of students (fewer than 100). The tool we validated in this study has more potential in light of its agility and applicability for many students.
Other tools used in a number of studies, as in the case of Alhassan (2016) and Al-Emran et al. (2016), took into account the use of mobile devices in higher education as well as the attitudes of students and teachers towards the general use of mobile technologies in the classroom. Both cases confirm the need to adapt to the new reality and the benefits of using mobile technologies based on how apps improve the teaching-learning processes.
On the other hand, the setting plays a major role in acquiring those competencies. In any case, MOBILE-APPS introduces the combination of mobile-information literacy as a strength and an innovative aspect, as opposed to these two tools, which focus on using mobile devices, without directly including the mobile IL aspect that we believe is at the core of this study.
Lastly, it is important to consider the tool created by Doyle et al. (2018). PILS (Students' Perception of Information Literacy Skills) is based on the six fundamental pillars of the Framework for Information Literacy in Higher Education. The authors use those pillars to establish a scale of descriptors that define the level of information competency in students for every section, with seven different levels ranging from beginner to expert. Along these lines, we find commonalities with MOBILE-APPS since this frame of reference is fundamental in both instances. However, unlike the tools mentioned earlier, PILS only takes into account IL, without considering the use of mobile devices in the classroom and everything resulting from digital literacy.
In summary, MOBILE-APPS is a tool that combines the two basic aspects in the area of higher education (information literacy and mobile technologies), it is easy to manage and, when combined with the MOBILE-APP questionnaire (for teachers), can offer a dual instructorstudent perspective about the perceived self-efficacy regarding the importance of information competencies and attitudes towards m-learning. This integrated perspective makes MOBILE-APP an innovative tool with an intriguing potential for wide-range studies.