What Can OER Do for Me? Evaluating the Claims for OER

What Can OER Do for Me? Evaluating the Claims for OER Book Section How to cite: Weller, Martin; de los Arcos, Beatriz; Farrow, Robert; Pitt, Rebecca and McAndrew, Patrick (2017). What Can OER Do for Me? Evaluating the Claims for OER. In: Jhangiani, R. S. and Biswas-Diener, R. eds. Open: The Philosophy and Practices that are Revolutionizing Education and Science. London, UK: Ubiquity Press, pp. 67–78.


Introduction
This chapter will explore the role of research in the emerging OER discipline, with a particular focus on the work of the OER Research Hub at the UK Open University. OER have the potential to impact upon many aspects of education, such as improved performance, cost savings and development of new approaches to teaching. In this chapter we will explore some of the claims made for the benefit of OER, and the research that is now emerging to support these claims. The benefits of OER are often deemed to be self-evident -a free textbook is 'evidently' better for a student than one which costs US$100, for example but at the start of the OER movement this evidence was often lacking. As the field matures, the importance of research to demonstrate the actual impact of OER in expected, and unexpected, ways becomes useful in deciding how the can be most effectively deployed.
The start of the OER movement is often given as the announcement of the MIT OpenCourseWare project in 2001, although this itself grew out of other movements such as learning objects, open source and open universities. 1 However, if we accept 2001 as the start of the OER movement in earnest, then it is now some 15 years old. This is still relatively young in terms of academic disciplines, but it provides a useful timeframe to examine how research has changed over that period.
In terms of establishing itself as a global movement, OER has been something of a success story compared with many educational developments. There are repositories in most major languages, considerable funding has been provided by foundations such as Hewlett, and national bodies such as JISC (the Joint Information Systems Committee) in the UK, and there have been a number of policies at the national, and institutional level setting out a programme for releasing or funding OER. 2 The advent of open textbooks is an example of an area that has seen particularly clear gains. These are electronic versions of standard textbooks that are freely available and can be modified by users. The physical versions of such books are available at a low cost to cover printing, for as little as US$5. 3 The motivations for doing so are particularly evident in the United States, where the cost of textbooks accounts for 26% of a 4-year degree programme 4 creating a strong economic argument for their adoption. Projects such as OpenStax have targeted subject areas with large national student populations, for example 'Introduction to Psychology, ' 'Concepts of Biology, ' 'Introduction to Sociology, ' etc. The books are co-authored and authors are paid a fee to author the books, which are peer-reviewed. The books are released under a CC-BY license, and educators are encouraged to modify the textbooks to suit their own needs. In terms of adoption the OpenStax textbooks had been downloaded over 120,000 times and 200 institutions had decided to formally adopt OpenStax materials, leading to an estimated saving of over US$30 million in a little over two years. 5 However, open textbooks are only one instantiation of the OER approach, and projects such as the Open University's OpenLearn focus on releasing more online, distance education focused content.
In summary then the OER movement has managed to grow substantially over the past fifteen years. It has released a vast amount of educational material, and established a range of implementation projects across the globe. In this time we have witnessed different phases, from startup, to growth and sustainability. This has happened in parallel with a number of related developments in the open education movement, namely the success of open access publishing, particularly through national mandates, 6 and the more recent popular attention garnered by so-called massive open online courses (MOOCs). This has created a context in which the OER movement views the next phase as one of becoming mainstream in educational practice. For example, the Hewlett Foundation White Paper (2013) on OER states that its goal is 'to pave the way towards mainstream adoption of OER in a manner that promotes greater, sustainable educational capacity, ' and the theme of the 2015 OER conference in the UK was 'mainstreaming open education ' . 7 Research in OER is seen as a key component in facilitating this mainstreaming, as it provides evidence for benefits. OER Research has been increasing as the field matures. The OER Knowledge Cloud is a project gathering together all publications that relate to OER (https://oerknowledgecloud.org/), and an examination of the number of publications per year is shown in Table 1.
The data in Table 1 demonstrate a considerable increase in publications in 2007, as the initial projects became established and follow-on ones were implemented. This can be seen as the first expansion of the OER movement from a few key projects. Then in 2010 there is another sharp increase in the number of publications, which can be seen as a further expansion and acceptance of the OER approach.
Much of the early period of the OER movement was characterised by a lack of rigorous research however. The emphasis was on establishing OER projects, and developing content. For instance, a content analysis performed by the authors of the 2007 publications in the OER Knowledge Cloud revealed the categories and number of articles shown in Table 2.
This demonstrates that as the field began to grow, resources were focused on developing the projects and infrastructure required, along with theorising about the application of OERs. The sort of impact data that is desired by decision makers was often not resourced, or performed with a lack of independence and objectivism.  However, empirical data to evidence these beliefs was usually absent. This was the impetus for the founding of the OER Research Hub.

OER Research Hub
The OER Research Hub is a project at the UK Open University which was funded by the Hewlett Foundation in 2012, to address this perceived need to develop a more robust evidence base for the impact of OERs. Drawing on previous research and in dialogue with the Hewlett Foundation, the project developed eleven hypotheses which represented some commonly stated beliefs and motivations regarding OERs. These were derived from previous experience, consultation with Hewlett Foundation and stakeholders, and analysis of common claims in OER literature. The full set of hypotheses is: A -Performance: Use of OER leads to improvement in student performance and satisfaction. B -Openness: The Open Aspect of OER creates different usage and adoption patterns than other online resources. C -Access: Open Education models lead to more equitable access to education, serving a broader base of learners than traditional education. D -Retention: Use of OER is an effective method for improving retention for at-risk students. E -Reflection: Use of OER leads to critical reflection by educators, with evidence of improvement in their practice.

Category No Publications
Project case study/announcement 6 Technical infrastructure 6 OER general discussion, guidelines 11 Research with impact data 3 for the lack of formal support, which can be supported in open courses. I -Transition: Open education acts as a bridge to formal education, and is complementary, not competitive, with it. J -Policy: Participation in OER pilots and programs leads to policy change at an institutional level. K -Assessment: Informal means of assessment are motivators to learning with OER.
The project adopted a mixed methods approach. As well as gathering existing evidence onto an impact map (oermap.org), the project worked with 15 different collaborations, across four sectors: K-12 (kindergarten through secondary school), Community College, Higher Education, and Informal Learning. Interviews, case studies, and quantitative data were gathered, but this chapter mainly reports on responses to surveys. A set of survey questions was created, addressing the 11 hypotheses. Although slight variations were permitted depending on context, the same pool of questions was used across a wide range of respondents. These included students in formal education, informal learners, educators at K12, Community College and Higher Education level and librarians. In total 21 surveys were conducted, with nearly 7,500 responses.

Key findings
This section will provide an overview of some of the key findings, according to three types of OER users: informal learners, formal learners and educators.

Formal learners
Formal learners in this instance are categorized as anyone who indicated they were enrolled in a formal offering with an education institution. The key factors influencing formal learners decision to use OER were all related to cost and ease of access, with 88.1% of respondents stating it was the opportunity to study at reduced or no cost that was a key factor, followed by the material being available any time (79.6%) and online (79.3%). The survey revealed that many students (30% of those we surveyed) had studied their current formal subject in OER prior to signing up for a course and 52.7% supplemented their current study by using OER from another institution. There was broad coverage across all disciplines, with science being the most popular subject studied via OER (43.4%). Similarly there was a broad use of different types of OER, with video (79.2%) and open textbooks (79.1%) being the most popular formats.
The reported impact was often around attitudes to study rather than performance, with 61.9% stating that use of OER increased their interest in their subject, 60.7% increased satisfaction with the learning experience and 60.4% increased enthusiasm for future study. This is compared with only 38.9% who felt that it had improved their grades. When selecting OERs, the key indicators in choice were relevance to the students' particular needs or interests (72.6%), clear learning objectives (65.5%), and the reputation of the provider (60.6%).
Users were satisfied with their experience of OERs, with 83.5% stating they would study again with OER and 80% that they would recommend them.

Informal learners
In this context informal learners are defined as a learner who is not enrolled in a formal course of study. In reality, formal learners in one discipline can be informal learners in another, but this becomes a problematic interpretation, and so the clearer distinction was used in this work. For informal learners similar reasons for deciding to use OER were given, although the desire to study at no cost was more significant here with 89% of respondents stating this as a key factor. As with formal learners there was a broad coverage of topics, but the dominant disciplines were computer science (31.8%) and economics (30.6%). The same three elements guided choice as for formal learners, namely relevance, learning objectives and reputation. Here, OER can be perceived as plugging a specific gap in knowledge or skills. Less important were reviews of OER or personal recommendations. Open or Creative Commons licensing allowing adaptation was only thought important by a minority of 13.9%. Informal learners used a wide range of support methods, such as notetaking, discussion in social media and blogging, although not more than the formal learners.
As with formal learners, informal learners were likely to study using OER again (78.5%) and recommend them to others (80.4%). Only 24.6% stated that studying OER would make them more likely to take a paid for course, but this could still represent a very effective recruitment method, compared with other forms of marketing. Given the potential large numbers of OER users, this could translate into substantial numbers of new students for universities. However, this should be countered with the 19% who stated that studying OER had made them less likely to take a paid for course, either because the OER were sufficient to meet their learning requirements, or they decided study was not something they wished to pursue. These future study patterns for formal and informal learners are summarized in Figure 1.

Educators
When asked about the impact of OER on their teaching, there was strong agreement from both full and part-time educators working at a range of different institutions that OER gave them a broader range of teaching methods (64.3%) and caused them to reflect on their practice (59.4%). Corresponding to the views of learners themselves, educators believed that use of OER increased learners' satisfaction with the subject (62.1%), increased their interest (60.8%), and increased experimentation with new ways of learning (60.3%).
By far the most prominent motivation for using OER was to get new ideas and inspiration for teaching (78.2%). This is compared with only 40.7% who stated that their motivation was to get assets for use in the classroom, which is often presumed to be the main driver for OER.
The same three factors were deemed important for educators as for learners in choosing a resource, namely relevance, learning outcomes, and reputation. Only 43% stated that having an open license was significant. The open licenses of OER are seen as a key differentiator between them and general online resources, as it allows users to reuse and adapt. Whilst the presence of such a license may not be that significant to learners, it might have been presumed to be of significance to educators, but it was ranked 12th out of 16 factors in importance.
Most educators believed that use of OER saved their students money (73.1%), although this was lower for formal learners themselves (60.9%). This may arise because some formal learners purchased the physical textbook also.  For all users, the biggest barrier to use of OER was finding suitable resources, and OER repositories were used very little compared with resources such as YouTube, Khan Academy, and TED talks. Another finding that was consistent across all groups was the comparatively high level of adaptation. This might be expected to be high amongst educators (79.8%), but was also found in formal learners (77.3%), and informal learners (84.7%). What constitutes adaptation varies for these users, and is an area that requires further investigation. This is in contrast to other research which found previously low levels of adaptation. 8 For some users adaptation means using the resources as inspiration for creating their own material, as this quote illustrates:

Math Teacher, Grade 11
For other users, adaptation is more direct, editing or reversioning the original, or aggregating elements from different sources to create a more relevant one, as this quote demonstrates: Math & Science Teacher, Grades 7-8 And for others, adaptation may be taking an existing resource and placing it in a different context within their own material, for example: 'I will maybe look and find an instructional video that's maybe 2 or 3 minutes long that gets to the point better than I could, and I would use it, or I will look for lessons and if they are for Grade 5

Math teacher, Grade 2
What this suggests is that one impact of openness is that it allows a continuum of adaptation to develop, ranging from adapting ideas for teachers' own material to full reversioning of content.

Discussion
A general picture of OER usage emerges from this research. There was a high degree of satisfaction with OER across all types of user, with a large percentage willing to access further OER and to recommend them. However, OER brand recognition was weak compared with other popular resource sites, and finding appropriate OER was a major obstacle. Use of OER increases satisfaction and engagement with learning and is seen as saving students money. Users look for relevance, reputation and clear learning outcomes when selecting OER. The use of OER is not confined to one or two disciplines, with all subjects well represented, and a range of formats are accessed, although video remains the most significant.
These findings themselves would be significant and useful to the OER community. But what this research also reveals is a more nuanced, subtle picture of OER usage than had often been supposed. Many of the benefits of OER being portrayed in this usage are not the primary ones of improved cost savings or improved performance. While these are important gains, and ones that will have strong leverage on policy, the benefits discovered by the OER Research Hub may have more long lasting effects. For example, the use by students in formal education to both trial and then supplement their learning may have an impact on student retention. Combined with a quarter of informal learners stating that they were more likely to study a paid for course after using OER, and the sustainability model for OER may be one that can be couched in terms of recruitment and retention (compared with the sale of additional services as seen with MOOCs).
Another factor that is important to students, but also increasingly used as a metric to rank universities, is student satisfaction. The impact of OER on emotive aspects related to learning such as satisfaction, enthusiasm, and confidence could be of greater relevance than cost savings.
A similarly under-reported benefit for educators is the manner in which OER cause them to reflect on their own practice, and to broaden their teaching approaches. The use of OER is often couched in terms of benefit to the learner, but the impact on educators could be equally significant. This needs to be balanced with the relative unimportance users placed on open licenses. An open license potentially allows an educator to take existing content from several different sources (for example, different open textbooks, videos) and adapt this to their own context, to produce something that is ideally suited to their specific learning outcomes in a way that a generic textbook can never be. However, if awareness of open licensing remains relatively low, these pedagogic possibilities of open content will not be realized.
What this research highlights beyond the use of OER, is the significance of research itself in an emerging discipline. Open education in its current guise is still in its infancy as a discipline, and research plays an important role in how it evolves. In the early stages researchers are often in the role of advocates, but as the field matures, more objective research can be undertaken. It is necessary for a field to gain momentum for it to commence from a set of beliefs and assumptions about the potential impact. These can later be tested as the OER Research Hub and other groups such as the Open Education Research Group 9 , and ROER4D 10 have done. Just as important however is to research into unexpected usage. The type of actual use of OER that this research found helps inform the sector. For an emerging discipline seeking to become part of mainstream this is important as it helps inform strategy, policy, and direction in a more direct manner than is possible with more established domains.
In order to realize the ambition of mainstreaming OER then there are two research related aspects. The first is that this type of objective, impact research becomes part of all implementation projects. The second is that it is communicated effectively to help shape strategy. In addition, a number of specific recommendations might be drawn from this work. We believe that increasing the 'brand' of OER through joint initiatives, sequenced activities, and improved marketing could significantly improve uptake and adoption. Making the business case for OER beyond the immediate cost savings of textbooks is also important, for instance emphasizing the manner in which all students benefit by trialing subjects and complementing formal study, and educators use OER for development. 1 Weller, 2014. 2 Wiki, n.d. 3 Wiley, 2011. 4 GAO, 2005.