definition of evaluation by different authors

There is a distinction between academic impact understood as the intellectual contribution to ones field of study within academia and external socio-economic impact beyond academia. The process of evaluation involves figuring out how well the goals have been accomplished. 0000009507 00000 n Thalidomide has since been found to have beneficial effects in the treatment of certain types of cancer. While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context. Evidence of academic impact may be derived through various bibliometric methods, one example of which is the H index, which has incorporated factors such as the number of publications and citations. However, it must be remembered that in the case of the UK REF, impact is only considered that is based on research that has taken place within the institution submitting the case study. 60 0 obj << /Linearized 1 /O 63 /H [ 1325 558 ] /L 397637 /E 348326 /N 12 /T 396319 >> endobj xref 60 37 0000000016 00000 n Replicated from (Hughes and Martin 2012). Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. Again the objective and perspective of the individuals and organizations assessing impact will be key to understanding how temporal and dissipated impact will be valued in comparison with longer-term impact. Assessment refers to a related series of measures used to determine a complex attribute of an individual or group of individuals. This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). Time, attribution, impact. This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. The Author 2013. Any information on the context of the data will be valuable to understanding the degree to which impact has taken place. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, 0000002868 00000 n Wigley (1988, p 21) defines it as "a data reduction process that involves the . By asking academics to consider the impact of the research they undertake and by reviewing and funding them accordingly, the result may be to compromise research by steering it away from the imaginative and creative quest for knowledge. different things to different people, and it is primarily a function of the application, as will be seen in the following. 0000006922 00000 n Wooding et al. Not only are differences in segmentation algorithm, boundary definition, and tissue contrast a likely cause of the poor correlation , but also the two different software packages used in this study are not comparable from a technical point of view. 2010; Hanney and Gonzlez-Block 2011) and can be thought of in two parts: a model that allows the research and subsequent dissemination process to be broken into specific components within which the benefits of research can be studied, and second, a multi-dimensional classification scheme into which the various outputs, outcomes, and impacts can be placed (Hanney and Gonzalez Block 2011). Productive interactions, which can perhaps be viewed as instances of knowledge exchange, are widely valued and supported internationally as mechanisms for enabling impact and are often supported financially for example by Canadas Social Sciences and Humanities Research Council, which aims to support knowledge exchange (financially) with a view to enabling long-term impact. In the Brunel model, depth refers to the degree to which the research has influenced or caused change, whereas spread refers to the extent to which the change has occurred and influenced end users. The . (2005), Wooding et al. While the case study is a useful way of showcasing impact, its limitations must be understood if we are to use this for evaluation purposes. Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. 8. The Goldsmith report (Cooke and Nadim 2011) recommended making indicators value free, enabling the value or quality to be established in an impact descriptor that could be assessed by expert panels. Classroom Assessment -- (sometime referred to as Course-based Assessment) - is a process of gathering data on student learning during the educational experience, designed to help the instructor determine which concepts or skills the students are not learning well, so that steps may be taken to improve the students' learning while the course is 0000012122 00000 n 0000007777 00000 n In terms of research impact, organizations and stakeholders may be interested in specific aspects of impact, dependent on their focus. The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. There is . different meanings for different people in many different contexts. We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems. 0000004692 00000 n As part of this review, we aim to explore the following questions: What are the reasons behind trying to understand and evaluate research impact? This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. (2007:11-12), describes and explains the different types of value claim. It is a process that involves careful gathering and evaluating of data on the actions, features, and consequences of a program. The ability to write a persuasive well-evidenced case study may influence the assessment of impact. What are the reasons behind trying to understand and evaluate research impact? The definition problem in evaluation has been around for decades (as early as Carter, 1971), and multiple definitions of evaluation have been offered throughout the years (see Table 1 for some examples). stream (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from benefit to impact when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. They aim to enable the instructors to determine how much the learners have understood what the teacher has taught in the class and how much they can apply the knowledge of what has been taught in the class as well. Understand. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide, This PDF is available to Subscribers Only. Without measuring and evaluating their performance, teachers will not be able to determine how much the students have learned. We will focus attention towards generating results that enable boxes to be ticked rather than delivering real value for money and innovative research. A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). SIAMPI has been used within the Netherlands Institute for health Services Research (SIAMPI n.d.). 0000004019 00000 n Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. Metrics in themselves cannot convey the full impact; however, they are often viewed as powerful and unequivocal forms of evidence. As such research outputs, for example, knowledge generated and publications, can be translated into outcomes, for example, new products and services, and impacts or added value (Duryea et al. Consortia for Advancing Standards in Research Administration Information, for example, has put together a data dictionary with the aim of setting the standards for terminology used to describe impact and indicators that can be incorporated into systems internationally and seems to be building a certain momentum in this area. 0000007559 00000 n Introduction, what is meant by impact? What indicators, evidence, and impacts need to be captured within developing systems? This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. This is recognized as being particularly problematic within the social sciences where informing policy is a likely impact of research. 2007). (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. Evaluation is a procedure that reviews a program critically. Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle. What emerged on testing the MICE taxonomy (Cooke and Nadim 2011), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. 1.3. Oxford University Press is a department of the University of Oxford. The justification for a university is that it preserves the connection between knowledge and the zest of life, by uniting the young and the old in the imaginative consideration of learning. In the UK, UK Department for Business, Innovation, and Skills provided funding of 150 million for knowledge exchange in 201112 to help universities and colleges support the economic recovery and growth, and contribute to wider society (Department for Business, Innovation and Skills 2012). Assessment is the collection of relevant information that may be relied on for making decisions., 3. HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). It is important to emphasize that Not everyone within the higher education sector itself is convinced that evaluation of higher education activity is a worthwhile task (Kelly and McNicoll 2011). SIAMPI is based on the widely held assumption that interactions between researchers and stakeholder are an important pre-requisite to achieving impact (Donovan 2011; Hughes and Martin 2012; Spaapen et al. On the societal impact of publicly funded Circular Bioeconomy research in Europe, Devices of evaluation: Institutionalization and impactIntroduction to the special issue, The rocky road to translational science: An analysis of Clinical and Translational Science Awards, The nexus between research impact and sustainability assessment: From stakeholders perspective. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. Definitions of Evaluation ( by different authors) According to Hanna- "The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". Researchers were asked to evidence the economic, societal, environmental, and cultural impact of their research within broad categories, which were then verified by an expert panel (Duryea et al. Definition of Evaluation by Different Authors Tuckman: Evaluation is a process wherein the parts, processes, or outcomes of a programme are examined to see whether they are satisfactory, particularly with reference to the stated objectives of the programme our own expectations, or our own standards of excellence. In this article, we draw on a broad range of examples with a focus on methods of evaluation for research impact within Higher Education Institutions (HEIs). Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. 0000003495 00000 n n.d.). 0000007967 00000 n 2006; Nason et al. Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. However, the . What indicators, evidence, and impacts need to be captured within developing systems. Why should this be the case? There are a couple of types of authorship to be aware of. There has been a drive from the UK government through Higher Education Funding Council for England (HEFCE) and the Research Councils (HM Treasury 2004) to account for the spending of public money by demonstrating the value of research to tax payers, voters, and the public in terms of socio-economic benefits (European Science Foundation 2009), in effect, justifying this expenditure (Davies Nutley, and Walter 2005; Hanney and Gonzlez-Block 2011). The most appropriate type of evaluation will vary according to the stakeholder whom we are wishing to inform. The growing trend for accountability within the university system is not limited to research and is mirrored in assessments of teaching quality, which now feed into evaluation of universities to ensure fee-paying students satisfaction. n.d.). Impact has become the term of choice in the UK for research influence beyond academia. A collation of several indicators of impact may be enough to convince that an impact has taken place. Measurement assessment and evaluation helps the teachers to determine the learning progress of the students. Describe and use several methods for finding previous research on a particular research idea or question. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. In education, the term assessment refers to the wide variety of methods or tools that educators use to evaluate, measure, and document the academic readiness, learning progress, skill acquisition, or educational needs of students. It is desirable that the assignation of administrative tasks to researchers is limited, and therefore, to assist the tracking and collating of impact data, systems are being developed involving numerous projects and developments internationally, including Star Metrics in the USA, the ERC (European Research Council) Research Information System, and Lattes in Brazil (Lane 2010; Mugabushaka and Papazoglou 2012). In the UK, evaluation of academic and broader socio-economic impact takes place separately. A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. In the majority of cases, a number of types of evidence will be required to provide an overview of impact. In this sense, when reading an opinion piece, you must decide if you agree or disagree with the writer by making an informed judgment. Teacher Education: Pre-Service and In-Service, Introduction to Educational Research Methodology, Teacher Education: Pre-Service & In-Service, Difference and Relationship Between Measurement, Assessment and Evaluation in Education, Concept and Importance of Measurement Assessment and Evaluation in Education, Purpose, Aims and Objective of Assessment and Evaluation in Education, Main Types of Assessment in Education and their Purposes, Main Types of Evaluation in Education with Examples, Critical Review of Current Evaluation Practices B.Ed Notes, Compare and Contrast Formative and Summative Evaluation in Curriculum Development B.ED Notes, Difference Between Prognostic and Diagnostic Evaluation in Education with Examples, Similarities and Difference Between Norm-Referenced Test and Criterion-Referenced Test with Examples, Difference Between Quantitative and Qualitative Evaluation in Education, Difference between Blooms Taxonomy and Revised Blooms Taxonomy by Anderson 2001, Cognitive Affective and Psychomotor Domains of Learning Revised Blooms Taxonomy 2001, Revised Blooms Taxonomy of Educational Objectives, 7 Types and Forms of Questions with its Advantages, VSA, SA, ET, Objective Type and Situation Based Questions, Definition and Characteristics of Achievement Test B.Ed Notes, Steps, Procedure and Uses of Achievement Test B.Ed Notes, Meaning, Types and Characteristics of diagnostic test in Education B.ED Notes, Advantages and Disadvantages of Diagnostic Test in Education B.ED Notes, Types of Tasks: Projects, Assignments, Performances B.ED Notes, Need and Importance of CCE: Continuous and Comprehensive Evaluation B.Ed Notes, Characteristics & Problems Faced by Teachers in Continuous and Comprehensive Evaluation, Meaning and Construction of Process Oriented Tools B.ED Notes, Components, Advantages and Disadvantages of Observation Schedule, Observation Techniques of Checklist and Rating Scale, Advantages and Disadvantages of Checklist and Rating Scale, Anecdotal Records Advantages and Disadvantages B.ED Notes, Types and Importance of Group Processes and Group Dynamics, Types, Uses, Advantages & Disadvantages of Sociometric Techniques, Stages of Group Processes & Development: Forming, Storming, Norming, Performing, Adjourning, Assessment Criteria of Social Skills in Collaborative or Cooperative Learning Situations, Portfolio Assessment: Meaning, Scope and Uses for Students Performance, Different Methods and Steps Involved in Developing Assessment Portfolio, Characteristics & Development of Rubrics as Tools of Assessment, Types of Rubrics as an Assessment Tool B.ED Notes, Advantages and Disadvantages of Rubrics in Assessment, Types & Importance of Descriptive Statistics B.ED Notes, What is the Difference Between Descriptive and Inferential Statistics with Examples, Central Tendency and Variability Measures & Difference, What are the Different Types of Graphical Representation & its importance for Performance Assessment, Properties and Uses of Normal Probability Curve (NPC) in Interpretation of Test Scores, Meaning & Types of Grading System in Education, Grading System in Education Advantages and Disadvantages B.ED Notes, 7 Types of Feedback in Education & Advantages and Disadvantages, Role of Feedback in Teaching Learning Process, How to Identify Learners Strengths and Weaknesses, Difference between Assessment of Learning and Assessment for Learning in Tabular Form, Critical Review of Current Evaluation Practices and their Assumptions about Learning and Development, The Concept of Test, Measurement, Assessment and Evaluation in Education. Capturing knowledge exchange events would greatly assist the linking of research with impact. As a result, numerous and widely varying models and frameworks for assessing impact exist. Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on. Recommendations from the REF pilot were that the panel should be able to extend the time frame where appropriate; this, however, poses difficult decisions when submitting a case study to the REF as to what the view of the panel will be and whether if deemed inappropriate this will render the case study unclassified.

definition of evaluation by different authors 2023