Edtech505: Reflection #1

Up to now I have read the first six chapters of The ABCs of Evaluation for the course EdTech 505. These are the issues that I found most interesting.

Although I have already worked with different concepts of efficiency and effectiveness, I enjoyed the way the authors define both concepts: efficiency measures the relationship between costs and results, while effectiveness measures the relationship between goals and results. But most interesting, pointed by the authors, is that besides that, when evaluating a program we should also focus on its impact, that is to say, how it has changed behavior in an extended period of time. We usually do not measure that, when evaluating courses.

The idea that evaluation should be embeded into a program since its beginning is also interesting, as it forces us to start designing evaluation since we start designing the program itself, and not just when the program is already designed and even working.

The text also presents several evaluation models, which I was not aware of, as distinct models: discrepancy model, goal-free model, transaction model, decision-making model, and goal-based model. I am still not sure if choosing one of these models can really help in designing evaluation, or if the best strategy should be to mix several characteristics of them, which we can do even instinctively. I hope I can test this suspicion during this semester.

The idea of an evaluation design format is also interesting, including among other components: responsibilities of the evaluator and the audience for whom the report is intended.

Although the notion of quantitative data is easy to grasp, as it directly relates to numbers, the same is not true for qualitative data, which the authors relate to verbal narratives based on observations, interviews, surveys, case studies, and existing artifacts and documents. This is a concept I always wanted to explore more deeply, and it seems I will be able to do that during the practical projects for this course.

Another conceptual difference that was not clear to me was the one among nominal, ordinal, interval, and ratio data. Anyway, I am still not sure that understanding this conceptual difference helps too much in designing evaluation.

The suggestions for developing instruments to collect new data in the book are interesting and deeply explored, including: interviews (which can be structured, semistructured, or unstructured), scales (such as the Likert scale), sentence completion (presented in several different ways), standardized tests (with several examples), and observational analysis (also presented with several examples). I should be able to differentiate and use them, consciously, during this quarter.

I will be back with more reflection, as the course and my reading go on!

Compartilhar