Tag Archives: Ed Tech

Syndicated, Virginia Group

Systematic Inquiry: A design perspective

The following post consists of the lightning round remarks I made as part of the OLC Accelerate 2018 Research Summit.  I was asked to speak on systematic inquiry – what is it and why it is important – to a group of higher ed online instructional designers, faculty, and administrators. 

I was an interesting choice as a panelist. The other members of the panel were CAOs, CEOs, and internationally known researchers.  I am a designer, not a researcher.  I am not a CAO or a CEO or a director. However, I am an online learning research super-user, a sometimes action-researcher, and a sympathetic critic who wants something more for our field and the people in and around it. These were the credentials I took up with me when I got on the stage.

This talk resonated with about 10% of the room and therefore cannot be considered well-received.  However, I’m used to being different and I’m glad I participated – I said exactly what I needed to say. 

Also, I owe much of what I said to this essay by RE Wyllys.  I credited Dr. Wyllys throughout my talk, but you should read his essay for yourself.

 

 

When I was asked to speak on systematic inquiry, I think I probably did the same thing that you all would have done: I googled it, read some articles and essays from a variety of disciplines, and then checked in with my Twitter followers just to make sure I wasn’t missing something big in terms of a discipline-specific meaning or theoretical framework.

Ultimately, I found a definition that seemed to make everyone happy – which may be the first and only time that happens on Twitter.  All sources seemed to agree that systematic inquiry is the careful, deliberate effort to understand a problem or an unknown.  It is the primary way in which we develop formal knowledge or practice within a discipline or field of study. In general, systematic inquiry is what people typically mean when they say something is “good” research, scholarship, or evaluation.

 

Slide2

 

 

In Zen and the Art of Motorcycle Maintenance, Robert Pirsig describes systematic inquiry as an artful combination of inductive and deductive logic that is best laid out formally to help promote reflection and sharing. He goes on to list the classic scientific method as what one should write down in their notebook.

 

Slide3

 

The scientific method is a great example of systematic inquiry, but when we consider what systematic inquiry means beyond a single question – through time and across networks of people – we need an orientation that goes beyond the scientific method.  Systematic inquiry can also be considered as a way of being in and interacting with a world that is relational, fluid, ethical, and emergent. It hinges on the act of questioning everything.

What are the values that drive our questions and designs? What are the politics of our descriptions? Why am we valuing these methods over others? What is our purpose and how does it align with our impact? For whom are we writing and why?

These are the sorts of questions that we must ask and answer as part of a longitudinal or systems-level vision of systematic inquiry.

 

Slide5

 

There are reasons we strive for systematic inquiry. The first can be found in Zen and the Art of Motorcycle Maintenance (although Albert Einstein also said something very similar):

“The real purpose of the scientific method is to make sure Nature hasn’t misled you into thinking you know something you don’t actually know.”

The world is complicated and we must approach it methodically or else we will get lost in the web of independent, dependent, constant, confounding, mediating, co-variables, and other things that are currently unimaginable that make up most things worth understanding.

 

Slide6

 

The second argument for systematic inquiry is deeply rooted in the social sciences. The act of reflecting on and seeking out diverse forms of feedback about every aspect of our work not only helps protect us from Nature, but also helps protect us from ourselves.

The cautionary tale of the streetlight effect is real; many of us make the mistake of focusing our attention on areas which are already familiar or easier to measure while ignoring other possibilities that are more difficult to pursue.

 

Slide7

 

Finally, systematic inquiry provides an infrastructure of shared language and practices that supports other forms of knowledge discovery and propels them forward. Roald Hoffman, the Nobel-prize winning chemist, writes that serendipitous discovery will occur when the mind has been prepared by intense, preferably interdisciplinary study, our institutions have provided space and opportunity for knowledge sharing, we have cultivated our senses of curiosity and wonder, and we have committed ourselves to hard work.

In considering the state of systematic inquiry in our field’s research literature, I am grateful to the online learning researchers who engage in systematic inquiry.  In fact, we have some of the best online learning researchers here in this room with us today.

However, as a field, online learning has arrived at a point in its development when it is time to reconsider how we approach online learning research – particularly at the level of teaching, learning, and instructional design – so that it is meaningful and useful to others in our field and beyond.

 

Slide8

 

It would be impossible to cover all aspects of online learning in a lightning speech, so I am going to focus on educational social media research: the body of literature that addresses the use of social media – including wikis, blogs, Twitter, YouTube, etc. – in the higher ed classroom.

As part of an ongoing project, Drs. Jessica Knot, Laura Pasquini and I recently reviewed almost 500 empirical articles on the use of social media in higher ed.

Our initial analysis indicates that educational social media literature is dominated by perception and use surveys and descriptive studies.  In general, these descriptive studies are written by practitioner-researchers who implement a social media tool in the classroom and are reporting on their experience and the learner’s perceptions of their experience.  Less than 10% of studies involved quasi-experimental or experimental design. Relatively few of the descriptive studies identified a specific qualitative research methodology and those that did rarely met baseline standards for rigor (or even discussed rigor).

 

Slide10

 

Our findings were consistent with Kimmerle et al (2015), which also looked at educational social media research: many of the articles written about social media use in the classroom focused on the technological aspects and specific functionalities of particular tools without consideration for their pedagogical purpose or associated learner outcomes.

As a field, our literature is hyperfocused on tools and often fails to place digital tools in the larger context of teaching, learning, and instructional design.

We cannot claim to be supporting systematic inquiry into online learning until we diversify our research questions and enhance the overall rigor of our research methodologies.

How do we fix this problem?  How do we design our systems and foster a culture to promote a systematic understanding of online teaching and learning?

 

Slide13

 

I have more questions than answers and I’m hoping we can continue the discussion through the breakout sessions, but here are a few observations and ideas to get us started.

Based on the hyperfocus on tool functionality, the limited discussions of instructional design, and thin applications and descriptions of research methodologies, I question how much pre-planning goes into the action research that accounts for a large portion of the online teaching and learning literature.  I also question the support many of these faculty, who are first and foremost subject matter experts, are receiving from instructional designers and methodologists as they write these articles.

At the theoretical level, collective IQ plays a large role in the value system of digital pedagogies and participatory culture.  So how can we operationalize collective IQ in a knowledge development context? How do we increase the inclusion of instructional designers, methodologists, learners, as well as instructors in the research process?

 

Slide14

 

One of the biggest weaknesses in the online teaching and learning literature is a lack of communication; there appear to be major differences in how researchers are defining key terms, instructional approaches, and research methodologies. I think a lot of that comes from the interdisciplinary nature of our field.

How do we foster interdependency and community within our interdisciplinary field so that we might create a shared understanding of key terms and methodologies?

Alternatively, if this is not possible, how can we promote transparency and thick descriptions so that readers can understand what is being described and judge the results for themselves?  Is that something that could be encouraged at the functional level of peer reviewers, editors, and journals?

 

Slide15

 

And finally, if we enhance the rigor around what counts as research, how can we make room and promote value for other important types of knowledge creation?  The sharing of design ideas and anecdotal experiences are valuable to other educators and designers, but perhaps they belong in other places or formats beyond the empirical journal article.  How do we create those spaces and encourage others to value them as they do a peer-reviewed research journal?

 

Slide16

 

These are some of the questions, I hope we will address today and through future conversations.  Thank you.

 

Syndicated, Virginia Group

A Post-IT Post

A post-IT post before I leave Newark.

During my  panel presentation at #WCET16 this AM, I heard some colleagues slip into tech-first talk, despite the fact I had been espousing a pedagogy-first perspective (and they had been nodding vigorously) for almost an hour. To make matters worse, some of them would start sentences with pedagogy and end with tech; the tools-first paradigm is THAT ingrained in these people.  So, in my final moments, I pulled out my VCU example:

Desire to facilitate Connected Learning –> let’s try blogging and tweeting to promote connected learning –> let’s assess for connected learning.  EVERYTHING, is done and considered in the context of the pedagogy, connected learning.  I defined the terms, of course. I offered an example of a specific course (Collaborative Curiosity – you know it :)). Administrators around the room were nodding and smiling…it really is a great course design. So intentional.  People were drawn in by the course but some of them actually got my point: Connected Learning first, blogging/tweeting second.

Then a vendor in the room stood up and asked me if we had ever considered using his product (some sort of gated discussion forum…whatever) to promote student safety, security, and FERPA in Collaborative Curiosity.

giphy-3

I just kinda looked at him for a second and said “No.  I explained our pedagogical approach.  We needed a public forum to promote dialogue beyond the classroom. The walled garden might be fine for some pedagogical goals, but it was not consistent with our agenda, and it would have been a horrible pedagogical choice.”

Then I firmly shut down his FERPA talk and my friend the moderator shut me down before I really got going on the vendor.  It was probably for the best.

There were some smart people in that room. I’m really hoping that exchange drove home my point. Pedagogy first.  Tools second. Otherwise you do stupid shit like letting tech get in the way of your pedagogy because a vendor told you to.

 

 

 

 

 

Syndicated, Virginia Group

The Problem with the EduSocMedia Lit: A presentation for WCET 2016

Tomorrow, Laura Pasquini, Ron Hannaford, and I will be presenting at the WCET Annual Meeting on methods, practices, and implications of Social Media Research. This came about because Ron, Director of Distance Learning at Biola University, was inspired to coordinate a panel to address the fact that higher education institutions need empirical data to direct and support faculty use of social media in classrooms.  He and Laura Pasquini had already worked together in the context of educational technology; Ron and I shared several heated discussions on the principles and practice of digital pedagogy at Online Learning Consortium’s Innovate Conference in April.

Although Laura, Ron, and I champion online learning, we bring very different perspectives to the table – perspectives that are fueled by our respective professional positions, educational backgrounds, and philosophical approaches. I can’t speak for them here, but I can provide you with some of my own ideas on the subject.

In titling our presentation “Social Media Research: Methods, Practices, & Implications,” I think we are dangerously close to missing the entire point.  We need to step back from a conversation of tools and technology to consider the big picture of the pedagogies we aim to develop and the nature of story we hope to describe. 

Let me back up. Over the summer, I worked with Laura Pasquini and Jessica Knott on a systematic literature review on the educational use of social media in higher education.  Laura P. and Jessica have been working on this for a little over a year, collecting and examining different aspects of the literature; I was brought in this summer specifically to help examine the research methods and study designs found within. It was a massive project, and as I’ve written previously I read over 500 articles, many of them twice.

Based on a preliminary analysis, two types of study dominate the edusocmedia research landscape: the quantitative survey and the action research/single site case study (represented here as a single entity because they almost always consisted of the same thing: descriptive pieces written by a practitioner who had developed and implemented something in their own classroom). Regardless of the study design, a large number of studies focus on either use or perceptions of social media (in general or for learning) by students, faculty, and/or administration.

I find this unfortunate, or, at the very least, unfinished.

In reviewing the edusocmedia at the depth that I did, I found a striking absence of experimental designs. I believe this is the point some of my colleagues would like to drive home and then change. In the U.S., we tend to privilege the wisdom of experiments, or at least well designed quasi-experimental designs, over other forms of research.  Many of my colleagues are looking for more (quasi)experiments involving structured cohorts and systematic approaches to interventions – at the tool level.  What impact does classroom Twitter have on the test score of group A versus group B? How does use of Facebook in that class increase student engagement (via validated instrument) in this group versus that? Does Twitter impact critical thinking more than the Facebook when implemented sequentially or simultaneously?

Or similar.

This is where I tend to veer away from some of my colleagues.  I like (quasi)experimental designs – I really do – and I find the lack of (quasi)experimental designs in the literature problematic.  However, I do not consider them the gold standard, because I believe in a balanced relationship between (quality) experimental and (quality) nonexperimental, (quality) qualitative and (quality) quantitative work.

I believe in a richly woven (and quality) story.  I believe it is a mistake to hold a discussion about the need for more (quasi) experimental designs in edusocmedia literature without situating it in a larger critique of the literature, including the questions we ask, the outcomes we seek, and the quality of the work we produce.

This is why I’m on the panel.

Let’s talk about quality of the literature for a moment. Systematically speaking, the edusocmedia research has a quality problem. To be clear, I don’t think it’s isolated to edusocmedia or even educational technology.  Furthermore, there are some beautiful articles out there. I am speaking at a general level.

Sampling. Surveys are frequently given to convenience sample populations. Qualitative work often involves extremely small sample sizes.  I am trained in qualitative research; I understand and live by terms such as snowballing, saturation, in-depth case studies and similar.  However, a one-time “focus group” of less than five students (without additional data) does not seem like the basis for a publication.

Research questions.  As previously mentioned, we tend to focus on perceptions and current use of social media among students, faculty, and administrators.  These are important questions; their answers lay a foundation for work on access, systems level change, and considerations for pedagogical scaffolding. However, I believe the literature is saturated in these areas and offer a relatively solid picture over the last decade on a fairly international scale (one exception is South America, but I’m unsure whether this is related to a lack of published research or a lack of research published in English). I am not suggesting you should not do your own needs analysis when necessary; I’m suggesting that you do not publish it unless it is remarkably different than what has already been published.

Unfortunately, given the consistently low number of article citations noted via Google Scholar, I wonder if we are reading the literature at any depth prior to publishing our work.  #Edusocmedia literature seems to be the thing that everyone writes but nobody reads…or cites.

Study designs and collection. We tend to fall back on a nonexperimental descriptive design by default. How many designs could be strengthened by the use of a pre-/post intervention survey rather than the use of an end-of-course evaluation? In terms of qualitative research – this is where my heart just plain hurts. Qualitative research requires knowledge and skills, just as quantitative work does.  There are definitions and standards to uphold.  There are indicators of rigor to follow. As a group of scholars, we need to raise our understanding and skill level in qualitative research.

Analysis and Reporting.  Transparency and replicability. Reported research is not actually research unless enough information is provided so that a reader to repeat the process. Methods sections.  Explanations of how data were analyzed – qualitative and quantitative.  This is essential.  As a group of scholars, we need to take our own research seriously. We need to identify our study designs correctly and report our data analysis methods (not just our course designs) with enough detail that others can replicate our work.

The Big Picture

I am concerned about improving the quality of what we do.  However, I am also concerned about building a compelling, cohesive body of literature that is as comprehensive as possible. In other words, we – as a body of scholars –  not only need to have a talk about how we do research, but why we do research.

We need to rethink our research questions, moving beyond use and perception of social media; however, we also need to move beyond social media. 

Social media is a collection of tools.  It is not a pedagogy. To show you what I mean, I’ve pulled a graphic from one of my other posts to serve as an illustration. Worldview. Pedagogy. Practice. Tool. Assessments.  Worldview and pedagogical assumptions drive the use of tools, such as social media.  To research and discuss social media without the broader context of philosophy and pedagogy is – at best – a waste of time and – at worst – harmful to collective knowledge. If we limit our research to the level of the tool (e.g. do students perceive a specific social media tool to be an effective learning tool?), we risk turning that tool into a pedagogical “black box:” We will have failed to analyze what makes that particular tool effective…and why.

picture1

When we situate ourselves entirely in specific forms of social media, we risk irrelevance, particularly if we are publishing in traditional journals that might take a year or more to see the light of publication. 

Tools change quickly – when faculty choose them, they are often doing so at the whim of administration (I’m thinking about learning management systems or clickers in particular) or the social media industry (Twitter, Facebook. They have a history of making significant changes and not once did they consult me about how it might impact my students or classroom).

I applaud the #edusocmedia field for exploring how students perceive the use of Pinterest (for example) in different disciplines and contexts.  However, it is just as important to ask what pedagogical actions Pinterest facilitates (collaborative resource curation…maybe?) and how those actions impact student learning (enhancing students’ ability to critique information sources…maybe?).  If we can elevate the discussion of tools to how they function (e.g. curation, collaboration, hyperlinks, multimodal creativity) rather than what they are, then our work will likely stay more relevant for longer periods of time.