How To: Move from Exploring to Investigating

As you hopefully know by now, this class is heavily driven by a project you yourself will propose about a third of the way through the semester. Some of you have likely entered the class already knowing exactly what you want to work on. Others of you may have some general interests, but you need to refine them down to a specific idea. Still others of you are taking this to fill specialization credit and have no initial ideas at all.

This guide covers, at a high level, how to move from wherever you start to landing on a specific problem.

What is Educational Technology?

But first, it’s important for us to define in advance what we mean by “educational technology”. A narrow, but common and entirely reasonable, view of educational technology is that educational technology is the use of technology in the act of teaching. This includes technology used in typical classrooms like Smartboards, online technology for teaching like simulations or MOOCs, artificial intelligence to give feedback like intelligent tutoring systems, virtual reality systems for creating immersive learning environments, and so on. Most projects in this class will likely come in this area: technology used in the act of teaching and learning.

In this class, however, we also adopt a slightly broader view of educational technology: for us, educational technology can also be technology in support of teaching and learning, even if it is not directly involved in the act of teaching and learning. Take, for example, academic advising: academic advising does not itself constitute a teaching or learning activity, but it serves a critical function that must be filled to allow teaching and learning to take place. We can see the relevance of this in the literature: the last two years of ACM’s Learning @ Scale conference, for instance, have included papers on course planning and TA hiring.

So, for us, educational technology is technology used at any place to support the overall endeavor of education. Academic advising, plagiarism detection, student communities, and other ideas are within our scope for this reason.

One key to note though: this class is about human learning. As we’ve taught this class more and more, we have encountered many students who strictly think of learning in the machine learning sense. You’re welcome, of course, to tackle a project of applying machine learning to education, but you must be tackling human learning in some way.

So where do you start?

If you’re entering the class with absolutely no ideas on what to work on, that’s ok. You still have some places to start.

First, you’re a student in an online Master of Science in Computer Science program. Unless this is your first semester, you’ve already taken some online classes. Even if this is your first semester, you’ve got four years of Bachelor’s and 13 years of grade school to draw from. What are some places that technology could have improved your experience?

Note that you’re unlikely to think of anything completely original: the world is a big place with lots of people, and every good idea has been pursued in some way. But at the same time, every past effort opens up new questions and new room for improvement. Your goal here isn’t to grab one idea and run with solving it: your goal here is to settle on ideas you want to pursue, and start exploring what has already been done in that space.

This is also why we have supplied the Course Library. You can instead start with the work that has already been done and get some ideas for where you want to go. Sometimes the greatest inspiration comes from seeing people working on a problem that you didn’t know could be solved, or even more, that you didn’t even know existed.

And if you’re still having trouble coming up with something to pursue, try talking to others! Your classmates, your friends, your family, other people online,  and more will all have ideas about problems they feel need to be solved. Education is universal to all of us. They might bring up the biases experienced by women and ESL students in the online education marketplace. They might reference the isolation online students may experience, and the need to build peripheral indicators of community online. They might reference modern difficulties teachers experience with addressing new and clever methods for unauthorized collaboration. They might reference specific content they find difficult to teach, like financial literacy or growth mindsets or complex systems. They might mention none of these things, but now I have, so you’ve got somewhere to start.

Ok, I’ve got some initial ideas. What next?

Your next goal, as mentioned above, is to find out what others have done in the area. The biggest mistake people often make in this area is to jump straight to trying to solve a problem without understanding what others have done in the area. This means you risk making the same mistakes others have made, or risk creating a solution that isn’t in any way differentiated from other options out there. This applies whether you care more about academia or more about industry. In academia, a key part of research is putting your findings and ideas in the context of others’ to move the field as a whole forward. In industry, to succeed you need to understand why others have failed. If you take any entrepreneurship course, you’ll receive the advice that if you tell an investor that you have no competitors, then they’ll know you haven’t done your research. Every problem worth solving has people trying to solve it: the question is, how are you different?

“Different”, however, doesn’t necessarily mean they failed where you want to succeed, though. Different could also include taking a solution that worked in one domain and transferring it to another. It could include taking a solution that worked for a certain population and finding how to make it work for a different one. It could include taking an idea that has some supporting evidence and testing it further to strengthen the case.

Your goal, in a nutshell, is to ensure that what project you land on can contribute to the world. To do that, you need to understand what the world needs and what others have provided. That’s where a separate guide comes in: how to find papers to read. Papers aren’t everything, of course, and it’s okay to refer to other companies, apps, etc. in some places, but you should focus on the literature: academic literature holds as its motivations transparency, comprehensiveness, and peer review. You’ll find details of implementation, rigorous peer-reviewed evaluation, and more, while industry typically presents its ideas with more bias and opacity.

The Cycle of Inquiry

This process proceeds in a cycle. It’s a cycle that isn’t dissimilar from the scientific method itself. You start with an idea from somewhere, whether it come from your own prior motivations, conversations with others, the course library, or somewhere else. You investigate it and find work that others have done. Finding that work brings up more questions: what are the current weaknesses? How does this apply to different domains? What do we still not know? What can we still not do?

Those questions then feed further investigation: you continue to search, explore, and find what still others have done. Ideally, you cross domains: you find that there are problems common in both medicine and education (such as teaching simulations) or in education and the workplace (such team dynamics). You keep reading and keep coming up with new questions. As you go, your questions become narrower and more specific, until eventually, you come up with a question to which you can’t seem to find a satisfactory answer. That’s when you know you’re close to a problem you might want to investigate yourself.

Of course, there’s very often a challenge that arises: many of the problems you want to investigate may exist in a domain with a lot of infrastructure and existing software. Take, for example, ASSISTments, a platform developed at Worcester Polytechnic Institute with fantastic results in improving math learning. You might ask: how can reinforcement schedules be used in ASSISTments to further improve student learning? That’s a difficult question to answer, however, without being able to actually modify the platform itself. In those cases, your question becomes: what can you build or do to investigate that question? Could you, for example, build a very lightweight approximation of the system targeting only one problem type, and use that to investigate reinforcement schedules?

Research, Development, and Content Tracks

This class offers three tracks for your final project: research, development, and content. The research track involves investigating some phenomenon; the development track involves building something to solve a problem; and the content track involves teaching some material using technology. These are not mutually exclusive, of course: many research projects require some development, developing content may involve researching its effectiveness, and developing an educational tool often involves developing a curriculum to use with the tool. These are instead organizing structures to help you understand what the primary emphasis of your work will be.

As you explore your ideas, you’ll likely start to see these things starting to come up. For example, this paper focuses on building a model of instructional design in online courses. This one explores understanding how professors structure their exams. Something exists in the world, and the paper aims to understand it better; these are research papers (in our parlance). This paper, on the other hand, build a tool to make it easier to teach code online. This paper constructs a tool to randomize asynchronous exams to avoid answer-sharing. These are more development papers: there existed a problem, and they build a tool to solve it. Of course, part of solving it is researching the effectiveness of the solution, but in this class you’ll likely find that in a single semester, you can only build the tool; testing would have to come later. There are content papers, too, like this one: it also includes evaluation, but the focus is on creating content.

Again, the lines here are blurred. Any solution or content could be evaluated, which would bring up more of a research angle. The development track and content track can have significant overlap as well: in the paper referenced above, a significant focus is on developing content according to a specific methodology, including short videos, AI assessment, and frequent evaluation. Is that a development project that includes a curriculum, or is that a content project that includes some development? The lines are blurred, and that’s ok; these tracks are meant to help you structure your exploration, but they shouldn’t limit you.

From Exploring to Digging: An Example

I’ve been speaking in very broad, general terms throughout this because the scope of things you might choose to investigate is so large. However, to give you something more concrete to go on, let me walk you through an example from my own dissertation work.

First, to start with the inspiration, the project I worked on existed before I ever started my PhD. What inspired those initial investigators? They were inspired by the observation that scientists working in real world investigations use models to make sense of the world. Students in science classes in grade school, however, are very often taught science as a collection of disjoint formulas, definitions, and facts. This carries two weaknesses: one, the understanding that students come away with is structurally different from the kinds of understandings real world scientists use, and two, it ignores the investigative quality of science in favor of teaching only established knowledge. Those investigators wanted to address this problem by teaching students to investigate systems and construct understandings that mirrored the kinds that real-world scientists would use. To do that, they constructed a software environment called ACT (Aquarium Construction Toolkit) that would allow students to build scientific models of the complex systems inside an aquarium. They deployed that in a middle school in New Jersey with an accompanying curriculum. You can read about it here.

That was when I entered the project, and what I found particularly interesting was the fact that while students very quickly took to constructing models of scientific phenomena, they did not naturally demonstrate the tendency to test their models against new evidence. They seemed to like to take one initial idea and run with it, regardless of the new information they observed. that became the problem that I wanted to address: not just teaching them to think in terms of scientific models, but to use those models as a way to investigate the world, and then to use what they learned in the world to inform revisions to their models.

But to do that, I needed to understand several things:

• How do scientists do scientific inquiry?
• How do scientists defend their models with evidence (which I found here)?
• How do you teach a skill like “scientific inquiry”? It’s not a skill like math, but more metacognitive.
• How do you teach skills like that scalably?

There were several other questions, of course, but these formed the brunt of things. I came from an AI perspective, so I knew I wanted AI-style solutions, and so I was led to metacognitive tutoring, a subfield of intelligent tutoring systems (itself a subfield of AI in education). Exploring these questions allowed me to find what had been done in the area. It also allowed me to see that I’d found a sufficient niche: people had tried to teach model-based reasoning, scientific inquiry, and metacognition with AI, but no one had yet used AI to teach scientific modeling and inquiry.

My dissertation then went on to become “Metacognitive Tutoring for Inquiry-Driven Modeling”. Notably, I wasn’t the only one working in the space: there were other attempts to use AI to teach inquiry, other approaches to teach inquiry in exploratory learning environments, and other approaches to teaching students to construct and revise models. However, no one had tackled things quite the way I did: I had enough similar ideas to draw from, but I wasn’t reinventing something someone else had already done.

You may not find something with so little overlap: you may need to reinvent a bit to get to something useful. You may find that the majority of what you do is walk through reimplementing something someone else has already done, but your reason for doing it will be to address one particular niche, question, or need that wasn’t addressed. Your result doesn’t need to be better than everything out there, but it needs to contribute to our general collective knowledgebase and toolkit.