Assignment 1 (Summer 2021)

Assignment 1 has two parts. The first, the Research Log, is a structured opportunity for you to report to your mentor and classmates the progress you’ve made this week in exploring the literature and refining your idea. Each assignment until the Qualifier Question, you’ll submit a new Research Log documenting your research progress since the last week. The second, the Activity, is a more structured opportunity to practice one of the skills you’ll need as you move forward in the class. Each part is worth 50% of this assignment’s grade.

Research Log

First, if you haven’t already, you probably want to peek forward at the Qualifier Question and the Project Proposal assignments. These first several assignments build toward those, so it would be useful to understand where you’re trying to get!

The goal of these first several assignments is have a general project idea in time to receive a Qualifier Question. However, different students will enter the class at different places in that pursuit. Some of you entered the class already knowing exactly what you want to work on. Others of you need some time to explore. In either case, though, there is research to be done: if you’re in the former group, your research is to find what others have done in your area so you know how to contribute. If you’re in the latter group, your research is to find what you might be interested in doing and how you can contribute to the community.

The Research Guide is there to help you explore the literature no matter what ideas you have as you enter the class. In each of the next three weeks, you’re going to complete a Research Log that documents your growing understanding of the literature. The goal of this research log is to give you an ordered, formal structure for organizing your growing understanding of the research domain you want to enter. Each of the three research logs will follow the same structure, intended to capture your progress regardless of where in the process you are—both entering the class and as the class progress.

Your Research Log should contain 5 sections:

  • Background: In about half a page, summarize your current state. For this Research Log, this is likely a summary of your prior interests. If you already know what you are interested in doing, you’d write about that; if you don’t, you’d write about your more general interests in Educational Technology.
  • Papers: As you walk through the Research Guide, you’ll be finding lots of papers to read. Here, you’ll make a list of the papers you come across and give considerable attention to. We would expect the Research Log to include at least 15-20 sources (though more is fine as well), and at least 12 (preferably more) should be academic and peer-reviewed. You may include blog posts, newspaper articles, etc. as well, but you should have at least 12 academic sources, too. In the list, for each source, you’ll provide:
    • The paper’s bibliographic information (its APA citation, typically)
    • In around one sentence, how you found it (a Google Scholar search? From a conference’s proceedings? From another paper’s references? Something else?)
    • In around three sentences, a brief, original summary in your own words
    • In around three sentences, the main takeaways going forward

    You should never copy text directly from the source (including its abstract) unless you are quoting it with quotation marks and in-line citation; see How To: Avoid Plagiarism for more. Your summaries should be in your own words; if you want to quote the source directly, make sure to follow APA guidelines for in-line quotes and citations.

  • Synthesis: In about a page, summarize the overall body of work you’ve put together. What are the high-level trends, large takeaways, or open questions you’ve found? If you’ve narrowed in on a particular domain, summarize that domain; if you’re still exploring, discuss the overall direction these efforts are leading you toward. Most importantly, anchor this synthesis in the papers you provided above, citing them where appropriate.
  • Reflection: In about half a page, reflect on the process of finding sources, reading papers, synthesizing their contents, and building your understand. What was difficult, and what was easy? What are you finding yourself interested in going forward?
  • Planning: In about half a page, provide a plan for what you expect to do next week. What threads or ideas will you pursue? What questions will you seek answers to in the literature?

The goal of this Research Log is three-fold: to structure (in conjunction with the Research Guide) your exploration of the literature, to report to your mentor your progress in an externalizable and organized way, and to provide enough information for feedback from your mentor and classmates. The process of building your understanding of the literature is a personal journey that is difficult to assess, but we feel confident that if you can externalize the outcomes above, you’re taking steps in the right direction. We will expect your Research Log to show that you’re following the advice prescribed by the Research Guide.

Activity

In a couple weeks, you’re going to be asked to propose your project for the semester. For some of you, that’s going to be building something that solves a problem (Development Track). For others, it’s going to be researching a phenomenon (Research Track), to identify it (“Are cat-owners more likely to succeed in OMSCS than dog-owners?”) and/or to explain it (“Why are cat-owners more likely to succeed in OMSCS than dog-owners?”). For others, you’re going to be teaching some content (Content Track), which in its own way is building something that solves a problem.

No matter what you’re doing, though, there are some commonalities. There will be a need you’re addressing, whether the need is a problem to be solved, a topic to be taught, or a phenomenon to be explained. There will be some audience, either to learn your material, to use your tool, or to participate (actively or passively) in your investigation. There will be some method you use to try to address that need, whether it be implementing some software, conducting some study, or developing some content. There will be a result: you might not reach the point of evaluating your result this semester, but you should be developing something that could be evaluated because that’s simply good design.

Mapping these together, however, can be a difficult task. One of the major things new researchers must learn is how to ensure a method is actually addressing their need and audience, as well as how to ensure an evaluation actually identifies success or failure. These are not trivial concepts.

One of the best ways to learn these things is to take CS6750, but either you’ve already done that or it’s probably too late to plan to do that before you move forward. The next best way, though, is to look at how others have addressed this.

So, for this week’s activity: select five papers. These can be papers you’ve reviewed as part of your Research Logs, or they can be papers you seek out solely for this activity. Note that not all papers will be compatible with this activity: you need to select papers where the authors were clearly solving a problem or investigating a phenomenon. Don’t worry, this still makes up a large number of papers.

Then, for each paper, introduce it with its full APA citation information and include a link to where the paper can be found. Then, create a logic chart/Guzdial Chart/Blumenfeld Chart. These charts are ways of mapping need, audience, method, and results in order to ensure they connect correctly. For this activity, yours will be a little more general than that link: that link is specifically for research projects, while we’ll include development projects as well.

Your chart should have four “columns”, each with 2-3 sentences answering the following questions. You do not have to actually make them columns, you can do this as subsequent subsections, bullets, or some other format, as long as all these parts are here and answered in sufficient depth:

  • What is the need? What problem is trying to be solved or phenomenon trying to be investigated?
  • What is method? What did they build to solve the problem? What did they do to investigate the phenomenon?
  • Who is the audience? Who are the participants or subjects?
  • What were the results? What did they find?

Then, for each paper, critique the alignment between need, audience, method, and results. Are the results justified by the method? Does the method address the need? Is the audience fitting for the need? Overall, how strong is the alignment between need, audience, method, and results?

For example, here would be what you might write about one of my papers, “Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate”:

Paper

Joyner, D. A. (2018). Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. London, United Kingdom. ACM Press.

Need

There is a need for scalable computer science education that preserves the rigor and reputability of an on-campus program. Modern MOOC platforms make content available, but they are often weaker on assessment, so the credential they generate is not as good as college credit.

Method

The authors developed an online CS1 course that is used as the basis for a for-credit class at Georgia Tech, as well as offered in MOOC form. To investigate its success, the authors had students in the for-credit online section complete a set of surveys and assessments that would allow them to be compared to the for-credit traditional section.

Audience

The audience for the MOOC overall is anyone interested in learning CS, but the audience for this study are undergraduates at Georgia Tech. Some of these undergraduates are in this online section, while some are in a traditional on-campus section.

Results

The authors found that students in the online for-credit section achieved comparable learning outcomes with students in the on-campus for-credit section, and appeared to enjoy the experience more, rating the course more highly. They also reported spending less time per week on the course.

Critique

The study gives pretty compelling evidence that the online for-credit students learn as much as the on-campus for-credit students. There are possible issues with the pre-test and post-test because it is difficult to know how much effort students invest in them, but that should be consistent across both sections, so it doesn’t bias things one way or the other. However, the method and results fall short of answering the core goal. Although this shows that online for-credit students learn as much as on-campus for-credit students, it does not demonstrate that MOOC students learn just as much as well. Without investigating the MOOC students’ actual learning outcomes, the authors can’t claim the MOOC students learn just as much (which they do not actually claim, but simply leave as a possibility).

You may find that some papers are a bit difficult to fit into this structure. For example, the paper used in the example above could be thought of in two ways: as a Development/Content study, where there was a need for scalable CS education and a MOOC was chosen as the method to solve that problem; or as a Research study, where there exists a scalable CS course and there is a need to demonstrate its equivalence with a traditional course through the method of shared assessments. That’s okay: there are no objective answers here. This is an organizing structure to allow you to structure your thinking and investigate your selected papers in depth.

The length of your deliverable will depend in part on your formatting, although we would generally expect 3-4 pages, a bit less than one page to answer these questions for each of the 5 papers.

Submission Instructions

Complete your assignment using JDF, then save your submission as a PDF. Assignments should be submitted to the corresponding assignment submission page in Canvas. You should submit a single PDF for this assignment. This PDF will be ported over to Peer Feedback for peer review by your classmates. If your assignment involves things (like videos, working software prototypes, etc.) that cannot be provided in PDF, you should provide them separately (through OneDrive, Google Drive, Dropbox, etc.) and submit a PDF that links to or otherwise describes how to access that material.

This is an individual assignment. Even if you already plan to work on a team for the project, this assignment should still be completed individually.

Late work is not accepted without advanced agreement except in cases of medical or family emergencies. In the case of such an emergency, please contact the Dean of Students.

Grading Information

As with all assignments in this class, this assignment will be graded on an 11-point scale (0 to 10), in accordance with the grading policy outlined in the syllabus. If your deliverable receives below a 9, you may revise and resubmit it once within one week of receiving a grade. Resubmissions may receive up to a 9. Note that this should not be treated as a de facto free pass to submit sorely lacking work initially; we reserve the right to deny resubmission or grade a resubmission more harshly if we perceive the original submission was lacking in earnest effort.

Peer Review

After submission, your assignment will be ported to Peer Feedback for review by your mentor and classmates. Grading is not the primary function of this peer review process; the primary function is simply to give you the opportunity to read and comment on your classmates’ ideas. All grades will come from the mentors alone.

You will typically be assigned four classmates to review. You receive 1.5 participation points for completing a peer review by the end of the day Thursday; 1.0 for completing a peer review by the end of the day Sunday; and 0.5 for completing it after Sunday but before the end of the semester. For more details, see the participation policy.