Link Search Menu Expand Document
CS190J (S21)

ULAs in CS: Research Paper Analysis

Table of contents

  1. Assignment
    1. Part 1: Main Paper by Mirza et al. (2019)
    2. Part 2: Analyze a Paper from Citations (paper1)
      1. Select a practice or a benefit
      2. Select and claim a citation (paper1)
      3. Write an essay about your selected citation paper (paper1)
    3. Part 3: Select a new paper (paper2) that cited the work that you analyzed (paper1)
      1. Find which papers cited a paper
      2. Augment your analysis
  2. Writing guidelines
  3. Submission instructions

Assignment

For this assignment, we ask that you

  1. read the paper by Mirza et al. (2019),
  2. analyze a paper from the list of citations in the work by Mirza et al. (2019),
  3. compare it to another paper that cited the work that you selected.

You are allowed to work alone, in pairs, or in groups of up to 3 students. When you get to Part 2, you might find other students who are interested in the same paper that you chose, so you might contact them and work together on the analysis and references. In your submission, clearly identify the names of all contributors.

We recommend looking through the Table of Contents and reading these instructions top to bottom to get an idea of the steps that you need to take for this assignment.

The goals of this assignment are:

  • Identify the main takeways of research papers
  • Critically evaluate the methodology of a research paper
  • Reflect on our current ULA program
  • Practice writing in LaTeX (optional but highly recommended)

Part 1: Main Paper by Mirza et al. (2019)

For this assignment, we ask that you read the following paper:

Diba Mirza, Phillip T. Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate Teaching Assistants in Computer Science: A Systematic Literature Review. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER ‘19). Association for Computing Machinery, New York, NY, USA, 31–40. https://doi.org/10.1145/3291279.3339422

Here’s a PDF link.

There are some sections of the paper that you should read carefully, while others you can just skim. Here’s a guide:

  • Section 1: Read carefully. Understand the purpose of the paper, and the three main research questions addressed.
  • Sections 2 and 3: Ok to Skim. These sections address the methodology of the paper, i.e how one goes about doing a structure literature review. This is important from a research standpoint, and to assess the validity of the results, but for purposes of this class, we are more interested in the results themselves.
  • Section 4: Read Carefully. This is the heart of the paper, and the section that we’ll be focusing on.
  • Section 5: Ok to Skim. Another section about methodology. A “threats to validity” section is an important part of any paper, you are encouraged to read it, but it isn’t necessary for the assignment.
  • Sections 6: Read Carefully. This points towards future work, and is part of the assignment.

The research questions in the paper deal with “practices” and “benefits”. In class, we will be discussing

  • What is meant by “practices”? Be prepared to discuss a few examples to illustrate the answer.
  • What is meant by “benefits”? Be prepared to discuss a few examples to illustrate the answer.

Additionally, be prepared to discuss what the paper found about a practice or a benefit from among all of those that were cited in the paper:

  • Tables 5,6,7,8 and 9 deal with practices. They list questions about practices in various areas, and then provide a list of different choices for each of those questions.
  • Tables 10,11,12,13 deal with benefits.

Part 2: Analyze a Paper from Citations (paper1)

For this part of the assignment, we ask that you select one of the papers from the list of citations in the work by Mirza et al. (2019).

You may need to go to the following sites to get the URL:

To get access without having to pay for articles, use the UCSB VPN:

Select a practice or a benefit

Choose a practice or a benefit from among all those in the tables 5-13. In your essay, briefly write what the main paper found about this practice or benefit.

Select and claim a citation (paper1)

Sign up to analyze a paper on this spreadsheet: for the citation number (from the paper by Mirza et al. (2019)), add the title and the year of the paper that you claimed. Once you find the paper online, include its URL as well.

Write an essay about your selected citation paper (paper1)

Read this paper (paper1) and update your essay with your analysis.

In your essay discuss the answers to the following questions:

  • What is the research question that the paper is investigating?
  • What are the key findings of the paper?
  • How did the researchers measure these outcomes? (If they didn’t adequately measure, and only described in general terms their anecdotal experiences, note this.)
  • How do you think we can apply this research to the course you are tutoring, or more generally, in our undergraduate program at UCSB?

Part 3: Select a new paper (paper2) that cited the work that you analyzed (paper1)

In this part of the assignment, you are basically trying to figure out whether there were other works that also cited the paper that you analyzed (paper1). Once you find it and select another paper that cited paper1, you can compare its context to paper1 and see potentially another angle in which someone cited this work.

To find another paper (i.e., paper2), you need to get a list of papers that cited your selected paper (i.e., paper1) and choose from that list a new paper that you are interested in analyzing.

Find which papers cited a paper

There are multiple ways to find which papers cited a specific paper. Here are two alternatives:

  • If you search for the title of the paper on https://scholar.google.com, then underneath the excerpt from the abstract you’ll see the two quotes with “Cited by” and the count next to it. If you click on it, you’ll be redirected to the page that lists the papers that cited your paper (that’s now shown as a link at the top of the page).
  • (Alternatively) If you follow the doi link (e.g., like the one that we included above for the main paper), in the paper citation, you’ll see two blue quotation marks with a number next to them: that number shows how many papers cited this paper. If you click on the “Bibliometrics and Citations” button on that page (the squiggly up arrow, on the vertical menu to the right of the Abstract), the “Citations” tab will tell you which papers cited it.

Augment your analysis

In your essay, apply the questions from the analysis of paper1 to your paper2.

Additionally, explain in what context was paper1 cited in paper2: How similar are their research questions? Where and how do they differ?

Writing guidelines

This is a writing assignment, so we emphasize on producing a coherent and well structured piece. Below are some guidelines on what I am looking for in your essay:

  • Clearly identify the main findings
    • There is a lot of information in a research paper. In conveying your finding you will need to distill the main takeaways of the paper.
    • Figure out what is new and interesting about the work.
  • Describe the findings and methodology at the right level of abstraction
    • We have not specified “how many pages” or “how many words”, because we don’t want you to aim for that.
    • Write just enough to answer the questions below, and no more. If that’s 1 page, fine. If it’s three pages fine. Make it as short as possible, and no shorter. Less than one page is probably too short and up to five pages.
  • Make clear arguments
    • Connect the research finding with an actual practice that we are doing in our program
    • Provide supporting data from the paper when critiquing results
  • Be original
    • Organize your essay in whatever way you like (except not by simply repeating the questions and answering each one in a paragraph; make it a coherent essay.)
    • Don’t just copy/paste from the paper or the abstract. Where you use exact wording, be sure to use “quotation marks”, but do this sparingly, only where using the exact wording of the original source is really necessary to convey the idea.
    • Summarize and critically reflect on the results of the paper.
    • Include personal accounts and reflection where appropriate.
  • Make it engaging
    • As a tutor you have the opportunity to really engage students. Doing so requires a willingness on your part to be engaging. Try to bring it into your writing.

Submission instructions

  • Type out your essay.

  • Make sure that you properly cite both papers within your essay (as shown in the template).

  • Insert your paper1 and paper2 as an appendix at the end of your essay.
    • Upload the PDF of papers to your Overleaf account
    • Use the pdfpages package: At the top of the LaTex file (after the \documentclass) add \usepackage{pdfpages}
    • To include all the pages in the PDF file: \includepdf[pages=-]{myfile.pdf} replacing myfile.pdf with the name of the PDF that you uploaded (using the “upload” icon at the top of the list of the files on the left pane).

  • Save your essay as a PDF (if using Overleaf, there’s a “Download PDF” icon on the top-left corner of the document preview pane).

  • Upload your file to Gauchospace to the “ULAs in CS” assignment before Friday, June 4 10am (Pacific time).