Resources: FAQ for Learning Plan and Replication Project / From previous semesters: Example learning plan Example replication studies / Peer-reviewed example replication studies / Late submission /

1. Plagiarisms test

The first assignment of this course is to pass the plagiarism test and obtain a certificate at the master and doctoral level. Plagiarism is a serious academic misconduct. You will receive zero grade on plagiarized work and there may be other consequences. We have been told not to do this maybe since primary school, and we are always assuming we know what plagiarism is. However, we may assume we know too much (e.g., famous cases of plagiarism).

You do not need to take this test if you have comparable certification, but the validity of your certification needs to be approved by the instructor.

“All assignments in this course may be processed by TurnItIn, a tool that compares submitted material to an archived database of published work to check for potential plagiarism. Other methods may also be used to determine if a paper is the student’s original work. Regardless of the results of any TurnItIn submission, the faculty member will make the final determination as to whether or not a paper has been plagiarized” (Statement from the Faculty Writing Committee: Guidelines for Preventing Plagiarism).

For this assignment, please submit your certificate as a file via Canvas

2. Participation

You are expected to read and annotate the course materials of each week before class (1st week is exempt). The purpose is to have a chance to interact with your classmates, and I will have a better idea of your “pain points” before class. At least have one comment on each article, and respond to at least one comment from another classmate. This regular on-going assignment uses online annotation platform, and is due before the class day.

2.2 In class participation

Your participation is assessed by the instructor and TA based on your performance in class during the lecture, practice, and TA sessions.

2.3 Group project participation

Your participation and responsibilities will be assessed by your group members using this form.

3. Customized learning

You may want to develop your own skill set over the semester. For example, learning how to use Python and R rather than using Stata or Excel. This assignment accommodates such a need. This is a DIY and self-motivated module.

Expected deliverables:

- Learning plan proposal.
- Self-evaluation report of completion.


3.1 Learning plan proposal

You are responsible to develop an actionable and feasible learning plan. There is no required structure of the plan, but it should have the following contents:

1. An analysis of your background, career goals, and needs of learning.
3. Break down your overall goals into specific and actionable items.
4. The plan should cover the major themes of our weekly sessions. Your plan does not need to synchronize with our weekly topics.
5. A feasible timeline.
6. A concrete plan for assessing your performance.
7. Maximum length 2 pages, use the space efficiently and keep your plan concise.

You can share learning plans with others. You can even assemble learning groups. But this is NOT a group assignment, and you will be assessed individually.

You cannot substantially revise your proposal to lower the requirements after it is finalized.

3.2 Completion

You will assess your own performance using the evaluation plan you proposed. Submit your completion report via Canvas. No required format and page number for the report, but it should at least include the following items:

1. Evidence showing your completion of the planned modules/courses (e.g., screenshot, certificate, etc.; changing the page layout & resizing any screenshots and certificates is okay).

A completion report in any file format is fine (e.g., PPT/Excel/Word). Although there is no page limit, considering a report that is simple and concise (e.g., equivalent to a 2-page Word document).

3.3 Resources for learning

• Free licenses for DataCamp and Brilliant are available via Canvas.
• Stata track: http://geocenter.github.io/StataTraining/

4. Research replication

Replication is critical to scientific discovery, however, a substantial mount of studies cannot be replicated, leading to the so-called “reproducibility crisis’’ in both natural and social sciences (Baker, 2016; Hardwicke et al., 2020). The replication projects serve two purposes: 1) get familiar with real-word research practices, which are often messy and unclear; 2) practice the empirical methods we learned in class. The reading materials in the first week are important to guiding the replication exercises.

Treat the group replication project as a serious manuscript to be submitted to an academic journal. Set your target high and also be realistic. A good project needs to be a finished project in the first place.

Expected deliverables:

Maximum 4 students per group.

- A replication plan.
- A presentation of your replication plan in class.
- A replication report.
- A presentation of your replication study.
- A verification report of another group's replication study.


4.1.1 Choosing the right article(s) for replication

You should keep three criteria in mind while searching for the articles for replication (Stojmenovska et al., 2019, p. 305):

1. Interest. You are motivated to replicate the study.
2. Impact. The article should be published in a disciplinary flagship journal and/or highly cited, making sure you are “learning from the best.”
3. Feasibility. Your replication plan should be feasible regarding data access and complexity of analysis.

4.1.2 Replication plan

You can think from five aspects while writing your replication plan: “(1) the premise of the paper, (2) the data and empirical design, (3) the key results, (4) why [you] think the paper would be a good candidate for the replication course, and (5) potential problems that might arise in the replication” (Stojmenovska et al., 2019, p. 306).

A substantial part of the replication plan will go to your final replication report. The contents of the plan may vary, but they should include (but not limited to):

1. Significance of replicating the article(s) and your novel contribution.
2. Analysis of the original article from: a) theorization, b) research design (e.g., hypothesis and operationalization), c) analysis methods, d) results and conclusions, e) data management and documentation (data management checklist).
3. Replication plan: How are you going to replicate and extend the original study? You can develop the plan using the structure for analyzing the original article.
• The following components are required in your replication plan: (a) descriptive statistics, (b) hypothesis testing, (c) regression analysis.
• The following methods are optional in replication: qualitative interviews and coding, additional surveys, computational social science methods.
4. Expected results in comparison to original article.
5. Knowledge necessary for replicating the article, and how the knowledge is connected to your learning plan (explain by group member). You may not understand all the analysis methods used in the original article, set a plan to learn the new skills.
6. Effective teamwork and project management plan. Not all of the members need to be good at running an advanced statistical analysis or designing a rigorous empirical study, but you need to identify your strength and know how to collaborate with and complement each other. This is a more realistic scenario in the future regardless of the career you hope to pursue.

Prepare a 10-minute presentation of your replication plan. In the presentation, I expect you will discuss, but are not limited to, the following items:

1. Overview of the original study.
2. Where you will spend most of your time and why.
3. The design of your replication.
4. Expected results.
5. How it ties to your learning plan.
6. Teamwork and division of labor.

Your presentation will be assessed from 1) significance, 2) novel contribution, 3) analysis of the original article, 4) feasibility of replication, 5) teamwork.

Prepare a 15-minute presentation of your replication, which should emphasize:

1. Overview of the original study and its significance.
2. Analysis of your replication. (a) The design of your replication, what are the differences in comparison to the original study, and why you are making these changes. (b) The results of your replication, the differences in comparison to the original study, and your interpretations about the differences. (c) Your data management and documentation, and how they support the replication and transparency of your project.
3. Teamwork and division of labor.
4. Your intellectual growth through the customized learning and replication project.

The class audience will use this form to evaluate your presentation, which will be used in grading.

4.3 Verify replications

Your replication report itself needs to be replicable. Each group will be assigned to verify another group’s replication and provide a verification report. Ideally, your group should be able to replicate another group’s project with the information provided in the replication report. The verification report should point out, for example:

• Whether you can or cannot run through the replication study, and why. You don’t have be crystal clear about the other group’s analysis, but there should be no obstacles for running the analysis through.
• Whether the theorization and operationalization are sound and clear.
• Whether the analysis is robust.
• Whether the findings and conclusions are valid and robust.
• How the replication study can be improved.

The purpose of the verification report is to provide concrete feedback on how to improve the replication report. So be constructive, insightful, and concise. No page limit but 2 pages are expected if excluding tables and figures. All groups will revise and finalize their replication reports using the feedback from the verification reports.

4.3 Final replication report

Use the published replication studies that are close to your areas of interest as references.

Generally, your manuscript should include two parts: a main text published as a regular printed journal article, and an appendix usually published online.

The main text summarizes, for example:

a. Theories and significance of the research topic/question.
b. How original study(s) responded to the research gaps and challenges.
d. The research design of your replication, and how it ties to your contribution.
e. Research findings, comparison to original findings, and interpretations.
f. Discuss the implications and limitations of your replication, and the directions for future studies (either replication or original study).


The appendix reports all technical details that are necessary for replicating and verifying your work. Please also include a Contributor Roles Taxonomy (e.g., page 3 of this document) and any other documents that you want to present to us.

There is no required format for the report, but a few suggestions on formatting and submission:

- Include page numbers, all figures and tables need a title (see other studies as examples).
- Do not use excessive decimal positions when presenting numbers (up to 3 valid decimal positions is more than enough, e.g., 0.000123).
- Tables and figures should be mentioned in the context of the report (A figure/table does not explain by itself, you guide the reader to understand/read/interpret tables and figures).
- Organize all the files in one single directory with full documentation (e.g., "readme" files and annotations in code scripts). You are expected to submit that single directory as a compressed file (e.g., .zip file) via Canvas.


4.5 Replication resources

The replication movement has been progressing very well in psychology, but you should be able to find replication studies in policy and public administration related areas. Here are some resources for your reference:

5 Late submission

All work is due as indicated on the course schedule. Late submission will be taken a 30% off the total possible points as late penalty. For example, for an assignment with 100 total possible points, if you submit late and graded 80, your final grade for this assignment will be 50 (= 80 − 100 ∗ 30%). Please send me a notice if you have an emergency (neither travel arrangements nor computer problems are emergencies). The last week of class is the final makeup date for all late assignments.

References

1. Baker, M. (2016). 1,500 Scientists Lift the Lid on Reproducibility. Nature News, 533(7604), 452. https://doi.org/10.1038/533452a
2. Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020). An Empirical Assessment of Transparency and Reproducibility-Related Research Practices in the Social Sciences (2014–2017). Royal Society Open Science, 7(2), 190806. https://doi.org/10.1098/rsos.190806
3. Stojmenovska, D., Bol, T., & Leopold, T. (2019). Teaching Replication to Graduate Students. Teaching Sociology, 47(4), 303–313. https://doi.org/10.1177/0092055X19867996