Experiment Design in Computer Science (01CH740)

Lecture notes for the "Experiment Design in Computer Sciences" class at Tsukuba University

Experiment Design in Computer Science (01CH740)

2020 Version

Main Instructor

  • Name: Claus Aranha
  • e-mail: caranha (at) cs.tsukuba.ac.jp
  • URL: caranha.github.io/ExperimentDesignCS
  • Office Hours: by appointment through manaba

Teaching Assistant

  • Name: Yifan He
  • e-mail: Please contact through manaba

Course Information:

  • Type: Basic Course (基本科目)
  • Style: Lectures
  • Term: Spring AB, Friday 5,6
  • Room: 3B301
  • Keywords: Experiment Design, Statistical Testing, Statistics, Philosophy of Science, Ethics, R Programming Language

Prerequesites

No particular requirements for registration.

This Course expect students to be familiar with basic statistical concepts (ex: Mean, Median, Standard Deviation, Random Variables, Distributions, etc). The code examples in the course will use the R language, so familiarity with this language is helpful. Some helper materials with these concepts will be available, but teaching basic statistics and R is not one of the goals of the course.

Degree Program Competences:

Knowledge Utilization Skills, Management Skills, Teamwork Skills, International Skills, Research Skills, Ethics

Outline

The collection and analysis of data through experiments is one of the cornerstones of the scientific method. In this course, we study the general philosophy and methods behind experimentalism: Why do we perform experiments, what is a good/rigorous experiment, how to plan and design a rigorous experiment, and how to perform statistical analysis on experimental data.

This course is centered around lectures with plenty of examples and study cases. The students will be invited to apply the techniques studied in this lecture to experiment of their own design.

Course plan

  • Topic 01 – Course Introduction, What is Experimentation
  • Topic 02 – Point and Interval Indicators
  • Topic 03 – Inference Testing I
  • Review – Class Review and Discussion of Report I
  • Topic 04 – Inference Testing II
  • Topic 05 – Inference Testing III
  • Topic 06 – Sample Size and Experiment Power
  • Topic 07 – Block and Factorial Designs I
  • Topic 08 – Block and Factorial Designs II
  • Review – Class Review and Consultation about Report II
  • Final Exam

Textbooks

No designated textbook: The course will be based on lecture notes to be distributed electronically on manaba. Some of the main references are:

  • D.C. Montgomery, “Design and Analysis of Experiments”, Wiley, 2005,
  • Felipe Campelo “Lecture Notes on Design and Analysis of Experiments”, Online, 2018,
  • Peter Hoff, “Lecture Notes on Design of Experiments”, Online
  • Other recommended readings on available on manaba

Evaluation

The grade is based on two reports and a final examination.

Each report is a case study, where the student will have to: Design an experiment to answer a scientific question, obtain experimental data (usually by performing an experiment, sometimes from the instructor), analyze the experimental data using the tools studied in the lecture, and prepare a report discussing the experimental conclusions.

Please see manaba for more information about the reports.

The report is evaluated on the quality of the experiment design, the correctness of the statistical analysis, and the quality of the discussion of the results (note that positive or negative results do not factor in the grade of the report).

  • The first report is worth 20% of the grade;
  • The second report is worth 40% of the grade;
  • The final exam is worth 40% of the grade;

Notes:

Lecture in English