Chapter 5: Carry out science laboratory research

5.1 Scientific study design

A study design is the plan that links a research question to the data that will answer it and to the analysis that will be run on those data. It is fixed before the first sample is taken, because a weak design cannot be repaired by clever statistics afterwards. Example: a research laboratory wants to know whether a new vial cap reduces the loss of solvent by evaporation. If the technician tests the old cap on a cool Monday and the new cap on a hot Friday, the change in room temperature may explain the difference just as well as the cap. A design that tests both caps side by side, on the same days and in the same rack, removes that doubt.

Fitting the design to the question

The main designs, experimental, quasi-experimental, descriptive, correlational, case study, cross-sectional and longitudinal, are introduced in the topic on Introduction to research methods. Working out the design of a particular study starts from the question and follows a short sequence of decisions.

• Must the study show cause? Then the design is experimental: the researcher assigns the treatments to units at random and measures the response. Where random assignment is impossible, as when a cleaning schedule can be changed only for a whole production line, the design is quasi-experimental and its weakness is stated.
• Are two groups that already exist to be compared? Then the design is comparative and observational, such as the calcium content of tap water from two supply zones. The researcher does not control which unit belongs to which group, so a difference found may be due to something other than the factor under study.
• Do two methods have to be shown to agree? Then the design is a method comparison: the same samples are measured by both methods, over the range of values the method will meet, and the differences are analysed. Comparing the means of two separate sets of samples would not answer the question.
• Is a level needed now? Then the design is cross-sectional: many units measured once, for example the alkalinity of water from 40 storage tanks sampled in one week.
• Is a change over time the subject? Then the design is longitudinal, with the same sites sampled at fixed intervals, for example the sulphate in the water at one tap point sampled every week for six months.

An observational design can show that two things go together but cannot, on its own, show that one causes the other. Only an experiment in which the researcher assigns the treatments at random supports a statement of cause and effect.

Principles of a sound design

Variables, controls and the basic steps of an experiment are introduced in the topic on Experiments. The principles below decide whether the data collected can be trusted.

• Control: A comparison group or condition that differs from the treated group in the factor under study only. A reagent blank and a certified reference material are controls in analytical work.
• Randomisation: Treatments, positions on the bench and the order of analysis are assigned by chance, not by choice. This stops the technician's preferences, bench position or time of day from lining up with one treatment.
• Replication: Each treatment is applied to several independent units. Replicates measure natural variation, and without them no test of a difference is possible.
• Blocking: Units are grouped by a known source of variation (analysis day, batch of reagent, instrument) and every treatment appears in every block. The block effect is then removed from the comparison.
• Blinding: The person measuring or recording does not know which treatment a sample received, so that expectation does not shape a reading or a judgement.
• Standardisation: Everything not under study (volumes, temperatures, times, instrument settings) is held the same for all units and written in the procedure.

Replicates and pseudoreplication

A true replicate is an independent unit that receives the treatment separately: one flask, one soil core, one bottle from a different position in the batch. Repeated readings on one prepared solution measure the instrument, not the treatment. Treating three readings from one flask as three replicates is pseudoreplication. It makes the sample look larger than it is and makes differences look more certain than they are.

Example: a technician prepares one digest of a dried leaf sample and reads it three times on the spectrophotometer. The study has one replicate with three readings. To have three replicates, three separate portions of the leaf material are weighed, digested and read.

Common experimental layouts

• Completely randomised design: Units are alike and every unit has an equal chance of receiving every treatment. It is used when conditions in the laboratory are uniform.
• Randomised complete block design: Units are first grouped into blocks, and every treatment is applied once in each block, in random order within the block. Analysis day is a frequent block when the work cannot be done in one session.
• Factorial design: Two or more factors are varied together, for example extraction time at two levels and temperature at two levels, so that each factor and their interaction can be studied in one experiment.

Worked example: allocating twelve flasks to three treatments (A, B and C), four flasks each, in a completely randomised design.

1. Number the flasks 1 to 12.
2. Put the numbers in random order with a random number table, a spreadsheet RAND column sorted from smallest to largest, or a statistical package. One random order obtained is 3, 6, 1, 7, 11, 5, 2, 4, 10, 12, 9, 8.
3. Give the first four numbers to treatment A, the next four to B and the last four to C.
4. Result: A receives flasks 3, 6, 1 and 7; B receives flasks 11, 5, 2 and 4; C receives flasks 10, 12, 9 and 8.
5. Record the allocation in the laboratory notebook before any treatment is applied, with the method used to generate the random order.

Steps in designing a study

1. State the research question and the objective in measurable terms, for example the difference in hardness removed from water by three softening resins.
2. Identify the response to be measured and the factors that may change it.
3. Choose the design type from the question, using the decisions above.
4. Decide the number of replicates and the sample size, using the methods in the topic on Sample size.
5. Plan randomisation, blocking and blinding, and write the allocation procedure.
6. Select the measuring methods and instruments, and plan the controls and blanks.
7. Choose the statistical analysis before collecting data, so that the data are collected in a form the analysis needs.
8. Run a small pilot of the whole design to expose problems, as described in the topic on Piloting of research instruments.
9. Write the full design in the study protocol and keep it unchanged while data are collected, recording any deviation with its reason.

Threats to a sound design

• Selection bias: The units studied differ from the target population because of how they were chosen, for example sampling only the points that can be reached on foot in the dry season.
• Confounding: A second factor changes together with the treatment and cannot be separated from it, such as treated samples all being analysed on the warmer afternoon.
• Measurement bias: An uncalibrated instrument or a method that reads high gives errors that all point the same way.
• Contamination between units: Carry-over between samples or shared glassware moves material from one treatment to another.
• Loss of units: Spoiled samples or failed runs, if they happen more in one treatment, leave an unbalanced study and a biased result.