This checklist is based on checklists from:
Spencer L. Ritchie J, Lewis J et al. [2003] Quality in qualitative evaluation: a framework for assessing research evidence. London: Government Chief Social Researcher's Office
Public Health Resource Unit England [2006] Critical Appraisal Skills Programme [CASP] – making sense of evidence: 10 questions to help you make sense of qualitative research
National Training and Research Appraisal Group [NTRAG]; contact:
British Sociological Association [BSA]
After completion of quality appraisal using the checklist, the included studies can be presented in a 'Quality of the included studies' table, which summarises the quality of each study under the main criteria of population, methods and analysis, and also the relevance of the study to the population being considered in the guideline.
Table 1 is an example table from the full guideline for Management of stable angina [NICE clinical guideline 126] for the review question 'What are the information needs of people with stable angina?'. In order to answer this question, three qualitative studies were included and assessed for quality.
The studies covered by this checklist are those that collect and analyse qualitative data – usually [but not exclusively] textual [written], spoken or observational data. Qualitative data are occasionally collected using structured questionnaires [for example, as thematically organised free-text comments] but such research needs to be scrutinised carefully because it may not meet acceptable quality criteria for consideration as a qualitative study.
There is considerable debate over which quality criteria should be used to assess qualitative studies. Quality in qualitative research can be assessed using the same broad concepts of validity [or trustworthiness] used for quantitative research, but these need to be put in a different contextual framework to take into account the aims of qualitative research. This checklist is based on the broadly accepted principles that characterise qualitative research and that may affect its validity; it is concerned with adequate reporting of key factors that affect the quality of qualitative research studies. The questions in the checklist are framed to encompass the variety of ways in which qualitative research is conducted. Care must be taken to apply the checklist in a way that matches the research methodology.
The following notes provide suggestions for completing the checklist. A list of publications on qualitative research is provided at the end of these notes for further reading on this topic.
Note that the sub-questions given as examples under each question in the checklist are intended to highlight some of the key issues to be considered for that question – they are not intended to be exhaustive. Please add any additional considerations in the comments box.
This section deals with the underlying theory and principles applied to the research.
A qualitative approach can be judged to be appropriate when the research sets out to investigate phenomena that are not easy to quantify or measure accurately, or where such measurement would be arbitrary and inexact.
Qualitative research in a health setting commonly measures:
-
personal experiences [for example, of a condition, treatment or situation]
-
processes [for example, action research, practitioner or patient views on the acceptability of using new technology]
-
personal values and beliefs [for example, about death, birth, disability]
-
interactions and relationships [for example, the quality of the GP–patient relationship, the openness of a psychotherapeutic relationship]
-
service evaluations [for example, what was good or bad about patients' experiences of a smoking cessation group].
If clear numerical measures could reasonably have been put in place, then consider whether a quantitative approach may have been more appropriate.
The design of qualitative research tends to be 'theory generative' rather than 'theory testing'; it is therefore unlikely that a research question will be found in the form of a hypothesis or null hypothesis in the way that you would expect in traditional quantitative research. Nevertheless, what the study is investigating should still be set out early and clearly. The research question should be set in context, with a summary of the background literature and the study's underpinning values and assumptions.
This section considers the robustness of the design of the research project.
There are a large number of qualitative methodologies, and a tendency in healthcare studies to 'mix' aspects of different methodologies or to use a generic qualitative method. From a qualitative perspective, this should not compromise the quality of the study if the research design captures appropriate data and has an appropriate plan of analysis for the subject under investigation.
Sampling in qualitative research can be purposive. Qualitative research is not experimental and does not purport to be generalisable, and therefore does not require a large or random sample. People are usually 'chosen' for qualitative research based on being key informers. The choice of sample and sampling method should be described, ideally including any shortcomings of the sample.
Assess whether the methods of data collection are described with details of the following:
-
how the data were collected
-
how the data were recorded and transcribed [if verbal data]
-
how the data were stored
-
what records were kept of the data collection.
Were these appropriate methods of data collection to use, given the aims of the research?
Assessing the validity of qualitative research is very different from assessing that of quantitative research. Qualitative research is much more focused on demonstrating the causes of bias rather than eliminating them. The report should include sections discussing the reflexive position of the researcher [their 'role' in the research], the context in which the research was conducted and the reliability of the actual data.
It is important when gauging the validity of qualitative data to consider whether the data are plausible and realistic. To make an accurate assessment of this, it is important to describe the context of the research in terms of the physical context [for example, youth club, GP surgery, gang headquarters] and who else was there [for example, participants are likely to position themselves very differently, and thus to respond very differently, in a discussion with parents present compared with a discussion with peers present]. The participants should be described in enough detail to allow some insight into their life and situation and any potential context bias considered by the authors [that is, interpretation of the influence of the setting].
It is important that the method used to collect the data is appropriate for the research question and that the data generated map well to the aims of the study. Ideally, more than one method should have been used to collect data.
Qualitative data analysis is very different from quantitative analysis. This does not mean that it should not be systematic and rigorous; however, systematisation and rigour require different methods of assessment.
Qualitative researchers use the adjective 'rich' to describe data that are in-depth, convincing, compelling and detailed enough that they can provide some insight into the research participants' experience. It is also important to know the 'context' of the data – where they came from, what prompted them, what they pertain to, and so on.
The analysis of data can be made more reliable by the researchers putting checks in place. Sections of data should be coded by another researcher or, as a minimum, a second researcher should check the coding for consistency. Participants may also verify the transcripts of their interview [or other data collection, if appropriate]. Negative or discrepant results should be highlighted and discussed.
The results of the research should be convincing or credible. Findings should be presented clearly and organised logically and the authors should consider and explain any contradictions. Extracts from original data should be included where possible to give a fuller sense of the findings. These data should be appropriately referenced – although you would expect data to be anonymised, they still need to be referenced in relevant ways [for example, if sex differences were important, then you would expect extracts to be marked male or female].
This section is self explanatory.
All qualitative research involves ethical considerations, and these should be considered within any research report. Ideally there should be a full discussion of ethics, although this is rare because of space constraints in peer-reviewed journals. Any qualitative research should be approved by a research ethics committee, and this should be stated in the report so that it is clear that every care was taken to protect research participants.
The researcher should have considered their role in the research; for example, as a reader, interviewer or observer. This is often referred to as 'reflexivity'. The 'status' of the researcher can profoundly affect the data. For example, a middle-aged woman and an 18-year-old man are likely to get different responses to questions about sexual activity when interviewing a group of teenage boys. It is important to consider age, gender, ethnicity and 'insider' status [such as whether the interviewer or researcher is part of the group being researched or has the same condition or illness]. The researcher can also profoundly influence the data by use of questions, opinions, judgements and so on, so it is important to know what the researcher's position is in this regard, and how the researcher introduced and talked about the research with the participants.
Barbour RS [2001] Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? British Medical Journal 322: 1115–7
Daly J, Willis K, Small R et al. [2007] A hierarchy of evidence for assessing qualitative health research. Journal of Clinical Epidemiology 60: 43–9
Mays N, Pope C [2000] Assessing quality in qualitative research. British Medical Journal 320: 50–2
Miller G, Dingwall R, editors [1997] Context and method in qualitative research. London: Sage