Received date: 23 December 2011; Accepted date: 15 April 2012
BackgroundTransparency in diabetes care requires quality indicators that are of interest to stakeholders in order to optimise their usage. Indicator development is often focused on consensus, and little is known about stakeholders’ preferences for information on quality. AimTo explore the preferences of consumers, providers, purchasers and policy makers for different quality domains and indicators in relation to the intended use of quality indicators. MethodsBetween June and December 2009, 14 semi-structured interviews were held with stakeholders who have a decisive vote in the selection of the national indicator set for diabetes care in the Netherlands. The following subjects were explored: (1) the aims of using information on quality; (2) the interpretation of and preferences for the quality domains of safety, timeliness, effectiveness and patient-centredness in relation to the user aims; and (3) the preferences for structure, process or outcome indicators. Content analysis was used to analyse qualitative data. ResultsStakeholders had similar and different aims according to their roles. The interpretations of quality domains varied greatly between the stakeholders. Besides differences in interpretation, their preferences were similar. Most stakeholders prioritized patient-centredness above the other domains of quality, ranked in order of priority as safety, effectiveness and timeliness, whereas purchasers also prioritised efficiency. All stakeholders preferred to use process indicators or a mix of process and outcome indicators. ConclusionsThe preferences of the stakeholders for quality indicators seem to be neither wellrefined nor congruent. The implementation of an indicator set can probably be improved if the stakeholders’ definitions and preferences for quality domains become more explicit during the selection process for indicators.
diabetes mellitus, information aims, quality domain, quality indicator, stakeholder preferences
Transparency in health care has become essential for healthcare consumers, healthcare providers, purchasers and policy makers. These stakeholders may use quality information for different purposes, including internal quality improvement, cost containment and accountability. [1,2] Currently, a great deal of information on quality has become available but is not optimally used.[1,3–5] This information is mostly expressed through quality indicators, beingmeasurable elements of healthcare performance. The presentation format of quality indicators can be a barrier to their use by healthcare consumers.[1,6,7] Another barrier is that physicians may doubt the accuracy and relevance of these quality indicators. Ideally, quality indicators are developed in consensus procedures, such as Delphi techniques, in which stakeholders are extensively involved and heard. Before arriving at a consensus on specific indicators, it is essential to learn more about stakeholders’ perspectives on quality indicators. Without this knowledge, one could easily develop a set of public indicators that does not adequately fit with the aims of the different stakeholders, thus creating a barrier to their implementation. Despite the importance of such knowledge,[11,12] there is little research in this area.
We therefore conducted a qualitative study to explore stakeholder preferences for quality domains in relation to the intended use of quality indicators. To compare these preferences, we felt it important to explore stakeholders’ interpretation of the quality domains. We hypothesised that different stakeholders with different interests may have different interpretations of quality domains and therefore different preferences. We restricted our study to primary diabetes care because several quality indicators for diabetes care are already available, although a national set was still lacking. We used the framework for quality assessment from the Institute of Medicine (IOM).
We also explored preferences for structure, process, or outcome quality indicators for diabetes care.
The present study draws on 14 semi-structured interviews with people from seven different organisations involved in the assessment of diabetes care quality using quality indicators. We aimed to include professionals who had a decisive influence on the inclusion of quality indicators in a national indicator set. This is not a very large group of people in the Netherlands. The interviewees represented: (1) healthcare consumers, (2) purchasers, (3) healthcare providers and (4) policy makers (Table 1). Purposive sampling was used to identify senior staff members who were engaged in quality of care measurement and improvement using quality indicators. For healthcare consumer perspectives, we contacted the federation of patient and consumer organisations, because this organisation deals with patient rights and preferences in theNetherlands.Purchasers’ perspectives were covered by the three main health insurance companies. For a healthcare provider perspective, we selected active members (diabetes nurses, primary care physicians and medical specialists) of the different professional organisations involved in healthcare quality improvement (Table 1). For policy makers’ perspectives, we contacted the health inspectorate, the authority within the Ministry of Health,Welfare and Sports responsible for the quality and safety of health care provided in the Netherlands. All participants received a letter containing information about the aim and methodology of the study in advance.
The interview guide was based on the quality framework of the IOM, comprising quality domains of safety, effectiveness, patient-centredness and timeliness. According to the IOM, the vast majority of quality indicators address effectiveness and safety, a smaller number examine timeliness and patient-centredness, and very few indicators assess the efficiency or equity of care. The IOM found that aspects of efficiency are reflected in other components of quality, with the exception of cost per unit service. There is also considerable confusion on how efficiency is defined: sometimes re-admission rates, procedure rates or costeffectiveness have been considered to be efficiency indicators. McGlynn’s report on the definition of efficiency defines efficiency as a relationship between a specific product (output) of health care and the resources (input) used to create the product. We left out efficiency because of this misunderstanding and its incorporation in other components. With regard to equity of care, the IOM stated that the four domains could differ between different groups in the population or across geographic areas, and therefore equity was not considered as a separate quality domain.
We used Donabedian’s model of quality measures addressing three levels of health care and its intended results: structure, process and outcome. A semistructured guide (Appendix) was used, including openended questions on: (1) the aims of using information on quality in order to clarify stakeholder perspectives on such information; (2) the interpretation of and preferences for the quality domains in relation to specific user aims, with a possibility of submitting additional domains; and (3) the preferences for information on quality in terms of structure, process and outcome, again in relation to specific user aims. The interview guide was pilot tested prior to the data collection.
Data collection and analysis
Fourteen face-to-face interviews were held between July and December 2009. All participants gave written consent to participate in the interviews. Interviews lasted between one and two hours. The interviews were conducted by two researchers: one conducted the interviews and the other took notes. All interviews were recorded on digital recorders with the permission of the participants. In order to ensure data accuracy, interviews were translated verbatim by the two researchers present at the interviews. The transcripts were then analysed independently by both researchers and emerging codes were systematically discussed by the research group. In case of disagreement, an independent (third) researcher would check the corresponding part of the transcript against the original interview recording. Any points of disagreement were resolved through discussion by the research group. Finally, summaries of the interviews including the emerging themes and the conclusions were sent back to interviewees, who were asked to check on consistency and accuracy.
The transcripts of each interview were analysed using content analysis18 according to a predetermined framework. This framework consisted of two main themes: the IOM quality domains and the Donabedian approach for quality indicators. Within these predefined themes, we identified parts of text (concepts) that related to categories. Next, we coded data by giving descriptive code names to the concepts (for example, GP, aim ‘internal quality improvement’, category ‘interpretation safety’, concept ‘patient level’). Later, we grouped similar codes under the larger, main concept. Finally, we organised our data by stakeholder in order to identify similarities and differences.
Qualitative analysis software (ATLAS.ti Win 6.1) was used to facilitate organising the data into concepts, categories and themes.19
Aims of using quality indicators
All stakeholders cited ‘improving the quality of care’ as the overarching goal in the use of quality measures. Stakeholders had different, as well as overlapping, aims and the number of aims per stakeholder varied from one to four. Healthcare consumers indicated that their interest in information on quality was for selecting care providers and services. Healthcare providers stated that they used information on quality for internal quality improvements (e.g. through audit, feedback and benchmarking). Policy makers and purchasers indicated that they used information on quality to detect areas of improvement and to inform healthcare consumers. In addition, policy makers aided quality improvement by stimulating cooperation between providers. Purchasers also mentioned that they used information on quality for performance- based reimbursement programmes to select or reward providers for meeting targets. Finally, information on quality was used for contracting purposes with healthcare providers.
Interpretations of the four quality domains
Safety has been defined as avoiding injuries or harm to patients from care that is intended to help them.13 Healthcare consumers and healthcare providers defined safety as reducing the possibilities of injuries or harm to patients. In addition, the healthcare consumers mentioned an extra aspect, namely:
That the patient feels safe with the healthcare provider. (HC2)
Healthcare providers interpreted safety at an organisational level:
It is important to have an organisation that prevents you from making mistakes or forgetting things. (HP1)
Policy makers and purchasers gave their interpretation at the national level:
Maintain the agreements of safety at a national level. (P2)
Effectiveness refers to providing services based on scientific knowledge to all who could benefit (avoiding overuse and underuse).13 All stakeholders interpreted effective care as reaching goals agreed between professionals and patients. When refining the effectiveness domain, however, interviewees mentioned different aspects, such as adherence to ‘evidence-based medicine’, treatment needs to be selected to do least harm or reaching good health outcomes at low costs.
Patient-centredness concerns health care that establishes a partnership among practitioners, patients and their families to ensure that decisions respect patients’ wants, needs and preferences; and that patients have the education and support they need to make decisions and participate in their own care.13 All interviewees mentioned ‘partnership’ in their interpretation of patient-centredness, followed by the degree to which patients’ expectations are met:
Do I offer the patient what he wants, and is he able to utilise it. (HP5)
In addition, the healthcare consumers indicated that each patient should be treated as an individual and should receive care that meets the individual needs.
Timeliness has been defined as obtaining care needed and minimising unnecessary delays in getting that care.13 Most respondents defined timeliness as no delays or waiting times before receiving care. From the healthcare consumers’ point of view, ‘a smooth and continuous flow of information through different stages of care’ should be added to this definition. Finally, two providers (HP1, HP4) were not able to define timeliness. Another provider (HP3) considered timeliness to be an inappropriate concept and suggested the use of ‘organisation of care’ instead.
Need for additional domains
All stakeholders added ‘efficiency’ as a separate domain of quality. They described efficiency as avoiding waste and overuse of healthcare services and an acceptable cost per unit of service in the providing of qualitative health care. Furthermore, ‘teamwork’, ‘innovation’ and ‘prevention’ were put forth as separate domains of quality.
Patient-centredness was considered to be the most important domain by all stakeholders, except purchasers. This was followed by safety, effectiveness and timeliness. One GP stated:
If care is not patient-centred, you will not achieve your aim whatever you do. (HP6)
Although purchasers underlined the importance of patient-centredness, according to their aim of cost containment, efficiency was rated as the most important quality domain, followed by patient-centredness.
Preferences for structure, process or outcome indicators
Healthcare consumers, providers and purchasers all indicated their preference for process indicators, because these were less sensitive to patient case-mix and to incomplete recording of clinical information. Outcome indicators were considered to have little relation with the quality of provided care and, for that reason, were listed second. Case-mix correction was believed relevant when interpreting results for the public domain and comparing performance. Policy makers, healthcare consumers and healthcare providers all preferred a set of process and outcome indicators. In the long run, purchasers would also prefer process and outcome indicators. Structural indicators were believed to provide basic information without sufficient detail on quality of care provided. One provider stated:
Structure forms the basics, while process is more straightforward. But of course, to fulfil my duties, I have to look at outcomes. (HP1)
In conclusion, process indicators were preferred, and when using advanced data collection and statistical methods, a mix of process and outcome indicators would be preferred.
Summary of the main findings
The aims of the stakeholders for using quality indicators varied and overlapped amongstakeholder groups. Stakeholders presented different interpretations of the quality domain safety, primarily due to the scope of their work (which varied from supervising healthcare providers to healthcare providers themselves). When refining definitions, there were differences in definition of effectiveness. In addition, the meaning of the quality domain timeliness was not well understood. Patient-centredness appeared to have a clear, shared focus, but because it was formulated very generally, this agreement was easily achieved. Despite stakeholders’ interpretation of domains varying greatly, their preferences for quality domains appeared to be concordant.
Overall, stakeholders viewed process indicators as most feasible, whereas for the public domain, a mix of process and outcome diabetes indicators was preferred. Some interviewees wanted to include process indicators only, because outcome indicators did not truly assess provider performance if case-mix was not corrected for. These interviewees were not convinced that this shortcoming was clear to all stakeholders.
Strengths and limitations
The number of interviews was small, but we included the most relevant organisations that participate in diabetes quality measurement and improvement, and within these organisations we interviewed those employees whose tasks were most closely related to quality of healthcare assessment or improvement. Even in this group of organisations, with a decisive vote in the selection procedure for national indicator sets, differences in views were substantial. Including more organisations, who would be involved less directly in the indicator selection procedure, would probably have revealed even more variation. Our study design involved a semi-structured interview because we hypothesised that only a discussion could help us answer our questions. In hindsight, an interview lasting one and a half hours is likely to be sufficient considering the schedule of the interviewees, but the discussion of the definition of the quality domains could have easily been longer in most cases.
The aims of information on quality explored were in line with other studies showing that healthcare providers use quality information for internal quality improvement as well as for the provision of quality information for their healthcare consumers.2,20 Healthcare consumers wanted information on the best providers, waiting times or insurance coverage.21 Furthermore, our results showed that although purchasers felt that they were customer-driven, the domain of patient-centrednesswas not their top priority. Their aims were to increase consumers’ healthcare choices through provision of comparative information on cost and quality and to provide financial incentives to providers for keeping costs low.22–24 Several studies also showed that purchasers and policy makers used quality information to detect areas for improvement, and to implement performance-based reimbursement programmes for healthcare providers. 25,26 Despite differences in the aims of providing information on quality, stakeholders’ preferences for quality domains overlapped to a large extent.
Our study showed that although stakeholders’ interpretations of the domains varied greatly, they had similar preferences for quality domains. Little research has been conducted on this subject, but recently, the Agency for Healthcare Research and Quality (AHRQ) has concluded that the information needs of stakeholders are neither well-refined nor stable.27 The results of our study confirm this. After considering the definitions given by the IOM,13 we conclude that the descriptions given by participants in our study for safety and effectiveness were very general. A proper description of the quality domain effectiveness, which implied evidence-based practice,28 was only mentioned by one provider. According to the IOM, patient-centredness is defined as focusing on the patient’s experience of illness and health care, and on the systems that work or fail to work to meet individual patient’s needs.29 Because of stakeholders’ general answers, their interpretations met this definition; however, we recommend further research on this subject. Delayed care, or timeliness, is a persistent and undesirable feature of current healthcare systems. 29 Timeliness is therefore a legitimate quality domain in health care.30 However, three providers experienced difficulties in interpreting this domain. Furthermore, some interviewees deviated from the IOM definition and this may explain why stakeholders indicated timeliness as being less important.
Needs for additional domains
Cost per unit of service was added as a quality domain by all stakeholders, which could be the result of rising costs, not only in real terms, but also as a percentage of gross domestic product.10 In some definitions of the IOM framework on quality domains, efficiency (including cost) is defined as a separate domain.28 Teamwork, prevention and innovation were other additional quality domains that were suggested by stakeholders. However, teamwork is an organisational aspect, which could be adequately covered by the domains safety and effectiveness.31,32 Prevention and innovation can be considered to be components of effectiveness that are based on the use of systematically acquired evidence to determine whether an intervention, such as a preventive service, produces better outcomes than alternatives. 28 These topics are not sufficiently apparent in current domains of quality and it would be desirable to develop quality indicators that reflect these aspects. The results of our study thus suggest that indicators for general public purposes should reflect the quality domains patient-centredness, safety, effectiveness, timeliness and costs.
Structure, process or outcome indicators
Donabedian’s classic paradigm for assessing quality of care consists of three levels of health care: structure, process and outcome of care. Stakeholders appeared to prefer easily measured process indicators that linked to sound evidence on outcomes. This finding was in line with the results from other studies showing that adjustment to patient case-mix is an important concern for physicians when publishing quality information. 33,34
Indicator selection or development procedures should become more focused on the definitions of quality domains, as well as on the differences in the stakeholders’ perspectives. Different interpretations will lead to different definitions of indicators in the same quality domain. Not addressing these issues could encourage stakeholders to develop new sets of indicators to satisfy their specific aims, creating an additional administrative burden and resulting in a rash of ineffective indicator sets.10 We believe that this problem is not restricted to diabetes care, and probably applies to all quality indicators, as the quality domains in our study are relevant to every health problem. One solution would be to deal with different perspectives during indicator selection or development by taking time to explain definitions and raise different perspectives. This could result in a more efficient development and use of quality indicator sets by stakeholders. In addition, we need to be aware that the relative importance of different quality domains and quality indicators may vary not only between stakeholders, but also over time, which underlines the importance of this exploration as an explicit step in the selection procedure.
We thank Ellen Keizer, IQ healthcare, for checking the accuracy of all transcripts against the original recordings.
The Netherlands Organisation for Health Research and Development (ZonMw).
Ethical approval is not necessary for interviews with healthcare professionals in the Netherlands.
Not commissioned; externally peer reviewed.