Evaluating a Health Measurement: The User’s Perspective

By: Ian McDowell

We recognize that everyone would like a guidebook to make recommendations about the “best buy.” This, of course, is a difficult judgment to make without knowing about the study in which the reader intends to use the method. We give several indications of the relative merits of the scales, but all methods we review have different strengths, so we can only make suggestions to the reader as to how to use the information in this article to choose an appropriate measurement.

The user must decide exactly what is required of the measurement. For example, will it be used to evaluate a program of care or to study individual patients? What type of patient will be assessed (e.g., diagnosis, age group, level of disability)? What time frame must the assessment cover: acute or long-term conditions? How broad ranging an assessment must be made, and how detailed does the information need to be? For example, would a single rating of pain level suffice, or is a more extensive description of the type as well as the intensity of the pain needed?

Bear in mind that this may require 15 minutes of the patient’s time and that a person in pain will be unenthusiastic about answering a lengthy questionnaire. The user must decide, in sum, on the appropriate balance to strike between the detail and accuracy required, and the effort of collecting it.

Turning to how the user evaluates the published information on a measurement method, the following characteristics of a method should be considered:

1. Is the purpose of the method fully explained and is it appropriate for the intended use? The method should have been tested on the type of person who will be taking it.

2. Is the method broad enough for the intended application, asking neither too many nor too few questions? Is it capable of identifying levels of positive health where this is relevant?

3. What is the conceptual approach to the measurement topic? For example, which theory of pain does it follow, and is its approach consonant with the orientation of the study? Is the theory well-established (e.g., Maslow’s hierarchy of needs) or is it an idiosyncratic notion that may not correspond to a broader body of knowledge?

4. How feasible is the method to administer? How long does administration take? Can it be self-administered? Is professional expertise required to apply or interpret the instrument? Does it use readily available data (e.g., information already contained in medical records) and will the measure be readily acceptable to respondents? What response rates have been achieved using the method? Is the questionnaire readily available? Is at free? Above all, does the instruction manual clearly specify how the questions should be asked?

5. Is it clear how the method is scored? Is the numerical quality of the scores suited to the type of statistical analyses planned? If the method uses an overall score, how is this to be interpreted?

6. What degree of change can be detected by the method, and is this adequate for the purpose? Does the method detect qualitative changes only, or does it provide quantitative data? Might it produce a false-negative result due to insensitivity to change (e.g., in a study comparing two types of therapy)? Is it suitable as a screening test only, or can it provide sufficiently detailed information to indicate diagnoses?

7. How strong is the available evidence for reliability and validity? How many different forms of quality testing have been carried out? How many other indices has it been compared with? How many different users have tested the method, and did they obtain similar results? How do these compare to the quality of other scales?

One difficulty commonly encountered in comparing two indices is where one shows excellent validity results in one or two studies and the other is more widely tested but shows somewhat less adequate validity. The reader should pay attention to the size of the validation studies: frequently, apparently excellent results obtained from initial smaller samples are not repeated in larger studies.

Ultimately the selection of a measurement contains an element of skill and even luck; it is often prudent to apply more than one measurement. This has the advantage of reinforcing the conclusions of the study when the results from ostensibly similar methods are in agreement, and it also serves to increase our general understanding of the comparability of the measurements.

References

Ian McDowell,  MEASURING HEALTH, A Guide to Rating Scales and Questionnaires, third edition, 2006, Oxford University Press

Read Also

Principles and Practice of assessment in primary care settings

Barriers to the implementation of Standardized Screening and outcomes measurement

Questions to consider when deciding which measurement instrument to use

Brief measures for screening and measuring mental Health outcomes

Factors Influencing Assessment

Potential considerations in Selection of a diagnostic interview

WHODAS: Assessing Disability

The Current Status of Health Measurement
Types of Health Measurements
Identifying and Controlling Biases in Subjective Judgments in health measurement