Interim findings from a new recovery measurement tool

Many of you will know that we've been piloting a new self-report measurement tool intended to more accurately capture the state of recovery of a person following a musculoskeletal injury.  The impetus for the project is the findings that I and others have reported on previously - there is currently no universally accepted indicator of 'recovery', and many of the indicators that have been used to date are arguably flawed in some way.  I won't go into too much detail about the prototype tool right now (since some of you may be completing it, and you're not meant to know which is the new tool and which is the exisiting one against which we're comparing), but suffice it to say that the items were generated through a series of focus sessions with people in pain and clinicians, and has been refined through consultation with other international experts in the field.

Recruitment has been slow but steady.  Our target is 150 in order to allow us to confidently identify items that are redundant or problematic in other ways.  We've just hit 50 respondents, so I've just gone and done some preliminary analyses more to check the fidelity of the data than to draw any firm conclusions.  That said, there are two interesting tidbits I want to share now.

1.  We pose a question in the study (part of neither of the tools under review) which asks about worst and best pain over the past 2 days, and a third question asking respondents to finish the following sentence: "I would feel satisfied if my symptoms were generally no more intense than..."  In other words, what's the highest pain level that people would accept before they consider themselves satisfied?  Again, I stress that this is preliminary data and not to be considered the final word on this.  But, of the 50 that have responded so far, what's the mean pain intensity value that would be considered satisfactory?  2 out of 10.  The range is from 0 to 7 so far.  Surprised?  A subsequent question asks people to indicate how recovered they currently feel (0% to 100%), and then what minimum level of recovery would be required for them to feel satisfied.  So far the mean current state of recovery of the respondents has been 47%, and the minimum level for satisfaction: 73% (range of 10% to 100%).  There are far more questions than answers that will come out of this data once we have enough respondents, but I find this personally interesting, and in keeping with some of the research in the area to date.  The tool is also intended to help us evaluate the phenomenon known as 'response shift', which I imagine would be identified if we were to follow people over time with even these two questions.  Someone whose pain is currently 10 out of 10 would probably think they'd be satisfied if it never got higher than a 7.  But of course, once it hit a 7, the level for satisfaction would likely shift, which speaks to the value of revisiting goals frequently in people with pain and injury.

2.  For validation purposes, I've embedded two questions into the survey which simply require the respondent to choose a specific number in that row.  So without giving too much away, the item reads something to effect of 'Please choose [number] in this row'.  This is really just to make sure people are paying attention and are able to follow simple commands when completing the survey.  Often we assume that respondents are focusing on our questionnaires, but who knows what's really going on when people complete a questionnaire - I know I've been distracted in the past and could imagine entering an incorrect response.  But when evaluating the properties of a scale, it's important to know that you're respondents are at least paying attention.  It's a simple approach that is used surprisingly infrequently.  Anyway, so far out of 50 respondents, 4 have gotten one of these two 'validation' questions frankly wrong (chose the wrong number), and 3 more have left those two questions blank - I'm not sure how to interpret that.  That means that AT BEST 4 of 50 (8%) and AT WORST 7 of 50 (14%) of surveys now have to be considered invalid and removed from the database.  Since the cost of this study isn't high, I'm not losing any sleep over it.  But this does beg the question: how often might this happen in other, more costly studies?  For example, many RCTs use patient self-reported outcomes as primary dependent variables.  Thinking of some of the larger studies, if even 8% of say 500 respondents have just filled out the outcome questionnaires without really paying attention, that would mean that a full 40 responses are invalid and should have been removed from the analysis.  Using my worst-case scenario here of 14% invalid, 70 of 500 responses in our hypothetical (but very possible) study are invalid.  Certainly enough to affect the likelihood of finding statistically significant differences between groups (a construct termed 'assay sensitivity').  This is an interesting and often overlooked concern for any study that uses self-report as an outcome.

So there you have it, some interesting findings from preliminary analysis.  Nothing firm yet, but certainly nuggets for conversation.