Pragmatism in manual therapy trials for knee osteoarthritis: a systematic review

Search strategy

A literature search was performed using PubMed, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Central Register of Controlled Trials (CENTRAL), and Embase to identify trials published prior to April 2023. In addition to these databases, the authors performed manual searches by cross-referencing trials included in related systematic reviews to capture all relevant studies in order to maximize the quality of this review. The systematic reviews that were examined consisted of any related to the eligibility criteria for this review.

Search strategies were developed using medical subject headings (MeSH) and keywords pertaining to the knee, OA, manual therapy, randomized controlled trials, and adult/young adult. Medical librarians assisted with the searches (Supplementary material, Appendix A). The primary search methods used were appropriate to each database, which included MeSH terms, CINAHL headings, subject headings, and keywords and their synonyms. Truncation and wildcards were used to account for different spellings and alternative words that may be used to describe our keywords (e.g., arthr* to identify arthritis or arthrosis). The Boolean operators “AND” and “OR” were used to combine search terms. Filters for the English language and the time frame of 1975 to April 2023 were used.

Study selection

Randomized clinical trials where the primary focus was assessing the effect of manual therapy interventions for adult patients with knee OA were included. Full text of all trials had to be available in the English language. Animal trials, trials that included subjects with diagnoses other than knee OA in any compartment, or trials where subjects had any surgery in the past 6 months or had undergone a knee arthroplasty in the involved knee at any time were excluded (Supplementary material, Appendix B).

Because the label of manual therapy can be broad and extensive (e.g., includes massage, lymphatic drainage, passive range of motion) (27), we deliberately limited the definition of manual therapy for this review as a treatment primarily consisting of joint mobilizations or manipulations performed by a healthcare provider, even if it was part of a multimodal intervention as long as the effect of the manual therapy intervention was being assessed. Trials including other forms of manual therapy (e.g., massage, soft tissue mobilization, lymphatic massage/drainage, cupping, dry needling, acupuncture, acupressure, and stretching) in the absence of joint mobilization or manipulation were excluded. Trials assessing manual therapy as part of a group of interventions where the effect of manual therapy was not assessed (e.g., a trial where everyone received manual therapy as part of standard care and the purpose of the trial was to assess the effect of other interventions, such as exercise, education, medications, etc.) were also excluded from the review. The eligibility requirements for this review were chosen to maximize the relevance and overall quality of this review.

Data management

Covidence data management software (Veritas Health Innovation Ltd, Melbourne, Australia) was used for study screening, full-text review, and data extraction (28).

Data extraction

Two reviewers independently screened all titles and abstracts to determine eligibility for full-text review. Any disagreements were discussed for resolution, and a third reviewer was consulted for final disposition, as necessary. Upon completion of title and abstract screenings, the remaining full-text trials were screened by the same two reviewers using the predetermined eligibility criteria. Reasons for exclusion were documented within Covidence (Supplementary material, Appendix C).

Data extracted included total number of subjects, mean age in years, mean body mass index, proportion of males and females, and the year the trial was published. In addition, the RITES tool was used to rate the efficacy-effectiveness nature of each study. Descriptors of maximal efficacy and maximal efficiency are provided in Table 1 (19). The RITES tool is used to rate the efficacy-effectiveness nature of trials by assessing four different domains (participant characteristics, trial settings, flexibility of interventions, and clinical relevance of experimental and comparison interventions) using a 5-point Likert scale, with 1 indicating a strong emphasis on efficacy (more explanatory), and 5 indicating a strong emphasis on effectiveness (more pragmatic) (19). The two reviewers independently scored each study using the RITES tool and consulted with a third reviewer when there was a lack of consensus.

Risk of bias assessment

The Cochrane Collaboration Risk of Bias 2.0 assessment tool (RoB-2) was used to assess the risk of bias across five separate domains: randomization process, deviations from the intended interventions, missing outcome data, measurement of the outcome, and selection of the reported results (29,30). Each domain was rated as having low risk, high risk, or some concerns regarding the risk of bias for that trial. The two reviewers independently scored each trial using RoB-2 to determine the potential risk for bias when looking at the results. In the event of a difference in opinion, consensus was reached by consulting with a third reviewer.

Efficacy-effectiveness spectrum

The RITES tool was used to assess where the components of each trial fell along the efficacy-effectiveness continuum (14). It was developed for post hoc assessment of trials in a systematic review based on efficacy-effectiveness continuum along four domains: participant characteristics, trial settings, flexibility of intervention(s), and clinical relevance of experimental and comparison intervention(s). A Likert scale ranging from 1 (strong emphasis on efficacy) to 5 (strong emphasis on effectiveness) is used in scoring. A rating of not applicable (N/A) may be given when information for a domain is unavailable. Trials typically cannot be completely categorized as explanatory or pragmatic as a whole, but instead rated along a continuum. In addition, different components of a trial design may fall in different places along the efficacy-effectiveness continuum. Thus, each domain is scored independently, without putting forth an overall score for a trial.

Data synthesis and analysis

Interrater reliability between reviewers was calculated for title and abstract and full-text screening using Cohen’s kappa. Levels of agreement were defined as <0 = no agreement, 0-0.20 = slight agreement, 0.21-0.40 = fair agreement, 0.41-0.60 = moderate agreement, 0.61-0.80 = substantial agreement, and 0.81-1.0 = almost perfect agreement (31). Descriptive statistics were calculated for RITES tool scores that included the count and percentage within each of the four domains. For each RITES domain, the results were separated into three different groups, those that had more emphasis on efficacy (scores of 1-2), those with more emphasis on effectiveness (scores of 4-5), and those that were balanced or neutral (scores of 3). In addition, the four domain scores from each trial were averaged together to determine if the individual trial design, with all domain scores considered together, leaned more toward efficacy or efficiency. For RoB-2, count data and percentage were calculated for all trials based on ratings of low risk, high risk, or some concerns regarding the risk of bias. Interrater reliability between reviewers was assessed for all four domains of the RITES tool and final RoB-2 scores. Finally, all trials were classified as being positive or null based on the primary outcome and then assessed for associations with trial design emphasis on efficacy vs. effectiveness using contingency tables and Fisher’s exact test. IBM SPSS Statistics (version 28; Chicago, IL) was used for all analyses.

Deviations from prospective protocol registration

There were no deviations from the prospective protocol registration.

Search results

The initial search yielded 2,656 citations, and after removing 1,584 duplicates, 1,074 titles and abstracts required screening. After title and abstract screening and full-text review, 36 trials (13,14,32-65) were included in the final review (Fig. 1). Specific details regarding each trial that was excluded can be found in Supplementary material, Appendix C. Features of the 36 trials are included in Supplementary material, Appendices D and E.

RITES domain scores

Overall RITES scores by domain are provided in Figure 2. A higher percentage of trials had a greater emphasis on efficacy within all four domains: participant characteristics (75.0%; n = 27) (13,34-42,44,45,48-50,52-55,57-62,64,65), trial setting (77.8%; n = 28) (13,14,32-36,40,43-49,51-55,57-60,62-65), flexibility of intervention (58.3%; n = 21) (33-36,38,40,43,44,46-48,50-52,54,55,58,60,62,63,65), and clinical relevance of experimental and comparison intervention (47.2%; n = 17) (14,34-36,38,40,43,47,48,51,52,54,55,57,61,62,64). In addition, when the RITES scores for all four domains of each trial were averaged, 29 trials were more oriented toward efficacy (mean [SD] of 2.2 [0.4] and range 1 to 3) (13,14,34-36,38,40,43-55,57-65), whereas five trials were more oriented toward effectiveness (mean [SD] of 3.9 [0.5] and range 3 to 5) (32,37,41,42,56). The remaining two trials had a mean score of 3.0 across the four domains, indicating a balanced emphasis between efficacy and effectiveness (33,39). Despite this overall emphasis on efficacy, 20 of the 36 trials had at least one domain with a score greater on the effectiveness spectrum (13,14,32,33,37-39,41,42,44,46,47,49,51,56,57,60,61,63,65) (Tab. 2).

For the participant characteristics domain, 27 trials (75.0%) had scores that emphasized efficacy (13,34-42,44,45,48-50,52-55,57-62,64), five trials (13.9%) emphasized effectiveness (32,33,47,51,63), and four trials (11.1%) had a balanced emphasis between efficacy and effectiveness (14,43,46,56). In the trial setting domain, 28 trials (77.8%) had scores that emphasized efficacy (13,14,32-36,40,43-49,51-55,57-60,62-65), six trials (16.7%) emphasized effectiveness (37-39,41,42,61), and two trials (5.5%) had a balanced emphasis between efficacy and effectiveness (50,56). The flexibility of intervention(s) domain had 21 trials (58.3%) that emphasized efficacy (33-36,38,40,43,44,46-48,50-52,54,55,58,60,62,63,65), nine trials (25.0%) emphasizing effectiveness (13,14,32,37,39,41,42,56,57), and six trials (16.7%) that exhibited a balanced emphasis between efficacy and effectiveness (45,49,53,59,61,64). Finally, the clinical relevance of experimental and comparison intervention(s) domain had 17 trials (47.2%) that emphasized efficacy (14,34-36,38,40,43,47,48,51,52,54,55,57,61,62,64), 13 trials (36.1%) emphasized effectiveness (13,32,33,37,41,42,44,46,49,56,60,63,65), and six trials (16.7%) had a balanced emphasis between efficacy and effectiveness (39,45,50,53,58,59).

Of the 36 trials, only seven had null findings (41,42,45,48,63-65). Fisher’s exact test revealed no statistically significant relationship between where studies fell on the efficacy-effectiveness spectrum and a positive outcome of the primary outcome (p = 0.27).

Risk of bias for included trials

Five of the included trials (13.9%) had low risk of bias (13,36,54,59,65), 15 trials (41.7%) had high risk of bias (44-52,55,57,58,62-64), and 16 trials (44.4%) had some concerns for risk of bias (14,32-35,37-43,53,56,60,61) (Supplementary material, Appendix F). The most common cause for bias included measurement of the outcome (44,45,47,49,51,52,55,57,58,64), and the least amount of bias was in the selection of reported outcome (13,14,32-65). When comparing risk of bias across the trials, all five of those with low risk of bias also had an emphasis on efficacy (13,36,54,59,65).

Rater agreement

Interrater reliability was к = 0.25 (fair agreement) for title and abstract screening and к = 0.31 (fair agreement) for full-text screening. Interrater reliability between reviewers for the participants’ characteristics domain was к = 0.45 (fair agreement), к = 0.39 (fair agreement) for trial settings, к = 0.34 (fair agreement) for flexibility of interventions, and к = 0.32 (fair agreement) for clinical relevance of experimental and comparison interventions. Interrater reliability between the reviewers for RoB-2 was к = 0.04 (slight agreement). These values were related to initial agreement when reviewing the trials. It is important to note that consensus was reached on all initial ratings, and a third reviewer needed to be consulted for only three trials (8.3%).

Participant characteristics

A large percentage of trials (75.0%) were higher on the explanatory end of the spectrum in the participant characteristics domain, with the primary reason being related to their exclusion criteria (13,34-42,44,45,48-50,52-55,57-65). Patients were most commonly excluded from trials due to the presence of other diagnoses or comorbidities, and while this could confound treatment effect, it is a more accurate representation of patients seeking care for knee OA. For example, vascular and cardiovascular disease, obesity, acute or chronic pain in the spine, hip, or ankle, and those using anti-inflammatory drugs beyond simple pain relievers are common presentations for individuals with knee OA (35,40). Excluding these individuals could result in conclusions that may not be relevant to the types of patients seen in most clinics (16,18). To achieve a more pragmatic rating would have required a study population that included patients with diagnoses, comorbidities, symptom durations, and age ranges similar to common knee OA patients that seek care (16,18).

Trial setting

The majority of trials (77.8%) had an emphasis on efficacy in the trial setting domain due to the trials being carried out in settings that were dissimilar from common practice (13,14,32-36,40,43-49,51-55,57-60,62-65). These included specialized clinics, specialized trial or academic centers, and military clinics and settings, and also used a limited number of clinicians who were often specifically trained for the interventions being assessed. While this may enable researchers to better determine the effect of the interventions without compromising internal validity, it limits external validity (15,18). To achieve more pragmatic trial settings, researchers should strive to use a broad array of clinics and clinicians that better mimic typical medical providers and healthcare settings (15,16).

Flexibility of interventions

The flexibility of interventions domain had an emphasis on efficacy. The majority of clinical trials (58.3%) required strict manual therapy protocols with little flexibility or prohibited cointerventions (33-36,38,40,43,44,46-48,50-52,54,55,58,60,62,63,65). Some reasons for a strict protocol include the ability to better attribute the treatment effect to the intervention being assessed, rather than an influence from other confounders. Even efforts to control or improve intervention adherence may lead to different results than can be expected in real-world settings (15,16). On the other hand, nine of the trials had a more pragmatic emphasis (13,14,32,37,39,41,42,56,57), which was accomplished by allowing more flexibility with the interventions between the trial populations. This approach allowed clinicians to manage patients based on their perceived needs with greater flexibility.

Clinical relevance of experimental and comparison intervention

Clinical relevance of experimental and comparison of interventions slightly favored efficacy (47.2%) (14,34-36,38,40,43,47,48,51,52,54,55,57,61,62,64) compared to those with designs more focused on effectiveness (36.1%) (13,32,33,37,41,42,44,46,49,56,60,63,65), and those that had a balanced emphasis on efficacy and effectiveness (16.7%) (39,45,50,53,58,59). Trials with a more explanatory design were less likely to have one of the treatment arms considered clinically relevant or best practice, such as using controls, placebo, or sham interventions, all of which provide a less than desirable comparison when considering generalizability to real-world settings (19). Treatment duration may also have been much shorter than the duration of treatments used in real-world practice (19). On the other hand, trials emphasizing a pragmatic approach use flexibility with interventions that mimic typical practice, and they use comparison groups that are often considered to represent best practice or usual care (19).

Outcomes relative to efficacy-effectiveness spectrum

When examining trial outcomes relative to where studies fell on the efficacy-effectiveness spectrum, there were no significant associations. This means that trial design has no bearing on whether a study showed a treatment effect. However, definitive conclusions cannot be made because there were only seven null trials out of the 36 trials (41,42,45,48,63-65).

Clinical implications

The majority of trials investigating manual therapy for knee OA were on the explanatory end of the spectrum across all four RITES domains. This is similar to what was reported by Maddox et al (66) for individuals with low back pain, except their review found a rather strong emphasis toward the pragmatic end of the spectrum with the domain related to clinical relevance of experimental and comparison interventions.

The role of explanatory trials is to analyze the mechanism of interventions under controlled circumstances (19). In this instance, explanatory trials help to determine if manual therapy is an effective treatment for knee OA. However, they lack the ability to generalize the results because the settings are often not representative of real-world clinical practice. That is where pragmatic trials provide their value by providing clinicians with the ability to know if manual therapy can be beneficial for patients with knee OA in real-world settings (15-17). The result of this review demonstrates lack of generalizability with the majority of studies examining manual therapy for knee OA.

Recommendation for future research

While the current body of literature demonstrates potential benefits when using manual therapy for individuals with knee OA, many of those recommendations come from trials that are more explanatory than pragmatic, making them less generalizable (5,9). Additional pragmatic studies examining manual therapy for knee OA in real-world scenarios and across a variety of settings and clinicians would help improve the applicability and implementation of these interventions. For example, the study design could include patient population with some comorbidities, especially those commonly associated with knee OA (diabetes and obesity); multiple/diverse trial settings or general clinical practice settings, not specialty treatment clinics; and flexibility of interventions, allowing cross-treatments whenever/if needed while ensuring that the methodology of interest is systematically and objectively directed toward best practice. Finally, it is worth noting that manual therapy may not be unique here, and these findings may be very similar to what is observed for trial designs of other interventions for knee OA.

Limitations

This review had the primary goal of assessing where this body of evidence falls on the efficacy-effectiveness spectrum, with no intention to examine the effectiveness of manual therapy for knee OA. Therefore, conclusions should not be inferred regarding pooled treatment effects or the value of manual therapy interventions for knee OA.