Evolving trends of systematic reviews on virtual reality for stroke rehabilitation

Study design

We conducted a cross-sectional study based on an overview of reviews (11). We adapted items from the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) (12), assuming as units of analysis the included SRs (see Supplementary 1).

Search strategy and data collection

We moved from a recent overview of reviews (11) investigating VR on stroke (CRD42022329263), which included 58 SRs and 345 unique primary studies.

The search was launched in multiple databases (the Cochrane Database of Systematic Reviews, EMBASE, MEDLINE, Scopus, ISI Web of Science, CINAHL, PsycINFO, PEDro, Otseeker, Healthevidence.org, Epistemonikos), including PROSPERO for ongoing SRs from inception up to January 17, 2023. We selected SRs published in English including adults with any diagnosis of stroke. The treatment investigated was any kind of immersive, semi-immersive, or non-immersive VR intervention, either with or without conventional therapy (e.g., usual care, exercises). Details of eligibility criteria are reported in our previous publication (11).

Data collection

We obtain the dataset of the related overview (11) to collect information about the general characteristics and methodological quality of SRs. Particularly, we used the following general characteristics: years of publication, countries of the corresponding author, description of outcomes, references and year of included primary studies, number of included primary studies, sample size, journal of publication, and journal impact factor (JIF), methodological quality appraised by A Measurement Tool to Assess SR (AMSTAR) 2 and categorized into critically low, low, moderate, and high methodological quality (13).

Data synthesis

We used descriptive statistics for general characteristics and methodological quality expressed as median and interquartile range (IQR) or absolute value and frequency.

The data chart and Pearson’s correlation coefficient were used to assume linearity assumption. Data were transformed into a logarithmic scale. Prediction of the percentage of published and ongoing SR data based on year was computed through linear regression models plotted with confidence and prediction intervals at 95%. All analyses were performed using the R Core Team (2023) (14).

We then visually described the trend of the following variables: median JIF, number of included primary studies and participants, outcomes, and methodological quality using Microsoft Excel 2019 and RAWGraphs 2.0 (15).

Publication trend

Overall, the data set consisted of 58 SRs published between 2007 and 2022, including 345 primary studies published between 1999 and 2021, and 45 ongoing SRs from 2014 to 2022. We found a significant increase in the number of published and ongoing SRs over the years (R² = 0.8654; R² = 0.747) respectively) (Figure 1). In Supplementary 2, we reported the trend of SRs and primary studies publication for each year (Figure S1-S2, respectively).

General characteristics of published systematic reviews

In Table 1 we showed overall general characteristics of published SRs. Most SRs include RCTs only (81%). Overall, 42 SRs (72.4%) conducted meta-analyses. Many SRs (69%) included mixed onset of stroke. The most common journal of publication was the Journal of Stroke and Cerebrovascular Diseases (8.6%) (Supplementary 2, Table S1). The median JIF was 3.25 (2.18-4.61). The distribution of the median JIF over the years is reported in Supplementary 2, Figure S3.

Primary studies and participants of included SRs

The number of primary studies included in an SR ranged from a minimum of three to a maximum of 87, with a median of 17 per SR. More than 85% of primary studies were RCTs. The number of participants included in the SRs ranged from a minimum of 60 to a maximum of 3540, with a median of 493. Supplementary 2, Figure S4-S5 shows the distribution of primary studies and participants over the years.

Countries

As regards the Asian states, China is the first country for the number of reviews, with 15 publications (25.9% of all SRs). In Europe, 16 SRs were published in eight different countries; Spain holds 5 publications (8.6% of all SRs), whereas Italy and Belgium hold 3 reviews each one (5.2%). America is the third continent for publications of SRs, with Brazil involved in 6 studies (10.3%), Canada in 4 (6.9%), and the USA in 3 (5.2%). In Oceania, 4 SRs were published. Only one review was conducted in Africa. The distribution of the continents where SRs have been conducted both generally and over the years is illustrated in Figure 2A-B. In the last three years we found an absolute increase in publications in Asia (n = 18, 31% of the overall sample).

Outcomes

Overall, the main outcome assessed was Upper Extremity Function and Activity in 67.2% of SRs (n = 39), followed by Activities of Daily Living (n = 37) and Gait and Balance (n = 36). Less investigated outcomes were Participation (n = 28), Cognitive and Mental function (n = 20), and Adverse Events (n = 18) (Table 1). Figure 3 shows the outcome distribution over the years. In the last three years, 28 SRs (48.3% of the overall sample) assessed Upper Limb Function and Activity.

Methodological quality

According to the AMSTAR 2 tool, 45 SRs (77.6%) were judged critically low, 12 low (20.7%), and one high (1.7%). Overall, 69% of SRs have no recorded protocol, and 89.7% do not describe motivations for the excluded primary studies. The 96.6% of SRs accomplished exhaustive bibliographic research. As regards the quality of the primary studies included in the 58 SRs, all authors used an adequate tool to measure the risk of bias (100%), and 74.1% included the assessment of primary studies in their SRs’ results (Supplementary File 2, Figure S6). Figure 4 shows the methodological quality distribution over the years.

Main findings

We analyzed the frequency and characteristics of 58 published SRs and 45 ongoing SRs covering the scientific diffusion of VR technologies for stroke from 2007 to 2022 and from 2014 to 2022, respectively, with an increasing trend over the years.

VR research is becoming more influential around the world, with over one-fifth of all countries involved in the scientific progress in this field (16). Asia represents the most influential country regarding VR rehabilitation in stroke adult people, with a particular increase in SR’s publications in the last three years (31%). In particular, China has been the leader in SR publications since it started in 2007 to cover not only the VR scientific field but also all medical fields, as it is in the top publishing countries, as reported by recent publications (17,18). With its aging population, China faces increasing challenges for stroke care and prevention, with a prevalence of stroke survivors of 58.1 million, four times higher than other countries (19). It has been argued that this rapid increase in the number of SRs could be due to multiple reasons, including the easiest widely accessible tools for doing SRs and meta-analyses (20), the pressures of academia (21), or industry of contracting companies “operating in the domain of evidence synthesis” to produce these publications, many of which probably remain unpublished (22). However, we cannot exclude that this increase is due to the growth of collaborative research with China and other countries, as already reported (23)

As the number of SRs has increased over time, the number of participants included in SRs and the number of included primary studies increased. This phenomenon might be conditioned by the fact that the latest published SRs can also include primary studies as well as participants of the oldest SRs. However, the number of primary studies retrieved from SRs may be underestimated considering that a time span exists from running the search strategy and publication of SRs (24); therefore, we cannot exclude that some published primary studies were not included in the SRs as well as new primary studies might be actually still ongoing. The year 2019-2020 was represented by a global pandemic emergency with limited possibilities to undergo primary studies. This might explain the substantial growth of secondary literature studies rather than primary studies in the last recent years.

Looking at the health outcomes, upper limb function and activity, participation, and cognitive and mental function are becoming more assessed in SRs as a sign of implementation of a core outcome set that can optimize the quality of poststroke rehabilitation (24).

We also found that the methodological quality of SRs remained critically low over time, even if SRs were published in journals with a median JIF of 3.25. It has been found that JIF may have little to no association with study results or methodological quality (25). Low methodological quality can be mainly due to some critical flaws in the protocol and in the excluded studied justification items. In fact, according to AMSTAR 2, 69% of our included SRs have no explicit statement that the study methods were established prior to the conduct of the review. On one hand, these findings are discouraging, as more than half of the authors (55.8%) register their protocol prior to publishing their systematic review/meta-analysis (26). Pre-registration does not guarantee that the protocol is complete but allows readers to be aware of methods for conducting and reporting in a transparent way (22). On the other hand, in the last years, we collected many ongoing SRs pre-registered in PROSPERO as a starting point of improvement for this flaw. However, authors should use registers such as PROSPERO to check overlapping questions covered by already existing SRs to avoid redundant meta-analyses with inconsistency and discordant findings. As well, journal editors should keep in mind that much has already been published. Multiple overlapping SRs can facilitate the origin of disputes. This is well known also in other fields such as the case of thrombolytic therapy for pulmonary embolism (27). As well, the large increase in the number of published and ongoing SRs over the years raises concerns about research waste. It should be ethical and reasonable to systematically review what is already known before deciding to perform any new study (28).

The other important flaw in SRs is related to the non-reporting of studies excluded from SRs. We found the 89.7% don’t provide a list of the excluded studies and the motivation. This can lead to selective inclusion for outcome non-reporting bias impacting meta-analytic effects (2930-31).

Recently, a meta-epidemiological study found that 58,8% of the included SRs (n = 131) excluded studies due to “no relevant outcome data” (32) despite it not being recommended by the scientific community (33) since this may be a consequence of selective outcome reporting and therefore compromise the systematic review reliability.

Strength and limitation

This review aimed to summarize all the publications and trends about the application of VR in the neurorehabilitation field in adult people with stroke. To analyze this specific sample, we included a total of 103 papers (58 published and 45 ongoing SRs) without limits in a publication year. An exhaustive research was conducted in many different databases. This trend study is linked to an extensive overview of reviews in accordance with the Cochrane Guidelines; moreover, the review protocol was registered in the PROSPERO database.

Limitations of this work need to be acknowledged. The research was conducted for only English-language publications, and the children population was excluded from the sample. We did not extract data from primary studies focused on VR rehabilitation, but we reported the trend of publications of those included in the SRs. Ongoing SRs were searched on PROSPERO and not on other registries. Nevertheless, PROSPERO seems to be the most common database used for protocol registration [71.3%, n = 270 (26)]. Thus, we cannot be sure that all SRs included the whole body of evidence.

Conclusion

The highest synthesis of evidence as published SRs, including those ongoing on VR, has risen over time in terms of the number of publications, sample size, and eligible outcomes, with some concerns about methodological quality and representation of countries around the world. To avoid waste of research, authors should check protocol registries before embarking on a new systematic review. As well they should consider planning innovative research methods for synthetizing the amount of literature available.