Bring Your Own Device ePRO: Hold the relish, or no holds barred?

• Sponsored by:
• 

There is a drive to design more patient-centric trials that make study participation more engaging and more convenient. One approach is to leverage patients’ own devices to enable the collection of patient-reported outcomes (PRO) data (“Bring Your Own Device” – BYOD). This may be more convenient for the patient as it eliminates the burden of carrying and maintaining a second device for the duration of the study, and may have other advantages due to increased familiarity compared to a provisioned device.

There is a drive to design more patient-centric trials that make study participation more engaging and more convenient. One approach is to leverage patients’ own devices to enable the collection of patient-reported outcomes (PRO) data (“Bring Your Own Device” – BYOD). This may be more convenient for the patient as it eliminates the burden of carrying and maintaining a second device for the duration of the study, and may have other advantages due to increased familiarity compared to a provisioned device. In addition, because the ePRO data will be collected on the patient’s primary device, it is thought that this may improve ePRO compliance rates. Because alarms/reminders and the app itself will appear on the device predominantly used by the patient throughout the day, this may lead to a reduced number of missed entries. In addition, Sponsors may benefit from reduced costs and simplified trial logistics that may be associated with the BYOD approach.

Despite these benefits, however, our industry has been slow to adopt BYOD for ePRO in phase 2 and 3 trials. The primary scientific/regulatory reason for this is a concern regarding the measurement properties of instruments when migrated to devices of varying makes, models and screen sizes. When using unknown devices, can we be sure that differences in the display and appearance of the instrument will not affect the way patients interpret and answer the instrument’s questions?

In this article we look at why these concerns about mea­surement equivalence are important, and we summarise the results of a recently published study that was designed to evaluate the impact of BYOD on measurement properties of patient-reported outcome measures (PROMs). We also explore considerations for BYOD with other patient devices such as wearables and sensors.

The importance of conserving measurement equivalence

The FDA have provided comprehensive guidance on the development of PROMs[1]. Much work is performed during instrument development to ensure that the questions are well understood and that they measure the concepts of in­terest they are intended to measure. Aside from a few recent examples, this psychometric work is normally performed during the development of a PROM designed to be complet­ed on paper. When migrating a PROM to a different format, for example to appear on a smartphone or tablet screen, it is important to ensure that the way a patient understands, interprets and responds to the PROM items is unaffected by the change in format. Typically, a change in format may be seen in minor wording changes (e.g. replacing “please tick or circle the response” with “please select the response”); and layout changes in terms of font size, spacing and the points at which text wraps between lines, and the number of questions displayed together on a single screen. In 2009, the ISPOR ePRO Good Research Practices Task Force pub­lished recommendations on the evidence needed to support measurement equivalence when migrating from paper to electronic formats[2]. This task force recommended that such minor changes to an instrument due to migration should require a cognitive interview and usability study in the target patient population to demonstrate measurement equivalence. Neither “target population” nor the exact requirements for a cognitive interview have been adequately defined and hence there are various approaches to this process. However, be­cause of the importance of maintaining PROM measurement properties, these recommendations have been largely adopted by the industry.

While it’s relatively straight forward to conduct a cognitive interview study when provisioning a standard handset to all patients, in a BYOD approach it simply isn’t possible to test the conservation of measurement properties on all possible device sizes, shapes and platforms that patient’s may use in the conduct of a clinical trial. Some aspects of ePRO design are easy for an app to maintain, independent of device. These include, for example, maintaining equal size of the real-estate taken up by each response option for a numeric response scale (NRS) or verbal response scale (VRS) item; or consistent use of normal, bold, italicised or underlined text with the paper original. However, other properties are less easy to keep consistent across different device sizes – such as the point at which text wraps to a new line, or whether a user needs to scroll the display to see all the text. Without evidence that such differences do not affect the measurement properties of well established PROMs, it is easy to take a conservative view and elect to use a provisioned device approach.

This level of rigour is not shared by the implementation of other eClinical technologies. EDC systems, for example, are applied on a variety of site hardware. However, the funda­mental difference is that (a) we are typically not collecting source data using EDC – more commonly sites transcribe data from handwritten worksheets, and (b) there is less subjective assessment in the recordings entered into EDC, and where this does exist study training ensures that site personnel have a common understanding of item meanings.

BYOD equivalence

Until recently, no comprehensive study had been conducted to explore the impact of BYOD on the measurement properties of a PROM. Without that evidence, it has been difficult to de­finitively dismiss the potential criticism that changes due to BYOD may alter the measurement properties of an instrument.

For that reason, we recently conducted and reported a quantitative study to explore the equivalence of PROMs con­ducted on paper, using a provisioned device and using the pa­tient’s own device (BYOD). This study was recently published in the Value in Health journal[3], and is summarised below.

The study recruited 155 patients suffering from a condition resulting in some form of chronic pain. Patients were aged between 19 and 69, with 32 in the over-sixty age group. To enable generalizability of our results, we selected a PROM composed of the common response scale types, specifically:

• Verbal response scales (VRS). These comprise a question prompt and an associated list of response options, ordered in a logical scale order, for example: mild, moderate, severe.

• Numeric response scales (NRS). These scales combine question text with a horizontal list of numbers reflecting the degree of association with the construct measured, such as pain severity. Meaning is normally provided for certain numeric scores, for example 0 may represent “no pain” and 10 may represent “worst possible pain”.
• Visual analogue scales (VAS). Visual analogue scales use a straight horizontal line on which the respondent marks their assessment of a specific construct. The scale interpretation is typically anchored using a text description to describe each end of the horizontal line, for example “no pain” to “worst possible pain”.

We analysed each instrument item separately, as opposed to looking only at the instrument total or sub scores, to enable us to generalise our findings to other instruments containing different questions but using the same response scale types.

The study was a three-way crossover design – with the order of the three administrations randomized (See figure 1).

Figure 1. BYOD Equivalence Study Design

Our statistical analysis showed very strong association in the responses to each question collected. In equivalence studies it’s common to look for an Intraclass Correlation Coefficient (ICC) exceeding 0.7 as an equivalence acceptance criterion. In our study, the ICCs of each PROM item in the overall test exceeded 0.816, with the lower bound of the 95% confidence exceeding 0.77 in all cases.

Our study also reported patient attitudes towards BYOD and experience in downloading and using the study app, which can be read in the journal article[3]. Our results provide strong evidence supporting the use of BYOD for PROM collec­tion in terms of the conservation of instrument measurement equivalence across the most widely used response scale types. This is an important piece of evidence supporting the use of BYOD to collect ePRO data in regulatory clinical trials.

Extending BYOD beyond ePRO

There are other opportunities for a BYOD approach in clin­ical trials and post-marketing studies. Mobile devices, for example, also offer a unique way to measure objective health outcomes in large populations by leveraging their inbuilt sensors and components. We see this potential in the growing number of applications launched using Apple ResearchKit that include performance tests and passive monitoring to collect objective outcomes longitudinally[4].

In addition, consumer wearable platforms often provide the capability to share personal health data. Fitbit (San Francisco, CA) and other consumer activity trackers, for example, enable app developers to access activity data with the user’s permission. In such cases, a study app may request permission to access wearable data and this authorisation, when granted by the user, is managed through the web service connection to the wearable device platform. This provides the opportunity to collect additional outcomes data using wearable devices owned by the patient in clinical trials and post-marketing studies.

Do we face similar concerns over measurement equivalence when using wearables and sensors provided by the patient? Undoubtedly, yes – especially in Phase 2 and 3 trials. However, additional knowledge though robust research may enable us to see ways forward in these areas also.

Let’s consider measures of activity collected using different consumer wearable devices as an example. Essentially most devices are constructed using similar sensor sets. Larger dif­ferences occur in the algorithms applied to the data to provide outcome measures to the user, such as the number of steps. Where devices permit access to the raw acceleration data, it becomes possible to apply common algorithms to interpret the data independent of the collecting device. This approach is being applied within open cloud analytics platforms such as the MOVEeCloud platform being developed by researchers at Newcastle University.

Conclusions

Our study extends the growing body of evidence supporting the use of BYOD to collect ePRO data, and is the industry’s first comprehensive formal assessment of the equivalence of BYOD compared to paper and standard provisioned devices.

Because we explored the measurement equivalence of in­dividual response scale types as opposed to total instrument scores, we conclude that our results are more widely gener­alizable to the use of BYOD for other instruments. PROMs are constructed from a number of instrument items, each composed of common response scale types such as VRS, NRS and VAS. Our study has uniquely demonstrated the measure­ment equivalence of these commonly used response scale types between administrations on paper, and ePRO using both BYOD and a common device provided to the patient at site.

We propose that researchers and regulatory agencies might make reference to this evidence as they consider the suitabil­ity of adopting more patient-centric approaches to clinical trials – such as enabling the patient to use their own mobile device to collect ePRO data.

At ICON, we continue to be committed to disseminating the growing body of supportive evidence, and conducting new research, to facilitate the adoption of new patient-cen­tric approaches such as BYOD. In the case of BYOD ePRO, there is now less need to hold the relish as we hope soon to see research teams apply the approach with no holds barred.

References

[1] Food and Drug Administration. Guidance for Industry: Patient-Reported Outcome Measures – Use in Medical Product Development to Support Labeling Claims, 2009. Available from: http://www.fda.gov/downloads/Drugs/.../Guidances/UCM193282.pdf.

[2] Coons SJ, Gwaltney CJ, Hays RD et al. Recommendations On Evidence Needed To Support Measurement Equivalence Between Electronic And Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR ePRO Good Research Practices Task Force Report. Value in Health 2009; 12, 419-429.

[3] Byrom B, Doll H, Muehlhausen W et al. Measurement Equivalence of Patient-Reported Outcome Measure Response Scale Types Collected Using Bring Your Own Device Compared to Paper and a Provisioned Device: Results of a Randomized Equivalence Trial. Value in health (in press). Available from: http://www.valueinhealthjournal.com/article/S1098-3015(17)33615-X/pdf.

[4] Byrom B, Simpson L, Lee J et al. Applicability of Apple ResearchKit to deliver cognitive testing in clinical trials: results of a pilot study. ISPOR 22nd Annual International Meeting, Boston, MA. May 2016. Available from: https://www.ispor.org/research_pdfs/55/pdffiles/PRM22.pdf.

Bill Byrom, Senior Director Of Product Innovation, Icon Clinical Research, Marlow, UK.
Bill serves as a Senior Director within Product Innova­tion at ICON, focusing on harnessing new technologies for clinical trials. He has worked in the Pharma industry for over 25 years and is the author of 60+ publications and an industry textbook on electronic Patient Reported Outcomes. Bill contributes to the C-PATH ePRO Consor­tium and the eClinical Forum. He holds a PhD in disease simulation.

Willie Muehlhausen, Head Of Innovation, Icon Clinical Research, Dublin, Ireland.
Willie has been involved in clinical research for more than 20 years and has held positions in Data Management, Clinical Research Operations and Project Management, Business Development and Product Management and Innovation. During his career he has specialised in devel­oping new products and services for CROs and technology providers. He has represented the ePRO vendors as the inaugural Vice Chair of the ePRO consortium and over­seen the development of important guidelines regarding the development and migration of questionnaires from paper to electronic formats.