 | We interviewed Dr. Seitz about his experience with lending devices — ranging from low-cost tablets to VR headsets — to research participants. His research explores cognition and cognitive training using custom-built software. To make participation accessible and inclusive, his lab ships pre-configured devices directly to participants’ homes. We wanted to get a slice of his experience to share with other researchers who may be considering lending devices as part of their study. |
Can you briefly describe your research and how lending devices to participants plays a role in your study?
The fundamentals of my research are how to better understand people’s cognitive needs and to develop interventions that could help them achieve their goals. To address this, we develop software systems that are designed to more comprehensively and reliably measure cognition and also to train cognition. The goal is that our measures will inform how people do complex tasks in their natural environments and that our interventions will transfer to practical benefits that will aid people’s daily activities. Our view is that it is necessary to understanding cognitive processes spanning many brain functions ranging from basic perception, to attention and memory, to decision making processes and motor responses. For example, when understanding vision or hearing, one needs to go beyond just basic clinical tests of eye and ear functioning, to understand the multiple brain processes involved in deconstructing and reconstructing the perceptual scenes that we experience, to derive meaning from these and to make appropriate decisions based upon what we perceive. To accomplish this research, we seek to make our studies accessible so that we can reach many people who have different cognitive needs and to gain large datasets that will help us understand how people are similar and different and to ultimately personalize our systems to each individual. This task would be next to impossible if we required everyone to travel to our lab and thus we strive to reach people where they are by lending-out devices and running most of our studies in people’s homes.
How did you choose the specific devices used in the study? What kind of factors do you consider when selecting a device?
The choice of device depends on the population and the goals of the study. The overall goal is to find devices that are inexpensive, durable and that will provide good data quality. We have benefited from the fact that most consumer devices are sold advertising the ability to watch movies, listen to music and play video-games, all of which require good sound and visual fidelity. In many cases we can use inexpensive tablet computers (often in the $100-200 range), but this depends a lot on the study and target population. For example, while kids and college students are often adept at using devices with relatively smaller screens, for older adults we typically will use more expensive devices with larger screens so that we can accommodate age-related changes in vision. Further, for some studies that we are conducting focused on people with low-vision, we are using VR headsets that have built-in eye-tracking (often $1000+ per device), which allows us to fully control everything that the person sees and make sure to present stimuli at precise locations on their retinas controlling for gaze position. On the other hand for some studies focused on hearing, we’ve found that we can get high-quality data by people downloading our software and running it on their own devices. Each research project often goes through a development cycle where we first validate on a target device, then extend the results to a range of other devices, with the long-term goal that we can make the research available to as many people as possible using a range of devices.
Are there any logistical challenges to navigate in purchasing and configuring devices? What kinds of things are you thinking about when preparing devices to be sent to participants?
Yes, always. These range from basic challenges of university purchasing policies, where some vendors are more difficult to access than others, to ensuring that our software runs properly on the given device, to creating a set of participant facing instructions that provide an easy way for people to get started once they receive their equipment. A challenge is that every company is constantly updating their software, not always for the best, and so we need to keep up with these changes and make sure that they don’t mess-up the study. This often involves turning off location trackers, automatic updates, notifications, and many other software systems that companies install by default on computers. There are additional challenges such as making sure that the devices are connected to the internet (of note, we always set-up our devices so that they can work even when not connected), when using peripherals (such as keyboards, headphones, amplifiers, eye-trackers, physiological monitoring devices, etc) or addressing any form of communication between the multiple devices that might be needed in a given study. For each study we create detailed documentation and train research assistants on how to address the issues that may occur.
What kind of logistical challenges are there to distributing, tracking, and facilitating the return of devices? What approaches have worked and what approaches do you recommend avoiding?
Here one must ensure that there is good communication with the participant and that the process for delivery and return is easy. The trick is to make it as convenient as possible for the participant. At the beginning of the study we explain to the participant what they should expect and have them sign a loaner agreement that informs them of what they will receive, that it is only for use in the study, and that they agree to return the equipment. Then in most cases we send them the equipment through the mail together with instructions on how to set-up and then return the device (including a return packing slip). However, in some cases where the set-up is more complicated, we drop off the equipment, set it up in their homes and pick it up after the study is complete. Generally, we have found that people are very receptive to these approaches and have had fewer breakages and losses of equipment that we’ve loaned out than equipment used in the lab.
What steps do you take to ensure privacy and security of the devices? Do you implement any restrictions of the devices and, if so, how do you think about these or what types of restrictions do you implement?
Again this depends on the study. In some cases we will use kiosk mode, where we can restrict people to only use the apps that we set-up for them. In other cases, we just configure the home-screen so that the only things that they see are the apps that they are supposed to use (typically our software and then zoom) with the rest of the apps deleted or hidden. We then will have the device configuration password protected so that they cannot make any changes on their own. Importantly, our software is pre-installed on the devices, we have it set-up so that they can click an icon that will open zoom and automatically take them to a zoom room that we pre-configured for study and also have set-up HIPAA compliant servers on Amazon Web Services where the data is securely transferred so that we can monitor, and in some cases remotely reconfigure studies remotely.
What IRB considerations or concerns arise related to lending devices and how did you navigate them?
There are a number of issues here:
1). We need to make sure that each of our processes are clearly described so that they can be understood and reviewed by the IRB. While this is true of any study, we took special care to explain how loaning devices increases equity and access, lowers participant burden, and provides more representative results.
2). We needed to create an Equipment Loan Agreement Form, where we consulted with the IRB to ensure that it didn’t provide burden or risk to the participants (e.g. we encourage participants to return the devices without putting them under legal obligation or financial penalty) and also with Office of General Council to ensure that it was compliant with University regulations.
3). We needed to ensure that the devices that we lent people are safe for this use-pattern. For example, if someone is using a device at-home that does electrical brain stimulation, then what efforts are needed to reduce risk of the equipment causing harms? If it is a regulated device is it consider safe for this use pattern? ,etc.
4). Since people are never coming into the lab then we need to make sure that we had a method of paying participants remotely. We typically use gift cards for this, but sometimes will send people checks.
How often are devices not returned and how do you navigate this situation? How do you encourage the return of devices?
We’ve only had a couple cases that devices were not returned. In both cases, these were due to a loss of contact with the individuals that we believe were related to health issues where the participant was no longer in a position to respond to our communications. We find that most people that agree to participate in the studies are well-intentioned and we motivate them to return the devices by explaining that we need the equipment to conduct research that is beneficial and aims to help people with cognitive needs. We also work hard to ensure that we are responsive to our participants’ needs, and, as stated above, that the return process is easy.
If another researcher were considering lending devices, what advice would you give them?
My advice is to carefully think through each part of the process and document it well. This ranges from making sure that you have good packing material both for delivery and return, that there are clear instructions to the participant on everything that is required of them (from receipt and unboxing, to set-up and running the study, to re-boxing and return), and that you have a good communication process so that at each step in the study instructions are given when they are needed and there is an opportunity for participants to ask questions. We will often get on the phone with participants when they receive the package, help them with unboxing and setting up wifi, and then have Zoom pre-configured so that when they start the study that we can share their screen and provide instructions based upon what they are seeing on their screen and the responses that they are making. Documentation is equally important for any research assistant to make sure that they know how to troubleshoot challenges, and whom to go to when there are problems that are more difficult to resolve. The devil is in the details and documentation is your friend.