Joel Rizzo is a student programmer in the Center for Modeling and Simulation, humans and synthetics lab with Dr. Majorie Zielke. He is currently working on an Emotive Virtual Patient (EVP) project which is a collaborative research effort between UT Dallas and UT Southwestern to create an augmented-reality patient with natural language processing (NLP) that can be used to train medical students in their discourse and diagnosis procedure with patients. He and Walter (the virtual patient) took a beat to discuss how their research could revolutionize the medical education industry.

Describe your research for a lay audience.

We are trying to put together a training tool for medical students. It’s a virtual simulation where there’s a virtual patient, visible via augmented reality—so, a hologram—before the student. The student is able to verbally interact with the patient.

The student has a physical bed in front of them, but the virtual patient sits on the bed via augmented reality and they can speak to it and it will understand what they are saying via NLP. Then, the student conducts their examination of this virtual patient. At the same time this is all happening, footage of what the student is doing is sent over the internet to an instructor.

The instructor doesn’t have to be in the same building—they can be anywhere—and this instructor gets a live video feed of the student, what they’re doing with the patient; it’s all superimposed. Then the instructor has the opportunity to provide feedback to the student live as well. So the student can actually see in the corner of the frame video of the instructor and the instructor can send messages or can talk via audio and of course they’re visible via camera so it’s sort of like a two-way Skype application except that the focus is all around this virtual patient that they’re supposed to be diagnosing and giving a medical exam to.

We’re developing the entire system; we have a prototype but we’re continuing further development on it at this point. It has a bunch of funding behind it now, so we’re hoping we will be able to turn it into something impressive.

What led you to pursue your research?

It was actually assigned to me; because I work in a research lab and I’m the programmer here. The thing that led me to this lab, specifically, was that I was referred here by a previous employee of the lab. Someone who was a UTD graduate left and became a professor elsewhere. That person ended up being my professor. We kind of got along well, became friends, and he recommended me here. I took the position and I’ve kept it since for over three years now.

The reason for that is because I’m actually somewhat afraid to get into the industry proper and leave the lab, because the environment is so relaxed and enjoyable; and plus, the work is just so fulfilling. As a programmer, I anticipate that work in the industry behaves very much as: I’m assigned a task, and this is what I’ve got to do for eight hours a day; I’m going to churn out code and it’s not going to be a lot of fun. Whereas here, I’m given a vague notion of we want, a piece of medical software, an educational tool that does something. How I make it, under what time frame I make it, and the pieces I decide to use are ultimately up to me. It gives me a lot of freedom and it gives me a lot of right to experiment with different software techniques.

For example, the EVP project I just told you about requires a whole range of very sophisticated things, but we’re not tethered to or tied down to the demands of something specific. I don’t have to use a specific platform. I can make use of recent advances in the industry.

I get to learn new algorithms; I get to make myself a better programmer. So, it’s such a fulfilling role that I never get up in the morning and feel like I’m going to work. So, I don’t want to leave the lab because I feel like if I go anywhere else it’s going to feel exactly like that: work; And I don’t want that to happen.

What are some of the challenges you faced?

It would be more accurate to ask “what are the challenges that have been faced prior in the project,” because I just recently got assigned to it. There have been other people developing it in the past and, granted, we only have a very prototypical system. It works. I can show it to people, but it’s a very distant shadow from what it could be and we’re hoping it becomes.

The challenges with that have primarily been in funding. It’s a very big, complicated system and it takes a lot of core pieces to work and we’ve not had the time nor the money to invest in properly building each piece of the system up as an independent software unit. We’ve cobbled together as quickly and cheaply as possible the pieces that prove the concept, and we did that successfully. The people who came before me that worked on the project did a really good job with that but it’s been an outstanding challenge. If I were to take you and show you the prototype we have, it would be obvious that it’s a very cobbled together thing and that’s what I’m going to try to fix going forward.

Another challenge is in the transfer of learning from the previous people who worked on it before me. I’ve had to reverse engineer a lot of what the people before me have done because everyone thinks differently. I had to learn what they were thinking when they made certain choices. I had to evaluate “OK, why is this like this,” “is this the best way for it to be” (usually yes), and then if there’s parts of it that can be improved I have to figure out what the current solution is and then figure out other ways I can take that to make it better.

There’s an exploratory research element to just figuring out how to proceed in the first place. That’s primarily what my research has been so far. We’ve identified several places in the system that need improvement that are problematic. There’s things we can do about it, but most of my research so far has been focused on: what do we have now, what are the things in the industry, what are the algorithms, standards, and practices that have changed in the past few years that can be applied to this project; and then I have to learn what those are for myself and how to implement them in our specific context.

How do you see your research making an impact in the future?

The impact the research is going to have on the future, I think, is somewhat self-evident from the nature of the technology that it is. You have an over-the-internet system allowing one doctor to teach a future doctor or medical professional how to do something in an examining environment. In a very broad sense, no matter what it is, no matter what the specific application we use for it is – right now the virtual patient is a stroke victim meant specifically for neurology students but that can be adapted to any environment—practically any medical situation where there’s some sort of verbal component, verbal analysis of a patient; it could in theory revolutionize the way that medical training in general is done. You don’t have to have your instructor and student in the same room, and you don’t have to have live patients with real problems to practice on. Now, you just have fake patients with real problems and that gives students a great degree of practice.

Naturally, it’s not meant to be a real replacement for real patients but it’s a very powerful supplement. We’ve had other projects in the lab that were like that as well, where what we’re making is a supplement to the real thing and we’ve shown that it was very practical and successful in the past. So, I expect that if this takes off, and it gets funded, and we get to develop it, it’s the sort of thing that big medical universities could use for training new students much more effectively with much less liability involved. It could revolutionize the entire medical education industry, I would think.

What advice would you give the younger version of yourself?

What I would the say to the younger version of myself is that not everything in life is going to come just from the classes. That there is a lot of extraneous, extracurricular activities that are going to happen. People that I’m going to meet that are going to be much more important in my life than I would’ve ever realized. I have a tendency to underestimate the [importance of] people that I meet and think that it’s just going to be a person that I meet briefly, then they go away. But in truth, my life has been dramatically and radically changed by meeting people just for a short amount of time and things happening as a result.

So, I would tell my younger self to keep more of an open mind about things that are nonacademic.