Part 1 — 20 Years in Healthcare Analytics & Data Warehousing: What Did We Learn? What’s the Future?


Hi everyone. Thanks for joining us today. This is a pleasure for me. As I get older and grayer, which by the way,
my picture’s outdated. I’m considerably grayer than that now, Chris. We need to fix that, or maybe not. It’s a nice picture. It really is a reflective point in my career. I can’t thank Lee and Shakeeb enough for their
participation. Both of them had to, they inherited and had
to deal with the problems that I created leading the data warehousing effort at Intermountain
and at Northwestern, so it’s great to have them here. This is a very sincere look back on not only
my career, but the decisions that I made, how they affected Lee and Shakeeb, and how
can we apply those lessons learned going forward? This is as much for me as it is anyone else,
and I thank everyone for joining us. Hopefully, you’ll find it useful as well. We’re going to break this into a couple of
parts here. We’ll go through the traditional people processes
and technology and ask ourselves in each of those, “What did we do right? What did we do wrong? What’s the future in each of those?” Again, I want to thank Lee and Shakeeb, and
want to emphasize too that we want this to be a fireside chat and a conversation, so
Lee and Shakeeb, be sure to interrupt me and prompt me with comments, and I will do the
same as you know well. I will interrupt you guys too with thoughts
and comments as we progress. All right, so let’s jump into this. Generally speaking, I learned what was right
by first doing what was wrong in life in general, but certainly in analytics and data warehousing. The nice thing is, I occasionally got to watch
others do what was wrong and learn from them without suffering their wounds as mine. In part, that’s what this all about too, talking
to Lee and Shakeeb about how things evolved at Northwestern and Intermountain. I’m sure they’re learned a lot of things in
the time that I’ve been away from both organizations that I can learn from, so this is really an
important part of the conversation today. Okay, so jumping into things, the people side
of things. Here are my thoughts. By the way, I want to emphasize, we purposely
did not rehearse this. I don’t know what Lee and Shakeeb are going
to talk about. We didn’t want to coordinate or bias the comments,
so it’ll be interesting to see what we agree and disagree on. I have no idea what to expect, so it’ll be
kind of fun. On the people side of things, here are the
things that I felt like we did right. In retrospect, I think we did a good job of
balancing and hiring for a combination of social skills, domain skills and technical
skills. Anytime I would interview and hire and try
to build out my team at the individual level as well as the macro level, I interviewed
around those three axes. The social skills were really important because
the nature of data being what it is, it can be a very sensitive topic, it could be very,
you can create defensive conversations if you don’t have good social skills, so that
social side of the technical personality is not always that easy to find, right? The traditional stereotype of IT, analytic
people is that we are not always great at the social parts of interactions, and I fault
myself with that as well. The other thing that I think we did well is
that we hired for domain knowledge. I think it’s really important in healthcare
that you can’t just be a technologist. You have to understand the domain that you’re
operating within. On occasion, I made mistakes I’ll talk about
later, trying to hire an intellectually capable person in analytics and force-feeding them
into healthcare domain, and they had a hard time picking it up. They had a hard time with the social skills. Likewise, I had a hard time, on occasion,
hiring nurses and clinicians, administrators, and teaching them SQL. They didn’t always have a great technical
background. On to the next bullet. I think we did a good job of balancing centralized
and decentralized development of these skills. I generally had a 60/40 split in mind, and
that was I wanted 60 percent of the human resources associated with the analytic platforms
in a centralized budget and a centralized team. I also wanted 40 percent of that distributed
and funded by the operating units in the organization, knowing that we always had top-down priorities,
that’s the central funding model that I encourage with that 60/40 split. The innovation and ideas and business success
start from the ground up in most cases, and so enabling 40 percent of the skills and also
the funding to come from the ground up was really important and worked out pretty well
for us. What could I have done differently? I overlooked the cultural issues of data when
I first came into healthcare. I came in from the military and national intelligence
communities, which have their own sensitivities around data, but I underappreciated how sensitive
the topic of data was in the healthcare setting. What a lot of people don’t appreciate is that
at Intermountain, the enterprise data warehouse is quite threatening at first because it stepped
on the legacy source systems teams quite often. They saw the data warehouse as a threat. You have to realize this was in 1997 when
data warehouses were first emerging, especially in healthcare. They thought we were trying to replace the
source systems, quite often. I would quite often communicate to them that
we don’t exist without those source systems. “We’re not trying to replace you,” but it
took a while for them to understand that. My personality probably didn’t help assuage
their fears. I can be sort of aggressive and a bull in
a china closet sometimes. Lee is laughing about that, I’m sure. The other thing that caught me by surprise
was the difference between Intermountain as an IDN and Northwestern as an academic medical
center. Northwestern recruited me, essentially, to
help build out an analytic capability and finish off the implementation of Epic and
to some degree Cerner. One of the first meetings I had there was
the analytics and data warehousing strategy for the campus. I talked all about clinical variability reduction
and best practices and evidence-based care and reducing variability, all that sort of
thing, which we did at Intermountain. Lou Landsberg, the Dean of the school … By
the way, the audience was dead silent as I presented this grand vision. I could tell it didn’t go over well. Lou Landsberg, who was the Dean at the time,
pulled me aside afterwards and he said, “Dale, you have to understand. All of the staff and the faculty here are
here to essentially practice controlled variability, so what you just told them is you’re taking
them away from the core mission of academic medicine, which is essentially experimentation
and hypothesis testing.” Didn’t take me long to realize that I had
stubbed my toe in a big way, and so we pivoted at that point and focused on the research
needs of the organization, less on clinical operations and clinic variability reduction. I’ll be interested to hear from Shakeeb the
state of affairs in that regard today. I depended too much on matrixed technology
resources, especially DBAs and Sys Admins, who were skilled in transaction databases,
not analytics. It’s a specialty, analytics is a specialty,
so the first thing that I did that was a mistake was rely too much on matrixed resources. In that regard, I would advocate that the
best way to organize any kind of team is to create autonomous teams that have all of the
skills and the resources necessary to execute a mission and then leave them alone. That’s something I learned in the military,
it works very well. Matrixed resources and org charts were a fad
in the ’90s that didn’t work. I brought that with me, and applied it, and
allowed it to happen at Intermountain especially, and it created a lot of problems. Create autonomous teams wherever you can and
realize that analytics is a specialty. You can’t easily convert transaction-based
Sys Admins and DBAs into an analytic environment. I don’t think we have that problem as much
as we had years ago. I think we’re past those biases now, I would
imagine. What are we thinking about the people side
of things in the future? It’s certainly going to be influenced by data
science, so traditional SQL programming will always be around in our lifetime, but the
folks that are involved in SQL programming now need to start building out their data
science skills, their machine learning skills, and building out their non-relational technical
skills in the world of big data. I’m advocating what I call the role of a digitician. If I were still a practicing CIO and chief
data guy in an organization today operationally, I would be creating this person and this skillset
that exists between the physician and the patient. It’s the digitician’s job to make sure that
the digital profile of the patient is constantly updated and maintained in a different way
than the way we’re approaching it today. Today, we only see a patient on an average
of three times per year. That’s just not enough to understand that
patient in a digital sense. That digitician’s job would be to round out
that patient digital profile, even when they are not seeing and being treated in a traditional
encounter. We’ll talk more about the role of digitizing
the patient in some of the technology slides. The other is that from a people perspective,
executives in healthcare still are largely not IT and data savvy. There are some, but I would say two-thirds
to three-fourths of the executives and administrators in healthcare still don’t appreciate the importance
of data and technology to the future of the business. I call it the eight percent folly, which is
there’s a belief that we’ve invested billions of dollars as a country and as organizations
in electronic health records, but if you look at the amount of data that’s collected and
retained in an electronic health record today, it’s at best eight percent of the data that
we need to understand the personalized care of a patient. There is this feeling right now in the industry
that, “We’ve invested in the HRs, that’s all we have to do.” That’s the leadership view of things. That’s why this is in the people category. The reality is, the HR is just the beginning
of the digitization of health care, and it’s actually a quite limited appreciation for
the entire digital view of the patient. That’s the eight percent data phenomenon. Okay. Guys, any reaction to any of that? Shakeeb and Lee? I think Dale, from my perspective … It’s
Shakeeb. I think I definitely agree with a lot of the
points I’m hearing, particularly around balancing a combination of the skills required and really
focusing on … It’s very difficult to understate the importance of the social skills in complementing
the technical skills when you are the person that is facilitating business requirements
from a clinician, and helping them understand the data landscape and what these technical
terms and three letter acronyms mean, to help them understand the impact that data analytics
can have on patient care and clinical, financial operations, so definitely agree with that
point there. Yeah. This is Lee, and I … Dale, the hiring formula
that you outlined has continued over the years at Intermountain. It’s something that I personally have adopted. I have many times referred to those social
skills also as … I’d also add team fit, because it’s ability to work also within a
team, in the data warehouse and analytics teams. Because having the right people that are committed
and get along and can work well with customers, which is all encompassed in those social skills,
I think, is absolutely critical, but that as a hiring formula has definitely continued,
so appreciate the foundation that you laid around these, for sure. Thanks, Lee. What about the … At Intermountain, you might
recall Lee, that we sponsored and we funded SQL programming skills, data analysis skills
in classes, right? To facilitate data literacy and …
Yeah. Shakeeb, I’d be interested to hear your thoughts. Have those classes continued? Is there a need for that anymore? At Intermountain, they have continued. I think there is a need. At one point, I think one of our lessons learned
related to that is we need to be a little more, a little more careful about screening
ahead of time so we get the right people in the class. Because what would happen is we would teach
a SQL class to … I remember a time where we taught it to a bunch of our laboratory
leaders and technicians who were interested and wanted and sign up. They wanted access to data, but as we looked
at their utilization of actually logging into the database and running SQL queries, it was
in the end a small percent actually used those skills that were taught. For that small percentage, it was absolutely
critical to their success and being able to continue to be, get the data that they needed. The classes continue. I think we’ve refined the criteria for who
attends though. Yeah. Shakeeb? Yeah, from a Northwestern perspective, I think
we certainly have those classes. We kind of, we just went through this entire
exercise of actually identifying different tiers of users and security as well as training
associated with the tier of user that you may be in your experience with data. Ranging from you just need access to queues
and tabular models, so you have a self-service analytics access and that’s it, you don’t
have direct access to the database. Secondarily, we have power users that have
access, SQL access to the database. For them, we’re in the process of reformatting
our power user program and putting some things that Lee talked about. We do have an onboarding SQL exam already,
but we’re tightening the process by introducing some other mechanisms by which we can monitor
their access. Some of those are monitoring their usage,
so if you have not accessed the database for 90 days, your access gets revoked. If you don’t renew your certification on a
yearly basis by taking a class, then your access gets revoked and you have to reapply. I think those are having a large impact. From a power user perspective, I think, one
of the things that we’ve noticed is that classes are definitely very, very helpful. It’s not just the SQL skill per se, but it’s
also traversing the large amount of data that we have at EDW, and knowing which data structures
and tables to use for a particular thing. If you think about Cerner data, seven medication
tables and which one do you pick? Interesting. One of the things we just did is we … This
is going a little away from people and more into kind of technology, but we implemented
a metadata explorer tool that really just lets them analyze the data that we have in
our integrated data structures and provides them some sort of description and data lineage. It helps them with that. Those things paired have kind of helped the
adoption of the EDW for analytics purposes. Great. Awesome, friends. I’m realizing we could spend an hour on each
slide, couldn’t we? Okay, we’ll move on to the other 30 slides
we have. Thank you guys, very interesting. Okay, so this is what I would advocate to
the executives, right? This is the cartoon that I’ve toted around
with me for a number of years. I would say that this has to be the strategic
data acquisition roadmap for every CEO in healthcare. Right now, we’re still operating in the lower
left for the most part. We’re starting to see some progress in genomics
data. Geisinger just made a great announcement in
that regard. If you’re going to think about data as a strategic
corporate asset, you’d lay this down on a timeline and you’d say, “This is the resources. This is when we’re going to acquire this data,
this is how we’re going to apply it back to healthcare.” That’s my cartoon for the day. All right. Lee, back to you, friend. All right. Well, let me go ahead and dive in. What did we do right? Well, we touched on the hiring formula already,
but the fact is we hired some amazing data professionals, dedicated, smart committed,
and using the hiring formula that Dale outlined. We underemphasize the importance of the people
part of this, people process and technology. Each of those aspects are critical, but if
you don’t get the people right, it’s going to be a challenge. Then what I would add also was that it started
from the early days. I believe we hired the right people and then
treated them right. I think that has to continue. Culture of a team and keeping committed to
what is right and working towards those things is so very important. Another item, I think, related to people is
early on at Intermountain, Dale and Steve Barlow and others that were part of the early,
early data analytic efforts they embraced the early support from business and clinical
leaders that really allowed, I think, us to start early. Some of those leaders are like Dr. Brent James
and Dr. Homer Warner, who laid a foundation for not just collection of meaningful data
but also use of that data, and how does it make a difference in the business? I guess I would add to this my personal journey
as it began as one of the early team members. The influence that Dale personally had on
me is something that I can never thank him enough for. I would probably, Dale, be staring in the
mouth of a patient right now as a practicing dentist if you hadn’t had that discussion
with me around, “There’s really a future in this healthcare analytics stuff.” Not that there’s anything wrong with dentist. That’s a good profession too. Yeah, true. For me personally, man, I love coming to work
every day doing healthcare analytics. At the same … That was not the right thing,
and you had a wonderful influence on me that day in your office, so thank you. Well, you’ve excelled at it friend, you’ve
excelled. Thank you. I think we also partnered with and empowered
the data analysts from the beginning. I think that’s something we did right, to
be the primary producers of analytics. Those are the power users that Shakeeb referred
to. We really enabled them and I think did a very
good job at making sure that they are able to get their jobs done and the training was
part of that. We also engaged business leaders in data and
analytics strategy and execution. This came a few years later where we had proven
the value, I believe, of data and analytics, of the data warehouse, and had some really
good successes. As we grew, the capability and the people,
we really needed more input from business and clinical leaders within the organization. The way that we did that is forming appropriate
governance committees. What Gartner had referred to as a BI competency
center, where one of those functions is to get business leadership involved in caring
about data and use of data and analytics and use of those insights that are generated. I think that’s something that we did right
that really helped us continue to grow and develop and mature the capability. Next is what would we do different? If you could advance to the next bullet points. Yep. Let’s see here friend. There we go. All right. Lee, not to, I don’t want to bother you too
much, friend. Your mic is dropping out every once in a while. Is it? Okay. Yeah. I don’t know if there’s anything you can adjust. You’re still audible and intelligible, friends. It’s just a little distracting, but we can
still proceed. Is that a compliment? Intelligible? Intelligible, yes. Lee is very intelligible and intelligent. That’s right, nice compliment there. Yes. There you go. How about I finish this slide and then see
if I can make some adjustments? I’m not sure what it is. That sounds good, yeah. What would we have done differently as I think
about people? I think we would have had a more thoughtful
and coordinated growth strategy related to hiring data analysts across the business. I think we, what we have done later is related
to standardizing job descriptions and trying to really have our arms around not just the
centralized, Dale, what you referred to, but the decentralized teams. A lot of effort in these last five, six years
has been put into better understanding and helping the analyst community. I think we would have been more thoughtful
around the growth, maybe smarter growth in helping the business understand, “Where do
we really need to add resources, and where don’t we need to do that?” Then I think we could have taken more bold
steps to organize analytics as a centrally managed and locally deployed, and this, we
did good, did great work related to centralizing the building of the enterprise data warehouse,
but I think related to the data analysts, that 40 percent of the business, I really
think that a model of centrally managed, locally deployed could have avoided a lot of headaches
that we had to deal with. I wish we would have been a little more bolder
leader in organizing that way. Would the split, would it have been different,
Lee? Would you have said maybe 80 percent distributed,
20 percent centralized, that extreme? When I say centralized, again it’s, I think
of it more as a … The importance of still having them locally deployed is critical to
be able to keep them within the context of the business and clinical units that they
support. I actually think that you could even have
a 100 percent centrally managed, meaning budgets and other, and best practices and such, but
make sure that you continue to have locally deployed, sitting with the business units
that they support. I see. The thought being because wherever the funding
is, that’s where the priorities are going to be and it just becomes difficult to manage
if you keep … Anyway, I don’t know exactly what that looks like, but I really like the
centrally managed, but keeping them in the business locally deployed model, I think,
avoids headaches that grew over time at Intermountain. Interesting. Yeah, I just like hear about that. We view this, or I kind of use the term “embedded
analytics.” It’s really, I’ve seen the concept, Lee, that
you’re talking about. It’s really having centrally managed function
that shows you best practices and how to transform data and make it usable for analytics, but
then locally deployed. I think what’s worked well for us, and we’re
moving more and more in that direction is directly funded resources by the lines of
business that are 100 percent dedicated to their needs. They feel like they have all the control,
and we feel like we have the ability to teach them, and grow them, and provide them the
support from a team perspective, and the training from a technology perspective to continue
their growth. I mean that, for us, it seems like a good
marriage. Yeah. Yeah, I can see that. One of the things that we also, just referring
to the future, and I know we need to keep moving, related to that is who you have those
analytic resources reporting to. Do you have somebody that they’re reporting
to that actually understands the work that they’re doing? Part of the reason for the centrally managed
is if the business units hire their own folks, but they really are reporting to somebody
that doesn’t understand analytics or the work that they should be doing, so there’s a partnership
there somewhere. It’s not a … It’s a pendulum of, from fully
centralized to fully decentralized and, of course, sits in the middle somewhere. There are advantages, I believe, in having
analytic resources formally in an HR relationship, reporting to people that understand the work
that they do. That avoids some additional headaches. Yeah, completely agree. Yeah. I’ll admit a problem I think we have at Health
Catalyst in this regard, guys. I think the way we typically engage with clients,
we tend to encourage a more centrally top-down driven analytics strategy. As a consequence, I don’t think we’re delivering
as much value to our clients as we should. Right. You can see it in the utilization of our platform. There’s only a handful of our clients that
are utilizing the platform in the degree that Northwestern and Intermountain do, for example. I think it’s in part, the way we engage, encourages
a central model. Yeah, yeah. There’s definitely a balance to be found there. As far as the future, the thing that stands
out to me most and, Dale, you touched on this, but really improving data literacy for leaders
within an organization. They need to know how to ask for analytics
help, and we need to help them ask the right questions. How do they actually use the insights to improve
decision making and change the processes that they’re trying to impact? Because that’s really where the value is. All of this is just a means to an end, which
should be resulting in improved decision making. I think data literacy or just how do you use
the insights generated and how to ask the right questions, I think is really critical
to the future of improving the value that analytics provides in healthcare. Yeah. Thank you, Lee. Thank you, Lee. Yep. Okay, Shakeeb. Great. From Northwestern’s perspective, I think some
of the things where we did it right is, one of the things that I think that created a
lot of dividends for us is having a small team at the start that was really agile and
consisted of a variety of skillsets. We had people that weren’t necessarily specialists
in one role, but more versatile and had a lot of the skills and could play in each of
those areas. Whether it’s ETL development or database design,
architecture, a business analyst, understanding the source system or business operations. Those, all those things, having a small, nimble
team really started us off on the right path. The EDW here, from a Northwestern perspective,
grew up on the research side of the house, but we did partner with a lot of clinical
folks. To show the value of the EDW quickly, it was
really important to have a team that kind of got what it took to build analytical products
that would show value very quickly. I think a small, nimble team was able to bring
those products to market faster and essentially provide a higher level of customer satisfaction. I think that was one thing that helped us
quite a bit in our infancy. I think the other thing, I think, Dale, you
spoke a little bit about this. I was thinking about this bullet point when
you were talking about autonomous teams. This is exactly what I’m referring to here,
is that we had as part of the EDW growing up, other roles such as data architecture,
but then also ETL development, application development to develop a reporting portal,
which allows you to deploy your ports for consumption. Research analytics, so having analysts hired
that actually extract data out of the EDW and do data quality and provide those PIs
at Northwestern. All of those roles sat and still actually
currently sit within the EDW team. I think what that’s caused us to do is be
very self-sufficient and be able to help our analytics counterparts with providing one
team that can serve all of their needs. I think we’ll see this a little bit later
when we talk about what we would have done differently, and it will be the opposite side
of the coin, the cons associated with that, but definitely has had a lot of value for
us there. Then last but not least, I think one of the
things that has really helped us is, I think Lee and Dale both talked about recruiting
folks within the organization to, when you’re first starting the journey down the data warehousing
and analytics path, people the right people that have a blend of experience with clinical
operations, in healthcare, potentially experience with the EMR and the workflows there, and
some BI background. I think what has worked really well for us,
historically, here is that we’ve hired people that have some sort of clinical background
and, or have been prior application analysts that have dabbled in the reporting world,
whether it be writing ETL for query data out of Cerner, they had some exposure to that,
and brought them over and made them data architects and started training them to think about things
on a much larger scale from a data warehousing perspective. That worked really well for us. Then from a, what we would have done differently,
I think you’ll see the opposite side of this is develop additional roles, so going from
that small, highly skilled team of versatile players to specialized roles. It’s kind of a little bit of victim of our
own success in the sense that EDW grew up on the research side and now we are the data
warehousing platform for FSM, our Feinberg School of Medicine, as well as all clinical
and financial operational analytics for the health system, so it’s a very different type
of tiering and support required to be able to support the health system analytics function. We are trying to specialize those roles into
specific responsibilities across the team in ETL development, data architecture, et
cetera, and so just to help manage the workload there. I think lastly, recruiting clinical and operational
SMEs to assist with understanding of data. I think what we’ve found is that having folks
that are technically skilled or having people that are experienced with the EMR do provide
value, but then having, I think, one of the things that we learned is having somebody
in your data warehousing and analytics shop that really understands healthcare data, right? Like DRG codes and IC-9s and 10s and how they’re
used, and claims data and those types of things that really can accelerate the development
times as well as help you put in controls like standardized vocabularies and those types
of things, which, our perspective is that the EDW should not be making the call for
that but also, data governance is not sexy and it’s hard to get off the ground, and so
having somebody that could bridge the gap would be a huge help. We would have liked to have that earlier. I think last, everything about the future,
we talked about the development of specialized roles and really blurring the lines between
we have EDW and analytics as two separate departments that report to separate leaders
in the health system, and so our challenge is blurring the lines between those resources
to create hybrid resources that can essentially get the work done from an analytics perspective
regardless of what the ask is. If it’s a data mart required or a queue or
a tabular model or a report or a visualization or some other tool, it could perhaps, if we’re
thinking about this role as a BI architect that can traverse multiple areas of expertise
and be able to solve for any one of those solutions. That’s kind of what we think about for the
future and I think also from a people perspective, we’re thinking about how to, always thinking
about how to more fully engage the business and meet their needs. I’ll talk a little bit about that in, related
to data ops and our processes that we’re putting in place to make sure that we’re listening
to the customer and focused on customer satisfaction with analytics products as a measure. How many, probably a tough question to answer
precisely, friend, but how many people do you have on the EDW team now at Northwestern? The EDW team, I actually just gave a presentation
on this this morning, so it’s very clear. We have 28 folks on the EDW team. I see, interesting. That’s still, that feels like a healthy number
actually. It’s not too small, not too many for the size
of Northwestern. How many monthly active users do you have? That’s a little bit of a harder question to
answer. I think we have about 2,000 users who are
active on our portal as a gauge of that, so about 2,000 folks. I think last time we looked at the numbers,
about 10 percent of the employee population. That’s a great metric. 10 percent, that’s a great metric for the
audience to take away and use as a goal. Right. Including some of our clients. We should tweet that actually, guys. Lee, what are your answers to that, friend? How many, when you left Intermountain, which
was just a few months ago, how many people on the EDW team? How many monthly average users? The organization related to data and analytics,
we kind of rebranded. It no longer was just the EDW data architects
and resources. Across our data and analytics services, which
is what our teams became, it was a combination of the EDW resources, traditionally with data
architects, the data governance, the function, the sum of the terminology services and experts,
and then the BI developers and starting to organize the data analysts as that centralized
function. When I left, the total team was about between
70 and 80 individuals. As far as data architects, today, it was between,
it was right around 30 on the data architects, and then broken down in different categories
from there. As far as … Man, I haven’t looked at those
stats around monthly users, but certainly the goal is … We used to count it specifically
related to the direct connections to the data warehouse, that’s what we used to call a user,
but really the analytics have generated were solutions. I think that 10 percent, I think is probably
good. You figure Intermountain Healthcare, 39,000
employees and 10 percent of that on any given month that might be utilizing the analytics
solutions that are generated as out of the data and analytics services function, I think
is probably appropriate. That’s really good metrics, guys. I want to emphasize to the audience that if
you have to a data warehouse right now, it’s what amounts to an internal small business
because analytics is still somewhat voluntary. If you’re running a small business, you think
about market penetration. I think a 10 percent goal for market penetration,
10 percent of your total population of employees is a good goal. I hope that we’re listening, Health Catalyst
is listening to this and I hope some of our clients. We need to get to that 10 percent goal for
some of our clients, so great, thank you guys for unwittingly furthering one of my agenda
items. Okay. I can tell right now, I don’t know, there’s
no way we’re going to get through all these slides. We’re going to go over. Can you guys go past the top of the hour? I can, yes. Yeah, I’m just taking a look at my calendar. I’ve got a hard one. Got a hard stop? Okay, friend. Well, we’ll do our best and who knows? Maybe we have to have version two of this
to get through these. Depends on how well the audience appreciates
what we’ve gone through so far. Okay, here we go. Processes now. What do we do right around processes? Design and code reviews were a critical part
of what we started in Health Catalyst. I hope Intermountain and Northwestern, I hope
they’re still going on. There was an article recently about a mistake
made in the mammography screening algorithms in the national health system of the UK. About 450,000 patients failed to get proper
mammography screening because the analytics algorithm had an error in it, so there’s real
patient safety issues associated with analytics that we don’t always appreciate. Part of what we were trying to encourage with
the design and code reviews was reliability around safety and just accuracy of what we’re
doing. Lightweight data governance, my philosophy
on this is govern to the least extent necessary for the greatest common good. I see way, I see two extremes of data governance
in the industry, too little and too much. What I see more often now is too much. There’s a reason that not every case goes
to the Supreme Court in our justice system. Create what amounts to a Supreme Court of
data governance, but then try to implement the principles and the values of data governance
at a very distributed level. One of the creative things that we did when
I moved from Intermountain to Northwestern was the utilization of EHR logs to analyze
and understand workflow. It was really the first generation of meaningful
use. We wanted to understand how well the EHRs
are being used but we also wanted to understand timing and throughput of clinical workflow. Shakeeb, I don’t know if that’s still going
on or not, but that’s actually one thing that we don’t do routinely in Health Catalyst that
I want to encourage our teams and our clients to start leveraging, the platform to understand
workflow from the EHR logs. Then I mentioned… Yeah, go ahead, Shakeeb. I was going to say, I don’t know … Yeah,
I think that’s a great idea. We’re not doing that now, but we’re looking
to get access directly into the EHR so we can channel workflows ourselves. Yeah, we’re doing it in a roundabout way. Interesting, friend. Thanks. We did it at the time because both Cerner
and Epic were being underutilized and we wanted to show with data how they should be utilized. Thanks went to David Liebovitz and Gary Martin
for the help that they played in that. I talked about running the data warehouses,
small business. You got to do that, you have to have a small
business mentality. It’s not like in the HR. You can’t force the utilization of a data
warehouse on a culture. You have to make it encouraging, motivating
and inspiring. What would we have done differently? Me, prioritization management, I always believed
that we could put some sort of governance process in that would formalize that and make,
turn it into a science. The reality is, you’re always going to have
greater demand than you have capacity, and trying to formalize that too much just creates
barriers to adoption. Just be realistic about it. You’re always going to be a little less than
what you’d rather be. I would have managed the expectations around
data and analytic quality validation a little better. I’m comfortable, my personality is comfortable
revealing early versions of just about anything as a point of departure for improvement and
conversation, but when you reveal data too early to clinicians and researchers, you can
set the agenda back if you’re not careful about managing expectations. I still am a believer in revealing data early
as a means to increase its quality, but do so very carefully and manage expectations
and personalities very carefully. The future, AI, data science is going to change
almost everything in terms of processes including design reviews. It’s not going to be about data governance
anymore, it’s going to be about algorithm and model governance, and it’s going to make
analytic validation very challenging. In-particular to my IT colleagues in the audience,
there’s a different notion of dev ops when it comes AI, and so you need to start studying
what it means to apply dev ops concepts to AI machine learning algorithms. It’s very different. Okay. We’re breezing along here. Any major comments on any of that, guys? None? No. Okay. Sounds good. All right, so here’s my future of diagnosis
and treatment, speaking of things that are going to be different. It’ll be enabled by bio-integrated sensors,
so patients are going to hold more data about themselves in the future than the healthcare
system, so you’re going to see a shift towards the patient in terms of the data power. It’s going to move away from the healthcare
system. That data’s going to constantly be updated
and uploaded to Cloud-based AI algorithms. Those algorithms will diagnose the patient’s
condition, calculate a composite health risk score and recommend options for treatment
or maintaining health. The algorithm will also suggest options for
best fit care provider based upon quality of care, volume of care, that kind of thing,
and it will provide the ability to socially interact with other patients like them, so
they’re now extended members of the care team. The patient will go into the care provider
enabled with this conversation of AI algorithms. There’s my second cartoon for the day showing
this highly complicated system in four arrows. It’s not that far away, to be honest, friends. We had versions of this at Intermountain around
the HELP system. What will be different is the precision of
the conversation, obviously, but it’s going to happen. Shakeeb and Lee, any thoughts there, friends? I couldn’t agree more. That’s one of my future bullet points is related
to this very thing. It’s actually happening in some areas so I
agree, Dale, that it’s not that far off. It is the future and I think it needs to become
more of a standard way that data and analytics is used. Yeah. Yeah, I agree too. I think particularly around the utilization
of sensors and patients holding more data about themselves. I think that’s going to be the trend that
we’re going to be seeing as well. Yeah. I would encourage the audience, and Shakeeb
and Lee, if you don’t know, John Rogers at Northwestern is at the leading edge of the
bio-integrated sensors world. If you Google John Rogers, we should probably
tweet that too, John Rogers at Northwestern is giving us a glimpse into the near future
of these bio-integrated sensors and patients really becoming the center of their own data
universe. Okay. Lee, processes. All right. We, I think at Intermountain, process, something
that we did right is had a vision from the beginning around the use of data. It was specific to clinical quality improvement. We had a reason to build what we built. I think also, we focused on real business
and clinical use cases and needs. It wasn’t just an IT project to stand something
up, even though the very early days we had started to show from a technology perspective
that it could be done, but as we went on, it didn’t become, the project itself did not
become, “Well, let’s build a data warehouse.” It was about, “How do we support real business
needs?” I think that was a win, and something that
should continue to be repeated within healthcare. I think we had the discipline at Intermountain
in many use cases, not across the board, and to implement the insights to help close the
loop in a slide that I’ll just refer to quickly in just a minute. That’s a big deal. Again, this is if you’re not using the insights
that are generated, why do it? Then as time has gone on, I think we have
done a good job at being practical around data governance. How to be project-based, use case driven and
where the business also sees value. We have, in the last year, a project around
an executive level dashboard with a new CEO that we were able to apply data governance
discipline around, data stewardship, data quality, best practices, data definitions
and where those are stored. It became very, very powerful because it was
applied to that particular project. There have been successes over the years,
but more to come. What are some things that we would have done
differently? I think that asking upfront what we expect
the return on investment to be and how an analytics project is going to be used and
how we’re going to measure that use, I think is very important, then communicate that measurement
and then repeat. I think that they would continue to get more
and more support for our efforts if we did that more, a more formal manner. It takes time, but I think there’s value in
sharing those successes and telling those successes if we’re measuring that in a meaningful
way. Establishing and enforcing better standard
development practices. I would say the reason for this is, in the
end to reduce maintenance required after you develop and roll it out, specifically in the
EDW and ETL development. Every process that you build, any new ETL
job requires maintenance. The team’s done a really good job in recent
years in continuing to develop. The challenge then is enforcing those standards,
so that’s something I think we could have done even better. Then I think we should have been more bold
about data governance, value and investment. It needs to be practical, it needs to be at
the right level, but there is definite value. I think at times we were too patient with
the business as we put forth the request for tools or resources to support data governance. I think we’d be much further along, Intermountain
would, if we would have been a little more bold in moving those efforts forward. The future comes down to the process of closing
the loop. We need more business discipline to use the
insights generated with more focus on the real value that we’re going to achieve. Why do everything that we’re part of, is part
of our professional lives in managing and leveraging data if we can’t actually get those
insights implemented and change decisions? The next slide, just 30 seconds on it. The core of this is something that Jamison
Jones, who is a medical informaticist and just a friend who worked at Intermountain
and then leads clinical analytics at Kaiser, that he put together, and then I continue
to add to it. The core of this is thinking about all of
the data that patient care generates. The integration steps are so critical in order
to then generate the insights. Tools and technologies are advancing and the
generation of insights, but closing the loop, committing to some kind of action, and which
ultimately leads back to patient care. I think more and more it’s going to be the
personal level kind of insights that the action, the person taking action needs to be the patient. That’s really going to lead to improved health. What sits above this is the data and analytics
program governance, and how you train and organize those people. Process is part of that as well as data governance
that has to underpin the primary applications as well as the use of that data. Those are just some additional thoughts related
to process. I like your notion of project-based data governance,
friend. I think that kind of lines up with mine, which
is govern when you need it, not go looking for opportunities, unless they’re compelling. Exactly. You know what? I’m going to make a suggestion, friends. We’re going to run out of time, so …
If you like, my 1:00 was moved, so … Oh. I can push it back if we need to. Okay. Awesome, friend. Well, let’s continue on. I bet we’re still going to run out of time. There’s a long list of questions starting
to pour in. Sorry, audience. This is the danger of being unrehearsed and
spontaneous. That’s just the way it goes. Sorry about that. If there’s demand, we’ll be happy to support
… I’m volunteering you guys now. We’ll be happy to support another one of these
discussions. This is really interesting. Okay, so you’re going to keep going, Dale. I do want to remind those who attended, we
are giving away a couple of HIMSS passes at the very end if you can stick around. We’re going to do that poll when they’re all
done. Not HIMSS passes, Healthcare Analytics Summit
passes, right? Yes. Thank you, that was a slip. Sorry about that. We can give away a HIMSS pass. It won’t do any good. Okay, processes. Shakeeb, what’d you do right? All right, so there’s a quick couple things. I mean I think one is, I think we found ourselves
being unique in a situation across the country as we’ve talked to our peers, is that we have
a single EDW for research and operational analytics, right? That is incredibly unique still, which I find
stunning. Right. We have really been, obviously, the leaders
that have put this together, Dale, yourself and others, really were committed to that,
and we have continued to commit to that, to really be that, be a core part of our strategy. Moving forward, and I think it’s paid dividends,
it’s hard to measure, because all of the data, it’s very difficult to find any other institution
that has all data available for clinical and operationality, also be available to the medical
school and I think that’s a huge win. That is something we’ve definitely done right. It definitely requires a lot more oversight
and management, but I think at the end of the day, it’s worth it. I’m going to skip the second bullet point. Other thing I want to talk about is having
an EDW oversight committee. I don’t have this on the slide, but was thinking
about that as were talking. Really, we’ve had an EDW oversight committee
that is deeply involved in what the EDW is working on to the point of allocating a project
list at the beginning of the year, breaking that up, identifying the level of effort,
have voting take place on which projects make the cut, and then assigning those specific
resources in EDW. I think that’s made us accountable for our
work to the business, and aligned that with business needs. I think that’s definitely the stuff that was
done. Who’s on your oversight committee now, Shakeeb? We have a combination. I think, when we were the faculty foundation
at NMHC and FSM, they were, the oversight committee consisted of three members of each
institution. Since then NMHC and NMFF have merged, so we
have Rex Chisholm who’s our Vice Dean of Scientific Affairs at the University. We have Donald Lloyd-Jones, who runs our NUCATS
program, which is our Clinical and Applied Translational Science Institute. Jay Anderson was on the committee as our Senior
VP of Analytics. We also have Danny Sama, who’s a VP of analytics
as well as physician Gary Martin. I’m sure I’m missing some names. Carl Christensen, our CIO and our Deputy CIAO,
Andrew Winter for FSM. Then, yeah …
Shakeeb, do you also have … Does it also govern the use of data and analytics or is
it just primarily focused on the EDW as you referred to it? Yeah, that’s a good question. We dabble in a little bit of both. It’s not the … We do prioritize the work
that analytics will be working on, and then the data architect resources will be working
on specifically, so it is more tilted towards prioritizing the time for the EDW resources,
but it does relate to what analytics projects are also getting completed. Sure. I guess I would, on your previous slide you
had a blurring the lines between the EDW management analytics and at the governance level, something
that I think is an opportunity to do that, to say this is, in the end, this all has to
be kind of governed together in order to really get the value that we want. Yeah, and that’s something we … That’s a
great point. That’s something we’re definitely considering,
with having senior leaders across both institutions kind of work on advanced analytics projects,
so it would be high value. Yeah, mm-hmm (affirmative). Cool. The other thing we’ve also, I think has really
helped, is the data steward approval process that we have. We have a release, a process that audits all
data releases for research purposes, use of clinical data for research purposes. It has to be, we have data stewards assigned
for practically every type of data source you can think of that has to approve that
data release specifically, and then obviously, has to through the IRB and data security plan,
all of that. I think that has been a huge success because
it allows people to get the data that they need, but also applies the right level of
control and security on that data. I think report deployment process, another
one that we’ve been able to leverage our power users as well as develop a very streamlined
process for the deployer portion of the portal for consumption across analytics. I think that’s helped quite a bit in the adoption
of analytics over the years. Something we would have done differently. I mean, I think one of the things I talked
about that, there was that committee planning all of the work, we’re doing that a year in
advance. I think that leaves this waterfall tendency
of project management and as we know, I think, and you guys probably agree, analytics, everything
is very iterative. People think they know what they want, but
they don’t really until they see the product at least once. Really focusing on agile processes and methodologies
is where we’re moving towards as a team, both in the analytics and the EDW. I’ll talk a little bit about that in the data
ops slide that I have next. Fail faster and more often. What we really discovered as we work through
these projects is the faster you can get a product or the data loaded into a design,
conceptual or logical or physical, in front of a customer, the sooner you will find problems
and you’ll be able to self-correct, instead of waiting until you are, you think you are
close to an end-product. Then lastly, bringing the customer to the
table. That’s one of the things that continue to
push the team on and making EWD and analytics resources meet with the customer directly
to get their input on the product and not what we think the product should be for the
business. Then last thing, I think, what do we think
about the future? One of the things that we’ve really been implementing
here is what I call data ops, it’s a term in the industry. It’s really agile product teams that are working
together. Combine skillsets of folks that takes … The
best way to describe it is to take data from source to solution, all skills required for
that are on that product team. They are dedicated to a particular line of
business, whether it be rev cycle, acute care, ambulatory, et cetera, that they take their
direction from. That reduces a lot of the bottleneck that
we see from project prioritization and really makes the customer feel like, “This is my
analytics team, dedicated to meet my needs.” With that, we’re also, we have implemented
a plethora of tools to do source control, get Virtual Studio and then automated. What we’re really working through now is the
dev ops process for automated testing and deployment and release management. We have implemented that. We definitely had some kinks, and it definitely
slowed down some of our work, but we’re in a place where we’re scaled so far and the
team has gotten so large, that we really need to monitor changes in our environment, so
that’s what we’re doing there. Really looking at data management as a holistic
practice. Really focusing on data quality and master
data management. We report on things today, we don’t have automated
data frameworks, testing frameworks to monitor the data quality of the data that’s being
reported out of EDW, and a closed loop process, going back to the application to say there
is data that is inaccurate or not of quality in the application, having that feedback loop
to get it corrected in the source as well. We’re definitely investing in that and trying
to build that robust processes, tools and technologies to help with that. Then I talked about embracing the agile, two
little methods. I know we’re over time so I don’t want to
take too much time here, but this, this slide should actually say “data ops.” I think, it got changed accidentally. This is really the notion of data ops, it’s
really cousin of dev ops that we’re trying to implement here at Northwestern for the
development of data products. It’s really commitment to a few goals. Standardizing the tool sets that we all use
as a team and the processes that we use to develop, test and deploy. Cross-functional teams that break down silos
between the EDW and analytics teams, and really get all the skillsets to work together. Whether it’s architects, data scientists,
consultants, report writers, ETL developers, to meet the customers’ needs. Customer focus is our largest metric for this,
is really … I’m sorry. Customer satisfaction is our largest metric
for this. What this allows us to do is really be customer
focused because you don’t have, we’ve noticed, architects jumping from ambulatory data mart
one day to rev cycle. Those are very different domains and it’s
very difficult to understand both, so we’re really getting people the training they need
to support the line of business that they support, and then embed them in those operational
workflows, which eventually leads to the customer satisfaction we’re looking for. That’s, I like that a lot, Shakeeb. Thanks for sharing that. That’s really cool. Yeah, I like it. I like the notion … I haven’t called it
data ops, but that makes a lot of sense. A cousin of dev ops and …
Yeah. Essentially what we’re trying to enable at
Health Catalyst too is that kind of thing. You know what? Friends, I would like to suggest … Oh, wait. You’ve got a dev ops team. Let’s look at this team here, Shakeeb. The team structure. Sure. Sure, yeah. This is kind of just talking, visually representing
what I was talking through, the data ops team containing all the skills required to transform
data from its raw format to an end-user analytical deliverable, whatever that might be. A report, dashboard, KPI, analytical predictive
application, those types of things. What we tried to do is create product teams,
and we’re very much in our infancy in getting this started, is an Analytics Manager that’s
really the voice of the customer and is managing the team timelines, et cetera, and facilitating
the requirements with this Senior Analytics Associate and the Analytics Associate. The Senior Data Architect is responsible for
enterprise data modeling, design of the data mart, defining the best tool, making sure
the solution is scalable, those types of things, and that responsibility and then managing
the data architects who are doing the ETL development, the data quality checks, and
actually creating and executing on the vision of the team. The goal, the aspiration here is that this
is kind of going for that embedded analytics, deployed analytics that you guys were talking
about is, it’s no longer a team of data warehousing analytics professionals sitting far away from
the businesses. “We’re here, we’re next to you.” We’re adopting agile methodologies to do two-week
sprints that then tell the customer what we’re planning on delivering in the next two weeks. It also makes us very nimble if something
comes up in those two weeks to respond to customer needs, but all-in-all, it makes us
very nimble, agile and be able to respond to customer needs when we need to with all
resources focused on one common goal. Yeah. Remove a lot of them. I like that. That’s back to the autonomous teams. All the research you need to execute a mission. That’s the way people appreciate working no
matter what they’re doing. That’s great, friend. Good to hear that. Yep, agreed. Here’s what I’d like to do, friends. If Lee and Shakeeb, I don’t want to volunteer
you if you can’t do it, but if we did a part two of the webinar, and we talked about the
technology lessons learned, in a week or two? Is that possible? Here’s the challenge to get you all scheduled. If you have time to do a continuation right
now, we’ll make this on-demand available, and we’ll also do a table of contents so people
can quickly get to this, especially those who have to drop now. We could record it and make it available,
so that’s the fastest way to present this information. Okay. We could try and schedule a part two. Yeah, that’s hard, isn’t it? It’s up to you. The problem is we’re half way through the
slides. We did 17 to 31 slides and we have a whole
bunch of questions. We do have someone voting here. Part two’s okay with Don, Don and others. Todd Hockenberry. We’re seeing some folks say part two would
be helpful. Yeah, it’s up to you. Yeah, Dale, I’m okay to do a part two or continue
now, but if part two is what will work best for the audience, that’s okay too. Yeah, I could-
We’re seeing lots of part twos here, guys coming in, so let’s do that. Here’s what I … Let’s do this. Let’s take at least a couple of minutes, ’til
quarter past the hour to answer a couple of questions. We’ll save all these questions, we’ll bring
them into part two. Okay? Does that sound good? Yeah. All right. Thanks for the feedback and the comments here,
friends, on the part two thing, and thank you, Shakeeb and Lee for being agile. Okay, so let’s pick off a couple, let’s pick
off three of the top questions here. If you’ll navigate to that Chris, I will read
them off. This is from Russel Porter. “You noted Cerner and Epic, off-topic but,
maybe. Why do they do such a poor job with EMR documentation
for rehab service, PT, OT, ST? They are a huge drain on staff productivity. Other providers that you … WebPT and ReDoc,
have perfect examples of what a therapy EMR should do.” Well, I couldn’t answer that Russ, but Lee,
given your background in PT, OT, maybe you have an observation there, I don’t know. Yeah. I actually, that is where I started 22 and
a half years ago at Intermountain, and actually have continued to be involved kind of on the
side because they’re still, I believe, an underserved part of healthcare. I’d be happy to talk to you offline, but my
initial reaction is Epic and Cerner and others, it’s PT, OT, speech or just they’re not high
on the priority list. Time just hasn’t been put to it. The tools could be used to effectively create
good documentation for PT, OT, speech, but they’re just, it’s the stepchild of healthcare
disciplines, unfortunately. That hurts me, because I have so many friends
that are part of that discipline. At the same time, I believe that there are
some unique analytics opportunities, that there needs some collaboration in rehabilitation. That’s something that I’d be happy to have
a follow-up discussion on, because there is some great work that’s been done at Intermountain,
and starting to collaborate with others across the country on that topic. Thanks, Lee. Yeah. One other …
I wouldn’t be able to provide any insight into … I was just going to say, I wouldn’t
be able to provide any insight to that one. I was a patient of rehab a couple of years
ago when I blew my knee out in a spectacular Lindsey Vonn style ski wreck. Sorry for that. You don’t remember that? I do remember. I remember that. Yes, but anyway, I was on the down side of
that where her .. She had a great EHR, completely disconnected from the rest of the EHR environment. Frankly, it was the most quantitative treatment
as a patient I’ve ever received in healthcare. Yep. Okay. “How do you differentiate between the digitician
and the informaticist?” Fadi Islim asked. That’s a great question. I think it’s the evolution of the informaticist,
Fadi. We could, I think the digitician is a cooler
name, only because I came up with it, but it’s the evolution of the informaticist. Is it a little different than the traditional
medical informatics degree. You do have a growing set of words that you
own. Yeah, yeah. Too bad I can’t make money on those. Any comments on that, guys? On the digitician, informatics term? This is Lee. I guess the … I would agree with you, Dale. I mean I hadn’t heard the digitician, use
of that word until this webinar, but the evolution of informaticists who are usually people that
are, have the technical skills and domain, clinical expertise, and many times, to be
able to apply that to data and analytics in a more meaningful way where it’s not as much
about just the EMR, I think, is the future. Call them what you want, I think it’s still
an important skillset. As long as it’s not too academic, it’s an
important skillset that can be applied to health data and analytics. Yeah. Okay, one more question and then we’ll wrap
things up and we’ll say thank you. This is from Samuel Ayisi. “Did you have to resort to offshore outsourcing? If so, what impact did it have? Any lessons learned?” At Intermountain, I didn’t do any outsourcing. At Northwestern we did do some around ETL
and it worked out great. At Health Catalyst, we are about to engage
in our first experiment with outsourcing, primarily around ETL, maybe some app development. I’ve had great success with it. There’s plenty of success stories out there,
so I think it can be done quite well. Just an FYI, Latin America is actually a very
good source for outsourcing now where, I think, traditionally we’ve looked to India for a
lot of our outsourcing in the U.S. because of the skills base there. Latin American now and Central America are
becoming great resources too, and the time zone issues aren’t quite as challenging. Shakeeb and Lee? Yeah, so at Intermountain in the last year
and a half, two years, we’ve done a couple of experiments with a partner that uses overseas
resources. Some success and some just the opposite of
success. Really what it came down to, I think, is maturity
of the development processes, because you have to be able to give the overseas teams
very specific requirements for them to go technically do something and deliver it back
to you. With the model that we have traditionally
used, where it’s more of a high touch, working with the customer, having those discussions,
iterating and not necessarily documenting requirements as thoroughly as you would need
to in order to hand it off, that became a challenge with ETL. In the analytics space, again it’s quite high
touch and so, but our motivation really with trying it was to try and find some of the
mundane activities, maintenance activities, migration of ETL jobs when there’s upgrades
or whatever. Things that we want our team members to work
at top of license and let’s use a lower cost resource to do some of those lower level items. In theory, I still think that should be a
success, but you have to take into account that that handoff around requirements, the
amount of work that it takes to document that so that an overseas resource can do the work
effectively, don’t overlook that. It’s not a one-to-one, one ETL developer overseas
to one ETL developer locally. It hasn’t played out that way, at least in
experiments that we have run so far. Yeah, and I think from my perspective, leveraging
those resources, we have very stringent guidelines on being able to use offshore resources, and
so we have not pursued that just because of the large amount of time it takes to get the
approvals from the pairs in order to be able to utilize those resources. Yeah, very interesting. All right, thanks guys. We’ll put a pause, and we’ll come back starting
with technology in part two, and turn things back over to Chris to wrap up. Can’t thank you guys enough, Shakeeb and Lee,
for doing this. Thank you so much.

Leave a Reply

Your email address will not be published. Required fields are marked *