00:00  The Excitement of AI Coaching & Virtual Assistants

Das Rush: To get started, like, what has made you so excited right now about this space of AI coaching and this potential we have to build, like, true virtual assistants and coaches? 

Jeff Dalton: What's really exciting to me, and I think a lot of people in the field right now, is just the pace of change is just absolutely amazing. Like, there's talk about, like, kind of the exponential growth in terms of the AI capability that we're seeing. So even what we the kind of pace of change that we saw, what we saw three years ago, and what we have now, it's like ten years of change just almost overnight. And so what that means is what we could do before, even a few months ago, suddenly, that was impossible is suddenly now possible. 

And what really makes that exciting is the fact that many of us have had this vision, as you mentioned, for a long time, twenty years, ten years, and some people for a lot longer, some people for fifty or even sixty years of having this AI assistant, that can actually be with us. And suddenly, many of these things are much closer to reality in a way that feels very tangible and exciting. My vision for what we've had for assistants is also the fact that we—assistants matter because they help us accomplish a goal that matters to you or to us. Right? So in coaching, whether that's dealing with a coworker who's struggling, whether it's helping us achieve your next career goal or your next big promotion or your five-year plan for what that looks like: what's a coach that's gonna be, like, help you along that path, right? 

01:27  Applying AI to Build a Virtual Coach

‍Das: Now that you've come over and are working on Valence and building Nadia, kind of what are one or two of the ideas that you worked on earlier that you're finding now are really applicable to building this AI coach? 

‍Jeff: Yeah. Certainly. So I've been around for a little bit. I did my PhD now probably, kind of fifteen years ago, really working at working on intelligent search systems. So how could they know more about us and know more about the world? And we did that using something called knowledge graphs, which were a structured form of a kind of a geeky way of, like, how machine we could encode facts for the machines to be able to understand and do question answering. And I continued that when I went to Google. I worked on health search. And so we take something that says, "I've got gunk in my eye," and we've turned that into something that we can give an answer to, leveraging a knowledge graph. Right? And, hopefully, it make your health a little bit more reliable in the process. 

And what we quickly realized is the fact that what was in the search box, that wasn't enough. We needed what was outside of the search box. We needed an AI assistant that had a plan that could talk to us, that could reach out and have, and then have that. And so I went to work on the Google Assistant, and we tried to build some of those technologies and tools. At the time, the technology is, like, just wasn't there. But the underlying element of having machines understand us, leveraging domain experts to be able to really deeply understand the world, are all fundamental parts of components of, like, the future AI assistance that we're building. 

02:48  Breaking (and Rebuilding) Virtual Assistants

‍Das: You know, you mentioned there, Google, like, your medical assistant. And one of my other favorite assistants that I know you built was a virtual kitchen assistant. Could you tell that story? 

‍Jeff: Yeah. So not too long ago now, around 2021, my research group competed in the Amazon Alexa task bot challenge. And our goal there was to do something that hadn't been done before. The goal was to have something that you could cook along to do something real in your kitchen. So not just talk to your assistant, but actually, see what your system was doing, use a screen, use rich interfaces to be able to do something in the world, and to have a coach there with you along the way. And along the way, we realized that pretty much everything that we had for the current assistant was broken. So we broke the speech, the voice, the real time voice. None of that existed. And so we had to build it, and we had to build a whole new open assistant toolkit. 

What was really a key challenge there that was really transformative for us was we were right on the cusp of the LM transition. So we just hit the first generation GPT three where we could take in and actually transform a recipe. So you're like, "I now live in Colorado, transform this for high altitude," and then that one could make that possible. So really kind of wow moments that really demonstrate just the potential for the assistant to be adaptable at the next level. 

Solving Key Challenges in AI Coaching

‍Das: It's, you know, you mentioned breaking the assistant. Right? Like, having to kinda break every piece of technology to build the assistant. And I think that ties into something that's been kind of a refrain at Valence, which is, you know, AI has a strong central promise of being able to personalize technology to us and to make it far more natural to use. 

At the same time, it's not fully formed, and there's a lot of rough edges that we're smoothing out right now and a lot of problems that need to be solved and worked on. What are some of the most important problems that you and your team right now are working on as you build Nadia? 

‍Jeff: Like any new technology, we still have a lot of rough edges. We're still working out a lot of the details of what the capabilities are, and I can talk to you probably for a whole hour just about the different kind of challenges and aspects of what we're going to need to do to build the coach of the future. So I'm just gonna give you a little bit of a taste for what that looks like and then maybe talk a little bit about how Nadia works today. 

So the first thing is how can we build an AI that you can build a relationship with, that you can trust over time so that Nadia can grow with you and adapt with you in the long term? Second, how can we scale Nadia so that Nadia is not just a coach for you, but it's a coach for everyone across all organizations, across all different types of domains? And the third challenge is how can we build an adaptive and proactive coach that's going to change and become more personalized with you over time so that your Nadia experience today is very different from your Nadia experience six months from now or a year from now in ways that are fundamentally different than what we have. 

And as we're kinda working on those challenges, I wanted to just kind of talk through a little bit about where we are today. Here's some information of the overall architecture of Nadia, a little bit more of a technical or conceptual level.

05:50  Nadia’s AI Architecture & Capabilities

Here at the base, again, we have foundation models. So those are the large language models that you probably have heard about. There's not just one large language model. It's many different types of models, small models, big models, state-of-the-art models, multimodal models, reasoning models, all these different models that, when you're using Nadia, you're using a suite of the best-in-breed, state-of-the-art language understanding building blocks that go into building the next-generation assistant. 

Next level that we have is memory. So Nadia has a couple different kinds of memory: memory about you, memory about your organization, and memory about coaching. So memory about you are things like that you expect the coach to remember. You expect them to remember your past conversations, the last time you shared your calendar with them, the documents and information that you uploaded, as well as information that you've shared and talked to about the people in the network that you have. 

For your organization, we—Nadia is custom for and bespoke to your organization. So knows your OKRs, knows your company values, your training documents. So the coaching that we have with Nadia is bespoke to your organization that's leveraging the right frameworks so we have the coaching be most effective for your team and for your overall organization.

And lastly, for me, really fundamentally, coaching. Nadia is nothing without her deep expertise and really differentiated expertise with expert coaches. So we work with a set of expert coaches to curate knowledge, to curate situational understanding that now you can use in future conversations, knowing the best frameworks to use so that it's not just a generic coach that you would have off the shelf. On top of that, memory alone is just not enough. Memory is only as good if you know how to use it. So on top of memory, we have an intelligence layer or hypothesis engine for just building a plan for this conversation and for future conversations, leveraging just the right elements of memory to be able to pull them in at the right time to be able to make that plan and to be able to have a successful conversation. 

On top of that, we have that interface layer. We have Nadia's core capabilities that we have to be able to execute that plan, whether today may be a skill building plan, tomorrow, it might be a reflection on feedback that you have from coworkers and have those orchestrated as part of the conversation, of course, in a rich multimodal interface experience. And around all that, of course, we have the safety and guardrails because one of the fundamental pillars of Nadia is the fact that this is an enterprise tool. We want people to feel safe and trust it, so you can talk about anything and Nadia will, if you can't talk about it, Nadia will stop you and prevent you from talking about things that aren't allowed in ways that are gentle, that are coaching approved, and that maintain a fundamental kind of trust and kind of safety of the coaching experience. 

And with that, I think we'll probably be talking about some of the new and exciting kind of product features that we're building on top of this architecture. 

‍Das: Jeff, thank you so much for joining us for this, and looking forward to seeing what you and the team build in the next coming months.