Jack Shay: Developing Products in EdTech: Qualitative and Quantitive Metrics, Regulatory and Ethical Frameworks

Summary

Today my guest is Jack Shay. Jack is one of the most admired and respected product leaders in marketplace growth and edtech innovation. He is the former Chief of Product at CourseKey, the leading student retention platform supporting over 200 colleges.

Prior to CourseKey, Jack worked for OpenTable where he led multiple large-scale product launches totaling >$XXM annually through IPO. He went on to lead product in a diversity of industries at places like, CrowdFlower (Figure 8), Pipefy, and Tech for Campaigns.

In our chat, we get deep into:

• What makes building in edtech different from other industries,

• Including why ethical and regulatory frameworks challenge innovation,

• The proven practices to drive qualitative and quantitative metrics and KPIs,

• Establishing product development organization with transparency,

• What opportunities product and cross functional teams have with new ML summarization,

• What’s behind the new innovative tech that’s addressing long-standing barriers to advancing education delivery and

• Why accessibility to these solutions is key for meeting students, teachers and tutors of today and next generations,

• We also talk about, demand for personalization in education and

• How it can (and should be) met with AI, and

• Predictions for the future of edtech and beyond.

If you enjoy this podcast, don’t forget to subscribe on YouTube, of follow us on Monterey AI.

Speakers

Where to find Jack Shay:

• LinkedIn: https://www.linkedin.com/in/jackshay/

Where to find Chun Jiang:

• LinkedIn: https://www.linkedin.com/in/chunonline

• Website: https://www.monterey.ai

Transcripts

Jack:

 Sure. Hey, thanks. Thanks for having me. so, my background has generally been, product management, even, even before I went to business school and came crawling back to Silicon Valley afterwards from New York, but, was born and raised in the Bay area mostly and, certainly professionally. and I've, I've really tried to go deep on scaling, and leading product development functions. I've been at OpenTable. I've been at CrowdFlower, which is now part of Appen and acquired, Pipeify. most recently, although it's not my first stint in ed tech with CourseKey. which is a vocational, solution for student retention and success, and compliance. And, so EdTech is, is on my mind as is AI and, and how you build a good product function within there.  So that's what we've been vibing on and I'm excited to talk to you about it today.

Chun:

Tell me a little bit about your career path. When did you start get interested in EdTech, and what led you to lead product at CourseKey.

Jack:

Sure, sure. Well, education just for me in general is how my, how my family has gotten to the point of, you know, being able to put food on the table and, and all that sort of thing for generations. And so that was always impressed upon me as the, the first thing to invest in and, And, and that's kind of carried me through, both as, growing up as a kid and then, and then afterwards I did, but I got my first kind of professional opportunity, to work with ed tech as a strategy consultant at Booz Allen Hamilton.

Jack:

And, we had a couple of the, confidential, of course, but, educational testing,  organizations, the ones that you. Probably know and hate from when you were taking standardized tests either to get into grad school or high school or whatever it might've been. And,  and so we worked with them and that kind of ignited a flame where, where I've been messing around with it for a long time. But, CourseQ was my first time really jumping in with both feet.

Chun:

I know we bounced ideas around what do you see the way companies use qualitative data, and how are they different in general tech companies, versus in education tech companies. I would love to hear more about that.

Jack:

Great. Yeah, that's a big, big question. I'll try not to make the answer even bigger. I think, you know, thinking about this, Ed Tech, it is, it is a little different. And and we'll talk about that. But just in general, as a. Product leader, and especially for the startups that I've been working for, now, 15 or so years, I would say that the, the best practices that the priority should always be the quantitative stuff, right?

If you can, if you can set up quantitative metrics and, and specifically when it comes to figuring out what the right path is, when you can do multivariate testing. A B testing, as really the, the first and best way of doing it. If you have the scale, which sometimes startups don't, to do good multivariate testing and you have a feasible methodology of doing it, that's, that's generally always the best way to really prove, that something is a driver or a hypothesis or whatever.

And, and, you know, just in general also, I think, as the cost of data and measurement and all that sort of thing has gone down, you know, there should always be. KPIs, key performance indicators that show your, your, your development process, your product development process and, and your production systems are, you know, operating nominally and optimally, right?

You're sometimes you're going to balance costs and speed and quality and all those sorts of things. but you should have those dashboards and, and not fly blind. And if things are statistically significant, that that's really the best way to do it. But,  you know, I, I've, I've been in organizations that, that do kind of poopoo the quality side of the qualitative side of things.

And I think it can never be zero. You know, you don't know, especially when you're trying to build a brand, if that brand promise is ever at risk,   You don't know where customer expectations are kind of less than being met or, or, just the question.  I'm sure you've been on a sales call before.

Why didn't they buy? You know, that is a real hard qualitative question, hard to put numbers behind it. And so,  best practices and sentiment analysis, best practices and just sort of being open to discovery. I think are, you know, they're out there and you really got to be looking for that. Even if you are just, hey, I think we're optimizing a funnel.

You know, open table was laser focused on the funnel, but there's just so much more to it. and I would think also proxy metrics are important too. You can't always measure everything, right? The data is not always there. And, and so if you're not asking qualitatively, how do we know, you know, Quick quickly, if we're on the right track without over building a whole measurement system without overtaxing instrumentation as part of the project, you know, that can kill you that can slow you down that that that can just, you know, lead to over investment real fast.

And so, you know, to me, If, if a definition of success is applied to every story, every epic, every, every initiative, and, and the, you know, it's the smart metrics, right? Smart being the, what is that? And now I'm going to have to remember a specific measurable, attainable, realistic, and time bound that acronym, for those of you who don't know it.

I think that's really that's really fine. Anything can be qualitative. You're trying to get customers to say a certain thing, use certain words, are the ideal press release statements that you put out when you, when you sort of wrote the, wrote the initial plan, hypothetically, are those true?

You know, those, those are all wins. And. You know, specific, I would say, just, you know, to turn to ed tech, you would expect, you know, for that to be more on the qualitative side, right? You know, index more proportionally towards qualitative and quantitative,  but it also depends on your product market fit equation, how you make money and all of that.

But, you know, in ed tech, I'd say  specifically it's qualitative because, you know, there's ethical frameworks around, educating folks, there's regulatory ones. and, and it's just about whether or not. People get it and can demonstrate the knowledge and prove the knowledge and all of that. But, you know, at the end of the day, most educational institutions are looking at a retention metric for students.

They want them to be successful and, and it's deterministic, right? You fail, you fail. You, you, you withdraw, you withdraw. And so even there, there's, there's plenty of quantitative to work around and You know, and on the other side, machine learning, right?  You know this well at Monterey, right? It's massively quantitative.

It's all math, right?  the quality metrics, for evaluating your training data, your outputs, in models there, you know, generally speaking, you know, they've got to be human categorized at some level. You've got to be taking just as a best practice you know, that we promulgated a crowdflower and, that it would have to be at least 10 percent kind of reviewed.

even at scale by humans. And so, you know, it's, it's never zero, I guess, is what I'm saying. It's always an optimum that's based on your needs.

Chun:

What an interesting trend. Or kind of the change of like how I think about this space. Um, so when I first, I started my job, um, my career and like at Uber, Uber was a massive, right? Like everything is like data driven. Uh, we track like literally everything to like improve the product, like experience there, uh, and then to second them by like geo areas by their time, by different, like type of cars you're taking.

Um, but then they always find interesting. Thing that like human stories always resonate way more with ICs. So like one scenario, you could think that, Oh, it might take us probably a lot of different dashboards to convince that, okay, engineer to work on like one thing or product team to work on thing. But if we literally like jack the driver and to let the driver, right.

Talk with someone who is building like the, uh, driver app. Okay. Or it's like platform in general, people resonate with starry, like an experience, like right away. Um, I assume the reason that why like qualitative data is getting like, look at more, um, in education industry, because they're like, just like way more, way more that star is a human started that can have like this kind of deep touch with the public or with the team in general.

Jack:

So it's, yeah,  it's why so many people get into it, right?  Like it is, it's to help people stand on shoulders and, and be able to provide for their family or go to college for the first time, you know, it's. Those highlight reel things. If you've ever done that as a product manager to sell people on your idea, when you clip the interviews, all that, like that is, you're right.

That, that really gets the light on for people. It's a, it's a lovely example.

Chun:
You mentioned regulation and compliance constraints. Would you say that somehow makes qualitative data more important, or in someway, make it even harder to quantify the qualitative data?

Jack:

 Yeah, I mean, I think it's a little bit of both. You could, you could take, you could take that discussion in a lot of different areas. I mean, it's, the compliance paper trail you need. Right, that that is that is qualitative and that's just making sure that things are written down that  the educational plan is there that you have a record of the students attendance interactions and and and all of that sort of thing because you need to, you know, when you're when you're saying, hey, I'm going  to draw down a percentage of this loan because this is how long the students been here.

You have to be able to prove that. But I mean, in terms of in terms of the qualitative piece of things, it's. You know, those are the things that are, when, when asynchronous education comes in, right? When you're, when you're doing an online portion of things and, and you're not able to see everybody in the room and interact with them, or even when you're online or when you're, when you're on ground, let's say, and, you want to be sure that people are paying attention and engaged and all that sort of thing.

Those are, those are opportunities to really, you know, not just get those stories, but really understand, you know, when the light goes on for, for a student. And, you know, just pedagogically. Yeah, I think it's, that's not dissimilar product management and I guess educational instruction that way.

Chun:

What about marketplace? Can you tell us a bit more about what are the functions in the organization that you work with to get the qualitative data from both sides of the marketplace? 

Jack:

sure. Well, I think, you know, if you're if you're in a bigger corporate environment, and I've been in those a couple times, although not too recently, I would say you have to deal with it. But but, you know, getting the voice of the customer data streams going right? even if it is analog or batch as opposed to an API or whatever, you have to do that.

And so the tool owners of your CRM, you know, which is often sales, especially in, in startups, right. The, the, you know, customer success for your support tickets and your support chats and all that kind of stuff. the zoom recordings, you know, all that kind of, all that kind of thing that you can, that you can get your hands on, which as you know, is a.

Product management person yourself or a  product person yourself, how hard it was to get ahold of all of that data. You know, what were you missing? What conversations were you not there for that? That's possible now to really get into one tool and, and whether it's Monterey specifically just to digest those sorts of things, you know, fit for purpose.

 to kind of get the qualitative feedback issue surveys, do research, all that sort of thing, but, but taking the next, step and tying it to the road map, I think, that's, that's where, you know, you're not talking to anybody, you're just trying to run a good product development organization. And so the, you know, the product boards and intercoms and, all those kinds of things you can integrate together and start to associate, you know, those conversations or those insights to those customers, those deals.

You know into product initiatives and so everybody can see asynchronously who we're working on what for why and what they're saying about it and how how How annoying that bug really is even if it doesn't sound like a big deal, you know, whatever it is. So that's that's you know, being able to show that help sales close deals where it makes sense and kind of real time You know, that's great.

If you can do that with alerts and all those sorts of things that are possible now with ML summarization, ML based summarization, I just, that to me has made this so much less of a fishing expedition,  where you spend the time, like I, I require at least two hours of, of practice. Product people on my team every week, just looking at the stream, right?

It could be talking to a customer that counts, right? But it's, it's looking at the feedback.   and being able to actually have, a large language model, simply summarize the themes and allow you to traverse the knowledge graph, the, you know, the vector store, whatever you want to call for who else has said things like that.

it just makes it so much more effective. it makes the, you know, the hypotheses that are important that you do go. Do a user research campaign around work. And so, you know, for me, those are conversations that help move deals forward that help you justify your value and why you're working on the squishy stuff about the design that some people say, why does that matter?

And that is really the cross functional leadership piece that the product has to do. So you should be working with anybody who's talking to a customer, and playing it back to them about what you're going to do with it that way.        

Chun:

The, um, value problem I always tell like a product team that we're selling to is like, Hey, if we can help you to get rid of this, of this, like boring work of just looking through your data and show you top things. And instead you can spend like all the time to talking, like really talking with like customers would tell you like, Hey, these are the five users you should go talk to.

This is what they are talking about. This is how you should like, you know, either dive deeper into the questions or give them kind of like solutions. I think there's. Always this interesting perspective of how do you keep like customers in the loop? Even from like either regular regulations or compliance point that, you know, you can not really deliver what they are asking for in short time.

Jack:

If you're. Value prop is around compliance.  You know, that that that's also become easier these days to sort of plug and play. I mean, I remember when it would take a year to get, you know, socks, compliance projects through the snake, you know, in order to go public, right?

That's, that's an example. But now you can go to, one of these you know, sort of compliance, Startup software as a service thing and sort of get certified very quickly. Right. And so, so it's, it's important that you walk the talk, but, you know, being able to deliver those things, I think is faster, but if you are essentially the one that they get to blame, if something goes wrong and they're, engaging you to make sure that they, that they are compliant and that when auditors or whoever would come knocking, you know, you have, you are part of giving them the paper trail in a, in an effective way.

Yeah. You know, that that's not an option. You have to lead. I think, by example, you have to have thought leadership around, you know, the new regulations that are coming out and how they're going to affect you when they actually go and force, you know, ed tech, for instance, like, or of course, in particular, there's a lot of, new regulation around, for instance, a borrower defense to repayment.

Which, you know, if you have a student loan, basically, if the school can't prove, that they've done their bit, the student is not liable for the loan, and the federal government, asks the school to pay back the money, right? Just to, you know, title four is what it's called. And that's massive, right?

Like you, you have to have your ducks in a row for that one. cause you can lose your accreditation, all that sort of thing. And, and so, we kind of made a practice of, being conversant on those things before they were actually implemented because, you usually have some heads up before it's really going to be enforced and you're going to be accountable for it.

 Chun:

How do you, this kind of kid, this kind of like momentum, how do you orchestrate those kind of like communications? Have you noticed like any kind of like, not behavior change, but trend of like how students or users like this space provide like feedback or have like a advocate there, like request, uh, compel like to maybe make a change.

Maybe like five years or like 10 years ago. So like one trend, like it's very typical, like very obvious. Okay. Everyone record like takes off videos or like go on like a video style of communication. But have you noticed this kind of behavior change, uh, of communication in terms of like giving feedback to the services that they are using.

Jack:

Yeah, sure. Sure. I mean, we have, we have some of the marketplace dynamics. I think you used that word before, in that the students are the users, but the customers are the institutions, right? And that's, that's frequently the, that's frequently the case.  Whether you're this, you know, not the student information system, that's very much administrative, but learning management systems, e learning systems are often procured on behalf of the students.

And then the students are the users, right? Like a lot of business users are kind of not, they don't get to choose which, whether they're using modern AI or something way worse than that. So you know, but yeah, absolutely. You know, day one that. That I got there that just like there's no, email is completely ineffective these days, right?

It's all it's all text based.  they do like video, I'd say, especially if it's if it's in career education, specifically cosmetology. a lot of the time you have to prove by taking a picture of, you know, the haircut you gave or whatever else it is, but, you know, they want to use that stuff.

It wants to be, it wants to be on Instagram in addition to, just taking the picture to prove you did the work. Right. So,  yeah, I think learning styles also, dictate that. Right. you know, We had a lot of hypotheses and sort of, product directions that we want, that we're planning to take, you know, at, at, School's BS to say we need to make sure that we're meeting students where they're at, right?

And that that would be true in K through 12 as well. You need to be sure you have visual aids. for visual learners, you know, or written learners or whatever. And and so. You know, A. I. parsing through student feedback, saying, I don't get this. I don't get this or bad grades or whatever would, you know, you could very easily find a series of recommendations on.

 Hey, you need more visual aids. You need more. You need to spend more time and review, you know, maybe. You know, what, whatever those sort of recommendations would be, you could, you could really, have something quickly show you a couple of opportunities to improve, either an individual students outcome or, or just in general, the program.

 I think students, especially students in those situations and higher education where you're a bit more of a customer, of the school than you are just. A pupil, let's say, I think,  being able to respond to that feedback is a huge opportunity and look through the classroom chat and try to derive meaning out of those things.

but yeah, don't do it in email.

Chun:

So we started talking about this kind of technical innovations or advancements in education in general. And I know we talked about a lot about digital literacy and inclusivity of that. I would love just to hear a bit more about your general thoughts and view of how this like really fast pacing of AI is affecting like the evolution of education in general for students.

Jack:    

Sure. Sure. Well maybe  I think one of the things I found most surprising rolling up my sleeves into this was, there's, you know, there's been ed tech startups and we've seen them in Silicon Valley and, they, you know, the outcomes have not necessarily been there. for students say nothing of, you know, VCs and the startups themselves, right?

Like it's, it's been an underperforming sector and, when I looked at, you know, the technological stack of, of lots of schools, right? Including my own kids schools, you know, this stuff is all, you know, sometimes 25 years old, right? Like the, the original student information system from when they first moved away from paper.

Is is the same one now it's anthology, although it used to be called nexus. It's been flipped over three times in private equity deal, so, so, you know, while I think, digitization may have saved some trees or whatever it is, it, it, you know, the barrier's just been so high in terms of switching costs.

And, and the risks have been high in terms of, you can't degrade your program, because you're having technical issues. And so those startups have a hard time breaking in, in these things. And, it has driven all kinds of democratization, right. You know, online courses, just being able to show and verify your skills.

It's kind of, you know, Khan Academy stuff, the ability to sort of work on it on a smartphone, all those kinds of, that, that's real and that's there. so even if the investment outcomes and even the investment itself has been lower and lower because, in some cases, people are thinking demographics will actually shrink the market for higher education software and all that sort of thing.

AI is just lowered. Also, I think more than that, the barrier to entry and the barrier to switching costs. And all of that you can handle, it's much easier to get a sense of the data hygiene from a large language model, telling you how many gaps there are in the thing or, in the distribution of data, you can set up those pipelines and not even have to know SQL.

Sometimes to do that now. And so, you know, to me, I think, it's likely to revive investment, both institutionally and, and for students who, you know, are enterprising and, and want to take advantage of these things. I mean, that's, that's, that's my, that's my view. There's definitely headwinds, in that, but it's just, it's just so fast and so much easier to, to demonstrate value and student outcomes and in, in real time that, without coding, without, without a.

Big data migration project that takes 18 months, it's just going to be a compelling problem.

Chun:

Thank you. I was, I was watching a video, a TikTok video last night. Uh, there's like one student whispering to Alexa, asking for like mathematical questions. Like, Hey, Alexa, what is like two plus six? And Alexa will literally like whisper back, say like, Hey, two plus six is eight. Um, and it is just like funny to see how fast Kids and students like, uh, adapt, like to adapt to AI.

Jack:   

Whisper.

Chun:
But it's like, we all know, like, AI is not to the point that provide the absolute accuracy or truth. Even like, you know, we don't really know who provides the real truth by the end of the day. But there's like this kind of risk and danger of assuming like accuracy of like new technologies. For both sides, for students and also institutions, what are, what are some like, uh, perspectives that you have on that topic?

Jack:

 I mean, AI, I think, , if, anyone who's ever tried it in a professional setting, right. It, it, one of the best use cases it has is. the sort of get me off of the ground. Explain to me very quickly as somebody who's semi technical or very technical or whatever, like what this is all about, what this topic is all about, what this paper is saying, you know, all that sort of thing.

That sort of lowest common denominator, getting students to basically, understand the basics very quickly, is, you know, subject to literacy and all of that basic literacy, nevermind digital or high literacy is is just going to be amazing. I think for for underserved students or students who need more repetition or just extra extra time.

But but the literacy, literacy, literacy, to understand whether or not the A. I. Is on track is obviously it can't be the students, purview to do that. They don't know they that they definitionally are not able to evaluate, what they're being told or can't be expected to. Let's say maybe they can.

In terms of evaluation workflows, in terms of, you know, LLMs that can be specifically built to evaluate independently, another LLM sort of initial output you know, students and and institutions thinking about the workflow and the guardrails around that. seems like there's been a lot of innovation, you know, on a slower, I think, or a later timeline, but very interesting around evaluation in particular, not just the chat bot, but being able to evaluate it at scale, mathematically with new, new kind of best practice measures.

And then just also, as I mentioned, you know, with humans,   hopefully the teacher is looking at this stuff, is explaining You know that if it doesn't sound right, you should be asking, right? Like you can ask for another explanation. You can you can say that doesn't sound right. What's your source?

You know, all that kind of stuff and get an answer. And so,  to me, I think that's probably a next wave is when we can get comfortable that we know what we don't know about what it's telling us. a lot of really cool innovation in there with, with a bunch of startups, you know, that, that you probably know of Galileo reminds is one of them, that, that I'm really excited about and, and, yeah, there's just, there's a ton.    

Chun:

One very exciting thing about like AI is I really get to the point of like, okay, the education can be personalized to your life skills, to what you feel like most comfortable with, or to like your goals in life or the career like trajectory there. Love to hear more about the trends or like predictions or the future of like e learning in general from you.

Jack:

Sure. Yeah. Yeah, yeah, yeah. Well,  mean, I do think students have to drive that sort of thing, not institutions, everything I mentioned about not being able to run tests or give student a, a different experience than student B. the students, you know, academic freedom, equity, all those things really important.

And, and regardless of where you are on the journey and so, you know, there's gonna be, I think, you know, the enterprising students that understand,   these tools are here to help and, and you know, they'll. I'll sort of hopefully be empowered on how to use them,  even as I think we do have a lot of.

K through 12 education, pressure politicization, right? Different states doing different things. Some, some of which I just, I can't wrap my head around really, but . But the folks who, especially very cheaply, can hopefully afford some access to some form of, of of.

Rapidly commoditizing large language model. I think, those students are going to be able to rise to the top. But, you know, going forward, the focus is always going to be on the students who are being left behind, like the struggling student, getting making sure that they can pass, but the trick is that hopefully that with some literacy and with, you know, hopefully a good, strong. You know, or or at least not critically weakened education system. People can decide or be in be in the be in the situation where they get to their job is sometimes at least telling the machine what to do, as opposed to being told by the machine what to do, and those are the jobs you want.

Even if they're still going to be elite institutions that, you know, only the very special can get into and that they're going to, you know, guard their value, with their lives, it's, it's not going to be something that's a public good. I think, I think there's still at least an opportunity, if, if we have the civic education and, and,  the political will continue to make these things accessible, you know, the, the, the return on AI, the ROI, I, If you want to, I don't know if you've ever had to use that phrase Monterey, but, but it just, it is, like I said, so,  much more immediately demonstrable that, you've, you've helped the student or that a student is now you know, been able to use this tool or not. But you know, What's going to happen in the middle, I guess, is the question to me there and, and, and my, my hypothesis here, what we've seen a little bit of, I think already. In particularly high demand skill sets  or worker types is that corporations, especially the big ones, you know, in our corporate governed capitalist world there, you know,  they're going to take over some of that middle, middle area where, you know, the skill oriented education, that they, they just know what they need and, they're not going to be as burdened.

By the same rules, the labor department is still very interested in this stuff. In fact, in some of the vocational schools or states, there have been, moves to sort of treat students as workers, right? Because they do internships or externships. And, you know, there's, there's. A little bit of overlap there that I think is, an area of regulatory probing, I think, but, but corporations, you know, as long as they're compliant with labor law, they're going to be able to do this stuff.

They're going to be able to do it faster. And I think that the, the, that we will see a lot more of, just because they're, they're going to be more, innovative and able to, able to take advantage of those things, these tools that have been. Yeah. Provided, but, schools, I think, like I said, the switching costs for these legacy systems are going down, and hopefully it finally will flip because there is some great technology out there.

It's not that hard to build these systems of record if you really think about  it. And and the trick has always been, how do you get from the old one to the new one and, and, and I see a lot of promise there,   individuals being able to prove it online, having, having AI help you skill up and Put a website out there with no code, you know, kind of thing that shows that you know how to do these things.

These are, these have always been somewhat possible, but they're, you know, in the last year, it's just been, exponentially more easier to do.   What about you? I mean, you, you, you find your AI no doubt having to educate product managers. How do you, how do you see it kind of unfolding that way as a teaching tool?

Chun:

And that's the hardest part of my job. So once we like go to like a cell, we give you a pretty clear of ROI of like, okay, how much time we're saving, uh, from analytics, from a triage of automation, we can do there. The real thing, like, if I know like, okay, this person could understand what I'm talking about, I would start this kind of like competition about like, Hey, what we really wanted to do is to help you to learn more about the Product insights help you to do like data analytics, to understand the concept of like how to use data to find that root causes without learning about like code, without learning about query, I think the traditional like a platforms for the last decade.

Jack:      

Well, it is. It is. I mean, it's, I love the idea that it's not just like, Hey, this is the theme that you should be looking at, or this is the theme of all of this feedback. But, you know, if this is, if this is the goal, you know, then, you know, have you thought about this? Have you thought about that? That's, you know, I love that the, the, what am I missing prompts when I write a spec, right?

Like, I love that. To just be able to, you know, what have I failed to mention? Oh yeah. You know, all that kind of thing. But the,  ability to teach it on the fly, right. Whether it's a, you know, retrieval, retrieval, augmented generation of rag pipeline, you know, of like context that you've given before, just simply the thumbs up, thumbs down afterwards of like, yes, I'm, you're on the right track AI, keep going, that loop being that tight is amazing.

to me, and yeah, I think I think you're kind of doing a similar job as you would, for a student in that world was product managers. Show us show us a little bit about how it's done.    

Chun:

Uh, everyone's trying to. like fix this or solve a very specific pain points or pain problem. Same kind of like education. It's like, okay, we need to learn like a coding. We need to like, uh, I don't like what drawing is different, but coding is a good example. Um, but like on a high level, it's about like, okay, how we understand the world, how we understand a user, how we understand data, how we view things, how we like ship things.

These are the things that like, I do believe like as AI status or like products of platform these days, we have more responsibility of like educating users on that front, instead of like educating them like how to use a tool, like which button to click to get the report you need this model. Okay, what do you need this report for?

Jack:

Yes, well, good old templates, templates. It's you know, product led growth 101, right? Is give people templates, give people you know, degenerate example they can work with. with, you know, all that sort of thing that said, you know, and that is, you know, to me, that's what I think I does great at is gets you, gets you a little bit off the ground, gets you to cover the basics, think about the basics.

And, and you know, and that's, and that's what you're trying to get out of a lot of students is just, Hey, make sure that you  have a basic understanding of, of why this is important. What's the key learning? And so and this is what I expect in your paper, right?

You haven't written a conclusion, right? You can write the paper and have AI tell you how to make it better. Like, that's fair game. And no one would consider that unethical,  right? Unless they've written a policy that says you can't use AI at all, I guess. But, but but I only know of one instance. And they've been roundly mocked and I bet they're going to change it.

Forget, I'm not going to name, I'm not going to accuse someone I don't remember the name of, but it's a university you've heard of before. They tried it that way, I think, in one of their graduate programs and people were just like, no, let's, let's deal with this thing. because it can make us better.

Chun:

I would love to you. Wrap up with one more question. So I asked this like to everyone, uh, do you have, what has been your like favorite book, uh, recently that you think that will help like folks that understand, uh, more about like education or technology or like AI or like infancy in general, but yeah, one book recommendation.

Jack:    

One book recommendation. That's hard. That's hard. I mean, I,  I, let me do two. Can I do two? Really quickly. So, so I just read.  So the, the one that I think was just an incredible read and will keep you humble, no matter what, about all of this hype stuff and all of that is going infinite is the Sam Beckman Fried thing about, crypto  and FDX written by a great writers names.

Michael Lewis did money ball and some other things like that. That one just reminds, it's just a real. Object lesson in,  sometimes people aren't what they say they are seem to be. They're not things don't make as much sense as people would have you believe just a real sobering account of something just crazy that happened. and when, when, when all was said and done, how, how little there was there. Just just amazing. So, so that one.  but also, I just think as long as we've been talking about a I if if if you're interested in the topic and you haven't read, faith Ali's, book, the world's I see.

Have you? Have you seen that?

It's, it's, it's a really good  compact history of just, of also somebody who's come from nothing. Right. You wanna talk about like, education kind of being, being the way out, you know, all that sort of thing. I, I thought she wrote an amazing book. She's a, you know, obviously a Stanford, AI Pioneer, worked at Google for a while.

for F-E-I-F-E-I, Lee is LI. By the way, if anybody's looking at that, but, but yeah, just a really good personal story and just had a real seat at the table for and an ability to recount the history of how we got to this amazing moment with, transformers and GPUs and all the kinds of things we can do now, how we went from there, from, you know,  really rudimentary, crazy stuff.

So I think she just did a really nice job with the history of how we're here.

Yeah, yeah, right. Or we're doomed to repeat it. Either way.    

No, no, like, likewise, I,   📍 I, well, yes, there is actually, but but we'll, we'll have to take that offline because, yeah, after, after we both go read this again, and everything you've said and all that sort of thing, I'm going to have questions, none for now. Thanks for having me. It was a great chat.

Absolutely. Cheers, JChun