Hey, Megan, let's do a podcast. Great idea. What

should we talk about?

Hey, it's Megan from TorranceLearning and another

installment in our Tangents with TorranceLearning Podcast. And one

of the things that is super, super cool about my

job and the work that I do is I get to talk to all sorts

of really interesting people. And the

cool part about having the podcast is that I get to share those conversations

with everybody. And so what I'd love to do is

introduce my new big nerd of a

friend, Chris Grady, and have a conversation that

may feel at first like it comes out of left field, and then all of

a sudden it's going to say, oh, my gosh, this is like what we do

here in Allen Date. So, Chris, welcome.

Well, thanks for having me. And thank you for introducing me as a giant nerd.

That is my official title. Chris Grady, Giant Nerd. I

think maybe you need a little bit more of an official title than that, but

I don't mind Giant nerd. Yeah, well, for people listening, I'm a

researcher, and I'm a former senior advisor at usaid, where I help design and evaluate

development programs, which, one might say has

nothing to do with learning development other than that the word development

is in both. And yet in both cases.

Stop me if I'm wrong here. In both cases, we are doing a

whole lot of effort aimed at a whole bunch of people

or a targeted group of people in order to get them to do or

change a behavior that we feel is

good for them or good for us or good for something. Is that a fair

description? Yeah, that's fair. I mean, most interventions at

USAID were about behavior change. Right. We want people to do something different than

they're doing. And that's most of the interventions in the world. Right. If you're doing

an intervention, it's because you want something to change, and that's usually people's behavior.

And then the key question is, did it work? And how do we know if

it worked? And so that's what I specialize in. Okay, that's

fascinating. And one of the things that so you

presented earlier this year at an

ATD intensive on measurement and

analytics, and we. What was really cool was your

entire conversation around experiments.

And experiment is a word that generally

feels very unsettling in the learning and development world because we want to

come across as professionals. We know what is right for us.

And while we love to iterate,

that's part of what we do in our Agile process.

We don't often call it an experiment.

What do you mean when you say the word experiment?

I mean something, I guess, very technical. You've got some group of people

and you split them and then you do something to one group and don't do

something to the other group. And that way you can learn the effect of the

thing you did. Boom. Experiment.

So if I were to say, gosh, it

sounds like drug trials or university

research or something like that. Is that a good

analogy? Exactly. Drug trials, university

research, they all almost always use experiments as their main tool for

learning. So drug trials, right. You give the drug to some people and not others,

randomly assign who gets it and who doesn't, so you can learn the effect of

the drug. You can apply that same logic anywhere. Right. The same logic applies

whether it's a social innovation, a drug trial, a diet, or

anything. Anything you want you can do an experiment on and learn the effect of

it. Okay, this kind of sounds

like the scientific method. Yes. Yeah, it's very motivated

by the scientific method. Right. You have a hypothesis and you want to test that

hypothesis, and the experiment is the test. Okay. And

then generally don't you end up with more questions at the end of all these

things? Yeah, that's the best part. Every

experiment, you analyze it, you get results, and then you want to know why did

that happen? Or something will pique your interest in your brain, your curiosity.

Then you can do another experiment. A never ending cycle of experiments. That's my

dream. Megan, you are a big nerd.

Okay, okay. So, all right. When

I'm thinking about this process and how it unfolds, when does

somebody decide they need to test

this or measure this or design an experiment? Right.

In my mind, not really knowing a lot about how these things work,

somebody comes up with a good idea, somebody goes and they gets grant money, or

they get budget money, or they get money money to be able

to go do a thing that they think is a good idea and they start

doing it. At what point in that process

is the experiment designed? Like at the beginning, before

you start doing the thing, before you start even designing the thing. Or does

somebody say at the end, like, hey, I wonder if that worked? Which is, by

the way, where we unfortunately often

end up in learning and development, although we're trying to work our way back up

that cycle. Yeah, unfortunately it's true. Most people start

to wonder if it worked after they did it. But that is too late for

an experiment. Right. Because to do an experiment, you have to randomize who gets the

thing. And so you can't, after you've Already done the

thing, it's too late to randomize it. So what often happens is

you do a thing for usaid, it would be some development

program, right? So a way to like help a country collect more taxes.

And then you want to know if it worked. So what you have to do

then is do it again, right? Maybe in a different place, in a different location

where you can randomize it. And so I think that's actually a good process

because you do it once almost as a pilot. You kind of figure out how

to implement something and you get a sense that it might have worked, right?

And so then you want to know, you want some rigorous evidence that it did

work, and then you do an experiment on it. So I think that's a good

workflow. Like first do something as kind of a pilot, and then if you think

it works, you want to scale it up and do it everywhere and do it

a lot, then it experiment. So you've got more than just a proof of

concept, you have rigorous evidence that it was effective. Okay, so this is a

multi step. So this is interesting. So the way

learning project works, we're often like, go do this thing and we do a bunch

of analysis and we do some design and some development work. And then

there's generally a pilot group if we're smart, or a beta test or something.

And you're saying maybe

you're not saying this, I'm interpreting this, maybe like

we do that pilot, we do that beta, but then

that next iteration is actually an experiment where we

say like, is this actually. So we've proven it kind of works, and then we

should test whether or not it actually has the intended effect on the

audience. Yeah, absolutely. Because it's very easy to

delude ourselves into thinking something works when it doesn't. Because we want it to work.

Right. If I'm at USAID and I've helped design a program, I

think it's going to work. I designed it to be effective. And so

unless I have rigorous evidence that it doesn't work, I'm going to assume it does.

And so sometimes we need an experiment to check ourselves. And another good

reason to pilot is that you want to know if the intervention

works or doesn't work. Not because you're

learning to implement it, but because it being implemented well is working

or not. Because something you might try and think of some example, but

something might be failing not because it doesn't work, because you don't know how to

do it yet, like riding a bike, right? It might take some time to Figure

out how to ride the bike. And then after that, you want to test if

you're faster or slower riding than running. If you only test it, when someone's

learning to ride a bike, it's going to look like they're slower because they're falling

off the bike. But once you learn to ride the bike, it's much faster,

obviously. So you have that same issue when you're piloting a program, right? When

people pilot, they're essentially falling off the bike over and over until they learn how

to pedal. And so you don't want to be testing them falling off the bike.

You want to test the bicycle. You know what I mean? I totally do. I

totally do. And it occurs to me this is, we're falling

into this. It's a little bit intentional, but

a client of ours, government client, asked us

to build an AI practice module. Yeah,

it's pretty fancy. It's actually super constrained

because for us, we want to make sure that it works and it doesn't go,

like, off the rails. But

the, the first asked was, can you build an, an

mvp, basically a proof of concept. Can it do its thing? And we

shopped that proof of concept around a bunch of, you know, different

stakeholders. And at the same time, we're, every time we're, we're

shopping it and demoing it, we're punching that. Kicking the

tires, whatever you do to tires, right? Don't puncture tires. That's a bad idea. But

we're kicking the tires and, and, and, and really

kind of working on this. And then our next

iteration, we're actually building out because

this audience happens to be really, really

pure, like as, as pure as you're going to get in L and D. And

we're going to talk about that in a second. I'll tell you a little bit

about our world, but we have a large

population, about a thousand learners who all have exactly

the same job, and they're taking exactly the same learning

program. And we are going to give half of them a randomized group. Half

of them will get the AI practice, half of them will get the same

exact module, but it won't be AI. It'll just be

straight up. And then we'll be able to assess

their performance. At the end on. I think we have eight different metrics

that we're looking at. But all of that

is before we spend the big money to scale it up

100 times the size to a much, much larger audience. So

it was interesting in the design of that. I was hanging out with another big

Nerd. And I said, oh,

I'm kind of feeling awkward here because

we've got this AI practice. We're going to give it to half the audience and

the other half isn't going to get it. But what if the other half, like,

but then they might fail the test and they might not get their job and

all this stuff. And this big nerd's

response was, you're assuming that this is going to be better.

Exactly. Like, oh, my gosh, what a butt punch.

Right? I was like, oh.

But yeah, like, yeah, I was assuming that this beautiful

thing we're making was going to be better. And how much better to

test it than to make that assumption? Right. So,

so how do we. And this kind

of came up during the ATD intensive too. Right. Most people

say it might be infeasible. We don't have the budgets, we don't have the

time. It's unfair to, you know, if, if.

How do we. How do we make ourselves feel better? Or how do we

design an experiment that's feasible in an

environment? Well, we have to train everybody. Got

any ideas? How have you tackled this? Yeah, it's a good question.

Also, the ethics of an experiment came up a lot at usaid,

but I think that this ethical question comes up a lot when you

assume that the thing works, because then it would be wrong to not give it

to everybody. So what we've done is, well, we don't know if it works yet,

so let's do the experiment and then if it works, give it to the control

group who didn't receive it. You can always give it to them later. Right.

And you also find often, if you think about it, it's

almost unethical not to do the experiment, because what if what you're doing

not only doesn't have any effect, but it's harmful and you've

actually done something harmful to people. You want to know that and you want to

not be able to do it. And if something's ineffective, you don't want to waste

a bunch of money on it because that's not helping people. So the best way

to help people do the experiment early, figure out if something works or not, and

then roll it out to everybody. So that same logic should apply in

training. Like your example with an AI training module.

What if the AI was worse and it made people worse at their jobs? You'd

want to know that. So you don't roll it out to everyone. And if it's

completely ineffective, you'd also probably want to know that too, because it's probably costly to

roll it out to everyone and so that money could be better spent elsewhere.

Totally, totally. Because it's a lot more expensive to build AI training than

straight up elearning. Yeah. Okay,

so that's really helpful perspective and

probably tweaks the messaging that a lot of L

and D people need to be engaging with their business

on learning and development is

often. Especially when it's employee development. Right. It's a, it's a.

Well, it is not even often. It is a cost center. It is an

expense to the business and not often

seen as a source of competitive advantage.

It's not often seen as an. Yeah, we say it's an investment in

people and it is an investment in people. But

when times are tough, what gets cut? It's training. Right.

Marketing is also an investment in the business that doesn't get cut as

much. So we're kind of in this tricky

spot. What that means though, is that a

lot of times people want to, when they say like, oh,

Megan, I want to measure some stuff, I say, oh, great, why do you want

to measure? And they want to prove their worth to the

organization does that. To me, that sounds like, oh,

that sounds biased. Am I thinking of that right? Yeah.

When you're doing the experiment, you. You want to be accepting of

either result. Like if something works or doesn't work.

Yeah. And you wouldn't want to go in desiring a certain

outcome. Right. Because you're going to. There's unconscious ways you can

kind of make that outcome be achieved. There's lots of

little effects. Like if you bring someone into a lab and

you want a certain, you want them to respond a certain way, they tend

to pick up on that, even if what you're doing is unconscious. And then they'll

respond that way. And so it would look like whatever you did had the effect.

But it's people's social intelligence coming out. That's why you often see

in medical trials and other rigorous studies, double blind.

So the experimenter, when someone comes in, the experimenter doesn't know if

that person's in the treatment group or control group because the experimenter knowing

biases the respondent. That's why in a drug trial, they'll often

give people a pill in the control group, but it's just a salt pill. It

doesn't have any medicine in it. Because they don't want either

side to know which group they're in because people will manifest something themselves.

That sounds like you might have a similar problem here. If you want to Find

something, you're often going to find it. But it's difficult then to

unbiase yourself. How do we unbiased ourselves? We're human beings, we're all biased.

So, yeah, I don't have a good answer for how to do that unless you

can find some way to bind yourself. Maybe you

remove yourself a bit from the actual analysis of the

experiment or implementing it and have somebody who

doesn't have a stake in the results being kind of a positive outcome do

that. But that, yeah, that's tough because who

else is going to implement and analyze the experiment if not, you don't

have a good answer. Megan? Well, you got me in a pickle.

We say, like all good questions spawn more questions.

Well, I think one of the things that in many organizations,

right, we're starting to get data scientists brought into,

or at least data analysts brought into the learning and development

team. I even heard last year a team that

had combined and they called it Learning, Design and Analysis. And I thought,

oh, my gosh, that's amazing. And, and, and some people

are finding in their organizations right there. They have a

finance team or a marketing teams already doing this kind of analysis

on surveys and stuff, or

they're engineers or they're R and D folks who can have maybe a little bit

more. They're a little bit more detached, but they also have the statistical

analysis and tools to be able to do some of this work

that is helpful. It also

occurred to me, I wonder if there's like college interns or

people who would love to have to work on this kind of thing.

Yeah, I bet college interns would. I mean. So I've been going back

thinking about what you would ask. I think we need to, in our minds, often

reframe things we think. Like, if you do something and you find

no effect, that's bad. No, that's great. You've learned that

you don't need to do that, you know, and that's great. You can do something

else. That gives you the freedom to, okay, let me design something new. Let me

design something better. So we shouldn't go into it thinking, oh, if I find

that what I did was ineffective, that I failed. No, that's like, that gives you

another opportunity. You've learned something valuable and that's really helpful to the business to know

they don't need to. Don't throw your money down that hole. Right. That is

fantastic. Yes. Find out which money, which holes to throw the money at.

Which hole? Does the money become a money tree or. I don't Know that,

that, that works totally. So

if we think about how

we gather data, right. One

of the things, and I'm think surveys as

a. There are lots of different ways in which we can gather data.

The learning and development team often doesn't have access

to actual on the job performance data or

doesn't have access to or the on the job performance is

not quantified, it's not instrumented to be quantified and

measurable. And so

surveys is a tool that is often at our

disposal. We can send surveys, we know who's taken a training, we can

send them a survey. Any pitfalls there? As we think about

designing surveys to send out to people, oh my gosh, there's

so many pitfalls with any, any measurement is going to have lots of pitfalls. So

there's a whole field called measurement validity where they try to validate

measures and surveys. Obviously there's, there's going to be several possible

issues that come up. The first thing, people just

might not know the, you know, you're going to ask them their attitude on something.

They'll come up with some attitude, but that might not

be their real attitude. First, they might not know. Second, they might not want to

tell you and they might not even let themselves know that they

don't want to tell you. So are people lying to you or to themselves?

There's like a lot of, in political science, we often study if

people voted or not. If you ask the average person,

they're going to tell you they voted. Something like 75% of people tell you they

voted even when the actual voter turnout's like 50%

because it's socially desirable to say you voted. And so if you just ask that

question, clearly measurement validity is going to be low because

we know that it doesn't correspond to people's actual behavior. So you have to figure

out a way to ask people the question in a way that they can and

want to respond honestly. So there's lots of tools for doing

that. But that's a huge challenge. One, the simplest one,

is to make the socially undesirable things seem totally acceptable. Right?

So you do that with framing. You frame the question in a way that it's,

you know, hey, we know a lot of people don't have time to vote. Did

you do you have to have time to vote this year? And so then if

they say no, it's like you've already kind of pre built in the excuse.

So it's okay if you didn't vote because we know lots of people have

Trouble getting there. So it's not that you're lazy or you didn't want to vote.

It's like your life is busy. So little things like that can make it more

acceptable for people to answer honestly. And then there's tools to

grant people anonymity so that there's no way to

trace their answer back to themselves. There's several ways.

Okay, now we're going to get into really deep measurement validity, but there's several ways

people do this. One is called, like, a randomized response technique. And the

simplest way to explain this is you give someone a. You ask someone a yes

or no question, then you give them a coin. You have them flip the coin

so you don't see it. You say, if it's heads, just say yes. Just say

yes. It doesn't matter what your actual response is. If it's a tails, answer

this question. And so then if they say yes or no,

or if they say yes, you're not actually sure if they said yes because of

the coin or because that was their real answer. So they have plausible deniability

that if they say yes to the socially undesirable thing, like, did they smoke

marijuana? It's not that they actually smoked, it's that the coin told

them to say yes. Right. There's other techniques that are similar, but those

are ways if people don't want to tell you something,

but you can get them to tell you by granting them anonymity.

That's kind of cool. Well, I was even thinking, your first way, you're like, hey,

there's a lot of reasons why people will come to a class and not be

able to apply that on the job. Were you able to

apply anything on the job that seems like a perfectly plausible thing?

Yeah. They're going to give you way more accurate and honest

responses than if you just say, did you apply this on the job? Because it's

clear if you ask that you want them to say yes. You know, clear,

your sociopaths will say, like, no. Yeah,

okay. This is fascinating. And we could be here

all day long, so looking

forward to the opportunity to continue talking with you about it, But I want to

shift gears slightly. We like to. To wrap up these.

These conversations because you are more than your work. But I'm

fascinated with just, like, work rhythms and

rituals and the way people engage with their craft. And

you've spent a lot of time in academia, like getting a PhD and all that.

Right. Does that influence your. Your work

rhythms, do you think in semesters or trimesters, what Are your

fidgets? What are your work snacks? What if you were to

sum up, what is Chris Grady's work style?

What's it like? What's it like inside your head?

Intense focus, which is different than any other. In my

personal life, I'm very non focused and kind of chaotic. Go with the flow.

When I work, I just disappear into my mind for hours. I often,

I'll like even forget to eat and eight hours will go by and I'll

realize I haven't moved in eight hours, which is very rare because if I'm not

working, I'm constantly moving. I exercise constantly. I can't sit and watch

tv. If my wife wants to watch a program, she wants to cuddle. She's usually

disappointed because I'll wind up doing yoga during the program. But while I'm working,

I just laser focus. Especially if I'm coding and doing analysis,

the hours fly by. And I think that that was very

beneficial in grad school where you need often like a lot of deep thinking and

focused time for long periods of time. And

it's harder to do that in a work world, right. Where I'm like, oh, I

have half hour now, then I have a meeting for 30 minutes and then I

need to, I don't know, I have 30 more minutes to do something and then

an hour meeting. So I think for me academia was great because it let

me use my natural rhythm of just I'm going to work on one thing for

eight hours straight and that was a benefit instead of a detriment, which in the

work world I've got to adjust and be more flexible. I don't know, there's a

lot of people who would love the ability to focus like that. Yeah, I have

reverse add.

That's fantastic. Chris, this has been awesome.

A window into your mind, window into your work.

Applicability to my own work and probably the work

of a lot of people who are listening. So I just want to thank you

so much and looking forward to continuing the

conversation. Thank you a lot, Megan. It was fun.

So how'd that go, Megan? You know, that was super fun because it's the first

time we've talked to somebody outside of the L and D

industry and yet all the things that he

does and talks about are applicable and appropriate

for us. And we could just lift some of these techniques

and there's some, some really easy, easy takeaways. Even if you took

only one thing away, there's probably half a dozen one things that you could

take away and do without any hardship

or at least to have that conversation with your leader. So I think that

was, that was super fun. And maybe we should find some other folks outside of

L and D and see what we can learn from them. I know you've got

one more thing to share, Megan. Yeah, there is one more thing.

So here's, here's one more thing that I was thinking about as we're having this

conversation with Chris is thinking

back to the conversation with Will Telheimer. And as he's got

a lot of work around, right? The learning transfer evaluation model. And

it's, you know, where I've been really moving into. It's his, his wording around,

right. Learning is competitive advantage for the business. And that's really resonated with me.

And I know I brought that up here. But also

connecting in, I want to go back now and reread performance

Focused learning surveys by Will, because I think the combo

of what Will's talking about and what Chris Grady is talking about

is a really powerful combination for getting really

valid measurement in our industry. This is Meg

Fairchild and Megan Torrance, and this has been a

podcast from Torrance Learning. Tangents is the official

podcast of Torrance Learning, as though we have an unofficial one.

Tangents is hosted by Meg Fairchild and Megan Torrance.

It's produced by Dean Casteel and Meg Fairchild,

engineered and edited by Dean Casteel with original

music also by Dean Castile. This episode was

fact checked by Meg Fairchild.