A Stranger in a Strange Land (with apologies to Robert Heinlein)

I had a really interesting discussion today with a kids’ learning app developer. She was upset because her app, which got rave reviews from a number of other review sites, had received a letter grade of “F” from us at Balefire Labs. To her credit, she wanted to talk about and better understand our review criteria. I took her through our criteria and explained where and how her app didn’t meet them. I gave examples of how her team could make some simple (and some not-so-simple) changes to their app so that it would meet our criteria and earn a higher grade. We chatted at some length about how to design instruction to achieve student mastery. She seemed to get it. But at the end of the call she said to me, “Well, that’s not really what our app is intended to do. It’s not really supposed to be instructional. We’re trying to expose students to more so that they can learn more and be more creative.” Sigh.

It’s challenging for a behavior analyst like me to be in an industry in which most decision-making is not based on (and sometimes doesn’t even consider) student learning outcomes or instructional effectiveness.

Mainstream educational technology might not be the ideal place for a behavior analyst. The technology part is okay…scientists and engineers busily arranging and measuring technical events and communications. But the education part…ah, there’s the rub. It’s challenging for a behavior analyst like me to be in an industry in which most decision-making is not based on (and sometimes doesn’t even consider) student learning outcomes or instructional effectiveness. But I’m a slow learner.

Back in 2012 it was the early days of using iPads in the classroom…there were no Android tablets at that time and iPads were really the only player in education. There were only around 25,000 “educational” apps in iTunes and, at that time, 25,000 seemed like a lot! Already teachers and parents were struggling with how to make sense of the selection of apps…the marketing claims all sounded the same and made big promises about helping kids learn. Not until an app was downloaded and tried might the truth that it was junk be revealed. And almost no apps had empirical data showing that they were effective in generating performance improvement.

The idea was to provide, in the absence of empirical research studies, objective reviews of learning apps and games based on what behavioral science has shown to be important features for effectiveness. Sort of like nutrition labels for edtech products…

In 2012, decision-making about educational apps and games wasn’t helped by the kinds of review sites that were available. They were sites run by well-intentioned teachers or parents who gave their opinions about different apps they and their kids had tried. These opinions ran the gamut, from the prettiness of the graphics, to the fun experienced by the learners, to how “engaging” the material presented might be. In short, subjective descriptions that communicated little or nothing about whether or not the kids had learned anything. I thought to myself, “There’s a better way to do this.” After all, we have decades of behavioral data that tell us what matters in good instruction, right?

So I started Balefire Labs. The idea was to provide, in the absence of empirical research studies, objective reviews of learning apps and games based on what behavioral science has shown to be important features for effectiveness. Sort of like nutrition labels for edtech products that kids use in the classroom…a list of what a given app has or doesn’t have in its instructional design and usability design. I recruited my old grad school roommate (Missy Reed, also a former Julie Vargas student) and we put together a rubric and checked it twice. We started applying the rubric to apps, with both of us reviewing each app and modifying the rubric’s definitions until we had greater than 80% interobserver agreement on each criterion (using a moving window of the most recent ten apps reviewed). Then we trained a couple of new reviewers. It took about four months to achieve a rubric that we all could apply reliably across apps.

We were naïve…When it comes to apps, parents don’t care that much about learning; they mostly care about “fun.” And teachers, by and large, are still using mobile devices and apps in the classroom for “reward time.”

Looking back, it seems so quaint that we thought parents and teachers, once provided with “nutrition labels,” would start opting for the healthier choices in educational apps. Ours wasn’t an “if we build it, they will come” attitude. We knew we had to do good marketing and outreach and increase our visibility. But we had taken teachers at their word when they told us what a terrible time they were having finding something good in the app stores. We thought they really meant it when they said that they’d love a resource that would help them sort out the apps that would really help their kids learn. And most of all, we were convinced that they really didn’t want to be spending hours vetting horrible apps themselves. We were naïve.

Now, four years later and with more than 5,000 learning app and game reviews under our collective belt, here’s what we’ve learned: When it comes to apps, parents don’t care that much about learning; they mostly care about “fun.” And teachers, by and large, are still using mobile devices and apps in the classroom for “reward time.”

The contingencies are more complicated than we had anticipated or hoped. We knew that teachers, especially, had a lot of competing contingencies…keeping kids busy and quiet, preparing for lessons, integrating technology with the existing curriculum, and dealing with (usually) a few devices shared by a whole class. Not to mention the inertia that’s involved in learning to do something differently than it’s been done in the past. But we had hoped that if we did the heavy lifting of vetting the products that we could at least help teachers find and select them in an easier and more effective way. And to some extent that’s happening, but change comes to classrooms at its own glacial pace.

Then there are the developers. Our secondary mission at Balefire Labs is to feed meaningful information about apps and how they can be improved, instructionally, back to the developers. We even offer to give them detailed suggestions for how to make those improvements, hoping that together we can move the needle of quality standards in the market. But the most common question we get from developers is, “Can you prove that I’ll sell more copies of my app if I make the changes you recommend?” And the answer is that we can’t. Because that’s not how teacher and school purchasing decisions are made. So where’s the motivation for these developers to make those changes? Implementing adaptive strategies (that require strict data tracking, decision-making algorithms and complex coding), mastery-based approaches and error remediation paths aren’t cheap. It’s a big investment for a developer. They might even agree that they’d like to do it, but if they can’t make a good business case for it then it is very unlikely to happen. No, Virginia, there aren’t very many Headsprouts in the world.

…mainstream edtech is starting to recognize it needs the “learning sciences”…It’s up to us to seize this opportunity and claim that category for ourselves.

None of this is to say that we behavior analysts should give up on edtech. And there are some leading, non-behavioral organizations, like Digital Promise, that are advocating for many of the same data-based and outcomes-driven solutions that we would like to see. But they still don’t really understand data or performance or even learning objectives the way we do. Bigger organizations are just now starting to see that they need people who do understand those things. Not that they see a need for “behavior analysts,” mind you… but that they need people who understand mastery and how to define competencies for competency-based learning and who know the right questions to be asking about these vast quantities of data that companies are collecting as part of their “learning analytics” efforts. Also, they need people who understand function, not just structure…though they don’t know to call it that.

So is there a place for us in mainstream edtech? There is. But if you want to play in the big sandbox with the big kids in edtech it will require a bit of rebranding. Because mainstream edtech is starting to recognize that it needs the “learning sciences.” They don’t really understand what all could fall under that umbrella and they don’t really care. It’s up to us to seize this opportunity and claim that category for ourselves. It will still be incremental and we’ll still be battling with complex contingencies in the market as we strive to change how products are designed and how purchasing decisions are made. But this latent need for “learning scientists” in the context of a burgeoning edtech market (estimated at $8.3 Billion in 2015) could be a perfect storm for behavior analysis to make real inroads in edtech.


Balefire LabsKaren L. Mahon, Ed.D. is President and Founder of Balefire Labs, an award-winning, web-based service that helps parents and teachers find kids’ learning apps and games with the highest instructional quality. Karen is an Educational Psychologist and Instructional Designer with more than 15 years’ experience in educational technology. She was Principal Investigator and Research Scientist at Praxis, Inc., a Waltham (MA) edtech startup that produced instructional software for children with severe and profound disabilities. While at Praxis, Karen successfully generated National Institutes of Health Small Business Innovation Research grants totaling close to $4M. Other previous appointments have included Global Senior Manager of Learning Sciences at Mimio Interactive Teaching Technologies and Research Assistant Professor at the University of Kansas, Parsons Research Center. Karen received her Ed.D. and M.A. in Educational Psychology from West Virginia University, her Product Management Certificate from the University of California, Berkeley, Haas School of Business, and her B.A. in Psychology from the University of California, San Diego. She is on the Advisory Boards of the Center on Innovations in Learning at Temple University and the Cambridge Center for Behavioral Studies. Tweet with Karen at @KarenLMahon or with Balefire Labs at @BalefireLabs or connect with Karen on LinkedIn here.

2 comments

  1. Reblogged this on disrupt learning! and commented:
    Wanted to do a quick share of an article I just wrote for another blog….hope you all enjoy it. Please let me know your thoughts! /k

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: