What Language-App Efficacy Studies Actually Measure

Executive finding

The phrase “this language app works” is incomplete. A study may show that a selected group improved on reading and listening items after completing a particular course segment, while a headline implies conversational fluency. Evaluating an efficacy claim requires matching population, intervention, comparison, outcome, time, and transfer. If any of those changes between the study and the claim, confidence should fall.

This brief provides a reusable audit for product teams, educators, journalists, and buyers. It does not rank apps and should not be used to imply that the absence of broad evidence proves a product ineffective.

Start with the exact claim

Rewrite the headline as a testable sentence:

For [which learners], using [which version and course] for [how long], compared with [what alternative], improves [which measured skill] on [which task], and the gain lasts [how long].

“Beginners completing units 1–7 improved on an external reading and listening test over three months” is interpretable. “Learn a language with the world's most effective method” is not.

Seven questions that change the conclusion

1. Who entered the study?

Check age, first language, target language, starting level, country, education, motivation, previous study, and device access. Independent app learners who volunteer for research may be more persistent than the typical new registrant. A result for English-speaking adults learning beginner Spanish does not establish the same effect for children learning Japanese.

2. Who remained?

Report enrolment, exclusions, dropout, completion, and missing-test data. If only highly engaged completers take the final test, the result describes that group. An intention-to-treat analysis answers a different question from a completer- only analysis.

3. What was the treatment?

Record the app version, course path, features used, target language, teacher support, reminders, and actual time on task. “Used the app for three months” can mean ten minutes twice a week or an hour every day. Product changes can also make an old study irreproducible.

4. What did the comparison group do?

No control group means improvement could come from time, outside study, test practice, or participant expectations. A waitlist controls for little. A strong active comparison receives equal time and attention through another credible method. Classroom equivalence also requires comparable contact hours, teacher support, homework, and assessment stakes.

5. What outcome was measured?

Separate:

• in-app accuracy;
• vocabulary recognition;
• reading comprehension;
• listening comprehension;
• prompted writing;
• monologic speaking;
• interactive speaking;
• real-world task completion;
• confidence, enjoyment, or intention to continue.

These outcomes are all legitimate, but none is a synonym for total language proficiency. A vocabulary quiz that reuses trained items tests near transfer; an unprepared call with a new listener tests a more demanding transfer.

6. Who scored the outcome?

For speaking or writing, check rater training, number of raters, reliability, blinding to group and test occasion, and whether the task was created by the product team. Automated scores need published validity evidence for the relevant language, accents, level, devices, and population.

7. Did the result last and transfer?

An immediate post-test can capture short-term practice effects. Look for delayed testing and tasks that were not rehearsed. The most commercially impressive claim—real-world communication—requires evidence beyond the same exercise format used during treatment.

A study-to-headline translation table

Study result	Supported wording	Unsupported leap
Higher score on trained vocabulary after 8 weeks	Improved performance on the study's vocabulary measure	Became conversationally fluent
Better reading/listening scores among completers	Completers improved on tested receptive skills	Replaced four university semesters in all respects
Lower self-reported speaking anxiety in an AI condition	Participants reported less anxiety in the measured context	Eliminates fear of real conversations
Higher rated monologue performance	Improved on the sampled speaking task	Can handle spontaneous dialogue at the same level
Users with more minutes score higher	Usage correlates with outcome	Extra app time caused the difference

Example: interpreting an app study carefully

An Arizona State University publication record describes a three-month study of 48 independent learners using a Spanish course for English speakers, examining general proficiency and receptive and productive skills. That is more informative than an in-app completion statistic because it names a population, period, and multiple outcomes. It still should be read at the article level for recruitment, attrition, comparison condition, external study, scoring, uncertainty, and author/funder roles before generating a marketing claim. Source: https://asu.elsevierpure.com/en/publications/the-effectiveness-of-duolingo-in-developing-receptive-and-product/

An earlier critical review of Duolingo efficacy claims highlights concerns such as selection and validity across cited studies. A critical paper is not itself the final answer, but it identifies checks a reader should apply to the methods and headline. Source: https://callej.org/index.php/journal/article/view/419

These examples illustrate the process, not a verdict on one company.

The “hours of university study” trap

Equivalence claims often compare a test score after app use with a score associated with a course level. They may not show that the experiences are equivalent in curriculum, speaking interaction, writing feedback, cultural learning, assessment range, or learner support.

Ask:

• Were learners randomly assigned to app and course conditions?
• Were contact and homework hours measured the same way?
• Did both groups take the same external assessment?
• Did the assessment cover every skill named in the headline?
• Is the comparison a causal result or a benchmark analogy?

If the study did not directly compare the two conditions, write “performed at a similar level on this measure,” not “replaced a semester.”

Minimum evidence card for every efficacy claim

Publish a compact card beside the claim:

Field	Required disclosure
Population	N, age, L1/L2, starting level, recruitment
Treatment	Product/version, duration, actual usage, co-interventions
Comparison	Active, waitlist, pre/post only, or none
Outcome	Exact task, skill, scorer, scale, trained/untrained
Result	Group estimates, effect size, uncertainty, attrition
Transfer	New task, human interaction, delayed test
Independence	Authors, funding, preregistration, data/code access
Boundaries	Populations, languages, and claims not supported

What LinguaLive should measure

For a speaking-practice product, the primary outcomes should match the product's job:

• completed speaking minutes and sessions;
• performance on unpractised but parallel speaking tasks;
• listener-rated intelligibility;
• speed, breakdown, and repair profiles;
• interactive task completion;
• delayed retention;
• transfer to a teacher, examiner, or unfamiliar human partner;
• differential performance across accents, devices, disabilities, and levels;
• privacy, safety, and adverse user experiences.

Streaks, satisfaction, and self-reported confidence are supporting measures, not substitutes for performance.

Editorial disclosure

This is a research-literacy framework, not an independent efficacy evaluation. LinguaLive has a commercial interest in language-app measurement. Before publication, a research-methods reviewer should check every example and a legal reviewer should approve comparative or equivalence wording.