About Readability
What is ?
Readability is simply another word for text difficulty.
What are ?
Sometimes teachers may wonder “is this material too difficult for my students?” Or they might want to find an appropriate text for their students amidst a sea of texts available online. That’s why texts for students are often categorized into reading levels: to help teachers (and students) find the text that’s just right.
CommonLit categorizes texts by grade levels and Lexile scores among other features
However, It’s often difficult to pinpoint the difficulty of a text. A book may look good enough for the “average” 4th grade reader (if such a thing exists). Or maybe, on second read, it looks more appropriate for 6th graders. Who can tell for sure? If we give a text to a group of people and ask them to decide on its appropriate reading level, there’s surely going to be a lot of disagreement. And it’s going to take a long time if we want to repeat this process for every single text we might want to let our children read.
That’s why we often rely on measures of readability formulas produced by computers. They are by no means perfect, but they are fast and they’re consistent, which is a good start.
One example of these measures is the Flesch-Kincaid Grade Level. Flesch-Kincaid Grade Level can predict the difficulty of texts and convert them into grade levels. It’s quite convenient which is why a lot of people use it. It’s included as a feature in Microsoft Word if you look hard enough.
The ?
That all sounds quite nice, but let’s dig a little bit deeper. After all, can we trust whatever these measures are telling us?
Below is an image of how Flesch-Kincaid Grade Level predicts the text difficulties of two different text excerpts: The Town Mouse and the Country Mouse by Aesop and the Black Cat by Edgar Allan Poe.
There are a couple of things to notice. First, Flesch-Kincaid Grade Level thinks that this particular excerpt from the Town Mouse and the Country Mouse is at a 6th grade reading level, which a lot of people might not agree with. Second, you can see that it’s basing its predictions on two things, the number of syllables per word, and the number of words per sentence (they’re multiplied by certain numbers, but those details aren’t shown here), which sounds a bit too simplified. Third, you can also see that the two texts are rather similar in terms of the number of words per sentence. This means that according to Flesch-Kincaid Grade Level, the difference between the difficulty levels of the two texts only comes from one factor: the number of syllables per sentence.
Let’s take a look at predictions made by another metric, the New Dale-Chall.
This metric predicts that the Town Mouse and the Country Mouse story is at a 1st grade reading level, and the Black Cat is at around an 8th grade level, which seems to make more sense, but the results are very different from those reported by Flesch-Kincaid Grade Level . But we can still see that the prediction is only based on two factors: word difficulty and the average number of words per sentence.
Can we do
One problem with many of the popular readability formulas like Flesch-Kincaid Grade Level and New Dale-Chall is that they are dated and were developed using simple language features for prediction. They were also trained on very small datasets, which were all that were available when the Flesch-Kincaid Grade Level was developed in the 1970s. This means that what may have been the best available options at the time are not the best option presently. Brilliant minds back then put their heads together to come up with these metrics, but we have larger collections of texts and better methods to measure language features that allow us to look at factors that are more in-depth than average sentence length and average number of words.
As a result, we now have more advanced readability metrics like the Crowdsourced Algorithm of Reading Comprehension and the Coh-Metrix L2 Readability Index, both of which can be accessed easily through this website.
make things better
If Flesch-Kincaid Grade Level and other popular readability metrics are outdated, why are people still using them? And why have I never heard of these newer, improved, metrics?
There can be many different reasons, and one of them is that newer metrics developed by the research community are often not really accessible to the people who might actually need them: actual teachers, practitioners, and students. That’s because these newer metrics often involve the use of more complex programming tools than the older measures.
ARTE was developed to address this problem: to make newer readability formulas much more accessible to everyone who might be interested and bridge the research and practitioner communities.
If you’re a teacher or an educational content creator, you can look through the predictions made by the many different available readability formulas and decide for yourself which makes the most sense.
If you’re a researcher and would like to have your formula included in this model, reach out to us and we can help your research reach a wider audience.
Browse through the charts below to get a better sense of how all these different readability metrics work.
s
So what is ARTE?
ARTE is an online tool that provides you with a variety of different readability formulas that you can use to assess the difficulty of your texts.
Some of these metrics just give me a random number and it doesn’t make sense.
Some of these formulas are not standardized to a grade level. This means that their predictions only make sense when you’re comparing one text to another. What would be the best use for these formulas is to find a text that you’re absolutely sure is appropriately difficult for your students, and then compare whether other texts you may potentially be using are also within a reasonable range of that target text’s readability score.
There are too many readability formulas to choose from. Which one do I use?
We recommend that you try out the deep-learning-based formula, CommonLevel. Or the Crowdsourced Algorithm of Reading Comprehension.
I’m a researcher and I want to try out the API.
Reach out to us at sacrossley@gmail.com or jchoi92@gsu.edu and we’d be happy to discuss this with you.
Automated Readability Index (1975)
The Town Mouse rather turned up his long nose at this country fare, and said: "I cannot understand, Cousin, how you can put up with such poor food as this, but of course you cannot expect anything better in the country; come you with me and I will show you how to live. When you have been in town a week you will wonder how you could ever have stood a country life." No sooner said than done: the two mice set off for the town and arrived at the Town Mouse's residence late at night. "You will want some refreshment after our long journey," said the polite Town Mouse, and took his friend into the grand dining-room. There they found the remains of a fine feast, and soon the two mice were eating up jellies and cakes and all that was nice. Suddenly they heard growling and barking. "What is that?" said the Country Mouse. "It is only the dogs of the house," answered the other. "Only!" said the Country Mouse. "I do not like that music at my dinner."
CAREC: Crowdsourced Algorithm of Reading Comprehension (2019)
Note: A higher score means that the text is predicted to be more difficult by this model.
The Town Mouse rather turned up his long nose at this country fare, and said: "I cannot understand, Cousin, how you can put up with such poor food as this, but of course you cannot expect anything better in the country; come you with me and I will show you how to live. When you have been in town a week you will wonder how you could ever have stood a country life." No sooner said than done: the two mice set off for the town and arrived at the Town Mouse's residence late at night. "You will want some refreshment after our long journey," said the polite Town Mouse, and took his friend into the grand dining-room. There they found the remains of a fine feast, and soon the two mice were eating up jellies and cakes and all that was nice. Suddenly they heard growling and barking. "What is that?" said the Country Mouse. "It is only the dogs of the house," answered the other. "Only!" said the Country Mouse. "I do not like that music at my dinner."
Coh-Metrix L2 Reading Index (2008)
Note: A higher score means that the text is predicted to be easier by this model. This is an approximation of the original model.
The Town Mouse rather turned up his long nose at this country fare, and said: "I cannot understand, Cousin, how you can put up with such poor food as this, but of course you cannot expect anything better in the country; come you with me and I will show you how to live. When you have been in town a week you will wonder how you could ever have stood a country life." No sooner said than done: the two mice set off for the town and arrived at the Town Mouse's residence late at night. "You will want some refreshment after our long journey," said the polite Town Mouse, and took his friend into the grand dining-room. There they found the remains of a fine feast, and soon the two mice were eating up jellies and cakes and all that was nice. Suddenly they heard growling and barking. "What is that?" said the Country Mouse. "It is only the dogs of the house," answered the other. "Only!" said the Country Mouse. "I do not like that music at my dinner."
Flesch-Kincaid Grade Level (1975)
The Town Mouse rather turned up his long nose at this country fare, and said: "I cannot understand, Cousin, how you can put up with such poor food as this, but of course you cannot expect anything better in the country; come you with me and I will show you how to live. When you have been in town a week you will wonder how you could ever have stood a country life." No sooner said than done: the two mice set off for the town and arrived at the Town Mouse's residence late at night. "You will want some refreshment after our long journey," said the polite Town Mouse, and took his friend into the grand dining-room. There they found the remains of a fine feast, and soon the two mice were eating up jellies and cakes and all that was nice. Suddenly they heard growling and barking. "What is that?" said the Country Mouse. "It is only the dogs of the house," answered the other. "Only!" said the Country Mouse. "I do not like that music at my dinner."
New Dale-Chall (1995)
The Town Mouse rather turned up his long nose at this country fare, and said: "I cannot understand, Cousin, how you can put up with such poor food as this, but of course you cannot expect anything better in the country; come you with me and I will show you how to live. When you have been in town a week you will wonder how you could ever have stood a country life." No sooner said than done: the two mice set off for the town and arrived at the Town Mouse's residence late at night. "You will want some refreshment after our long journey," said the polite Town Mouse, and took his friend into the grand dining-room. There they found the remains of a fine feast, and soon the two mice were eating up jellies and cakes and all that was nice. Suddenly they heard growling and barking. "What is that?" said the Country Mouse. "It is only the dogs of the house," answered the other. "Only!" said the Country Mouse. "I do not like that music at my dinner."
SMOG (1969)
The Town Mouse rather turned up his long nose at this country fare, and said: "I cannot understand, Cousin, how you can put up with such poor food as this, but of course you cannot expect anything better in the country; come you with me and I will show you how to live. When you have been in town a week you will wonder how you could ever have stood a country life." No sooner said than done: the two mice set off for the town and arrived at the Town Mouse's residence late at night. "You will want some refreshment after our long journey," said the polite Town Mouse, and took his friend into the grand dining-room. There they found the remains of a fine feast, and soon the two mice were eating up jellies and cakes and all that was nice. Suddenly they heard growling and barking. "What is that?" said the Country Mouse. "It is only the dogs of the house," answered the other. "Only!" said the Country Mouse. "I do not like that music at my dinner."
SBERT (CommonLevel)
The deep-learning-based readability model is now available. This model performs well, but its decision making process cannot be broken down into human-understandable linguistic features.Read more about the Kaggle competition where deep learning based readability models were developed at this link.