Overview

Transcribed by Kanako Suwa

[ Introduction music plays ]

Welcome to the SENIA Happy Hour with your host, Lori Boll. We know you’re busy so we bring you 1 hour’s worth of content in under thirty minutes, leaving you with time for a true happy hour. 

Lori: Hello everyone. Today, I have the pleasure of speaking with Dr. Jack Naglieri. So some of you may recognise that last name. Yes. He is the creator of the Naglieri Non-Verbal Ability Test, which many of us have probably given in our experiences. So Jack and I speak about how traditional IQ tests do not actually measure how students think. Rather, they measure what they know which is inherently biased and an unfair practice. Today’s conversation is really eye-opening and hopefully gets us all thinking about changing the traditional system and the benefits of doing so. And now, onto the show. 

Hello Jack and welcome to the podcast. 

Dr. Jack: Hi Lori, nice to be here. Thank you for inviting me. 

Lori: Yeah, you bet! So I wanna start by framing the conversation around one of the problems you’ve set out to solve. And that is intelligence testing that evaluates students fairly and equitably. Tell us how you became interested in fair assessment of intelligence. 

Dr. Jack: This is really an important question and has a lot of parts. So I’ll start from the beginning. I actually first noticed that the intelligence test that I was taught to give was very similar to the achievement test I was taught to give when I worked as a school psychologist in 1975. And I was only a beginning 25 year old in the field but something didn’t feel right. And I remember thinking, these two tests are supposed to be measuring something different but I have a vocabulary test on the intelligence test and there’s vocabulary on the achievement test. There’s math word problems on the intelligence test and there’s math word problems on the achievement test and so on. When I went on for my PhD at the University of Georgia, there was some discussion about whether or not the so-called Verbal Intelligence Scale, or the WISC, was really measuring knowledge or measuring intelligence. But what really struck me was when I began working in Northern Arizona as a professor at Northern Arizona University and we started doing some work with Native American students. And I actually worked in the Supai village which is located inside the Grand Canyon. 

Lori: Oh wow. 

Dr. Jack: Working with students there who lived in a community, a tiny little community where there were no cars, where they were really isolated, really from the rest of the people in the state, and I was doing what I was taught to do. Giving a WISC to a student who hardly spoke English and mostly Supai language, she got a score on the verbal part of the WISC that was like, a 2nd percentile score. But her score on the performance, now what we would call non-verbal skill, was pretty average. And I wrote in my report that that score should not count as a measure of intelligence and I got a lot of grief from my colleagues who said “no, that measures intelligence. It’s verbal intelligence!” and these Native American people who, you know, live out in the desert, they have to have good spatial skills so their right brain is developed and left brain isn’t, and… they went on and on with these, kind of, explanations to which I responded, “that’s completely absurd”. You know, it’s just not right. 

So I published my first paper on equity in 1982, and my first test that I published, which eventually became the Naglieri Non-Verbal Ability Test, I published that in 1985. And I explicitly stated that measuring general intelligence with these kinds of tests is more valid and more fair because it’s not confounded by what a student knows. It measures how well a student can think, and a vocabulary test can measure how well a student can think, if they’ve been exposed to the content. But if they haven’t, then the basic assumption that the vocabulary is a good way to measure intelligence is violated. 

Lori: Very true. 

Dr. Jack: So that’s why, to me, critically important to always separate out when we’re trying to measure intelligence, to separate out how well a student can think from how well a student can solve problems based on knowledge that they should’ve or typically have been able to acquire in schools or in other contexts where advantage makes a difference. 

Lori: Right – and interestingly, much of our audience comes from the international field. Many of us are international school teachers or professionals or parents in the international circuit and you know, many of us have given your assessment to our students just for that reason. Because their language is different than what we’re assessing their intelligence test in, and it just made no sense to keep assessing them and having them score so dismally when we all knew that wasn’t their true score. 

Dr. Jack: That’s right. I would argue even more that even a student who knows the language that the intelligence test may be given in… let’s say, a WISC that’s been translated into some other language, and that language is used in a country where that language is spoken and the student knows the language, I would argue that still inappropriate. Because we should not be measuring intelligence in ways that depend on what people know. Because that’s two influence or variables that… you know, to be really honest and historically accurate, using those kinds of tests to evaluate people was pretty much the first approach in 1900’s, a hundred years ago and it was completely atheoretical and it was all based upon predicting achievement, which is a circular argument in the first place.

Lori: Right. So your whole position is that intelligence should measure how well a person can think to answer the questions on the test but they don’t do that very well so the majority of our intelligence tests that we’re giving students are basing it on what they know and not how they think. 

Dr. Jack: Yes. That’s true of all the tests. So the WISC, the Stanford-Binet, the Differential Ability Scales, the Woodcock Johnson, any of these kinds of tests that have been developed really in the last 100 years that demand knowledge. All these tests, in my opinion, are inaccurate. And think about this, for a second. We’ve experienced COVID. We know that students have not benefited from education the way they did before COVID. That renders all of those tests inaccurate. Because now you’re comparing students who are two years older but didn’t really get two years of education to students who didn’t have a gap like that. So this, this problem of the impact of knowledge on the intelligence scores is really serious and in fact, I have a research project that is almost finished, pre-post COVID project, where we show that in fact, if we measure intelligence in a way that is not influenced by what as student has learned in school, there is not that much difference pre- and post- COVID. But of course, when you look at achievement scores pre- and post- COVID, there are big differences. 

Lori: Ohhh I see. So that’s the impact that you were discussing. Okay. Got it. So when it comes to equitable identification of all students, it seems, that we keep doing the same thing and getting the same results, which as you were saying, are not great. So, we haven’t seen widespread change. So what are the obstacles causing that and how are you going about dismantling those? 

Dr. Jack: Yeah, so it’s a really fascinating and frustrating at the same time for me. Fascinating because it’s unfortunate, the extent to which people are unwilling to embrace new ideas about intelligence. People often say to me, well, we know what intelligence is, it’s what intelligence tests measure. And I ask, why would you assume that to be true? And the implications are so profound, especially as it relates to equity. So, fr example, I like to read old books about intelligence and I have a whole stack of them here in my office. Because it’s time travel, right? I have one book that I particularly love, it’s called “Intelligence Testing of Children at Ellis Island”. 

Lori: Oh. Wow.

Dr. Jack: Yeah, yeah. It’s great. And when I read it, I’m looking at the, what they said back then vis a vis perspective they had back then. So for example, if you look at all the early books, they were right from the very beginning comparing people. So Turnman is a good example – the author of Stanford Binet. What he knows and it was clearly racist and eugenics like many people were at that time period. And what do they do? They actually categorized people by backgrounds. Florence Goodenough, who I talk about a lot because I use the work in another context but I didn’t realize until recently that she did a lot of work on comparing racial and ethnic groups and one of her papers, she literally has a table of.. Average scores by white people, by Hispacnic people, by Native American people, by Irish people, by Polish people, Italian people, and so on. And you know what you’ve got.

Lori: Of course…

Dr. Jack: You got exactly what people expected. Well, of course these groups aren’t as smart as us. 

Lori: Right… rather than thinking of the test itself and what it means.

Dr. Jack: Right, exactly. What they did was they used the results of these new tests that they just created and they assumed that the tests were reflecting differences in people when the test was really reflecting the content of those tests, which are inappropriate for those people. But they arrogantly thought they were measuring intelligence and that’s why they said, these Italian people are stupid, you know, etc, etc, etc. Right? And it comes back to what they, what method they used to measure intelligence. Which is what basically what Binet and Army Alpha and Army Beta were all about. And this is somehow, it got solidified by Wechsler. Wechsler was a military examiner, he worked for the US military and the US military developed the Army Alpha and the Army Beta and what Wechsler did was take that test and make that into tests that clinicians can use. That was his original contribution when he worked at the Bellevue Hospital in Manhattan.

Lori: Wow. That is fascinating. 

Dr. Jack: And so Wechsler’s grip on this concept of intelligence, verbal and non-verbal, with some quantitative stuff, wasn’t his original idea. It came from the US military in 1917. And you know, people still think Wechsler is the gold standard. And I say, not even close. Because if you understand the history and you understand the lack of theoretical perspective that drove these tests, and in the way in which they were validating the predetermined views of the people of the time, it all starts to fit together. And why haven’t we been able to change? Because people think, “this is the way it is” and I took some sense of historical perspective when I read the book, “Guns, Germs, and Steel” and the author talked about automobiles and cars, horses and automobiles were on the same roads for 50 years before they managed to change it, you know. 

But a 100 years of intelligence testing and we’re still doing the same thing. It’s inexcusable and even given the evidence, there’s so much evidence about the damage that this approach creates. It’s still hard to get people to change. What I have done, is to help people understand the historical problem but then show all the evidence that I’ve accumulated with all my non-verbal work and also with my work on cognitive assessment system, which is a theoretically developed intelligence test based on brain function, not based upon what Wechsler did or what Binet did, and I’ve been able to show with decades of research, that when you change your conceptualization and your thinking in a way that is not confounded by knowledge, not only do you get equity but you get much more sensitivity from children with ADHD, children with dyslexia, children with autism and so on and so forth. 

So it’s frustrating, as I said frustrating and fascinating at the same time, but if you look at the data, if you look at it objectively, it’s really clear that we need to be thinking of intelligence from a) a theoretical perspective and b) a theoretical perspective based upon the brain and we need to have an instrument that is developed specifically across that, according to that perspective and we need to have the research that shows the validity of the different perspective. And that’s always been my approach. I don’t, I don’t advocate in the advancement of science. For example, CHC is a very popular approach these days in advocating for the advancement of science. Even if you read Carl Jones book, as some people call it bible now for cognitive abilities, in the last chapter first paragraph, he says of course we haven’t looked at the validity of these facts. Minor detail…

Lori: Very minor… 

Dr. Jack: Yeah, so that’s my perspective. And you know, I’ve taken, really decades of research to get to this point where I’m completely comfortable saying to people, you know what, what we’ve been doing is not the way it should be done, look at the evidence that I’ve developed with my Naglieri Non-Verbal and my cognitive assessment system, which clearly show benefits changing what we’ve done to measure thinking in a way that’s not confounded by knowing. 

Lori: Great, I mean… so you’ve created tests now that were carefully designed to measure thinking. 

Dr. Jack: Yes.

Lori: versus the knowing. So the test were designed to significantly decrease the amount of formal knowledge required so tests measure how well a student thinks. I think most people listening to this interview will really be excited to hear more about this. So let’s start by talking more about how you went about designing this intelligence test this way. 

Dr. Jack: Okay. I’m going to talk about the, really, the 3 groups of tasks that I’ve developed. The initial Naglieri Non-Verbal, which has been around since, actually 1985, it was originally called the Matrix Analogy test, that was the first version. In that first version, we measure general ability, this concept is a vague concept really but it has a lot of empirical support, we measure general ability using diagrams and figures. More recently, I’ve taken that approach and just actually just this month, we’ll publish 3 new non-verbal tests of general ability that are similar to my Naglieri Non-Verbal, we have a non-verbal version but we also have a verbal and a quantitative version. The thing is, the verbal and the quantitative can be solved regardless of the language the person speaks. The directions are given with an animated video instead of oral instructions. And none of the content in that quantitative or the verbal involves a specific language. And we’ve shown with research studies that these 3 tests don’t yield gender differences, race differences, ethnic differences, or parental education level differences. And that’s the study, that’s actually 3 studies with about 7000 students, K-12, who were carefully selected to match the US population for generalisability to the US. 

Lori: Wow, that’s really fascinating. 

Dr. Jack: So you can create traditional general ability tests that are more equitable – I’ve done it. But here’s the thing. If you want to understand why a student can read, if you want to understand why is a student impulsive and disorganized and seem incapable of getting things done, if you want to understand if it’s a cognitive component to that student who can focus really really well but can’t shift from one thing to the other, when you want to understand why students who can read just fine but can’t comprehend what they’re reading, you have to use a different approach. And that’s where my cognitive assessment system comes in. Because with that, my colleague and I, in the mid 80’s, we conceptualized intelligence on the basis of brain function. 

So what does that mean? It sounds kind of… almost scary. But it’s really simple. Because Lorea’s work and a lot of the work thereafter show that basically 4 main regions of our brains. The first is the brain stem, which of course is important for cortical arousal and focus of attention. The back part of the brain, occipital and parietal is what we use when we understand how things go together. In other words, you can understand it as visual and spatial but that doesn’t have to be visual, it could be words that describe relationships. 

Lori: Like concepts.

Dr. Jack: The temporal part of our brain is all about sequencing. Sequencing of anything – words, numbers, speech articulation, reading decoding, phonological tests. The front part of the brain is how you do what you decide to do. That’s this concept of executive function. So what we did was we said intelligence has 4 components and we met and we developed tasks, which are now in the second edition, to measure those 4 components which is called the Cognitive Assessment System, Second Edition. And we measure these 4 abilities and we find that students with dyslexia, students who can’t decode the words, they have trouble on all sequencing tasks. Temporal part. Students who have trouble with reading comprehension, they have trouble with the simultaneous tasks, which is the back part of the brain. Students with ADHD, hyper-active impulsive types, they have trouble with the planning portion of the test, which is the front part of the brain. But the inattentive type of students, they have trouble with the attention scale on our test. 

Lori: That is fascinating. That is really interesting. 

Dr. Jack: Different profiles for different students based upon really their learning profiles. Because that’s really what these four brain concepts are all about. That’s what intelligence should be about. How well you can learn. 

Lori: Well, I think what it also tells me is something we can… when we know what area it is, we can do something about it. So often we get these scores that really tell us nothing. So… what an important component. 

Dr. Jack: Yeah, that’s correct. And I’ve published a book of interventions that go along with the theory and the task. Interventions that are designed explicitly for teachers and parents, and even for students themselves, to use their strengths to manage their weaknesses. So a student who has trouble with decoding of words because they have trouble with sequencing, we would first help them understand the nature of that problem and say, okay, now when you get stuck, think smart and use a plan. A plan could be chunking the information into smaller segments. 

Lori: Yep. Perfect. Let me ask you a question. For someone, let’s say, has been identified as you know, this is kind of old terminology, but profound autism or severe autism, can your test work for someone on that scale to measure? 

Dr. Jack: Yeah, we’ve used the diagnostic assessments for decades for a full range, very low ability to a very high ability. 

Lori: Hm. Fascinating. So what’s the benefit? Well, I think the benefit is, I think I can answer this, is that by having the assessment that you’ve now created, we can use their strengths to support them and use the strategies you’ve come up with to support them – the interventions. Is that correct? 

Dr. Jack: Yes. We can see the students for who they are and we can communicate to the students what we found in ordinary language – I mean, think about it. Isn’t that the most important person to tell? When you’ve done an evaluation of a student? And when that student says to you, Oh yeah, that’s right, I just can’t concentrate… that’s an important step for that student to understand here’s the problem and let’s talk about what we’re gonna do about it. 

Lori: I love it. So is there a book or a film or any other resources that you have that have had an influence on you either personally or professionally that you’d recommend to us? 

Dr. Jack: The books that I’ve read, you mean? 

Lori: Yeah, that you recommend! 

Dr. Jack: Well, I think that Luria’s book on language and cognition is one of my favorites and I think it’s, for a lot of reasons. It talks a lot about what I call past for planning, attention simultaneous, and success to processing. I think going back and reading some of the older books on non-verbal assessment, that was a really important one for me. I’ll tell you something interesting, I found out that Rudolph Pitner was a professor at Ohio State University as I was, in the same building that I was in, about 40 years before me. So I thought that was kind of interesting! 

Lori: Wow! 

Dr. Jack: Yeah, but anyway, I think that… that says all. 

Lori: Yeah, I think that’s cool! We’ll add those resources to the show notes so people can grab them. 

Dr. Jack: And I would encourage people to go to my two websites – www.Dr. Jacknaglieri.com – that website has a lot of information about my work on the cognitive assessment system and other things. I work on autism spectrum rating scale, my executive functioning rating scale, I work with a social emotional test and so on. And for more specifically for gifted, look at www.naglierigiftedtests.com and people can also get in touch with me via those websites. 

Lori: Okay, great! We’ll put those on our resource list as well. So, Jack. You are going to be a speaker at our upcoming virtual conference. 

Dr. Jack: That’s right, I’m looking forward to that.

Lori: Yeah, we’re really excited and I think it’s going to help so many of us. So can you just allude to what you will be speaking about? 

Dr. Jack: Yeah. What I would really like to talk about is this concept of measuring thinking rather than knowing and as it relates to trying to understand the strengths and struggles of students, we really need to talk about posterior intelligence, which is what the Cognitive Assessment System was designed to measure, and where it came from, how well it can be used to really help us do a better job of helping students learn. 

Lori: That’s great. I think that all of us would benefit and hopefully create some positive change after a 100 years of not changing so positive change to work with the students’ strength. Okay, well, that’s a great way to end so thank you so much for your time and knowledge! 

Dr. Jack: You’re quite welcome and I’m really glad we had the chance to chat today. 

Lori: Yeah, me too! Thanks. 

Dr. Jack: Bye. 

Thank you for joining us for today’s show. For more information including how to subscribe and show notes, please head to our website. That’s SENIAinternational.org/podcast. Until next time, cheers.