Does AI actually help students learn? A recent experiment in a high school provides a cautionary tale.
Researchers at the University of Pennsylvania found that Turkish high school students who had access to ChatGPT while doing practice math problems did worse on a math test compared with students who didn’t have access to ChatGPT. Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning.
A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better. Students who just did their practice problems the old fashioned way — on their own — matched their test scores.
I’ve found AI helpful in asking for it to explain stuff. Why is the problem solved like this, why did you use this and not that, could you put it in simpler terms and so on. Much like you might ask a teacher.
To an extent, but it’s often just wrong about stuff.
It’s been a good second step for things I have questions about that I can’t immediately find good search results for. I don’t wanna get off topic but I have major beef with Stack Overflow and posting questions there makes me anxious as hell because I’ll do so much diligence to make sure it is clear, reproducible, and not a duplicate only for my questions to still get closed. It’s a major fucking waste of my time. Why put all that effort in when it’s still going to get closed?? Anyways – ChatGPT never gets mad at me. Sure, it’s often wrong as hell but it never berates me or makes me feel stupid for asking a question. It generally gets me close enough on topics that I can search for other terms in search engines and get different results that are more helpful.
Yep. My first interaction with GPT pro lasted 36 hours and I nearly changed my religion.
AI is the best thing to come to learning, ever. If you are a curious person, this is bigger than Gutenberg, IMO.
That sounds like a manic episode
I think this works great if the student is interested in the subject, but if you’re just trying to work through a bunch of problems so you can stop working through a bunch of problems, it ain’t gonna help you.
I have personally learned so much from LLMs (although you can’t really take anything at face value and have to look things up independently, but it gives you a great starting place), but it comes from a genuine interest in the questions I’m asking and things I dig at.
I have personally learned so much from LLMs
No offense but that’s what the article is also highlighting, naming that students, even the good, believe they did learn. Once it’s time to pass a test designed to evaluate if they actually did, it’s not that positive.
Because AI and previously google searches are not a substitute for having knowledge and experience. You can learn by googling something and reading about how something works so you can figure out answers for yourself. But googling for answers will not teach you much. Even if it solves a problem, you won’t learn how. And won’t be able to fix something in the future without googling th answer again.
If you dont learn how to do something, you won’t be experienced enough to know when you are doing it wrong.
I use google to give me answers all the time when im problem solving. But i have to spend a lot more time after the fact to learn why what i did fixed the problem.
Nope, this doesn’t work like this. sometimes you need someone to explain, specially on math, youtube can take that spot, but not always.
That’s what i am saying. You need to learn it. If someone explains it to you, then you are learning. If someone gives you the answer, then you dont understand it, so you are less good at said something.
You agree with me…
This isn’t a new issue. Wolfram alpha has been around for 15 years and can easily handle high school level math problems.
Except wolfram alpha is able to correctly explain step by step solutions. Which was an aid in my education.
Only old farts still use Wolfram
What do young idiots use?
I can’t remember, but my dad said before he retired he would just pirate Wolfram because he was too old to bother learning whatever they were using. He spent 25 years in academia teaching graduate chem-e before moving to the private sector. He very briefly worked with one of the Wolfram founders at UIUC.
ChatGPT apparently lol
Where did you think you were?
Perhaps unsurprisingly. Any sort of “assistance” with answers will do that.
Students have to learn why things work the way they do, and they won’t be able to grasp it without going ahead and doing every piece manually.
Taking too many shortcuts doesn’t help anyone learn anything.
Kids who take shortcuts and don’t learn suck at recalling knowledge they never had…
The only reason we’re trying to somehow compromise and allow or even incorporate cheating software into student education is because the tech-bros and singularity cultists have been hyping this technology like it’s the new, unstoppable force of nature that is going to wash over all things and bring about the new Golden Age of humanity as none of us have to work ever again.
Meanwhile, 80% of AI startups sink and something like 75% of the “new techs” like AI drive-thru orders and AI phone support go to call centers in India and Philippines. The only thing we seem to have gotten is the absolute rotting destruction of all content on the internet and children growing up thinking it’s normal to consume this watered-down, plagiarized, worthless content.
Actually if you read the article ChatGPT is horrible at math a modified version where chatGPT was fed the correct answers with the problem didn’t make the kids stupider but it didn’t make them any better either because they mostly just asked it for the answers.
Good tl;dr
I took German in high school and cheated by inventing my own runic script. I would draw elaborate fantasy/sci-fi drawings on the covers of my notebooks with the German verb declensions and whatnot written all over monoliths or knight’s armor or dueling spaceships, using my own script instead of regular characters, and then have these notebook sitting on my desk while taking the tests. I got 100% on every test and now the only German I can speak is the bullshit I remember Nightcrawler from the X-Men saying. Unglaublich!
I just wrote really small on a paper in my glasses case, or hidden data in the depths of my TI86.
We love Nightcrawler in this house.
Meanwhile the teacher was thinking, “interesting tactic you’ve got there, admiring your art in the middle of a test”
God knows what he would have done to me if he’d caught me. He once threw an eraser at my head for speaking German with a Texas accent. In his defense, he grew up in a post-war Yugoslavian concentration camp.
Unsurprised
I.would have no problem with AI if the shit actually worked
No, I think the point here is that the kids never learned the material, not that AI taught them the wrong material (though there is a high possibility of that).
Yes yet there is indeed a deeper point. If the AI is to be used as a teaching tool it still has to give genuinely useful advice. No good sounding advice that might actually still be wrong. LLMs can feed wrong final answers but they can also make poor suggestions on the process itself too. So there are both problematic, how the tool is used but also its intrinsic limitations.
There are a part here that sounds interesting
The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better.
Do you think that these students that used ChatGPT can do the exercises “the old fashioned way”? For me it was a nightmare try to resolve a calculus problem just with the trash books that doesn’t explain a damn fuck, I have to go to different resources, wolphram, youtube, but what happened when there was a problem that wasnt well explained in any resource?. I hate openAI, I want to punch Altman in the face. But this doesn’t mean we have to bait this hard in the title.
Maybe it’s an accident but you left out your fellow students and the teacher, in my eyes the most useful resources
It’s not about using it. It’s about using it ina helpful and constructive manner. Obviously no one’s going to learn anything if all they do is blatantly asking for an answer and writings.
LLM has been a wonderful tool for me to further understand various topics.
Obviously no one’s going to learn anything if all they do is blatantly asking for an answer and writings.
You should try reading the article instead of just the headline.
The researchers believe the problem is that students are using the chatbot as a “crutch.” When they analyzed the questions that students typed into ChatGPT, students often simply asked for the answer. Students were not building the skills that come from solving the problems themselves.
I did? What are you trying to say?
If you actually read the article you will see that they tested both allowing the students to ask for answers from the LLM, and then limiting the students to just ask for guidance from the LLM. In the first case the students did significantly worse than their peers that didn’t use the LLM. In the second one they performed the same as students who didn’t use it. So, if the results of this study can be replicated, this shows that LLMs are at best useless for learning and most likely harmful. Most students are not going to limit their use of LLMs for guidance.
You AI shills are just ridiculous, you defend this technology without even bothering to read the points under discussion. Or maybe you read an LLM generated summary? Hahahaha. In any case, do better man.
If you’d have read tye article, you would have learned that there were three groups, one with no gpt, one where they just had gpt access, and another gpt that would only give hints and clues to the answer, but wouldn’t directly give it.
That third group tied the first group in test scores. The issue was that chat gpt is dumb and was often giving incorrect instructions on how to solve the answer, or came up with the wrong answer. I’m sure if gpt were capable of not giving the answer away and actually correctly giving instructions on how to solve each problem, that group would have beaten the no gpt group, easily.
This! Don’t blame the tech, blame the grown ups not able to teach the young how to use tech!
The study is still valuable, this is a math class not a technology class, so understanding it’s impact is important.
Yea, did not read that promptengineered chatGPT was better than non chatGPT class 😄 but I guess that proofs my point as well, because if students in group with normal chatGPT were teached how to prompt normal ChatGPT so that it answer in a more teacher style, I bet they would have similar results as students with promtengineered chatGPT
Can I blame the tech for using massive amounts of electricity, making e.g. Ireland use more fossil fuels again?
I mean, is it really that surprising? You’re not analyzing anything, an algorithm just spits text at you. You’re not gonna learn much from that.
In the study they said they used a modified version that acted as a tutor, that refused to give direct answers and gave hints to the solution instead.
So it’s still not surprising since ChatGPT doesn’t give you factual information. It just gives you what it statistically thinks you want to read.
That’s like cheating with extra steps.
Ain’t getting hints on your in class exam.
You could always try reading the article
Which, in a fun bit of meta, is a decent description of artificial “intelligence” too.
Maybe the real ChatGPT was the children we tested along the way
At work we give a 16/17 year old, work experience over the summer. He was using chatgpt and not understanding the code that was outputing.
I his last week he asked why he doing print statement something like
print (f"message {thing} ")Sounds like operator error because he could have asked chatGPT and gotten the correct answer about python f strings…
Students first need to learn to:
- Break down the line of code, then
- Ask the right questions
The student in question probably didn’t develop the mental faculties required to think, “Hmm… what the ‘f’?”
A similar thingy happened to me having to teach a BTech grad with 2 years of prior exp. At first, I found it hard to believe how someone couldn’t ask such questions from themselves, by themselves. I am repeatedly dumbfounded at how someone manages to be so ignorant of something they are typing and recently realising (after interaction with multiple such people) that this is actually the norm[1].
and that I am the weirdo for trying hard and visualising the C++ abstract machine in my mind ↩︎
No. Printing statements, using console inputs and building little games like tic tac toe and crosswords isn’t the right way to learn Computer Science. It is the way things are currently done, but you learn much more through open source code and trying to build useful things yourself. I would never go back to doing those little chores to get a grade.
I would never go back to doing those little chores to get a grade.
So either you have finished obtaining all the academic certifications that require said chores, or you are going to fail at getting a grade.
It all depends on how and what you ask it, plus an element of randomness. Remember that it’s essentially a massive text predictor. The same question asked in different ways can lead it into predicting text based on different conversations it trained on. There’s a ton of people talking about python, some know it well, others not as well. And the LLM can end up giving some kind of hybrid of multiple other answers.
It doesn’t understand anything, it’s just built a massive network of correlations such that if you type “Python”, it will “want” to “talk” about scripting or snakes (just tried it, it preferred the scripting language, even when I said “snake”, it asked me if I wanted help implementing the snake game in Python 😂).
So it is very possible for it to give accurate responses sometimes and wildly different responses in other times. Like with the African countries that start with “K” question, I’ve seen reasonable responses and meme ones. It’s even said there are none while also acknowledging Kenya in the same response.
Im afraid to ask, but whats wrong with that line? In the right context thats fine to do no?
Yea, this highlights a fundamental tension I think: sometimes, perhaps oftentimes, the point of doing something is the doing itself, not the result.
Tech is hyper focused on removing the “doing” and reproducing the result. Now that it’s trying to put itself into the “thinking” part of human work, this tension is making itself unavoidable.
I think we can all take it as a given that we don’t want to hand total control to machines, simply because of accountability issues. Which means we want a human “in the loop” to ensure things stay sensible. But the ability of that human to keep things sensible requires skills, experience and insight. And all of the focus our education system now has on grades and certificates has lead us astray into thinking that the practice and experience doesn’t mean that much. In a way the labour market and employers are relevant here in their insistence on experience (to the point of absurdity sometimes).
Bottom line is that we humans are doing machines, and we learn through practice and experience, in ways I suspect much closer to building intuitions. Being stuck on a problem, being confused and getting things wrong are all part of this experience. Making it easier to get the right answer is not making education better. LLMs likely have no good role to play in education and I wouldn’t be surprised if banning them outright in what may become a harshly fought battle isn’t too far away.
All that being said, I also think LLMs raise questions about what it is we’re doing with our education and tests and whether the simple response to their existence is to conclude that anything an LLM can easily do well isn’t worth assessing. Of course, as I’ve said above, that’s likely manifestly rubbish … building up an intelligent and capable human likely requires getting them to do things an LLM could easily do. But the question still stands I think about whether we need to also find a way to focus more on the less mechanical parts of human intelligence and education.
LLMs likely have no good role to play in education and I wouldn’t be surprised if banning them outright in what may become a harshly fought battle isn’t too far away.
While I agree that LLMs have no place in education, you’re not going to be able to do more than just ban them in class unfortunately. Students will be able to use them at home, and the alleged “LLM detection” applications are no better than throwing a dart at the wall. You may catch a couple students, but you’re going to falsely accuse many more. The only surefire way to catch them is them being stupid and not bothering to edit what they turn in.
Would kids do better if the AI doesn’t hallucinate?
Would snails be happier if it kept raining? What can we do to make it rain forever and all time?
Paradoxically, they would probably do better if the AI hallucinated more. When you realize your tutor is capable of making mistakes, you can’t just blindly follow their process; you have to analyze and verify their work, which forces a more complete understanding of the concept, and some insight into what errors can occur and how they might affect outcomes.
God I miss ytmnd https://owleyes.ytmnd.com/
Haven’t seen that in ages. Thanks.
Kids using an AI system trained on edgelord Reddit posts aren’t doing well on tests?
Ya don’t say.
