Misconceptions Study Results, Teaching How LLMs Work
by matthias.hauswirth＠usi.ch
Dear all,
It’s been over half a year without news from us. We have been busy here at USI…
Misconceptions Study Results!
Many of you participated in the big misconceptions study, and we promised we would share the results. Now is the time! We finished the analysis and wrote it up as a paper titled “Surveying Upper-Secondary Teachers on Programming Misconceptions”, which was accepted at ICER. Remember, the study involved 16 Python misconceptions, and you had to work through a whopping 55 pages of questions? The survey consisted of three parts: (1) demographics, (2) conceptual questions, and (3) report on observations and importance.
Overall, 97 high school informatics teachers from the German, French, and Italian-speaking parts of Switzerland participated. Quite a few participants mentioned in the comments that they might have had some misconceptions themselves. One of the reasons behind the second part (conceptual questions) was to determine which participants had which misconceptions themselves, so we could then separate the results of the third part (observations and importance) based on participants’ understanding. Prior studies did not distinguish responses based on whether or not a respondent had that specific misconceptions themselves. One of the take-home messages of our study is that neglecting this distinction can significantly affect the results. In the paper you can see this effect. You also can see that you definitely were not alone if you made a mistake somewhere.
Besides that, the paper reports all the specific data you probably were wondering about: which misconceptions were most common amongst your peers, which misconceptions are reported as most prevalent, which ones are considered most important, and, maybe most interesting, which strategies and techniques your peers use to spot and correct those misconceptions in their students?
You find answers to all these questions in the paper, which you can find here:
https://chiodini.org/publications/icer25-misconceptions-survey.pdf
Once again, we would like to profoundly thank everyone who participated in this study! We read and analyzed every single response in detail, and it was great to see how thoughtful everyone was. We hope your students are equally caring and thoughtful when they answer your questions.
If you’d like to look at the different misconceptions, and at some questions you could ask to detect them, check out the PyTamaro curriculum we put together about the 16 misconceptions we included in the study (one activity per misconception, plus a couple of extras, with links to progmiscon). In the activities you can experiment with the example code to convince yourself of the right answer.
https://pytamaro.si.usi.ch/curricula/luce/misconceptions
Oh, and we integrated the results from the study into the misconception pages on progmiscon. Here is an example:
https://progmiscon.org/misconceptions/Python/VariablesHoldExpressions/
Like I stated in an earlier post, I do not think that programming education should exclusively focus on misconceptions. Being competent in programming requires more than a correct conceptual understanding.
How Does ChatGPT Work, Behind the Scenes?
You teach informatics, and you probably discuss "Artificial Intelligence" with your students. Products such as ChatGPT are widely used today, especially by students. But how do they work? Behind the scenes, these products are powered by "Large Language Models" (LLMs). Understanding what language models are is key to understanding the strengths and the limitations of these products.
Luca put together a wonderful activity focusing on the essence of language models. Starting from scratch, just with Python, you and/or your students can build a *Little* Language Model (which you can still call an LLM :) ) that generates text sounding like Dante, or like Shakespeare, depending on what you train it on (we haven’t tried Goethe, or Molière, but if you teach in German or French, they would be great examples). The model is little, so training takes only a few seconds (it’s short enough so you can even run it on the PyTamaro Web site). Luca’s activity guides you through a progression of language models, from the most trivial model to the one that sounds surprisingly good.
https://pytamaro.si.usi.ch/curricula/luce/language-model
We have used a short version of the activity in hour-long presentations to families visiting USI, and a longer version in workshops where participants implemented the key parts themselves (in a short curriculum on PyTamaro Web). Even within an hour, if you essentially do live programming in front of your students, and ask them questions throughout, you can get a great discussion going that involves aspects such as bias in training and privacy, in addition to instilling a dose of “wonder” about how something so simple can lead to such surprising results.
Have a great summer! All the best from Lugano,
Matthias and the LuCE team
PS: You can find this and the previous message in our mailing list archive:
https://lists.usi.ch/mailman3/hyperkitty/list/luce-news@lists.usi.ch/
If you would like to unsubscribe from our list, or if you would like to update your email address, simply let me (Matthias.Hauswirth(a)usi.ch) know. You also can do it yourself here:
https://lists.usi.ch/mailman3/postorius/lists/luce-news.lists.usi.ch/
1 week