Time: Summer term 2023, Mondays 12:15-13:45; start: 24. April 2023 Venues: at HU: Bernstein Center Lecture Hall at HU Berlin; at FU: lecture hall KL 32/123, Habelschwerdter Allee 45; on zoom: (link TBC)

Language has been investigated from a range of perspectives. Linguists have described it as a formal system focusing on levels that range from phonology to syntax, semantics and pragmatics. Both linguists and psychologists worked on models focusing on the time course of linguistic processing, so that these psycholinguistic models could be tested in behavioral experiments. Neuro- and cognitive scientists have attempted to spell out the brain mechanisms of language in terms of neuronal structure and function by specifying language-relevant areas, ‘networks’, neuronal assemblies and their interactions. Most recently, explicit biologically inspired modelling and neural network research aim at imitating and explaining language circuits in the human brain, following Feynman’s insight that “What I cannot create, I do not understand”. These efforts are founded in neuroscience data about the event-related brain potentials and the brain loci that activate when specific linguistic operations occur, the time course of their activation and the linguistic effects of focal brain lesions. The lecture series will provide a broad introduction into these linguistic, psycholinguistic and neurolinguistic research streams and highlight a variety of cutting-edge behavioral, neuroscience and computational findings addressing a broad range of linguistic issues, including, for example, the recognition of words, the parsing of sentences, the computation of the meaning and of the communicative function of language. Likewise, language development and language disorders will be in focus. Further emphasis will lie on theoretical and computational models of language processing built by psycho- and neurolinguists, which range from abstract box-and-arrow diagrams of the language (processing) system to computationally implemented models and neural network models mimicking the structure and function of the human brain. To evaluate these models, we will review experimental findings involving a broad range of behavioral (reaction time studies, eye tracking), neuroimaging (EEG, MEG, fMRI, NIRS) and neuropsychological methods (patient studies, TMS, tDCS). Complementing the lecture series, a tutorial will be offered jointly by Johanna Knechtges, research assistant at the Brain Language Laboratory of the Freie Universität, and Angela Patarroyo, PhD candidate in the SFB 1412 “Register”. The tutorial will deepen the lecture contents, in part by discussing relevant articles with theoretical and experimental focus. Together with the lectures, the tutorial will familiarize students with current research in the field of language and the brain. This lecture series is open to students at the Berlin School of Mind and Brain as well as for students of linguistics at HU and FU Berlin.

Readings (recommended for course preparation): Knoeferle, P., & Guerra, E. (2016). Visually situated language comprehension. Linguistics and Language Compass, 10(2), 66–82. doi: 10.1111/lnc3.12177 Knoeferle, P. (2021). Grounding language processing: The added value of specifying linguistic/compositional representations and processes. Journal of Cognition, 4, 1-14, doi: 10.5334/joc.155. Pulvermüller, F. (2018). Neural reuse of action perception circuits for language, concepts and communication. Progress in Neurobiology, 160, 1-44. doi: 10.1016/j.pneurobio.2017.07.001 Pulvermüller, F., Tomasello, R., Henningsen-Schomers, M. R., & Wennekers, T. (2021). Biological constraints on neural network models of cognitive function. Nature Reviews Neuroscience, 22(8), 488-502. doi: 10.1038/s41583-021-00473-5 Lecture Series “Language and the Brain”

Berlin School of Mind and Brain, Summer Term 2023

Lecturers: Friedemann Pulvermüller (FP) & Pia Knoeferle (PK)

FP’s lectures will be held at FU, PK’s ones at HU. Lectures will be broadcasted to the respective other location. Please note: References highlighted by an asterisk (*) are required readings and form, together with the content of the lectures, the basis of the final exam. Additional recommendations for readings are given for interested participants.

01. FP From Brain Structure to Linguistic Function 24.04 *Pulvermüller, F. (2002). The neuroscience of language: on brain circuits of words and serial order (pp. 9–32). Cambridge University Press, Chapters 2 and 3. Schomers, M. R., Garagnani, M., & Pulvermüller, F. (2017). Neurocomputational consequences of evolutionary connectivity changes in perisylvian language cortex. Journal of Neuroscience, 2693-16. doi:10.1523/JNEUROSCI.2693-16.2017 (in depth: Pulvermüller, F. (2018). Neural reuse of action perception circuits for language, concepts and communication. Progress in Neurobiology, 160, 1-44: Sections 1 & 2. doi: 10.1016/j.pneurobio.2017.07.001)

02. FP Meaning in mind and brain 08.05 * Pulvermüller, F. (2013). How neurons make meaning: Brain mechanisms for embodied and abstract-symbolic semantics. Trends in Cognitive Sciences, 17(9), 458-470. doi: 10.1016/j.tics.2013.06.004 Grisoni, L., Tomasello, R., & Pulvermüller, F. (2021). Correlated Brain Indexes of Semantic Prediction and Prediction Error: Brain Localization and Category Specificity. Cerebral Cortex, 31(3), 1553-1568. doi: 10.1093/cercor/bhaa308 (in depth: Pulvermüller, F. (2018). Neural reuse of action perception circuits for language, concepts and communication. Progress in Neurobiology, 160, 1-44, Section 3.2. doi: 10.1016/j.pneurobio.2017.07.001)

03. FP Speech Acts and Communication 15.05 *Tomasello, R., Kim, C., Dreyer, F. R., Grisoni, L., & Pulvermüller, F. (2019). Neurophysiological evidence for rapid processing of verbal and gestural information in understanding communicative actions. Scientific Reports, 9(1), 16285. doi:10.1038/s41598-019-52158-w Hagoort, P., & Levinson, S. C. (2014). Neuropragmatics. In M. S. Gazzaniga (Ed.), The cognitive neurosciences (pp. 667-674). Boston, MA: MIT Press. 

04. FP Language Breakdown and Therapy 22.05 *Blumstein, S. E. (2016). Psycholinguistic approaches to the study of syndromes and symptoms of aphasia. In G. Hickok & S. L. Small (Eds.), Neurobiology of language (pp. 923-933). Amsterdam: Elsevier. Pulvermüller, F., Mohr, B., & Taub, E. (2016). Constraint-induced aphasia therapy: A neuroscience-centered translational method. In G. Hickok & S. L. Small (Eds.), Neurobiology of language (pp. 1025-1034). Amsterdam: Elsevier.

05. FP Brain Constrained Neural Language Modelling 05.06 * Pulvermüller, F., Tomasello, R., Henningsen-Schomers, M. R., & Wennekers, T. (2021). Biological constraints on neural network models of cognitive function. Nature Reviews Neuroscience, 22(8), 488-502. doi:10.1038/s41583-021-00473-5 Henningsen-Schomers, M. R., Garagnani, M., & Pulvermüller, F. (2023). Influence of language on perception and concept formation in a brain-constrained deep neural network model. Philos Trans R Soc Lond B Biol Sci, 378(1870), 20210373. https://doi.org/10.1098/rstb.2021.0373 Tomasello, R., Wennekers, T., Garagnani, M., & Pulvermüller, F. (2019). Visual cortex recruitment during language processing in blind individuals is explained by Hebbian learning. Scientific Reports, 9(1), 3579. doi:10.1038/s41598-019-39864-1

06. PK Knowledge and processing 12.06 Traxler, M. J. (2012). Speech Production and Comprehension. In: Introduction to Psycholinguistics: Understanding Language Science (First, pp. 37–78). Blackwell Publishing Ltd. Knoeferle, P., & Guerra, E. (2016). Visually situated language comprehension. Linguistics and Language Compass, 10(2), 66–82. doi: 10.1111/lnc3.12177 *Knoeferle, P. (2021). Grounding language processing: The added value of specifying linguistic/compositional representations and processes. Journal of Cognition, 4, 1-14, doi: 10.5334/joc.155.

07. PK Methods: Eye-tracking and ERPs 19.06 Cooper, R. (1974). The Control of Eye Fixation by the Meaning of Spoken Language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84–107. *Pyykkönen-Klauck, Pirita, and Matthew W. Crocker. (2016). Attention and eye movement metrics in visual world eye tracking. In: P. Knoeferle, P. Pykkönnen-Klauck and M.W. Crocker (eds). Visually situated language comprehension. (67-82). Amsterdam: John Benjamins Publishing. Ronderos, C.R., Münster, K., Guerra, E., Kreysa, H., Rodriguez, A., Kröger, J., Kluth, T., Burigo, M., Abashidze, D., Nunnemann, N., & Knoeferle, P. (2018). Eye Tracking during visually-situated language comprehension: Flexibility and limitations in uncovering visual context effects. Journal of Visualized Experiments, 141, e57694, doi:10.3791/57694 *Kutas, M., & Federmeier, K. D. (2007). Event-related brain potential (ERP) studies of sentence processing. In: G. Gaskell (Ed.), The Oxford Handbook of Psycholinguistics. doi:10.1093/oxfordhb/9780198568971.013.0023

08. PK Processing syntactic structure 26.06 C. Clifton Jr., & A. Staub (2011). Syntactic influences on eye movements during reading. In: Simon Liversedge, Iain Gilchrist, and Stefan Everling (eds). The Oxford Handbook of Eye movements. (pp. 895-910). Oxford University Press. Knoeferle, P., Habets, B., Crocker, M. W., & Münte, T. F. (2008). Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18(4), 789–795. doi:10.1093/cercor/bhm121. *Van Gompel, R., & J. Järvikivi (2016). The role of syntax in sentence and referential processing. In: P. Knoeferle, P. Pykkönnen-Klauck and M.W. Crocker (eds). Visually situated language comprehension. (83-126). Amsterdam: John Benjamins Publishing.

09. PK Semantic interpretation 03.07 Chambers, C. (2016). The role of affordances in visually situated language comprehension. In: P. Knoeferle, P. Pykkönnen-Klauck and M.W. Crocker (eds). Visually situated language comprehension. (205-226). Amsterdam: John Benjamins Publishing. *Engelhardt, P. and F. Ferreira (2016). Reaching sentence and reference meaning. In: P. Knoeferle, P. Pykkönnen-Klauck and M.W. Crocker (eds). Visually situated language comprehension. (127-150). Amsterdam: John Benjamins Publishing. Knoeferle, P., Urbach, T. P., & Kutas, M. (2014). Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification. Acta Psychologica, 152, 133–148. doi:10.1016/j.actpsy.2014.08.004

10. PK Social aspects in language processing 10.07 Mishra, R. K., Singh, N., Pandey, A., & Huettig, F. (2012). Spoken language-mediated anticipatory eye movements are modulated by reading ability: Evidence from Indian low and high literates. Journal of Eye Movement Research, 5(1), 1–10. doi:10.16910/jemr.5.1.3 *Münster, K., & Knoeferle, P. (2018). Extending situated language comprehension (accounts) with speaker and comprehender characteristics: Toward socially situated interpretation. Frontiers in Psychology, 8. Van Berkum, J. J. A., Van den Brink, D., Tesink, C., Kos, M., and Hagoort, P. (2008). The neural integration of speaker and message. Joural of Cognitive Neuroscience, 20, 580–591. doi:10.1162/jocn.2008.20054

11. PK & FP Exam Review (at HU) 17.07

12. Final exam (at HU) 24.07