September, 2015: Our paper on a distributional learning approach to selective adaptation is in press at Psychonomic Bulletin and Review! It's based on recent work on distributional learning and on sensory adaptation as a computational property of sensory systems, rather than a mechanistic one. Read the accepted version here and let us know what you think!

August, 2015: I've posted an R package that has the data and analyses for the studies on supervised and unsupervised phonetic adaptation that I presented at CUNY and CogSci this summer.

March, 2015: My paper with Florian Jaeger on modeling phonetic adaptation as Bayesian inference is in press at Psychological Review! See the HLP Lab blog post for a summary or read the final version.

The big picture

Broadly speaking, I'm interested in how the brain manages to efficiently allocate representational resources in a world where the statistics of sensory features changes from situation to situation. I'm particularly interested in how the structure in this sensory non-stationarity makes it possible to adapt to such changes more efficiently. Speech perception serves as an excellent model organism, because the statistics of the speech signal depend both on what is being said and who is saying it, both of which introduce highly structured variability that listeners are sensitive to.

My work aims to develop explicit, computational models of perception and adaptation, with a particular emphasis on speech. I think that good theories and models draw on insights from—and try to make connections between—neuroscience, behavioral data, and broader computational-level cognitive modeling.

Topics that I have or am working on or am interested in include: perceptual category learning, phonetic adaptation/recalibration, acquisition of phonetic categories, and cue combination and complex acoustic feature extraction. I'm also particularly interested in increasing awareness and appreciation of Bayesian methods for modeling (although I don't consider myself a capital-B-Bayesian) and—especially—data analysis.

Phonetic adaptation/belief updating

Adaptation is ubiquitous in speech perception and production, but it's usually treated as a bunch of phenomena rather than a theoretically interesting entity in its own right. Working with Florian Jaeger, I've developed a Bayesian model of phonetic adaptation which instantiates the idea that listeners update their beliefs about how each category sounds by combining their prior beliefs with the observations they have in the relevant context. This model captures some interesting data on how cumulative exposure affects perceptual recalibration and, in at least one case, selective adaptation of phonetic categories.

We're working on extending this approach to situations where there are multiple speakers or contexts, and when new categories or cues must be learned.

  • Kleinschmidt, D. F., & Jaeger, T. F. (2015). Re-examining selective adaptation: Fatiguing feature detectors, or distributional learning? Psychonomic Bulletin and Review, in press.[link]
  • Kleinschmidt, D. F., & Jaeger, T. F. (2015). Robust speech perception: Recognize the familiar, generalize to the similar, and adapt to the novel. Psychological Review, 122(2). doi:10.1037/a0038695 [link]
  • Kleinschmidt, D. F., & Jaeger, T. F. (2013). Modeling adaptation to multiple speakers as (Bayesian) belief updating. Poster presented at the Workshop on Current Issues and Methods in Speaker Adaptation, The Ohio State University, Columbus, OH. [pdf]
  • Kleinschmidt, D. F., & Jaeger, T. F. (2012). A continuum of phonetic adaptation: Evaluating an incremental belief-updating model of recalibration and selective adaptation. Proceedings of the 34th Annual Conference of the Cognitive Science Society. Sapporo, Japan. [pdf]
  • Kleinschmidt, D. F., & Jaeger, T. F. (2011). A Bayesian belief updating model of phonetic recalibration and selective adaptation. 2nd ACL Workshop on Cognitive Modeling and Computational Linguistics. [pdf]

Phonetics experiments on the web

I've developed a framework for phonetic adaptation experiments on the web, over Mechanical Turk. It's written in Javascript and the soure code will be available (soon!) as a git repository, or see a demo here.

Using this paradigm, we've replicated a study on the effect of cumulative exposure to audio-visual speech on phonetic adaptation. (Vroomen et al., 2007, Neuropsychologia)

  • Kleinschmidt, D. F., & Jaeger, T. F. (2012). A continuum of phonetic adaptation: Evaluating an incremental belief-updating model of recalibration and selective adaptation. Proceedings of the 34th Annual Conference of the Cognitive Science Society. Sapporo, Japan. [pdf]

Syntactic expectations/belief updating

After we found that belief updating (in a Bayesian framework) gave us some traction in thinking about phonetic adaptation, we realized that the same principles we used to build that model could just as easily be applied to other levels of language processing. The first extension was to syntactic expectation adaptation, based on some data on a long-term training study carried out by Alex Fine. We showed that a Bayesian belief updating model also captured this data well, despite the fact that it is of a very different sort than the phonetic adaptation data in our first model.

  • Kleinschmidt, D. F., Fine, A. B., & Jaeger, T. F. (2012). A belief-updating model of adaptation and cue combination in syntactic comprehension. Proceedings of the 34th Annual Conference of the Cognitive Science Society. Sapporo, Japan. [pdf]

Phylogenetic approaches to language typology

  • Croft, W., Bhattacharya, T., Kleinschmidt, D., Smith, D. E., & Jaeger, T. F. (2011). Greenbergian universals, diachrony, and statistical analyses. Linguistic Typology, 15(2), 433-453. doi:10.1515/LITY.2011.029 [pdf]

Efficient coding of sounds in the auditory cortex

The visual system is tuned to the statistical properties of natural images, at multiple levels. Such sensitivity to the statistics of natural scenes enables the visual system to efficiently represent information, enhancing interesting information and suppressing redundant information. These types of considerations are relevant to all perceptual systems, and so it is likely that other sensory systems are similarly attuned to the statistics of their natural input. This project aims to apply models which embody efficient coding principles to acoustic stimuli at the level of the auditory cortex. Preliminary work has shown that representations of natural sounds, including speech sounds, learned by such a model are very similar to the representations used by the auditory cortex (as indicated by A1 receptive fields).

Future work aims to apply more sophisticated models to this data, and to relate the salient statistical regularities of speech sounds learned by these models to phonetic categories.

  • Kleinschmidt, D. (2010). Efficient coding of speech in the auditory cortex. Poster presented at the 2nd Annual Neurobiology of Language Conference. [poster pdf]

Unsupervised learning of phonetic categories

As a Baggett Fellow at the University of Maryland Linguistics Department in 2009-2010, I worked with Bill Idsardi, Brian Dillon, Ewan Dunbar, and others on investigating and developing hierarchical Bayesian models of unsupervised learning of phonetic categories. This work is motivated by the fact that infants acquire their native-language phonetic categories at an age before they have acquired a substantial vocabulary, and so it is unlikely that they acquire these distinctions via contrastive minimal pairs. One alternative view holds that infants learn phonetic and phonological categories based on the statistical distribution of acoustic cues in the input. However, learning these categories based on the statistical distribution of acoustic input alone is very difficult (if not impossible), given that noise and asymmetries in frequency between different sounds often obscure phonemic contrasts in the input distribution. Moreover allophonic variation introduces additional non-phonemic contrasts, which have to be ignored or processed out at a later stage of processing.

Thus, for a distributional learning model to be reasonable, some additional structure must be built into the learning mechanism. Hierarchical Bayesian models allow additional structure and constrains on learning to be built into a distributional learner in a non-arbitrary way, and moreover allow precise, mathematically justified, and quantitative inference based on observed data. We are currently exploring a number of possible ways in which such structure might be incorporated, including general knowledge of phonological rules (as transformations in acoustic space), auditory-articulatory coupling, and a rudimentary lexicon (motivated by the work of Naomi Feldman and colleagues, and the fact that infants become sensitive to word boundaries around the same time that they become sensitive to their native phonetic categories).

An ARTful model of phonotactic and neighborhood effects

For my undergrad honors thesis, I extended the ARTWORD model of lexical access to account for phonotactic probability and lexical neighborhood density effects in spoken nonword and word perception. By analyzing the a priori ability of the model to generate different qualitative data patterns when different levels of sublexical representations were included, I found tentative evidence for the importance of larger sublexical representations (biphones, etc.) in speech perception.

  • Kleinschmidt, D. (2009). An ARTful case for the existence of sublexical representations in speech perception. Cognitive Science honors thesis, Williams College [pdf]