457
Views
10
CrossRef citations to date
0
Altmetric
Articles

Rage against the machine: Evaluation metrics in the 21st century

Pages 100-125 | Received 30 Nov 2016, Accepted 04 Dec 2016, Published online: 20 Mar 2017

References

  • Anderson, J. R. (1991). The adaptive nature of human categorization. Psychological Review, 98(3):409.
  • Anderson, S. R. (1969). West Scandinavian vowel systems and the ordering of phonological rules. PhD thesis, MIT.
  • Angluin, D. (1982). Inference of reversible languages. Journal of the ACM, 29(3):741–765.
  • Aronoff, M. (1976). Word formation in generative grammar. MIT Press, Cambridge, MA.
  • Baerman, M., Corbett, G. G., and Brown, D., editors (2010). Defective paradigms: Missing forms and what they tell us. Oxford University Press, Oxford.
  • Baker, C. L. (1979). Syntactic theory and the projection problem. Linguistic Inquiry, 10(4):533–581.
  • Baker, M. (2001). The atoms of language: The mind’s hidden rules of grammar. Basic Books, New York.
  • Berko, J. (1958). The child’s learning of English morphology. Word, 14(2–3):150–177.
  • Berwick, R. (1985). The acquisition of syntactic knowledge. MIT Press, Cambridge, MA.
  • Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent dirichlet allocation. the Journal of machine Learning research, 3:993–1022.
  • Bonawitz, E., Denison, S., Gopnik, A., and Griffiths, T. L. (2014). Win-stay, lose-sample: A simple sequential algorithm for approximating bayesian inference. Cognitive psychology, 74:35–65.
  • Bowerman, M. (1988). The ‘no negative evidence’ problem: How do children avoid constructing an overly general grammar? In Hawkins, J. A., editor, Explaining language universals, pages 73–101. Basil Blackwell, Oxford.
  • Bowers, J. S. and Davis, C. J. (2012). Bayesian just-so stories in psychology and neuroscience. Psychological bulletin, 138(3):389.
  • Boyd, J. K. and Goldberg, A. E. (2011). Learning what not to say: The role of statistical preemption and categorization in a-adjective production. Language, 87(1):55–83.
  • Brown, R. (1973). A first language: The early stages. Harvard University Press, Cambridge, MA.
  • Brown, R. and Hanlon, C. (1970). Derivational complexity and the order of acquisition in child speech. In Hayes, J. R., editor, Cognition and the development of language, pages 11–53. Wiley, New York.
  • Bush, R. R. and Mosteller, F. (1951). A mathematical model for simple learning. Psychological Review, 68(3):313–323.
  • Chater, N. and Vitányi, P. (2007). Ideal learning of natural language: Positive results about learning from positive evidence. Journal of Mathematical Psychology, 51(3):135–163.
  • Chickering, M., Heckerman, D., and Meek, C. (2004). Large-sample learning of Bayesian networks is NP-hard. Journal of Machine Learning Research, 5:1287–1330.
  • Chomsky, N. (1951). Morphophonemics of Modern Hebrew. Master’s thesis, University of Pennsylvania. Published by Garland, New York, 1979.
  • Chomsky, N. (1955). The logical structure of linguistic theory. Ms., Harvard University and MIT. Revised version published by Plenum, New York, 1975.
  • Chomsky, N. (1957). Syntactic structures. Mouton, The Hague.
  • Chomsky, N. (1965). Aspects of the theory of syntax. MIT Press, Cambridge, MA.
  • Chomsky, N. (1981). Lectures in government and binding. Foris, Dordrecht.
  • Chomsky, N. (1986). Knowledge of language: Its nature, origins, and use. Praeger, New York.
  • Chomsky, N. (2005). Three factors in language design. Linguistic Inquiry, 36(1):1–22.
  • Chomsky, N. and Halle, M. (1968). The sound pattern of English. MIT Press, Cambridge, MA.
  • Clark, A. and Eyraud, R. (2007). Polynomial identification in the limit of context-free substitutable languages. Journal of Machine Learning Research, 8:1725–1745.
  • Clark, E. V. (1987). The principle of contrast: A constraint on language acquisition. In MacWhinney, B., editor, Mechanisms of language acquisition, pages 1–33. Erlbaum, Hillsdale, NJ.
  • Conwell, E. and Demuth, K. (2007). Early syntactic productivity: Evidence from dative shift. Cognition, 103(2):163–179.
  • Crain, S. (2012). The emergence of meaning, volume 135. Cambridge University Press.
  • Crain, S. and Thornton, R. (2000). Investigations in universal grammar: A guide to experiments on the acquisition of syntax and semantics. MIT Press, Cambridge, MA.
  • Culbertson, J., Smolensky, P., and Legendre, G. (2012). Learning biases predict a word order universal. Cognition, 122(3):306–329.
  • D abrowska, E. (2001). Learning a morphological system without a default: The Polish genitive. Journal of Child Language, 28(3):545–574.
  • Dagum, P. and Luby, M. (1993). Approximating probabilistic inference in bayesian belief networks is np-hard. Artificial intelligence, 60(1):141–153.
  • de Marcken, C. (1996). Unsupervised language acquisition. PhD thesis, MIT.
  • Dillon, B., Dunbar, E., and Idsardi, W. (2013). A single-stage approach to learning phonological categories: Insights from inuktitut. Cognitive Science, 37(2):344–377.
  • Dresher, B. E. and Kaye, J. (1990). A computational learning model for metrical phonology. Cognition, 34:137–195.
  • Eberhardt, F. and Danks, D. (2011). Confirmation in the cognitive sciences: The problematic case of bayesian models. Minds and Machines, 21(3):389–410.
  • Estes, W. K. (1950). Toward a statistical theory of learning. Psychological review, 57(2):94.
  • Fazly, A., Alishahi, A., and Stevenson, S. (2010). A probabilistic computational model of cross-situational word learning. Cognitive Science, 34(6):1017–1063.
  • Feldman, N. H., Griffiths, T. L., Goldwater, S., and Morgan, J. L. (2013). A role for the developing lexicon in phonetic category acquisition. Psychological Review, 120(4):751–778.
  • Fodor, J. D. and Sakas, W. G. (2005). The subset principle in syntax: Costs of compliance. Journal of Linguistics, 41(3):513–569.
  • Frank, M. C., Goodman, N. D., and Tenenbaum, J. B. (2009). Using speakers’ referential intentions to model early cross-situational word learning. Psychological Science, 20(5):578–585.
  • Geman, S. and Geman, D. (1984). Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 6(6):721–741.
  • Gibson, E. and Wexler, K. (1994). Triggers. Linguistic Inquiry, 25(3):407–454.
  • Gilks, W. R., Richardson, S., and Spiegelhalter, D. J. (1996). Markov chain Monte Carlo in practice. London: Chapman and Hall.
  • Gold, E. M. (1967). Language identification in the limit. Information and Control, 10:447–474.
  • Goldsmith, J. (2001). Unsupervised learning of morphology of a natural language. Computational Linguistics, 27(2):153–198.
  • Goldwater, S., Griffiths, T. L., and Johnson, M. (2009). A bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112(1):21–54.
  • Goodman, N. (1955). Fact, fiction, and forecast. Harvard University Press, Cambridge, MA.
  • Goodman, N. D., Frank, M. C., Griffiths, T. L., Tenenbaum, J. B., Battaglia, P. W., and Hamrick, J. B. (2015). Relevant and robust a response to marcus and davis (2013). Psychological science, 26(4):539–541.
  • Green, G. M. (1974). Semantics and syntactic regularity. Indiana University Press.
  • Grimshaw, J. (1990). Argument structure. MIT Press, Cambridge, MA.
  • Grimson, W. E. L. (1981). From images to surfaces: A computational study of the human early visual system. MIT press.
  • Gropen, J., Pinker, S., Hollander, M., Goldberg, R., and Wilson, R. (1989). The learnability and acquisition of the dative alternation in English. Language, 65(2):203–257.
  • Halle, M. (1973). Prolegomena to a theory of word formation. Linguistic Inquiry, 4(1):3–16.
  • Han, C.-h., Musolino, J., and Lidz, J. (2016). Endogenous sources of variation in language acquisition. Proceedings of the National Academy of Sciences, 113(4):942–947.
  • Hart, B. and Risley, T. R. (1995). Meaningful differences in the everyday experience of young American children. Paul H Brookes Publishing, Baltimore, MD.
  • Heath, S. B. (1983). Ways with words: Language, life and work in communities and classrooms. Cambridge University Press, Cambridge.
  • Horning, J. J. (1969). A study of grammatical inference. Technical report, Stanford University, Stanford, CA.
  • Huybregts, R. (1984). The weak inadequacy of context-free phrase structure grammars. Van periferie naar kern, pages 81–99.
  • Jackendoff, R. S. (1990). Semantic structures. MIT Press, Cambridge, MA.
  • Jarmulowicz, L. (2002). English derivational suffix frequency and children’s stress judgements. Brain and Language, 81(1–3):192–204.
  • Johnson, M., Griffiths, T. L., and Goldwater, S. (2007). Bayesian inference for pcfgs via markov chain monte carlo. In HLT-NAACL, pages 139–146.
  • Jones, M. and Love, B. C. (2011). Bayesian Fundamentalism or Enlightenment? on the explanatory status and theoretical contributions of Bayesian models of cognition. Behavioral and Brain Sciences, 34(?):169–231.
  • Julesz, B. (1971). Foundations of cyclopean perception. University of Chicago Press, Chicago.
  • Kanazawa, M. (1998). Learnable Classes of Categorial Grammars. Center for the Study of Language and Information, Stanford, CA.
  • Kemp, C., Perfors, A., and Tenenbaum, J. B. (2007). Learning overhypotheses with hierarchical bayesian models. Developmental Science, 10(3):307–321.
  • Kiparsky, P. (1973). Elsewhere in phonology. In Anderson, S. R. and Kiparsky, P., editors, A festschrift for Morris Halle, pages 93–106. Holt, Rinehart and Winston, New York.
  • Kirby, S., Dowman, M., and Griffiths, T. L. (2007). Innateness and culture in the evolution of language. Proceedings of the National Academy of Sciences, 104(12):5241–5245.
  • Köhne, J., Trueswell, J. C., and Gleitman, L. R. (2013). Multiple proposal memory in observational word learning. In Proceedings of the 35th Annual meeting of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
  • Krifka, M. (1999). Manner in dative alternation. In West Coast Conference on Formal Linguistics, volume 18, pages 260–271.
  • Kroch, A. (1989). Reflexes of grammar in patterns of language change. Language Variation and Change, 1(3):199–244.
  • Kroch, A. (1995). Dialect and style in upper class Philadelphia. In Guy, G., Feagin, C., Schiffrin, D., and Baugh, J., editors, Towards a social science of language: Papers in honor of William Labov, volume 1, pages 23–45. John Benjamins, Philadelphia.
  • Kuhl, P. K., Williams, K. A., Lacerda, F., Stevens, K. N., and Lindblom, B. (1992). Linguistic experience alters phonetic perception in infants by 6 months of age. Science, 255(5044):606–608.
  • Kwisthout, J., Wareham, T., and van Rooij, I. (2011). Bayesian intractability is not an ailment that approximation can cure. Cognitive Science, 35(5):779–784.
  • Labov, W. (1972). Sociolinguistic patterns. University of Pennsylvania Press, Philadelphia.
  • Labov, W. (1989). The child as linguistic historian. Language Variation and Change, 1(1):85–97.
  • Labov, W. (2007). Transmission and diffusion. Language, 83(2):344–387.
  • Levin, B. (1993). English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago.
  • Lignos, C. (2010). Learning from unseen data. In Proceedings of the Morpho Challenge 2010 Workshop, pages 35–38.
  • Lignos, C. (2013). Modeling words in the mind. PhD thesis, University of Pennsylvania.
  • MacWhinney, B. (2000). The CHILDES project: Tools for analyzing talk. Lawrence Erlbaum, Mahwah, NJ, 3rd edition.
  • Manning, C. and Schütze, H. (1999). Foundations of statistical natural language processing. MIT Press, Cambridge.
  • Marcus, G. F. and Davis, E. (2013). How robust are probabilistic models of higher-level cognition? Psychological science, 24(12):2351–2360.
  • Markman, E. M. and Wachtel, G. F. (1988). Children’s use of mutual exclusivity to constrain the meanings of words. Cognitive psychology, 20(2):121–157.
  • Marr, D. (2010). Vision: A computational investigation into the human representation and processing of visual information. MIT Press, Cambridge, MA. Originally published in 1982 by Freeman, San Francisco, CA.
  • Marr, D., Palm, G., and Poggio, T. (1978). Analysis of a cooperative stereo algorithm. Biological Cybernetics, 28(4):223–239.
  • Marr, D. and Poggio, T. (1976). Cooperative computation of stereo disparity. Science, 194(4262):283–287.
  • Marr, D. and Poggio, T. (1979). A computational theory of human stereo vision. Proceedings of the Royal Society of London B, 204:301–328.
  • Medina, T. N., Snedeker, J., Trueswell, J. C., and Gleitman, L. R. (2011). How words can and cannot be learned by observation. Proceedings of the National Academy of Sciences, 108(22):9014–9019.
  • Miller, K. L. and Schmitt, C. (2012). Variable input and the acquisition of plural morphology. Language Acquisition, 19(3):223–261.
  • Mysln, M. and Levy, R. (2016). Comprehension priming as rational expectation for repetition: Evidence from syntactic processing. Cognition, 147:29–56.
  • Niyogi, P. (2006). The computational nature of language learning and evolution. MIT Press, Cambridge, MA.
  • Niyogi, P. and Berwick, R. C. (2009). The proper treatment of language acquisition and change in a population setting. Proceedings of the National Academy of Sciences, 106(25):10124–10129.
  • Nosofsky, R. M., Palmeri, T. J., and McKinley, S. C. (1994). Rule-plus-exception model of classification learning. Psychological Review, 101(1):53.
  • O’Donnell, T. (2015). Productivity and reuse in language. MIT Press, Cambridge, MA.
  • Oehrle, R. T. (1976). The grammatical status of the English dative alternation. PhD thesis, Massachusetts Institute of Technology.
  • Osherson, D. N., Stob, M., and Weinstein, S. (1986). Systems that learn: An introduction to learning theory for cognitive and computer scientists. MIT Press, Cambridge, MA.
  • Pearl, L. and Sprouse, J. (2013). Syntactic islands and learning biases: Combining experimental syntax and computational modeling to investigate the language acquisition problem. Language Acquisition, 20(1):23–68.
  • Perfors, A., Tenenbaum, J. B., and Regier, T. (2011). The learnability of abstract syntactic principles. Cognition, 118(3):306–338.
  • Perfors, A., Tenenbaum, J. B., and Wonnacott, E. (2010). Variability, negative evidence, and the acquisition of verb argument constructions. Journal of Child Language, 37(3):607–642.
  • Pesetsky, D. (1995). Zero syntax: Experiencer and Cascade. MIT Press, Cambridge, MA.
  • Pinker, S. (1989). Learnability and cognition: The acquisition of argument structure. MIT Press, Cambridge, MA.
  • Pinker, S. (1999). Words and rules: The ingredients of language. Basic Books, New York.
  • Pintzuk, S. (1999). Phrase structures in competition: Variation and change in Old English word order. Routledge.
  • Prince, A. and Smolensky, P. (2004). Optimality Theory: Constraint interaction in generative grammar. MIT Press, Cambridge, MA.
  • Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5):465–471.
  • Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535.
  • Roberts, J. and Labov, W. (1995). Learning to talk Philadelphian: acquisition of short a by preschool children. Language Variation and Change, 7:101–112.
  • Roeper, T. and Williams, E. (1987). Parameter setting. Springer, Berlin.
  • Sakas, W. G. and Fodor, J. D. (2001). The structural triggers learner. In Bertolo, S., editor, Language acquistion and language learnability, pages 172–233. Cambridge University Press.
  • Sakas, W. G. and Fodor, J. D. (2012). Disambiguating syntactic triggers. Language Acquisition, 19(2):83–143.
  • Sankoff, G. and Blondeau, H. (2007). Language change across the lifespan: /r/in Montreal French. Language, 83(3):560–588.
  • Santorini, B. (1992). Variation and change in yiddish subordinate clause word order. Natural Language & Linguistic Theory, 10(4):595–640.
  • Schuler, K., Yang, C., and Newport, E. (2016). Testing the Tolerance Principle: Children form productive rules when it is more computationally efficient to do so. In The 38th Cognitive Society Annual Meeting, Philadelphia, PA.
  • Shi, R. and Melançon, A. (2010). Syntactic categorization in French-learning infants. Infancy, 15(5):517–533.
  • Shieber, S. (1985). Evidence against the context-freeness of natural language. Linguistics and Philosophy, 8(3):333–343.
  • Sirts, K. and Goldwater, S. (2013). Minimally-supervised morphological segmentation using adaptor grammars. Transactions of the Association for Computational Linguistics, 1:255–266.
  • Slobin, D. I. (1997). The crosslinguistic study of language acquisition, volume 4. Psychology Press.
  • Smith, J., Durham, M., and Fortune, L. (2009). Universal and dialect-specific pathways of acquisition: Caregivers, children, and t/d deletion. Language Variation and Change, 21(1):69–95.
  • Sober, E. (1975). Simplicity. Oxford University Press, New York.
  • Stefanowitsch, A. (2008). Negative entrenchment: A usage-based approach to negative evidence. Cognitive Linguistics, 19(3):513–531.
  • Stevens, J., Trueswell, J., Yang, C., and Gleitman, L. (2016). The pursuit of word meanings. In Cognitive Science. doi: 10.1111/cogs.12416.
  • Steyvers, M., Lee, M. D., and Wagenmakers, E.-J. (2009). A bayesian analysis of human decision-making on bandit problems. Journal of Mathematical Psychology, 53(3):168–179.
  • Suppes, P. (1966). Concept formation and bayesian decisions. In Hintkikka, J. and Suppes, P., editors, Aspects of inductive logic, pages 21–48. North-Holland.
  • Sutton, R. S. and Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge University Press.
  • Tardif, T., Shatz, M., and Naigles, L. (1997). Caregiver speech and children’s use of nouns versus verbs: A comparison of English, Italian, and Mandarin. Journal of Child Language, 24(3):535–565.
  • Taylor, A. (1994). Variation in past tense formation in the history of English. In Izvorski, R., Meyerhoff, M., Reynolds, B., and Tredinnick, V., editors, Penn Working Papers in Linguistics 1, pages 143–158. Penn Linguistics Club, Philadelphia.
  • Tenenbaum, J. B. and Griffiths, T. L. (2001). Generalization, similarity and bayesian inference. Behavioral and Brain Sciences, 24(4):629–640.
  • Tenenbaum, J. B., Kemp, C., Griffiths, T. L., and Goodman, N. D. (2011). How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285.
  • Trueswell, J. C., Medina, T. N., Hafri, A., and Gleitman, L. R. (2013). Propose but verify: Fast mapping meets cross-situational word learning. Cognitive psychology, 66(1):126–156.
  • Tyler, A. and Nagy, W. (1989). The acquisition of English derivational morphology. Journal of Memory and Language, 28(6):649–667.
  • Valian, V. (1986). Syntactic categories in the speech of young children. Developmental Psychology, 22(4):562.
  • Villavicencio, A., Idiart, M., Berwick, R. C., and Malioutov, I. (2013). Language acquisiiton and probabilistic models: Keeping it simple. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1321–1330.
  • Weinreich, U., Labov, W., and Herzog, M. (1968). Empirical foundations for a theory of language change. In Lehmann, W., editor, Directions for historical linguistics: A symposium, pages 95–195. University of Texas Press, Austin.
  • Werker, J. F. and Tees, R. C. (1984). Cross-language speech perception: Evidence for perceptual reorganization during the first year of life. Infant Behavior and Development, 7(1):49–63.
  • Wexler, K. and Culicover, P. (1980). Formal principles of language acquisition. MIT Press, Cambridge, MA.
  • Willshaw, D., Dayan, P., and Morris, R. (2015). Memory, modelling and marr: a commentary on marr (1971) ‘simple memory: a theory of archicortex’. Phil. Trans. R. Soc. B, 370(1666):20140383.
  • Xu, F. and Tenenbaum, J. B. (2007). Word learning as Bayesian inference. Psychological Review, 114(2):245.
  • Yang, C. (2000). Internal and external forces in language change. Language Variation and Change, 12(3):231–250.
  • Yang, C. (2002a). Knowledge and learning in natural language. Oxford University Press, Oxford.
  • Yang, C. (2002b). A principle of word storage. Manuscript: Yale University.
  • Yang, C. (2005). On productivity. Linguistic Variation Yearbook, 5(1):333–370.
  • Yang, C. (2013). Ontogeny and phylogeny of language. Proceedings of the National Academy of Sciences, 110(16):6324–6327
  • Yang, C. (2015). Negative knowledge from positive evidence. Language, 91(4):938–953.
  • Yang, C. (2016). The price of linguistic productivity: How children learn to break rules of language. MIT Press, Cambridge, MA.
  • Yang, C., Ellman, A., and Legate, J. A. (2015). Input and its structural description. In Ott, D. and Gallego, A., editors, 50th anniversary of Noam Chomsky’s Aspects of the Theory of Syntax. MITWPL.
  • Yu, C. and Smith, L. B. (2007). Rapid word learning under uncertainty via cross-situational statistics. Psychological Science, 18(5):414–420.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.