Background. Becoming competent with numbers is important for handling the demands of everyday life and progression in many subjects. Research and debate on why primary schoolchildren differ so much in their development of number competence has long been a part of developmental psychology. Evidence of the continuity between primary number competence and both secondary mathematics and adult numeracy broadens the audience for this research. However, confusion over the meanings of mathematics and the implications of research can lead to misunderstandings of the genuine contribution developmental psychology can make. Aims. We illustrate the potential for confusion with recent media coverage of psychological research and issues from the beginnings of mathematics. After outlining current concerns in England, we consider possible factors which may contribute to differences in number competence observed among children, both within and across nations. Results. The factors identified as potential sources of difference include sociocultural variables such as beliefs about development, environmental variables such as support and experience at home and in school, cognitive characteristics of individuals such as short-term memory functioning, and specific numerical skills. A central problem is that the correlational nature of most research does not justify inferring causal connections from the links identified between these factors and number competence. Conclusions. Despite its limitations, existing research indicates that there is no single cause of difficulty in mathematics and that both factors within the child and external factors are likely to be important. The success of interventions provides grounds for believing that difficulties can be overcome.