Caste-based discrimination has been a feature of Indian society for many centuries, and despite sustained efforts by the Indian government, caste-based inequity still  persists in socio-economic status, educational attainment and in health outcomes. A potential source of caste inequity, which is as yet unexplored, is in the differential development of cognitive skills during early childhood. These skills have been associated with performance in school and even with longer-term employment and wage outcomes. Current research highlights the fact that cognitive skills are malleable and can be influenced by external environmental factors such as parental education, household socio-economic status and child's health, especially during the initial years of a child's life. This paper examines the link between caste and these external environmental factors in order to highlight the channels through which caste might affect the cognitive development of young children. We use recent longitudinal data for a cohort of approximately 2000 young children from the Young Lives survey in the state of Andhra Pradesh in southern India. We find significant evidence of a differential in cognitive skill development by caste, and also that caste acts largely through lower parental education and lower household socio-economic status, suggesting an intergenerational mechanism of transmission of outcomes.