An extension of the Rasch model with correlated latent variables is proposed to model correlated binary data within families. The latent variables have the classical correlation structure of Fisher (1918) and the model parameters thus have genetic interpretations. The proposed model is fitted to data using a hybrid of the Metropolis-Hastings algorithm and the MCEM modification of the EM-algorithm and is illustrated using genotype-phenotype data on a psychological subtest in families where some members are affected by the genetic disorder fragile X. In addition, hypothesis testing and model selection methods based on the Wald statistic are discussed.