Most of the current methods to assess effective connectivity from functional magnetic resonance imaging (fMRI) rely on the assumption that all relevant brain regions are entered into the analysis. If this assumption is untenable, which we believe is most often the case, then spurious connections between brain regions can appear. In this paper we propose to use an ancestral graph to model connectivity, which provides a way to avoid spurious connections. The ancestral graph is determined from trial-by-trial variation and not from the time series. A random effects model is defined for ancestral graphs which allows for individual differences in terms of graph parameters (e.g., connection strength). Procedures for model selection, model fit, and hypothesis testing of ancestral graphs are proposed. The hypothesis test can be used to find differences in connection strength between, for example, conditions. Monte Carlo simulations show that the ancestral graph is appropriate to model connectivity from fMRI condition specific trial data. To assess the accuracy further, the proposed method is applied to real fMRI data to determine how brain regions interact during speech monitoring.