Long-term experiments are commonly-used tools in agronomy, soil science, and other disciplines for comparing the effects of different treatment regimes over an extended length of time. Periodic measurements, typically annual but sometimes over other periods, are taken on experimental units and are often analyzed using customary tools and models for repeated measures. It is demonstrated that this analysis approach is invalid because of a fundamental flaw in the design of most long-term trials that fails to account for the common effects of environments. The models contain nothing that accounts for the random environmental variations that typically affect all experimental units simultaneously and can interact with treatment effects. The experiment is essentially unreplicated, because all plots are exposed to a single sequence of random environmental effects, and this analysis uses the wrong variability in the error term for tests.

This added variability can dominate that from all other sources, and can lead to completely incorrect analysis results. A small simulation shows that type I error (false rejection) chances can be close to 100% under realistic circumstances. Possible solutions are reviewed and recommendations are made for improving statistical analysis and interpretation in the presence of these extra random variations. It is demonstrated that a modified experimental design avoids all of the analysis problems and allows valid statistical inferences on all effects.