We present our experience with building test corpora using actual usage data from a general domain MT system. We compare this to the more traditional approach of using held out data from the same domain as the training data, and discuss some of the challenges and results.