ARTool

Jacob O. Wobbrock
Leah Findlater
Darren Gergle
James J. Higgins

Download

Current Version 1.1.4

Windows executable: ARTool 1.1.4

ARTool requires you to have the Microsoft .NET 2.0 Framework or later installed. Get the latest version of the .NET Framework.

This software is distributed under the "New BSD License" agreement.

Source code: ARTool.zip.

Web Version

ARTool has been built as ARTweb, a Web-based Java program that accomplishes the same thing as ARTool.

Publication

Wobbrock, J.O., Findlater, L., Gergle, D. and Higgins, J.J. (2011). The Aligned Rank Transform for nonparametric factorial analyses using only ANOVA procedures. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '11). Vancouver, British Columbia (May 7-12, 2011). New York: ACM Press, pp. 143-146.

About

The need for a general nonparametric factorial analysis is acute for many types of data obtained in human-computer interaction (HCI) studies, especially for repeated measures data. The Friedman test handles only 1 factor of arbitrary levels, and therefore cannot be used to examine interaction effects. Logistic regression can be used for some designs, but has assumptions about the independence of responses and is often not suitable for repeated measures. The same is true of typical χ² tests. Examples of data warranting nonparametric factorial analyses are those obtained from Likert scales, error rates that occur in human performance studies, or preference tallies. These measures often cannot be transformed for suitability to ANOVA, e.g., with the popular log_e(Y) or log_e(Y+c) transforms (Aitchison & Brown 1957; Berry 1987).

But isn't there a standard analysis equivalent to the factorial F-test for nonparametric data? Surely there must be a nonparametric equivalent to the F-test. Surprisingly, such an analysis is elusive, and although there has, of course, been work by researchers on nonparametric factorial methods, those methods remain relatively uncommon, obscure, or only partially vetted. For a review of some methods, see, e.g., Sawilowsky (1990).

To illustrate the point, consider this useful table of analyses from U.C.L.A.; you will see question marks ("???") where a factorial nonparametric analysis should be. Similarly, this useful page, which explains the rationale for just about every common statistical analysis, does not describe a factorial nonparametric approach to repeated measures data.

The popular Rank Transform (RT) method of Conover and Iman (1981) applies ranks, averaged in the case of ties, over the entire data set, and then uses parametric ANOVA on ranks, resulting in a nonparametric factorial procedure. However, it was found that this process only produces accurate results for main effects, and interactions are subject to increases of Type I errors (Salter & Fawcett 1993; Higgins & Tashtoush 1994).

The Aligned Rank Transform (ART) procedure was found to correct this problem. For each main effect or interaction, the response variable (Y) is "aligned," a process that "strips" from Y all effects but the one of interest. This aligned response we call Y_aligned. The aligned responses are then assigned ranks, averaged in the case of ties, and the new response we might call Y_art. Then a full factorial ANOVA is run on the Y_art responses, but only the effect for which Y was aligned (Y_aligned) is examined. Thus, for each main effect or interaction, a new aligned column (Y_aligned) and a new ranked column (Y_art) is necessary. For example, with two factors and their interaction, we need three additional aligned (and then three additional ranked) columns, one for each main effect, and one for the interaction effect. In general, for N factors, we need 2^N-1 aligned columns. The program provided here, ARTool, creates those columns, and their ranked results, automatically for you.

Most modern statistical packages lack a built-in feature for aligning data. (Many do have features for assigning averaged ranks.) Aligning data is laborious to do by hand.

ARTool takes a comma-delimited (*.csv) file as input. This file must represent a long-format data table (one Y response per row, in the last column). The first row should be column names. The first column should be the experimental unit, e.g., Subject (i.e., s01, s02, s03, etc.). This column is not currently used in the mathematical calculations, but is useful for clarity when reading the output, and for retaining in repeated measures designs where the same experimental unit is listed on multiple rows. The last column must be the sole numeric response (Y) from the original data. Every column in between S and Y represents one factor (X_n) from the experiment. All possible main effects and interactions are given a new column in the output.

The alignment process used is that for a completely randomized design. This can result in reduced power for other designs like split-plots, as described in Higgins et al. (1990). But this is the simplest and most easlily generalized algorithm to implement. As it may only reduce power, any significant results can be trusted. For more on this issue, see Higgins et al. (1990) and Higgins & Tashtoush (1994).

The output of ARTool is a new comma-delimited (*.csv) file with a custom extension, e.g., *.art.csv, appended to its name. This file will have, for each effect, an "aligned" column showing the aligned data (Y_aligned) and an "ART" column (Y_art), showing the averaged ranks applied to the corresponding aligned column. As the original table's columns are retained, the output data table will have (2+N) + 2*(2^N-1) columns. A verification step is automatically performed by ARTool to ensure that each aligned column sums to zero. Users of ARTool can perform a further sanity check by running a full-factorial ANOVA on the aligned columns. All effects other than the one for which the column was created should be close to, if not exactly, F=0.00 and p=1.00.

The long-format *.csv file produced by ARTool can be opened directly by Microsoft Excel. From there, the data can be copied-and-pasted into one's favorite statistics package. At that point, a traditional ANOVA can be run on the ART columns using the full-factorial model, and interpreting the effects only for their relevant columns. Alternatively, the long-format table can be directly used by most statistical packages in a mixed-effects model analysis of variance using the REstricted (or REsidual) Maximum Likelihood method (REML). In this case, the experimental unit, e.g., Subject, should be made a random effect, leaving the other factors as fixed effects. In SAS, the well-known command for this is PROC MIXED (see Littell et al. 1998).

You should verify that an analysis on the aligned data (not its averaged ranks) shows all effects except the one of interest for a given column "stripped out," indicating the correctness of the ART procedure. The one exception is the effect of the experimental unit, e.g., Subject, which is usually ignored. In my preferred method of using REML-based mixed-effects model analyses of variance, the experimental unit is treated as a random effect and therefore not included in the fixed-effects results table. Mixed-effects models are particularly useful for repeated measures designs, which occur often in the field of human-computer interaction. Although mixed-effects models have become common in biology, ecology, biostatistics, and epidemiology, they have not yet become common in the analysis of human-computer interaction experiments, although they ought to be. See the papers by Frederick (1999), Littell et al. (1998), and Schuster & von Eye (2001) for more on mixed-effects models and their relevance to repeated measures data.

Warning. In general, because the aligning process strips all but one effect from the data, the ANOVA on ranks should also show close to F=0.00 and p=1.00 for all "other" effects except the one corresponding to the given column. (However, it will rarely be exactly F=0.00 and p=1.00.) If this is not the case, then it may be that the data is not suitable for the ART procedure. Proceed with caution in this case, and perhaps consider an alternative nonparametric approach (e.g., a robust rank-based approach, a bootstrap approach, Generalized Linear Mixed Models (GLMM), or Generalized Estimating Equations (GEE)). See, e.g., Sawilowsky (1990) or Higgins (2004).

Mathematics

The mathematics for this general nonparametric factorial analysis were worked out by Higgins & Tashtoush (1994). Dr. Higgins was kind enough to translate the mathematics of his article into a personal communication to me that was easier for me to understand and code. Note that the literature on the Aligned Rank Transform does not present a general formulation for an arbitrary number of factors; most publications deal with only two factors. It was for the purpose of creating ARTool that Dr. Higgins worked out the mathematics for an arbitrary number of factors.

Dr. Higgins' five steps for turning a raw response Y into Y_aligned are as follows:

Step 1. For each raw response Y, compute its residual as

The cell mean is the mean Y for that cell, i.e., over all data points whose levels of the factors (X's) match that of the Y response.

Step 2. Compute the "estimated effect." Example: Let A, B, C, D be factors with levels

Let A_i indicate the mean response Y_i only for rows where factor A is at level i. Let A_iB_j indicate the mean response Y_ij only for rows where factor A is at level i and factor B is at level j. And so forth. Let μ = the grand mean of Y over all rows.

Main effects

The estimated effect for factor A with response Y_i is

Two-way effects

The estimated effect for the A*B interaction with response Y_ij is

Three-way effects

The estimated effect for the A*B*C interaction with response Y_ijk is

Four-way effects

The estimated effect for the A*B*C*D interaction with response Y_ijkℓ is

N-way effects

The estimated effect for an N-way interaction is

Step 3. Compute the aligned data point Y_aligned as the replacement for raw data point Y for the effect of interest as

Step 4. Assign averaged ranks to all aligned observations Y_aligned for each new aligned column such that Y_aligned becomes Y_art. With averaged ranks, "if a value is unique, its averaged rank is the same as its rank. If a value occurs k times, the average rank is computed as the sum of the value's ranks divided by k" (SAS® JMP 7.0 help documentation).

As noted above, ARTool computes aligned data columns (for inspection) and the averaged ranks for each of these columns.

Step 5. This step is one not performed by ARTool: Perform a full-factorial ANOVA, or mixed-effects model analysis of variance, on the aligned ranks data (Y_art) produced by ARTool.

Sample Data

Four example data sets are included in the ARTool\data\ folder. The first two are from Higgins et al. (1990). The first of these, named Higgins1990-Table1.csv, shows a mock data set with two between-subjects factors named Row and Column. Each factor has 3 levels. Although in Higgins et al. (1990) this table is represented in wide-format, ARTool requires long-format tables, so it has been rendered as such. After using ARTool on it, an output file named Higgins1990-Table1.art.csv is created. This data has also been put in an SAS® JMP table, Higgins1990-Table1.art.JMP, which contains saved analyses of variance for inspection. One can verify that the aligned ranks and the test results agree with those found in Higgins et al. (1990).

A second example is in Higgins1990-Table5.csv. The output file created by ARTool is Higgins1990-Table5.art.csv. This data is from a real study of moisture levels and fertilizer as it affects the dry matter created in peat. It has two factors, Moisture and Fertilizer. Moisture is a between-subjects factor of 3 levels, while Fertilizer is a within-subjects factor of 4 levels. Twelve trays containing four pots of peat each were put in a different moisture condition. Each peat pot on a tray was subjected to a seperate Fertilizer. The Tray is therefore regarded as the experimental unit (the "Subject"), and each peat pot on each tray is a "trial." The response variable is the amount of dry matter produced in the pot. In agricultural-statistical terminology, this is a classic split-plot design, with Moisture as the whole-plot factor and Fertilizer as the subplot factor. It is instructive to compare the layout of Table 5 in Higgins et al. (1990) to the long-format layout in Higgins1990-Table5.csv. The aligned data has been put in an SAS® JMP table named Higgins1990-Table5.art.JMP. Analyses have been saved to the table and match the results in Higgins et al. (1990).

A third example is HigginsABC.csv, which is a mock data set with two between-subjects factors, A and B, and a third within-subjects factor, C. The aligned table is HigginsABC.art.csv, and in SAS® JMP it is HigginsABC.art.JMP. An analysis of variance will show that all main effects and the A*B interaction are significant. An analysis of variance on the aligned-ranks data (i.e., the "ART" columns) will show that the same significance conclusions are drawn.

A fourth example is HigginsABC.csv renamed to 'Produces Error.csv' and given an invalid non-numeric response ("X") on the 3rd data row. When analyzed by ARTool, a red-text error is produced. In general, ARTool produces as descriptive error messages as possible, identifying where errors occur so they can be quickly remedied.

A trial version of SAS® JMP can be downloaded from http://www.jmp.com/software/.

Further Reading

1. Aitchison, J. and Brown, J. A. C. (1957). The Lognormal Distribution. Cambridge, England: Cambridge University Press.
2. Akritas, M. G. and Brunner, E. (1997). A unified approach to rank tests for mixed models. Journal of Statistical Planning and Inference 61 (2), pp. 249-277.
3. Akritas, M. G. and Osgood, D. W. (2002). Guest editors' introduction to the special issue on nonparametric models. Sociological Methods and Research 30 (3), pp. 303-308.
4. Berry, D. A. (1987). Logarithmic transformations in ANOVA. Biometrics 43 (2), pp. 439-456.
5. Conover, W. J. and Iman, R. L. (1981). Rank transformations as a bridge between parametric and nonparametric statistics The American Statistician 35 (3), pp. 124-129.
6. Fawcett, R. F. and Salter, K. C. (1984). A monte carlo study of the F test and three tests based on ranks of treatment effects in randomized block designs. Communications in Statistics: Simulation and Computation 13 (2), pp. 213-225.
7. Frederick, B. N. (1999). Fixed-, random-, and mixed-effects ANOVA models: A user-friendly guide for increasing the generalizability of ANOVA results. In Advances in Social Science Methodology, B. Thompson (ed). Stamford, Connecticut: JAI Press, pp. 111-122.
8. Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association 32 (200), pp. 675-701.
9. Higgins, J. J., Blair, R. C. and Tashtoush, S. (1990). The aligned rank transform procedure. Proceedings of the Conference on Applied Statistics in Agriculture. Manhattan, Kansas: Kansas State University, pp. 185-195.
10. Higgins, J. J. and Tashtoush, S. (1994). An aligned rank transform test for interaction. Nonlinear World 1 (2), pp. 201-211.
11. Higgins, J. J. (2004). Introduction to Modern Nonparametric Statistics. Pacific Grove, California: Duxbury Press.
12. Kaptein, M., Nass, C. and Markopoulos, P. (2010). Powerful and consistent analysis of Likert-type rating scales. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '10). New York: ACM Press, pp. 2391-2394.
13. Lehmann, E. L. (2006). Nonparametrics: Statistical Methods Based on Ranks. New York: Springer.
14. Littell, R. C., Henry, P. R. and Ammerman, C. B. (1998). Statistical analysis of repeated measures data using SAS procedures. Journal of Animal Science 76 (4), pp. 1216-1231.
15. Mansouri, H. (1999). Aligned rank transform tests in linear models. Journal of Statistical Planning and Inference 79 (1), pp. 141-155.
16. Mansouri, H. (1999). Multifactor analysis of variance based on the aligned rank transform technique. Computational Statistics and Data Analysis 29 (2), pp. 177-189.
17. Mansouri, H., Paige, R. L. and Surles, J. G. (2004). Aligned rank transform techniques for analysis of variance and multiple comparisons. Communications in Statistics: Theory and Methods 33 (9), pp. 2217-2232.
18. Richter, S. J. (1999). Nearly exact tests in factorial experiments using the aligned rank transform. Journal of Applied Statistics 26 (2), pp. 203-217.
19. Salter, K. C. and Fawcett, R. F. (1985). A robust and powerful rank test of treatment effects in balanced incomplete block designs. Communications in Statistics: Simulation and Computation 14 (4), pp. 807-828.
20. Salter, K. C. and Fawcett, R. F. (1993). The ART test of interaction: A robust and powerful rank test of interaction in factorial models. Communications in Statistics: Simulation and Computation 22 (1), pp. 137-153.
21. Sawilowsky, S. S. (1990). Nonparametric tests of interaction in experimental design. Review of Educational Research 60 (1), 91-126.
22. Schuster, C. and von Eye, A. (2001). The relationship of ANOVA models with random effects and repeated measurement designs. Journal of Adolescent Research 16 (2), pp. 205-220.
23. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin 1 (6), pp. 80-83.

Acknowledgement

This work was supported in part by the National Science Foundation under grants IIS-0811884 and IIS-0811063. Any opinions, findings, conclusions or recommendations expressed in this work are those of the authors and do not necessarily reflect those of the National Science Foundation.