Oberlin Winter Term 2013

Posted by Katie Kuksenok on April 02, 2013

For the month of January, three Oberlin College undergraduates – Dan Barella, Sayer Rippey, and Eli Rose – joined SCCL to work on extending our command-line tool for affect detection using machine learning, ALOE. The Winter Term internship was initially conceived by Katie Kuksenok, one of the two Oberlin alumni in SCCL; the other, Michael Brooks, also helped in mentoring the students while they were on campus.


Each of the visiting Obies contributed a new functionality and compared its performance to that reported in our CSCW report; Dan implemented a novel segmentation algorithm, Sayer extended feature extraction to process French chat messages rather than only English, and Eli worked on HMM classification. Having returned to Oberlin, Sayer continues to work on analyzing the French portions of the dataset as an independent research project, collaborating over distance.

It has been an incredible month. Besides being blown away by the Seattle public transit system, I got to learn so much about machine learning, language, and grad school, and I got to meet a lot of smart, passionate, inspiring people.
The work I did applying the ALOE pipeline to French was completely fascinating. It was great because I got to be doing something very practical, trying to get the labeler working for the rest of the pipeline, but it also brought up some really interesting differences between French and English.

- Sayer Rippey

So, here I am at the end of Winter Term. I’m already nostalgic! This project was really enrapturing, and the whole experience thoroughly enjoyable. … I will say that, I’m proud of the work I’ve done. There are some places where I know there’s room for improvement, but to believe otherwise would perhaps be worse. I can’t claim that it’s all perfect, but I can claim that I did nearly all that I set out to do, and then some that I hadn’t expected to do. I didn’t expect I’d have to put together a profiling script to test my project, and yet this turned out to be one of the most invaluable tools I’ve had for code analysis (hopefully for others as well). I didn’t expect to find such a subtle tradeoff between a small tweaking of time variables, and yet this became a central issue of the last two weeks of my project. I didn’t think comparing pipeline statistics would be so nuanced, but now I’m beginning to see all the ways that a visualization can change the way people perceive information. I could go on, but what I’m really trying to say is: I learned so many new things!

But the most exciting parts of this Winter Term were not the components of my project. They were the incredible people at the SCCL, who brought me to lectures and talks on the nature of artificial intelligence and information visualization, who always provided novel viewpoints and provoking discussions, who were dedicated to sharing their unbelievable experience in so many topics. I was honored to work with Eli, Sayer, Katie, Michael, Megan, Cecilia, and the rest of this great team. They’ve humbled and challenged me, and for that I thank all of them; as this term comes to a close, I hope only that I should be so lucky in pursuit of future endeavors as I was in finding this one. So to everyone at the SCCL, so long, and thanks for all the fish!

- Dan Barella

Trends in Crowdsourcing

Posted by Katie Kuksenok on April 09, 2012

These several recent years have seen the rise of crowdsourcing as an exciting new tool for getting things done. For many, it was a way to get tedious tasks done quickly, such as transcribing audio. For many others, it was a way to get data: labeled image data, transcription correction data, and so on. But there is also a layer of meta-inquiry: what constitutes crowdsourcing? Who is in the crowd, and why? What can they accomplish, and how might the software that supports crowdsourcing be designed in a way to help them accomplish more?

Each of the last two conferences I have attended, CSCW2012 and UIST2011, had a “crowdsourcing session,” spanning a range of crowdsourcing-related research. But only a short while before that, the far bigger CHI conference contained only one or two instances of “crowdsourcing papers.” So what happened in the last few years?

At some point in the last decade, crowdsourcing emerged both as a method for getting lots of tedious work done cheaply, and a field of inquiry that resonated with human-computer interaction researchers. Arguably, this point historically coincided with the unveiling of Amazon Mechanical Turk platform, which allowed employers, or “requesters,” to list small, low-paid tasks, or “human-intelligence tasks (HITs)” for anonymous online contractors, or “workers,” to complete. In Amazon’s words, this enabled “artificial artificial intelligence” – the capacity to cheaply get answers to questions that cannot be automated.

And so this crowdsourcing thing took academic literature by storm, evidenced by growth in yearly additions to work exposed via Google scholar, as the related terms “crowdsourcing,” “mechanical turk,” and “human computation” seemed to grow rapidly at roughly the same time:

Cumulative term use over time in Google Scholar

This point in time, barring what could be argued as noise in the extremely coarse metric of Google scholar yearly result count, pretty much coincides with the 2008 unveiling of Mechanical Turk:

Google Trends

There is a distinction between HCI research that (1) uses crowdsourcing; (2) investigates crowdsourcing as a platform is capable of, which is yet distinct from that which (3) investigates what crowdsourcing is, or ought to be. Kittur, Chi, and Suh authored one of the first papers of the second variety, explaining how crowdsourcing – via Mechanical Turk in particular – could be used as a method for running user studies distinct from typical approaches in HCI literature (“Crowdsourcing User Studies with Mechanical Turk,” CHI 2007). Later, at CHI2010, a paper of the third variety – characterizing the demographics of Mechanical Turk workforce – was presented at an alt.chi session by Ross et al from UC Irvine (“Who are the Crowdworkers? Shifting Demographics in Mechanical Turk,” alt.chi 2010). Since, the third kind of research has begun to re-examine aspects of crowdsourcing that have been taken as nearly axiomatic as a result of its initial synonymy with Mechanical Turk.

Adversarial workers and the “arms race.” The kind of task that today is said to use crowdsourcing was already happening well before Mechanical Turk took the world by storm. Luis von Ahn had launched the ESP game and reCAPTCHA several years before. Still, it wasn’t until 2008 that Science published the paper on reCAPTCHA (“reCAPTCHA: Human-Based Character Recognition via Web Security Measures,” Science online 2008). Recently, the term human computation and crowdsourcing became used in lieu of one another frequently enough to warrant a 2011 survey of what those words mean in CHI by Quinn and Bederson* (“Human computation: a survey and taxonomy of a growing field,” CHI2011). The historical roots of crowdsourcing in Amazon Mechanical Turk seem to have resulted in a view of the stranger-workers as particularly adversarial strangers who can be used to generate data for any and all tasks. This has led to a body of research sometimes referred to as the “crowdsourcing arms race:” the tension between the worker’s [presumed] desire to do as little work as quickly as possible, and the requester’s [presumed] desire to paid as little as possible for as much work as possible. For example, quality control has moved from asking workers to fill out surveys to recently using biometrics to identify intentionally shoddy work, or “cheating” (J. Rzeszotarski and A. Kittur. “Instrumenting the Crowd: Using Implicit Behavioral Measures to Predict Task Performance,” UIST2011).

Beyond monetary incentives. Another artifact of historical connection into Amazon Mechanical Turk has been conflating crowdsourcing and human computation with paid work markets, and of monetary incentives. But there are other models – such as the initial Luis von Ahn games, where human computation is fuelled by fun. Or Jeff Bigham’s VizWiz, where motivation can border altruism – helping blind people interpret certain objects quickly – despite having monetary incentives via Mechanical Turk. Salman Ahmad and colleagues’ work on Jabberwocky crowd programming environment is a particularly deliberate departure from the monetary incentive model, recasting itself as “structured social computing” despite overlapping computational elements with crowdsourcing (“The Jabbrwocky Programming Environment for Structured Social Computing,” UIST2011). When I saw this presentation at UIST2011, it was followed by a question: wouldn’t you get more critical mass if you pay workers? and the response was telling of the future direction of the view of crowdsourcing as a field of inquiry: yes, but the second one incentivizes a system monetarily, it’s impossible to move away into other incentive structures, which is what we want to investigate more structured social computing. (Disclaimer: this is a many-months-old paraphrase!)

Reconsidering the notion of collaboration in crowdsourcing. Along with the assumption of adversarial behavior came the assumption of individual, asynchronous work. To enable accomplishing more high-level, larger tasks, research has begun to consider alternative models of collaboration within crowdsourcing platforms. Jeff Bigham’s work in trying to get masses of strangers to negotiate and operate an interface simultaneously (introduced by Lasecki et al, “Real-time Crowd Control of Existing Interfaces,” UIST2011). In my mind, this sits in contrast to work on platforms exploring novel approaches to organizing small work contributions in complex workflows to enable the completion of more high-level tasks, such as three CSCW2012 publications: Turkomatic by Kulkarni et al; CrowdWeaver by Kittur et al; and Shepherd by Dow et al; as well as a UIST2011 publication by Kittur et al on CrowdForge.

* For example, in a possibly overly-broad view, Wikipedia is sort of like a really unstructured crowdsourcing, because there’s these people doing small tasks to contribute to the construction of an accurate and complex encyclopedia. However, this more aptly termed social computing, according to Quinn and Bederson’s taxonomical distinctions.

Written by lab member, student, and deliciousness enthusiast Katie Kuksenok. Read more of her posts here on the SCCL blog, or on her own research blog, interactive everything.

Colors in Visualization

Posted by Katie Kuksenok on April 03, 2012

In Russian, there is a name for a light blue, and a name for a dark blue, but not for what English-speaking people call blue. Indeed, language shapes our understanding of color differences and categories.

Although it can be entertaining to think about how other factors, like gender, affect categorization of color, the survey deployed and analyzed by xkcd’s Randall Munroe showed that chromosomal gender mostly doesn’t matter (and that nobody can spell fuchsia).

Color is used widely in scientific visualization. The influence of language/culture on color perception impacts the interpretation of such visualization to some extent, but other measures can be taken to improve the use of color in visualizations:

Written by lab member, student, and deliciousness enthusiast Katie Kuksenok. Read more of her posts here on the SCCL blog, or on her own research blog, interactive everything.