From Flash Teams to Flash Fiction

Posted by Cecilia Aragon on January 13, 2015

When Andrés Monroy-Hernandez invited me to the Microsoft Research FUSE Labs 2015 Social Computing Symposium, I was thrilled.

Then he said, “We thought it would be fun to have a debate about the utopian and dystopian views around crowdsourcing. We’ll have [brilliant crowdsourcing expert] Michael Bernstein taking the utopian side. How would you like to take the dystopian side and lead off with a 10-minute presentation?”

Uh… thanks, Andrés?

Say again what you thought my qualifications in crowdsourcing were?

After the vision of being publicly humiliated subsided, I found myself saying, “That sounds like too much fun.”

So I wrote a science fiction story.

I figured I’d need to read it out loud in 8 minutes. At my normal reading rate of 150 words per minute, that gave me 1200 words to play with. In other words, flash fiction territory. Perfect.

(Recent work by Michael and his team at Stanford, Expert Crowdsourcing with Flash Teams, won a best paper award at UIST 2014.)

Thanks again to Andrés and Michael for an awesome session, including fantastic work on crowdsourcing by Niloufar Salehi, Meredith Ringel Morris, Yanni Antonellis, and Michael Galpert, as well as a discussion led by crowdworkers from TaskRabbit, AMT, and oDesk. (More details and video at

Afterwards, some folks asked me to post my crowdsourcing dystopia, so here it is.



Green or Not At All

I was born in debt.

Well, all children are, nowadays in the 2070s. It costs money to install an Oracle RFID, a Google Eye. The surgery is delicate, on a newborn.

But it has to be done. How else would you survive? No way to connect to the net, no way to work, no way to earn your living. No way to pay for food, water, air. Since the Taxation Ban of 2041, signed into law by the first World Bank president, a fundamental tenet of our Constitution has been that no free citizen should be required to pay for another’s upkeep. We are all free to make our own future, to live without the onerous burden of providing for someone else too lazy to work.

So as kids, we start getting out of debt as soon as possible. We’re lucky that, with the passing of the Children’s Freedom Act of 2049, co-sponsored by the oil senators, we’re now legally allowed to work from birth.

I was four when I worked my first hit. My parents were so proud of me. They told me my debt counter had gone down for the first time in my life. They pointed out the blinking red numbers on my wrist, but I didn’t really understand yet. All I knew is that I’d flicked my Eye over the Dasher just like I’d been practicing, and eyetyped in the blurry numbers I read from the image captcha on my eyescreen.

Sixty seconds of work, and it earned me my first dollar. My mom smiled for the first time that week and took me into her warm arms for a big hug, and my dad tapped my head with his knuckle, saying “This brain works.” My debt counter, which had barely reached a megadollar by then, had taken its first downward step.

The first of many, I vowed.

When I was old enough for kindergarten, every day I would skip down the hall from our cube to the elementary school on corridor B6, scan my wrist RFID at the big blue door, get it debited for my daily education charge, and join a hundred kids from our neighborhood sitting at carrels learning to read, eyetype, do math, and program; or if we were lucky, playing on the shaggy orange rug in front of the Window.

Just like all the other kids, I loved pressing my nose to the Window and looking out over the city. As far as the eye could see, tall buildings speared up into the pale brown sky until their edges were lost in the distance, shimmering in the heat. The sun glared down like an old-time furnace, and you could feel its searing blaze even through the thick filtered plexiglass. Every now and then, an Outside worker in all their protective gear would come rappelling down past the glass, waving at the children who clustered around in excitement.

“I’m going to work Outside when I’m grown,” bragged Willy, a tall boy whose debt counter was, amazingly, below a megadollar.

Kim sniffed. “You’ll never be able to afford the gear,” she scoffed. Nobody looked at her. Everyone knew Kim had the highest debt counter in the class. How she would dig herself out of that hole enough to qualify for a loan on the hugely expensive Outside gear, especially when everyone knew she was behind on her hit homework, was pretty much impossible. She didn’t work enough hits fast enough even to pay for her education, much less start slowing her debt counter’s ballooning growth.

But still, we all had big dreams. Don’t all kindergartners? I worked all my hit homework and then some, studiously avoiding the game tab on my Eye until I had surpassed my goal for the day. Still, my debt counter crept slowly upward. School, food, oxygen, all took their daily bites out of those numbers on my wrist. I just wasn’t good enough, at six years old, to work the higher-paying hits. But I played the hit lottery each day, dreaming of that magical hit that would make me a big winner.

“Play the Amazon Lottery! It only takes one hit to win! Imagine your debt counter, and your entire family’s, being reset to zero. You too could join these lucky winners in the world of green numbers!” The images of laughing families proudly displaying their wrists with the astonishing winking green of the rich flashed across my Eye, and I sighed and closed my eyes to work faster.

I would be rich one day. Anybody could be rich if they worked hard enough, if they earned a high enough rating to qualify for the more lucrative hits. Of course, you had to be careful. At six years old, Kim had already been blacklisted by many of the major corporations for doing poor quality work. If her rating kept going lower, she wouldn’t ever have a chance to get out of debt. And even worse, she might be crowdshunned, blacklisted by other workers, who didn’t want to work a team hit with her. Once you were crowdshunned, that was it. There was no chance of turning the numbers around. When she turned eighteen and couldn’t stay with her parents anymore, she would be kicked out of their cube and sent to one of the lower floors, where they couldn’t afford to pay for hallway air conditioning and where the oxygen levels were low. I’d been there on field trips, seen the people slumped in hot, dim corners, their wrists blinking red with unimaginably high numbers, and returned with a headache from lack of oxygen, sweaty and scared.

But I was going to be rich. Not only would I be rich, I would be an Employee one day, actually drawing a salary from JP Morgan, or Comcast, or Microsoft. My favorite net show was “Green Employees,” a sitcom about a group of friends who worked in one of the domes, played games on real grass, swam in pools and had private toilets, who actually received money every week that made their green numbers go up. The main character was a kid who had once been just like us. Just like me, he lay in his slot at night with the panel closed and worked hits until he fell asleep.

The villain of that show was a terrorist, who had once been the friend of the main character when they were kids. But he had tried to get the workers of their neighborhood to organize, to refuse to work hits that he said paid too little. But that made the requesters complain that if their labor costs went up too high, they would go out of business and stop creating jobs. They started withholding hits to that neighborhood. So the other workers crowdshunned the terrorist and he ran away, hiding somewhere in the city and trying to sabotage his former classmate. The main character was always getting in trouble because he had a soft spot for his terrorist friend.

As I caught the latest episode on my Eye while standing in line for the toilet one morning, I grimaced at the antics of the main character. When I grow up, I’ll never act soft like that.

After all, I’m from an upper-middle-class family. We have a responsibility to those less fortunate.

Tags: ,

Highlights of the First Annual CHI Play Conference

Posted by Daniel Perry on November 11, 2014

The Participatory Design Workshop at CHI Play

Posted by PhD student Daniel Perry

In late October I attended the first annual conference on Computer-Human Interaction in Play (CHI Play 2014) in Toronto. The conference brought together some 150 researchers, academics, and game designers across numerous areas of serious game research and HCI. Sessions at the conference covered a variety of topic areas, including collaboration and communication in serious games, games for health, gamification and education, and game analytics. Research I conducted on co-design with high school youth for the bioinformatics educational game MAX5 was one of several accepted abstracts in a workshop titled Participatory Design for Serious Game Design.

The Participatory Design for Serious Game Design workshop marked one of the highlights of the conference for me, as it brought together participants from all over the world (Malta, Belgium, Germany, Taiwan, Canada, and the U.S to name just a few) to explore the philosophical and methodological challenges of integrating participatory design and serious games. Participatory design (PD) has its roots in Scandinavia some forty years ago as way to empower workers and involve them more directly in the software design process. PD techniques have been adapted and used in a variety of fields and contexts with communities ranging from toddlers to the elderly. While PD often works hand-in-hand with user-centered design approaches, PD’s focus on directly integrating participant design concepts takes a more democratic stance on the design process. In the workshop, we worked in small groups mapping the field of participatory design as we saw it in our own research on games. Topics that emerged included: the use of PD to design games, as well as the use of games as a methodological way to design other types of software systems; the challenge of conveying domain knowledge to participants for learning games (in my own research, this involved getting high school youth up-to-speed with biology and computer science topics in our game); and deciding what design outcomes should be integrated into the final game. While there were no easy answers that came out of the workshop, the importance of transparency in design, as well as providing a tangible sense of participant contribution came up as important issues to address. It was exciting to feel that in many ways we were putting forth a new global agenda for the future of PD in serious games research.

Another highlight of the conference included a keynote by Mike Ambinder, an experimental psychologist at Valve Software (the Seattle-area company behind a host of game favorites including Portal and Left for Dead). In his talk, Mike discussed the current state of the art in gathering game data, and the frequent biases and challenges inherent in the process. He encouraged the audience to imagine the tools and methods that would fill in the data gaps in an ideal research world. I was left with the impression that if a game powerhouse like Valve was facing a daunting data landscape, there is much to gain from further discussions between industry veterans and the academic researcher community. I’m looking forward to attending CHI Play next year.



Hacked Ethnographic Fieldnotes from Astro Hack Week

Posted by Cecilia Aragon on October 29, 2014

First posted at the Astrohackweek blog

What is data science ethnography anyway?

As an ethnographer of data science, I immerse myself in particular communities to understand how they make sense of the world, how they communicate, what motivates them, and how they work together. I spent a week at astro data hack week, which might as well have been a foreign culture to me. I participated as an active listener, trying to sensitize myself to the culture and discern patterns that may not be self-evident to people within the community. Ethnography can have the effect of making the ordinary strange, such that the norms, objects, and practices that the community takes for granted become fascinating, informative sites for learning and discovery. Many of the astro hackers were probably thinking, “Why is this woman hanging around watching me code on my laptop? There is nothing interesting here.” But I assured them it was interesting to me because I was seeing their everyday practice in the context of a complex social and technical world that is in flux.

Ethnography can be thought of as a form of big data. Typically hundreds of pages of fieldnotes, interview transcripts, and artifacts from the field would be recorded over a long period of time until the ethnographer determines they have reached a point of saturation. The analysis process co-occurs with the data collection, iteratively shaping the focus of the research and observation strategy. Across this massive dataset with an abundance of unwieldy dimensions, the ethnographer has to make sense. The ethnographer works with members of the community to help them interpret what they are observing. Ethnographic insights, what many may term “findings”, emerge as patterns and themes are detected. Theory and new questions are generated, rather than tested. In this process I also acknowledge my own biases and prior assumptions and use them as ways to probe deeper and understand through them rather than ignore them. For instance, I came to astro data hack week not understanding much of anything people were talking about. It made me prone to feeling intimidated and I recognized with this intimidation my own reticence to ask questions. My own experience with this feeling helped me identify in others that were also feeling variations of this and also be able to identify what helped transform that feeling throughout the week into a more comfortable and curious state.

I only spent 5 days among the community of astro hackers, but in the spirit of hacking, I have a few “hacked” fieldnotes to share. Sharing is a key component of the hack week and as a participant I feel it is important to follow suit. But bear in mind these thoughts are preliminary. So, what have I been working on this week?

Initial descriptive observations from an outsider (a little tongue-in-cheek, forgive me):

  • Astro hackers live in a very dusty, dirty, and noisy environment! Very hard to keep clean and elaborate measures are taken to obtain a signal. But when the signal is too strong or the data too clean, there is a feeling of mistrust.
  • The common language is Python, although there are many other dialects, some entirely made of acronyms, others sound like common names, such as George and Julia.
  • When talking there is always some form of data, documentation or model that mediates the conversation, whether it is on the white board, on the screen, or through representational gestures.
  • Although most people are studying something that has to do with astronomy, they can literally be operating on “different wavelengths”!
  • Astro hackers play with “toys” and “fake data” as much as “real world data”!
  • Coffee and beer fuel interactivity!


Josh Bloom teaches Machine Learning

Data science at the community level: From T to Pi to Gamma-shaped (Josh Bloom’s term) scientists: Across the group I heard over and over again in various ways reference and deference to those who are more expert, those who are smarter or those who know more than I do. Granted, this is a somewhat common occurrence in the culture of academia as we are continuously humbled by the expertise around us. However, I found this particularly acute and concentrated within this community. What I heard across students, postdocs, and research scientists was more than the typical imposter syndrome. It was the feeling that they are expected to be experts or at the very least fluent in a range of computing and statistical areas in addition to their own domain. While this motivates people to be at a hack week such as this, it can also have the unintended effect of making people intimidated and overwhelmed with having to know everything themselves. This can have a chilling effect across the community. This means the feeling that other people know more than they do is pervasive and this often leads to thinking their questions aren’t valuable for the rest of the group, and therefore, not worth sharing. This is a negative thing and we want to ensure this effect is minimized. Not only is it bad for morale; it is bad for science. We should consider who feels comfortable taking a risk in these settings? A risk might be asking a question that they fear isn’t scientifically interesting for others. Or sharing something that isn’t complete or isn’t perfect. If we take what Josh Bloom says, that we might be better off thinking about data science on the community level, happening in a more distributed way, rather than data science on the individual level, we can begin to paint a different picture and change some of the expectations that may trigger this negative effect.

Josh Bloom’s lecture on machine learning explained the popular idea of “Pi-shaped” individuals (a buzz word for the academic data science community) and his preference, for talking about “Gamma-shaped” individuals. Rather than promote the idea that there is an expectation of individuals having expert-level depth in two domains, which is unrealistic for the majority of people, what if we thought of people as Gamma-shaped? These people would have expert-level depth in one domain and also be versed and proficient in other domains. Someone with their PhD in biology may be conversant in the language and culture of computer science enough to have conversations and collaborate, but they don’t necessarily need to be an expert in computer science to the extent that they are able to advance the discipline. These Gamma-shaped individuals can work with each other to bridge multiple domains of expertise. This Gamma symbol better reflects the makeup of individuals in this astro hack week community and this view of data science allows for the expectations to shift to the community and to the collaborative interactions between people. This shift is important and has implications for thinking about how to better structure hack week. For instance, with these tweaked expectations a learning goal of the hack week might be working together across Gamma-shaped individuals.

Categorizing hacking interactions I categorized the different kinds of hacking interactions I observed over the course of the week. This list is not meant to be exhaustive, but it might be helpful in understanding the diversity of interactions and how to facilitate the types of hacking interactions desired.

  • Resource Seeking: An individual works on their hack idea and uses other people as sources of expertise when they need help
  • Asymmetical Synergy: A pair or small group joins together to work on a hack idea in which one person is learning something, such as an algorithm, and the other has more advanced knowledge and is exploring what that algorithm can do. They are generating something together but getting different things out of the activity.
  • Symmetrical Synergy: A pair or small group joins together to work on a hack idea and iteratively discovers how their expertise informs the other, or how interests synergize. Then, they generate something new together.
  • Comparing Notes: An individual works on their hack idea and shares it with others based on their common interest. A form of comparing notes in which they are talking about the work more broadly and loosely.
  • Learning Collective: A semi-structured activity that draws multiple people in to learn something collectively, thus creating a learning collective.

The Importance of “Connective Tissue”

Across this community there is great diversity across institution, dataset, data source, methodology, computing tools and packages, statistical approach, status within academia, and level of knowledge in different arenas. This creates many opportunities for discovering connections, for sharing, and working together. Yet this also presents challenges for forging these connections especially within the broader academic environment which in many ways doesn’t incentivize collaboration and “failing fast”. Failing fast refers to the capacity to be highly experimental, to take risks, and invest a little bit often, such that when things don’t work, it is framed much more as part of the iterative process rather than as a significant loss. In a culture where people are failing fast, people are more likely to take risks and learning can happen more rapidly.

A key and essential role that emerged this week was the set of capacities for facilitating connection across people and ideas, what Fernando Perez has called the “connective tissue”. There is a need both the people and the organizational structure that supports social and technical resonances across a wide range of people and can facilitate connections among them. These people can play a role of translation across ideas that might appear otherwise unrelated. They also provide coaching (as opposed to teaching) to help both identify and achieve their goals. We should all be learning from these people so that we can all contribute to the connective tissue. This connective tissue developed further throughout the week. Specifically, the more semi-structured collective learning activities and the emphasis on working in pairs greatly increased the productivity across the group (there was more to show at the end of the day) and the interaction (fewer people with earphones in and more talking). I also observed many more small and big shared victories. I hadn’t yet seen a high five and I saw two instances on Thursday, which reflected the overall sense that the victory was about more than an individual completing the hack, rather it was shared and celebrated together.

This hack week performs as a kind of lab space where people can take risks and work together in new ways that they might not be incentivized to do otherwise. It is an opportunity to change the incentives for a short period of time. In fact, the frictions that we see emerge in this hack week (i.e. people needing to work towards publications) reflect some of the default incentives clashing with hack week incentives. For future hack weeks it might be important to advocate failing fast through normalizing it and facilitating a supportive environment for risk taking. In addition, part of the goal of a future hack week might be more explicitly to learn about how to work together and what it takes to develop connective tissue through incentivizing a range of different hacking interactions.

Work Life Balance

Posted by Katie Kuksenok on October 13, 2014

During the first meeting of the new quarter, our lab meeting consisted of each member talking about what they did this summer: be it professional achievement or a personal one. We laughed together. We ate pizza and a root vegetable medley made by one of the students, as per last year’s tradition to share food during meetings which had to be during mealtimes due to our excessively overwhelming schedules. We applauded for especially noteworthy remarks, such as: making a plan to graduate soon (2x), submitting a paper to the most recent Big Deal Conference deadline (4x), getting married (1x),and managing to have an actual honest-to-goodness vacation (3x). In our meetings for the last few years, we have allowed the unrelated to seep in, and I think it has improved both the variety and the caliber of our work. Instead of seeing these asides as distractions, we engaged with each other about a huge variety of research topics, as well as human topics.

In my own multi-year struggle with work-life balance (aka, “four years of grad school”), I have found it useful to have one core assumption. Even though I work on a million of seemingly-unrelated projects, they are necessarily and fundamentally related: because they are mine, and are built on the same body of knowledge. In this sense, every intellectually-stimulating conversation that grabs my attention is, by definition, relevant. It is relevant to my perception of the world, and I take note of it. Incidentally, when I began to pursue this sense of “wholeness,” it helped to ease the dreaded (and all-too-common) “impostor syndrome,” the haunting sense of being found out as far less competent than I appear. On the one hand, yes, with anything that I do, there are many people in the world who are much better at that thing than I am. But they are all not me, they do not have the combined idiosyncratic background I bring to the table: the whole has more creative variety to draw from than the sum of its parts. So I can feel both more secure in myself, and relieved that there is always someone to save you from excruciating (and boring) intellectual solitude with advice, feedback, and debate.

“How did you get over anxiety during giving talks?” one of the students asked Cecilia in an aside in a meeting a few years ago. “Well, when you’ve flown a plane straight at the ground at 250 mph at an airshow with hundreds of thousands of people watching, it’s difficult to be too stressed out about other things.” Professor Aragon leads our lab, teaches classes, and occasionally shares what she learned from the time she was an aerobatic champion. Instead of viewing “work life balance” as something of a separation between our “work” selves and our “life” selves, we’re building empathy within the group, as well as sharing with one another our wonderful variety of experiences and lessons.