Hacked Ethnographic Fieldnotes from Astro Hack Week

Posted by fioreb on October 29, 2014

First posted at the Astrohackweek blog

What is data science ethnography anyway?

As an ethnographer of data science, I immerse myself in particular communities to understand how they make sense of the world, how they communicate, what motivates them, and how they work together. I spent a week at astro data hack week, which might as well have been a foreign culture to me. I participated as an active listener, trying to sensitize myself to the culture and discern patterns that may not be self-evident to people within the community. Ethnography can have the effect of making the ordinary strange, such that the norms, objects, and practices that the community takes for granted become fascinating, informative sites for learning and discovery. Many of the astro hackers were probably thinking, “Why is this woman hanging around watching me code on my laptop? There is nothing interesting here.” But I assured them it was interesting to me because I was seeing their everyday practice in the context of a complex social and technical world that is in flux.

Ethnography can be thought of as a form of big data. Typically hundreds of pages of fieldnotes, interview transcripts, and artifacts from the field would be recorded over a long period of time until the ethnographer determines they have reached a point of saturation. The analysis process co-occurs with the data collection, iteratively shaping the focus of the research and observation strategy. Across this massive dataset with an abundance of unwieldy dimensions, the ethnographer has to make sense. The ethnographer works with members of the community to help them interpret what they are observing. Ethnographic insights, what many may term “findings”, emerge as patterns and themes are detected. Theory and new questions are generated, rather than tested. In this process I also acknowledge my own biases and prior assumptions and use them as ways to probe deeper and understand through them rather than ignore them. For instance, I came to astro data hack week not understanding much of anything people were talking about. It made me prone to feeling intimidated and I recognized with this intimidation my own reticence to ask questions. My own experience with this feeling helped me identify in others that were also feeling variations of this and also be able to identify what helped transform that feeling throughout the week into a more comfortable and curious state.

I only spent 5 days among the community of astro hackers, but in the spirit of hacking, I have a few “hacked” fieldnotes to share. Sharing is a key component of the hack week and as a participant I feel it is important to follow suit. But bear in mind these thoughts are preliminary. So, what have I been working on this week?

Initial descriptive observations from an outsider (a little tongue-in-cheek, forgive me):

  • Astro hackers live in a very dusty, dirty, and noisy environment! Very hard to keep clean and elaborate measures are taken to obtain a signal. But when the signal is too strong or the data too clean, there is a feeling of mistrust.
  • The common language is Python, although there are many other dialects, some entirely made of acronyms, others sound like common names, such as George and Julia.
  • When talking there is always some form of data, documentation or model that mediates the conversation, whether it is on the white board, on the screen, or through representational gestures.
  • Although most people are studying something that has to do with astronomy, they can literally be operating on “different wavelengths”!
  • Astro hackers play with “toys” and “fake data” as much as “real world data”!
  • Coffee and beer fuel interactivity!

Themes


Josh Bloom teaches Machine Learning

Data science at the community level: From T to Pi to Gamma-shaped (Josh Bloom’s term) scientists: Across the group I heard over and over again in various ways reference and deference to those who are more expert, those who are smarter or those who know more than I do. Granted, this is a somewhat common occurrence in the culture of academia as we are continuously humbled by the expertise around us. However, I found this particularly acute and concentrated within this community. What I heard across students, postdocs, and research scientists was more than the typical imposter syndrome. It was the feeling that they are expected to be experts or at the very least fluent in a range of computing and statistical areas in addition to their own domain. While this motivates people to be at a hack week such as this, it can also have the unintended effect of making people intimidated and overwhelmed with having to know everything themselves. This can have a chilling effect across the community. This means the feeling that other people know more than they do is pervasive and this often leads to thinking their questions aren’t valuable for the rest of the group, and therefore, not worth sharing. This is a negative thing and we want to ensure this effect is minimized. Not only is it bad for morale; it is bad for science. We should consider who feels comfortable taking a risk in these settings? A risk might be asking a question that they fear isn’t scientifically interesting for others. Or sharing something that isn’t complete or isn’t perfect. If we take what Josh Bloom says, that we might be better off thinking about data science on the community level, happening in a more distributed way, rather than data science on the individual level, we can begin to paint a different picture and change some of the expectations that may trigger this negative effect.

Josh Bloom’s lecture on machine learning explained the popular idea of “Pi-shaped” individuals (a buzz word for the academic data science community) and his preference, for talking about “Gamma-shaped” individuals. Rather than promote the idea that there is an expectation of individuals having expert-level depth in two domains, which is unrealistic for the majority of people, what if we thought of people as Gamma-shaped? These people would have expert-level depth in one domain and also be versed and proficient in other domains. Someone with their PhD in biology may be conversant in the language and culture of computer science enough to have conversations and collaborate, but they don’t necessarily need to be an expert in computer science to the extent that they are able to advance the discipline. These Gamma-shaped individuals can work with each other to bridge multiple domains of expertise. This Gamma symbol better reflects the makeup of individuals in this astro hack week community and this view of data science allows for the expectations to shift to the community and to the collaborative interactions between people. This shift is important and has implications for thinking about how to better structure hack week. For instance, with these tweaked expectations a learning goal of the hack week might be working together across Gamma-shaped individuals.

Categorizing hacking interactions I categorized the different kinds of hacking interactions I observed over the course of the week. This list is not meant to be exhaustive, but it might be helpful in understanding the diversity of interactions and how to facilitate the types of hacking interactions desired.

  • Resource Seeking: An individual works on their hack idea and uses other people as sources of expertise when they need help
  • Asymmetical Synergy: A pair or small group joins together to work on a hack idea in which one person is learning something, such as an algorithm, and the other has more advanced knowledge and is exploring what that algorithm can do. They are generating something together but getting different things out of the activity.
  • Symmetrical Synergy: A pair or small group joins together to work on a hack idea and iteratively discovers how their expertise informs the other, or how interests synergize. Then, they generate something new together.
  • Comparing Notes: An individual works on their hack idea and shares it with others based on their common interest. A form of comparing notes in which they are talking about the work more broadly and loosely.
  • Learning Collective: A semi-structured activity that draws multiple people in to learn something collectively, thus creating a learning collective.

The Importance of “Connective Tissue”

Across this community there is great diversity across institution, dataset, data source, methodology, computing tools and packages, statistical approach, status within academia, and level of knowledge in different arenas. This creates many opportunities for discovering connections, for sharing, and working together. Yet this also presents challenges for forging these connections especially within the broader academic environment which in many ways doesn’t incentivize collaboration and “failing fast”. Failing fast refers to the capacity to be highly experimental, to take risks, and invest a little bit often, such that when things don’t work, it is framed much more as part of the iterative process rather than as a significant loss. In a culture where people are failing fast, people are more likely to take risks and learning can happen more rapidly.

A key and essential role that emerged this week was the set of capacities for facilitating connection across people and ideas, what Fernando Perez has called the “connective tissue”. There is a need both the people and the organizational structure that supports social and technical resonances across a wide range of people and can facilitate connections among them. These people can play a role of translation across ideas that might appear otherwise unrelated. They also provide coaching (as opposed to teaching) to help both identify and achieve their goals. We should all be learning from these people so that we can all contribute to the connective tissue. This connective tissue developed further throughout the week. Specifically, the more semi-structured collective learning activities and the emphasis on working in pairs greatly increased the productivity across the group (there was more to show at the end of the day) and the interaction (fewer people with earphones in and more talking). I also observed many more small and big shared victories. I hadn’t yet seen a high five and I saw two instances on Thursday, which reflected the overall sense that the victory was about more than an individual completing the hack, rather it was shared and celebrated together.

This hack week performs as a kind of lab space where people can take risks and work together in new ways that they might not be incentivized to do otherwise. It is an opportunity to change the incentives for a short period of time. In fact, the frictions that we see emerge in this hack week (i.e. people needing to work towards publications) reflect some of the default incentives clashing with hack week incentives. For future hack weeks it might be important to advocate failing fast through normalizing it and facilitating a supportive environment for risk taking. In addition, part of the goal of a future hack week might be more explicitly to learn about how to work together and what it takes to develop connective tissue through incentivizing a range of different hacking interactions.

Work Life Balance

Posted by Katie Kuksenok on October 13, 2014

During the first meeting of the new quarter, our lab meeting consisted of each member talking about what they did this summer: be it professional achievement or a personal one. We laughed together. We ate pizza and a root vegetable medley made by one of the students, as per last year’s tradition to share food during meetings which had to be during mealtimes due to our excessively overwhelming schedules. We applauded for especially noteworthy remarks, such as: making a plan to graduate soon (2x), submitting a paper to the most recent Big Deal Conference deadline (4x), getting married (1x),and managing to have an actual honest-to-goodness vacation (3x). In our meetings for the last few years, we have allowed the unrelated to seep in, and I think it has improved both the variety and the caliber of our work. Instead of seeing these asides as distractions, we engaged with each other about a huge variety of research topics, as well as human topics.

In my own multi-year struggle with work-life balance (aka, “four years of grad school”), I have found it useful to have one core assumption. Even though I work on a million of seemingly-unrelated projects, they are necessarily and fundamentally related: because they are mine, and are built on the same body of knowledge. In this sense, every intellectually-stimulating conversation that grabs my attention is, by definition, relevant. It is relevant to my perception of the world, and I take note of it. Incidentally, when I began to pursue this sense of “wholeness,” it helped to ease the dreaded (and all-too-common) “impostor syndrome,” the haunting sense of being found out as far less competent than I appear. On the one hand, yes, with anything that I do, there are many people in the world who are much better at that thing than I am. But they are all not me, they do not have the combined idiosyncratic background I bring to the table: the whole has more creative variety to draw from than the sum of its parts. So I can feel both more secure in myself, and relieved that there is always someone to save you from excruciating (and boring) intellectual solitude with advice, feedback, and debate.

“How did you get over anxiety during giving talks?” one of the students asked Cecilia in an aside in a meeting a few years ago. “Well, when you’ve flown a plane straight at the ground at 250 mph at an airshow with hundreds of thousands of people watching, it’s difficult to be too stressed out about other things.” Professor Aragon leads our lab, teaches classes, and occasionally shares what she learned from the time she was an aerobatic champion. Instead of viewing “work life balance” as something of a separation between our “work” selves and our “life” selves, we’re building empathy within the group, as well as sharing with one another our wonderful variety of experiences and lessons.

Oberlin Winter Term 2013

Posted by Katie Kuksenok on April 02, 2013

For the month of January, three Oberlin College undergraduates – Dan Barella, Sayer Rippey, and Eli Rose – joined SCCL to work on extending our command-line tool for affect detection using machine learning, ALOE. The Winter Term internship was initially conceived by Katie Kuksenok, one of the two Oberlin alumni in SCCL; the other, Michael Brooks, also helped in mentoring the students while they were on campus.

obies2013

Each of the visiting Obies contributed a new functionality and compared its performance to that reported in our CSCW report; Dan implemented a novel segmentation algorithm, Sayer extended feature extraction to process French chat messages rather than only English, and Eli worked on HMM classification. Having returned to Oberlin, Sayer continues to work on analyzing the French portions of the dataset as an independent research project, collaborating over distance.

It has been an incredible month. Besides being blown away by the Seattle public transit system, I got to learn so much about machine learning, language, and grad school, and I got to meet a lot of smart, passionate, inspiring people.
The work I did applying the ALOE pipeline to French was completely fascinating. It was great because I got to be doing something very practical, trying to get the labeler working for the rest of the pipeline, but it also brought up some really interesting differences between French and English.

- Sayer Rippey

So, here I am at the end of Winter Term. I’m already nostalgic! This project was really enrapturing, and the whole experience thoroughly enjoyable. … I will say that, I’m proud of the work I’ve done. There are some places where I know there’s room for improvement, but to believe otherwise would perhaps be worse. I can’t claim that it’s all perfect, but I can claim that I did nearly all that I set out to do, and then some that I hadn’t expected to do. I didn’t expect I’d have to put together a profiling script to test my project, and yet this turned out to be one of the most invaluable tools I’ve had for code analysis (hopefully for others as well). I didn’t expect to find such a subtle tradeoff between a small tweaking of time variables, and yet this became a central issue of the last two weeks of my project. I didn’t think comparing pipeline statistics would be so nuanced, but now I’m beginning to see all the ways that a visualization can change the way people perceive information. I could go on, but what I’m really trying to say is: I learned so many new things!

But the most exciting parts of this Winter Term were not the components of my project. They were the incredible people at the SCCL, who brought me to lectures and talks on the nature of artificial intelligence and information visualization, who always provided novel viewpoints and provoking discussions, who were dedicated to sharing their unbelievable experience in so many topics. I was honored to work with Eli, Sayer, Katie, Michael, Megan, Cecilia, and the rest of this great team. They’ve humbled and challenged me, and for that I thank all of them; as this term comes to a close, I hope only that I should be so lucky in pursuit of future endeavors as I was in finding this one. So to everyone at the SCCL, so long, and thanks for all the fish!

- Dan Barella

Trends in Crowdsourcing

Posted by Katie Kuksenok on April 09, 2012

These several recent years have seen the rise of crowdsourcing as an exciting new tool for getting things done. For many, it was a way to get tedious tasks done quickly, such as transcribing audio. For many others, it was a way to get data: labeled image data, transcription correction data, and so on. But there is also a layer of meta-inquiry: what constitutes crowdsourcing? Who is in the crowd, and why? What can they accomplish, and how might the software that supports crowdsourcing be designed in a way to help them accomplish more?

Each of the last two conferences I have attended, CSCW2012 and UIST2011, had a “crowdsourcing session,” spanning a range of crowdsourcing-related research. But only a short while before that, the far bigger CHI conference contained only one or two instances of “crowdsourcing papers.” So what happened in the last few years?

At some point in the last decade, crowdsourcing emerged both as a method for getting lots of tedious work done cheaply, and a field of inquiry that resonated with human-computer interaction researchers. Arguably, this point historically coincided with the unveiling of Amazon Mechanical Turk platform, which allowed employers, or “requesters,” to list small, low-paid tasks, or “human-intelligence tasks (HITs)” for anonymous online contractors, or “workers,” to complete. In Amazon’s words, this enabled “artificial artificial intelligence” – the capacity to cheaply get answers to questions that cannot be automated.

And so this crowdsourcing thing took academic literature by storm, evidenced by growth in yearly additions to work exposed via Google scholar, as the related terms “crowdsourcing,” “mechanical turk,” and “human computation” seemed to grow rapidly at roughly the same time:

Cumulative term use over time in Google Scholar

This point in time, barring what could be argued as noise in the extremely coarse metric of Google scholar yearly result count, pretty much coincides with the 2008 unveiling of Mechanical Turk:

Google Trends

There is a distinction between HCI research that (1) uses crowdsourcing; (2) investigates crowdsourcing as a platform is capable of, which is yet distinct from that which (3) investigates what crowdsourcing is, or ought to be. Kittur, Chi, and Suh authored one of the first papers of the second variety, explaining how crowdsourcing – via Mechanical Turk in particular – could be used as a method for running user studies distinct from typical approaches in HCI literature (“Crowdsourcing User Studies with Mechanical Turk,” CHI 2007). Later, at CHI2010, a paper of the third variety – characterizing the demographics of Mechanical Turk workforce – was presented at an alt.chi session by Ross et al from UC Irvine (“Who are the Crowdworkers? Shifting Demographics in Mechanical Turk,” alt.chi 2010). Since, the third kind of research has begun to re-examine aspects of crowdsourcing that have been taken as nearly axiomatic as a result of its initial synonymy with Mechanical Turk.

Adversarial workers and the “arms race.” The kind of task that today is said to use crowdsourcing was already happening well before Mechanical Turk took the world by storm. Luis von Ahn had launched the ESP game and reCAPTCHA several years before. Still, it wasn’t until 2008 that Science published the paper on reCAPTCHA (“reCAPTCHA: Human-Based Character Recognition via Web Security Measures,” Science online 2008). Recently, the term human computation and crowdsourcing became used in lieu of one another frequently enough to warrant a 2011 survey of what those words mean in CHI by Quinn and Bederson* (“Human computation: a survey and taxonomy of a growing field,” CHI2011). The historical roots of crowdsourcing in Amazon Mechanical Turk seem to have resulted in a view of the stranger-workers as particularly adversarial strangers who can be used to generate data for any and all tasks. This has led to a body of research sometimes referred to as the “crowdsourcing arms race:” the tension between the worker’s [presumed] desire to do as little work as quickly as possible, and the requester’s [presumed] desire to paid as little as possible for as much work as possible. For example, quality control has moved from asking workers to fill out surveys to recently using biometrics to identify intentionally shoddy work, or “cheating” (J. Rzeszotarski and A. Kittur. “Instrumenting the Crowd: Using Implicit Behavioral Measures to Predict Task Performance,” UIST2011).

Beyond monetary incentives. Another artifact of historical connection into Amazon Mechanical Turk has been conflating crowdsourcing and human computation with paid work markets, and of monetary incentives. But there are other models – such as the initial Luis von Ahn games, where human computation is fuelled by fun. Or Jeff Bigham’s VizWiz, where motivation can border altruism – helping blind people interpret certain objects quickly – despite having monetary incentives via Mechanical Turk. Salman Ahmad and colleagues’ work on Jabberwocky crowd programming environment is a particularly deliberate departure from the monetary incentive model, recasting itself as “structured social computing” despite overlapping computational elements with crowdsourcing (“The Jabbrwocky Programming Environment for Structured Social Computing,” UIST2011). When I saw this presentation at UIST2011, it was followed by a question: wouldn’t you get more critical mass if you pay workers? and the response was telling of the future direction of the view of crowdsourcing as a field of inquiry: yes, but the second one incentivizes a system monetarily, it’s impossible to move away into other incentive structures, which is what we want to investigate more structured social computing. (Disclaimer: this is a many-months-old paraphrase!)

Reconsidering the notion of collaboration in crowdsourcing. Along with the assumption of adversarial behavior came the assumption of individual, asynchronous work. To enable accomplishing more high-level, larger tasks, research has begun to consider alternative models of collaboration within crowdsourcing platforms. Jeff Bigham’s work in trying to get masses of strangers to negotiate and operate an interface simultaneously (introduced by Lasecki et al, “Real-time Crowd Control of Existing Interfaces,” UIST2011). In my mind, this sits in contrast to work on platforms exploring novel approaches to organizing small work contributions in complex workflows to enable the completion of more high-level tasks, such as three CSCW2012 publications: Turkomatic by Kulkarni et al; CrowdWeaver by Kittur et al; and Shepherd by Dow et al; as well as a UIST2011 publication by Kittur et al on CrowdForge.

* For example, in a possibly overly-broad view, Wikipedia is sort of like a really unstructured crowdsourcing, because there’s these people doing small tasks to contribute to the construction of an accurate and complex encyclopedia. However, this more aptly termed social computing, according to Quinn and Bederson’s taxonomical distinctions.

Written by lab member, student, and deliciousness enthusiast Katie Kuksenok. Read more of her posts here on the SCCL blog, or on her own research blog, interactive everything.