SCC Lab member and CSE PhD student Katie Kuksenok and SCC Lab director Cecilia Aragon were co-authors on a recent article accepted to the Journal of Surgical Research. The article titled “Crowd-Sourced Assessment of Technical Skills (C-SATS): A Novel Method to Evaluate Surgical Performance” marks the first use of crowdsourcing for surgical skills assessment. The article was co-authored by Carolyn Chen, Lee White, Timothy Kowalewski, Rajesh Aggarwal, Chris Lintott, Bryan Comstock, Katie Kuksenok, Cecilia Aragon, Daniel Holst, and Thomas Lendvay.
The article explores the effectiveness of large crowds sampled from two on-line crowdsourcing venues, Amazon.com’s Mechanical Turk and Facebook, testing the hypothesis that crowdsourcing of technical skills using validated surgical assessment tools is equivalent to assessment by experienced surgeon educators. Their results show that not only could crowds, presumably unfamiliar with surgical education, rate a common robotic surgery suturing task equivalent to experienced surgeons’ ratings, but that the crowds could also be honed to identify crowd workers who demonstrated markers of critical thinking making the workers more accurate. While this research finding does not presume that such a rating can assess surgical judgement, they note that this observation is not unlike being able to identify good from bad athletic performances in a sport one may have no ability to play.
This research represents a departure from conventional wisdom and practice in the area of procedural skills education. CSATS may provide a potential opportunity to disseminate basic technical skills assessment rapidly and globally while preserving educator resources ‘on the ground’ for refined, tailored advanced technical skills curricula to accelerate individual learning curves.
