Submissions List TBI Interagency Conference

Predictive Model for Accessible Web Automation

  • Borodin, Yevgen


Applications of technology to the needs of people with disabilities


We will report on the progress made by Charmtech Labs LLC toward developing an intelligent web automation interface that learns to predict user actions in repetitive web browsing tasks. We will present the results of our experiments and user studies, discuss technical and scientific challenges, and outline future work.


We often find ourselves repeating the same browsing steps every time we shop, make reservations, or pay bills online. While this is not a major time-sink for most sighted people, it can take a visually impaired person up to ten times as long to go through the same online transactions. This is because blind people use screen readers, which narrate the content of the screen sequentially, restricting the user to a serial mode of interaction. Thus, while sighted people can visually find the form fields they need to fill and the buttons they need to press (e.g., the “check out” button, “shipping address” field, etc.), blind people have to complete the same tasks often listening through volumes of irrelevant content. This problem substantially reduces the productivity and even inhibits the employability of blind people for jobs requiring repetitive web transactions such as purchasing.

In the U.S. alone, there are over 20 million people with low vision, and approximately 1.3 million are considered legally blind, as reported by the American Foundation for the Blind. Many of these people have come to rely on the Web for transactions such as shopping, banking, making travel arrangements, applying for college or employment, etc. Web automation has the potential to bridge the accessibility divide between the ways blind and sighted people perform Web transactions; specifically, automation can enable blind people to breeze through repetitive web browsing tasks that beforehand were slow, hard, or even impossible to achieve.

Typical automation interfaces require that the user explicitly record a macro, a useful sequence of browsing steps so that these steps can be replayed in the future. Additionally, the existing automation tools are not sufficiently flexible to recognize the fact that the user may need to deviate from the prerecorded transaction steps or that the user may have several options in each step of the transaction. Unfortunately, these constraints limit the adoption of web automation technology by blind users. As a result, blind users continue to waste their time on repetitive web browsing.

In this talk, we will report on the progress made by Charmtech Labs LLC toward developing an intuitive and accessible web automation interface that does not require that the user record macros. We will propose an intelligent interface and a model that learns to predict user actions in repetitive web browsing tasks by analyzing the user browsing history. We will present our results of the preliminary experiments that exhibit high accuracy of prediction. We will also summarize the results of the human-subjects experiments demonstrating that the interface has the potential to improve significantly accessibility and usability of web pages by reducing interaction time and by increasing user satisfaction.

The predictive automation approach has been developed under a grant from the Department of Education, the NIDRR grant number H133S110023: “Goal-Oriented Browsing: The Next Generation Web Accessibility Technology for People with Visual Impairments”. However, the contents do not represent the policy of the Department of Education, and you should not assume the endorsement by the Federal Government.