Archive for February, 2010

API keys (tokens) to track api usage

As many of you know we have a set of public APIs for the Student Web Service. Public APIs mean that access to those APIs do not require any form of authentication. For months now I have seen a trend that has been bothering me. Our percentage of http requests using public apis who are not setting an application useragent or using a X.509 cert has been steady increasing to 14% of all SWS requests. Note that using an application useragent or X.509 cert is not required for public API access but if made available those are two ways that we can track usage. Yes, its a good thing we are seeing more traffic to our Student Web Service however some of us are concerned that we are losing our ability to track usage of our apis and are unable to support those clients properly.

Members of the ROA-technical team recently got together to discuss pros and cons of whether we should require the use of API keys or tokens (keys and tokens are used interchangeably throughout this post) to access our public APIs. Here is what we discussed but did not come to a decision because that was NOT an agenda item for our meeting. Your input to this blog can help influence what future decision we decide to take on this issue.

What is the benefit of including a token?

The benefit of registering for an api key or token is simply better client support and capturing application usage data so that service developers know what services to keep, not keep, make better, or not make better.

What would be some things to think about when implementing the use of an API key/token?

    If implemented:

  • We should only require an API key for public apis. The addition of using a cert and an API key for access to private resources seem overkill.
  • The key should be required to be passed to the api on every request.
  • The api key can be reused among many applications and there is no way to enforce the use of one token per web application. We can however set a policy that a developer keep the token a secret and only uses it for apps written within their department. If we do not set this policy then our application usage tracking data will likely become skewed and unusable.
  • The api key should be embedded in the URI and not in the HTTP header. This makes it easier for users that just want to browse the service in their browser.

Would introducing the requirement for a token be a violation of the contract with developers?

  • The answer is a resounding YES; if we made it mandatory.
  • It was recommended that we make the key/token optional which we all came to agreement on. If you get a token then developers are in a better position to be supported while those who dont are not. This policy can be stated explicitly while developers are registering their application.
  • So does the benefit of requiring an API key (token) outweigh the costs?

    Benefits:

  • Better usage tracking
  • Better client support
  • Costs:

  • Add new features to the UW Web Services Registry to allow for app registrations. This would require some user interface additions and a token/key validation web service.
  • Update documentation and communicate to our clients on our new API key policy
  • Update all web services to accept API keys on every request and validate them
  • Let us know what you think.

    4 Comments

    A look at chatty vs chunky RESTful web services

    While helping develop the new Student Web Service Enrollment resource, released just last week, the development team came across a RESTful design decision related to making our services more chatty or chunky. So what is the difference?

    Chatty services tend to be ones that return simplified information and use more fine-grained operations. Chunky services tend to return complex hierarchies of information and use coarse operations. In other words, the difference is that chatty services require more calls than chunky services to return the same information but allow more flexibility to return only the information that is actually required.
    - Rory Primrose (http://www.neovolve.com/page/WCF-service-contract-design.aspx)

    This is a discussion we often have while designing our RESTful web services here at the UW. There is no clear cut answer on which is better to use because your design decision depends on the use cases of the consuming applications. For the Enrollment resource, we knew most applications would want all the student’s transcript data in one GET request, which led us to design a chunky interface.

    Since we were retrieving an entire student’s academic history in one call our performance numbers for this resource was much slower than our other resources. Given our immediate clients wanted this exact design and our clients understood that retrieving years worth of data is not a fast operation; we decided that the performance numbers for our chunky design was okay. However we did not want to handcuff our future users and have them suffer the performance penalty if they only wanted a subset of the data like a student’s Majors data. With this in mind, we decided to offer a non-chunky (chatty) interface that provided the Majors and ClassLevel data and included a href link to the other half of a student’s transcript data. This was a significant performance improvement for the resource and incurred very little development time to offer this additional chatty pay load.

    The educational cost of offering additional options to the client developers meant we had to help guide choices of when its better to use a chatty or chunky interface. I put together some performance related data in hopes that it would help client developers decide on when to use a chatty interface vs a chunky interface. The data below is a combination or real and calculated data. Due to the lack of time I was not able to develop an appropriate test harness to provide real numbers for all test cases.

    The numbers below are based on a student with 9 enrollments totaling 35 registrations. There are 4 SWS resources here in play – Enrollment Search, Enrollment, Registration Search and Registration. Here is the call path to get to all the data when making multiple chatty calls back to the web service. EnrollmentSearch -> Enrollment (includes Registration Search) -> Registration.

    Enrollment Search (Chunky vs Chatty) Comparision

    Enrollment Search (Chunky vs Chatty) Comparision

    To help with chart translation:
    Chatty = Verbose Off
    Chunky = Verbose On

    Resource Type Seconds Notes
    Enrollment Search Chunky 4.52 Based on real load data from our test environment
    Enrollment Search Chatty Sequential GETS 12.1 (9 * Enrollment Chatty) + (35 * Registration Resource) + Enrollment Search Chatty
    Enrollment Search Chatty Multi-Threaded GETS 2.5 (2 * Enrollment Chatty) + (6 * Registration Resource) + Enrollment Search Chatty
    Resource Type Response Time (Seconds) – One Request
    Enrollment Search Chatty 0.25
    Enrollment Chatty (Includes a Registration Search call) 0.50
    Registration Resource 0.21

    The mainframe only serves 6 concurrent requests at a time (possibly only 4 in our test environments).

    The purpose of this analysis is to help a client developer strategize their calls to a RESTful web service. It also allows service developers to plan on future RESTFul designs when thinking about payload verbosity as it relates to performance.

    I hope this helps. It has helped me learn more about RESTFul web services by going thru this exercise.

    1 Comment

    Repost: New Student Web Service Enrollment resource enables better data for developers, advisers

    The following entry was published on the Office of the University Registrar’s blog earlier this week. It’s relevant to On the ROA readers, so it’s reposted here for convenience.

    Early this month, the Office of Information Management (OIM) released to the University community a new component to its Student Web Services (SWS): the Enrollment resource. This new data set allows campus developers—following a strict security policy—access students’ transcript data for improved tools for faculty and staff.

    The College of Education had been an early tester of the service and therefore has already updated one of its web applications to leverage the Enrollment resource. Student Tracking of Educational Progress (STEP) is an advising application that combines University-level data (including the new Enrollment resource) with College-level data into one unified tool.

    “The Student Web Service provides an ideal resource to bring in university data. It simplifies the process by allowing our web application to connect directly to the data source, eliminating an ETL process [Extract, transform, and load].  The direct connection means we can provide more up-to-date information in our local web application.”

    ~ Paul Hanisko, College of Education

    Both the Grad School and Computer Science and Engineering have committed to building applications the rely on the resource.

    Getting access to this FERPA-protected student data requires signing the SWS Access Agreement and obtaining and installing an X509 browser certificate. Any web application using the Enrollment resource must also undergo a code-security review prior to launch.

    The Enrollment resource builds upon a foundation of the existing SWS resources, such as curriculum, course, and section data. For this reason, developers who obtain access to the Enrollment resource should find its structure familiar and be able to easily use the data it provides.

    The availability of this enrollment data, long requested by members of the University developer community, is a result of the hard work by many people. The Office of the University Registrar would like to thank those who worked with them to make it possible, including everyone on the SWS and Student Database teams at OIM, the College of Education, the Graduate School, and Computer Science and Engineering.

    No Comments