What should go into the Registry?

• Something that is accessible over the web (in other words uses http)
• Contains UW data
• The content is machine parseable (json,xml,xhtml)

What should NOT go into the Registry?

• Oren’s Blog (RSS/Atom feed)
• A link to an HTML page that is NOT XHTML strict

This question is becoming more pressing as UW web services such as the Student Service, Financial Service, and ID Card Service begin to expand their format offerings. These services began with just XHTML, largely due to how conveniently it renders in a browser, but a growing number of developers have vocalized their desire for a more terse XML format that works with their libraries and tooling that read XML, but not XHTML.

In our recent discussion of this issue on the ROA Technical team, we considered several options for how to request content type:

1. URI query string
   
   GET /books/12345?format=xhtml HTTP/1.1

2. URI file name extension
   
   GET /books/12345.xhtml HTTP/1.1

3. Content negotiation using the Accept header
   
   GET /books/12345 HTTP/1.1
   Accept: application/xhtml+xml

There is plenty of debate about which is the better approach.

In their book, Richardson and Ruby prefer the explicit URI specification over the content negotiation:

Unlike humans, computer programs are very bad at dealing with representations they didn’t expect. I think an automated web client should be as explicit as possible about the representation it wants. This almost always means specifying a representation in the URL (RESTful Web Services, 94).

We lean that way, too, and prefer the URI file name extension option. It’s intuitive from all of our Web browser years; it’s simple; and it’s easy to test in a browser (you don’t have to monkey with headers).

We also think services should expose a default extension-less URI that either redirects to the .xhtml/.xml resource or returns a location header to it.


GET /books/12345 HTTP/1.1

Even though we think that the URI file name extension option is preferable, we’re not too religious about it. As long as the contract is made clear, any of these three methods are acceptable.

Let us know what you think.

The following entry was published on the Office of the University Registrar’s blog earlier this week. It’s relevant to On the ROA readers, so it’s reposted here for convenience.

The term “web services” has been used frequently on this blog when discussing new tools for the University community. Some examples include m.UW, the UW’s iPhone app; an improved course catalog search; and Kuali, the next-generation student software initiative. With the fourth version of the Student Web Services (SWS) open and available for use, it’s time that the Office of the University Registrar officially invite interested developers (and their managers!) to dive and start creating new, useful tools.

Okay, but how do I start?

That’s a good question. Here are the ingredients necessary to get a SWS project off the ground:

1. Join the community – The UW’s web services community is strong, and if you’re going to develop something using SWS you should get to know it. Read On the ROA, the UW’s web services blog; review ideas from other developers at UserVoice; and stop by at a Web Services Discussion Group meeting. Sign up on the “appdev@u” mailing list to be notified of meeting dates and locations.
2. Identify a need – Have you wished there was a site that did X? Are your students asking for Y? Want to find a better way to display Z? Once you’ve identified something to build, fix, or improve upon, you can plan a web site, iPhone application—or something else—to accomplish it using the data available to you (see number 3).
3. Research the services at your disposal – The Web Services Registry is a maintained list of UW web services, including a description and links to documentation and a contact person. You can also submit your own UW-centric web service to the registry. But don’t limit your idea to UW-specific data; maybe there’s another dataset that you could mash up it up with?
4. Build it – Web services really shine when it comes to accessing data. If you’re using public information you can simply access the service you want and start using the data returned. And it’s easy to do so regardless of your preferred language: PHP, Python, .NET, Ruby on Rails, etc. There’s a PHP class already available to simplify things even further; a .NET version is in the works.

What about an example?

Part of the reason for inviting the community to built tools with SWS is the “serendipity” factor. With pubicly-available data and a whole community of smart people, the sky’s the limit on what sort of useful tools might emerge.

An example is the recent improvements to the University’s course catalog search. A developer in the Office of the University Registrar saw the data available, knew of the issues with the current Google-based search, and built a prototype replacement in just a few days. A presentation of this tool’s development was recently given at the Office of Information Management’s Community Forum (the developer’s slides are available for download).

So go ahead: wow your students and the University as a whole with your creation. Show us the tool we didn’t know we couldn’t live without.

These ideas submitted to UserVoice have become a reality or are underway:

* Course description available in SWS

* Course url available in SWS

* Publicly available student Web services

* Person Web service

* Publicly available IdCard Web service

* Student transcript data

* Classroom scheduling Web service using R25 data

 Make your voice heard at: http://ontheroa.uservoice.com/

Amazon EC2 was chosen for hosting the project for its infinite scalability and general developer friendliness. Tony had some prior experience with the environment which certainly helped smooth many of the potential pitfalls. The control panel is really a breeze to use and the whole migration was really straightforward.

There certainly was a process as it was more involved than other hosting environments, but once done is really a set and forget operation. We mapped out the backup strategy to make sure we had a path forward just in case by using some of the built-in tools. We used a really basic Ubuntu image and then installed the environment for the application.

The application switch occurred several weeks ago and we’ve had no major problems. We look forward to continually improve the application and now we have a stable platform for some really great changes. Until then please submit some services to the registry, you’ll be glad you did.

With the release of the IdCard web service this quarter, the work involved has dropped to near zero.  We can now automatically retrieve a student’s photo from the IdCard service rather than taking our own (note that by policy, we can only retrieve student photos, not those of staff or faculty).  Our undergrad program alone admitted 105 new students this quarter, and if we assume 10 minutes per photo, it would take over 17 staff-hours to collect all photos.  I was able to modify our existing infrastructure to take advantage of the IdCard service in one afternoon. This quarter alone, the time savings is huge.

The process was simplified because access to our local store of photos all went through a single PHP web app.  Students have the right to opt-out of having their photos displayed to the public or their peers, and one of the main jobs of this application was to control access to the photos depending on the person making the request.  My strategy is first to look in our local store of photos, and if one is not found then to try the IdCard service.  If found there, the photo is retrieved, normalized to our standards, and then stored locally.  The next time the photo is requested, it is found in local “cache” and no request to the web service is needed.

The design of the IdCard service makes this task quite easy.  The photo resource lets you retrieve the photo in the size that’s most convenient to you.  I request the photos in our standard dimensions and then use the PHP interface to the GD image processing functions to add a student’s name to the image.  That image is then stored locally along with all the others, and all of our existing infrastructure now takes full advantage of the IdCard’s photo resource.  If we wish to “override” one of the ID card photos, we simply take and upload a person’s photograph as before, and the new one is served preferentially.

One side-note is that, as with most web resources concerning students, the native identifier for a person is his regid.  We had never before used regids, keeping instead system_key and uwnetid for each user.  In order to collect regids, we use the person resource of the SWS to do a system_key to regid translation, and then save the regid locally so that this translation only happens once per person.

The listing of the IdCard service in the registry is awaiting some polishing of the documentation (restricted) but expected soon.  For access, contact Tony Chang or Paul Schurr.  To use this service, you will need to obtain a test and a development client certificate for your application which you will use to authenticate to the IdCard service.   Permission to use the resource is granted by the registrar, and before getting the final OK, you and they will do a short security review of your application.  It’s all easier than it sounds, and if you get stuck you can contact roa-technical@u.washington.edu and they’ll hook you up with someone who can answer your questions.

You can download the PhpRestClient and associated SampleApp from GitHub. BTW Github is the source control repository that we are using to host our UW Web Services Registry code.

http://github.com/tonychang/PhpRestClient (click on the download button)

You can simply untar the files into the same directory and run the sample app from any webserver with a current version of PHP on it.

Since this code was contributed by developers at multiple UW departments we consider this an UW Community Source project. Yea I just made up the term right now but think it has a nice ring to it. If you want to help contribute to the project just let me know and I can get you added as a GIT contributor.

This is just a first iteration of it and its possible that someone will add additional features like supporting all the HTTP methods (vs just GET and POST) and allowing for Basic/Digest HTTP authentication.  Stay tuned for something similar to help .NET devs.

The code is not officially supported but we are happy to answer any questions you may have.

Developers:

Dan Boren – Computer Science and Engineering

Tony Chang – Office of Information Management

Paul Hanisko – College of Education

You, dear On the ROA reader, are. Let’s dive in.

Background

First, a little background: The OUR has wanted a better, data-driven interface to the course catalog (CC) for years. Though the usefulness of moving away from dozens upon dozens of static HTML files was clear, the complexity of establishing such a system—not to mention the time and cost—stymied attempts to set something up. Enter the then-unnamed UW iPhone app. Given that it needed to consume course data from somewhere and the app had support from the President, the “course search” resource was implemented in SWS v4. That Herculean effort by the Office of Information Management (OIM), UW Tech, and the OUR cannot be understated. Thank you everyone!

Getting data from the SWS

The course resource itself presents a simple search form. Interacting with it as a human is a simple (albeit plain) task: enter a search term(s) and constraints, hit search, and revel in the resulting list of matching courses. Neat. But how can a system interact with it?

Answering that question fully is beyond the scope of this article (besides, there are already good resources available on that topic). However, I can outline how the CC search works:

• I’m a PHP developer, so I chose PHP’s cURL library to consume data from the SWS. (Thanks to Paul Hanisko in the College of Education for sharing his web service code with me early on.)
• SimpleXML is an extension to PHP that allows for easy manipulation of XML data as a PHP object.
• With these tools set up, I was able to write a PHP class, Http_SwsCourse, that extended cURL’s HTTP class. I fed the class the base URL for the public course resource and wrote some helper functions (such as getting the count of the returned results) and I was all set to build the interface.
• The interface is explained in more detail below, but at its heart is just an HTML form. When submitted (via GET), the interface creates a new instance of the Http_SwsCourse class and sends the sanitized user input to the SWS. In essence, the class builds a query string just like the one visible in the browser’s address bar when using the course resource directly.
• The resulting XML payload from the webservice is then manipulated—using SimpleXML functions—by the class’s functions. Really, only two are of consequence. One retrieves all matching courses, the other “dives deeper” and retrieves course details using the specific course syntax: /course/year,qtr,curriculm,course-number

Building the search interface

Though it’s not directly related to the course resource or the SWS itself, a brief discussion of the creation of the interface for the CC search is warranted. The overall goal for the improved search was simplicity.

• Streamlined inputs – A single text-input box for the user’s query dominates the page; advanced options are hidden away. Even the advanced options are presented in a streamlined fashion. For example, I added a “search from start” checkbox rather than a second input field, and the year input is a drop-down menu listing only the current and upcoming years.
• Validation and sanitation – jQuery enforces some sophisticated field validation to match the SWS’s needs. For instance, a search term is required unless a curriculum is entered; and a year is required if a quarter is selected (and vice-versa). All variables are “sanitized” (whitespace trimmed, slashes and other undesirable characters converted, etc.) before being submitted. This is already done by the SWS but it never hurts to ensure user input won’t do any harm!
• Ajax – Mirroring the course resource’s structure, the resulting list of matching courses offer links to a given course’s details.  Rather than send the user to another page to view those details, I implemented an ajax call (again using jQuery) to retrieve and display that information. After years of static HTML pages, I thought the UW’s students and faculty deserved a little eye-candy along with their information.
• Feedback – The free version of a user-feedback service, Kampyle, provided an opportunity for users to point out any issues or offer suggestions. It’s a wise idea with any application or website, but especially so when it’s new and admittedly a beta.
• Accessibility – As noted in the beta disclaimer, the beta CC search is does not yet fully meet accessibility standards. It can—and will—when it is ready to replace the primary (Google-based) CC search. It should be relatively easy to do so, as the markup is clean and all ajax content is accessible with regular anchors and URLs.

Thoughts on the SWS course resource

Given the time constraints the OIM was under it’s amazing there aren’t dozens of problems with the web service. In my experience with it, there are none. The minor discrepancies noted by users of the CC search were related to the data or the interface I created, not the webservice. I experienced no issues with the production version and found my first foray into programming that consumes from a webservice very smooth. Again, congratulations are due to those who made this possible.

That said, there are features that I—and many, many users—would like to see in the course resource:

1. Description search – The most requested feature. Currently it only searches long- and short titles.
2. Attribute search – Searching by various course attributes was the runner-up in terms of volume of requests. For example, “show me courses that satisfy I&S or NW” or “I only want to see 3-credit courses.”
3. Sorting and paging – In building the search interface, I found that sorting options other than the default were not available. I was able to pseudo-sort results once they were in an array. This method, however, is dependent on the page-size variable in the SWS. I’d like to see more robust sort options: by number of credits; alphabetically by course or curriculum code; by course number; etc.
   As a workaround, the option to retrieve all results in one request would allow proper sorting and alternatives to paging (like an “endless scroll“). It might, however, be a heavy burden on the webservice, as some searches return many hundreds of results.
4. Instructor Course Descriptions – If these could be linked to or otherwise added to the course resource, the webservice would be sufficient to replace the static HTML version of the course catalog.

Conclusion

The beta CC search is really just the tip of the iceberg. Webservices are “game changers,” and I’m happy to have had the opportunity to build one of the early tools on campus that consumes them. The process was easy, flexible, and can only get better from here. Again, I’d like to thank the OIM, UW Tech, and the OUR for making the SWS a reality on this campus.

And I’d like to thank Tony Chang for inviting me to write a guest entry for the On the ROA blog. I enjoyed writing it; I hope you enjoyed reading it.

PS – If anyone would like to see the PHP code I wrote, let me know. I’m happy to share.