Adding XML & JSON Reps: Sprint Update II

9/16

XHTML deprecation

While working thru the technical details of offering both our default XHTML and upcoming new JSON and XML representations; the team began to more fully understand the long-term costs of supporting all three different representation formats.
As a team we have decided today to decrement xhtml in future releases only after we have spoken to more clients to understand timing. In the short term we will offer and support all three representation formats.

Why decrement XHTML?

  • All our client research (uservoice and developer interviews) has pointed to the fact that developers would rather program against XML or JSON than XHTML.
  • XML and JSON are the most common formats on the internet and we would like to stick with that trend and help our developers take advantage of frameworks that can easily parse one or the other.
  • The long term costs of supporting XHTML in our code seemed to outweigh the benefits for our clients.

Although many of us like the human readability of XHTML in a browser; we felt the feature did not warrant the costs of continued support. In addition, we felt that those who have already coded against XHTML would have marginal difficulty consuming our XML representations. We plan on giving the UW plenty of time to move over to the new representations.

We plan on starting our community outreach immediately so we can talk to folks on when we can effectively decrement XHTML. A method of transition will be for us to start discouraging the use of XHTML by new clients – especially in documentation by making the xhtml format content less prominent on the page and highlighting the fact it will be unsupported in the future.

ATOM pub research

While investigating what common XML representation we wanted for all our services we briefly researched the use of ATOM. Google’s Gdata and Microsoft’s OData is based on Atom pub for generic data delivery so wanted to see how they were using it. We also liked ATOM’s default rendering in all major browsers; making our web services human readable in a browser. Something we liked about our XHTML payload but like I mentioned above planned on decrementing.

After looking through the SyndicationFeed class in C# we felt that it did not offer the flexibility we needed. We wanted to get ATOM pub for “free” meaning that we can just simply serialize our model objects into ATOM using native .NET framework classes. Again the extra work and support or costs to make ATOM happen didn’t justify its use.

As Scott already mentioned in the last blog post we will use a very simple no attribute (all elements) XML representation.

Visual Studio 2010 Conversion

 Today we successfully upgraded our UW.Web.Services Core framework to VS 2010 and .NET 4.0 using EntLib 5.0. This was a successful milestone. Shortly after we successfully retrofitted Person Web Service with the new  core and complied the service against .NET 4.0 with no problems.

We are getting there and now we need to convert our other services and begin creating our new view models to support JSON and XML.

2 Comments

Adding XML & JSON Reps: Sprint Update

We began a two week sprint began on 9/14/2010 with the primary goal of adding XML & JSON representations to all resources in all the existing services to complement the XHTML offering.  The addition of XML is the #1 requested feature on UserVoice  and follow-up polling indicated that JSON was similarly desired and can be done without too much additional effort. Other technology renewal work will be done concurrently, including upgrading to .NET 4.0, Visual Studio 2010 IDE and Enterprise Library 5.0.

9/14: Discussed the possible tasks, prioritized each, decided which will actually be done/attempted, and assigned pairs to work on each task.  Work began in earnest.

9/15: Accomplished/decided the following:

  • Gathered baseline performance numbers for several of the services. We will compare the new versions against this baseline to be sure we haven’t introduced performance problems during our work.
  • Technical deep dive compared the XMLSerializer with the DataContractSerializer to decide which one we will use to automatically generate the XML representation.  We decided to stick with the DataContractSerializer for a number of compelling reasons, even though it means we won’t use element attributes, just elements.
  • Alpha versions of the Financial Web Service Budget and Budget Search resources delivering XML and JSON are working.
  • .NET 4.0 and Enterprise Library 5 upgrades are proceeding nicely.

We chose the DataContractSerializer(DCS) over the XMLSerializer(XS) for these primary reasons:

  • Microsoft guidance, and beyond, recommends the DCS over the XS. It’s about 10% faster, for starters.
  • DCS supports both JSON and XML serialization and the XS does only XML; using one serializer means much less possibility for introducing unforseen bugs.
  • DCS is an ‘opt-in’ serializer, meaning that properties that are to be included in the serialized XML must be decorated with an attribute. XS is ‘opt-out’, serializing anything that is public. Opt-in is a better model for us since we are dealing with explicit contracts.
  • XS requires a public setter on each property to be serialized.  In read-only services (the majority) we don’t want this, and having it could introduce bugs.

The effort is progressing nicely and so far we are optimistic we can complete it in the two week window.

No Comments

When to use a Web Service or get a Report?

Folks who develop Web Services often get asked the question; when is it better to use one or the other. On a very high level serving data via a Web Service is for an application to consume while serving data via a report (ie. Microsoft Reporting Services) is for a person to consume.

Here is a cool page that some of us created to help people decipher between the two. Scott Bush put together the web page while others in Decision Support & Application Integration Services helped out with the content.

No Comments

Differential authorization of data within a given resource

The ROA Tech team recently had a discussion about the form a payload might take if you were authorized to see only part of the data requested.  Imagine the following hypothetical resource and its XML representation:

/employee/{id} returns:


<employee>
      <id/>
      <name/>
      <phone/>
      <paygrade/>
      <sickleave-balance/>
</employee>

For example, suppose that a receptionist requested such a record and that he was authorized to see name and phone number, but not the paygrade or sick leave balance.  What should be returned? There are several obivous ways of handling this situation.

One approach might be thought of as a “variable representation” solution.  The requester only gets back fields he is allowed to see.  Our receptionist would get back this simplified payload, with no mention of the forbidden data:

<employee>
      <id/>
      <name/>
      <phone/>
</employee>

If we chose instead always to return a fixed-form representation, several possibilities arise.  We might implement access to the resource by separate public and private URIs, returning only the appropriate fields:

/employee/public/{id}
/employee/private/{id}

Alternatively, we could stick with the single URI and an isomorphic payload, but fill the disallowed fields either with nulls or some sort of error value to indicate  that access is not allowed (a sort of per-field 403).

Is one of these solutions better than the others?  Each solution has its own strengths, and no consensus emerged during our discussion.

The “variable representation” solution is the one found in our own person webservice– you only get back what you’re allowed to see.  This keeps the security close to the datasource.  But it can also be hard to code against if the client needed to be smart enough to know that some fields might be invisible to some users.

The private/public approach is straightforward in its expectations, but cumbersome in its own way.  Now, the client might need to be smart enough to know which resource to request.   And is the public version always a subset of the public version?  A no answer has the possibility to complicate parsing of the payloads.

Isomorphic payloads are probably the easiest for the client to handle because the client can understand that payload without need of external knowledge of the resource’s structure or authorization model.  If all fields in the resource are to be returned (and ease of client use is the goal), it is preferable to include an error value for disallowed fields in order to disambiguate between data that are truly empty compared to those that are forbidden.

One final question troubles me about this last solution, though: are there cases where merely acknowledging the existence of a disallowed field represents a security concern?  Is it potentially worse to disclose that a datum exists but can’t be viewed than it is to omit it silently?  I have not been able to come up with an example of such a field, nor have I come up with a convincing argument that
it should not be a concern. Anybody want to chime in?

3 Comments

R25 Web Service now available

Today the Office the University Registrar and UWIT have released the R25 Class Room Scheduling Web Service to be available for public use. Please see documentation on how to use the Web Service.

Background

It was back in June of 2009 that we first blogged about the R25 web service. At that time we started talking to folks on campus to help us understand the value proposition for offering classroom scheduling data available as a web service. Due to the low cost in providing the web service, a free web service interface to the R25 database provided by Collegenet, we determined that the return on investment for this solution was compelling.

In the past, the only people who could see when a particular classroom was available at a certain date/time were classroom schedulers on campus who had the R25 Windows desktop application installed on their PC. There are more than 100 individuals that help this University with room and course scheduling, but not all of them have R25 installed on their PC. This made scheduling classrooms challenging to say the least since all questions had to go through these limited set of folks. With the availability of this data through a web service interface now all departments on campus can get access to the data programmatically.

The Room Class Schedule Finder

In addition to our Web Service release, a client web application has been released along side the web service called the Room Class Schedule Finder (note that you must have access provided by the Office the University Registrar). This tool was written by our friend Scott Bush at the Office of the University Registrar. Please see his blog posting on how the web application was written.

room scheduler

This allows anyone on campus, with the appropriate access, the ability to type in a classroom and see the room availability.

Why Web Services?

Web Services help developers decouple their code from database schemas so that  updates to the schema do not impact client applications that rely on the data. RESTful Web Services like this one offer an easy to use programmatic interface to get the data you need in real time that is technology agnostic. Scott’s web application is developed in PHP and jQuery however you can use any popular programming language to get access to the R25 Web Service.

Future plans
In the future the Office the University Registrar would like to allow for people to submit scheduling requests to classrooms via the Web Service interface and/or the Room Class Schedule Finder.

2 Comments

An initial Academic Calendar web service now available via MyUW

An academic calendar web service has been long requested by campus developers who needed ways to populate departmental web pages and incorporate important dates into web apps. With the availability of academic calendar data on MyUW’s calendaring tool, developers now have a method to access the information they need—without resorting to screen-scraping or other jury-rigged techniques.

MyUW’s calendaring tool provides multiple export options for creating data feeds. Supported formats are iCal, webcal, RSS, Atom, XML, and nicely-formatted plain text; each is available for a given date range. In this way, developers can obtain all the academic calendar dates previously only available on a static, table-laden HTML page maintained by the Registrar’s office. (A few academic calendar dates were available through the Student Web Service (SWS) term resource because they existed in the SDB; a quarter’s first- and census days are prime examples.) This in itself is a huge step forward for developers who can easily get at the data, and for students who can now subscribe to the calendar on their iPhones, Google Calendar, or whatever calendaring device/service they prefer.

A few examples of how campus developers might use this web service:

  • Integrate campus-wide registration periods and deadlines with important dates specific to their department;
  • Display an alert on a web page when an important academic calendar date is nearing;
  • Trigger an e-mail, text message, blog entry, etc. based on the UW’s academic calendar; and
  • Populate “what’s happening” calendars with religious holidays, which are now part of the Academic Calendar.

But  this approach has drawbacks when compared to a full web service such as the SWS resources UWIT developed. So why didn’t UWIT develop a resource for this information? In a word, bandwidth; or rather, a lack thereof. Reduced budgets and higher-priority projects precluded the option of building  a dedicated SWS resource. Leveraging the existing MyUW tool was the only viable way to open up the data in the short term.

Some issues with this approach include:

  • No date ranges – MyUW’s calendaring tool does not support date ranges, so ranges like “Registration Period II” must be entered as two discrete entries: a beginning date and an end date.
  • Start and end times – What’s more, these dates are themselves represented as start and end times. Entries that do not include time information (which describes all academic calendar entries) default to an end time of midnight, which pushes the entry’s end date to the following day. It’s an easily-avoided problem: just use the start date for each entry. But it’s something that must be looked out for when using the data in an application.
  • No non-date information – Questions like “What quarter is it now?” or “What general catalog was in use Spring quarter 2009?” cannot be answered directly via this tool because it only contains discrete dates. (It is possible to use the SWS’s Term resource and some logic to answer these questions, however.) Developers are limited to retrieving a list of significant events (grades are due, late registration starts, etc.) and their dates; or, querying a specific date to see what events take place on that date.
  • XML issues – An early tester of the services noted an issue with how the XML parser deals with certain accented characters, and that the VCALENDAR-based XML format does not lend itself to parsing.

Definitely not an ideal solution, but certainly one that represents a significant step forward. So retire your screen-scraping tools and instead dive directly into that feed to start playing with the data. UWIT and the Registrar’s office would love to hear how you’ll use the academic calendar in your apps, and we’ll gladly listen to ideas on ways to improve this MyUW-based academic calendar web service.

, ,

3 Comments

Web Service Discussion Group Ignites This Thursday!

You liked the last Ignite Web Services! session so much, we decided to do it again.

What: Ignite-style presentations from UW people using Web services – 5 minutes and 20 slides each! We also decided to mix it up a bit. Each presentation will be followed by a 5 minute Q&A period.

When: Thursday, May 20 from 3-5pm.

Where: Odegaard Undergraduate Library, Room 220

Who…

1) Access Agreement Service
-Rupert Berk, UWIT Information Management

2) Let Them Eat Data: Unlocking Room Availability Information
-Scott Bush, Office of the Registrar

3) SQLShare: Query Services for Ad Hoc Research Data
-Garret Cole & Bill Howe, eScience Institute

4) A New Source for the Person Registry: Math Placement Testers
-Trevor Leffler, Office of Educational Assessment

5) YARN: Yet Another REST Narration
-Patrick Michaud, UWIT Learning & Scholarly Technologies

6) Just Can’t Decide? There’s a Web Service for That!
-Scott Stephenson, UWIT Information Management

———————————–
Web services discussion group info
———————————–

We meet once per quarter, from 3:00-5:00 PM in Odegaard Undergraduate Library, room 220. Our meetings will begin again in Fall Quarter.

To discuss Web services of all flavors, please email questions, suggestions, challenges, etc. to: appdev@u.washington.edu

Our Web service blog is located at: http://depts.washington.edu/ontheroa/

The UW Web services Registry is located at:

http://webservices.washington.edu/

And finally, don’t forget a free service available to the UW developer community–Web Service Design Reviews. If you are designing, developing, or integrating a Web service and would like some help, call on us! More information is available here:

https://sig.washington.edu/itsigs/Web_Services_Design_Review

1 Comment

Decision Support Web Service v1.0 Launched

The Decision Support Web Service (DSWS) v1 is live in production. The service contains resources to view the Enterprise Data Warehouse (EDW) Data Availability Summary – the date and time data was last loaded and load frequencies — and to search for reports available in the EDW.

The web service is utilized by the newly launched Decision Support web site – “Delivering data to meet your business needs” — but it is completely public, requiring no authentication or authorization to access.

No Comments

Request and obtain a UW CA X.509 certificate

Many of our UW Web Services require a X.509 certificate for access. Up until now instructions were split among multiple wiki docs and we did not have a single page to tell developers how to request one. The UW IT Identity and Access Management team recently created a page to do just that.

Request and obtain a UW CA X.509 certificate

Our Web Services client developer docs have all been updated to reference this new page.

No Comments

Person Web Service v1.0 Launched

Person Web Service (PWS) v1 is live in production. The joint collaboration between AIS and IAM in UWIT has produced a tool to offer an authoritative set of web-based resources for the UW person data.

The current API provides read access to the PDS (Person Directory Service) data, and exposes person and entity resources including search capabilities. Entities are data objects that hold UW NetIDs (with minor exceptions, like people who haven’t yet created a UW NetID). The vast majority of entities are persons. A person resource reflects a single real-life individual with one or more UW affiliations. Person resources carry data attributes about the person’s affiliation(s) and contact information. Some entities are not persons. Non-person entities include things like Departmental, Shared or Temporary UW NetID accounts, and carry very few attributes.

An entity resource, and therefore a person resource (since persons are a subset of all entities), can be fetched by its primary key, UW RegID, or by UW NetID. Entity searches support queries by name, while Person Searches support queries by name or by additional affiliation-specific identifiers.

More detailed documentation about the service, resources and sample .NET client code can be found on our support wiki here. It also explains how to get access to PWS. For further information please feel free to contact iam-support@uw.edu .

No Comments