9/17 – 9/24
Web Service Release
We plan on releasing either FWS and/or SWS with these new formats soon (within a couple months). PWS, IdCard, and DecisionSupport release time frames that includes XML/JSON has yet to be determined. The additional work required to complete the conversion is outlined below.
Visual Studio 2010 Conversion Issues
We were able to convert all our web services to VS 2010, .NET 4, and EntLib5.0. With that said, this took more time than any of us on the dev team anticipated. We found very unusual behavior in VS 2010′s checkin capabilities that confused developers on what files were actually being checked out for use and checked in for saving. For example on more than one occasion we found that VS2010 was automatically checking in our bin folder and all dlls contained within them without explicit knowledge from the developer. We also found checkins and outs to TFS (Team Foundation Server) to be horribly slow.
We found a .NET4 bug related to the XslCompliedStylesheet class that took some time to debug and plan on submitting a bug report to Microsoft.
We also found that our core framework when complied in Release mode did not inject (using Unity) our LogManager objects consistently. We still do not have solution to this problem however are currently looking into it. Right now all projects are being run with a Debug dll.
The last and most concerning problem related to this technology renewal upgrade is that we found the IIS7 w3wp.exe process pegging the CPU intermittently when our services were executed. We have yet to find a repeatable test case to help us debug this problem however have confirmed that all developers working on our sprint have experienced the problem on their workstations. We plan to test more on our test servers to isolate whether this problem is a Visual Studio 2010 desktop issue or a general issue that will show itself on our servers as well.
Good news is that we did not find any degradation in performance by moving to .NET 4.0 and Entlib 5.0.
While working thru the technical details of offering both our default XHTML and upcoming new JSON and XML representations; the team began to more fully understand the long-term costs of supporting all three different representation formats.
As a team we have decided today to decrement xhtml in future releases only after we have spoken to more clients to understand timing. In the short term we will offer and support all three representation formats.
Why decrement XHTML?
- All our client research (uservoice and developer interviews) has pointed to the fact that developers would rather program against XML or JSON than XHTML.
- XML and JSON are the most common formats on the internet and we would like to stick with that trend and help our developers take advantage of frameworks that can easily parse one or the other.
- The long term costs of supporting XHTML in our code seemed to outweigh the benefits for our clients.
Although many of us like the human readability of XHTML in a browser; we felt the feature did not warrant the costs of continued support. In addition, we felt that those who have already coded against XHTML would have marginal difficulty consuming our XML representations. We plan on giving the UW plenty of time to move over to the new representations.
We plan on starting our community outreach immediately so we can talk to folks on when we can effectively decrement XHTML. A method of transition will be for us to start discouraging the use of XHTML by new clients – especially in documentation by making the xhtml format content less prominent on the page and highlighting the fact it will be unsupported in the future.
ATOM pub research
While investigating what common XML representation we wanted for all our services we briefly researched the use of ATOM. Google’s Gdata and Microsoft’s OData is based on Atom pub for generic data delivery so wanted to see how they were using it. We also liked ATOM’s default rendering in all major browsers; making our web services human readable in a browser. Something we liked about our XHTML payload but like I mentioned above planned on decrementing.
After looking through the SyndicationFeed class in C# we felt that it did not offer the flexibility we needed. We wanted to get ATOM pub for “free” meaning that we can just simply serialize our model objects into ATOM using native .NET framework classes. Again the extra work and support or costs to make ATOM happen didn’t justify its use.
As Scott already mentioned in the last blog post we will use a very simple no attribute (all elements) XML representation.
Visual Studio 2010 Conversion
Today we successfully upgraded our UW.Web.Services Core framework to VS 2010 and .NET 4.0 using EntLib 5.0. This was a successful milestone. Shortly after we successfully retrofitted Person Web Service with the new core and complied the service against .NET 4.0 with no problems.
We are getting there and now we need to convert our other services and begin creating our new view models to support JSON and XML.
We began a two week sprint began on 9/14/2010 with the primary goal of adding XML & JSON representations to all resources in all the existing services to complement the XHTML offering. The addition of XML is the #1 requested feature on UserVoice and follow-up polling indicated that JSON was similarly desired and can be done without too much additional effort. Other technology renewal work will be done concurrently, including upgrading to .NET 4.0, Visual Studio 2010 IDE and Enterprise Library 5.0.
9/14: Discussed the possible tasks, prioritized each, decided which will actually be done/attempted, and assigned pairs to work on each task. Work began in earnest.
9/15: Accomplished/decided the following:
- Gathered baseline performance numbers for several of the services. We will compare the new versions against this baseline to be sure we haven’t introduced performance problems during our work.
- Technical deep dive compared the XMLSerializer with the DataContractSerializer to decide which one we will use to automatically generate the XML representation. We decided to stick with the DataContractSerializer for a number of compelling reasons, even though it means we won’t use element attributes, just elements.
- Alpha versions of the Financial Web Service Budget and Budget Search resources delivering XML and JSON are working.
- .NET 4.0 and Enterprise Library 5 upgrades are proceeding nicely.
We chose the DataContractSerializer(DCS) over the XMLSerializer(XS) for these primary reasons:
- Microsoft guidance, and beyond, recommends the DCS over the XS. It’s about 10% faster, for starters.
- DCS supports both JSON and XML serialization and the XS does only XML; using one serializer means much less possibility for introducing unforseen bugs.
- DCS is an ‘opt-in’ serializer, meaning that properties that are to be included in the serialized XML must be decorated with an attribute. XS is ‘opt-out’, serializing anything that is public. Opt-in is a better model for us since we are dealing with explicit contracts.
- XS requires a public setter on each property to be serialized. In read-only services (the majority) we don’t want this, and having it could introduce bugs.
The effort is progressing nicely and so far we are optimistic we can complete it in the two week window.
Folks who develop Web Services often get asked the question; when is it better to use one or the other. On a very high level serving data via a Web Service is for an application to consume while serving data via a report (ie. Microsoft Reporting Services) is for a person to consume.
Here is a cool page that some of us created to help people decipher between the two. Scott Bush put together the web page while others in Decision Support & Application Integration Services helped out with the content.
The ROA Tech team recently had a discussion about the form a payload might take if you were authorized to see only part of the data requested. Imagine the following hypothetical resource and its XML representation:
<employee> <id/> <name/> <phone/> <paygrade/> <sickleave-balance/> </employee>
For example, suppose that a receptionist requested such a record and that he was authorized to see name and phone number, but not the paygrade or sick leave balance. What should be returned? There are several obivous ways of handling this situation.
One approach might be thought of as a “variable representation” solution. The requester only gets back fields he is allowed to see. Our receptionist would get back this simplified payload, with no mention of the forbidden data:
<employee> <id/> <name/> <phone/> </employee>
If we chose instead always to return a fixed-form representation, several possibilities arise. We might implement access to the resource by separate public and private URIs, returning only the appropriate fields:
Alternatively, we could stick with the single URI and an isomorphic payload, but fill the disallowed fields either with nulls or some sort of error value to indicate that access is not allowed (a sort of per-field 403).
Is one of these solutions better than the others? Each solution has its own strengths, and no consensus emerged during our discussion.
The “variable representation” solution is the one found in our own person webservice– you only get back what you’re allowed to see. This keeps the security close to the datasource. But it can also be hard to code against if the client needed to be smart enough to know that some fields might be invisible to some users.
The private/public approach is straightforward in its expectations, but cumbersome in its own way. Now, the client might need to be smart enough to know which resource to request. And is the public version always a subset of the public version? A no answer has the possibility to complicate parsing of the payloads.
Isomorphic payloads are probably the easiest for the client to handle because the client can understand that payload without need of external knowledge of the resource’s structure or authorization model. If all fields in the resource are to be returned (and ease of client use is the goal), it is preferable to include an error value for disallowed fields in order to disambiguate between data that are truly empty compared to those that are forbidden.
One final question troubles me about this last solution, though: are there cases where merely acknowledging the existence of a disallowed field represents a security concern? Is it potentially worse to disclose that a datum exists but can’t be viewed than it is to omit it silently? I have not been able to come up with an example of such a field, nor have I come up with a convincing argument that
it should not be a concern. Anybody want to chime in?
Today the Office the University Registrar and UWIT have released the R25 Class Room Scheduling Web Service to be available for public use. Please see documentation on how to use the Web Service.
It was back in June of 2009 that we first blogged about the R25 web service. At that time we started talking to folks on campus to help us understand the value proposition for offering classroom scheduling data available as a web service. Due to the low cost in providing the web service, a free web service interface to the R25 database provided by Collegenet, we determined that the return on investment for this solution was compelling.
In the past, the only people who could see when a particular classroom was available at a certain date/time were classroom schedulers on campus who had the R25 Windows desktop application installed on their PC. There are more than 100 individuals that help this University with room and course scheduling, but not all of them have R25 installed on their PC. This made scheduling classrooms challenging to say the least since all questions had to go through these limited set of folks. With the availability of this data through a web service interface now all departments on campus can get access to the data programmatically.
The Room Class Schedule Finder
In addition to our Web Service release, a client web application has been released along side the web service called the Room Class Schedule Finder (note that you must have access provided by the Office the University Registrar). This tool was written by our friend Scott Bush at the Office of the University Registrar. Please see his blog posting on how the web application was written.
This allows anyone on campus, with the appropriate access, the ability to type in a classroom and see the room availability.
Why Web Services?
Web Services help developers decouple their code from database schemas so that updates to the schema do not impact client applications that rely on the data. RESTful Web Services like this one offer an easy to use programmatic interface to get the data you need in real time that is technology agnostic. Scott’s web application is developed in PHP and jQuery however you can use any popular programming language to get access to the R25 Web Service.
In the future the Office the University Registrar would like to allow for people to submit scheduling requests to classrooms via the Web Service interface and/or the Room Class Schedule Finder.
An academic calendar web service has been long requested by campus developers who needed ways to populate departmental web pages and incorporate important dates into web apps. With the availability of academic calendar data on MyUW’s calendaring tool, developers now have a method to access the information they need—without resorting to screen-scraping or other jury-rigged techniques.
MyUW’s calendaring tool provides multiple export options for creating data feeds. Supported formats are iCal, webcal, RSS, Atom, XML, and nicely-formatted plain text; each is available for a given date range. In this way, developers can obtain all the academic calendar dates previously only available on a static, table-laden HTML page maintained by the Registrar’s office. (A few academic calendar dates were available through the Student Web Service (SWS) term resource because they existed in the SDB; a quarter’s first- and census days are prime examples.) This in itself is a huge step forward for developers who can easily get at the data, and for students who can now subscribe to the calendar on their iPhones, Google Calendar, or whatever calendaring device/service they prefer.
A few examples of how campus developers might use this web service:
- Integrate campus-wide registration periods and deadlines with important dates specific to their department;
- Display an alert on a web page when an important academic calendar date is nearing;
- Trigger an e-mail, text message, blog entry, etc. based on the UW’s academic calendar; and
- Populate “what’s happening” calendars with religious holidays, which are now part of the Academic Calendar.
But this approach has drawbacks when compared to a full web service such as the SWS resources UWIT developed. So why didn’t UWIT develop a resource for this information? In a word, bandwidth; or rather, a lack thereof. Reduced budgets and higher-priority projects precluded the option of building a dedicated SWS resource. Leveraging the existing MyUW tool was the only viable way to open up the data in the short term.
Some issues with this approach include:
- No date ranges – MyUW’s calendaring tool does not support date ranges, so ranges like “Registration Period II” must be entered as two discrete entries: a beginning date and an end date.
- Start and end times – What’s more, these dates are themselves represented as start and end times. Entries that do not include time information (which describes all academic calendar entries) default to an end time of midnight, which pushes the entry’s end date to the following day. It’s an easily-avoided problem: just use the start date for each entry. But it’s something that must be looked out for when using the data in an application.
- No non-date information – Questions like “What quarter is it now?” or “What general catalog was in use Spring quarter 2009?” cannot be answered directly via this tool because it only contains discrete dates. (It is possible to use the SWS’s Term resource and some logic to answer these questions, however.) Developers are limited to retrieving a list of significant events (grades are due, late registration starts, etc.) and their dates; or, querying a specific date to see what events take place on that date.
- XML issues – An early tester of the services noted an issue with how the XML parser deals with certain accented characters, and that the VCALENDAR-based XML format does not lend itself to parsing.
Definitely not an ideal solution, but certainly one that represents a significant step forward. So retire your screen-scraping tools and instead dive directly into that feed to start playing with the data. UWIT and the Registrar’s office would love to hear how you’ll use the academic calendar in your apps, and we’ll gladly listen to ideas on ways to improve this MyUW-based academic calendar web service.
You liked the last Ignite Web Services! session so much, we decided to do it again.
What: Ignite-style presentations from UW people using Web services – 5 minutes and 20 slides each! We also decided to mix it up a bit. Each presentation will be followed by a 5 minute Q&A period.
When: Thursday, May 20 from 3-5pm.
Where: Odegaard Undergraduate Library, Room 220
1) Access Agreement Service
-Rupert Berk, UWIT Information Management
2) Let Them Eat Data: Unlocking Room Availability Information
-Scott Bush, Office of the Registrar
3) SQLShare: Query Services for Ad Hoc Research Data
-Garret Cole & Bill Howe, eScience Institute
4) A New Source for the Person Registry: Math Placement Testers
-Trevor Leffler, Office of Educational Assessment
5) YARN: Yet Another REST Narration
-Patrick Michaud, UWIT Learning & Scholarly Technologies
6) Just Can’t Decide? There’s a Web Service for That!
-Scott Stephenson, UWIT Information Management
Web services discussion group info
We meet once per quarter, from 3:00-5:00 PM in Odegaard Undergraduate Library, room 220. Our meetings will begin again in Fall Quarter.
To discuss Web services of all flavors, please email questions, suggestions, challenges, etc. to: email@example.com
Our Web service blog is located at: http://depts.washington.edu/ontheroa/
The UW Web services Registry is located at:
And finally, don’t forget a free service available to the UW developer community–Web Service Design Reviews. If you are designing, developing, or integrating a Web service and would like some help, call on us! More information is available here:
The Decision Support Web Service (DSWS) v1 is live in production. The service contains resources to view the Enterprise Data Warehouse (EDW) Data Availability Summary – the date and time data was last loaded and load frequencies — and to search for reports available in the EDW.
The web service is utilized by the newly launched Decision Support web site – “Delivering data to meet your business needs” — but it is completely public, requiring no authentication or authorization to access.
Many of our UW Web Services require a X.509 certificate for access. Up until now instructions were split among multiple wiki docs and we did not have a single page to tell developers how to request one. The UW IT Identity and Access Management team recently created a page to do just that.
Our Web Services client developer docs have all been updated to reference this new page.