Version 10 August 2005
This document contains notes that I am developing as I prepare the "public" course portfolio for the TCSS 360 Software Development course. It reflects my beliefs that there are two types of goals associated with the portfolio that I have read about in the literature. One concerns the self-reflection and learning goals associated with the preparation of the portfolio for the portfolio author. Portfolio preparers in the literature that I have seen (esp. in The Course Portfolio by Hutchings) consistently report on the new awarenesses, personal transformations, and/or improvements in teaching from a deep analysis and reflection on teaching that results from preparing a course portfolio. I see some of this, however, as a private matter. If one explicitly recognizes this as a goal, then self-discovery can be fostered through keeping some of the portfolio private; this provides greater safety leading to fuller disclosure and hence self-understanding. Knowing that this reflection does not get shared with others, while acknowledging this important function of portfolios, can foster it. It also helps to prevent the "drowning in data" problem that portfolio readers report, and helps keep the "public" aspects of the portfolio more manageable. Finally, it provides a place for those parts of the portfolio that one realizes one needs to cut, but which one does not want to lose.
This private document also documents the design rationale for the portfolio that I develop. There are a couple of primary things that I want to capture here concerning structure and content. The first is about documenting speaker, audience, and purpose. The next is about the kinds of questions about my teaching that I think the portfolio will answer. I then want to discuss what I will include in the portfolio to answer these questions. I then want to discuss in more detail the sorts of things that I will include in the "enactment" section, the evidence of student learning (outcomes) that I will include, and the sorts of analysis I plan to do. And then I'd like a section on "lessons learned" from portfolio preparation.
One is that I should be clear about "speaker, audience, and purpose", and make some of this explicit to the reader. I should identify who I believe this audience to be, as it will provide a basis for how much to make explicit and how much to keep implicit. And audience is directly related to purpose, so these are likely inseparable. The main audience and purpose is faculty in my department who will be teaching this course. But another audience and purpose is to use this as an example portfolio (or sections of it, anyway) for my pilot project (i.e. CS faculty in the region). And I think that I would like to use sections of this to share with faculty in my program as a means to document and reflect on teaching. Again, this primarily helps with the implicit/explicit distinction, and also what to include. If I were writing this portfolio for promotion (which I might eventually do), then I would probably want to make a few other things explicit.
In working with Qi on the construction of her portfolio, she has asked several questions that have echoed my own concerns. One thing that she pointed out was that initially, one needs introductory material and navigational aids to help individuals with different goals to quickly access those parts of the portfolio most relevant to their concerns. This material needs to be placed at the beginning of the portfolio, and should rely on internal hyperlinks. Having an initial table of contents will also help (which I already had); but including a brief sentence of explanation about each section might be useful in the introductory section. In this regard, it is much like the Introduction section to a scholarly paper.
The other thing that I need to do is to clarify to the reader the meaning of hyperlinks, i.e. whether they are an essential part of the portfolio or simply optional, or perhaps a relatively unimportant scholarly reference. This issue arose when Qi asked whether we should hyperlink material or include it inline. It most importantly impacts portfolio construction in my "Design and Rationale" section concerning how much of the syllabus to include, and also in the "Analysis of Outcomes" section concerning how much student work to include.
So my philosophy is this: that I want the portfolio to be a single file, a coherent and linear document that stands on its own. This means that I have the obligation to choose for my reader those parts of other documents that are necessary to include inline. So, for example, to simply make reference to my syllabus, which is a long document, does not by itself instruct the reader as to how important it is for the discussion in my portfolio for the reader to know my syllabus. She will not know whether she needs to read all of the syllabus or some of it, and if just some what parts, or perhaps none at all. Without some guidelines for the reader, she will lack a feeling of closure with the text, a sense of completeness and coherence. This is, I believe, a general difficulty with hyperdocuments. And so, I will need to provide some guides to the reader about this, and will do so up front. I also think that it will be helpful to include hyperlinks to each of the documents that I summarize all together as a set of Appendices.
The public part of the portfolio will address, at least to some extent, the five elements that Shulman states are entailed in a public account of the full act of teaching: "vision, design, interactions, outcomes, analysis." (p5, "Course Anatomy" from Hutchings's The Course Portfolio) I take vision to mean the overarching raison d'etre of the course. I take design to be the choice and sequence of activities, readings, assessments, etc. that I construct in order to achieve the vision. Also, I take design to also include the design rationale, what Kees Dorst (in Understanding Design, p29) calls "the story behind [the design]": "This story consists of all the choices you have made during your design project and the arguments that you used in making them. It is the justification of the design, which explains why the design is constructed in just the way it is." I see this as particularly important for discipline-specific course portfolios, i.e., where the audience and purpose are embedded within a discipline, so that the design rationale assumes a shared context between the portfolio writer and reader. Taking this design perspective in general about courses is a useful perspective. "By looking at things with a designers' [sic] eye, you get an idea of the reasoning and design process behind them. This reveals not only 'how things work', but also the 'why' behind them." (p.159, op cit).
I take the rationale as being the most important part of the design. Part of this is because I have made serious reference to various literatures to develop my concept of this course and to guide my course decisions. It is thus the important framing for the course; without it, the outcomes cannot be interpreted, at least not in the sense in which I intend.
I take interactions to mean the enactment of the course; Shulman comments that this section is to the course design as a research study is to its being carried out. In particular, he focuses on student/professor interactions, but it can (it seems to me) also include student/student interactions, and even more abstract sorts of interactions (student/guest speaker, or student/resources). One of the things that I did not initially realize but which Sally pointed out, is that teaching methods is an important part of the enactment of the course. Do I use labs, demo's, lecture, small groups? At which points, for which purposes?
On examining both Qi's and several other portfolios, I think that the best way to illustrate method is to 1) list the range of methods that I use, and 2) illustrate the interplay between these, and how each meets different goals for a particular "unit". I think that it is best to choose the "Design" part of the course, since I view this as the heart of the course, both in terms of its chronological ordering (it is smack in the middle), the design milestone is the middle of three milestones, and design ideas are the most important of the course in my view. One of the things that this highlights is whether I take it as an explicit objective that student designs exhibit characteristics of "good" design. Or perhaps that students are developing a design esthetic, or that they are coming to understand the relationship between design decisions and coding outcomes.
I take outcomes to be equivalent to the results of a research study, the kinds of student learning that one has evidence for. And I take analysis to be the interpretations of the outcomes with respect to vision and design. The advantage of focusing on the design unit, as I just mentioned, is that I can also focus on this unit for analysis of student outcomes. What I need to do is to consider each of my goals in sequence, and match the data/evidence to the goal. I will merge the "outcomes" and "analysis" sections together (like the "results and discussion" section of a paper). Best to try to interpret what the outcomes mean at the time I first discuss them.
I will also adopt three of the four "formats and themes" (or what Robin Adams would call lenses) on portfolio preparation that Shulman describes (p8): "the course as anatomical structure; the natural history of a course; [and] the ecology of courses". The anatomical structure also includes the physiology: what are the parts, how do they function, what is their structural relationship to one another, and how do they function together to carry out the course goals. This is very much related to issues of design and rationale mentioned above. The natural history provides a historical context for the course: how it has evolved over time to both adapt to changing situations and also to display the learning of the professor. And the ecological perspective provides a context for the course; how does it fit into the larger institutional goals, that can range all the way from the department, to the college and campus, to the discipline as a whole (e.g. disciplinary curriculum guidelines), to broader social goals (e.g. Dewey's view of the citizen in a democratic society). The Shulman "lens" that I didn't adopt, course portfolio as inquiry, is one that I have had the hardest time with. It is only on seeing Jennifer Robinson's, where she states her three challenges that I had a sense of how a portfolio can address a particular question about one's course. Inquiry portfolios are my favorite portfolios of those I've read, yet they don't suit my "disciplinary commons" purposes. As my portfolio grows, I wonder about whether I've got it wrong, i.e., perhaps the "inquiry" approach can also make sense in that context. Hard to tell from this vantage point.
To summarize the central "questions", then, that I perceive my "Commons" portfolio to answer are: what are the goals of my course, what is the content and structure, what are the teaching methods, how are these goals placed within a larger curricular context, why do I use the particular methods that I do, and how and why has my course evolved over time?
As mentioned above, I have always considered the rationale to be the most important part of my portfolio, a rationale not yet written at the time of writing this section but ever present. This is because, especially in the case of a course about a subject matter that I know, I want to know "why" whenever I look at someone's course. Why did you choose that topic, why did you put it there, why did you teach it that way? It is what is so glaringly missing when I evaluate the teaching of my colleagues at the university.
And although I often have questions about the particulars, I am most concerned about a teacher's underlying models and assumptions -- about cognition, about the discipline, about human learning -- that both shapes all subsequent choices and is embedded in the course as it unfolds. Interestingly, this is what Murray argues is the one essential and common ingredient to any teaching (as opposed to course) portfolio. It is often called a teaching philosophy or teaching statement. "Whatever it is called, faculty must examine the implicit and explicit assumptions they hold about teaching in order to reflect on what they know about the teaching and learning process" (p25,28).
So I need to address what are the things for which I need to provide a rationale, for surely I cannot justify every decision. First, many of the small-scale decisions can be discussed in the "Historical Trajectory" section, since these involved changes over time. Even such things as "why the waterfall" would go here. Second, I want to discuss my reasons for a focus on practice, esp. social practice. I think that cribbing a bit of the essay that I prepared for the teaching award might provide a context for the main models of teaching and learning that are floating around in my head in this regard. These mostly concern constructivism, active learning, collaborative learning, and communities of practice. Software design as iterated social activity. Third, I want to discuss reasons for how I structure groupwork.
One choice that I have struggled with concerns whether or not to include student work. Two things have guided me in deciding not to include student work, but rather to provide my interpretations of student work. The first is the fact that the main artifacts that students produce in this course are large documents, the final milestone generally consisting of around 50 - 100 files handed in on a CD. Thus, student artifacts are quite large.
The second consideration has been the use of the "research" analogy in conceptualizing this portfolio. When I publish research, I never include my raw data, except perhaps for illustrative snippets. Rather, I make explicit to my reader my method for collecting and analyzing the data, along with my interpretations of what the data means. In this way the reader will understand the inference chain between data and conclusions without having to construct this entire chain. But this isn't an individual idiosyncracy; published research almost never includes raw data, except for small illustrative pieces. By explicitness about method, I follow evolved social conventions for reporting, at the same time as I establish an ethos of honesty, so that the reader will (hopefully) believe those parts of the inference chain (i.e. the groundings in actual data) that are never revealed. Thus, I provide my interpretations without providing the raw data on which these are based.
Specific sections will include the following, keeping in mind the audience - CS faculty in my program who will teach the course - and goals - to disseminate the design, rationale, and evidence that students meet the learning objectives:
Note that the reason I include the historical trajectory after the course design is that it rests on having an understanding of the specifics of the course. One of the other things that I like about including the "historical trajectory" section, is that it points out how various aspects of the course -- particularly my refining of the group practices, grew from my responses to situations that I was observing and experiencing. So, although I can provide a philosophical rationale, this was never separate from, but informed by and informing, the lived situations with groups that I was encountering as teacher of the course.
As I complete this portfolio, I see no reason to include a "lessons learned" in the portfolio itself, though I think that such a section might be useful in this private portfolio. Trying to distill or summarize the portfolio will detract from the portfolio as a larger work, with many audiences and possible purposes.
Probably my main lesson about the course itself concerned how little evidence that I had that students transfer the practices around which the course centers outside the course setting. This is an implicit goal of all of my classes, and yet I would have to do something different than only looking at student work while they are in my course (the normal state of affairs) to find evidence (or not) of such transfer.
Another lesson was recognizing in the "goal setting" part of the portfolio, that as much as I would like students to build "good" software designs, that that is a secondary goal. Most important to me is that they engage in the practices of software designers, and begin to cultivate a design orientation, a design estethic, a "feel" for design.
Most of my lessons were not about the course, though, largely because I regularly evaluate this course, do post-mortems, reflect on why I teach the course the way I do and how to improve it. Further, since I have taught this course so many times, I have had ample opportunities to try a number of different topics, structures, and methods.
There were a few major lessons at the meta-level, however, about the use of course portfolios, some of which have found (and will find) their way into other forums. First is the importance of the Rationale, especially within a disciplinary "community of validation." The design view is crucial, and providing the "ex-plan-ation" (to use Eli Blevis's term) is one of the most important things that the portfolios can contain. I went into the project believing that this was the most important section, and continue to believe it is -- even more important than the "Outcomes" section. Note that it is not one of the portfolio parts that Dan Bernstein uses in his Peer Review of Teaching project, probably the most well-known of the large-scale portfolio projects to date (e.g. as described in the July/August issue of Academe). I believe that the rationale is most important because I have come to believe that the portfolio's primary effect will be to serve as a vehicle for "reflective practice" by the portfolio author (a point made clear to me by Susan Mosburg, research scientist at CELT at the University of Washington). Perhaps I feel this way, as well, because I had, for so long, wanted to make explicit many of the assumptions and decisions, my internal "folk pedagogy" (to use Bruner's term) that undergirded the entire course design. And so this was by far the most satisfying part of the portfolio to write, and I had a great sense of relief on externalizing all of those thoughts and reasons that had built up over many years.
I forget at what point I realized that there is actually a potential third document, what I call the "protected" portfolio. And this would include those parts of my portfolio that I do not want to make fully public, but would like to share with colleagues. The tripartite structure: a public, protected, and private portfolio, all interrelated, each for different audiences, reminds me of two things. The first are the three levels of the Scholarship of Teaching and Learning that Keith Trigwell discussed in his keynote speech at the 1st IS-SOTL conference in Bloomington in October, 2005. There is "level 1", which is scholarship that helps a teacher improve, mapping to the private portfolio. "Level 2" is scholarship directed toward a local setting, for one's colleagues in the discipline within a specific setting. This relates (though imperfectly) to the protected portfolio. And "level 3" scholarship is what is generally called "research": general and generalizable beyond specific contexts, which maps to the public portfolio.
I am also reminded of Kvale's discussion of three communities who validate qualitative research (from his book InterViews). There is direct validation during the interview itself, between the participants in the interview, validation by the general public, and validation by the community of research peers.
Another lesson concerned my struggle concerning validation, and my tentative resolution of this. This issue is highlighted in this (somewhat long) message that I sent to Sally Fincher, based on similar questions I had asked of Jennifer Robinson.
One thing, though, that I want to further discuss with you is the nature of the evidence of student learning, and what sort of analysis is appropriate. It echoes some of the "pragmatic" approach that you and Marian have advocated. Here is an instance of it: in my own portfolio, when I am looking for evidence that students have actually benefitted from the various "monitoring" mechanisms to ensure commitment to carry out their share of the groupwork, do I run a t test between pre- and post-course surveys on their attitudes toward commitment, or do I simply show that there is improvement in these scores (i.e. but not worry about significance). And what if I haven't captured pre/post attitudes at all? I think that this example concisely captures the ambiguity of issues of audience and the reasons for even looking at evidence of student learning. For if there isn't a clear peer community for whom I am writing the portfolio, with established evidential norms for establishing claims, then what leads me to choose my evidence on the spectrum between "they seemed to improve" and "on a 2-tailed, paired-samples t test between pre- and post-course attitudes toward group commitment, ..."? Clearly I can't subject all of my teaching choices to the highest level of scrutiny, of falsification; I need to be pragmatic in my choices. I do not have time to pour through the hundreds of pages of documents that I have collected to do a deep semantical analysis of their statements pertaining to groupwork and commitment. But if I am subjecting none of my choices to anything but my own hunches, speculations, or intutitions, then 1) why am I going through the point of documenting them, since I _already_ make hunches and speculations, call upon my intutions, which is almost always good enough for me, and 2) to what extent can we call any of this scholarly? From what I can tell, the "point", or the value, comes not in a scholarly review, but in both the individual metacognition that comes from making the implicit explicit and in the _collegial_ (but not "peer" in the sense of blind, refereed journals) review that happens in discussions with colleagues and peers. Done in a group setting, such as the Commons project, it has additional benefits for community building, knowledge dissemination about learning in our respective settings, modeling of a more deliberate and reasoned approach to teaching, and sharing tips, tricks, and best practices. All of these are good things. But few of them address the "scholarly" claims made by Dan [Bernstein] (and others) for course portfolios, and few of them address "level of evidence" for student learning that we should seek.
The resolution of this tension might come about from being clear about the "communities of validation" (to borrow Kvale's use), and to provide validation appropriate to the intended community of validation, depending on whether the claims are in the public, private, or protected portfolio.
The largest surprise from this project, and it came about only in interaction with a colleague who was similarly engaged in creating a portfolio, was in the nature and quality of the discussions that we had together in peer critiqueing one another's work. Our discussions were "teacherly", and embedded in the discipline. Essentially, the portfolio dropped away, and we found ourselves discussing important questions about our assumptions about student learning, in general as well as within our discipline. It was this experience, this use of the portfolio as a vehicle for discipline-specific community discussion, that gives me confidence that a larger-scale project such as Sally and my "Disciplinary Commons" project will have important outcomes for the participants.
I also found it particularly valuable to think through choices about selection, perhaps surprising to the reader who will find my portfolio voluminous. I was guided throughout by purpose and audience. But nonetheless, there are no set formulae for how to illustrate teaching method, for how many of the millions of choices in course design to leave out and which few to include, and for how much student work, of what sorts, analyzed in what way to include.
What provided coherence and selection for me was to focus thematically on the 3-week "unit" within this course on software design. I used that as a lens with which to explore method, detailing those three weeks but not others, and examining student outcomes for that unit. Further, it provided a sufficient mechanism at the end for returning to my original course objectives and asking "were these met"?
Qi on the other hand, used a "critical classes" focus, examining method and outcomes with respect to two critical class sessions. I found this vertical slice highly effective for her portfolio, and her illustration with actual student work and her commentary revealing. Perhaps the different choices that Qi and I made about focusing mechanism reflect differences in course content: Qi's course is about the introductory material, about small-scale tactical choices in programming, while my course examines larger-scale, strategical design choices. Regardless of mechanism, it is clear that the portfolio author must make these choices, and having some coherent mechanism for doing so (thematic focus, critical lesson, or some other way) is important. A contribution would be to enumerate useful ways for doing so, based on an examination of the UNL and IU course portfolio repositories.
Another happy aspect of having at least one other partner in the portfolio enterprise is the inspiration that one can draw from seeing another person sincerely reflect on their own teaching. I found myself inspired to complete my portfolio as a result.
This section documents the time I have taken to complete this portfolio, the tasks that I have done, and the dates in which I have done them.