Day 1We spent the morning doing a breakfast meet and great, where each person from a particular institution introduced the other from their institution. Prepping for this gave me a good chance to learn quite a bit more about JoAnn, including our shared love of critically panned/unpopular movies: in particular Joe Versus the Volcano and The Fifth Element. I also found that U of I is very well represented at this event, not just by JoAnn and myself, but also by U of I GSLIS grads/past employees (and current advanced degree students ;) Elizabeth Edwards and Jennifer Ward. You can see a full list of who was there at the bottom of this post (they where all super cool and agreed to let me mention them by name even without seeing what I was saying about them :) .
Following breakfast/introductions, Nancy Foster lead this workshop (she's is one of the few, and possibly only, people doing this type of research as it relates to libraries). She very generously consented to let me post my notes about what we did there. Nancy gave us some brief tips on how to do ethnographic observations, and then set us out into various locations on the George Washington (G Dub, apparently) campus, in pairs (separated from our institutional buddies). We mapped our spaces (quick sketched maps) and jotted down brief descriptions. Then we settled in to do about 45 minutes of general observations of the people who passed through and used those spaces, noting what they did, how they used the space, etc. I've got two pages of notes I typed up, which I won't bore you with (
but if you really want to see them, drop me a line and I'll send them along. Oops, I spoke too soon. Agreements between CLIR and host institutions don't allow us to share observational notes, even if taken in public areas, so you can't be bored by my notes even if you wanted to. ;P Thanks to Nancy for correcting me on this.).
We returned from our field trip, and reported what we had observed to the group. Nancy listened to our reports, and then analyzed them for common threads. She pointed out generalities that we had mentioned that have proven to be constants in student/faculty observations (and ethnographic observation in general). She helped us understand the process a bit and used our initial experiences as a way to prep us for our real interviews on day 2, when we'll actually be interviewing and observing a faculty member.
Nancy then gave us an overview of how she has successfully run faculty interview/observations in the past. She showed us some slides with examples from previous studies she had run (some of which you can find referenced in her publications, where IRB restrictions allow). We then headed out to the office of Professor S (I forgot to get permission to reference Prof S, so mystery abounds ;) , who turned out to be an incredibly relaxed, friendly interview candidate, even though we were too many people (13) in too small an office (8' by 7'). Nancy ran the interview, while we all observed and took turns practicing with the camcorders we'd be using during our own interview sessions. I was genuinely surprised to see how closely his own research methods mirrored my own- he would print out articles (here we differed, I am a mix of saving to my computer and printing, depending on where I think I'll be working/reading, and he said he never saved articles to his computer, just printed) but from here on we turned out to be eerily similar. He would read through articles quickly, then if the where meritorious, read through them again thoroughly/slowly, and either make notes in the margins, or on separate paper (here, slightly different, I highlight/mark printouts a little, but mostly copy/paste and directly annotate into a text document while reading). What was very interesting was the similarity in how we made notes. We both start by making an "annotated bibliography plus," where we have a mix of direct quotations of import (marked in both cases by double quotes - I mark page references, I didn't get to ask if he did as well), paraphrases (marked by [ ] in his case, and simple unadorned text block in mine) and our own commentary (in italics in his method, in [ ] in mine). Then, once we've completed our initial pass of all the literature, we both begin arranging quotes, paraphrases, and commentary thematically in a separate document, in a way that addresses our particular research topic. This then serves as the point from which we begin organizing materials into our own narrative (or in my case outline-like-thingy, again, we didn't have enough time for me to ask if he went straight to writing, or also used an outline). And here I thought my approach to lit review and analysis was just that- mine. Silly me- there nothing new under the sun.
We finished the official work part of our day by meeting back in Gelman Library, where we talked about the practice interview Nancy had run, and had the chance to ask her about her approach, and address any concerns we still had about running our own interviews/observations. A little after five we broke for a bit, then most of us met to have a dinner (provided by CLIR, thanks!) at the Peacock Grand Café.
Day 2Nancy started the day off over a light breakfast, giving us all a chance to ask any questions in preparation for our faculty interviews, as well as let us voice any concerns we had about the interview protocol or the roles we would play in the interview. We headed out around 10 to meet with our faculty interview subjects. The faculty had all been contacted by Alice Bishop (of CLIR) and agreed to be interviewed, but had purposefully not been given many details on what exactly we would be interviewing them about (in the hopes that they wouldn't alter their normal habits/practices to give us the "right" answers ;) . I was lucky enough to have Elizabeth Edwards and Kathy Magareell on my interview team.
Our faculty interviewee, Professor Z (I forgot to ask her permission to blog about her, so more mystery) was really a delight. She was incredibly laid back and open, eager to tell us all about her position, research interests, and everything that was going on with her department. She was so open, and had so much she wanted to talk about, that we found it difficult to keep the interview on track (something that Nancy had warned us may happen during our first few interviews). After 30 minutes (our technically allotted time) we were finally getting her back onto our subject matter, by starting to ask increasingly pointed (though hopefully still not leading) questions about her research process, so we decided to press on. Our questions seemed so straightforward and easy to get answers for going in to the interview, especially after watching how masterfully Nancy was able to navigate them with Professor S, that we hadn't anticipated this much difficulty going in. Here where the seven seemingly straightforward questions we were asking (to get at the topic "How do you conduct your own research and use the information you've gathered" especially with reference to library resources and services- but remember we didn't want to lead out subjects directly to telling us they used the library- again, giving us what they might feel was the "right" answer)- btw, I am paraphrasing from the actual questions here, since that is really Nancy's IP;
- We did some research about what you do, can you tell us more about some particular aspect you are currently researching?
- Do you have anything related to your research with you that you are reading right now? How did you find out about this item? How did you get it?
- How do you make notes/keep track of all the materials you are reading? Would you show us some examples? How will this be used/referenced by you later?
- Is there currently any information that you need to find for your research or teaching (if not, something topic you've recently researched, and how long ago was it)? Could you show us how you'd go about finding this information?
- Are you currently co-authoring anything, or have you ever co-authored anything? How do you handle working together on the same content/document/resource/etc?
- Can you give me a brief tour of your office, and maybe tell us a bit about how you use this space?
After taking turns attempting to rephrase the question without being too leading (it took about 25 more minutes) we managed to get some answers for all of our questions (except one). We finally had to wrap up, because we needed to analyze our interview and prepare a ten minute report for the entire group, to be presented shortly.
Over lunch, and for the next few hours, we took turns presenting and analyzing our interviews with the entire workshop group. One commonality that popped up was that we all had difficulties, to one degree or another, getting faculty members to actually show us how they did their research. They would all gladly talk about doing research (and what they researched) but generally in very abstract terms. Three groups had to repeatedly ask variations of some questions to try to gently nudge faculty into showing us their actual processes (and even the group that had an easier time with some process information, and this faculty members process was very clearly defined and involved, initially got a response equivalent to "I don't really have much of a process or method I use to do my research" :) . Nancy assured us that, with practice, we'd become more adroit at keeping the faculty engaged and excited about talking to us, while still getting them to show us as well as telling us about their research process.
Next, Nancy lead us through some exercises to show us how we could extract salient, pertinent, useful information from our interviews, even though or questions had been purposefully unfocused (not mentioning the library, library resources, etc even though that was at the heart of what we wanted to know) and despite the fact that many of us seemed to feel (I know I did) that our interviews hadn't been nearly as successful as Nancy's example interview.
Nancy opened by letting us know that if you are having difficulty getting a subject to show you how they research, rather than talk about it, first try to determine if they are just too uncomfortable to talk about the topic (you don't want to press them if they are uncomfortable and trying to avoid answering). If they seem to be comfortable, you can press/lead a bit on a question. However, rather than saying, "show me a keyword search in the library catalog/database" instead nudge them that direction by starting more simply and general, for instance ask the question "Could/Would you show me your library's home page" to get a simple task completed and get them using the computer. Once they are using the computer, try following this with, "When was the last time you used the library web page" and then "what for" and finally "can you show me?"
Nancy's own observations has shown a dichotomy between the way amateur scholars (undergrads) and established scholars (faculty, with grad students exhibiting a mix of behaviors as they transition) approach a field of research and published information;
Younger scholars, like undergrads, see information as a massive field of items related to a particular topic (published information) that if they have the right tools, and the right search strategies, they'll find items that answer their information needs based on the topic/keywords/content of the items.
Established scholars see information represented as a sea of established scholars in their field that they need to be aware of. They feel they need to make direct contact with them, and follow their work, and only read/be aware of the work of those known experts or works that those experts refer to.
Nancy then walked us through how in a real application of these techniques (with a much longer time frame, generally six months to a year of a few hours of work each week, and generally 20-24 30 minutes interviews) would work. After co-viewing sessions (by all interviews) of all the taped interviews, everyone notes what they've noticed as common elements of the interviews. Often they use post-its that are then collected and organized (sometimes passed around the table for each member to scan in case its triggers a thought or memory for them). It is vital that everyone takes the approach that no idea is wrong, and everyone either positively comment on people's observations, or just remain silent. At this point, no one needs to worry about wacky or contradictory ideas. Once all the ideas out there, and they are being reviewed by the group, the common/best will rise to the top, so no need to worry about the wacky (either fearing to mention it yourself, or being negative about someone else's idea you find wacky).
We didn't have the time to do this (the workshop was only two days, after all) so we proceeded to try some super-fast brainstorming sessions, similar to what Nancy would do, but which a much faster tempo.
First, the group reflected on what we had seen (remember, we hadn't had the time to co-view everyone's interview, just hear brief reports and see an occasional video clip from each group). Then, bearing that in mind, we had to come up with a list of
Things the faculty do very well/are very good at:- Publish
- Find (and remain aware of the work of) societies and organizations related to their research areas
- Keep up with (what they felt) was the literature in their field
- Quickly filter large masses of literature down to the ones that where actually relevant to their research
- Collect information about their field of research
- Organize information (both with physical and electronic materials)
- Collaborate with colleagues in their field (as well as within their institution)
- Very good when it comes to weeding & tossing materials they aren't using/planning on using (unlike, a few people noted, some librarians :)
- Teaching
- Working in more than one place (or diff spaces for diff pruposes)
- Research away from office
- Maintain clearly defined zones of activity (work office for some things they do, coffee shops for others, home for, often, research/reading)
- Networking
- Managing work
- Keyword searches
Then, Nancy asked us to some up with a list of
Things the faculty have trouble with:- Storing & saving materials
- Digitally organizing things
- Accessing saved files
- Finding unknown item (new materials in a field that they don't know the title or author of)
- Finding things in databases
- Using library tools
- Using librarians
- Thinking beyond their own spheres/disciplines
- Answering questions directly
- Finding literature outside their fields
- Conserving (paper, for the most part)
- Using technology with confidence (they used it, quite a lot in some cases, but constantly referred to themselves as low-tech or not very tech savvy)
- Managing citations
- Reading materials online/onscreen
- Collaborating
- Keeping their offices clean
- Using storage systems for physical items
You may notice, as we did, that the same, or markedly similar, items appear on both lists. Nancy let us know that this is very common (sometimes because different interview subjects have different strengths/weakness, but also because there are certain variations of a particular activity that they may be more or less good at, and these are all very basic generalizations of activity). Finally, Nancy asked us to put ourselves in our respective interview subjects place, and decide if we where them,
What superpower would you want to have that would help you do your research- Total awareness of what's going on in my field
- Know where all the treasures in the field are located (without having to dig around for them)
- Have an unlimited army of research assistants/the ability to clone helpful people
- Have access to any paper or electronic file they every written or read instantly
- All books/papers/web resources automatically re-organize themselves in a way that speaks directly to their own research interests
- [and I missed a few hear, because we were really trucking along…]
Following this, Nancy explained that after you make these types of generalizations/lists, you go back and look at which of these topics/ideas can be addressed within the constraints/goals of your upcoming projects and tasks. You don't want to even really consider the project(s) you are working on (and probably doing your ethnographic interview for) except in the most general sense when writing up your questions/survey tools. You want to wait until after the interviews, co-viewings, and brainstorming to do this, because we don't want to accidentally shape the outcomes/lists by biasing our questions, our interview protocols, our co-viewing sessions, or our brainstorming. These lists and brainstorming sessions in general allow you to identify areas you want to support, when they fall in your areas, as well as identify areas you could (or should) collaborate with others (units, people, etc) when the goal you've identified falls outside your own area.
Nancy then talked to us about common themes she saw from our presentations and brainstorming. These where: knowing everyone's work in your field, the need to read lots of abstracts (as opposed to full text),the need to track/follow citations, the need to judge the credibility of sources, and the need to work away from the traditional office.
Once you establish a list of commonalities, you then need to see if there is something you can pursue to build/provide, but before you choose one to pursue, you need to:
- Go back to the data and make sure the transcripts/data actually support those ideas/commonalities. If you have 24 transcripts and a team of 6, everyone carefully combs through 4 interviews and looks for support (or contradiction) of each point.
- You must consider if the ideas are actually feasible/accomplishable/affordable/timely. If it's too expensive, or will take too long, you'll need to toss that idea out. Only pursue what is feasible.
- Are the things you've identified as worth doing affordable, doable, and, finally, are they unique? If what you've identified as a worthwhile project/product/service is already available in another format (or from some other unit or vendor even), even if it that alternative isn’t quite exactly/perfectly what you envisioned for your project, it might not be worth pursuing. For example, let's say you think "I'd like to provide a way for people to share videos and tag them with keywords." Later you find out that YouTube does _most_ of what your envisioning, but won't quite perfectly accomplish your need. Even if you go forward and make your more perfect tool/service, that other approach/service (YouTube in our example) is already so prevalent and entrenched that it's unlikely your tool will gain any use (and almost certainly won't supplant it).
- Address the "Smile/Ugh!" factor. Ask yourself if the people you will need to leverage to achieve your goal, like your programmers, going to smile when you tell them about it or are they going sigh or release some guttural noise of contempt (Ugh!) and hate every second of time they spend working on your project.
Once your ideas have passed gotten through to this point, you need to move beyond ethnographic interviews and analysis, and start prototyping your new service/product, including iterative cycles of usability testing (like heuristic testing) and prototype refinement (often using advanced software that tracks mouse/keyboard use, etc.). Information gathering, using tools like ethnographic interviews, is a necessary and important beginning for your projects, but doesn't in and of itself produce any direct results. Instead, it informs all your future decisions.
You often want someone with experience doing this type of ethnographic research to help you with your project. You'll likely want to recruit someone (like an anthropologist or sociologist, occasionally a market analyst will work) who has experience making and executing these types of instruments to help you establish your initial question set and interview protocols. Often a very experienced person can come in and only very marginally alter your wording, question order, etc. to make sure that you aren’t inadvertently introducing leading questions and biasing the outcome.
Finally, we wrapped the workshop up by filling out a "Project Planning Worksheet" where we tried to identify ongoing or possible future projects that we could applied what we've learned at this workshop. We then shared our ideas with the group.
I have far more notes than even this excessive post represents, and I'd be happy to share them with anyone who's interested. Also, if I've made any mistakes in this overview, please let me know and I'll correct them. It was an incredibly busy two days, and I didn't have time until today to finally try to wrap all my scattered notes into a single review. The workshop was very informative (far more than most presentations, workshops, and conferences I've attended). It was well paced, and I was fortunate enough to take part in it with an amazing group of librarians and IT professionals who made it incredibly fun.
CLIR Workshop on Faculty Research Behavior Participants:Workshop Leader: Nancy Foster
CLIR Workshop Coordinator: Alice Bishop
Workshop participants: Elizabeth Edwards & David Bietila (George Washington University), Jill Hollingsworth & Molly Sorice (Georgetown University), Carrie Forbes & Greg Colati (University of Denver), JoAnn Jacoby & Robert Slater (University of Illinois), Kathy Magarrell & Jodi Scholl (University of Iowa), Amanda Hornby & Jennifer Ward (University of Washington)
