My offsider, Pauline, is embarking on the new world of question writing as we develop surveys for our Museum Shop and Science in the Suburbs. This got me thinking about what I would recommend as resources for question-writing.
Found a good online resource from the Committee on Audience Research and Evaluation called Visitor Studies 201 presented at the 2006 AAM, with a downloadable pdf on developing questions. There is a 'how to' on writing questions posted by the Evaluation and Visitor Studies SIG with a link to an online guide to designing and conducting visitor surveys by Julie Leones, The University of Arizona.
As mentioned before, Judy Diamond's 1999 book Practical Evaluation Guide: tools for museums and other informal educational settings (Walnut Creek: AltaMira Press) is a good starting point. For general advice on sampling and suggested questions for a whole range of areas try:
* McManus, P. (1991). Towards Understanding the Needs of Museum Visitors. In G. Lord & B. Lord (Eds.), The Manual of Museum Planning (pp. 35-52). London: HMSO. (I think there's a newer version of this book and it's in our Research Library).
You could also look at general social research texts - one I have used is:
* de Vaus, D. (1991). Surveys in Social Research (3rd ed.). London: UCL Press, especially Chapter 6 (also in our Research Library).
Good luck!
Monday, May 28, 2007
Friday, May 18, 2007
Museums and Web 2.0
While at the MA conference I have been asked by several people here what Web 2.0 is ... this is worrying! My straw poll of museum professionals at this conference revealed that noone has heard of Web 2.0, let alone doing any thinking about what it might mean for museums. So I promised to send them something, and have blogged it instead (I have been kind and sent them an email also!).
The three things I recommend people view/read are:
* The Machine is us/ing us
* Nina Simon's blog post about implementing Web 2.0 in museums
* paper discussing organisational barriers and Web 2.0 We are going to discuss this on our wiki
Of course there's lots more but I don't want to overwhelm people! Jerry's papers presented at the MA conference had a great summary of how strategy needs to come before tools so, at the risk of now overwhelming people I would add this to your list.
The three things I recommend people view/read are:
* The Machine is us/ing us
* Nina Simon's blog post about implementing Web 2.0 in museums
* paper discussing organisational barriers and Web 2.0 We are going to discuss this on our wiki
Of course there's lots more but I don't want to overwhelm people! Jerry's papers presented at the MA conference had a great summary of how strategy needs to come before tools so, at the risk of now overwhelming people I would add this to your list.
Museums Australia Conference 2007
HI all, am attending the annual Museums Australia conference in Canberra and am going to attempt to blog my experiences. There are no internet facilities at the conference (!!) so will check in when I can.
Tuesday, May 15, 2007
A review of literature on audience research and evaluation for museum websites
There is a clear conception within the literature of the potential within the museum area for online development, given the already inherent value a museum has in terms of its reservoir of high quality content, often with its own intellectual property rights, and an established reputation for quality information provision, objectivity, and authority. (Both Daniel Cunliffe et al and Tom Hennes make similar points).
The challenge for museums on the web is to take advantage of and develop this potential, creating an online entity that is not just a pale reflection of the physical museum, but an exciting expansion into new ways of accessing, interacting with, exploring and developing resources held by museums.
Historically website design has been focused on the visual design, and when overhauling their sites, institutions such as museums may be faced with redesigning a site that may still ‘look’ good, but behind the scenes is dysfunctional, based on obsolete code and almost impossible to add to or update in a simple manner. Added to this is the challenge of addressing web 2.0 issues, including a desire to make the site more interactive and to bring the user into content manipulation and production. This adds another area of concern, with challenges to issues of control and management.
The literature shows that there are basic criteria to assess website usability and audience/user profiles that have been developed over the last 10-15 years, and these are seen as working effectively when applied as a group, rather than depending on any one method or aspect. A clear consensus emerges in regards to key issues.
These include: the necessity of usability testing, assessment and research at all stages of the web development process; the development of user profiles, as virtual users cannot be assumed to match physical visitors, about whom much research has already been done; the importance of a conceptual approach to the site, where the site is not just a reflection of the physical museum but an active entity in itself; and finally situating the virtual museum within a marketplace context.
To achieve this, the agreement is that there needs to be a lot of preparation, research, testing, trials and then more testing.
The earliest article, Usability Evaluation for Museum Websites, offers a thorough analysis of these criteria, which include:
1. user testing- with prototypes within the museum, monitor and observe interaction behaviour, use feedback. The user is usually given a set of tasks to carry out and is monitored. Can be monitored by video, think aloud protocol, observer’s notebook (problems include finding a typical user).
2. interviews- questionnaires about a prototype, an existing site, or ongoing feedback. Could be done via a kiosk in the museum.
3. log file analysis- looking at the data gathered automatically, includes such info as number of hits, type of platform, where the user came from, time spent on page and quite a lot more (problems include amount of data, relevance of data, interpretation, identification of unique visitors, caching) Then how do you extrapolate intention from info – why from what?
4. online questionnaires/feedback – the two major concerns are: self selection of sample; and response rate to draw reliable conclusions. Often there is a low response rate (between 2 and 5%)
5. Heuristic evaluation – doesn’t involve users, evaluates site against a set of agreed criteria or guidelines, research already available.
6. Procedures for ongoing maintenance and development must also be considered.
The article Audiences, Visitors, Users: Reconceptualising Users of Museum On-line Content and Services expands on these criteria to develop a more formal set of principles for web site development and evaluation. They put museum web site development into the context of the marketplace and come up with six design principles:
• acknowledge the marketplace web sites exist within
• have clear site goals, from the perspectives of the market, user, and organisation
• design in relation to these goals
• design an information space, not a museum replica
• test with target users at all stages, many times
• evaluate in terms of your goals, not just according to user demographic or products on site.
There are many examples in the literature (SFMOMA, Museum of Victoria) of organisations that take these basic criteria and offer case studies on how they apply them to their own redesigns, the challenges that emerged, and how effective the process was.
Some general points that come from these case studies are:
o Usability is important in developing an effective, interesting and user friendly website.
o There is a connection between website usage and attendance, however the audiences do not necessarily correlate completely.
o The different approaches – heuristic, user surveys, questionnaires, log file data – all contribute to a more complete picture of users and usability.
o How a user gets to the site is largely based on internet searches, rather than access through the home page.
o Museums have a very active role to play in developing their physical content for the online environment, and need to look at issues of intellectual property, copyright, etc.
o There is a lot of potential for museum sites – both directly and indirectly related to their physical role (see Hennes, Sherry Hsi)
o There are more novel methods of assessing user interaction for example the kiosk approach of the Exploratorium that can have a role in website evaluation.
o Web 2.0 – do the users want it? How do museums implement it in ways that do not threaten users, or waste resources? (see SFMOMA article, fresh and new blog re percentage of active users vs passive)
o How then do museums assess the use of this interactive content – data file analysis has not yet caught up with server side includes, active content, feeds etc.
Overall, for those embarking on web site redesign, or initial design, there is already a good body of research and practical examples to draw on for methods, ideas and examples. The challenge of web 2.0 and how to assess the role it plays in website assessment and audience research wil be another are for research and assessment over the next few years, as museums continue to take up the challenge of implementing web 2.0 concepts and ideas.
The challenge for museums on the web is to take advantage of and develop this potential, creating an online entity that is not just a pale reflection of the physical museum, but an exciting expansion into new ways of accessing, interacting with, exploring and developing resources held by museums.
Historically website design has been focused on the visual design, and when overhauling their sites, institutions such as museums may be faced with redesigning a site that may still ‘look’ good, but behind the scenes is dysfunctional, based on obsolete code and almost impossible to add to or update in a simple manner. Added to this is the challenge of addressing web 2.0 issues, including a desire to make the site more interactive and to bring the user into content manipulation and production. This adds another area of concern, with challenges to issues of control and management.
The literature shows that there are basic criteria to assess website usability and audience/user profiles that have been developed over the last 10-15 years, and these are seen as working effectively when applied as a group, rather than depending on any one method or aspect. A clear consensus emerges in regards to key issues.
These include: the necessity of usability testing, assessment and research at all stages of the web development process; the development of user profiles, as virtual users cannot be assumed to match physical visitors, about whom much research has already been done; the importance of a conceptual approach to the site, where the site is not just a reflection of the physical museum but an active entity in itself; and finally situating the virtual museum within a marketplace context.
To achieve this, the agreement is that there needs to be a lot of preparation, research, testing, trials and then more testing.
The earliest article, Usability Evaluation for Museum Websites, offers a thorough analysis of these criteria, which include:
1. user testing- with prototypes within the museum, monitor and observe interaction behaviour, use feedback. The user is usually given a set of tasks to carry out and is monitored. Can be monitored by video, think aloud protocol, observer’s notebook (problems include finding a typical user).
2. interviews- questionnaires about a prototype, an existing site, or ongoing feedback. Could be done via a kiosk in the museum.
3. log file analysis- looking at the data gathered automatically, includes such info as number of hits, type of platform, where the user came from, time spent on page and quite a lot more (problems include amount of data, relevance of data, interpretation, identification of unique visitors, caching) Then how do you extrapolate intention from info – why from what?
4. online questionnaires/feedback – the two major concerns are: self selection of sample; and response rate to draw reliable conclusions. Often there is a low response rate (between 2 and 5%)
5. Heuristic evaluation – doesn’t involve users, evaluates site against a set of agreed criteria or guidelines, research already available.
6. Procedures for ongoing maintenance and development must also be considered.
The article Audiences, Visitors, Users: Reconceptualising Users of Museum On-line Content and Services expands on these criteria to develop a more formal set of principles for web site development and evaluation. They put museum web site development into the context of the marketplace and come up with six design principles:
• acknowledge the marketplace web sites exist within
• have clear site goals, from the perspectives of the market, user, and organisation
• design in relation to these goals
• design an information space, not a museum replica
• test with target users at all stages, many times
• evaluate in terms of your goals, not just according to user demographic or products on site.
There are many examples in the literature (SFMOMA, Museum of Victoria) of organisations that take these basic criteria and offer case studies on how they apply them to their own redesigns, the challenges that emerged, and how effective the process was.
Some general points that come from these case studies are:
o Usability is important in developing an effective, interesting and user friendly website.
o There is a connection between website usage and attendance, however the audiences do not necessarily correlate completely.
o The different approaches – heuristic, user surveys, questionnaires, log file data – all contribute to a more complete picture of users and usability.
o How a user gets to the site is largely based on internet searches, rather than access through the home page.
o Museums have a very active role to play in developing their physical content for the online environment, and need to look at issues of intellectual property, copyright, etc.
o There is a lot of potential for museum sites – both directly and indirectly related to their physical role (see Hennes, Sherry Hsi)
o There are more novel methods of assessing user interaction for example the kiosk approach of the Exploratorium that can have a role in website evaluation.
o Web 2.0 – do the users want it? How do museums implement it in ways that do not threaten users, or waste resources? (see SFMOMA article, fresh and new blog re percentage of active users vs passive)
o How then do museums assess the use of this interactive content – data file analysis has not yet caught up with server side includes, active content, feeds etc.
Overall, for those embarking on web site redesign, or initial design, there is already a good body of research and practical examples to draw on for methods, ideas and examples. The challenge of web 2.0 and how to assess the role it plays in website assessment and audience research wil be another are for research and assessment over the next few years, as museums continue to take up the challenge of implementing web 2.0 concepts and ideas.
Thursday, May 10, 2007
Evaluating teacher/student experiences at museums
My colleague here at the Museum, Pauline, is digging around for recent published studies of evaluations with teachers about their experiences, expectations and needs when visiting museums and cultural institutions generally.
The recent post I wrote in response to Alison's request is also useful background.
Pauline has promised to post some of the things she has found so far, but I was wondering if anyone had some studies, information, findings etc they'd like to share with us??
The recent post I wrote in response to Alison's request is also useful background.
Pauline has promised to post some of the things she has found so far, but I was wondering if anyone had some studies, information, findings etc they'd like to share with us??
Tuesday, May 08, 2007
What are the most effective ways to recruit new audiences and build repeat visitations to a cultural institution?
This question came from Alison Wishart, State Library of Queensland (who promises to blog a response!): "My assumption is that one of the most effective ways is to give school students, who visit the cultural institution with their school group, an exciting and rewarding experience so that the students then go home and tell their parents about the fantastic cultural institution and the family visits as a group again and again. Is there any written research in this area that you could point me to?"
Alison, research that I know of has continually found that the characteristic which most impacts on adult museum visits is whether they were taken to museums as children and the types of experiences they engaged in. The "big picture" references are:
* Falk, J., & Dierking, L. (1992). The Museum Experience. Washington: Whalesback Books.
* Falk, J., & Dierking, L. (2000). Learning from Museums: Visitor Experiences and the Making of Meaning. Walnut Creek: AltaMira Press.
Prince investigated a range of attitudes and perceptions that were key to museum visiting. He found that visitors’ previous experiences with museums, as well as with learning and education generally, determined whether people then visited museums and the subsequent experiences they remembered. The reference for his work is:
* Prince, D. (1990). Factors Influencing Museum Visits: An Empirical Evaluation of Audience Selection. Museum Management and Curatorship, 9, 149-168.
Other references for these conclusions from a family perspective are:
* Ellenbogen, K. (2002). Museums in Family Life: An Ethnographic Case Study. In G. Leinhardt & K. Crowley & K. Knutson (Eds.), Learning Conversations in Museums (pp. 81-101). Mahwah: Lawrence Erlbaum Associates.
* Ellenbogen, K., Luke, J., & Dierking, L. (2004). Family Learning Research in Museums: An Emerging Disciplinary Matrix? Science Education, 88(Supplement 1), S48-S58.
and from a school-visit perspective are:
* Falk, J., & Dierking, L. (1997). School Field trips: Assessing Their Long-Term Impact. Curator, 40(3), 211-218.
The latter is particularly useful. A general overview of school visits and museums, with a comprehensive reference list is:
* Griffin, J. (2004). Research on Students and Museums: Looking More Closely at the Students in School Groups. Science Education, 88(Supplement 1), S60-S70.
I'm not sure if there have been any long-term tracking studies of visitation, but certainly Ellen's work is worth checking out.
Of course your broader question about new audiences generally is a whole new area but I have started small here. The big issue of how to attract and keep youth audiences (for example) can be found on here on my website. We also found in our study of audiences with disabilities that if museums offered great learning experiences, not just accessible ones, these groups would visit more regularly (and bring a whole lot more people with them).
Look forward to hearing from you.
Alison, research that I know of has continually found that the characteristic which most impacts on adult museum visits is whether they were taken to museums as children and the types of experiences they engaged in. The "big picture" references are:
* Falk, J., & Dierking, L. (1992). The Museum Experience. Washington: Whalesback Books.
* Falk, J., & Dierking, L. (2000). Learning from Museums: Visitor Experiences and the Making of Meaning. Walnut Creek: AltaMira Press.
Prince investigated a range of attitudes and perceptions that were key to museum visiting. He found that visitors’ previous experiences with museums, as well as with learning and education generally, determined whether people then visited museums and the subsequent experiences they remembered. The reference for his work is:
* Prince, D. (1990). Factors Influencing Museum Visits: An Empirical Evaluation of Audience Selection. Museum Management and Curatorship, 9, 149-168.
Other references for these conclusions from a family perspective are:
* Ellenbogen, K. (2002). Museums in Family Life: An Ethnographic Case Study. In G. Leinhardt & K. Crowley & K. Knutson (Eds.), Learning Conversations in Museums (pp. 81-101). Mahwah: Lawrence Erlbaum Associates.
* Ellenbogen, K., Luke, J., & Dierking, L. (2004). Family Learning Research in Museums: An Emerging Disciplinary Matrix? Science Education, 88(Supplement 1), S48-S58.
and from a school-visit perspective are:
* Falk, J., & Dierking, L. (1997). School Field trips: Assessing Their Long-Term Impact. Curator, 40(3), 211-218.
The latter is particularly useful. A general overview of school visits and museums, with a comprehensive reference list is:
* Griffin, J. (2004). Research on Students and Museums: Looking More Closely at the Students in School Groups. Science Education, 88(Supplement 1), S60-S70.
I'm not sure if there have been any long-term tracking studies of visitation, but certainly Ellen's work is worth checking out.
Of course your broader question about new audiences generally is a whole new area but I have started small here. The big issue of how to attract and keep youth audiences (for example) can be found on here on my website. We also found in our study of audiences with disabilities that if museums offered great learning experiences, not just accessible ones, these groups would visit more regularly (and bring a whole lot more people with them).
Look forward to hearing from you.
audience research wiki
In the spirit of trying all forms of web 2.0 media, we have set up an audience research wiki , as a dicussion forum for audience research. Here we will post articles that particularly address museum websites and audience research. Please have a look, and add your comments and thoughts to our discussion pages. If you are new to wikis, have a look around, and also have a look at Chris Barry's education section wiki.
Saturday, May 05, 2007
Innovative audience research
This post from Chris Larry: "Thought you might find interesting the work we are doing at American History Workshop with our Telling Lives story capture technology. This technology allows audiences to answer question to a video camera that is than collected as a response database." I think that this is a very innovative approach to seeking audience feedback. Are there any others out there?
Chris: do you see this as a sustainable way to continue capturing feedback? How has the organisation used the information in their future development of programs?
Chris: do you see this as a sustainable way to continue capturing feedback? How has the organisation used the information in their future development of programs?
Doing audience research
Mel asked "What are traditional methods of finding out about your audience and is the internet affecting the way you do audience research?"
Audience research is a discipline of museum practice that provides information about visitors and non-visitors to museums and other cultural institutions, influencing the ways museums think about and meet the needs of their audiences and stakeholders. Audience research is also a strategic management tool that provides data to assist museums more effectively plan and develop exhibitions and programs; to meet their corporate goals; and to learn as organisations.
In audience research a range of methods are used to address issues such as who visits; motivation; behaviour; satisfaction and learning; as well as who does not visit and why. Areas examined include visitor demographics, visitor behaviour, leisure habits and learning strategies. Methods are chosen in accordance with the questions that need to be answered.
In a 1990 paper, Chan Screven noted that the most popular methods inlcude structured and open-ended interviews, questionnaires and rating scales, informal conversations, tracing visitor movements through exhibits and unobtrusive observations of stops, time spent, exhibit usage, and reactions to mocked-up exhibit components.
Other ways of conducting audience research include video-taping visitor behaviour and audio-taping conversations. Focus groups; telephone surveys; and online surveys and panels are also employed to research current and potential audiences as well as non-visitors.
A list of beginning readings about audience research in museums can be found on the
FAQ section of my website - on there you'll also find a downloadable list of beginning references.
I also wrote a paper on the history and practice of audience research in cultural institutions: Kelly, L. (2005). Evaluation, Research and Communities of Practice: Program Evaluation in Museums. Archival Science, 4(1-2), 45-69. This paper contains a good list of references. More information about it can be found at the publishers' website.
I'll get to the internet part of your question shortly.
Audience research is a discipline of museum practice that provides information about visitors and non-visitors to museums and other cultural institutions, influencing the ways museums think about and meet the needs of their audiences and stakeholders. Audience research is also a strategic management tool that provides data to assist museums more effectively plan and develop exhibitions and programs; to meet their corporate goals; and to learn as organisations.
In audience research a range of methods are used to address issues such as who visits; motivation; behaviour; satisfaction and learning; as well as who does not visit and why. Areas examined include visitor demographics, visitor behaviour, leisure habits and learning strategies. Methods are chosen in accordance with the questions that need to be answered.
In a 1990 paper, Chan Screven noted that the most popular methods inlcude structured and open-ended interviews, questionnaires and rating scales, informal conversations, tracing visitor movements through exhibits and unobtrusive observations of stops, time spent, exhibit usage, and reactions to mocked-up exhibit components.
Other ways of conducting audience research include video-taping visitor behaviour and audio-taping conversations. Focus groups; telephone surveys; and online surveys and panels are also employed to research current and potential audiences as well as non-visitors.
A list of beginning readings about audience research in museums can be found on the
FAQ section of my website - on there you'll also find a downloadable list of beginning references.
I also wrote a paper on the history and practice of audience research in cultural institutions: Kelly, L. (2005). Evaluation, Research and Communities of Practice: Program Evaluation in Museums. Archival Science, 4(1-2), 45-69. This paper contains a good list of references. More information about it can be found at the publishers' website.
I'll get to the internet part of your question shortly.
Research on museum websites developed for teachers
I received the following email from the VSG list:
"Jessica from Randi Korn & Associates is looking for fairly recent studies of museum Web sites developed for teachers. She is trying to comb the literature to get an idea of how teachers are using the Web, specifically museum Web sites."
Jessica, I did a detailed study in November 2005 to gain a better understanding of how emerging internet technologies are affecting the learning environment, with a specific focus on teachers and high-school students. The overall objective of the research project was to provide us with guidance on how to best develop a website that meets the needs of students and teachers in the primary and secondary levels across a range of curriculum areas. General objectives were to gain insights into how students and teachers are using the internet and what they are looking for when they access websites.
We conducted a series of five focus groups with students and teachers from a mix of public and private educational institutions. There were primary school teachers from years 5-6; secondary school teachers from years 7-10 (mainly science teachers and co-ordinators); students aged 13 to 16; and a group of teachers specialising in technology and computing. The aim of the discussion was to look at participants’ use of the internet, their knowledge of media, and their reaction to a range of websites.
Ahead of time, participants received a letter outlining the aims of the research and providing a list of websites to review. These museum, gallery and general sites had targeted materials for teachers and allowed a two-way exchange of information - these proved to be extremey useful as a way to seek general and specific feedback - here's the list of the sites we used. As well, participants brought along a list of the website ‘favourites’ they used for recreation and education, which again gave us useful insights into how they were using the web.
The paper from this study (Kelly and Breault, 2006) will be published in the proceedings of the 2006 ICOM-CECA conference this year. I'm happy to email this paper to anyone who wants it.
Also, I'd be interested to hear of others' experiences, BUT rather than email - which only reaches a select few - why not blog your responses for many to share??
"Jessica from Randi Korn & Associates is looking for fairly recent studies of museum Web sites developed for teachers. She is trying to comb the literature to get an idea of how teachers are using the Web, specifically museum Web sites."
Jessica, I did a detailed study in November 2005 to gain a better understanding of how emerging internet technologies are affecting the learning environment, with a specific focus on teachers and high-school students. The overall objective of the research project was to provide us with guidance on how to best develop a website that meets the needs of students and teachers in the primary and secondary levels across a range of curriculum areas. General objectives were to gain insights into how students and teachers are using the internet and what they are looking for when they access websites.
We conducted a series of five focus groups with students and teachers from a mix of public and private educational institutions. There were primary school teachers from years 5-6; secondary school teachers from years 7-10 (mainly science teachers and co-ordinators); students aged 13 to 16; and a group of teachers specialising in technology and computing. The aim of the discussion was to look at participants’ use of the internet, their knowledge of media, and their reaction to a range of websites.
Ahead of time, participants received a letter outlining the aims of the research and providing a list of websites to review. These museum, gallery and general sites had targeted materials for teachers and allowed a two-way exchange of information - these proved to be extremey useful as a way to seek general and specific feedback - here's the list of the sites we used. As well, participants brought along a list of the website ‘favourites’ they used for recreation and education, which again gave us useful insights into how they were using the web.
The paper from this study (Kelly and Breault, 2006) will be published in the proceedings of the 2006 ICOM-CECA conference this year. I'm happy to email this paper to anyone who wants it.
Also, I'd be interested to hear of others' experiences, BUT rather than email - which only reaches a select few - why not blog your responses for many to share??
Thursday, May 03, 2007
ExhibitFiles Website
Check out ExhibitFiles - a great new website for exhibitions and museum staff - top stuff!
There is an interview with the site developers on Nina Simon's site which gives good insights to the thinking behind the development of ExhibitFiles.
There is an interview with the site developers on Nina Simon's site which gives good insights to the thinking behind the development of ExhibitFiles.
Subscribe to:
Posts (Atom)
