Using Flickr as an evaluation tool

Check out the Flickr audience research site where childrens’ drawings of the African Impressions exhibition are stored. I’m experimenting with using these kind of sites as extra evaluation tools. Thanks to Ariel for doing all the scanning for me.

Any feedback welcome...

E-kids’ college: afternoon session

What we did this afternoon was for the students to interview some of the Museum's scientists about what they do and their personal and professional interests. Mark and Amanda fielded many questions about their work and how they got interested in their field of work.

There were some really provocative and interesting questions from the students. One lovely one was How do we know what people know and what people don't know? Mark answered that by reminding us that all the answers aren't always to be found on the internet and that there are really cool books that you can use too. They also asked What was the most interesting thing they had discovered? Mark discovered a new species of wallaby, and Amanda's friend also discovered a new species of fish. Other questions were: What is the most weirdest and wackiest thing you've ever done in your job? Any scary experiences? Where's the most interesting place you've been to? Thanks Mark and Amanda, I know I learned more about you two who I usually see every day! Now I understand more about your passions and why you got into doing what you do. Amanda reminded us to hang in there as it may take some time to get some paid work.

The discussions have been really productive, some ideas that emerged at the end were that they:

• don't want to use mobiles for accessing internet as they are too hard to use (with small keyboards, etc), the screens are no good and they don't have a mouse to use, although when I asked them about whether they'd like to use some kind of mobile device throughout the Museum to gather content they were really keen
• suggested that we have a blog with question of the week that they can email in
• would like to have a section on the website where they could leave a comment/feedback about their experiences and see what others have said
• really wanted to talk to scientists via some kind of video link, where you log in and talk to them in real time online
• felt we should be uploading video content both to our own site and YouTube (they admit their views on this had changed over the day)
  

The day finished with us going on a tour of an area that the Museum is thinking about turning into a new media space. And after that we thanked them, gave each school a certificate and finished!

Thanks so much for all your efforts as we now have a goldmine of information we can use when developing our new website and other digital technologies.

E-kids’ college: morning session

What is it like to learn using computers and digital technologies and what do we think about the web? You can do anything, go anywhere in the world, and do anything unexpected and find anything. Web is about bringing things together and mixing them (people and things). Web opens new doors and new discoveries. Finding unexpected things, using shortcuts, learning more about how to use it. The internet is big, lots of stuff to discover and find out. Wild and unexpected and creative.

Next we went on a visit – one to the Palaeontology collection and one to the Museum's search and discover area. We were curious to find out how objects in the Museum could be made more interesting by supplementing them through digital technologies. All ideas were great and really useful. Some from my group were:

• An interactive collection space where you click on a cupboard, drawer etc and then have objects to explore
  
• 3D mapping of objects that you could rotate and explore further
  

We also got them to complete a survey with ten of their friends before they came here. The survey asked them questions about what they do online in order to compare to data we have collected from adults in Australia and the US. I'm still working on that! They also completed the sentence: Not being able to access the web is like not being able to ..., and here's some of what they said:

• Walk, breathe, talk, eat, survive, socialise
  
• Get access to water
  
• Open a chest
  
• Get an education
  
• Access a whole new world
  
• Go to school
  
• Wear clothes and be fashionable
  
• See the sun
  
• Travel around the world, explore my inner self or broaden my horizon
  
• Read a simple book
  

The final activity we did this morning was to think about websites b y looking at a site called ShowMe. This got a really bad review unfortunately and they felt that the site was developed by older people who weren't in touch with them ... Since they didn't like it we asked them to show us what sites they did like. Two sites from my group were FreeRice, where you learn about vocabulary/words while contributing to a good cause. They loved the colours and clean design of that site too. The other was MiniClip Games, for storing games which they felt was really well-organised, easy to find the games and fun as well.

One thing that struck me as they talked was that they felt the Museum's site was the place where we should store our content, videos, images and so on, not necessarily YouTube, Flickr and other places. They felt we should separate ourselves from what they perceive as fun sites – I'm not sure about this myself and is something that I'll be exploring further.

E-kids’ College

It's an exciting day for us here at the Museum. Around 24 students from schools across NSW are attending the second of our Kids' Colleges. This one is looking at digital technologies, how young people are using the web and to explore how e-learning relate to museums and young people. I'll attempt to blog as we go with my impressions so this may be a long and fragmented post ...

Timing and pace of museum education visits

This question from Cathy Burke (UK): Dear Lynda, Susan Groundwater-Smith suggested I contact you to see if you know of any studies that have examined matters of timing and pace in the way that museum educators organised the experience of children visiting museums - especially on children's views on or experience of this - I am involved with a project that has generated this question at Manchester Art Gallery here in the UK. Looking for suitable literature. Feels like it is an important issue but I am finding it difficult to find relevant studies.

Hi Cathy. I don't know of any studies about this actually. A good resource is the Visitor Studies Group email alert where you can post a question. You could also look at Janette Griffin's thesis on school visits and museums, especially the lit review. It is available online at this link (I think!). If it doesn't work then go to the home page, search for University of Technology and her name and it should pop up. You could also email Janette (her details are on the UTS website).

There have been many general studies of timing in museums which may also be a good place to start. The Methodology chapter of my thesis has relevant lit and references (scroll down the page and it's there as a pdf).

I'd be really interested in what you find - can you let us all know via the blog please?

Education sector and museum domain workshop

Attended this roundtable today organised by the Le@rning Federation through the Curriculum Corporation. Was a large gathering of museum representatives and folks from various education domains. Really productive discussion and I learned lots about how different education domains access content through their own learning portals. This presents issues for institutions who may need to get their content accepted by a huge range of education jurisdictions, not only the state system but providers like the Catholic Education system. It was interesting to hear from Charles Morgan from Tasmania who described how the learning objects are actually used in the classroom, and this brought alive the TLF process to me. Tim Hart from Museum Victoria raised incredibly important issues around museum and digitisation and working with organisations like TLF. The points I took away were:

• Museums need to standardise process in order to play together in the digital space (what a great quote!)
  
• How much have museums learned about what they're doing online?
  
• Too focused on creating physical exhibitions (I may have put my own bias on this comment, don't know if that's what he actually said!)
  
• Currently we all go it alone too much
  
• We are at a point where we can combine resources for people to access online – for example why not a search that throws back an artwork, an object, a book, a piece of ephemera, and archival film and so on across all our collections?
  

To me the issue of metadata and standards are the most critical one facing the sector. When I raised the issue of how the various domains will deal with the future of the web and social networks and the way students learn and interact in the future, I found out that this actually won't be a problem as long as the metadata is right – we need to be able to map a common authentification according to Seb (I don't know what that really means but I'm sure someone's onto it!).

The wrap-up of the workshop was asking for one salient point from each working group. These were (from my perspective):

• Uncoordinated, ad hoc and disparate nature of what's out there – infrastructure, content, uses of material, therefore need system to facilitate sharing of resources
  
• Utilises Web 2.0 tools, enable museum staff to interact with users and facilitate teachers and students able to tell their own stories and interpretations of objects
  
• How to deal with the richness and vastness of the data, complications of findings and searching
  
• Multiplicity of portals, brokerage arrangements makes for unnecessary levels of complication – need a national shared portal
  
• Students want to be producers and creators using the resources of museums and tool s of Web 2.0 while still tied to curriculum outcomes and assessment
  
• Sustainable process for cultural institutions to develop their own content for teachers in the future
  
• Sustainability will need to come through collaboration
  
• New signposts and new directions – look at the national statements of learning as a starting point? New roles and ways of learning – how to resource? How to digitise? How to prioritise? What are the measures of success? Where will mobile technologies fit?
  

Afternoon was a symposium. First paper Pedagogy and digital content: findings from three years evaluation of TLF online curriculum content by Professor Peter Freebody, University of Sydney. His report will be on TLF website end of the month. Key points for me that are relevant to what the Museum's doing:

• Effective ICT use and integration: committed leadership to here and now and building up of skills over time; a champion; growing social networks of people working together without having to blaze the trail themselves; a working plan that involved personnel and resources; well-directed and high quality resources; lot of old pedagogies going on around new technologies
  
• One finding from school visits was that often school leaders deal with procedural issues rather than curriculum issues (this is very relevant – focusing on process over what you're actually there to be doing...)
  

Several niggles I had – I didn't get a sense of what the actual experience of teachers and students were? The processes of evaluation, while I accept they need to be long-term and rigorous have the potential to be left behind –by the time you've disseminated the findings the audience has actually move on. I also have problems with language used around "mature" users and waiting – I think this is rather paternalistic – who's to define what mature practice is? It needs to be end-user defined, not defined by the researcher?? Guess I'll need to read the report.

Professor Stephen Heppell
Transforming practice – what (else) does it take? Points:

• Writing policies can't keep up with the pace of change
  
• Now we have lots of social space and social spaces to put stuff, we have mashups that join things together
  
• The gadgets have real scale, globality and ubiquity (the iTouch is a good example)
  
• Question – what haven't we got that learning needs? We haven't got identity; we haven't got time; no certainty of longevity; ability to annotate as you go; narrative layers and ability to link/join threads together (shiftspace is starting to do threading)
  
• There is not a shortage of content anymore – the world is awash with good stuff, but would be better if could thread them together, annotate them, narrate them
  
• We are in a learning age (not an information or knowledge age – that's already there) – what a great opportunity!
  
• Asked children what does being literate mean? Answer manage a community in Facebook; upload a video to YouTube; edit a Wikipedia entry; choose a safe online payment site; subscribe to a podcast; turn predictive text on/off on a fone; manage a group's Flickr account; vote in an online poll; comment on a concern site
  
• The death of education and the dawn of learning – all the boxes and rigidity are going. Judgements will be made by the learner in future
  

This was one of the most stimulating, fun and useful talks about the future of the web I have ever heard I must say. There was so much more he said that I haven't blogged about but I will be incorporating them into a paper I'm writing about the organisational change process in museums through the implementation of Web 2.0.

Impact of museum health/well-being programs

May Redfern asks: I am currently writing a paper for a conference on measuring cost/benefit analysis for museums delivering health and well being agendas (part of the broader social value remit). I was therefore just wondering if you know of examples of best practice in this area? At the moment I'm focusing on the UK's MLA's generic social outcomes framework and the new economics foundation's measurements. It would be great if I could add some more 'real life' examples of health/well-being policy delivery and evaluation in a museum setting.

Hi May. This is a tough one. I don't know of any specific evaluations and programs but I'm sure there must be some in the UK especially given their social inclusion agenda. Maybe ask via the Visitor Studies Group – they have a great email list.

The other person who springs to mind is Lois Silverman who has researched museums as therapy and come up with some interesting findings. She has published quite a lot and a quick Google search brings up various pieces she has done. Lois also has a chapter in the Richard Sandell edited book Museums, Society and Social Inequality that may be a good start. Last I heard she was at Indiana University in the Department of Parks, Recreation etc, but have looked through their (frustrating!) site and can't uncover anything further.

Good luck with your abstract – is it for the Inclusive Museum conference next June?

Summative Evaluation

Anonymous asked: Dear Lynda, Does visitors book and feedback form that we found at the end of exhibition falls under summative evaluation?

Hi Anonymous. In my view visitors' books and feedback forms, when used strategically, can be a very valuable and easy form of summative evaluation. For example, we used feedback forms for a controversial exhibition, and by posting up others' forms coupled with media clippings found that we had a series of conversation happening – visitor-to-museum and visitor-to-visitor. Feedback forms can also help gather demographic information if you don't have the resources to do a targeted survey.

For more information about summative evaluation see the information sheet called Exhibition Evaluation on my website, and also the paper outlining the history and future of audience research that I have posted to my wiki – it is the one called Evaluation, Research and Communities of Practice: Program Evaluation in Museums (go down the page a bit).

Biodiversity Exhibition Evaluations

This question from Georgina Bishop, Natural History Museum, London: Hi Lynda, Hope all is well with you. I am gathering some information on Biodiversity exhibitions as we are in the beginning stages of developing a brief for one that will happen as part of the NHM's masterplanning. I think that you also have a Biodiversity exhibition and I wondered if it would be possible for you to send me any front-end research you did as part of this.

Hi Georgina and I am well thanks! I have chosen to blog my response so others can have access.

We did a series of studies for our former Biodiversity: Life Supporting Life exhibition as follows:

• Frontend evaluation report, 1997: concepts, themes, prior understanding and interpretive approaches
  
• Formative evaluation report 1 1997: concepts, population issues and interpretive strategies
  
• Formative evaluation report 2, 1998: topics and concepts, merchandise and exhibition title
  
• Formative evaluation report 3, 1998: text, label and graphics testing
  
• Summative evaluation, 1998
  

These files have now posted on my wiki at a new page called Evaluation reports. When reading them you need to keep in mind the time they were conducted. I think many of the topics and understandings that we tested at that time would be vastly different now. For example, the issue of climate change was not really even thought about then. We have done work on climate change which I have blogged about here, so I would also reference that work when you are looking at biodiversity.

There is also a project called Who Cares about the Environment? run by the NSW Department of Environment and Conservation that has been tracking attitudes to, knowledge of and skills and behaviour of adults regarding the environment over many years and is worth checking out. I'm sure there have been studies of this kind in the UK too. The Pew Internet/Exploratorium report The Internet as a Resource for News and Information about Science is also relevant as it showed that 20% Americans used the internet to find out information about science, including environment and climate change issues (second only to television at 41%) and this has implications for the conjunction of physical exhibitions and what museums provide on the web I believe.

Good luck.

Evaluating online audiences

This question from Martin Bazley and Nicky Boyd, London via the Visitor Studies Group email alert: We are doing some research into online audiences for the London Museums Hub. Have you done any evaluation of your online audiences, or do you know of any other data or research? They also detail a range of other questions around audience and methodology.

Martin and Nicky, this is a really growing field of evaluation and we have been active in this area over the past year or so. As part of her internship here, Mel Broe did literature review of online audiences and evaluation which she posted here on my blog. I also posted about a study I did with teachers and students and the web which has a series of extra comments and links. I also presented a paper with some very initial work on digital audiences and museums to a bunch of interesting folks in Tubingen, Germany, just last week. I posted that paper on my wiki (go to the second one). I am developing this paper more as we do further research with online audiences aged 18-30 (which is being presented to us today) and teenagers (via an e-kids' college we are running here in two weeks time). I promise to blog about these projects further.

I also suggest you go to the Museums and Web search papers site and have a look there.

One (of the many) outstanding evaluators in Australia, Carolyn Meehan from Museum Victoria in Melbourne, has done very interesting online work and you could email her.

Lifelong Learning Symposium

What a full day of fascinating papers delivered by passionate people! I took a wad of notes, so will report here on the main points as I saw them.
David's paper was (as always) a broad sweep of the intellectual and the practical. He suggested that we need to tackle barriers of poverty, education disadvantage, disabilities and other issues facing older peole. He reminded us that museums can still be seen as elitist places and that we need to cater for all audiences and not spend too much time always segmenting.
The thing I enjoyed abut Marie's talk was the emphasis on learning as pleasure, fun and enjoyment - words we often forget to relate to musuems! The NGI is running some amazing programs, not just for older audiences - one I think is really innovative is the young mothers tours.
Des's talk ranged widely, reminding us that often with ageing we focus on the negative rather than the positive. The ageing population shouldn't be seen as a problem to be solved but as a "demographic bounty" to be used. He cited many, many examples of creativity in older life - apart from artists and classical composers we also saw images of Mick Jagger and Paul McCartney! The three aspects regarding ageing he mentioned were:

• the positives - the creativity, wisdom and life experience of older people
  
• inter-individual variability and the need to have a wide palette - all older people are not the same and there is more variation between two 80 year olds as between two 40 year olds
• reduced reserve - therefore providing spaces to relax, recharge and contemplate are required
  

He reminded us that the majority of older people are fit, heatlty and contributing, however those that are sick do tend to have more complex health issues. He also gave us a nice quote (didn't get the author but worth reproducing): "If you design for the old you include the young, if you design for the young you exclude the old". The overall message I got was to have a complex palette on offer and design for all.
Catherine's talk on vounteering was full of information. The remarks she made about the 21st century volunteer were very salient - that they are looking for an experience and added-value and that there are so many more opportunities to volunteer now (I have seen the increase of volunteer tourism for example). She challenged us to remember that volunteers are changing so are we trying to fit them into a 20th century volunteer model or are we trying to change?
There were other papers and a great discussion at the end - too much to digest now, but overall a great success I feel.
Just a reminder that my paper is here

Lifelong Learning Conference Roundtable

Attended this session yesterday and found it really inspiring. There certainly is a lot of innovative work and thinking going on in Ireland about all audiences, not just older ones.
There was quite a focus on reminiscence, which is something that really interests me as it isn't dealt with in Australia at all I feel, although on reflection some of the digital stories we are doing with Indigenous communities and objects is getting there. The work of the Age Exchange in London was mentioned as a good resource, particularly for developing reminiscence kits.
Another thing that struck me was the work at the Ulster Museum and the Ireland Museum of Modern Art where they used older groups to actually produce product for them in terms of the reminiscence boxes at Ulster and various programming initiatives at IMMA. The poetry/writers group that uses the National Gallery of Ireland was also very inspiring, and often brought a tear to the eye (actually everything about the Gallery is incredibly impressive and has given me a lot to think about in terms of programming at my Museum).
Seems that collaboration is the key, while also considering (and planning for) sustainabilty after grant fudidng finishes. One of the challenges raised was that museums need to develop outreach programs for a variety of community groups, not just a traditional focus on schools. I think there is a lesson there for our Museum in a Box program - some of the potential audiences discussed at the roundtable were offenders; those with dementia; ethnic minorities; users of mental health services; those that are housebound (not just the elderly or with disabilities - think young mums) and intergenerational groups.
All in all a fine day, topped with a wonderful dinner in a beautiful part of the gallery.

Knowledge Media Research Centre, Tubingen, Germany

Spending some time learning about the innovative projects that are happening at the Knowledge Media Research Centre. One that I hadn't heard about was the project being undertaken by Lars, who is using video technologies for students to record what they are learning. He is also developing a electronic tracking system which I think is really exciting – it allows people who are tracking/observing visitors in exhibitions to use a PDA and set of icons/symbols that they click on when visitors are doing something (e.g. pointing, reading, talking, etc). These are also time-stamped so you can record what they are doing when and for how long. Lars is using open source software in designing the tracking system and hopes to make it available for others to test and give feedback.

Spoke with Daniel who is looking at mobile technologies and museums. He has co-authored a paper which has promised to send me and I'll post the link. Eva also mentioned a good paper from the ICHIM conference that summarised what museums were doing in the areas of handheld technologies. I think it's this paper, although there are some other good papers on this website! Also met Eva who is working on this project, as well as Christine doing work on discussion kiosks and controversial topics; and Mika who is working on film and inferences. Mika isn't part of the museum component of the KMRC but I still think what she's doing has relevance for us.

It's interesting as they have a whole lab here where they can set up experiments using small exhibitions and testing out technological solutions to measuring what visitors learn from an exhibition. I tried out the i-tracker which was this cool set of glasses with built-in camera that records what the visitor is looking at. Used in conjunction with other tools, such as observation and interviews, is a potential way to do some evaluation. They have only tried it a few times however, and seems to me a neat idea but I'm unsure of the practicalities of it (let alone the cost).

Overall a really useful visit and hope to have more to do with these researchers (plus come back to the lovely city of Tubingen).

Presentations: Tubingen (Germany) and National Gallery of Ireland (Dublin)

Am currently in transit to Germany (via Ireland!) to visit colleagues at the Knowledge Media Research Centre, who are undertaking a major research project: Learning in museums: the role of media for the recontextualisation of objects. The project ‘... aims to explore the potential of digital media for learning in natural science museums and exhibitions’. They are holding their first workshop later this week called Learning in Museums: International Positions. I’m giving a pre-workshop lecture called What does learning mean in museums and the digital environment? I’ll be presenting data from my thesis and some initial work I’ve been doing on “ladders of participation” and what Australians are doing online.
Will also be a good opportunity to find out about their progress on their many projects, which have synergy with the work we are doing under both the New Literacy, New Audiences project and our Social Media in Museums ARC grant.
On Wednesday I travel back to Dublin to be a keynote speaker at the National Gallery of Ireland’s Education Symposium Museums, Galleries and Lifelong Learning: are museums doing enough to attract older audiences?
Both my papers have been uploaded on my wiki at this link: audience-research » Conference papers.
It’s a cold and rainy day here in Dublin as I’m waiting for my flight to Frankfurt (after a long flite from Sydney...). Looking forward to a hot shower!

Young people and museums

Came across a query about young people and museums through Facebook of all places! Have done quite a few studies with this group which are published here on my website. An interesting study we did several years ago now was with Indigenous young people and museums – that report can be downloaded here. One of the key outcomes from this for me was that young people wanted to engage with museums on deeper levels than just as visitors. They wanted to work with the collections, work in the public program areas developing and running programs for all audiences and even as meet and greet people! One of their major bugbears was not seeing people like them working in museums, therefore felt unwelcome and alienated.

I also conducted several studies with Professor Susan Groundwater-Smith called the Museum I'd like. Happy to send on any information to interested people. Now that I've thought of it again will experiment with positing examples of their work to Flickr at some point. We are repeating this exercise this year with students aged around 12-18 years and their use of digital technologies.

I also think we need to keep abreast of the kinds of studies happening around teens and the internet, as the ways they engage with content, their peers and "formal" institutions will radically change based on this new form of learning. The blog for the New Literacy, New Audiences project contains many posts on this topic. Demos also did an important paper on this issue which can be found here: Demos | Publications | Their Space. I think that report is essential reading.

I must admit though getting tired of the focus on young people when it is older audiences that are increasing and should be the focus in the future in my view! Here's the report of the Older audiences and museums study we did in 2002. I'm giving a paper on this very topic at the Lifelong Learning in Museums and Galleries conference at the National Gallery of Ireland next week. Watch this space as I post my impressions as well as the paper I'm giving.