survey98
The Walks: 1998 Survey
This page describes problems with the initial survey, corrects a results table and concludes that the bid is damaged.
There have been two surveys, both included in stage one of the Walks project bid. This page only considers the first survey from 1998. Details of both are in Appendix 5 (Appendix volume C) of the phase one bid, edited by Jeff Clarke, Acting Head of Policy and Environment Division, BCKLWN Planning Department. One can view this in the planning department reception, Juniper House, Austin Street, King's Lynn.
Survey Design
I have reproduced the survey form as part of another project. This questionnaire was produced as an A4 3-fold leaflet, distributed from an exhibition in November and December 1998, placed on the council web site, and a newspaper coupon, printed in the Lynn News on 4 December 1998. The exhibition was housed in a portacabin at different locations in the Walks open 10am-3pm for 2 days, then in the town centre open 10-3 for 2 days, and finally at the library and the council offices for about a week and a half each, open longer hours.
The Problems
The problems can be gathered under the three headings of collection, analysis and use.
Data Collection
Time of year
The survey was launched near the end of November and finished shortly before Christmas. I would expect the Walks to be used less in winter. No data on user numbers was collected, so it is not possible to verify the level of use. Other data suggests that user numbers are greatly reduced by poor weather. The 2000 user survey saw a 25% reduction on a wet Wednesday compared to a sunny one.
I also expect more people to be busy with Christmas-related activities in December and less willing to spend time completing a questionnaire.
Self-selecting sample
Only visitors to the exhibition who choose to fill in a leaflet and newspaper readers who send in the coupon are included. This will over-represent people with a particular special interest in the survey. Best practice would suggest conducting interviews in the Walks according to a sampling plan, but maybe this was considered too expensive for a preliminary consultation.
Literacy and language
This is a written questionnaire covering a side of A4. As far as I can tell, the questionnaire was only available in English and in writing, neither of which are really necessary to use the Walks.
Venue Effect
Apparently it was not noted which venue submitted questionnaires were taken from. Without this, we cannot compare responses from people in the Walks, responses from council office visitors and responses from council web site visitors.
Exhibition Display
One fifth of the exhibition panels was devoted to "historical development". There was a computer-based presentation which emphasised "remembering its past". There was also a leaflet about the history of the Walks distributed from the same exhibition as the questionnaire. The reverse of the questionnaire said "Don't forget to send us your memories of the Walks or any old photographs you may have of the area." This would probably influence the questionnaire responses.
Media coverage
Before the consultation, there was coverage in the Lynn News in February 1998 which emphasised the history of the walks. There was coverage of the survey in newspapers during the period, including the Lynn News coupon and articles in the EDP. In the EDP on 27 November 1998, the council's James Cooper mentioned a "desire to see a more period feel." The questionnaires were not dated, so we cannot investigate the effect of these comments.
Staffing
The exhibition was staffed while in the Walks and unstaffed at the library and council offices. It is not stated what briefing the exhibition staff were given about the survey, if any, nor whether they were allowed to assist respondants with the questionnaire. Because no venue data is given, it is impossible to compare the staffed and unstaffed venues.
Methodology questions
Some of the detail appears to have been removed from the final report. The consultation aims to gather views from two "customer groups". The first is regular users of the walks, but the second is never stated.
Very little rationale is given for the chosen methodology. Why was a non- random sample used? Why were large open questions asked? Who is the target population for the survey? The personnel responsible for the project has changed since this survey was conducted and they did not answer these questions.
Data Analysis
Graphing Errors
Pie charts are used to present the reasons for visit, one for each different visit frequency class. Four identically-sized charts are used and no numbers are marked on them. It is impossible to tell by looking, but the fourth chart represents only about a third as many people as the other three.
Transcription Errors (1)
The headings on the results sheet are different to the questions asked on the questionnaire. For example, "What for" (as in "Do you use the Walks? What for?") becomes "Why".
Coding question
Not all of the questions are represented on the results sheet. I have asked for the full results sheet and raw data, but I have not received them yet. Without this, we cannot verify the coding used for the likes, dislikes and comments.
Grouping question
The third-placed heading is "boundary definitions, landscaping, maintenance, enhancement" which is very broad and contains four items. Most headings are a single topic or two topics and the two that contain three topics have two synonymous ones. This heading has four different topics.
If it was broken into four headings, "boundary definitions" would have 10 comments (placing it 18th), "landscaping" 3 (23rd), "maintenance" 12 (16th) and "enhancement" 8 (19th). It is arguable that comments about maintenance should be included in the "care for it" heading and that fountain enhancements should be included under the "water" heading. Why was this broad group heading used?
Calculation Errors
The analyst claims "by combining the likes, dislikes and comments it is possible to derive ranked themes". The full analysis is contained in Table 6, pages 34-39. However, Table 6 only combines dislikes and comments (D and C code responses) and ignores likes (L codes). This changed the results significantly (see corrected table below).
Transcription Errors (2)
Many of the headings in Table 6 are different to those that appear in Table 4. While both are arbitrary choices of the analyst, it counts requests for more paths towards "The need to drain the Recreation Ground in winter" and similar strange outcomes.
Transcription Errors (3)
There are 28 headings in Table 6. There are 27 headings in Table 4.
Biases
While the analyst suspects a campaign for recreation facilities, no reason is offered for this claim. The effects of literacy and self-selection are not noted. The effect of weather is not included, despite some obvious seasonal comments. Also, the effect of the media coverage and exhibition is not mentioned, despite very similar views appearing.
The analyst does correctly identify one aspect of the venue effect, but does not note more general implications of this.
Use of the Survey
Follow-up work
The title of the appendix containing this survey was "Preliminary Consultation", but little work was based on these results. The 2000 survey merely investigated whether opinion had changed and did not seek to collect new data on the themes raised.
Publicity material
The incorrect table 4 was published in a special supplement to the Norfolk Citizen in May 2000.
Ignoring the results
The analysis of this survey tries to justify ignoring some of the responses, while still combining them with other responses. This is very unusual.
Different purpose
This "Preliminary Consultation" survey is the only original quantitative research of public opinion about the Walks bid submitted to the Heritage Lottery Fund. Despite this, the council claims "broad consensus" for their plan (developed in 2003/4, long after this survey) and asserts that it is "a relatively small body of public opinion" which is unhappy. If these claims are based on this survey, they are not justified.
Corrections
It is impossible to correct many errors in the analysis, because the data is not available. Any corrections have to trust that the analyst's coding of answers and totalling is accurate. By doing that, it is possible to calculate a corrected table 4 from the results published in the stage one bid. It should have read:
Table 4: By combining the likes, dislikes and comments it is possible to
derive ranked themes. These are: Rank| Heading| Freq| Was
---|---|---|---
1| Don't spoil it/protect it/care for it| 81| (16th)
2| Rivers/water/associated wildlife| 64| (4th)
3| Trees| 55| (8th)
4| Requests for more recreational facilities| 47| (1st)
5| Boundary definitions, landscaping, maintenance, enhancement| 45| (3rd)
6| More safety/security| 39| (2nd)
7| Use of Walks by cyclists| 30| (6th)
8| Noise/Quiet| 24| (25th)
9| Poor quality lighting| 23| (5th)
10| Children's Play| 21| (19th)
11| Cafe/cafeteria| 20| (7th)
12=| Recreation ground/drainage| 18| (9th)
12=| Vandalism/bad behaviour| 18| (12th)
14=| Provide Park Keeper/Warden| 17| (10th)
14=| Toilets| 17| (11th)
14=| More music/entertainment| 17| (14th)
17=| Dog control/dog dirt| 15| (13th)
17=| Seating| 15| (15th)
19| Red Mount| 14| (20th)
20| Position/Location| 9| -
21| Car parking| 8| (17th)
22=| Return to period character| 2| (21st)
22=| Pedestrian Crossing Tennyson Road| 2| (23rd)
22=| No development (railway land)| 2| (24th)
22=| Tourism| 2| -
26=| KL has nothing else| 1| (26th)
26=| Provide new bridge over station| 1| (27th)
Method
Using the graphs and lists of the "Likes", it was possible to reconstruct the frequency table for L-codes and assign them to headings in table 4:
| Code | Freq | Heading |
|---|---|---|
| L1 | 29 | 16 |
| 1 | L8 | 5 |
| L14 | 6 | 3 |
| L20 | 1 | 16 |
| L26 | 1 | 19 |
| 2 | 19 | L33 |
| 16 | L39 | 1 |
It was necessary to discard L33 and heading 18 (dislike or comment on nothing) because it is not possible to know their overlap, which removes at most 9 responses. Heading 22 (graffiti) was combined into heading 12 (vandalism). Heading 8 was modified to include comments liking "quiet". L2 and L31 formed a new heading "Position/location".
Conclusion
The survey designer made some questionable design decisions. That is not unusual for a preliminary study, but the decisions and their reasons should be stated. Most importantly, why was a non-random sample used? Full details of the data collection methods should be included, to help later surveys build on the results.
The survey results are incomplete, so it is impossible to check most of the analysis. Some parts which can be checked contain errors. One table which can be corrected gives very different results to those published in newspapers and included in the bid to the Heritage Lottery Fund.
The survey is not properly evaluated. For example, there is only the briefest attempt to identify and explain sources of bias, despite the weather and promotion of historical information during the survey apparently being reflected in the responses.
Although this was called part of a preliminary consultation, no survey based on this has been included in the stage two bid. Given that lack of consultation, it is hard to tell whether public opinion supports the proposals or not. It is also difficult to explain the failure to cover popular issues from this survey, such as recreation facilties, or the inclusion of measures which will harm trees and wildlife, that seem likely to be unpopular with respondants to this survey,
While saving a small amount of money by not hiring a opinion poll specialist for stage two, this has seriously damaged the Walks bid.