Case Study: Public Involvement Evaluation: Brevard Metropolitan Planning Organization (Viera, Florida)

Back to Case Studies

Understanding the Purpose Upfront

photo: people sitting around tables discussing transportation planning

OVERVIEW

In November 2000, the Brevard Metropolitan Planning Organization (MPO) adopted a new Public Involvement Plan (PIP) and Evaluation Handbook. The PIP provides the policy to support evaluation and details the full complement of public involvement techniques and their application. The Evaluation Handbook takes this one step farther by delineating evaluation criteria, performance goals and methods to meet each goal for all of the techniques outlined in the PIP. By developing these documents together, the agency has created an effective framework to simultaneously conduct, evaluate and refine its public involvement policy and techniques.

The impetus for revising the PIP and creating the Handbook resulted largely from the failure of a consultant-led study to garner adequate public participation in the recommendations being proposed for an area transportation study. (This was due in part from confusion regarding the area of the study.)  Intent to build consensus, the MPO initiated a second round of agency-led outreach.

To help ensure the success of the second round of outreach for the area study and success in understanding and incorporating public concerns in future planning efforts, the MPO conducted an evaluation of the outreach activities associated with the first round of outreach for the Study.

This evaluation served as a foundation to the revision of the PIP and the development of the Evaluation Handbook. The evaluation is an integral part of both of these documents, and evaluations will be used to guide and assess the effectiveness of future outreach initiatives. The PIP and the Handbook are viewed as "living documents" that will continue to be refined. Both the current PIP and Evaluation Handbook can be accessed online. (Public Involvement).

WHAT HAPPENED

In 1997, the Brevard MPO staff began informally monitoring the agency's outreach activities. Then, in 1998, a consultant team working on the Northwest Palm Bay Transportation Study (NWPBTS), recommended to the MPO Board a series of actions to improve mobility and increase roadway capacity in and around the City of Palm Bay. The Study was not endorsed due to citizen opposition. Issues ranged from criticism of the individual changes (there were approximately 15 under consideration) to concern that the outreach efforts had not focused on the entire study area, including areas outside Palm Bay that could see significant impacts from the recommendations.

Like previous MPO studies, the project consultants had relied on a number of different methods to inform and engage the public. At the outset, a Citizen's Advisory Committee and Technical Advisory Committee were formed to guide the project. Initial stakeholder interviews were conducted, local businesses and property owners were notified, an interactive project web site was developed, project specific articles were placed in local publications, a newsletter was created, and a toll-free hotline was established.

To publicize the public meetings, press releases were sent to the local media outlets and meeting information was included in the project newsletter (with a distribution of about 23,000) and posted on the project web site. Over the course of the study, four public workshops and two neighborhood meetings were held to coincide with project milestones. More than 600 people attended these meetings and thousands of comments were recorded. Despite these efforts, it was evident that a portion of the public did not support in the proposed recommendations.

One issue that emerged as the recommendations were being developed was the concern that public outreach had not involved people throughout the entire study area. Because the study was titled the Northwest Palm Bay Transportation Study, there was confusion on the part of the public regarding who would be impacted by the study. Based on the name of the study, many people had made the assumption that the improvements being considered pertained only to the City of Palm Bay and had not participated in the meetings. In truth, the study extended beyond city limits. As such, recommendations were made that might potentially impact people that had exercised little voice in defining the improvements.

Without clear public support, and concerned about the perceived lack of inclusiveness of the process, the MPO Board postponed voting on the recommendations without further review. Initially, the Board requested an intensive, agency-led review of all of the public input recorded on the project. Once this was completed, the agency set up a series of three technical meetings to further familiarize the Board with the recommendations.

At the completion of these meetings, the Board requested that a public hearing be scheduled to solicit public comment.

At the hearing it became apparent that while citizens saw the need for improvements, they preferred options that went through someone else's backyard, and points of consensus were not readily apparent. Based on the input received at this meeting, the Board further postponed voting on the recommendations in favor of soliciting public comment on each and every alternative before voting individually on the options.

The MPO staff conducted six (6) public workshops over an eight-month (8) period. More than 1,500 individuals participated in these meetings. The result:  greater and more effective public participation from throughout the Study area.

Based on feedback from the initial round of public involvement, two changes were made to the public involvement process in the second round of agency-led outreach. First, the study was renamed the Southwest Brevard Transportation Study (SWBTS) to be more reflective of the area being considered for improvements. Second, rather than relying on publicizing public meetings in a newsletter or on the web, the MPO direct mailed meeting notices to over 26,000 individuals. Upon examination, it was revealed that the consultant's mailing list for the first round of outreach did not include all of the study area. As a result, an additional 3,000 names were added in the second round.

The first round of public involvement for the Study area was lead by a contractor and totaled about 10 percent of the contractor budget. The second round of public involvement was conducted entirely by MPO staff. Estimated cost to the MPO for the follow-on analysis and second round of meetings:  one and a half times the MPO's annual public involvement budget. Estimated staff time expended:  1,643 hours. The additional cost and time included the evaluation, which resulted in the revised PIP and Evaluation Handbook.

The need to conduct a second round of outreach provided the impetus for the agency to formalize its evaluation process and update its PIP. Before undertaking a second round of public involvement, however, the MPO conducted an evaluation of the Study outreach activities to gain insight into the agency's strengths and weaknesses in working with the public.

To gauge the effectiveness of the second round of outreach, the agency conducted both an external and internal evaluation of the process. External evaluation entailed conducting a five-minute telephone survey with the approximately 1,500 participants from the first round of meetings. The survey was developed to solicit their opinions on the techniques employed, the materials presented and the inclusiveness of the process. The response rate was approximately 30 percent of the target audience (about 500 people) with the following results:

  • Direct mailings were regarded to be the most successful means of meeting notification. (Measuring the success of this technique was particularly important given the budget expended to mail 26,000 notices)
  • The majority of those surveyed expressed the opinion that the information conveyed at the meetings was clear and informative and that the opportunities provided for public involvement were "just right" (from the choices "too many," "too few" and "just right")
  • The majority of those surveyed felt that their concerns were adequately conveyed to decision-makers (The MPO Board/DOT)
  • Most respondents indicated that they did not use or did not know about the local project hotline and the project web site that had been established by the consultant in the first round of outreach and continued to be employed in the second round
  • Several citizens expressed concern about the accuracy of coverage provided by the local newspaper
  • The best source of technical information about the project was considered to be public meetings followed by contacting MPO staff members. The Internet and project newsletters rated the lowest.
  • Participants overwhelmingly responded that they felt they understood the overall purpose of the study

Equally interested in the perceptions of the internal audience (the MPO staff, local governments involved in the study, and MPO Board members), the MPO staff prepared a written survey to solicit their input on the public involvement process, the project information provided, and the activities of the study committee. Additionally, the survey asked participants to identify areas of improvement and lessons that could be applied to future studies. The survey was sent to the MPO board and 10 MPO and local agency staff members who were involved in the project. The response rate was approximately fifty percent of the target audience with the following results:

  • The majority of the respondents agreed that citizens had received ample notification of meetings, had been provided with adequate project information, had ample opportunity to comment and were successful in conveying their concerns
  • Several respondents supported the continued use of direct mailings to notify the public of upcoming meetings
  • The majority of those surveyed indicated that meeting procedures, including scheduling, agenda packages, and the elimination of alternatives were appropriate and that the information provided to stakeholders was of the right amount and type
  • Most respondents agreed that having the study consultant available during the second round of outreach during decision-making process would have been beneficial. (The consultant's contract had expired by the time the second round of outreach was conducted.)
  • Many respondents expressed frustration about the length of time it took to reach a decision (eight months for the second round of outreach).
  • Upon examination of the general trends identified in both the external and internal surveys and a review of the consultant study process, the MPO staff made the following recommendations for implementation on future planning studies:
  • Notification about the study should be mailed directly to all current residents and property owners that may be affected by proposed alternatives
  • To be cost effective, the project hotline and web-site should be publicized more and the web-site should be kept up-to-date in order to be useful to the public
  • More frequent coverage should be sought from the local media. Staff should increase efforts to ensure that accurate information is published in local publications
  • The length of consultant contracts should be such that adequate time is allowed for review and refinement of the technical recommendations being proposed to maintain continuity and provide citizens and decision-makers with the best technical information available

Internal Audience Written Survey

The input received from both the external and internal surveys, a review of the consultant study process and an informal assessment of the MPO's prior public involvement activities, informed the revision of the PIP and the development of the Evaluation Handbook.

LESSON 1:  Understand The Purpose of Public Involvement Upfront

photo: group of people sitting in audotorium with another group standing in the back

Faced with the challenge of delivering projects on time and on budget while addressing the needs and concerns of the public is no small task. Agencies are often accused of not adequately soliciting the advice of the citizenry in the planning process.   Too often, agencies have relied on a set menu of activities to interact with citizens with little emphasis on the effectiveness of these techniques in a given circumstance. As such, additional resources may be deployed to expand the agency's ability to effectively engage the public in the decision-making process. Without policy and guidelines to direct these efforts, however, there is little information to substantiate an increase in efforts and guide their potential for success.   This can be costly, both in terms of budget and goodwill from the public.

The Brevard MPO increasingly found itself asking consultants to provide a lot of public involvement "extras" (e.g., expand mailing lists, create newsletters, provide project web sites) without having either the formal policy or evaluation criteria to justify these requests and expenditures. The current PIP and Evaluation Handbook provides this guidance and positions the agency to be able to continually assess and refine its approach to better meet the needs of the public. This in turn will enable the agency to more effectively target its resources. The alternative can be costly and result in funding activities that do not contribute to engaging the public effectively.

LESSON 2:  FIND OUT WHAT WORKS AND WHAT DOESN'T:   REVIEW OF A PROCESS

When faced with the task of evaluating the public involvement efforts during the Southwest Brevard Transportation Study, we were quite apprehensive about calling citizens at home. We were fearful that those citizens that were not pleased with the outcome of the study would not be objective when answering our questions. However, we were very pleasantly surprised when an overwhelming majority of the people that we contacted were simply thrilled that we were interested in using their input to help us to refine our process to better meet their needs.

--Kama Dobbs, transportation planner, Brevard MPO

By listening carefully to the feedback provided on the first round of outreach and conducting the telephone survey with citizens that had participated in the second round, the agency obtained valuable information about its outreach process.

In the first round, the name of the study had caused confusion about who would and would not be impacted and limited participation. Additionally, the consultant's mailing list for the project newsletter did not accurately reflect the entire study area. By listening to the public's concerns, the MPO was able to modify the study name, mailing list, and means of meeting notification before launching the second round of outreach.

At the completion of the second round of outreach, the MPO once again turned their ear to the public by conducting a telephone survey of those who had participated in the process. Based on the level of workshop participation, the comprehension level of the attendees, and the success of the direct mailings to notify stakeholders about the meetings, it became evident that the MPO had been successful in notifying stakeholders and conveying project information.

By carefully reviewing what had been done in the first round, and making modifications in the second round, the agency was successful in identifying public concerns and responding to them in the recommendations.

It should be noted that the phone survey was subjective in that it included only those citizens that had participated in the workshops. Also, certain groups will not be represented in any evaluation and, therefore, there exists a need for agencies to continually vary their sample when assessing their outreach activities. This might be done by conducting a random survey or including comment cards in all agency mailings and publications to solicit feedback from those who have not yet had the opportunity to participate in the decision making or who have not had the time to attend public meetings.

Citizen Telephone Survey

LESSON 3:  DESIGN A FRAMEWORK TO SUPPORT EFFECTIVE EVALUATION

The MPO shall strive to continuously improve public involvement. The MPO shall continuously evaluate public involvement techniques, according to the procedures contained in the Brevard MPO Public Involvement Evaluation Handbook.

-- Brevard MPO Public Involvement Plan (2000)

As the staff of the MPO began working on the Evaluation Handbook, it became apparent that the 1995 PIP did not provide the appropriate policy language to support evaluation and needed to be revised.

The recently completed PIP includes language specific to the evaluation of public involvement techniques based on the procedures outlined in the Evaluation Handbook and calls for a review of both the PIP and the Handbook at least every three years. By developing the PIP and Evaluation Handbook as companion pieces, the MPO has established an effective framework to conduct, evaluate and refine its public involvement policy and techniques on a regular basis. While the PIP provides the goals, policies and objectives of the MPO, as well as a list of public involvement techniques and forums, the Handbook details the actions required to evaluate the effectiveness of the techniques identified in the PIP.

The Handbook was developed over a period of a year and a half during which it was continually refined to reflect the activities of the MPO. Included in the Handbook is a Public Involvement Tools Evaluation Matrix, which lists specific techniques/tools, evaluation criteria, performance goals and methods to meet those goals.   Developed by the MPO staff, the evaluation criteria include indicators such as the number of calls made and letters sent to the public for hearings, open houses and workshops and the number of people who respond to direct mailings and newsletters. (Similar criteria could be applied to response from press releases, newspaper articles and editorials.)  The performance goals detail target levels of performance such as response by three to five percent of the affected population from a study area.

Assuring that the full range of issues raised by the public are identified and that they are reflected in decisions should be the ultimate goal. It is important that measures are developed to support Agency goals for effective public involvement and not be based solely on what is easy to measure.

Evaluation Matrix

LESSON 5:  DEFINE METHODS OF EVALUATION

Evaluating public involvement means assessing how well we communicate with our customers. We have three basic customer groups - elected officials, other agency staff and the public at large. Each has different expectations, information needs, roles and levels of understanding. Different evaluation tools are needed for each group.

--Bob Kamm, staff director, Brevard MPO

Once tools/techniques, evaluation criteria and performance goals have been identified, the instruments for both internal and external evaluation must be developed.   

INTERNAL

The Brevard MPO has developed a general evaluation form to be completed by agency staff each time an evaluation is conducted that specifies the type of study, the point at which the evaluation was conducted, the public involvement tools employed, the target audience and the type of evaluation conducted. A similar form is also used for project specific evaluations. Complementing these forms is an Improvement Strategies Form that has been developed to provide practitioners with the opportunity to provide input on potential improvements. Upon completion, these forms will be reviewed by the MPO staff and, where applicable, forwarded to the Florida Department of Transportation (FDOT). Copies of these evaluation forms will also be provided to the Technical Committee and the Citizen's Advisory Committee (advisory committees to the MPO Board) for review and comment with the aim of keeping staff and key stakeholders apprised of evaluation efforts and areas for improvement.

Recognizing the role of consultants in a number of its studies, the MPO has taken internal review one step farther by requiring consultants to provide an assessment of the public involvement process (what worked, what didn't, what they would do differently) as part of their final project write-up.

General Evaluation Form and Improvement Strategies

EXTERNAL

Once we began thinking about qualitative measures, rather than quantitative, our evaluations became much easier to develop. Answers to "How can we improve the workshop?" give us better guidance than will simply counting the number of people that show up.

--Bob Kamm, staff director, Brevard MPO

Externally, the agency has developed a workshop evaluation form that asks participants how they found out about the meeting, whether they felt the information provided was clear and informative, what their best source of information has been and how they would rate the public involvement process.

It also is important to find out the concerns of people who have not participated and mechanisms should be developed to determine who these people are and the reasons they are not involved.

LESSON 6:  USE THE EVALUATION INFORMATION TO MAKE REFINEMENTS

What happens once all of this information has been collected?  First and foremost a central point must be established to house the information collected. At this time, the Brevard MPO maintains binders of all evaluations conducted to date. It is anticipated, however, that this information will eventually be entered into a database. This would enable the agency to "group" findings in a manageable fashion where data could then be printed out, reviewed, debated and refined.

LESSON 7:  TRACK COSTS CAREFULLY

The Brevard MPO estimates that fifteen percent (15%) of its annual budget is expended for public involvement activities (staff time and direct costs). Averaged out, twenty to twenty-five percent (20-25%) of the agency's staff time is dedicated to outreach activities. Consultant costs for public involvement are estimated at ten percent (10%)

of the study contract. As the need for public involvement in transportation planning increases, so too will these numbers. Therefore it is more important than ever to carefully track staff time, direct costs (e.g., newsletter production and mailing, web site development and maintenance) and consultant costs associated with these efforts both in terms of measuring success/failure against expenditure as well as to quantify the "savings" of eliminating a particular approach so that these funds can be applied to other activities.

LESSON 8:  TAILOR PUBLIC INVOLVEMENT TO LOCAL NEEDS

The work done by the Brevard MPO provides a starting point for any agency intent on evaluating its public involvement process and outcomes. Recognizing that each agency needs to tailor its individual outreach activities, it is often helpful to review what other agencies have done as a means of comparison.

In the fall of 1999, members of the MPO staff attended a Public Involvement Training Course sponsored by the Florida Department of Transportation where they shared public involvement techniques with other MPO's in Florida. After completing the training, the staff decided to document some of the "best practices" identified so that they would have a list of public involvement "tools" to choose from for individual projects as well as everyday MPO activities. These tools are included in the PIP.

Brevard's Public Involvement Plan provides a comprehensive list of public involvement techniques and indicates when these techniques should be employed. Appendices to the Plan provide detailed examples of each of these tools, ranging from surveys, comment forms, and direct mail pieces to web site and database visuals.

The Evaluation Handbook is useful in the provision of both internal and external evaluation forms as well as providing detailed matrices against which these activities can be measured. Both of these documents provide an excellent starting point for practitioners intent on evaluating their public involvement activities.

CHALLENGES AHEAD

Public participation by definition is an open, ongoing, two-way communication, reflecting the local community of people and issues. It cannot be structured to any single right model or measured against any one size fits all criteria.

--Public Involvement Desk Reference, U.S. Department of Energy

Brevard is about to conduct its first evaluation since the adoption of the PIP and the Evaluation Handbook on the public involvement activities associated with the agency's Greenways and Trails Master Plan.

While Brevard has made great strides in the compilation of the Evaluation Handbook, the agency recognizes that it will need to be vigilant in continuing to assess what works and what doesn't. More importantly, to be successful in the eyes of the public, the MPO will need to be able to translate information into action.

The MPO identified three areas that might impede the agency's ability to effectively respond to the needs of the public in the future:

  1. Budget. As public involvement "add ons" are identified (e.g. the need for more meetings, direct mailings, etc.) the agency will need to be able to absorb the direct costs for these activities including staff time required to either implement or oversee these initiatives. Additionally, conducting proper evaluations to assess the effectiveness of these activities will also require an ongoing commitment of time and budget.
  2. Knowledge. Based on changing technologies (e.g., the Internet, video imaging) and the public's interest in receiving more "user-friendly" information, the agency will need to evaluate staff skills to meet these needs, provide training, if necessary, or rely on outside contractors to provide these services.
  3. Institutional Support. An evaluation of the Southwest Brevard Transportation Study revealed the effectiveness of direct mail to notify the public about upcoming meetings. Given the cost to mail thousands of notices, it will be important for the MPO to secure the support of its funding agencies to justify such expenditure for future projects.

Another challenge is identifying people who have not been involved and understanding why they have not. Are there specific obstacles that limit their interaction with the MPO?  Do they have concerns that are not being articulated?  Public agencies have the responsibility to serve all people so it is important that some people are not left out of the process unintentionally.

By revising the PIP to support evaluation and developing an Evaluation Handbook to provide guidance on the actions required to evaluate the effectiveness of the public involvement techniques identified in the PIP, the MPO has made a strong start to enhance its relationship with the public. The challenge will be in continuing to find and refine cost effective means to assure that the issues and concerns from all members of the public are identified and reflected in the transportation decision-making process.

Contact:

Kama Dobbs
Transportation Planner
Brevard Metropolitan Planning Organization
2725 Judge Fran Jamieson Way, Building A
Viera, FL  32940
Phone:  (321)690-6890
E-mail:  kamad@brevardmpo.com

The Brevard Metropolitan Planning Organization (MPO) is the agency designated by state and federal law responsible for the urban transportation planning and programming process for the Palm Bay-Melbourne-Titusville Urbanized Area. The jurisdiction of the MPO extends to the boundaries of Brevard County Florida, which is located along the east coast of the Central Florida region. Brevard County is approximately 20 miles wide, east to west and 72 miles long, north to south and is bordered by the Atlantic Ocean to the east and the St. Johns River to the west. Brevard County is adjacent to Indian River, Orange, Osceola, Seminole and Volusia Counties and located approximately halfway between Jacksonville and Miami and about 40 miles east of the greater Orlando area. NASA's Kennedy Space Center, Cape Canaveral Air Force Station, Patrick Air Force Base and the Port of Canaveral are all located along the east coast of Brevard County.

The estimated 2000 population of Brevard County is 483,000, with just over 27,000 residing in Melbourne and Palm Bay. Close to 87,000 (18% ) of Brevard County residents are over the age of 65. In 1990, approximately 90% of Brevard residents were white, 8% were black, with the other 2% being of asian, hispanic or other descent. The 1999 unemployment rate in Brevard was 4%.

Approximately 55% of all land in Brevard County is undeveloped.

map: Outline map of Florida showing county lines including a closeup of Brevard County, Florida