Is access to public transportation New York’s most important municipal service? Is the city’s poorest performance in the area of re-entry programs for former convicts? Is the public school system still faltering?
Yes, yes, and yes, a group of community leaders from the five boroughs told Public Advocate Betsy Gotbaum at a forum on citizen satisfaction with municipal services. While these respondents are hardly a cross section of all New Yorkers, their comments on city services indicate what public officials might learn if they were to ask city residents Mayor Ed Koch’s signature question, “How’m I doin’?”
Who is the best judge of how well city government performs -- the bureaucracy responsible for providing service or the citizens who are supposed to benefit from municipal programs? Many experts agree that citizens are as qualified to evaluate city services than government itself, if not more. “The best way to encourage government performance is to measure it, and the best indicator of government performance is citizen satisfaction,” says the International City and County Management Association.
Still, in New York, most standard measures focus on government’s own definitions of success. The city measures how much money and staff time are put to use addressing problems, as well as the number and scope of new programs launched in response. It looks far less frequently at the actual results of programs or residents’ experiences with and attitudes toward those programs. While the Bloomberg administration has made great strides in performance assessment and reporting, New York City still does not regularly and systematically survey its residents’ perceptions and assessments of the services they receive.
For example, the city knows that the number of calls to the 311 emergency hotline continues to rise. It also knows how quickly the calls were answered. But the government does not have information on how many of the 40 million callers to 311 have been satisfied with the city’s response to their problem, nor does it have information on how New Yorkers who do not call 311 feel about government services.
SURVEYING THE CITIZENS
Hundreds of U.S. municipalities, among them Phoenix, Miami-Dade County, Detroit, San Diego, San Jose, Austin, and Portland, Oregon use resident surveys. Gregg Van Ryzin, a scholar who follows municipal measurement trends closely, notes that the Government Accounting Standards Board, the International County Management Association and the National Academy of Public Administration have all championed customer surveys as an index of local government performance. Locally, a February 2002 “Policy Brief” by the New York City Independent Budget Office recommended that the city “identify and report on results that matter to the public and reflect the way the public sees and uses city services.” The office observed that “public accountability requires, in part, viewing an agency from the citizens’ perspective, understanding their expectations and collecting information that helps the public assess how well government is meeting those expectations.”
But effective, accurate and policy-relevant surveys are not easy to conduct. The challenge in New York is particularly daunting because of the sheer size of the city and the difficulty of distinguishing borough and neighborhood problems from those affecting the entire city. Surveys are expensive to design and write. Those drafting the survey must work to frame questions that reflect the public concerns â€“ and not the thinking of whoever is writing the questions. A good survey must be tested before the final version is released into the field, and a statistically representative sample of respondents must be selected.
DESIGNING A SURVEY FOR NEW YORKERS
Despite such obstacles, New Yorkers may soon have a chance to grade their government. Public Advocate Betsy Gotbaum has begun an ambitious new program to gauge resident satisfaction. Working with experts from the Baruch College's School of Public Affairs, (including the two of us), she has assembled a Civic Leader Board to develop a questionnaire that will ultimately be circulated among a broadly representative sample of New Yorkers. Dubbed the Public Advocacy Project, the program will provide both an ongoing forum for community leaders and a regular gauge of residents’ satisfaction with key government services.
Late last year, more than 60 civic leaders came to Baruch College to rate city services and to help provide topics for a future survey. Drawn from community organizations throughout the five boroughs, each leader brought a different perspective to the conversation. The structure was similar to the one employed in the summer of 2002 in the Civic Alliance’s “Listening to the City” program, in which thousands of New Yorkers gathered to express their views on the rebuilding of lower Manhattan after the 9/11 attacks.
At the forum, community leaders rated 100 city services as “extremely” “very” “somewhat” or “not that” important and on whether the service provided is “excellent,” “good,” “only fair” or “poor.” Each participant was provided with a handheld voting device. While voting was anonymous, the results of every vote were immediately visible on screen, providing immediate feedback for the participants. Within two hours, we had one roster of the services participants deemed most important, and another rating the city’s performance in each area (see tables).
There was remarkable agreement on the top 10 service areas, with each being cited by at least 85 percent of the respondents. Access to public transportation ranked first, closely followed by regular trash pick up and public education.
Five focus groups then discussed the 10 areas in which city services were perceived to be most in need of repair. Along with helping former prisoners re-enter society, the groups cited long-term senior care, affordable childcare and affordable housing as needing improvement. For each topic, the groups provided a detailed breakdown of the problem; a sense of the city responses to those problems; positive elements of the city’s response; references to other responders, such as nonprofit organizations that have a good reputation for service/response in the area; and illustrations of the problem in specific neighborhoods.
Participants then came back into plenary session to hear what happened in the groups and to talk about next steps. They also had an opportunity to hear from the public advocate and to provide her with their reactions to the process.
LEARNING FROM COMMUNITY LEADERS
While the session provided valuable information about what some of New York’s most active citizens think of city services, the discussion’s core purpose was to generate and focus questions for a telephone survey of the general population. Telephone surveys cannot cover the full range of issues facing residents because they simply take too long to administer. It took a very dedicated, knowledgeable group of respondents two hours to move very quickly through our 100 topics; most surveys have to be completed within less than 25 minutes to maintain participation. By soliciting input from community leaders, we were able to identify issues that we had not thought about ourselves and frame follow-up questions around problems identified by the leaders, not by scholars or government officials.
We found several things that we had not expected. For example, the initial list of 100 topics had an item on “police/fire department response times.” The community leaders insisted that those two categories be separated on the public survey given their strong sense that police response times were inadequate, “particularly on weekends.” That concern emerged as one of the 10 most important issues facing the city.
Large categories, like “crime” need to be broken into smaller pieces to get a clearer idea of how to improve city services. For example, “safety from violent crime” was only 24th on the “extremely important” list, but three crime-related categories ranked in the top 10: police response times (fifth), drug abuse and drug dealing (ninth) and police-community relations (tenth). The discussion with community leaders will prompt an expanded treatment of crime in the broader survey.
We found several benefits to involving community leaders in the design of the survey. First, it creates a forum for civic leaders to express their opinions to public officials. Second, it enhances those leaders’ roles as advocates, strengthening community organizations and improving their ability to speak on behalf of the residents they serve. Third, the rich detail and extensive examples elicited in the focus groups provided a wealth of material for inclusion on surveys and illustrative stories about the way that residents live with problems and the city’s efforts to solve them. For example, one discussant said of police response times, “They’re fine Monday through Friday, but terrible on the weekends.” Many people talked about a lack of information -- on school bus schedules, on sanitation schedules, on street repair, and other vital services. These specific cases will help sharpen questions for the survey.
The final version of the survey will be fielded via telephone to a representative sample of New Yorkers in the fall of 2007. The findings will be circulated widely, discussed in depth with the community leaders, and repeated, with emphases on different issues, as the survey evolves in subsequent years.
These citizen surveys will provide a robust set of data with which to approach elected officials and the leaders of city agencies. It will offer an important corrective to the “complaints-only” data available from 311 and move New York closer to a genuine measure of service outcomes. City officials will finally know â€“ from New Yorkers themselves â€“ how they’re doin’.Â