Grading the stations on cleanliness, security, announcements, crowding and ease of getting around, the riders gave their top score to the Yankee Stadium/161st Street station in the traditionally underserved Bronx â€“ despite renovation problems and complaints about rats. The 51st Street and Lexington Station came in dead last in the 2000 survey, even though in many respects it looked better than its Bronx counterpart. A possible conclusion: When it comes to rating stations, many riders would prefer vermin to overcrowding.
Citizens often have different criteria for rating services than experts or officials. Health administrators may evaluate a hospital based on the number of prestigious doctors or the amount of cutting edge equipment. Patients, though, tend to look at whether the triage nurse in the emergency room treated them with courtesy and respect. In San Jose, the heart of the high-technology Silicon Valley, city officials expected residents would prefer to get key information on public services via their computers. No, residents said in a survey: They wanted to speak to a “real” person on the phone.
As governments, including New York City’s, find new ways to assess their performance (for more, see Gotham Gazette’s section Measuring Up), some experts say that all too often officials ignore a key piece -- what the people who are supposed to benefit from government services think of those services. In response, academics, officials and others are striving to come up with ways to take the pulse of the people -- and use that data to improve government performance. The issue has attracted enough interest lately that last month, dozens of policy experts, academics and Bloomberg administration officials turned out early on a Monday morning for a forum at Baruch College on “giving New Yorkers a voice.”
THE MISSING NUMBERS
Since the 1970s, New York City has produced a welter of data to evaluate performance. Key to this is the Mayor’s Management Report, a document brimming with data on lots cleared of rodents, students graduating from high school, fire department response times and so on.
Beyond 311: Listening to New Yorkers
Most of this data focuses on what experts call inputs -- money, staff, resources -- and outputs â€“- usually statistics on services produced by government, such as tons of trash collected, number of potholes filled, the average length of a hospital stay, the number of bathrooms in city parks. But the data may not cover the outcome -- the quality of the service delivered.
It’s the difference between reporting on the number of potholes filled and the number of streets in excellent condition. So a sanitation department may report how much trash it collects but not whether, in the process, it dropped garbage or if it picked up trash from some buildings and not others.
To get that kind of information one often has to ask citizens.
So far, New York City has no comprehensive way to do this. Private groups (Straphangers and the Fund for the City of New York, to name two) have tried to gauge citizen satisfaction -- or dissatisfaction -- with government. Baruch College conducted two surveys for City Council in 2000 and 2001. And some city agencies try to get their own information. According to Jeffrey Kay, director of the Mayors Office of Operations, the Human Resources Administration, for example, asks people who go into their walk-in centers to evaluate the services they received, while the buildings department holds focus groups. (See The Web Site, 311, Surveys and Focus Groups.) And in the coming school year, the Department of Education plans to ask parents and older students to rate certain aspects of their schools. (See Parents Grading Schools.)
The city also culls some data from its 311 non-emergency helpline. But, however useful this might be, it does not provide an accurate picture of citizen satisfaction. “It is representative of who picks up the phone â€“ not necessarily of the broader population,” Diana Fortuna, president of the Citizens Budget Commission, said. (See Beyond 311: Listening to New Yorkers.)
And most people dial 311 only to report a problem. “Do people call 311 to pay the city compliments? Probably not, said Douglas Muzzio, a public policy professor at Baruch College. (What Residents Can Tell Government)
DO YOU NEED TO ASK?
Private companies, of course, frequently get satisfaction information from their customers. Sales figures give businesses a quick indication of whether people like their product. But companies also survey their customers. Customers of online business such as eBay or Amazon are asked for “feedback” and many hotels provide a customer response form.
“How strange that we don’t do this and in the private sector, they do it all the time, “ said Barbara Cohn Berman, director of the Center on Municipal Government Performance at the Fund for the City of New York. “They’re not going to make a purple car if no one’s going to buy it.”
It’s important for government to follow business’ example, many argue. “This kind of research can be eye-opening because it helps government to see things in a different way,” Berman said. When the fund conducted focus groups of New Yorkers in 1995 and in 2001, she said, “no one complained about taxes.” But, overwhelming majorities of people thought the condition of the city’s roads was “critically important” and were very dissatisfied with it.
Taking surveys also allows government to view data in various ways, considering, for example how different racial groups view a city service. “In a city as diverse as New York, such differences in how people experience government services are significant,” said Muzzio.
Getting citizen opinion has a number of advantages, said Marc Holzer, executive director of the National Center for Public Productivity at Rutgers University. “Citizens may have good ideas about government,” he said. Further, “You want citizens to feel they are connected with governmentâ€¦.and that government is a good use of their money.”
Despite this, Holzer said, many governments do not like to consult citizens. There is a tendency for this in New York City where, he said, “There’s a certain arrogance at City Hall.” As an example he cited the recent controversy over cell phones in public schools, where the city has persisted in enforcing a total ban, despite parents’ objections.
ASKING THE QUESTIONS
If one accepts that government should consider citizen ratings, the question then becomes how to get that information. It is far harder to do this in government than business, former Mayor Rudolph Giuliani said in 2001. “It requires a process of asking yourself not only why government exists but in the case of city government why certain agencies exist. What is their raison d’etre, what’s their purpose, and what is the reason the public wants those agencies,” he said.
To try to get answers to these questions, management experts, pollsters and statisticians have devised a number of methods. They include:
--Surveys of a random sampling of citizens: This has been used in a number of cities including San Jose. There the city contacted a statistically selected sample of residents -- not necessarily citizens -- and asked 100 questions about how they feel about the city, their concerns about the quality of life, safety and so on. (See The Way in San Jose.)
--Less random samples: Often these surveys are made available to a broad swath of citizens, any of whom can respond. In Broken Arrow, Oklahoma, a questionnaire on the police went out with the water bills, and in Sheboygan, Wisconsin, a response form was inserted in the local paper. While the results could prove useful, the self-selected sample could provide a skewed picture of citizen attitudes.
--User surveys: Residents who avail themselves of a certain city survey are then asked to evaluate. The Straphanger’s subway station survey is an example of this. Pranger said that, while that effort yielded useful results, it was very labor intensive. “It takes forever to hand out postcards,” she said. And the data has limitations, Pranger said. Responses to surveys, she said, may not be as reliable and standardized as data collected through observations and counts, such as the Straphangers reports on dirt in the subways.
Berman cautions that surveys, while they have value, can lead to erroneous conclusions. For example, she said many cities ask residents whether they feel safe. When people answer no, governments typically respond by increasing police presence. But, she said, the reason people feel vulnerable may have nothing to do with police presence: It could reflect press coverage of crime, poor street lighting or other factors. “You could be throwing good money after bad by putting more police there,” she said.
Focus groups: These are groups chosen from the population that are sampled through an open discussion â€“ rather than yes or no questions â€“ about a specific issue, product or, in this case, city services. The Fund for the City of New York has employed this approach, said Berman, because in a citizen survey “you can’t find out much about why people say what they do.”
Citizen driven government performance: In this approach, citizen groups collect data on a public service. Straphangers relies largely on this, sending out employees and others to gauge the cleanliness of subway cars or the operation of pay phones in subway stations. Other examples include New Yorkers for Parks, Inside Schools.
All of these methods, if use correctly, have their pluses and minuses. And then of course there’s the original citizen survey: The ballot box is open for public responses every November.
For more information:
“How the City Can Improve Tracking of Service Delivery,” Independent Budget Office, February 2002
"Listening to the Public: Adding the Voices of the People to Government Performance Measurement and Reporting" Barbara Cohn Berman of the Fund for the City of New York's Center on Municipal Government Performance discusses performance measures that reflect the public's needs, concerns, and perspective.
Government Accounting Standards Board. Focuses on government finances and accountability.
International City Council Management Association. It has a center for performance management
National center for Public Productivity. A research and public service organization devoted to improving productivity in the public sector. Â