When the Straphangers Campaign asked riders at 15 of New York’s busiest subway stops what they thought of their stations, the answers from almost 4,000 riders surprised the advocacy group.
Grading the stations on cleanliness, security, announcements, crowding and ease of getting around, the riders gave their top score to the Yankee Stadium/161st Street station in the traditionally underserved Bronx – despite renovation problems and complaints about rats. The 51st Street and Lexington Station came in dead last in the 2000 survey, even though in many respects it looked better than its Bronx counterpart. A possible conclusion: When it comes to rating stations, many riders would prefer vermin to overcrowding.
Citizens often have different criteria for rating services than experts or officials. Health administrators may evaluate a hospital based on the number of prestigious doctors or the amount of cutting edge equipment. Patients, though, tend to look at whether the triage nurse in the emergency room treated them with courtesy and respect. In San Jose, the heart of the high-technology Silicon Valley, city officials expected residents would prefer to get key information on public services via their computers. No, residents said in a survey: They wanted to speak to a “real” person on the phone.
As governments, including New York City’s, find new ways to assess their performance (for more, see Gotham Gazette’s section Measuring Up), some experts say that all too often officials ignore a key piece -- what the people who are supposed to benefit from government services think of those services. In response, academics, officials and others are striving to come up with ways to take the pulse of the people -- and use that data to improve government performance. The issue has attracted enough interest lately that last month, dozens of policy experts, academics and Bloomberg administration officials turned out early on a Monday morning for a forum at Baruch College on “giving New Yorkers a voice.”
Since the 1970s, New York City has produced a welter of data to evaluate performance. Key to this is the Mayor’s Management Report, a document brimming with data on lots cleared of rodents, students graduating from high school, fire department response times and so on.
Beyond 311: Listening to New Yorkers |
Most of this data focuses on what experts call inputs -- money, staff, resources -- and outputs –- usually statistics on services produced by government, such as tons of trash collected, number of potholes filled, the average length of a hospital stay, the number of bathrooms in city parks. But the data may not cover the outcome -- the quality of the service delivered.
It’s the difference between reporting on the number of potholes filled and the number of streets in excellent condition. So a sanitation department may report how much trash it collects but not whether, in the process, it dropped garbage or if it picked up trash from some buildings and not others.
To get that kind of information one often has to ask citizens.
So far, New York City has no comprehensive way to do this. Private groups (Straphangers and the Fund for the City of New York, to name two) have tried to gauge citizen satisfaction -- or dissatisfaction -- with government. Baruch College conducted two surveys for City Council in 2000 and 2001. And some city agencies try to get their own information. According to Jeffrey Kay, director of the Mayors Office of Operations, the Human Resources Administration, for example, asks people who go into their walk-in centers to evaluate the services they received, while the buildings department holds focus groups. (See The Web Site, 311, Surveys and Focus Groups.) And in the coming school year, the Department of Education plans to ask parents and older students to rate certain aspects of their schools. (See Parents Grading Schools.)
The city also culls some data from its 311 non-emergency helpline. But, however useful this might be, it does not provide an accurate picture of citizen satisfaction. “It is representative of who picks up the phone – not necessarily of the broader population,” Diana Fortuna, president of the Citizens Budget Commission, said. (See Beyond 311: Listening to New Yorkers.)
And most people dial 311 only to report a problem. “Do people call 311 to pay the city compliments? Probably not, said Douglas Muzzio, a public policy professor at Baruch College. (What Residents Can Tell Government)
Private companies, of course, frequently get satisfaction information from their customers. Sales figures give businesses a quick indication of whether people like their product. But companies also survey their customers. Customers of online business such as eBay or Amazon are asked for “feedback” and many hotels provide a customer response form.
“How strange that we don’t do this and in the private sector, they do it all the time, “ said Barbara Cohn Berman, director of the Center on Municipal Government Performance at the Fund for the City of New York. “They’re not going to make a purple car if no one’s going to buy it.”
It’s important for government to follow business’ example, many argue. “This kind of research can be eye-opening because it helps government to see things in a different way,” Berman said. When the fund conducted focus groups of New Yorkers in 1995 and in 2001, she said, “no one complained about taxes.” But, overwhelming majorities of people thought the condition of the city’s roads was “critically important” and were very dissatisfied with it.
Taking surveys also allows government to view data in various ways, considering, for example how different racial groups view a city service. “In a city as diverse as New York, such differences in how people experience government services are significant,” said Muzzio.
Getting citizen opinion has a number of advantages, said Marc Holzer, executive director of the National Center for Public Productivity at Rutgers University. “Citizens may have good ideas about government,” he said. Further, “You want citizens to feel they are connected with government….and that government is a good use of their money.”
Despite this, Holzer said, many governments do not like to consult citizens. There is a tendency for this in New York City where, he said, “There’s a certain arrogance at City Hall.” As an example he cited the recent controversy over cell phones in public schools, where the city has persisted in enforcing a total ban, despite parents’ objections.
If one accepts that government should consider citizen ratings, the question then becomes how to get that information. It is far harder to do this in government than business, former Mayor Rudolph Giuliani said in 2001. “It requires a process of asking yourself not only why government exists but in the case of city government why certain agencies exist. What is their raison d’etre, what’s their purpose, and what is the reason the public wants those agencies,” he said.
To try to get answers to these questions, management experts, pollsters and statisticians have devised a number of methods. They include:
--Surveys of a random sampling of citizens: This has been used in a number of cities including San Jose. There the city contacted a statistically selected sample of residents -- not necessarily citizens -- and asked 100 questions about how they feel about the city, their concerns about the quality of life, safety and so on. (See The Way in San Jose.)
--Less random samples: Often these surveys are made available to a broad swath of citizens, any of whom can respond. In Broken Arrow, Oklahoma, a questionnaire on the police went out with the water bills, and in Sheboygan, Wisconsin, a response form was inserted in the local paper. While the results could prove useful, the self-selected sample could provide a skewed picture of citizen attitudes.
--User surveys: Residents who avail themselves of a certain city survey are then asked to evaluate. The Straphanger’s subway station survey is an example of this. Pranger said that, while that effort yielded useful results, it was very labor intensive. “It takes forever to hand out postcards,” she said. And the data has limitations, Pranger said. Responses to surveys, she said, may not be as reliable and standardized as data collected through observations and counts, such as the Straphangers reports on dirt in the subways.
Berman cautions that surveys, while they have value, can lead to erroneous conclusions. For example, she said many cities ask residents whether they feel safe. When people answer no, governments typically respond by increasing police presence. But, she said, the reason people feel vulnerable may have nothing to do with police presence: It could reflect press coverage of crime, poor street lighting or other factors. “You could be throwing good money after bad by putting more police there,” she said.
Focus groups: These are groups chosen from the population that are sampled through an open discussion – rather than yes or no questions – about a specific issue, product or, in this case, city services. The Fund for the City of New York has employed this approach, said Berman, because in a citizen survey “you can’t find out much about why people say what they do.”
Citizen driven government performance: In this approach, citizen groups collect data on a public service. Straphangers relies largely on this, sending out employees and others to gauge the cleanliness of subway cars or the operation of pay phones in subway stations. Other examples include New Yorkers for Parks, Inside Schools.
All of these methods, if use correctly, have their pluses and minuses. And then of course there’s the original citizen survey: The ballot box is open for public responses every November.
“How the City Can Improve Tracking of Service Delivery,” Independent Budget Office, February 2002
"Listening to the Public: Adding the Voices of the People to Government Performance Measurement and Reporting" Barbara Cohn Berman of the Fund for the City of New York's Center on Municipal Government Performance discusses performance measures that reflect the public's needs, concerns, and perspective.
Center on Municipal Government Performance at the Fund for the City of New York
Government Accounting Standards Board. Focuses on government finances and accountability.
International City Council Management Association. It has a center for performance management
National center for Public Productivity. A research and public service organization devoted to improving productivity in the public sector. Â
Taking the Public Pulse |
How do you give New Yorkers a voice? How do you bring them into the system? How do you find out what they want? And then once you collect that data, how do you turn it around and how do you produce results?
nyc.gov is sort of the front line. Right now, every city agency and dozens of independent offices are included on the nyc.gov web site. They have their own links. Four hundred and ten forms are available. Fifty-five thousand online forms are submitted for use. People are obviously using the system. There are over 250,000,000 hits or page views. It's been ranked number one in usability for city government services and number two in digital governance.
You can contact agency heads. You can contact the mayor. There are 16 comments and feedback forms. Over 730,000 forms are submitted to nyc.gov annually. That's a significant amount of interaction between the citizens and government. And at the bottom, it provides easy access to the performance data in many different formats.
311 is a great base. It's taking 46,000 calls a weekday. To date, over 35 million calls have been made to 311. You can request information. You can ask easy questions. You can make a recommendation. You can also say, "I just passed a pothole" and that information goes directly into the system.
The average call waiting time is 15 seconds. Eighty eight percent of the calls are picked up within 30 seconds.
Now that begs the next question: Are we just seeing how quickly the call is picked up, or are we looking at the quality of it. Is there follow-up? Are you getting the right answer?
In terms of citizen surveys, we don't have a citywide strategy, but different agencies are doing different things. Our Department of Buildings has focus groups in each field office. The Department of Finance has a huge amount of customer survey data as does the Department of Parks. The Human Resources Administration has ongoing surveys in their walk-in centers: "Do you feel you got what you needed while you're here?" The Department of Aging has a senior advisory council.
Once you collect the data, once you go after what the citizens want, how do you turn that into producing results.
You can do it by holding agency and managers accountable for collecting this data. If they collect it, at least they have the information. That's the obvious first step.
Then you need to present it to the public. And you need to always refocus and always try to get a new sense of what people want.
Jeffrey Kay is director of the New York City Mayor's Office of OperationsÂ
Taking the Public Pulse |
Government officials, management experts, and program analysts agree that government services must be customer driven and results-oriented. Governments should pay attention to residents' perceptions and assessments of the quality of the services they receive. “The best way to encourage good performance is to measure it, and the best indicator of government performance is citizen satisfaction,” according to the International City and County Management Association
Hundreds of municipalities, including Phoenix, Miami-Dade County, Detroit, San Diego, San Francisco, Austin, San Jose, and Portland, Oregon, use resident surveys to evaluate the quality of services.
Resident surveys provide a unique perspective on municipal services that is not available from other sources. In particular, resident surveys can provide information on:
• constituents' satisfaction with the quality of specific services;
• facts on who uses – and does not use –various services, including how often people use the services and how to use them;
• the reasons specific services are disliked or not used
• high priority but inadequate services;
• potential demands for new services;
• residents' evaluation of and support for policy or service delivery options.
Resident surveys concentrate on the outcomes or results of government services – how satisfied people are with their schools and parks, how safe they feel in their neighborhoods. On the other hand, administrative measures including those found in New York City's Mayor's Management Report focus on inputs (money, staff), effort, and sometimes outputs (tons of trash collected, number of potholes filled). While these are important, what the public wants from government centers on results.
Furthermore, resident surveys allow for the analysis of individual differences in how people use and experience city services – for example, differences by race, ethnicity, age, and gender. In a city as diverse and dynamic as New York, such differences in how people experience government services are significant.
The city's 311 program provides much valuable data to managers and policy-makers as well as the city's diverse stakeholders. But 311 data are not outcome measures. And users of 311 are not representative of New York City residents generally. Callers self-select, violating the cardinal rule of survey research that calls for a random sample.
Interestingly, callers to 311 may not be the most disgruntled New Yorkers. In fact, most dissatisfied customers don't complain -- they just take their business elsewhere. In the public sector, the dissatisfied citizens may simply stop trusting government or move someplace else.
And what about the things New Yorkers like about their government – do people call 311 to pay the city compliments? Probably not. A citizen survey would help capture positive views and document areas where New York City government should keep up the good work.
Moreover, calls to 311 are motivated by a need for information or services, not to provide feedback. The system in no way enquires about residents' awareness, use and assessment of services, nor does it provide data on the demographic or socioeconomic characteristics of its users.
Noise complaints offer an example of how such information might be misused to diagnose where service problems exist. A map of noise complaint calls made to the city's quality of life line in fiscal year 2001 found more complaints from Staten Island's community boards than from all of the Bronx -- a borough three times Staten Island's size. As one Bronx community board member stated at the time, "The only reason why the number of [noise] complaints may not be as high is that many of the people in our community don't trust the system. And they don't believe that they are going to get results.”
Douglas Muzzio is a professor of public policy at Baruch College Â