Gotham Gazette

The Place for New York Policy and politics

Lost in the PDFs: Getting City Data


NY Map's new tech-offering, New York CityMap, helps New Yorkers to find basic services like firehouses, libraries, subway stations, senior centers, museums, and even Wi-Fi hotspots around a particular address.

Whatever else it does or does not do to keep construction safe in New York City, the Department of Buildings offers New Yorkers concerned about the projects in their midst a wealth of information. Simply typing in an address produces an array of material about the location, including certificate of occupancy, landmark status, number of complaints against the site and facts about its boilers and elevators.

This kind of sharing of information would seem to be exactly what Mayor Michael Bloomberg expects of agencies in his administration. Bloomberg has frequently stated he wants to use technology to make New York City's government more open. "More than 500 different measurements from 45 city agencies [are] all available with a few clicks of the mouse," he said in his 2008 State of the City address. "Good government is about transparency and accountability. We are doing everything we can to make them both permanent."

Meanwhile, the City Council has attempted to write online information sharing into law. In 2003, City Councilmember Gale Brewer, chair of the technology committee, authored a bill requiring that all city documents submitted to the Department of Records and Information Services be posted to the city's Web site within 10 days. The bill, Local Law 11, was the first legislation in the country to require a city government to use the Internet to provide access to official documents.

Providing this kind of access to government documents and information has been one of the important democratizing accomplishments of the Internet. New York City has been a leader in this area, but we are falling behind as a lack of standards and uneven performance across different agencies have left us with a confusing jumble of formats and numbers.

Five years after its passage, we are starting to see the limitations of Local Law 11. With no method for enforcement, compliance has been spotty. A survey of the Web sites of agencies suggests they are following Bloomberg's directive unevenly. Along with this, changes in technology have created new opportunities to make online information more accessible and useful. It might be time for an upgrade.

Opening the Government

The city puts out a lot of information that residents can use to monitor the performance of local government - so much information that, unless you know exactly what you're looking for, you will quickly drown in it.

As required by the City Charter, the mayor produces a report card for city services, called the Mayor's Management Report or MMR. Each mayor has done the report in his own style. Bloomberg's approach has been to trim it down to a list of indicators with totals or percentages, highlighting positive shifts and leaving negative one's unaddressed.

The mayor's Office of Operations has undertaken impressive efforts to craft a more user-friendly presentation for the information in the report. With its My Neighborhood Statistics Web site, visitors can sort the information by community board, or police precinct.

A separate mapping tool, NYCityMap allows New Yorkers to find basic services like firehouses, libraries, subway stations, senior centers, museums, and even Wi-Fi hotspots around a particular address.

Unfortunately, these tools present data in formats that are difficult to convert into other formats, limiting their usefulness. So, for example, if you want to sort the neighborhood statistics into council districts so you know who to complain to, you're out of luck. The map could allow you to merge it with other maps, an increasingly common online process known as a mashup. That could help apartment hunters, pizza hunters or anyone searching for information that you might want to see in a context of what city services are available.

Look But Don't Touch

The most common format across the city's Web portal is the PDF, which takes information and turns it into a two-dimensional document that cannot be manipulated or transformed into other kinds of data. An online PDF is definitely an improvement over going to the records department to get a printed copy of a document, but that's about it. Free PDF readers allow users to search within each document, but combining data from multiple documents has to be done manually.

For example, not only did the Office of Management and Budget publish the New York City's 2009 capital budget more than a week after the 10-day deadline mandated by Local Law 11, but when it did post the spending plan online it was as a 421-page PDF document.

Receiving the city budget as a PDF means that even the deftest programmer has to reverse engineer it to find such timely data such as which council members had requested what items.

Along with the mayor's budget every year, the Office of Management and Budget publishes an accompanying message filled with budget analysis. This too is a PDF, making it difficult to examine expenditures across departments or in some way other than how the mayor has presented the information. For example, the format provides no easy way to tally the expenditures on health programs not under the department of health. Getting that kind of information would require expensive data entry. If the city provided its budget in any number of more accessible formats, that data entry wouldn't be necessary.

In contrast, the state comptroller manages a Website called Open Book New York that provides easy access to New York State's financial information. It allows New Yorkers to search for individual state expenses across 113 agencies.

Ironically, the Department of Information Technology and Telecommunications is using inadequate technology for sharing its information, including 311 call center data.

Community boards, for example, have wanted information on noise complaints, The department of information technology is required by law to provide the boards with complaint data, but the Local Law 47 reports on 311 calls come in PDFs, which make it difficult to analyze trends in noise complaints or neighbors calling in construction violations.

"I've been working with the Community Boards endlessly to get them the 311 data in some form that they can manipulate. So far, we only have PDFs," Brewer said.

Some individual city departments have gone further with sharing information online. Along with data on individual sites, the Department of Buildings makes its weekly reports available in spreadsheets that can be searched and filtered for specific data and information.

The Department of Finance allows site users to subscribe to individual RSS feeds. RSS feeds provide automatic update notifications to other Web sites or applications so users get an alert in their inbox or web browser whenever there is new information. Combined with XML, a Web language that helps facilitate Web site data sharing, users can stay in constant contact with Web sites they subscribe to. New Yorkers who subscribe to the Department of Finance's Web site can receive data feeds with information on property, business taxes, or parking and vehicle news.

Importantly, XML feeds also allow people with sight or other impairments to access information through text readers and other helpful technologies. Publishing information as a PDF shuts that community out from their government.

Looking at Crime

The police department's data, on the other hand, is difficult to interpret. The department's weekly Comp Stat report breaks citywide data into seven broad categories and is only available to the public as a PDF. The department provides the information by precinct, but city residents cannot search for crime statistics by their ZIP code, neighborhood or even borough. Additionally, this link for "Department Statistics" was down at the time of researching for this article. The department did not respond to requests for comment.

Law enforcement agencies in other cities are much more forthcoming. The police department in Charlotte, N.C., for example, provides statistics for police calls requesting service by neighborhood, ZIP code and district, sorting the information in more than 20 different categories., which compiles information from agencies of several cities and presents it in a friendly format, has been trying to provide New York City residents with detailed information about crime. To get the statistics for New York, EveryBlock's "people person," Daniel O'Neil, has had to make phone calls and develop relationships.

In Chicago, where EveryBlock got its start, residents wanting to know what crimes have occurred in their area simply type in their address or neighborhood and receive a list of reported offenses. Clicking on the crime brings more information, including a locator map. This comes directly from the police department's records. By contrast, all the New York Every Block site offers is a weekly compendium of crimes by precinct, with no details on the reports and no indication of where in the precinct the alleged offense occurred.

While EveryBlock has not been able to persuade New York City's police department to make that information accessible, EveryBlock does provide access to New York restaurant inspections and building violations. The information is much more accessible on EveryBlock where anyone can search for violations in a particular ZIP code or by the name of a facility.

O'Neil says it is absurd that city agencies leave it to third party projects like his to provide information access. "It's unethical for cities to say that some external service is going to be essential to displaying city service requests," he said.

Making records accessible is not just a matter of honest government, O'Neil said; it's a matter of being cost effective. "The city spends an enormous amount of time and energy and human resources and technical resources to create reports that are practically worthless," he explained.

Opening the Information Portal

There are mechanisms intended to address this. The Commission on Public Information and Communication was created by the 1989 charter revision and is chaired by the public advocate. It is supposed to "make the administration more transparent to the citizens it serves and would encourage the public to become more actively involved in government," as Public Advocate Betsy Gotbaum said in a 2003 statement on the revival of the commission. Unfortunately, this revival never happened. The last meeting, of the commission according to the public advocate's Website was in September 2007.

"There are existing structures to handle this issue, but they've never functioned," said Thomas Lowenhaupt, who is promoting a .nyc top-level domain (the ending of a web address) to better connect people to information about the city.

The mayor continues to make improvements in its information-sharing practices, launching a Citywide Performance Reporting system earlier this year. The data for CPR are drawn from hundreds of indicators from agencies citywide, updated monthly. As Nicholas Sbordone from the information technology department explained, this "lets users really drill down into each indicator and compare trends across the past five fiscal years."

Some experts have questioned the indicators the administration has chosen to list. The Bloomberg administration could go further by adopting the best practices from other cities and using open standards across all agencies. That would help residents conduct their own analysis or convert it into the format that works best for them. And it would streamline City Council's job of oversight.

For its part, City Council could update Local Law 11, or at least be more vigilant in seeing that all agencies comply. "The office of operations needs to be involved in pursuing agencies that don't go on the web as they should," says Brewer. "I don't think you can fine agencies, but you have to have some carrot internally."

For now, though, we'll have to keep searching.

Joshua Breitbart and Kristofer Rios run the Digital Expansion Initiative at People's Production House.

Print Friendly and PDF

Editor's Choice

Gotham Gazette: Max Politics Podcast: Bill De Blasio's Legacy and Eric Adams' Agenda
Max Politics Podcast: Bill De Blasio's Legacy and Eric Adams' Agenda
Gotham Gazette: Emma Wolfe Reflects on the De Blasio Years
Emma Wolfe Reflects on the De Blasio Years
Gotham Gazette: City Continues Expansion of Online Access to Public Benefits
City Continues Expansion of Online Access to Public Benefits
Gotham Gazette: Max Politics Podcast Max Politics Podcast