Gotham Gazette Thu, 04 Mar 2021 22:08:07 +0000 Joomla! - Open Source Content Management en-gb (Webmaster) Opening Up New York’s COVID-19 Data Can Save Lives amid ongoning covid

(photo: governor's office)

WHEREAS, Open Data Can Make New York Government More Transparent and Accountable in Order to Promote Public Trust and Save Lives During the COVID-19 pandemic...

Something close to this is the preamble to Governor Cuomo’s 2013 Executive Order 95, which created the state’s open data program almost simultaneously with New York City’s pioneering Open Data Law. What does open data have to do with a massive pandemic and economic shutdown? Well, just about everything. 

The COVID-19 pandemic has highlighted the importance of public trust in government. Fortunately, New Yorkers have overwhelmingly heeded the governor’s and mayor’s mandates to socially distance and wear face coverings. This basic trust has slowed the virus in New York and saved lives. Unfortunately, there is a long way to go before New Yorkers have seen the last of COVID-19. The New York Times just reported the Centers for Disease Control is internally predicting a massive increase nationally in COVID-19 cases and fatalities in late May and June. 

New York is not an island. As long as COVID-19 cases are skyrocketing nationally, we will be at high risk of reinfection. Given how much death, misery, and economic suffering we have already experienced, it will require great trust and patience with our state and city governments for the public to keep heeding inconvenient and onerous public health rules. We must have trust in governments’ ability to test, track, and report to us.

One important way New York can earn public trust is to open up the huge amounts of data related to COVID-19. 

Last month, Reinvent Albany, BetaNYC, and six other watchdog groups sent a letter to the governor asking that all of the state’s data on COVID-19 be placed in the open data portal (a separate letter from seven groups went to the Mayor asking the same). New York State and City’s COVID-19 pages provide more data than most other governments, but they provide very little open data – data that is downloadable, sortable, and machine-readable. Despite all the charts on the New York state COVID-19 tracker, only a single dataset is provided in the state’s open data portal, whereas none of the city’s Department of Health and Mental Hygiene (DOHMH) data is centralized on the city’s open data portal.

Withholding open datasets creates an unnecessary burden for the public and sows doubt. If someone wants to analyze the data on New York state’s COVID-19 page, they must go through the tracker and enter all of the numbers into a spreadsheet manually. Some datasets – such as deaths by date – aren’t even provided in the tracker and can only be found in the governor’s daily powerpoint presentations. Likewise, too many of the city’s datasets are only available in PDF format, with some open data on GitHub, meaning that most city data also requires manual formatting. Both the state and the city could remove this time-consuming process by simply opening up all of the data on COVID-19. How can we trust the government's response if it isn't sharing data in an accessible way?

Both New York State and City need to make all of this information available in their Open Data portals, and the datasets must be as granular as possible, with discrepancies addressed and logged. The state should be sharing county-level data. The city should be sharing data down to the Neighborhood Tabulation Area (NTA) – the same level of data the DOHMH shares about 100 other measures via the New York City Neighborhood Health Atlas. There are legitimate concerns about the need to protect New Yorkers’ privacy during this crisis, but the DOHMH has a simple process for sanitizing the data to ensure civil liberties aren’t violated.

Releasing New York’s COVID-19 data will be crucial to prepare for second or third outbreaks, which experts say are likely to occur over the next year. To prevent more tragedy, New York will need to know where the virus travels, whom it is affecting, how it is getting there, and who is most at risk. This doesn’t mean that the city should limit its datasets to in-house experts – instead, the data should be opened up so that the informed public can share its own analysis and insight rather than being restricted to official narratives.

Open data is key for building public trust in government. Seeing how our governments are operating enables the public to better understand its operations. Trust is what gives government the latitude to do more good work. Times of crisis may seem to make open data less of a priority, but these times are in fact when open data is needed the most.

John Kaehny is executive director of Reinvent Albany. Noel Hidalgo is executive director of BetaNYC. On Twitter @JohnKaehny & @ReinventAlbany & @noneck & @BetaNYC.

Have an op-ed idea or submission for Gotham Gazette? Email

Opening Up New York’s COVID-19 Data Can Save Lives Sun, 10 May 2020 04:00:00 +0000
De Blasio Names New Chief Analytics Officer kelly jin 
Kelly Jin (photo via the mayor's office)

After a yearlong vacancy in the position, Mayor Bill de Blasio named Kelly Jin as the city’s new Chief Analytics Officer and Director of the Mayor’s Office of Data Analytics. Jin, who started in her new position on Monday, replaced Amen Ra Mashariki, who left the position in 2017 to work in the private sector at mapping firm Esri.

While tasked with helping the mayoral administration effectively use data in governing, the position’s most notable concrete responsibility is managing the city’s open data program, wherein city agencies upload datasets to a public portal accessible online to everyone, as required by law. The program began in 2012 under former Mayor Michael Bloomberg, and was expanded significantly by the de Blasio administration through the “Open Data for All” plan.

The Open Data Portal currently hosts over 2,200 datasets from 98 different city agencies. The most recent annual report from the agency, filed last month, showed 629 datasets having been added by 38 different agencies, including data on taxi and for-hire vehicle trips, intimate partner violence at the community board level, and a large-scale measure of poverty in the city.

Agencies from the NYPD to the Department of Education to 311 have datasets on the portal, with some agencies having more than others. However, the program has faced criticism for not doing enough to ensure that agencies upload their datasets on time, and for the generally slow progress that has been made.

“Our data analytics team is a crucial part of bringing important city initiatives to life, tracking their progress, and finding new ways to understand the city through data,” de Blasio said in a statement about Jin’s addition to his team. “I know Kelly has the experience and the drive to use our data resources to help City Hall and city agencies make better decisions and craft better policies, as well as manage the largest Open Data program in the country.”

Jin was most recently the “Director of Results-Driven Government” at the Laura and John Arnold Foundation, a philanthropic enterprise that promotes making decisions in government, science, and other areas by following evidence. She previously worked at the White House as a policy advisor to the US Chief Technology Officer under President Obama, and before that led the City of Boston’s Data Analytics Team.

In a Tuesday interview with Gotham Gazette, Jin identified improving data infrastructure and integration of data between agencies as goals for her tenure with New York City. She also said she would seek to create better implementation tools, and use data to help agencies make better decisions.

“What I’ve learned across all the different orgs is that executives and leadership of the various agencies, they’re looking for help around decision-making at the executive level,” Jin said. “So, this is really, ‘how can we use evidence and data and summarize that,’ because this is not waiting a few months to hear about results, but is really about there are priorities, clear priorities here within the city and with the mayor that this team is already tasked with and helping to get data into the hands of executives, so that they can ask and answer better questions.”

Jin noted work the unit has done with the Department of Transportation on Vision Zero as an example of the kind of data-driven policymaking she’d like to see expanded.

Jin, who reports to the Director of the Mayor’s Office of Operations, Jeff Thamkittikasem, also noted that, since she only began working in the city this week, she hopes to set up meetings with agency officials to discuss plans for more effective compliance with open data targets and deadlines, and gain insights into the intricacies involved with the office and open data.

“I think that it is setting up some initial meetings on my end,” Jin said. “I hope to get with those particular agencies pretty soon here as I get started.”

The 19-month long vacancy in the position Jin now holds apparently did had little impact on the work of the seven person-strong team she’s now leading in terms of the Open Data Portal: the 629 datasets published in 2018 represent a nearly fourfold increase over the 170 datasets released in 2017; 2016 saw 150 datasets uploaded to the portal. There are major deadlines to meet.

“The team, over the last year-and-a-half, has worked closely with the Mayor’s Office of Operations, and so when it comes to reporting and compliance on open data, I know that the team has pushed forward on that work in partnership with the agencies to release different datasets,” Jin said.

Jin said that she hopes to travel across the city to learn how data can be used to improve people’s lives, and said she would like to hold town hall meetings, something Mashariki also did.

“I’m open to ideas. If there’s a quote from me here, it’s I’m open and have open ears for additional ideas,” Jin said. “I think right now, the possibilities here of getting out and talking to different stakeholders, and town halls, definitely going to be doing a circuit through the different boroughs to get a better understanding of what data and what policies are impacting folks.”

by Ben Brachfeld, Gotham Gazette

Read more by this writer.

De Blasio Names New Chief Analytics Officer Tue, 16 Oct 2018 04:00:00 +0000
City Council Passes 'Open Budget' Bill Mayor Has Yet to Support 15610942274 2c68194eaf z

Council Member Ben Kallos (photo: William Alatriste/New York City Council)

The New York City Council passed legislation on Tuesday mandating that the government agency that oversees the city’s $85.2 billion annual budget provide all budget documents to the public in an easily accessible and machine-readable format.

The legislation, sponsored by Council Member Ben Kallos, chair of the governmental operations committee, requires the Office of Management and Budget to post documents released in the annual budget process to the city’s open data portal within 10 days of posting them on their website. OMB produces multiple iterations of the city’s budget each year as the annual expenditure plan is negotiated between Mayor Bill de Blasio and the City Council. These include the preliminary budget proposal, the executive budget proposal and the final adopted budget, each of which contain hundreds of pages with detailed breakdowns of agency spending and city revenues. The agency also issues a budget modification document in November.

“I want to know where every single penny in the budget is,” said Kallos. “We’ve already got a lot of this implemented, but the parts of the budget that matter are still hidden behind thousands and thousands and thousands of pages of pdfs that can’t be searched, let alone analyzed.”

Kallos’ bill would make reams of expenditure and revenue reports searchable online and downloadable in bulk, providing an added level of transparency to city budgeting, he says. “With this legislation, whether it’s a member of the City Council, City Council staff or any resident will be able to actually see where the dollars are being spent and really hold the administration accountable.”

An ardent advocate of government reform and a software developer himself, Kallos has consistently pushed the de Blasio’s administration to increase the amount of data available on the open data portal. The administration, for its part, has made significant progress since the city’s open data initiative was first launched in 2012 by former Mayor Michael Bloomberg.

Kallos believes this latest bill will build on that effort. “We spent years working with the administration, working with the existing financial management system to ensure that it could be done,” he said. “We’ve gotten a lot of the budget up.”

It’s unclear whether the mayor will sign the bill into law, although he has yet to veto a single bill passed by the City Council. “We look forward to reviewing the legislation,” said Freddi Goldstein, a mayoral spokesperson on budget matters, in an email.  

The bill would go into effect immediately if the mayor signs it, but it would only cover future budget documents and would not apply retroactively. Under de Blasio, the city’s budget has increased by 17.2 percent, from $72.7 billion in the last modification under Bloomberg to $85.2 billion for fiscal year 2018.

Kallos held out hope that past budgets would eventually be included on the open data website. “I would love to be able to compare our city’s budget going back as far as we have records for,” Kallos said.

[Related: City Releases Latest Progress on Open Data Plan]

by Samar Khurshid, senior reporter, Gotham Gazette

Read more by this writer.

City Council Passes 'Open Budget' Bill Mayor Has Yet to Support Tue, 31 Oct 2017 21:05:18 +0000
City Council Hears Testimony on 'Open Data 2.0' Bills c93 69luiaaiaii.jpg large

Council Member James Vacca (right)

At an oversight hearing on Wednesday of the City Council’s Committee on Technology, good government and civic technology experts praised a pair of bills aimed at updating the city’s 2012 open data law, but raised concerns that the proposals could have unforeseen ramifications.

The city’s current law mandates that city agencies publish non-confidential agency data on a public web portal for all New Yorkers to see and easily access. The open data portal is now home to over 1,600 data sets, with hundreds more data sets scheduled to be released in the coming years. The two “Open Data 2.0” proposals, sponsored by Council Member James Vacca, chair of the committee, and supported by Manhattan Borough President Gale Brewer, are intended to bolster open data reporting standards as well as improve certain areas of the current policy. “Access to government information ensures public institutions can be held accountable,” said Vacca, in his opening statement. “In New York City, the importance of public access to data is reflected in both law and practice.” He noted that New York City had been the first municipality in America to pass a law requiring public data be posted online in machine readable formats.

The hearing was a bittersweet occasion for Vacca, who is term-limited and will be leaving the City Council in December after 12 years. It was his final oversight hearing on open data policy, a project he has long championed.

The city’s open data policy is seen as a model not just for the United States, but the entire world, Brewer noted in her opening statement at the hearing. “I just met with elected officials in Spain... And they are looking at our law,” Brewer said. To illustrate one of the uses of open data, Brewer will be joining civic technology group BetaNYC to host a “311 Data Jam” on Saturday, employing the city’s dataset on the 311 helpline. It’s the third such event hosted by BetaNYC, and they will be introducing “BoardStat,” a tool for community boards to analyze visualized 311 data in order to discover community trends.

The bulk of Wednesday’s hearing covered Intro. 1707, which includes provisions requiring reviews of the “Technical Standards Manual,” which is the guideline for how agencies manage and present their datasets every two years, to ensure the datasets remain up to date.

“These standards are designed to make open data more usable to the maximum number of users,” said Albert Webber, director of Open Data for the Department of Information Technology and Telecommunications (DoITT). “We see the TSM as a living document, meant to keep up with advances in technology, data availability, and the passage of local laws that impact data disclosure. An official periodic review of this document would ensure that open data disclosure stays up-to-date, relevant, and true to the most current practices.”

The bill codifies a requirement for city agencies to appoint an “open data coordinator” to ensure agency compliance with the open data law. It also changes the deadline for agencies’ compliance plans from July to September, to avoid conflicting with the city’s budget season. The bill would also require that data analytics on public usage of the data portal be collected and studied.

Criticisms of the legislation stemmed not from opposition to any one plank, but rather from unintended consequences that certain provisions of Intro. 1707 could have in areas such as agency reporting, accountability, and protection of individual privacy.

One provision in the proposal raised eyebrows among good government groups and civic technologists. Noel Hidalgo, executive director of Beta NYC, and John Kaehny, executive director of Reinvent Albany, pointed out that the bill extends the open data compliance deadline for city agencies from 2018 to 2021, a full nine years after the law went into effect in 2012. Many datasets from numerous city agencies are currently scheduled for release on the current deadline, December 31, 2018.

Kaehny said allowing agencies to hold off on publishing data is akin to a high school student saying they’ll turn in all of their homework on the last day, a promise that can’t be seen as reliable. He made similar comments to Gotham Gazette in July after the city released its annual update to the open data plan. Kaehny referred to the extension to 2021 as a “homework extension” that will never be turned in.

Vacca pressed Webber on these concerns, and Webber insisted that the administration was committed to releasing the data sets on time. “Putting a deadline is always helpful,” he told the Council member. “But no, we do not have concerns that pushing it to 2021 will delay the release of some datasets.” In his testimony, Webber stated that the administration would “like to discuss how to improve the language to ensure that submissions meant to be disclosed by December 31, 2018 will still be held to that deadline.”

Another aspect of Intro. 1707 that received some critique was a provision calling for the collection of data analytics on visitors to the portal. Concerns were raised that recording data on unique visitors to the portal, especially those exploring data that could be construed as sensitive, might violate an individual’s privacy. The language of this provision in the bill was viewed as vague.

“Pageviews, great,” said Sumana Harihareswara, a programmer from Astoria who testified at the hearing. “Numbers of pageviews, great. Unique users? I don’t want to be identified uniquely in a public place for having looked up a certain piece of data. So number of unique users is perhaps what this ought to say.”

Those who testified otherwise generally praised the proposed biannual review of the TSM, the change in the annual compliance deadline from July to September, and codifying into law the agency requirement of an open data coordinator.

The bill also saw some critique for what it lacked, such as a “private right of action,” the ability of the public to sue agencies failing to perform their duties. Kaehny noted that the only course of action without the private right of action is to “name and shame” agencies considered to be “laggard” in regards to their dataset releases. According to Kaehny, open data is one of the few pieces of legislation in New York City that does not include a private right of action, making it highly unusual, especially for such a major city law.

Brewer echoed the call for a private right of action, emphasizing that future administrations may not be as committed to transparency. “I believe we must provide a private right-of-action to protect the New York City municipal open data apparatus from a future administration that does not wish to operate it in good faith as the current and previous administrations have done,” she said. She cited the Trump administration’s moves immediately post-inauguration to delete data that had been posted online under the previous administration; there is no federal law that establishes similar data protections as New York City’s policy.

Kaehny claimed that even naming and shaming is a difficult task, given that agency compliance with the law is not compiled in a single spreadsheet on the portal but rather in four separate spreadsheets. Compiling all relevant information about published and unpublished datasets into one spreadsheet was one of Reinvent Albany’s key recommendations to the committee. “What we’re seeing is fog when it comes to understanding exactly how well agencies are doing,” Kaehny said.

The administration also had suggestions for improving the bill. Echoing testimony from the advocates, DoITT’s Webber expressed dismay at the law’s lack of any licensing requirements for the data, hoping that “permissive licenses,” licenses with few, if any, redistributive restrictions, could be added to the program.

“Our mission to encourage public engagement with open data in the long term makes the ability to license very important,” Webber said. “Structured properly, licenses could allow us to formally make the city’s datasets free and open for public use in perpetuity, codifying our existing practice and ensuring that New Yorkers have access to open data for the long haul.”

Also under consideration at the hearing was Intro. 1528, an amendment to an existing open data law that would require agencies that process Freedom of Information Law requests involving non-confidential data to report and publish the names of the public datasets, rather than simply the metrics of the datasets. The measure received overall support from those testifying before the committee.

by Ben Brachfeld, Gotham Gazette

Read more by this writer.

City Council Hears Testimony on 'Open Data 2.0' Bills Thu, 21 Sep 2017 01:19:55 +0000
Open Data Promises New Era of Transparent Government, When Will It Arrive?

Vision Zero data (photo: @NYCMayor)

New York has had some issues when it comes to ethics and transparency in government, to say the least.

Whether it is the more than 30 politicians that have been convicted of corruption-related offenses, or countless challenges with getting public records from government at every level, it’s not hard to see that New York has a long way to go in becoming ethical, transparent, and accessible.

It will be a long road to repairing this mess. One concept that has the potential to be a huge help is Open Data.

Open data policies come in different shapes and sizes, but they all aim to take advantage of the digital age by posting data collected and maintained by government agencies to a public online portal. This provides citizens and watchdogs with easier access to information, and helps government agencies gain valuable insight into how they are performing.

It also reduces how often formal records requests must be filed under the Freedom of Information Law. For citizens, filing these requests can be difficult to manage, and potentially require costly lawsuits to see through. That’s not an option for many New Yorkers. Just think of the parents who recently protested the city Department of Education’s awful response time when they asked for records.

On the bright side, New York City’s Open Data plan is setting an example worth following for governments across the state.  

The city’s policy has been in effect since March 2012, when Mayor Michael Bloomberg signed Local Law 11 of 2012 “to make City data available online using open standards.” City agencies’ public data sets must be posted on a single web portal by the end of 2018, with many sets already up and certain benchmarks along the way.

This legislation made New York City a leader in the open data movement, but the city has hit bumps in the road in implementing it.

Agencies were first required to self-submit plans showing when they would publicly release data. This was thankfully bolstered by packages of amendments added in 2015 and 2016. These amendments required more timely responses to public requests for new data sets.

More significantly, they demanded that agencies review FOIL requests asking for data to see if they involved new public data sets that could be published. This may sound complicated, but it’s a very commonsense ‘if one person is asking a question, somebody else is thinking it too’ type of policy. It is a built-in way to constantly expand the data available to the public.

The Mayor’s Office of Data Analytics (MODA) examines three agencies each year to verify that all public data sets have been disclosed. This is nice, but operates at a snail’s pace.

Many agencies are still delinquent in their reporting. With 92 participating agencies, it’s clear that MODA’s examination of just three per year will not guarantee all 92 comply by the December 2018 deadline. It’s one thing to promise, but New Yorkers need to see government deliver.

The bottom-line is positive, though. The City’s Open Data Portal received an average of 140,000 monthly hits from 2016 to 2017. There is a clear demand for transparency, and people using the system can hold lagging agencies accountable.  

The City’s leadership on this issue has also gained a few followers.

Recently, the City of Syracuse launched its open data portal. Currently, the portal houses data about neighborhoods, housing, infrastructure, and lead risks. The city is requesting public input on what data should be added.

New York State enacted a similar open data policy in 2013, when Governor Cuomo signed Executive Order No. 95 establishing “an online Open Data Website for the collection and public dissemination of Publishable State data.”

However, the state’s plan is well behind New York City’s. Even the state’s trademark bragging tone cannot hide that its report shows incredibly important departments barely post anything. The state’s economic development arm, its Department of Education, and Government & Finance are all below 40 percent of records posted.

A look at specific departments finds a number that haven’t posted items recently. For example, the Dormitory Authority, which oversees Albany’s most notorious slush fund, the “State and Municipal Facilities Program” has not posted new data in over a year.

New Yorkers should be excited about Open Data’s potential to provide a window into what their governments are doing. They must get more involved as these policies evolve, and demand that deadlines and requirements are met – and that the state get caught up to speed.

Candice DiLavore is Reclaim New York Transparency Project Manager. On Twitter @cdilavorecdilavore & @ReclaimNewYork.

Have an op-ed idea or submission for Gotham Gazette? Email

Open Data Promises New Era of Transparent Government, When Will It Arrive? Thu, 17 Aug 2017 04:00:00 +0000
City Releases Latest Progress on Open Data Plan open data portal

(screenshot from Open Data Portal)

The city released its annual update to its Open Data plan on July 15, detailing the progress by city agencies to release all of their public data by 2018. The initiative, originally launched in 2012 under former Mayor Michael Bloomberg, was expanded by the de Blasio administration two years ago under its Open Data for All plan, which aimed to make the data more accessible to everyday New Yorkers. City agencies periodically add data sets to the portal for public access.

In a statement, Mayor Bill de Blasio lauded the transparency efforts intrinsic to the open data plan. “Transparency is vital for a healthy government and flourishing democracy,” he said. “That’s why Open Data for All is delivering more data, in more ways, to more New Yorkers than ever before.”

Under the Open Data plan, information from 92 participating agencies is compiled by the Mayor’s Office of Data Analytics and the Department of Information Technology and Telecommunications, and released online in datasets. According to the city, the portal has been visited 3 million times and is home to 1,700 data sets, 165 of which were added in the past year. The city also redesigned the portal in March, which it says led to a 367 percent increase in data inquiries from the public. According to Miguel Gamiño, New York City’s Chief Technology Officer, the portal is visited by 140,000 users each month, including 50,000 new users.

The new datasets released in the update include NYPD complaint data on felonies, misdemeanors and violent crimes reported between 2006 and 2016; details of City Council participatory budgeting projects from 2012 onwards; data on the programs, benefits and resources for 40 health and human services available to New Yorkers; and a Department of City Planning database of more than 35,000 records on public and private facilities from 50 sources. Other new aspects of the program include legal mandates for compliance with FOIL requests and on timing of responses to data requests.

Though FOIL requests involving data are being streamlined, City Council Member Ben Kallos, a longtime advocate of open data, thinks that it can be improved further by passing his “Open FOIL” bill, which would create “one searchable database of Freedom of Information Law requests sent to city agencies.” Kallos also believes that the city could do more outreach about the existence of the open data initiative.

"The City is getting better and better at getting the word out about Open Data,” Kallos told Gotham Gazette. “I for one want to see Open Data classes taught at our city libraries so anyone can learn how to use the data sets, not just techies." Indeed, while many data sets are available, they aren’t always easy to digest or utilize to find patterns or other takeaways.

Hundreds of datasets are currently scheduled for release. While some of these datasets are collected and released monthly, others involve large studies that will be aggregated on the portal for the first time, including a study on poverty in New York City that augments national spending to reflect the high cost of living in the city. The mayor’s office will also release an “inventory” of greenhouse gas emissions in the city in the ten-year period covering 2005 to 2015 on September 30. Large amounts of wage data from the comptroller’s office will be aggregated on the portal come November 3, along with other data from the comptroller’s office such as an updated quarterly measure of the city’s debt limit. Participation data in the Supplemental Nutrition Assistance Program (SNAP) or food stamps from the Human Resource Administration will be aggregated on the portal in December.

The continually increasing number of datasets has been praised by elected officials and good government groups.

“Open Data is a fantastic resource for civic organizations, local businesses, and City residents,” said City Council Member James Vacca, chair of the City Council’s Committee on Technology, in a statement accompanying the recent release. “I’m very happy to see that the number of datasets continues increasing and that the Open Data Portal is being made even more accessible to New Yorkers. I congratulate the City’s Open Data team on these promising results.”

However, the process is not without its criticisms. John Kaehny, co-chair of the NYC Transparency Working Group and Executive Director of Reinvent Albany, a good government group, has praised the open data law but says that many city agencies are kicking the can down the road in releasing some datasets. Many datasets are scheduled for release in December 2018, when the law expires. He recommended that the mayor and City Council prod agencies to get the data out on time.

“It’s like a teacher telling a kid that they can turn all their homework in on the last day of school,” Kaehny told Gotham Gazette in an interview, “but gives them the entire year to do the homework. And then the kid’s like ‘great, I’m gonna turn in all my homework on the last day of school.’ Well, you believe that, you know. They’re gonna finish all their assignments and turn it in on the last day? That’s not credible.”

by Ben Brachfeld, Gotham Gazette

Read more by this writer.

City Releases Latest Progress on Open Data Plan Tue, 25 Jul 2017 04:00:09 +0000
Open Data Compliance the Focus at Oversight Hearing james vacca technology open data

City Council Member Vacca (photo: @JamesVacca13)

The New York City Council’s Committee on Technology held an oversight hearing Tuesday to evaluate the progress being made by the city in tracking compliance by government agencies with the city’s open data law.

The Mayor’s Office of Data Analytics (MODA) has been conducting what is essentially an audit, the “Examination and Verification Plan,” to gauge each agency’s status regarding its publication of required data sets on the Open Data Portal, an online access point meant to open government to the public.

This means public information on everything from motor vehicle collisions to wifi hotspot locations to leading causes of death, which, in theory, allows regular New Yorkers to better evaluate city services, gives civic technologists more to work with in acting as an outside of government check, and encourages problem-solving using data.

Tuesday’s hearing, chaired by City Council Member James Vacca, was the first on the examination and verification plan, which was adopted early in 2016. It showed that progress is underway in ensuring the quality of data and availability of the appropriate data sets, but raised questions about levels of agency participation, public engagement, and even what constitutes a public data set.

While the portal is not lacking in data -- over 1,500 data sets from 60 different government agencies are currently accessible -- Local Law 11 of 2012, the city’s first open data law, mandates that all agencies publish their respective public data sets in entirety by 2019.

MODA is employing the Examination and Verification Plan to ensure that each agency is on pace to meet that deadline. The plan calls for evaluation of three agencies per year. In 2016, the first year of the plan, the Department of Corrections, Department of Sanitation, and Department Housing Preservation and Development were selected for evaluation.

In his opening statement at Tuesday’s hearing, MODA Director and Chief Analytics Officer Dr. Amen Ra Mashariki spoke to the importance of examination and verification. “Its purpose is to improve citywide compliance by creating a more systematic way to locate data sets that may have inadvertently or purposefully been excluded from agencies’ self reported data,” Mashariki said, adding that all three agencies evaluated this year are “in good standing.”

As Mashariki alluded to in his statement, part of the impetus behind the open data law is to increase government transparency. The preamble to Local Law 11 states: “Making city data available online using open standards will make the operation of city government more transparent, effective and accountable to the public.”

The aggregation of agency data also benefits those within government. Representatives from MODA explained at Tuesday’s hearing that when government agencies are required to share data, they often have had to go through a manual process, uploading files onto a hard drive or disk. However, now that agencies are posting their data in the portal, other departments are able to access it without going through the antiquated data exchange process.

This could allow, for example, the Department of Transportation and the NYPD to access each other’s data more easily, which could aid the implementation of Vision Zero. As Mashariki put it, “There’s not only intra- but also inter-agency usage of data.”

While the hearing touched on a number of benefits of the open data initiative, Council Member Vacca, a Bronx Democrat who chairs the technology committee, did raise several concerns.

First, Vacca wanted to know when the examination and verification of the Police Department would be conducted. Crime statistics are frequently sought and police records remain among the most hotly debated in the city. Mashariki explained that while the NYPD is not scheduled for the examination and verification plan, MODA has been working with the department to publish and improve its data sets, some of which can be found in the portal. Mashariki added that while the NYPD is not currently on the books for evaluation, MODA “can begin looking into ramping up other agencies.”

Vacca was also generally curious about non-mayoral agencies, such as the New York City Housing Authority (NYCHA) and the Health + Hospitals Corporation, and their involvement in the examination and verification process. Mashariki explained that though those agencies are not held to the same data-release requirements as other agencies, MODA has been working to get their data on the portal, pointing out that Health + Hospitals has several data sets on the portal.

Vacca was particularly interested in traffic to the Open Data Portal, stressing that it is crucial to know how many people are making use of the newly available data. “Once we get actual numbers, we can decide whether or not we have to do more outreach and more engagement of community groups.”

While MODA could not provide definitive numbers, Craig Campbell, a Senior Advisor at MODA, did say that the most-viewed data set on the portal has received over five million hits (though he would not name the data set). Still, Dr. Mashariki firmly agreed that raising public awareness around the availability and advantages of open data must be a core objective. “We need aggressive engagement no matter what the number says,” he said.

One way in which MODA is striving to improve accessibility and engagement is through a new data portal website launching in March. The website, according to Mashariki, “makes it easier than ever to request new data sets on the open data portal.” Users will be able track the status of their requests and “get in touch with the open data team on anything from technical inquiries to partnership opportunities.” The website will also ensure that data dictionaries and geospatial fields (interactive maps) accompany all data sets.

Also testifying at the hearing was a representative from Reinvent Albany, a good government group focused on open data. The Reinvent representative, Dominic Mauro, largely commended MODA's work, but did raise concerns about whether an agency can be deemed compliant if it possesses data sets that are listed as public but which are not present in the portal, as is the case with the Department of Sanitation.

This concern prompted Mashariki to delve into the evolving nature of the open data effort, in particular the working definitions of what should be labeled as public data and would should remain confidential. “This process has helped us think a little smarter about what should or shouldn’t be a public data set,” he said. “The way we saw it is that the process is working if it’s forcing us to begin to think about what’s public, what’s private, and, to be very honest...I think that’s a conversation that should consistently be had.”

As it turned out, the status of the data sets in question was pending -- they had not been classified as either public or not -- which is why they had not made it onto the portal at the time of the hearing.

The public versus private determination that Mashariki referenced speaks to the conceptual as well as practical considerations involved in an undertaking like the open data initiative. Not only must MODA coordinate with agencies’ open data coordinators to aggregate data and ensure quality, they must also consider the spectrum of ramifications that come with publicizing certain information.

Local Law 11’s agency compliance plan does include the following criteria for public classification:

“For purposes of prioritizing public data sets, agencies shall consider whether information embodied in the public data set: (1) can be used to increase agency accountability and responsiveness; (2) improves public knowledge of the agency and its operations; (3) furthers the mission of the agency; (4) creates economic opportunity; or (5) responds to a need or demand identified by public consultation.”

But questions remain: Could some data sets be exploited to the detriment of New York and its residents? Does the potential for positive opportunity outweigh the potential for exploitation? What about privacy? For example, the Department of Corrections data contains a lot of Personally Identifiable Information on inmates which could potentially compromise an individual.

This question of definitions was echoed in the “2016 Examination and Verification Findings Report” issued by MODA on December 1. “Creating a list of eligible datasets is not a cut and dried process, and raised questions regarding the definitions of ‘public,’ ‘data,’ and ‘dataset,’” the report states. It also says that MODA will be working with the Department of Investigations to more clearly define these terms and develop a strategy for classification moving forward.

by Luke Foley, Gotham Gazette

Open Data Compliance the Focus at Oversight Hearing Tue, 24 Jan 2017 05:00:00 +0000
Praise, and Recommendations, for City's Open Data Progress vacca tech open data cropped

Council Member Vacca (photo: @JamesVacca13)

There are two years to go before the deadline set by the city’s 2012 Open Data Law, which requires city agencies identify and publish all their citywide aggregated public data online. At a Wednesday afternoon oversight hearing of the City Council’s Committee on Technology, open data and civic technology advocates offered praise for the city’s progress thus far and recommendations for the city to meet its goals.

Open data is considered a vital transparency tool that allows citizens and government to evaluate and improve city services. New York City, considered a trendsetter in this area, committed to this goal in 2012 by passing a law that would make data easily accessible and consumable by the public. Through the city’s Open Data Portal, New Yorkers can access a range of information such as the number and type of complaints made to the 311 helpline, real-time traffic speed data, or where the City Council spends discretionary funding.

The Open Data Portal hosts nearly 1,600 datasets across all city agencies.

Last year, in the annual progress report on the city’s implementation of the Open Data Law,  Mayor Bill de Blasio released a new open data plan, “Open Data For All,” which aims at making the Open Data Portal more user-friendly, making it easily visualized for those with little experience in data or programming, and pushes for more citywide engagement.

Following that, the City Council passed a package of legislation to enhance the existing law and improve access to information through the city’s Open Data Portal maintained by the Department of Information Technology and Telecommunications (DoITT).

This was the first year which saw some of the new amendments put into effect. These new laws, seven in total, deal with retention of data, building data dictionaries, standardizing geospatial data, establishing timelines for agencies to respond to public requests on the Open Data Portal, mandating timely updates for public data, requiring publication of datasets obtained through Freedom of Information Law requests, and examination and verification of agency compliance.

“One of the things we’ve been able to do is build an open data ecosystem,” testified Dr. Amen Ra Mashariki, director of the Mayor’s Office of Data Analytics (MODA), who is also the chief analytics officer and chief open platform officer for the city. He said the new laws help “anchor the administration’s commitment to transparency and equitable uses of technology around open data.”

Ra Mashariki stressed that MODA’s focus has been on user research, feedback mechanisms, and technical standards, and he laid out the city’s progress on each while teasing updates that will be released later this year. These updates include a first-of-its-kind analysis with NYU Center for Urban Science and Progress on “data poverty” -- the lack of access or representation of communities in city data; a study with Columbia University’s School of International and Public Affairs on opportunities for community based organizations to utilize open data; and a new centralized mechanism for receiving feedback, with quicker responses and tracking.

Concluding his testimony, Ra Mashariki cited an important example of how open data can be useful to the public. He referenced Ben Wellington, a data scientist who runs the blog “I Quant NY.” Wellington examined parking violation data and found that the NYPD had been ticketing legally parked cars for millions of dollars. He took his analysis to the NYPD who confirmed it and took measures to prevent it in the future.

Testifying on behalf of DoITT, Albert Webber cited the city’s key accomplishments over the last year -- traffic on the Open Data Portal increased to more than 5 million hits; 116 datasets were made public, bringing the total to nearly 1,600; 100 new data sets are now automatically updated; and more than 40 unscheduled data sets were identified and published. He also said DoITT would soon fill three of four new dedicated positions for the Open Data Portal.

At the hearing, government transparency advocates and civic technology professionals wholeheartedly endorsed the city’s efforts, and also made a few recommendations for improvements.

John Kaehny, executive director of Reinvent Albany and co-chair of the New York City Transparency Working Group, emphasised “that City Council stay actively and energetically involved in pushing, cajoling and cheerleading for open data in this city,” without which the program can not be a success.

The Transparency Working Group is co-chaired by Gene Russianoff, senior attorney at the New York Public Interest Research Group, and its members include Common Cause New York and Citizens Union. “Open data and the idea behind open data is working in New York City,” Kaehny said, also stressing that city agencies will eventually save tens of millions of dollars by using open data.

He said the de Blasio administration has “started upping their game” in implementing open data but in his testimony Kaehny pointed out a number of outstanding issues. These included the need for a better process for remedying public complaints about data quality, more standardization of data, clearer agency criteria for publishing data, and publishing high-demand data sets, such as NYPD crime statistics or the Department of Transportation’s street paving schedule.

He also suggested other needed improvements to the overall system and the law. Currently, one of the new laws requires publishing the number of datasets requested under FOIL -- a method often used by advocates and journalists to examine city information. These, he said, should also be named. He said the city should also consider creating a data issues tracker to monitor errors in datasets reported by the public, similar to a tool used by the federal government.

Council Member James Vacca, chair of the technology committee, was generally satisfied with the administration although he raised a few concerns at the hearing. He was especially pleased that one dataset to be added soon will be of snowplow locations, updated every hour, which would give people quick answers during snowstorms when the city usually struggles to clear roads across the five boroughs. He did question the administration on whether certain agencies were fully complying with the law, particularly the New York Police Department.

Vacca also wondered later whether open data use was being taught to local community groups and Community Boards, which would then have a stronger ability to advocate for local issues. “I know communities that always say ‘We want more police, we want more traffic agents...but they don’t have the documentation, they don’t do the research,” he said. “Here it’s at their fingertips, here we can do the research.”

DoITT’s Webber said that agencies “have made strides across the board,” and that the NYPD had added a number of datasets to its website and the Open Data Portal links to them. “Overall I wouldn’t say any particular agency is not committed to doing what they’re supposed to do,” Webber said, adding that he would later follow up with the Council Member with specific NYPD datasets.

For civic technology advocates, the major concern is not only the availability of data but its usefulness to the public.

Noel Hidalgo, executive director of BetaNYC, a civic technology and open government group, made this clear even as he praised the administration and the Council for their push on open data. “The seven pieces of legislation that were passed are taking us in the right direction and have really strengthened the city’s open data practice,” he said. But Hidalgo pointed out that one big weakness, echoing Vacca’s words, is figuring out how to teach community groups to access, understand and utilize open data. “Frustratingly, there are zero best practices out there on how to teach the city’s open data to itself,” Hidalgo said.

Hidalgo added that under the provisions of the new laws passed last year, the data dictionaries need to contain tutorials and that there needs to be easier access to bulk data downloading. Finally, he stressed the need for the city to create an open data legacy that can last through successive administrations and City Council classes, with a unit dedicated to leadership, standardization, development, and education.

There was also consensus at the hearing among advocates that the very infrastructure of the Open Data Portal could be modernized. Joel Natividad, director of open data at OpenGov Inc., said the portal, currently run by a contracted vendor, needs to be an open source platform that can “support innovation and experimentation” by universities, non-governmental organizations and the private sector. This will allow the city to “tap the genius of the crowd,” he said. Kaehn, of Reinvent Albany, agreed, saying the current platform has severe limitations.

by Samar Khurshid, senior reporter, Gotham Gazette

Read more by this writer.

Praise, and Recommendations, for City's Open Data Progress Wed, 21 Sep 2016 04:00:00 +0000
Push for a Better Mayor's Management Report Continues 311-cust-serv 

The City Council’s Committee on Governmental Operations held an oversight hearing on Wednesday to evaluate the structure and content of two annual city reports that measure everything from the wait times on 311 calls and the conditions of city parks to the number of unsheltered homeless people and traffic crashes in the city each year. These key performance indicators help the City Council and the public keep city agencies and offices accountable. Those agencies, in turn, rely on these reports to improve their policies and functioning.

The administration releases two detailed reports each year - the Preliminary Mayor’s Management Report (PMMR), two weeks after the January financial plan, and the Mayor’s Management Report (MMR), in September. The reports include evaluations of annual goals set by 41 city agencies and 3 non-mayoral agencies that report directly to the Mayor, and whether these agencies are succeeding or failing to meet their objectives.

Wednesday’s hearing, the third one held by the governmental operations committee on this issue, gave Council members an opportunity to delve into the process of how agency goals are included in the reports and to reiterate their request for adding more data to future reports. Council Member Ben Kallos, chair of the committee, highlighted some of the key concerns the Council has about a lack of clarity in certain agency goals, agency indicators with no set targets, and the need to directly link agency performance with corresponding budget allocations.

The PMMR and MMR are produced over six to eight weeks by the Mayor’s Office of Operations (MOO). About 10 MOO staff members work with more than 150 senior staff across the agencies, including deputy mayors and agency commissioners, and coordinate efforts with City Hall to release the final products, which detail agency goals and indicators of whether those goals are being met year after year.

There are 1649 metrics by which agency performance is tracked in the two reports. Examples include quality-of-life summonses issued by the New York Police Department, the number of people on the city’s cash assistance rolls, the average daily attendance in city schools, and the number of families entering the city’s homeless shelter system.

The report tracked the NYPD’s goal to reduce the number of quality-of-life violations, a critical indicator without a set target. The four-month numbers (the timeframe for the PMMR) fell from 142,434 in Fiscal 2015 to 125, 685 in Fiscal 2016.

Under the Department of Education’s performance, the report looked at average daily attendance in city schools, which surpassed the 91.7 percent target in Fiscal 2016 (93.2 percent) and Fiscal 2015 (93.5 percent). Similarly, the report showed that the Department of Homeless Services had been successful in reducing the number of adult families entering the shelter system from 521 families to 466.

The data, which is available on an interactive website, includes comparisons with previous years, five-year trends and a look at desired directions for performance. This year’s PMMR data was also available on the city’s Open Data Portal, providing public access to all the information collected under the report. Kallos, an ardent supporter of transparent and accessible information and technology, praised MOO for this effort.

“We really take care not to take a one-size-fits-all approach to targets and indicators in the PMMR and MMR,” said Mindy Tarlow, MOO director, at one point during Wednesday’s hearing. “We try to see each indicator and each agency as a complex diverse case that’s in some ways reflective of the complex diverse city that we’re monitoring and reporting on.”

City Council members took turns questioning Tarlow and her colleague Tina Chiu, deputy director for performance management. The members usually stuck with issues germane to issue committees they chair or sit on, but the larger concerns voiced at the hearing were about the lack of targets for many critical performance indicators and the ways in which targets are set and the reports are improved.

Council Member Ben Kallos, who chairs the government operations committee, commended MOO for changes they had made to the PMMR based on recommendations from a December 2015 hearing about how targets were defined in the reports. This year’s PMMR included a more nuanced description of targets, which can be numeric or directional. Numeric targets set expected performance levels including a maximum that should not be exceeded or a minimum level that should be met. Directional targets indicate whether an indicator should go up or down. Many indicators had no target at all, which gave cause for concern to Kallos and Dick Dadey, executive director of government reform group Citizens Union, who testified at the hearing.

When Kallos asked why certain indicators had no targets, citing the number of lawsuits brought against the city as an example, Tarlow clarified that many of these indicators are “neutral” statistics out of their control, such as certain population numbers, for which it’s impossible to set directional targets.

“Although the committee is pleased that some improvements have been made,” Kallos said in his opening statement, “our review of the most recent PMMR suggests that further changes could make both of these publications more helpful tools for the Council, the public, as well as  the agencies.”

The City Council, in its Monday response to the mayor’s preliminary budget for fiscal year 2017, identified 57 indicators across 18 agencies that they want included in the MMR going forward. Kallos brought this up at the hearing, specifically pointing to indicators for the Board of Standards and Appeals, which falls under the Department of Citywide Administrative Services, which is under Kallos’ committee’s oversight. He also mentioned a City Council data study that found 107 indicators (70 critical ones) where there was a 28 percent discrepancy over last year’s PMMR and average performance over the prior three years.

kallos rules hearingKallos (pictured*) also questioned Tarlow and Chiu about indicators relating to the Department of Health and Mental Hygiene, the 311 helpline system and the Department of Homeless Services, but his main concerns were about holding people accountable for changes to the PMMR and MMR, further clarifying how targets are defined, minimizing the number of indicators without targets, and linking performance indicators in the PMMR to the corresponding allocations in the preliminary budget, as the City Charter mandates for the PMMR.

“Who has ultimate responsibility for the changes that are being implemented?” Kallos asked Tarlow, frustrated that Council members’ requests for additional indicators in the past had been routinely agreed to by agencies but not followed through on. “Is it with the agency or Operations, and how do those changes actually end up happening?

Tarlow stressed that the creation and addition of indicators was a collaborative process with agencies. “We don’t dictate play to the agencies and the agencies don’t just unilaterally make changes,” she said, agreeing to consider improvements in communication with Council members. She also said that they had started to work with the Mayor’s Office of Management and Budget to cross-reference and link the PMMR with the preliminary budget.

Kallos’ concerns were echoed by Dadey, who reiterated recommendations that Citizens Union had made in the past, which shows, he said, “a challenge that we face that we’ve been back here several times making the same recommendations and while some of them get adopted we think that some of the more common sense ones are not being embraced and we’re at a loss to understand why.”

Dadey floated three specific proposals: setting targets for half the indicators in the report which had no metrics to judge them by (the number of indicators with no target increased from 948 to 961 this year from last); providing more budget detail alongside performance indicators; and expanded reporting on cross-agency initiatives to track transparency (for instance Freedom of Information Law requests) and voting registration programs at agencies.

He called the lack of targets for half the proposals “disconcerting” as “it indicates one of two troubling possibilities. Either the agency experienced difficulty in setting goals in coordination with the office of the mayor or that these goals have been established and are possibly being concealed from the public. Neither is satisfactory,” he said.

To the administration’s credit, MOO has made effort to add sections on multi-agency initiatives, such as IDNYC, which was eventually folded into the report on the Human Resources Administration. “That’s what we want,” Tarlow said in response to questions from Council Member Carlos Menchaca. “To really feature a multi-agency new initiative and then have it be so routine that it becomes baked into the fabric of what we do on a day-to-day basis.”

Menchaca wanted to know how the PMMR and MMR reports evolve and about efforts to make the report machine readable. His larger concern, as chair of the Council’s immigration committee, was that the data be accessible to New York’s immigrant communities. “What do you do to get the report out in other languages?,” he asked.

Tarlow’s answer was unimpressive. “I actually don’t know,” she said.

Lisa Neary, general counsel for the Independent Budget Office, also testified, although less about the content of the reports than the process. IBO had testified at the December 2015 hearing as well, recommending legislation that would require citizen surveys to be included in the reports. The idea has been championed for a long time by Baruch Professor Doug Muzzio and a relevant bill has been introduced by Council Member Corey Johnson, though it has not received a vote.

On Wednesday, Neary focused on the timing of the two reports. She said the PMMR being released early in the year was inadequate since it only accounts for the first four months of the financial year (July to October).

“With only this partial picture in hand,” Neary said, “the Council lacks crucial information that would allow you to link objectives to resources, and resources to outcomes.” She cited the current PMMR’s data on Department of Education programs, where at least a dozen critical indicators were listed as “not available.” She said the MMR’s release was “even more poorly timed,” and should be some time between January and June to have maximum influence on budget decisions.

When Kallos asked her for suggestions, she said perhaps the MMR should be released with the executive budget in May and then a full accounting of the year should come out in October or November. “The key thing is to have as much information as you can at the time that you’re making resource allocation decisions,” she said.

Council Member Mark Levine, chair of the parks committee, questioned Tarlow and Chiu about PMMR indicators on park maintenance and capital park projects, as well as on the lack of information on eviction prevention and legal services. Council Member Vincent Gentile, in keeping with his position as chair of the oversight and investigations committee, asked MOO to consider tracking the performance of individual Inspectors General under the Department of Investigation. Tarlow had ready answers for both Council members and promised to discuss their concerns with individual agencies.

by Samar Khurshid, senior reporter, Gotham Gazette

Read more by this writer.

*Photo of Ben Kallos by William Alatriste

Note: Gotham Gazette is an independent publication of Citizens Union Foundation, sister organization to Citizens Union.

Push for a Better Mayor's Management Report Continues Thu, 07 Apr 2016 19:34:38 +0000