Gotham Gazette Gotham Gazette is an online publication covering New York policy and politics as well as news on public safety, transportation, education, finance and more. http://www.gothamgazette.com/component/tags/tag/committee-on-technology 2017-10-23T18:50:52+00:00 Webmaster webmaster@gothamgazette.com City Council Hears Testimony on 'Open Data 2.0' Bills 2017-09-21T01:19:55+00:00 2017-09-21T01:19:55+00:00 http://www.gothamgazette.com/?id=7209:city-council-hears-testimony-on-open-data-2-0-bills Samar Khurshid <p><img src="http://www.gothamgazette.com/images/c93-69luiaaiaii.jpg-large.jpeg" alt="c93 69luiaaiaii.jpg large" width="600" /></p> <p>Council Member James Vacca (right)</p> <hr /> <p>At an oversight hearing on Wednesday of the City Council’s Committee on Technology, good government and civic technology experts praised a pair of bills aimed at updating the city’s 2012 open data law, but raised concerns that the proposals could have unforeseen ramifications.</p> <p>The city’s current law mandates that city agencies publish non-confidential agency data on a public web portal for all New Yorkers to see and easily access. The <a href="https://opendata.cityofnewyork.us" target="_blank" rel="noopener noreferrer">open data portal</a> is now home to over 1,600 data sets, with hundreds more data sets scheduled to be released in the coming years. The two “Open Data 2.0” proposals, sponsored by Council Member James Vacca, chair of the committee, and supported by Manhattan Borough President Gale Brewer, are intended to bolster open data reporting standards as well as improve certain areas of the current policy. “Access to government information ensures public institutions can be held accountable,” said Vacca, in his opening statement. “In New York City, the importance of public access to data is reflected in both law and practice.” He noted that New York City had been the first municipality in America to pass a law requiring public data be posted online in machine readable formats.</p> <p>The hearing was a bittersweet occasion for Vacca, who is term-limited and will be leaving the City Council in December after 12 years. It was his final oversight hearing on open data policy, a project he has long championed.</p> <p>The city’s open data policy is seen as a model not just for the United States, but the entire world, Brewer noted in her opening statement at the hearing. “I just met with elected officials in Spain... And they are looking at our law,” Brewer said. To illustrate one of the uses of open data, Brewer will be joining civic technology group BetaNYC to host a “311 Data Jam” on Saturday, employing the city’s dataset on the 311 helpline. It’s the third such event hosted by BetaNYC, and they will be introducing “BoardStat,” a tool for community boards to analyze visualized 311 data in order to discover community trends.</p> <p>The bulk of Wednesday’s hearing covered <a href="http://legistar.council.nyc.gov/LegislationDetail.aspx?ID=3136796&amp;GUID=0144E84F-0BEC-4DB9-ABC6-B564DE4F717C&amp;Options=&amp;Search=" target="_blank" rel="noopener noreferrer">Intro. 1707</a>, which includes provisions requiring reviews of the “Technical Standards Manual,” which is the guideline for how agencies manage and present their datasets every two years, to ensure the datasets remain up to date.</p> <p>“These standards are designed to make open data more usable to the maximum number of users,” said Albert Webber, director of Open Data for the Department of Information Technology and Telecommunications (DoITT). “We see the TSM as a living document, meant to keep up with advances in technology, data availability, and the passage of local laws that impact data disclosure. An official periodic review of this document would ensure that open data disclosure stays up-to-date, relevant, and true to the most current practices.”</p> <p>The bill codifies a requirement for city agencies to appoint an “open data coordinator” to ensure agency compliance with the open data law. It also changes the deadline for agencies’ compliance plans from July to September, to avoid conflicting with the city’s budget season. The bill would also require that data analytics on public usage of the data portal be collected and studied.</p> <p>Criticisms of the legislation stemmed not from opposition to any one plank, but rather from unintended consequences that certain provisions of Intro. 1707 could have in areas such as agency reporting, accountability, and protection of individual privacy.</p> <p>One provision in the proposal raised eyebrows among good government groups and civic technologists. Noel Hidalgo, executive director of Beta NYC, and John Kaehny, executive director of Reinvent Albany, pointed out that the bill extends the open data compliance deadline for city agencies from 2018 to 2021, a full nine years after the law went into effect in 2012. Many datasets from numerous city agencies are currently scheduled for release on the current deadline, December 31, 2018.</p> <p>Kaehny said allowing agencies to hold off on publishing data is akin to a high school student saying they’ll turn in all of their homework on the last day, a promise that can’t be seen as reliable. He made similar comments to Gotham Gazette <a href="http://www.gothamgazette.com/city/7086-city-releases-latest-progress-on-open-data-plan" target="_blank" rel="noopener noreferrer">in July</a> after the city released its annual update to the open data plan. Kaehny referred to the extension to 2021 as a “homework extension” that will never be turned in.</p> <p>Vacca pressed Webber on these concerns, and Webber insisted that the administration was committed to releasing the data sets on time. “Putting a deadline is always helpful,” he told the Council member. “But no, we do not have concerns that pushing it to 2021 will delay the release of some datasets.” In his testimony, Webber stated that the administration would “like to discuss how to improve the language to ensure that submissions meant to be disclosed by December 31, 2018 will still be held to that deadline.”</p> <p>Another aspect of Intro. 1707 that received some critique was a provision calling for the collection of data analytics on visitors to the portal. Concerns were raised that recording data on unique visitors to the portal, especially those exploring data that could be construed as sensitive, might violate an individual’s privacy. The language of this provision in the bill was viewed as vague.</p> <p>“Pageviews, great,” said Sumana Harihareswara, a programmer from Astoria who testified at the hearing. “Numbers of pageviews, great. Unique users? I don’t want to be identified uniquely in a public place for having looked up a certain piece of data. So number of unique users is perhaps what this ought to say.”</p> <p>Those who testified otherwise generally praised the proposed biannual review of the TSM, the change in the annual compliance deadline from July to September, and codifying into law the agency requirement of an open data coordinator.</p> <p>The bill also saw some critique for what it lacked, such as a “private right of action,” the ability of the public to sue agencies failing to perform their duties. Kaehny noted that the only course of action without the private right of action is to “name and shame” agencies considered to be “laggard” in regards to their dataset releases. According to Kaehny, open data is one of the few pieces of legislation in New York City that does not include a private right of action, making it highly unusual, especially for such a major city law.</p> <p>Brewer echoed the call for a private right of action, emphasizing that future administrations may not be as committed to transparency. “I believe we must provide a private right-of-action to protect the New York City municipal open data apparatus from a future administration that does not wish to operate it in good faith as the current and previous administrations have done,” she said. She cited the Trump administration’s moves immediately post-inauguration to delete data that had been posted online under the previous administration; there is no federal law that establishes similar data protections as New York City’s policy.</p> <p>Kaehny claimed that even naming and shaming is a difficult task, given that agency compliance with the law is not compiled in a single spreadsheet on the portal but rather in four separate spreadsheets. Compiling all relevant information about published and unpublished datasets into one spreadsheet was one of Reinvent Albany’s key recommendations to the committee. “What we’re seeing is fog when it comes to understanding exactly how well agencies are doing,” Kaehny said.</p> <p>The administration also had suggestions for improving the bill. Echoing testimony from the advocates, DoITT’s Webber expressed dismay at the law’s lack of any licensing requirements for the data, hoping that “permissive licenses,” licenses with few, if any, redistributive restrictions, could be added to the program.</p> <p>“Our mission to encourage public engagement with open data in the long term makes the ability to license very important,” Webber said. “Structured properly, licenses could allow us to formally make the city’s datasets free and open for public use in perpetuity, codifying our existing practice and ensuring that New Yorkers have access to open data for the long haul.”</p> <p>Also under consideration at the hearing was <a href="http://legistar.council.nyc.gov/LegislationDetail.aspx?ID=2984659&amp;GUID=6B734B2A-47FA-49C7-BE83-C7C640B098D2&amp;Options=&amp;Search=" target="_blank" rel="noopener noreferrer">Intro. 1528</a>, an amendment to an existing open data law that would require agencies that process Freedom of Information Law requests involving non-confidential data to report and publish the names of the public datasets, rather than simply the metrics of the datasets. The measure received overall support from those testifying before the committee.</p> <p>

***
by Ben Brachfeld, Gotham Gazette

Read more by this writer.

</p>
<p><img src="http://www.gothamgazette.com/images/c93-69luiaaiaii.jpg-large.jpeg" alt="c93 69luiaaiaii.jpg large" width="600" /></p> <p>Council Member James Vacca (right)</p> <hr /> <p>At an oversight hearing on Wednesday of the City Council’s Committee on Technology, good government and civic technology experts praised a pair of bills aimed at updating the city’s 2012 open data law, but raised concerns that the proposals could have unforeseen ramifications.</p> <p>The city’s current law mandates that city agencies publish non-confidential agency data on a public web portal for all New Yorkers to see and easily access. The <a href="https://opendata.cityofnewyork.us" target="_blank" rel="noopener noreferrer">open data portal</a> is now home to over 1,600 data sets, with hundreds more data sets scheduled to be released in the coming years. The two “Open Data 2.0” proposals, sponsored by Council Member James Vacca, chair of the committee, and supported by Manhattan Borough President Gale Brewer, are intended to bolster open data reporting standards as well as improve certain areas of the current policy. “Access to government information ensures public institutions can be held accountable,” said Vacca, in his opening statement. “In New York City, the importance of public access to data is reflected in both law and practice.” He noted that New York City had been the first municipality in America to pass a law requiring public data be posted online in machine readable formats.</p> <p>The hearing was a bittersweet occasion for Vacca, who is term-limited and will be leaving the City Council in December after 12 years. It was his final oversight hearing on open data policy, a project he has long championed.</p> <p>The city’s open data policy is seen as a model not just for the United States, but the entire world, Brewer noted in her opening statement at the hearing. “I just met with elected officials in Spain... And they are looking at our law,” Brewer said. To illustrate one of the uses of open data, Brewer will be joining civic technology group BetaNYC to host a “311 Data Jam” on Saturday, employing the city’s dataset on the 311 helpline. It’s the third such event hosted by BetaNYC, and they will be introducing “BoardStat,” a tool for community boards to analyze visualized 311 data in order to discover community trends.</p> <p>The bulk of Wednesday’s hearing covered <a href="http://legistar.council.nyc.gov/LegislationDetail.aspx?ID=3136796&amp;GUID=0144E84F-0BEC-4DB9-ABC6-B564DE4F717C&amp;Options=&amp;Search=" target="_blank" rel="noopener noreferrer">Intro. 1707</a>, which includes provisions requiring reviews of the “Technical Standards Manual,” which is the guideline for how agencies manage and present their datasets every two years, to ensure the datasets remain up to date.</p> <p>“These standards are designed to make open data more usable to the maximum number of users,” said Albert Webber, director of Open Data for the Department of Information Technology and Telecommunications (DoITT). “We see the TSM as a living document, meant to keep up with advances in technology, data availability, and the passage of local laws that impact data disclosure. An official periodic review of this document would ensure that open data disclosure stays up-to-date, relevant, and true to the most current practices.”</p> <p>The bill codifies a requirement for city agencies to appoint an “open data coordinator” to ensure agency compliance with the open data law. It also changes the deadline for agencies’ compliance plans from July to September, to avoid conflicting with the city’s budget season. The bill would also require that data analytics on public usage of the data portal be collected and studied.</p> <p>Criticisms of the legislation stemmed not from opposition to any one plank, but rather from unintended consequences that certain provisions of Intro. 1707 could have in areas such as agency reporting, accountability, and protection of individual privacy.</p> <p>One provision in the proposal raised eyebrows among good government groups and civic technologists. Noel Hidalgo, executive director of Beta NYC, and John Kaehny, executive director of Reinvent Albany, pointed out that the bill extends the open data compliance deadline for city agencies from 2018 to 2021, a full nine years after the law went into effect in 2012. Many datasets from numerous city agencies are currently scheduled for release on the current deadline, December 31, 2018.</p> <p>Kaehny said allowing agencies to hold off on publishing data is akin to a high school student saying they’ll turn in all of their homework on the last day, a promise that can’t be seen as reliable. He made similar comments to Gotham Gazette <a href="http://www.gothamgazette.com/city/7086-city-releases-latest-progress-on-open-data-plan" target="_blank" rel="noopener noreferrer">in July</a> after the city released its annual update to the open data plan. Kaehny referred to the extension to 2021 as a “homework extension” that will never be turned in.</p> <p>Vacca pressed Webber on these concerns, and Webber insisted that the administration was committed to releasing the data sets on time. “Putting a deadline is always helpful,” he told the Council member. “But no, we do not have concerns that pushing it to 2021 will delay the release of some datasets.” In his testimony, Webber stated that the administration would “like to discuss how to improve the language to ensure that submissions meant to be disclosed by December 31, 2018 will still be held to that deadline.”</p> <p>Another aspect of Intro. 1707 that received some critique was a provision calling for the collection of data analytics on visitors to the portal. Concerns were raised that recording data on unique visitors to the portal, especially those exploring data that could be construed as sensitive, might violate an individual’s privacy. The language of this provision in the bill was viewed as vague.</p> <p>“Pageviews, great,” said Sumana Harihareswara, a programmer from Astoria who testified at the hearing. “Numbers of pageviews, great. Unique users? I don’t want to be identified uniquely in a public place for having looked up a certain piece of data. So number of unique users is perhaps what this ought to say.”</p> <p>Those who testified otherwise generally praised the proposed biannual review of the TSM, the change in the annual compliance deadline from July to September, and codifying into law the agency requirement of an open data coordinator.</p> <p>The bill also saw some critique for what it lacked, such as a “private right of action,” the ability of the public to sue agencies failing to perform their duties. Kaehny noted that the only course of action without the private right of action is to “name and shame” agencies considered to be “laggard” in regards to their dataset releases. According to Kaehny, open data is one of the few pieces of legislation in New York City that does not include a private right of action, making it highly unusual, especially for such a major city law.</p> <p>Brewer echoed the call for a private right of action, emphasizing that future administrations may not be as committed to transparency. “I believe we must provide a private right-of-action to protect the New York City municipal open data apparatus from a future administration that does not wish to operate it in good faith as the current and previous administrations have done,” she said. She cited the Trump administration’s moves immediately post-inauguration to delete data that had been posted online under the previous administration; there is no federal law that establishes similar data protections as New York City’s policy.</p> <p>Kaehny claimed that even naming and shaming is a difficult task, given that agency compliance with the law is not compiled in a single spreadsheet on the portal but rather in four separate spreadsheets. Compiling all relevant information about published and unpublished datasets into one spreadsheet was one of Reinvent Albany’s key recommendations to the committee. “What we’re seeing is fog when it comes to understanding exactly how well agencies are doing,” Kaehny said.</p> <p>The administration also had suggestions for improving the bill. Echoing testimony from the advocates, DoITT’s Webber expressed dismay at the law’s lack of any licensing requirements for the data, hoping that “permissive licenses,” licenses with few, if any, redistributive restrictions, could be added to the program.</p> <p>“Our mission to encourage public engagement with open data in the long term makes the ability to license very important,” Webber said. “Structured properly, licenses could allow us to formally make the city’s datasets free and open for public use in perpetuity, codifying our existing practice and ensuring that New Yorkers have access to open data for the long haul.”</p> <p>Also under consideration at the hearing was <a href="http://legistar.council.nyc.gov/LegislationDetail.aspx?ID=2984659&amp;GUID=6B734B2A-47FA-49C7-BE83-C7C640B098D2&amp;Options=&amp;Search=" target="_blank" rel="noopener noreferrer">Intro. 1528</a>, an amendment to an existing open data law that would require agencies that process Freedom of Information Law requests involving non-confidential data to report and publish the names of the public datasets, rather than simply the metrics of the datasets. The measure received overall support from those testifying before the committee.</p> <p>

***
by Ben Brachfeld, Gotham Gazette

Read more by this writer.

</p>
City Council Examines Data Privacy, Security 2017-04-26T04:00:00+00:00 2017-04-26T04:00:00+00:00 http://www.gothamgazette.com/?id=6892:city-council-examines-data-privacy-security Ben Max <p><img src="http://www.gothamgazette.com/images/graphics/2017/vacca_prep_computer.jpg" alt="vacca prep computer" width="600" height="465" /></p> <p>City Council Member Vacca (seated)</p> <hr /> <p>The Technology Committee of the New York City Council held an oversight hearing Monday on the city’s data privacy policies and data security protocols. A broad variety of more specific topics were covered during the hearing, chaired by Council Member James Vacca, including unauthorized data usage and security breaches; risks related to the expansion of high-speed internet availability and sidewalk wi-fi kiosks; the Worker Connect program that allows New Yorkers to search and apply for public benefits; new efforts at data exchange among city agencies; and the ‘Internet of Things,’ which refers to interconnectivity of various applications, devices, and information.</p> <p>As expected, a number of questions were raised at the hearing, only some of which were answered, about data privacy for New Yorkers and data security for city agencies.</p> <p>“I do think we came across some needs to have more of a formalized process on how we protect people’s privacy,” Vacca said after the hearing. “There should be an agency that’s tasked with this specific issue of privacy.”</p> <p>The main city agency tasked with data privacy concerns and security is the Department of Information Technology and Telecommunications, or DoITT. Commissioner Anne Roest testified Monday and told Vacca and the committee that the city is constantly experiencing hacking attacks. Since the current administration took office in January 2014, there were two instances when city data was stolen, Roest said, once from the Fire Department and once from the Administration for Children’s Services.</p> <p>“We’ve had, and I don’t have a number in front of me, but hundreds in the past year, of incidents where malware got into our system, but didn’t get anything out,” Roest said, emphasizing the quality of agencies cybersecurity threat response and prevention.</p> <p>A substantial part of the hearing revolved around LinkNYC, a network of interactive kiosks spread throughout the city, often where payphones used to be, providing free, fast wi-fi, access to various public services, and free phone calls anywhere in the United States. In January 2017, LinkNYC had over 1 million subscribers. Meanwhile, the <a href="https://data.cityofnewyork.us/dataset/LinkNYC-Usage-Statistics/69wu-b929/data" target="_blank" rel="noopener noreferrer">latest data</a> published on the city’s Open Data Portal indicates over 7 million sessions for the week of April 10. &nbsp;</p> <p>While the Link network is being used and expanded, there are privacy concerns. DoITT takes pride in the latest version of the privacy policy negotiated with the franchise partner, CityBridge LLC. Roest emphasized in her opening statement that LinkNYC does not track users’ browsing history and location, and therefore does not store or sell it.</p> <p>According to the privacy policy published on the LinkNYC <a href="https://www.link.nyc/privacy-policy.html" target="_blank" rel="noopener noreferrer">website</a>, the software does collect unique information about the device and browser being used, such as, for example, IP and MAC addresses, time zone settings, and other data (According to the privacy policy, this information is needed by LinkNYC to be able to provide internet connectivity). The storage period for this information varies from 60 days to a year. Each booth is also equipped with two cameras used to prevent or track vandalism, Roest said. The video is stored for seven days and then deleted. &nbsp;</p> <p>“Many national studies indicate that lower income people are disproportionately burned by data collection and privacy violations,” said Vacca in his opening statement at the hearing. He continued: “Last month Attorney General Eric Schneiderman announced that his office received approximately 1,300 notifications regarding data breaches of non-governmental systems in 2016. That is a 60 percent increase from the previous year.”</p> <p>“Privacy is not a luxury item, but a fundamental right of all people. With the recent federal repeal of the data privacy measures for internet service providers this discussion takes on added importance,” Vacca said about the decision to roll back a set of privacy regulations that obliged Internet providers to request consent from users before sharing and selling sensitive information, such as, for instance, healthcare and financial data, obtained by tracking their browsing behavior.</p> <p>The city’s Chief Technology Officer, Miguel A. Gamiño Jr., added during testimony that his office is planning to conduct a legal review to investigate the city’s potential to protect the privacy of New Yorkers, given ongoing reconsideration of Obama-era FCC regulations. “Specifically, this review includes an evaluation of the authority the city may have over the privacy policies of internet service providers, how and to what extent the city is exercising this authority currently, and whether the city can expand the exercise of its available authority to achieve, at minimum, the privacy protections for internet service consumers that Congress and the President have recently repealed,” said Gamiño. &nbsp;&nbsp;&nbsp;&nbsp;</p> <p>Officials identified public education as one of the key measures in preventing hacking attacks and data breaches in New York. Gamiño announced an “interagency work group on internet health and human rights” that will look for ways to educate consumers and businesses on how to protect their privacy. He also mentioned the city’s cooperation with public libraries on digital literacy materials for citizens and collaboration with Mozilla Foundation on a series of digital security workshops for organizations. Mozilla, a technology company known for its Firefox web browser, runs a nonprofit foundation that promotes open and accessible internet.</p> <p>Another topic covered by several meeting participants was an interagency data exchange strategy. Better data exchange among city agencies can improve transparency, efficiency, and broader accessibility to information for New Yorkers, but necessitates establishment of security and privacy protocols.</p> <p>DoITT Commissioner Roest discussed the importance of classifying collected data and applying safety protocols to the resulting categories. For instance, all data characterized as confidential cannot be stored or transferred unless it is protected by a special encryption technology.</p> <p>Finally, the Council committee discussed <a href="https://iot.cityofnewyork.us/" target="_blank" rel="noopener noreferrer">existing guidelines </a>and a future strategy for the ‘Internet of Things’, where New York City claims to be one of the global pioneers, and a recent launch of the <a href="http://innovation.nyc/" target="_blank" rel="noopener noreferrer">Neighborhood Innovation Lab</a> initiative, aimed to connect government, technologists, and communities in a joint effort to innovate and improve local communities. &nbsp;&nbsp;&nbsp;</p> <p>“Data privacy is a wide-ranging issue and seemingly touches almost every aspect of our city’s services,” Vacca concluded.</p> <p>***<br />by Natalia Erokhina for Gotham Gazette<br /><a href="https://twitter.com/GothamGazette" target="_blank" rel="noopener noreferrer">@GothamGazette</a></p> <p><img src="http://www.gothamgazette.com/images/graphics/2017/vacca_prep_computer.jpg" alt="vacca prep computer" width="600" height="465" /></p> <p>City Council Member Vacca (seated)</p> <hr /> <p>The Technology Committee of the New York City Council held an oversight hearing Monday on the city’s data privacy policies and data security protocols. A broad variety of more specific topics were covered during the hearing, chaired by Council Member James Vacca, including unauthorized data usage and security breaches; risks related to the expansion of high-speed internet availability and sidewalk wi-fi kiosks; the Worker Connect program that allows New Yorkers to search and apply for public benefits; new efforts at data exchange among city agencies; and the ‘Internet of Things,’ which refers to interconnectivity of various applications, devices, and information.</p> <p>As expected, a number of questions were raised at the hearing, only some of which were answered, about data privacy for New Yorkers and data security for city agencies.</p> <p>“I do think we came across some needs to have more of a formalized process on how we protect people’s privacy,” Vacca said after the hearing. “There should be an agency that’s tasked with this specific issue of privacy.”</p> <p>The main city agency tasked with data privacy concerns and security is the Department of Information Technology and Telecommunications, or DoITT. Commissioner Anne Roest testified Monday and told Vacca and the committee that the city is constantly experiencing hacking attacks. Since the current administration took office in January 2014, there were two instances when city data was stolen, Roest said, once from the Fire Department and once from the Administration for Children’s Services.</p> <p>“We’ve had, and I don’t have a number in front of me, but hundreds in the past year, of incidents where malware got into our system, but didn’t get anything out,” Roest said, emphasizing the quality of agencies cybersecurity threat response and prevention.</p> <p>A substantial part of the hearing revolved around LinkNYC, a network of interactive kiosks spread throughout the city, often where payphones used to be, providing free, fast wi-fi, access to various public services, and free phone calls anywhere in the United States. In January 2017, LinkNYC had over 1 million subscribers. Meanwhile, the <a href="https://data.cityofnewyork.us/dataset/LinkNYC-Usage-Statistics/69wu-b929/data" target="_blank" rel="noopener noreferrer">latest data</a> published on the city’s Open Data Portal indicates over 7 million sessions for the week of April 10. &nbsp;</p> <p>While the Link network is being used and expanded, there are privacy concerns. DoITT takes pride in the latest version of the privacy policy negotiated with the franchise partner, CityBridge LLC. Roest emphasized in her opening statement that LinkNYC does not track users’ browsing history and location, and therefore does not store or sell it.</p> <p>According to the privacy policy published on the LinkNYC <a href="https://www.link.nyc/privacy-policy.html" target="_blank" rel="noopener noreferrer">website</a>, the software does collect unique information about the device and browser being used, such as, for example, IP and MAC addresses, time zone settings, and other data (According to the privacy policy, this information is needed by LinkNYC to be able to provide internet connectivity). The storage period for this information varies from 60 days to a year. Each booth is also equipped with two cameras used to prevent or track vandalism, Roest said. The video is stored for seven days and then deleted. &nbsp;</p> <p>“Many national studies indicate that lower income people are disproportionately burned by data collection and privacy violations,” said Vacca in his opening statement at the hearing. He continued: “Last month Attorney General Eric Schneiderman announced that his office received approximately 1,300 notifications regarding data breaches of non-governmental systems in 2016. That is a 60 percent increase from the previous year.”</p> <p>“Privacy is not a luxury item, but a fundamental right of all people. With the recent federal repeal of the data privacy measures for internet service providers this discussion takes on added importance,” Vacca said about the decision to roll back a set of privacy regulations that obliged Internet providers to request consent from users before sharing and selling sensitive information, such as, for instance, healthcare and financial data, obtained by tracking their browsing behavior.</p> <p>The city’s Chief Technology Officer, Miguel A. Gamiño Jr., added during testimony that his office is planning to conduct a legal review to investigate the city’s potential to protect the privacy of New Yorkers, given ongoing reconsideration of Obama-era FCC regulations. “Specifically, this review includes an evaluation of the authority the city may have over the privacy policies of internet service providers, how and to what extent the city is exercising this authority currently, and whether the city can expand the exercise of its available authority to achieve, at minimum, the privacy protections for internet service consumers that Congress and the President have recently repealed,” said Gamiño. &nbsp;&nbsp;&nbsp;&nbsp;</p> <p>Officials identified public education as one of the key measures in preventing hacking attacks and data breaches in New York. Gamiño announced an “interagency work group on internet health and human rights” that will look for ways to educate consumers and businesses on how to protect their privacy. He also mentioned the city’s cooperation with public libraries on digital literacy materials for citizens and collaboration with Mozilla Foundation on a series of digital security workshops for organizations. Mozilla, a technology company known for its Firefox web browser, runs a nonprofit foundation that promotes open and accessible internet.</p> <p>Another topic covered by several meeting participants was an interagency data exchange strategy. Better data exchange among city agencies can improve transparency, efficiency, and broader accessibility to information for New Yorkers, but necessitates establishment of security and privacy protocols.</p> <p>DoITT Commissioner Roest discussed the importance of classifying collected data and applying safety protocols to the resulting categories. For instance, all data characterized as confidential cannot be stored or transferred unless it is protected by a special encryption technology.</p> <p>Finally, the Council committee discussed <a href="https://iot.cityofnewyork.us/" target="_blank" rel="noopener noreferrer">existing guidelines </a>and a future strategy for the ‘Internet of Things’, where New York City claims to be one of the global pioneers, and a recent launch of the <a href="http://innovation.nyc/" target="_blank" rel="noopener noreferrer">Neighborhood Innovation Lab</a> initiative, aimed to connect government, technologists, and communities in a joint effort to innovate and improve local communities. &nbsp;&nbsp;&nbsp;</p> <p>“Data privacy is a wide-ranging issue and seemingly touches almost every aspect of our city’s services,” Vacca concluded.</p> <p>***<br />by Natalia Erokhina for Gotham Gazette<br /><a href="https://twitter.com/GothamGazette" target="_blank" rel="noopener noreferrer">@GothamGazette</a></p> Open Data Compliance the Focus at Oversight Hearing 2017-01-24T05:00:00+00:00 2017-01-24T05:00:00+00:00 http://www.gothamgazette.com/?id=6728:open-data-compliance-the-focus-at-oversight-hearing Ben Max <p><img src="http://www.gothamgazette.com/images/vacca_1.jpg" alt="james vacca technology open data" width="600" height="378" /></p> <p>City Council Member Vacca (photo: @JamesVacca13)</p> <hr /> <p>The New York City Council’s Committee on Technology held an oversight hearing Tuesday to evaluate the progress being made by the city in tracking compliance by government agencies with the city’s open data law.</p> <p>The Mayor’s Office of Data Analytics (MODA) has been conducting what is essentially an audit, the “Examination and Verification Plan,” to gauge each agency’s status regarding its publication of required data sets on the <a href="https://nycopendata.socrata.com/" target="_blank">Open Data Portal</a>, an online access point meant to open government to the public.</p> <p>This means public information on everything from motor vehicle collisions to wifi hotspot locations to leading causes of death, which, in theory, allows regular New Yorkers to better evaluate city services, gives civic technologists more to work with in acting as an outside of government check, and encourages problem-solving using data.</p> <p>Tuesday’s hearing, chaired by City Council Member James Vacca, was the first on the examination and verification plan, which was adopted early in 2016. It showed that progress is underway in ensuring the quality of data and availability of the appropriate data sets, but raised questions about levels of agency participation, public engagement, and even what constitutes a public data set.</p> <p>While the portal is not lacking in data -- over 1,500 data sets from 60 different government agencies are currently accessible -- Local Law 11 of 2012, the city’s first open data law, mandates that all agencies publish their respective public data sets in entirety by 2019.</p> <p>MODA is employing the Examination and Verification Plan to ensure that each agency is on pace to meet that deadline. The plan calls for evaluation of three agencies per year. In 2016, the first year of the plan, the Department of Corrections, Department of Sanitation, and Department Housing Preservation and Development were selected for evaluation.</p> <p>In his opening statement at Tuesday’s hearing, MODA Director and Chief Analytics Officer Dr. Amen Ra Mashariki spoke to the importance of examination and verification. “Its purpose is to improve citywide compliance by creating a more systematic way to locate data sets that may have inadvertently or purposefully been excluded from agencies’ self reported data,” Mashariki said, adding that all three agencies evaluated this year are “in good standing.”</p> <p>As Mashariki alluded to in his statement, part of the impetus behind the open data law is to increase government transparency. The preamble to Local Law 11 states: “Making city data available online using open standards will make the operation of city government more transparent, effective and accountable to the public.”</p> <p>The aggregation of agency data also benefits those within government. Representatives from MODA explained at Tuesday’s hearing that when government agencies are required to share data, they often have had to go through a manual process, uploading files onto a hard drive or disk. However, now that agencies are posting their data in the portal, other departments are able to access it without going through the antiquated data exchange process.</p> <p>This could allow, for example, the Department of Transportation and the NYPD to access each other’s data more easily, which could aid the implementation of Vision Zero. As Mashariki put it, “There’s not only intra- but also inter-agency usage of data.”</p> <p>While the hearing touched on a number of benefits of the open data initiative, Council Member Vacca, a Bronx Democrat who chairs the technology committee, did raise several concerns.</p> <p>First, Vacca wanted to know when the examination and verification of the Police Department would be conducted. Crime statistics are frequently sought and police records remain among the most hotly debated in the city. Mashariki explained that while the NYPD is not scheduled for the examination and verification plan, MODA has been working with the department to publish and improve its data sets, some of which can be found in the portal. Mashariki added that while the NYPD is not currently on the books for evaluation, MODA “can begin looking into ramping up other agencies.”</p> <p>Vacca was also generally curious about non-mayoral agencies, such as the New York City Housing Authority (NYCHA) and the Health + Hospitals Corporation, and their involvement in the examination and verification process. Mashariki explained that though those agencies are not held to the same data-release requirements as other agencies, MODA has been working to get their data on the portal, pointing out that Health + Hospitals has several data sets on the portal.</p> <p>Vacca was particularly interested in traffic to the Open Data Portal, stressing that it is crucial to know how many people are making use of the newly available data. “Once we get actual numbers, we can decide whether or not we have to do more outreach and more engagement of community groups.”</p> <p>While MODA could not provide definitive numbers, Craig Campbell, a Senior Advisor at MODA, did say that the most-viewed data set on the portal has received over five million hits (though he would not name the data set). Still, Dr. Mashariki firmly agreed that raising public awareness around the availability and advantages of open data must be a core objective. “We need aggressive engagement no matter what the number says,” he said.</p> <p>One way in which MODA is striving to improve accessibility and engagement is through a new data portal website launching in March. The website, according to Mashariki, “makes it easier than ever to request new data sets on the open data portal.” Users will be able track the status of their requests and “get in touch with the open data team on anything from technical inquiries to partnership opportunities.” The website will also ensure that data dictionaries and geospatial fields (interactive maps) accompany all data sets.</p> <p>Also testifying at the hearing was a representative from Reinvent Albany, a good government group focused on open data. The Reinvent representative, Dominic Mauro, largely commended MODA's work, but did raise concerns about whether an agency can be deemed compliant if it possesses data sets that are listed as public but which are not present in the portal, as is the case with the Department of Sanitation.</p> <p>This concern prompted Mashariki to delve into the evolving nature of the open data effort, in particular the working definitions of what should be labeled as public data and would should remain confidential. “This process has helped us think a little smarter about what should or shouldn’t be a public data set,” he said. “The way we saw it is that the process is working if it’s forcing us to begin to think about what’s public, what’s private, and, to be very honest...I think that’s a conversation that should consistently be had.”</p> <p>As it turned out, the status of the data sets in question was pending -- they had not been classified as either public or not -- which is why they had not made it onto the portal at the time of the hearing.</p> <p>The public versus private determination that Mashariki referenced speaks to the conceptual as well as practical considerations involved in an undertaking like the open data initiative. Not only must MODA coordinate with agencies’ open data coordinators to aggregate data and ensure quality, they must also consider the spectrum of ramifications that come with publicizing certain information.</p> <p>Local Law 11’s agency compliance plan does include the following criteria for public classification:</p> <p>“For purposes of prioritizing public data sets, agencies shall consider whether information embodied in the public data set: (1) can be used to increase agency accountability and responsiveness; (2) improves public knowledge of the agency and its operations; (3) furthers the mission of the agency; (4) creates economic opportunity; or (5) responds to a need or demand identified by public consultation.”</p> <p>But questions remain: Could some data sets be exploited to the detriment of New York and its residents? Does the potential for positive opportunity outweigh the potential for exploitation? What about privacy? For example, the Department of Corrections data contains a lot of Personally Identifiable Information on inmates which could potentially compromise an individual.</p> <p>This question of definitions was echoed in the “2016 Examination and Verification Findings Report” issued by MODA on December 1. “Creating a list of eligible datasets is not a cut and dried process, and raised questions regarding the definitions of ‘public,’ ‘data,’ and ‘dataset,’” the report states. It also says that MODA will be working with the Department of Investigations to more clearly define these terms and develop a strategy for classification moving forward.</p> <p>

***
by Luke Foley, Gotham Gazette

</p>
<p><img src="http://www.gothamgazette.com/images/vacca_1.jpg" alt="james vacca technology open data" width="600" height="378" /></p> <p>City Council Member Vacca (photo: @JamesVacca13)</p> <hr /> <p>The New York City Council’s Committee on Technology held an oversight hearing Tuesday to evaluate the progress being made by the city in tracking compliance by government agencies with the city’s open data law.</p> <p>The Mayor’s Office of Data Analytics (MODA) has been conducting what is essentially an audit, the “Examination and Verification Plan,” to gauge each agency’s status regarding its publication of required data sets on the <a href="https://nycopendata.socrata.com/" target="_blank">Open Data Portal</a>, an online access point meant to open government to the public.</p> <p>This means public information on everything from motor vehicle collisions to wifi hotspot locations to leading causes of death, which, in theory, allows regular New Yorkers to better evaluate city services, gives civic technologists more to work with in acting as an outside of government check, and encourages problem-solving using data.</p> <p>Tuesday’s hearing, chaired by City Council Member James Vacca, was the first on the examination and verification plan, which was adopted early in 2016. It showed that progress is underway in ensuring the quality of data and availability of the appropriate data sets, but raised questions about levels of agency participation, public engagement, and even what constitutes a public data set.</p> <p>While the portal is not lacking in data -- over 1,500 data sets from 60 different government agencies are currently accessible -- Local Law 11 of 2012, the city’s first open data law, mandates that all agencies publish their respective public data sets in entirety by 2019.</p> <p>MODA is employing the Examination and Verification Plan to ensure that each agency is on pace to meet that deadline. The plan calls for evaluation of three agencies per year. In 2016, the first year of the plan, the Department of Corrections, Department of Sanitation, and Department Housing Preservation and Development were selected for evaluation.</p> <p>In his opening statement at Tuesday’s hearing, MODA Director and Chief Analytics Officer Dr. Amen Ra Mashariki spoke to the importance of examination and verification. “Its purpose is to improve citywide compliance by creating a more systematic way to locate data sets that may have inadvertently or purposefully been excluded from agencies’ self reported data,” Mashariki said, adding that all three agencies evaluated this year are “in good standing.”</p> <p>As Mashariki alluded to in his statement, part of the impetus behind the open data law is to increase government transparency. The preamble to Local Law 11 states: “Making city data available online using open standards will make the operation of city government more transparent, effective and accountable to the public.”</p> <p>The aggregation of agency data also benefits those within government. Representatives from MODA explained at Tuesday’s hearing that when government agencies are required to share data, they often have had to go through a manual process, uploading files onto a hard drive or disk. However, now that agencies are posting their data in the portal, other departments are able to access it without going through the antiquated data exchange process.</p> <p>This could allow, for example, the Department of Transportation and the NYPD to access each other’s data more easily, which could aid the implementation of Vision Zero. As Mashariki put it, “There’s not only intra- but also inter-agency usage of data.”</p> <p>While the hearing touched on a number of benefits of the open data initiative, Council Member Vacca, a Bronx Democrat who chairs the technology committee, did raise several concerns.</p> <p>First, Vacca wanted to know when the examination and verification of the Police Department would be conducted. Crime statistics are frequently sought and police records remain among the most hotly debated in the city. Mashariki explained that while the NYPD is not scheduled for the examination and verification plan, MODA has been working with the department to publish and improve its data sets, some of which can be found in the portal. Mashariki added that while the NYPD is not currently on the books for evaluation, MODA “can begin looking into ramping up other agencies.”</p> <p>Vacca was also generally curious about non-mayoral agencies, such as the New York City Housing Authority (NYCHA) and the Health + Hospitals Corporation, and their involvement in the examination and verification process. Mashariki explained that though those agencies are not held to the same data-release requirements as other agencies, MODA has been working to get their data on the portal, pointing out that Health + Hospitals has several data sets on the portal.</p> <p>Vacca was particularly interested in traffic to the Open Data Portal, stressing that it is crucial to know how many people are making use of the newly available data. “Once we get actual numbers, we can decide whether or not we have to do more outreach and more engagement of community groups.”</p> <p>While MODA could not provide definitive numbers, Craig Campbell, a Senior Advisor at MODA, did say that the most-viewed data set on the portal has received over five million hits (though he would not name the data set). Still, Dr. Mashariki firmly agreed that raising public awareness around the availability and advantages of open data must be a core objective. “We need aggressive engagement no matter what the number says,” he said.</p> <p>One way in which MODA is striving to improve accessibility and engagement is through a new data portal website launching in March. The website, according to Mashariki, “makes it easier than ever to request new data sets on the open data portal.” Users will be able track the status of their requests and “get in touch with the open data team on anything from technical inquiries to partnership opportunities.” The website will also ensure that data dictionaries and geospatial fields (interactive maps) accompany all data sets.</p> <p>Also testifying at the hearing was a representative from Reinvent Albany, a good government group focused on open data. The Reinvent representative, Dominic Mauro, largely commended MODA's work, but did raise concerns about whether an agency can be deemed compliant if it possesses data sets that are listed as public but which are not present in the portal, as is the case with the Department of Sanitation.</p> <p>This concern prompted Mashariki to delve into the evolving nature of the open data effort, in particular the working definitions of what should be labeled as public data and would should remain confidential. “This process has helped us think a little smarter about what should or shouldn’t be a public data set,” he said. “The way we saw it is that the process is working if it’s forcing us to begin to think about what’s public, what’s private, and, to be very honest...I think that’s a conversation that should consistently be had.”</p> <p>As it turned out, the status of the data sets in question was pending -- they had not been classified as either public or not -- which is why they had not made it onto the portal at the time of the hearing.</p> <p>The public versus private determination that Mashariki referenced speaks to the conceptual as well as practical considerations involved in an undertaking like the open data initiative. Not only must MODA coordinate with agencies’ open data coordinators to aggregate data and ensure quality, they must also consider the spectrum of ramifications that come with publicizing certain information.</p> <p>Local Law 11’s agency compliance plan does include the following criteria for public classification:</p> <p>“For purposes of prioritizing public data sets, agencies shall consider whether information embodied in the public data set: (1) can be used to increase agency accountability and responsiveness; (2) improves public knowledge of the agency and its operations; (3) furthers the mission of the agency; (4) creates economic opportunity; or (5) responds to a need or demand identified by public consultation.”</p> <p>But questions remain: Could some data sets be exploited to the detriment of New York and its residents? Does the potential for positive opportunity outweigh the potential for exploitation? What about privacy? For example, the Department of Corrections data contains a lot of Personally Identifiable Information on inmates which could potentially compromise an individual.</p> <p>This question of definitions was echoed in the “2016 Examination and Verification Findings Report” issued by MODA on December 1. “Creating a list of eligible datasets is not a cut and dried process, and raised questions regarding the definitions of ‘public,’ ‘data,’ and ‘dataset,’” the report states. It also says that MODA will be working with the Department of Investigations to more clearly define these terms and develop a strategy for classification moving forward.</p> <p>

***
by Luke Foley, Gotham Gazette

</p>
Praise, and Recommendations, for City's Open Data Progress 2016-09-21T04:00:00+00:00 2016-09-21T04:00:00+00:00 http://www.gothamgazette.com/?id=6541:praise-and-recommendations-for-city-s-open-data-progress Ben Max <p><img src="http://www.gothamgazette.com/images/graphics/2014/03/vacca_tech_open_data_cropped.jpg" alt="vacca tech open data cropped" width="600" height="536" /></p> <p>Council Member Vacca (photo: @JamesVacca13)</p> <hr /> <p dir="ltr">There are two years to go before the deadline set by the city&rsquo;s 2012 Open Data Law, which requires city agencies identify and publish all their citywide aggregated public data online. At a Wednesday afternoon oversight hearing of the City Council&rsquo;s Committee on Technology, open data and civic technology advocates offered praise for the city&rsquo;s progress thus far and recommendations for the city to meet its goals.</p> <p>Open data is considered a vital transparency tool that allows citizens and government to evaluate and improve city services. New York City, considered a trendsetter in this area, committed to this goal in 2012 by passing a law that would make data easily accessible and consumable by the public. Through the city&rsquo;s <a href="https://nycopendata.socrata.com" target="_blank">Open Data Portal</a>, New Yorkers can access a range of information such as the number and type of complaints made to the 311 helpline, real-time traffic speed data, or where the City Council spends discretionary funding.</p> <p dir="ltr">The Open Data Portal hosts nearly 1,600 datasets across all city agencies.</p> <p dir="ltr">Last year, in the annual progress report on the city&rsquo;s implementation of the Open Data Law, &nbsp;Mayor Bill de Blasio released a new open data plan, <a href="http://www1.nyc.gov/assets/home/downloads/pdf/reports/2015/NYC-Open-Data-Plan-2015.pdf" target="_blank">&ldquo;Open Data For All,&rdquo;</a> which aims at making the Open Data Portal more user-friendly, making it easily visualized for those with little experience in data or programming, and pushes for more citywide engagement.</p> <p dir="ltr">Following that, the City Council passed a package of legislation to enhance the existing law and improve access to information through the city&rsquo;s Open Data Portal maintained by the Department of Information Technology and Telecommunications (DoITT).</p> <p dir="ltr">This was the first year which saw some of the new amendments put into effect. These new laws, seven in total, deal with retention of data, building data dictionaries, standardizing geospatial data, establishing timelines for agencies to respond to public requests on the Open Data Portal, mandating timely updates for public data, requiring publication of datasets obtained through Freedom of Information Law requests, and examination and verification of agency compliance.</p> <p dir="ltr">&ldquo;One of the things we&rsquo;ve been able to do is build an open data ecosystem,&rdquo; testified Dr. Amen Ra Mashariki, director of the Mayor&rsquo;s Office of Data Analytics (MODA), who is also the chief analytics officer and chief open platform officer for the city. He said the new laws help &ldquo;anchor the administration&rsquo;s commitment to transparency and equitable uses of technology around open data.&rdquo;</p> <p dir="ltr">Ra Mashariki stressed that MODA&rsquo;s focus has been on user research, feedback mechanisms, and technical standards, and he laid out the city&rsquo;s progress on each while teasing updates that will be released later this year. These updates include a first-of-its-kind analysis with NYU Center for Urban Science and Progress on &ldquo;data poverty&rdquo; -- the lack of access or representation of communities in city data; a study with Columbia University&rsquo;s School of International and Public Affairs on opportunities for community based organizations to utilize open data; and a new centralized mechanism for receiving feedback, with quicker responses and tracking.</p> <p dir="ltr">Concluding his testimony, Ra Mashariki cited an important example of how open data can be useful to the public. He referenced Ben Wellington, a data scientist who runs the blog &ldquo;I Quant NY.&rdquo; Wellington examined parking violation data and <a href="http://iquantny.tumblr.com/post/144197004989/the-nypd-was-systematically-ticketing-legally" target="_blank">found that</a> the NYPD had been ticketing legally parked cars for millions of dollars. He took his analysis to the NYPD who confirmed it and took measures to prevent it in the future.</p> <p dir="ltr">Testifying on behalf of DoITT, Albert Webber cited the city&rsquo;s key accomplishments over the last year -- traffic on the Open Data Portal increased to more than 5 million hits; 116 datasets were made public, bringing the total to nearly 1,600; 100 new data sets are now automatically updated; and more than 40 unscheduled data sets were identified and published. He also said DoITT would soon fill three of four new dedicated positions for the Open Data Portal.</p> <p dir="ltr">At the hearing, government transparency advocates and civic technology professionals wholeheartedly endorsed the city&rsquo;s efforts, and also made a few recommendations for improvements.</p> <p dir="ltr">John Kaehny, executive director of Reinvent Albany and co-chair of the New York City Transparency Working Group, emphasised &ldquo;that City Council stay actively and energetically involved in pushing, cajoling and cheerleading for open data in this city,&rdquo; without which the program can not be a success.</p> <p dir="ltr">The Transparency Working Group is co-chaired by Gene Russianoff, senior attorney at the New York Public Interest Research Group, and its members include Common Cause New York and Citizens Union. &ldquo;Open data and the idea behind open data is working in New York City,&rdquo; Kaehny said, also stressing that city agencies will eventually save tens of millions of dollars by using open data.</p> <p dir="ltr">He said the de Blasio administration has &ldquo;started upping their game&rdquo; in implementing open data but in his testimony Kaehny pointed out a number of outstanding issues. These included the need for a better process for remedying public complaints about data quality, more standardization of data, clearer agency criteria for publishing data, and publishing high-demand data sets, such as NYPD crime statistics or the Department of Transportation&rsquo;s street paving schedule.</p> <p dir="ltr">He also suggested other needed improvements to the overall system and the law. Currently, one of the new laws requires publishing the number of datasets requested under FOIL -- a method often used by advocates and journalists to examine city information. These, he said, should also be named. He said the city should also consider creating a data issues tracker to monitor errors in datasets reported by the public, similar to a tool used by the federal government.</p> <p dir="ltr">Council Member James Vacca, chair of the technology committee, was generally satisfied with the administration although he raised a few concerns at the hearing. He was especially pleased that one dataset to be added soon will be of snowplow locations, updated every hour, which would give people quick answers during snowstorms when the city usually struggles to clear roads across the five boroughs. He did question the administration on whether certain agencies were fully complying with the law, particularly the New York Police Department.</p> <p dir="ltr">Vacca also wondered later whether open data use was being taught to local community groups and Community Boards, which would then have a stronger ability to advocate for local issues. &ldquo;I know communities that always say &lsquo;We want more police, we want more traffic agents...but they don&rsquo;t have the documentation, they don&rsquo;t do the research,&rdquo; he said. &ldquo;Here it&rsquo;s at their fingertips, here we can do the research.&rdquo;</p> <p dir="ltr">DoITT&rsquo;s Webber said that agencies &ldquo;have made strides across the board,&rdquo; and that the NYPD had added a number of datasets to its website and the Open Data Portal links to them. &ldquo;Overall I wouldn&rsquo;t say any particular agency is not committed to doing what they&rsquo;re supposed to do,&rdquo; Webber said, adding that he would later follow up with the Council Member with specific NYPD datasets.</p> <p dir="ltr">For civic technology advocates, the major concern is not only the availability of data but its usefulness to the public.</p> <p dir="ltr">Noel Hidalgo, executive director of BetaNYC, a civic technology and open government group, made this clear even as he praised the administration and the Council for their push on open data. &ldquo;The seven pieces of legislation that were passed are taking us in the right direction and have really strengthened the city&rsquo;s open data practice,&rdquo; he said. But Hidalgo pointed out that one big weakness, echoing Vacca&rsquo;s words, is figuring out how to teach community groups to access, understand and utilize open data. &ldquo;Frustratingly, there are zero best practices out there on how to teach the city&rsquo;s open data to itself,&rdquo; Hidalgo said.</p> <p dir="ltr">Hidalgo added that under the provisions of the new laws passed last year, the data dictionaries need to contain tutorials and that there needs to be easier access to bulk data downloading. Finally, he stressed the need for the city to create an open data legacy that can last through successive administrations and City Council classes, with a unit dedicated to leadership, standardization, development, and education.</p> <p>There was also consensus at the hearing among advocates that the very infrastructure of the Open Data Portal could be modernized. Joel Natividad, director of open data at OpenGov Inc., said the portal, currently run by a contracted vendor, needs to be an open source platform that can &ldquo;support innovation and experimentation&rdquo; by universities, non-governmental organizations and the private sector. This will allow the city to &ldquo;tap the genius of the crowd,&rdquo; he said. Kaehn, of Reinvent Albany, agreed, saying the current platform has severe limitations.</p> <p>

***
by Samar Khurshid, City government reporter, Gotham Gazette
     

Read more by this writer.

</p>
<p><img src="http://www.gothamgazette.com/images/graphics/2014/03/vacca_tech_open_data_cropped.jpg" alt="vacca tech open data cropped" width="600" height="536" /></p> <p>Council Member Vacca (photo: @JamesVacca13)</p> <hr /> <p dir="ltr">There are two years to go before the deadline set by the city&rsquo;s 2012 Open Data Law, which requires city agencies identify and publish all their citywide aggregated public data online. At a Wednesday afternoon oversight hearing of the City Council&rsquo;s Committee on Technology, open data and civic technology advocates offered praise for the city&rsquo;s progress thus far and recommendations for the city to meet its goals.</p> <p>Open data is considered a vital transparency tool that allows citizens and government to evaluate and improve city services. New York City, considered a trendsetter in this area, committed to this goal in 2012 by passing a law that would make data easily accessible and consumable by the public. Through the city&rsquo;s <a href="https://nycopendata.socrata.com" target="_blank">Open Data Portal</a>, New Yorkers can access a range of information such as the number and type of complaints made to the 311 helpline, real-time traffic speed data, or where the City Council spends discretionary funding.</p> <p dir="ltr">The Open Data Portal hosts nearly 1,600 datasets across all city agencies.</p> <p dir="ltr">Last year, in the annual progress report on the city&rsquo;s implementation of the Open Data Law, &nbsp;Mayor Bill de Blasio released a new open data plan, <a href="http://www1.nyc.gov/assets/home/downloads/pdf/reports/2015/NYC-Open-Data-Plan-2015.pdf" target="_blank">&ldquo;Open Data For All,&rdquo;</a> which aims at making the Open Data Portal more user-friendly, making it easily visualized for those with little experience in data or programming, and pushes for more citywide engagement.</p> <p dir="ltr">Following that, the City Council passed a package of legislation to enhance the existing law and improve access to information through the city&rsquo;s Open Data Portal maintained by the Department of Information Technology and Telecommunications (DoITT).</p> <p dir="ltr">This was the first year which saw some of the new amendments put into effect. These new laws, seven in total, deal with retention of data, building data dictionaries, standardizing geospatial data, establishing timelines for agencies to respond to public requests on the Open Data Portal, mandating timely updates for public data, requiring publication of datasets obtained through Freedom of Information Law requests, and examination and verification of agency compliance.</p> <p dir="ltr">&ldquo;One of the things we&rsquo;ve been able to do is build an open data ecosystem,&rdquo; testified Dr. Amen Ra Mashariki, director of the Mayor&rsquo;s Office of Data Analytics (MODA), who is also the chief analytics officer and chief open platform officer for the city. He said the new laws help &ldquo;anchor the administration&rsquo;s commitment to transparency and equitable uses of technology around open data.&rdquo;</p> <p dir="ltr">Ra Mashariki stressed that MODA&rsquo;s focus has been on user research, feedback mechanisms, and technical standards, and he laid out the city&rsquo;s progress on each while teasing updates that will be released later this year. These updates include a first-of-its-kind analysis with NYU Center for Urban Science and Progress on &ldquo;data poverty&rdquo; -- the lack of access or representation of communities in city data; a study with Columbia University&rsquo;s School of International and Public Affairs on opportunities for community based organizations to utilize open data; and a new centralized mechanism for receiving feedback, with quicker responses and tracking.</p> <p dir="ltr">Concluding his testimony, Ra Mashariki cited an important example of how open data can be useful to the public. He referenced Ben Wellington, a data scientist who runs the blog &ldquo;I Quant NY.&rdquo; Wellington examined parking violation data and <a href="http://iquantny.tumblr.com/post/144197004989/the-nypd-was-systematically-ticketing-legally" target="_blank">found that</a> the NYPD had been ticketing legally parked cars for millions of dollars. He took his analysis to the NYPD who confirmed it and took measures to prevent it in the future.</p> <p dir="ltr">Testifying on behalf of DoITT, Albert Webber cited the city&rsquo;s key accomplishments over the last year -- traffic on the Open Data Portal increased to more than 5 million hits; 116 datasets were made public, bringing the total to nearly 1,600; 100 new data sets are now automatically updated; and more than 40 unscheduled data sets were identified and published. He also said DoITT would soon fill three of four new dedicated positions for the Open Data Portal.</p> <p dir="ltr">At the hearing, government transparency advocates and civic technology professionals wholeheartedly endorsed the city&rsquo;s efforts, and also made a few recommendations for improvements.</p> <p dir="ltr">John Kaehny, executive director of Reinvent Albany and co-chair of the New York City Transparency Working Group, emphasised &ldquo;that City Council stay actively and energetically involved in pushing, cajoling and cheerleading for open data in this city,&rdquo; without which the program can not be a success.</p> <p dir="ltr">The Transparency Working Group is co-chaired by Gene Russianoff, senior attorney at the New York Public Interest Research Group, and its members include Common Cause New York and Citizens Union. &ldquo;Open data and the idea behind open data is working in New York City,&rdquo; Kaehny said, also stressing that city agencies will eventually save tens of millions of dollars by using open data.</p> <p dir="ltr">He said the de Blasio administration has &ldquo;started upping their game&rdquo; in implementing open data but in his testimony Kaehny pointed out a number of outstanding issues. These included the need for a better process for remedying public complaints about data quality, more standardization of data, clearer agency criteria for publishing data, and publishing high-demand data sets, such as NYPD crime statistics or the Department of Transportation&rsquo;s street paving schedule.</p> <p dir="ltr">He also suggested other needed improvements to the overall system and the law. Currently, one of the new laws requires publishing the number of datasets requested under FOIL -- a method often used by advocates and journalists to examine city information. These, he said, should also be named. He said the city should also consider creating a data issues tracker to monitor errors in datasets reported by the public, similar to a tool used by the federal government.</p> <p dir="ltr">Council Member James Vacca, chair of the technology committee, was generally satisfied with the administration although he raised a few concerns at the hearing. He was especially pleased that one dataset to be added soon will be of snowplow locations, updated every hour, which would give people quick answers during snowstorms when the city usually struggles to clear roads across the five boroughs. He did question the administration on whether certain agencies were fully complying with the law, particularly the New York Police Department.</p> <p dir="ltr">Vacca also wondered later whether open data use was being taught to local community groups and Community Boards, which would then have a stronger ability to advocate for local issues. &ldquo;I know communities that always say &lsquo;We want more police, we want more traffic agents...but they don&rsquo;t have the documentation, they don&rsquo;t do the research,&rdquo; he said. &ldquo;Here it&rsquo;s at their fingertips, here we can do the research.&rdquo;</p> <p dir="ltr">DoITT&rsquo;s Webber said that agencies &ldquo;have made strides across the board,&rdquo; and that the NYPD had added a number of datasets to its website and the Open Data Portal links to them. &ldquo;Overall I wouldn&rsquo;t say any particular agency is not committed to doing what they&rsquo;re supposed to do,&rdquo; Webber said, adding that he would later follow up with the Council Member with specific NYPD datasets.</p> <p dir="ltr">For civic technology advocates, the major concern is not only the availability of data but its usefulness to the public.</p> <p dir="ltr">Noel Hidalgo, executive director of BetaNYC, a civic technology and open government group, made this clear even as he praised the administration and the Council for their push on open data. &ldquo;The seven pieces of legislation that were passed are taking us in the right direction and have really strengthened the city&rsquo;s open data practice,&rdquo; he said. But Hidalgo pointed out that one big weakness, echoing Vacca&rsquo;s words, is figuring out how to teach community groups to access, understand and utilize open data. &ldquo;Frustratingly, there are zero best practices out there on how to teach the city&rsquo;s open data to itself,&rdquo; Hidalgo said.</p> <p dir="ltr">Hidalgo added that under the provisions of the new laws passed last year, the data dictionaries need to contain tutorials and that there needs to be easier access to bulk data downloading. Finally, he stressed the need for the city to create an open data legacy that can last through successive administrations and City Council classes, with a unit dedicated to leadership, standardization, development, and education.</p> <p>There was also consensus at the hearing among advocates that the very infrastructure of the Open Data Portal could be modernized. Joel Natividad, director of open data at OpenGov Inc., said the portal, currently run by a contracted vendor, needs to be an open source platform that can &ldquo;support innovation and experimentation&rdquo; by universities, non-governmental organizations and the private sector. This will allow the city to &ldquo;tap the genius of the crowd,&rdquo; he said. Kaehn, of Reinvent Albany, agreed, saying the current platform has severe limitations.</p> <p>

***
by Samar Khurshid, City government reporter, Gotham Gazette
     

Read more by this writer.

</p>