White House Upgrades Open Data Policy

On Thursday, President Obama issued an executive order building off his earlier orders and OMB memos like the Open Government Directive, the Managing Government Records Directive, and projects like the Open Government Platform. It’s called “Making Open and Machine Readable the New Default for Government Information“, and it continues to push federal agencies toward meaningful open data. The OMB has placed a number of major requirements on federal agencies:

  1. Create datasets from collected information with the expectation that data will be used by other sources; keep information machine-readable.
  2. All agencies must internally catalog and index their datasets in both human- and machine-readable formats.
  3. Agencies must publish the portions of the indexes containing datasets which could be made public. (e.g. which do not contain Federal secrets or social security numbers or the like.)
  4. Agencies must create a forum for public dialogue, where members of the public to request certain datasets be prioritized over others.

These are all promising steps, and the White House has created a Github repository with best practices and policies for agencies to learn from and share their experiences, as well as case studies, tools, and various written resources for officials and open government advocates alike.

NYC Reforms, Opens Subcontractor Reporting System

Today, New York City became the first municipality in the country to create a public database of subcontractors. In addition to recording payments to prime contractors, now subcontractors are recorded and tracked, and all this information is public. NYC Comptroller Liu’s office issued this press release, quoting TWG Co-Chairs Gene Russianoff and John Kaehny:

“New Yorkers will be getting a much more complete picture of how contractors and subcontractors are spending their tax dollars, thanks to Mayor Michael Bloomberg and Comptroller John Liu,” said Gene Russianoff, senior attorney for the New York Public Interest Research Group.


“Congratulations to Mayor Bloomberg and Comptroller Liu,” said John Kaehny, Executive Director of Reinvent Albany and Co-Chair of the NYC Transparency Working Group. “Digitizing and reporting subcontractor payments is a huge leap forward in accountability and transparency. Though somewhat dry and esoteric, this new reporting system has big implications for reducing corruption and improving efficiency, and when fully in place, will make New York City one of the most fiscally transparent cities in the world. When the subcontractor data is put into the Checkbook NYC platform, it will become instantly available for the rest of government and the public to use.”

Governor Launches Open Budget NY

Governor Andrew M. Cuomo today announced the launch of OpenBudget.ny.gov, a new website that provides unprecedented access and transparency to New York’s budget. Open Budget’s easy-to-use tools, charts, and information are available to the public today in coordination with the Governor’s Executive Budget address.

Open Budget is a first step in Open New York, an initiative outlined in the Governor’s 2013 State of the State address, which will use technology to promote transparency, improve government performance, and enhance citizen engagement.

“Open Budget is bringing the people back into government by taking budget data out of government file cabinets and making it available to the public for the first time in an easy-to-access, downloadable form. This will facilitate research, analysis, and innovation,” Governor Cuomo said. “As a first step in my Open New York initiative, Open Budget provides a powerful tool for transparency and accountability, fostering citizen engagement and enhancing the public’s trust in government.”

New York City Launches Checkbook 2.0

NYC Checkbook 2.0

In a press release today, NYC Comptroller John Liu announced the update of the city’s financial transparency platform, now called Checkbook 2.0:

“Checkbook NYC 2.0 is probably the most powerful fiscal transparency website in the United States, maybe the world. It’s a civic-technology home run for New Yorkers,” said John Kaehny, Co-Chair of the NYC Transparency Working Group and Executive Director of Reinvent Albany. “It hugely increases the usefulness of the City’s Financial Management System to watchdog groups and journalists, and allows the public to cast an independent eye on City spending. And, hugely to the Comptroller’s credit, Checkbook NYC 2.0 will be open source – which means governments and transparency groups across the country can reuse it to shine a light on their own city or state.”

MTA Increases Transparency with Digital Initiatives

In June of 2012, members of the nycTWG wrote to Joseph Lhota, Executive Director of the Metropolitan Transportation Authority, with seven transparency initiatives which would increase organizational efficiency, reduce costs, and eliminate archaic business practices. Shortly thereafter, Director Lhota responded favorably to our suggestions. Read the nycTWG’s letter here, and the response from the MTA here.

Today, on October 1, 2012, the MTA announced in a press release that they were following the nycTWG’s recommendation to implement an email and text alert archive for service disruptions, as well as a real-time RSS feed of service changes. Each entry contains the date and time of the message, the MTA agency that sent the alert, and the full text of the message.

Read the MTA’s press release here, and see the email/text alert archive here.

Groups Applaud Passage of NYC Open Data Bill

Today, New York City took a very important step towards getting more government digital data online when the NYC City Council passed Council Member Gale Brewer’s NYC Open Data Bill, Intro 29-A. The new law creates a process for putting truly massive amounts of city data online. Reinvent Albany congratulates Council Member Brewer, Council Speaker Christine Quinn, council lawyer Jeff Baker, DOITT Commissioner Carole Post, and the Bloomberg administration for negotiating a workable way forward for making vast amounts of city data usable by the public. The mayor has pledged to sign the bill when it arrives on his desk. The Open Data Law is the product of untold hours of work by city council and mayoral staff, and involved extensive consultation with the civic and technology groups in the NYC Transparency Working Group, many of which actively supported the bill.

We have high hopes for the NYC Open Data Law, and would like to see a New York State Open Data Law. The new law is important for both symbolic and practical reasons. It articulates a clear public principal that digital data should be available to the public and online – not just in response to Freedom of Information Law requests.

The law’s preamble says:

“It is in the best interest of New York City that its agencies and departments make their data available online using open standards. (This) will make the operation of city government more transparent, effective and accountable to the public…Streamline… communications within and between government, promote innovative strategies for social progress and create economic opportunities.”

For those concerned about implementation and actually getting new data, there is reason for optimism. The new law was developed jointly by staff to the mayor and council. They worked hard to create a realistic process grounded in public feedback. Both branches feel ownership, and have internal champions who want to see the Open Data Law work. Putting terabytes of government data online is a new and ambitious undertaking. It is a complex process and easily sabotaged by bureaucratic obstructionism.

The Open Data Law does four things:

  1. It empowers the NYC Department of Information Technology and Telecommunications (DOITT) to develop standards and oversee a step by step process for putting the city’s quantitative data* into a single web portal. (*The new law will not require government reports, the budget or ad hoc quantitative analysis be put online. For instance, the NYC Digital Road Map would not have to be put online under this bill — though it already is.)
  2. It calls for an inventory of all data sets. (Already called for under the NYC COPIC law, but is not being complied with.)
  3. Sets a timeline for agencies to put their quantitative data online at the NYC Open Data central data website. Within a year, agencies have to put all of the data they currently have online in this data site.
  4. It creates an online feedback process that agencies are supposed to heed. This is the key to the law’s success or failure — advocates know this and are going to energetically pursue something that gives full voice to public concerns and interests. NYC transparency advocates will build on what Portland is doing, and make sure that the mayor and his agencies know exactly what data the public wants to see online, and in what form.


TWG Comments for Commissioner Merchant

Dear Commissioner Merchant,

Congratulations on your appointment as New York City’s first Citywide Chief Information and Innovation Officer, and as Commissioner of the Department of Information, Technology and Telecommunications. Our groups are strong supporters of the city’s efforts to use Information Technology to become more transparent and efficient.

We write today to offer our strong support for DOITT’s efforts to implement Local Law 11, New York’s path breaking Open Data Law. We commend the work that DOITT’s staff has done developing the document, and DOITT’s open-minded and innovative approach to the Open Data Law. We believe the City’s Open Data Law, and its successful implementation, are important nationally and will be watched closely by state and local governments.

We applaud DOITT’s early efforts to engage the interested public, and experiment with collaboration tools like the online, fully editable, wiki version of the Standards document. The wiki is an innovative idea and is well worth trying. Our groups have made specific edits to that document. But as key civic stakeholders, we also wanted to provide you with the following comments and recommendations. These reflect a consensus among our groups regarding DOITT’s Open Data Technical Standards and Policies document.

We look forward to working with you to make the Open Data Law a big success. Sincerely,

Gene Russianoff
NY Public Interest Research Group

Phil Ashlock
Civic Commons

Susan Lerner
Common Cause

Adrienne Kivelson
League of Women Voters NYC

John Kaehny
Reinvent Albany

Rachael Fauss
Citizens Union

Frank Hebbert

Marjorie Shea
Women’s City Club

1. There Should Be a Clear Policy Statement on Open Data.

The Standards manual should include a clear goal statement about implementing the Open Data Law. We recommend the Standards document say:

“The goal of the City of New York’s Open Data Law is to make the city’s public data as easy to find and use as is possible, consistent with the New York State Freedom of Information Law.”

2. Our Groups Strongly Support Designating an Open Data Coordinator at Each Agency.

We strongly support the designation of an Open Data Coordinator at each City agency. The Standards Manual should clarify that the coordinators should be available to respond public inquiries about open data. Their names, phone numbers, and email addresses should be listed on agency websites and the open data portal.

3. Create An Open Data Dashboard.

Our groups strongly recommend that DOITT should create via the Standards Manual a public Open Data Dashboard to promote public and internal accountability. The site should track which data has been made available via the Open Data Law, and make it clear how well agencies are complying with Local Law 11. A good model is the White House Open Government Dashboard (See https://www.whitehouse.gov/open/around).

4. DOITT Should Use NYC Domain Names for the Open Data Portal(s)

DOITT should use the NYC domain name for all open data portals (and more generally online applications.) The city should establish a unique and permanent NYC domain for its open data website. (For example: https://opendata.nyc.gov.) The City should avoid being locked into the domain of a particular vendor, especially since an open data portal is a generic platform, not a social media channel.

5. The City Should Clarify and Revise its Information Classification

We strongly recommend that the City review its Information Classification standards and rewrite them to be more specific, and clearly consistent with the privacy, intellectual property, and security exceptions in the NYS Freedom of Information Law. The Classifications should make it clear to the public, City Council and agency data coordinators what data can and cannot be posted online. Further, by clearly articulating what exceptions exist in the Manual, the rationale for not releasing certain data sets will be more apparent to everyone at the outset. In this way, the City can foster an environment of trust between its agencies and the public.


Memo in Support of Intro 29A

Statement in Support:

Citizens Union, Civic Commons, Common Cause, New York Public Interest Research Group(NYPIRG), and Reinvent Albany – all members of the New York City Transparency Working Group – strongly support Intro 29A, which creates a process for New York City agencies to make certain data available to the public in a singular web portal between one year after the bill’s passage and 2018.

Our organizations believe that by making New York City government’s digital data more accessible and usable by the public, Intro 29A has the potential to make New York City government more transparent, accessible, and accountable. It builds on ongoing initiatives like NYC Open Data, the May 2011 Road Map for the Digital City, the NYC Big Apps competition, and NYC Stat.

We share the City Council and the City Administration belief that making more City agency data easily available has profound opportunities to make government more open and responsive to the public, to deliver better public services at a lower cost and to share information among government agencies.

Summary of Bill:

Section 23-502 requires City agencies with one year to make available their existing public data sets on the internet available through a single web portal that is linked to nyc.gov or any successor website maintained by, or on behalf of, the City. Such public data sets shall be made available in accordance with technical standards published by the NYC Department of Telecommunications and Technology (DoITT) pursuant to section 23-505 of this chapter and shall be in a format that permits automated processing and shall make use of appropriate technology to notify the public of all updates.

Subsection (c) requires that public data sets shall be updated as often as is necessary to preserve the integrity and usefulness of the data sets to the extent that the agency regularly maintains or updates the public data set and subsection (d) states that public data sets shall be made available without any registration requirement, license requirement or restrictions on their use except for measures required to ensure access to public data sets, to protect the single web site housing public data sets from unlawful abuse or attempts to damage or impair use of the web site or to analyze the types of data being used to improve service delivery.

Section 23-506 (a) of the bill requires DoITT to issue an “agency compliance plan” within eighteen months to the Council and mayor and make such plan available to the public on the web portal. The plan must include a summary description of public data sets under the control of City agencies and must prioritize these public data sets for inclusion on the single web portal on or before December 31, 2018. City agencies must cooperate with DoITT in its preparation of the compliance plan. Section 23-506 (c) requires annual updates to the public by DoITT.

Section 23-501 provides definitions for terms in the bill including defining “data” as meaning final versions of statistical or factual information (i) in alphanumeric form reflected in a list, table, graph, chart or other non-narrative form, that can be digitally transmitted or processed; and (ii) regularly created or maintained by or on behalf of and owned by an agency that records a measurement, transaction, or determination related to the mission of an agency. Such term shall not include information provided to an agency by other governmental entities, nor shall it include image files, such as designs, drawings, maps, photos, or scanned copies of original documents, provided that it shall include statistical or factual information about such image files and shall include geographic information system data. Nothing in this legislation prohibits an agency from voluntarily disclosing information not otherwise defined as “data”, nor shall it be deemed to prohibit an agency from making such voluntarily disclosed information accessible through the single web portal established pursuant to section 23-502.

Section 23-503 (c) requires DoITT to implement an on-line forum to solicit feedback from the public and to encourage public discussion on open data policies and public data set availability on the web portal. Under section (d), requests received via the on-line forum for inclusion of particular public data sets shall be considered by agencies in making determinations as to priority for public data set inclusion on the single web portal pursuant to paragraph v of subdivision b of section 23-506 of this chapter.