Two weeks ago, while riding BART from East Bay to San Francisco, I was offered a transit survey from a BART Survey Team member. Pointing to what seemed to me to be a large number of written-response questions for a survey being delivered on a moving train, I asked her, "Do you have to read and code all these surveys?"

The answer was on my mind as I was in the Bay Area to attend a Summit at Code for America, the organization where I will be serving as a Fellow for 2012. During this trip, I observed (though this is by no means a fresh observation) that geeks are drawn to architecting databases, "open" no less, without recognizing or paying much attention to the experience of the people who will be entering and maintaining the data.

As someone who has spent a large amount of life sorting (and resorting) spreadsheets in Excel---and creator of an open database of my own---the experience of maintaining data isn't particularly liberating (nor is compiling federal grant reports, for that matter). The experience reminds me of what Joel Johnson wrote for Wired Magazine in "1 Million Workers. 90 Million iPhones. 17 Suicides. Who’s to Blame?":

"...the work itself isn’t inhumane---unless you consider a repetitive, exhausting, and alienating workplace over which you have no influence or authority to be inhumane. And that would pretty much describe every single manufacturing or burger-flipping job ever."

Which is to say that maintaining a database is work and doubly so when the "data driven decisions" are not the maintainers' to make. I think of this whenever the licensing costs of "closed" commercial databases are cited as driving the need for "open"; I agree that data in the public interest should be "free",  but I recognize that freedom comes with a cost.

At the Summit, Tim O'Reilly said, "Data is infrastructure." If that's the case, I hope us data "architects" recognize the experience of people---the data pavers, data plumbers, and data janitors---who maintain such "virtual" infrastructure if we expect to gain greater liberties than our current physical landscape provides.

The BART Survey Team member's answer on that shaky train was "Yes".

Related posts:

1. Data divides and umbrellafication Jesse Lichtenstein in “Transparency for All”, writing for Wired: The concern that open data may simply empower the empowered is...
2. Crime and Data Leadership This afternoon I have been following the mis-framing of Chicago’s newly announced plan to release 10 years of… not crime...
3. Data-driven, content-first design I’m working on an app for the DonorChoose.org Hacking Education Contest. DonorsChoose works by having teachers submit classroom project/supply needs that...

Chicago to publish crime stats online

CHICAGO—Long a city with a reputation for withholding information, Chicago now wants to make public every crime over the past 10 years -- a highly unusual move among the nation's major police departments.

Starting Wednesday, millions of crime statistics dating to 2001 will be posted online in a searchable database. It will be updated daily, providing fodder for residents to evaluate their own neighborhoods, academics to study crime and techie types to create websites or apps.

I'll harp on that last sentence in which the "techie type's" activities are described so rudderlessly: this data will fuel a hundred red and green heatmaps, but probably provide little opportunity for reflection on the true nature of this data---police incident reports---let alone the politics, policies and policing that generated this data in the first place. Just as an example, this is from a WBEZ story entitled "The downside of hiring more cops in Chicago":

But there are also costs with increasing the number of police on the street and those costs can be tough to measure. "The good intentions of actually creating the uniformed presence to lower the immediate problems of crime may have an unintended result when you're looking further down the line," according to George Gascon. He's the district attorney for the city of San Francisco, and before that he was the chief of police. He says low-income, often minority communities, are flooded with police, and residents are over-criminalized. Lots of people are arrested, sometimes for small infractions.

Kids get criminal records, they're cut off from educational and employment opportunities, and all of that ultimately makes the crime problems worse. "I'm not saying that we should look the other way to crime, to the contrary. What I'm saying is that the strategies that we used in the past have not worked well, and we need to evolve away from that. In many neighborhoods basically we have been at war with our people," Gascon said.

An incident does not make an indictment, and an infraction need not be a crime, let alone a conviction; nor should we forget what goes unreported. And yet this dataset is being distributed and---this is the actual problem---represented by the media and (soon) a myriad of websites and apps as full-stop crime.

This is not a criticism of the police or the City of Chicago, who should be commended for being more transparent and making this information more readily available; this is a plea for the media and application makers to appropriately label this data and use it with an understanding of its limitations.

Ben Fry, on visualization future and data literacy looks toward the future:

"I think the real thing that's going to change is that we're going to start understanding that visualization isn't this sort of monolithic thing... I like to look at it a lot like writing. You have novels and poetry and haikus. You know there's lots of different types of writing and styles of writing — and I think the same thing happens in visualization... some things are tools for analysis and some things are purely for entertainment, and there's not so much a spectrum that there is different ways of addressing it.

But this understanding and widespread "data literacy" is not here yet. Its up to us so called techie types to exhibit "Data Leadership" and work to better interpret the and explain the complexity and nuance of our analysis---the absence of which I've complained about before.

Data leadership is appropriately labeling data. Data leadership is presenting data with a recognition of its limitations. Data leadership is consideration for how your presentation of data may be interpreted and responsibility for the consequences.

Data leadership is ultimately a recognition of the broader context of human experience and how information is collected, analyzed and integrated into our lives and decision-making processes---both individually and socially. I realize that's awfully heady for discussing glorified spreadsheets, but to riff off the old chestnut, you can't manage what you misrepresented.

(Thanks Justin, Bec and David for drawing my attention to this.)

Related posts:

1. Data divides and umbrellafication Jesse Lichtenstein in “Transparency for All”, writing for Wired: The concern that open data may simply empower the empowered is...
2. Uncrime Mapping I really don’t understand the appeal of crime maps. Trulia, a housing search tool, just launched a crime map, too...
3. Community mapping class interview & outcomes   Last year I interviewed Richard (Dick) Howe, Lowell’s Registrar of Deeds about the impact of his participation in a...

The concern that open data may simply empower the empowered is not an argument against open data; it's an argument against looking at open data as an end in itself. Massive data dumps and even friendly online government portals are insufficient. Ordinary people need to know what information is available, and they need the training to be conversant in it. And if people are to have more than theoretical access to the information, it needs to be easy and cheap to use. That means investing in the kinds of organizations doing outreach, advocacy, and education in the communities least familiar with the benefits of data transparency. If we want truly open government, we still have to do the hard work of addressing basic and stubborn inequalities. However freely it flows, the data alone isn't enough.

Yes and yes. Most government data is policy-level, which means to understand and act on that data, you not only need to be data literate, but also civically literate to transform knowledge into power. Call me a constructionist, but community organizing creates a stronger sense of agency than statistics.

And "cheap" data inconsiderately presented can be harmful. The worst data abuses come from trying to use policy-level, non-contextualized or incomplete data to inform individual decision-making. For example, I had to add this caveat to the Boston Bike Crash Map after getting several anxiety-producing inquiries:

This data alone is not appropriate for making routing decisions. The presence or absence of incidents in a location should not be used to determine the relative safety of biking there as this data does not include ridership or traffic information; i.e. a location may contain less incidents because bikers know to avoid it.

For decision making, I've come to call this phenomenon the "umbrellafication" of data---after the service that boils the weather forecast down to a simple yes/no answer to "Do you need an umbrella today?" Unfortunately, like trying to portray crime as a spectrum of green to red, issues and datasets that can be easily synthesized and presented are the exception, not the rule.

Related posts:

1. Crime and Data Leadership This afternoon I have been following the mis-framing of Chicago’s newly announced plan to release 10 years of… not crime...
2. Data-driven, content-first design I’m working on an app for the DonorChoose.org Hacking Education Contest. DonorsChoose works by having teachers submit classroom project/supply needs that...
3. The work itself isn’t inhumane Two weeks ago, while riding BART from East Bay to San Francisco, I was offered a transit survey from a...

I'm working on an app for the DonorChoose.org Hacking Education Contest. DonorsChoose works by having teachers submit classroom project/supply needs that people can then donate to pay for through the internet. Right now the only way to "share" those projects is through the usual email, Facebook and Twitter; my idea for am app is to create custom printable 4-up (quartersheet) flyers for handing out, and pull-tab sheets for posting-up. Despite the "power of teh internets", I think consideration of local resources and capacity is an important value to recognize---and helping teachers and allies advertise their needs (and the positive outcomes they hope to create) more easily within their physical communities would go far towards advancing DonorsChoose.org's mission.

This project has the typical technical difficulties: my goal is to use the DonorsChoose API to fetch a particular project and then populate an HTML+CSS preview---allowing the teacher to then live-edit the text (both for layout and because writing for a local audience is different than an internet one)---and then use dompdf to convert that HTML+CSS to a printable PDF. The dompdf library is nifty: it supports CSS 2d transformations, custom fonts, images (necessary for QR code hotness), light positioning (no floats, so tables it is), and media-queries  (@media screen and @media print) to fix all the little layout and typography issues in translating from the web to print, as well as dompdf's own myriad quirks. Basically it's everything I need to make a not-too-ugly flyer and pull-tab sheet; the image at the top of this post is the ugly proof-of-concept.

But the biggest challenge is designing a layout scheme that is flexible to the wide range of DonorsChoose project content. Designing for an 8.5 x 11" sheet of paper is way different than designing for the web: creating a balanced---or at least aesthetically-acceptable---design is no easy matter when there is no such thing as overflow. I hope to get around some issues by providing a live-preview so that teachers can fix any egregious text over-/under- runs, but the goal is to get teachers to Click-Print-Post as quickly as possible.

Fortunately I have a designer (the awesome Billy Brown) helping with the layout, but I need to give him some idea of what to expect. Fortunately DonorsChoose made available project data for ~296,000 projects. So I parsed through the lengths for the 4 main pieces of content I want to use in order to get the distribution of lengths. Sure, saying "Design for a title that is 10-50 characters" doesn't have the highest specificity, but it's a whole lot more useful than the alternative of blind experimentation. The data below is charts of those lengths---I probably will have to limit my data to just the past year or so since DonorsChoose has changed their requirements/text-fields over time, but its interesting so far to see the full distributions.

Related posts:

1. Print & Share: not everyone is a social media ninja (nor need they be) Today is the deadline for DonorsChoose’s Hacking Education Contest, and fortunately I have completed and submitted Print and Share (with no...
2. DonorsChoose Contest Update: Consolation Prize Edition DonorsChoose announced the winners for their Hacking Education contest today and unfortunately Print & Share, the app I developed with...
3. Belief-based design Matt Webb posted “Inbox Hero” about a month back (via AJ): Rand: The question isn’t who is going to let...

The City of Boston uses a Lagan CRM system to create and track tickets and cases. To feed that ticketing system the city offers constituent services in person, via telephone (the meeting was held next to the call center which had ~10 agents active at the time), via the web, and through the smartphone apps. The Lagan system tracks 150-170 types of tickets, but Boston currently exposes only 6 of them through Open311 API (streetlights, potholes, graffiti, 2 types of snow removal, and other); this decision was explained as being driven by the UI needs of the official smartphone apps. The Open311 system is only a data bridge into the Lagan CRM and thus won't support any additional metadata or external decisioning (this dismissed a Code for America fellow's suggestion of voting on tickets).

The API is not currently available. The city estimates 2 - 3 weeks until they have a test server up, and from there they will evaluate whether to give applications access to the live system. The test server will be a sandbox that is either refreshed every 24 hours, has new data streamed to it, or may simulate workflows (e.g. submit, review, comment, close)---it was still in discussion.

In addition to the Open311 API, the city also offers data dumps of its entire ticketing system, offset by 24 hours. Unfortunately, those data dumps don't include the channel through which they were inputted, e.g. it's not recorded whether the ticket came thru Open311, telephone, web or in person. This is allegedly through the city's Data Hub / Data Dashboard, but I can't find it.

The City of Boston is taking a more deliberate and restrictive approach than MassDOT/MBTA in opening up their data, though CRM tickets are clearly different than bus/train route and dispatch data. The university researcher's (Ben Clark) use case for the data was spot-on: determining who is utilizing these smartphone tools, and importantly who isn't. Time will tell how the influence of outside software developers will push the city's implementation---and how will it effect less-technology focused solutions.

I was really excited to meet the Code for America crew as they are bringing a lot of excitement and energy to the gov data scene. I did get the impression that they were unprepared for managing institutional forces: there was a question about why the city couldn't devote more IT resources to the project that were answered with some allusions to Dilbert (without acknowledging that the current capital budget was probably set 14 months ago). Open311 doesn't seem like CfA's primary focus in Boston, but if this was their first City of Boston meeting, I think they learned a lot.

Related posts:

1. 7 years in Boston This August marks the completion of my 7th year in Boston, with the loose exceptions of the 1 month I...
2. Apps off the approved vendor list I ran across a year-old article I had bookmarked from GovTech entitled “Do Apps for Democracy and Other Contests Create...
3. Crime and Data Leadership This afternoon I have been following the mis-framing of Chicago’s newly announced plan to release 10 years of… not crime...

I would advise against self reporting—because what you get is only crashes within whatever bike subculture is reached by the map and by the web. Boston did that before our map and the spread of accidents across the city is very different in each—and could give people the wrong ideas about priority streets and areas. (This is particularly true in communities of color, which were heavily under-reported in Boston's self-reported crash survey.)

Boston's bike crash map is generated from (somewhat-) public data on on ambulance runs and EMT reports; Portland has a self-reported bike incident map.

Related posts:

1. Data divides and umbrellafication Jesse Lichtenstein in “Transparency for All”, writing for Wired: The concern that open data may simply empower the empowered is...
2. Crime and Data Leadership This afternoon I have been following the mis-framing of Chicago’s newly announced plan to release 10 years of… not crime...
3. Data-driven, content-first design I’m working on an app for the DonorChoose.org Hacking Education Contest. DonorsChoose works by having teachers submit classroom project/supply needs that...

I’ve been spending some time at work scraping data. Long story short: government transparency is not transparent when the only access they give you is a pile of poorly structured html. That’s better than government opacity but not past the level of frosted glass: titillating but unsatisfying. If your expected audience is pencil pushers, please release your data in a spreadsheet. That’s what I did.

Notes for nerds:

Regular Expressions vs. Parsing Engines: I wrote a the first parser in Python with Regular Expressions, then rewrote it in BeautifulSoup (a Python parser). It took me about 2 hours to write it the first time with RegExp. It took me about 2 days to do it with BeautifulSoup. It’s slightly easier to maintain now, but you tell me which one is more semantically correct:

project_title = re.search('<tr><td><b>Project&nbsp;title</b></td><td>(.+)</td></tr>', line)

versus

project_title = app.find(text="Project title").parent.parent.nextSibling.string

Yep, it’s written in 2-column tables with each row being a different data-set: the first column holds a key (if there is a key; sometimes there isn’t) and the second column being the data . With RegExp, I know exactly what I’m looking for. With the parser, I have to find the element in the tree, then traverse up, over and down (if there isn’t a key, I have to go up, up, over, over, over, down, over, down). The data itself is a big set of applications (about 2000+ total) and each application has about 15 different data-sets (some with keys, some just follow a consistent-ish pattern).

Fortunately, I have an appreciative audience for my troubles and it lets me draw pretty maps like the ones above. Also done with Python by parsing an SVG vector image.

Michigan boaters beware: there is now an isthmus between Mackinaw City and St. Ignace. Rather than rewrite the process for grouped-shapes—Michigan being in 2 parts—it was good enough to make Michigan 1. Hawaii somehow endured.

No related posts.