Let’s start with some deep breathing. Give me a good deep breath and count to four with me.
Here we go. 1. 2. 3. 4. Now exhale. You can feel your eyes. Good, that’s exactly it.
Now let’s take a deep breath and, in your mind, draw a hippopotamus as fast as you can. Quick quick. His legs, his folds, his marshmallow teeth. Okay, done. Now exhale.
Take another deep breath and hold it tight. As you hold it tightly in your chest, imagine the tightness is shrinking you down into a bug. You’ve held your breath so hard that you’re an insect. And all the other bugs saw you shrink and they loved the stunt. They’re clapping and rubbing their feelers together madly. But you had an apple in your hand when you were big and it just caught up with you, crushed the whole crowd. You’re dead, too. Now exhale.
Give me a solid deep breath and imagine you live in a town where everything is made of telephone cords. The houses are all telephone cords, the shingles, the rafters. The doorways are a thick mass of telephone cords which you simply thrust yourself through. When you go to bed, the bedspread is telephone cords. And the mattress and box springs are telephone cords, too. Like I said, everything is made out of telephone cords. The telephone itself is made of telephone cords. But the telephone cord going to the telephone is made out of bread and a couple sticks. Now exhale.
Breathe in. 1. 2. 3. 4. Breathe out.
Breath in. 1. 2. Another short breath in. 3. 4. Imagine both of your hands snapping off at the wrists and flying into your computer screen and programming it from the inside. Exhale.
Big, big deep breath. Deep down inside you there is a submarine. It has a tongue. Exhale.
Breathe through your nostrils. Deep breath. Filter the air through your nostrils. Breathing through the nostrils gives you quality air. Your nostrils flare, you are taking breaths of nature’s air, the way God intended. Imagine a floppy disk drive clogged up with orphans. And while it chokes on orphans, you have good, wholesome God’s breath in your lungs. But that pleasurable, life-giving air will become a powerful toxin if held too long. Hurry, exhale God and nature’s air!
Now, you will wake up, smoothing out the creases of this page in your web browser. You will have full recollection of your whole life and not forgetting any one of the many adventures you have had in your life. You will feel rich and renewed and expert. You will have no remembrance of this short exercise, you will instead remember teaching a rabbit to use scissors from a great distance.
And as you will wake up with your eyes directed to the top of this exercise, you will begin again. But this time, try to imagine that even your shadow is a telephone cord.
I’ve spent the last 6 months working at OkCupid Labs building an iOS native mobile app I’m very proud of: Ravel!—”Share Photos. Meet People.” We’ve described it as a dating app for Instagrammers: your public profile is the photos that you pull in from Instagram, Facebook or your camera roll and you can introduce yourself and start chatting with other people based on the photos they share.
We built Ravel! using RubyMotion, a toolchain for writing native iOS apps in (nearly)Ruby. In building it, I learned a lot more both about Ruby and Objective-C along with the intricacies of developing for a mobile device. The mobile app is powered by a Rails-based API server, which is also the backend for the website (for sharing your photo and profile via the web). One of the strengths of RubyMotion is its wonderful automation tools; and the drawbacks is that it’s memory management still has requires some work and workarounds. Perhaps you’ve seen this quote: “It’s not just you. I’m experiencing these memory-related types of crashes (like SIGSEGV and SIGBUS) with about 10-20% of users in production.” That was me on the RubyMotion mailing list.
I also learned a lot of about instrumentation and building a consumer-facing app. Even though a lot of our focus was on in-app engagement, we also invested a lot of time in growth and retention. From optimizing the download and App Store texts, to adding events and triggers in-app and on the server, to adding sharing and open graph integration through Facebook and on the web.
As I cycle off of Ravel!, time will tell how it survives as a product, but I’m very proud of how we brought it to market.
2012 was a big year for Day of the Shirt: it moved from being one of a constellation of hobby projects to becoming the main focus of my spare development time and a largely self-sustaining business.
As a project
As a technology project, Day of the Shirt went through a lot of changes in 2012. Including:
- It started off the year featuring just 10 t-shirt vendors. By the end of the year, that number had grown to 21 vendors, including some that featured multiple shirt sales.
- The technology stack morphed a lot. Originally built in PHP, I first rewrote the backend in Node.JS (though I was still using PHP for HTML templating). Then, as the year closed out, I began the task of rewriting it for Ruby on Rails.
- In moving Day of the Shirt from PHP to Node (and towards Rails), I improved the instrumentation and error logging a lot. This meant I’m much more quickly able to respond to changes in t-shirt vendors websites and ensure that designs on Day of the Shirt are up-to-date.
- Added a daily RSS Feed
- Started pushing daily updates to Facebook (in addition to Twitter) with about 1,500 likes on the Facebook Page by year’s end.
As a business
2012 marked the first time that Day of the Shirt ever earned any money. Like:
- After 1.5 years of having Google Adsense advertisements on Day of the Shirt, I finally received my first payout in February.
- In addition to showing advertisements, I also started adding affiliate links to shirts. The majority of t-shirt vendors I list don’t have affiliate programs, but I went ahead and signed up as an affiliate for those that do.
- Because I was actually earning some revenue for the first time, in November I completed the process of registering Day of the Shirt as a Sole Proprietorship in the City of San Francisco.
As a passion
- Traffic went up, way up. Over the course of the year Day of the Shirt went from having 200 daily visitors in January to more than 8,000 by December.
- People started writing about Day of the Shirt on Reddit and online forums. And tweeting about it. And in November Day of the Shirt was featured on Lifehacker.
- People started emailing me (and we don’t feature our email address very prominently): thanking me for the website and suggesting new daily t-shirt vendors to add to the website.
2012 was a really important year for Day of the Shirt and has set a lot of the foundation for improvements and new features I’m adding in 2013.
So, approximately every year or so, someone launches Yet Another Fully Automated Statistical Tool Dashboard Thing that tries to show the skills & activity of a given open source contributor, or the “health” of a given open source project.
The next thing that happens then follows as surely as night follows day, or as unrequited nostalgia follows the latest James Bond release:
People try the tool, and say “Hmm, well, it’s wildly inaccurate about *me* (or about the projects I have expertise in), but maybe it’s useful to someone else.”
And maybe it is. But what’s really going on, I think, is that the developers of these tools are trying to solve too much of the problem.
Investigating the activity of a particular developer, or the health of an open source project, inevitably involves human judgement calls informed by experience and by out-of-band knowledge. These judgements can be tremendously improved by good tool support — one could imagine certain dashboard-like tools that would make such investigations orders of magnitude more convenient and accurate than they currently are.
But because that kind of tool deliberately doesn’t go the last few inches, it’s a lot harder to demo it. There’s no one-size-fits-all final screen displaying conclusions, because in reality that final screen can only be generated through a process of interaction with the human investigator, who tunes various queries leading up to the final screen, based on knowledge and on specific questions and concerns.
And because it’s harder to demo, people are less likely to write it, because, hey, it’s not going to be as easy to post about it on Slashdot or Hacker News :-). Well, also because it’s a lot more work: interactive tools are simply more complex, in absolute terms, than single-algorithm full-service dashboards that load a known data query and treat it the same way every time.
So what I’m saying is: Any tool that tries to do this, but where you just have to enter a person’s name or a project’s name and click “Go”, is going to be useless. If you didn’t shape the query in partnership with the tool, then whatever question the tool is answering is probably not the question you were interested in.
Thomas Aquinas, who devoted some two million words to spelling out, in the Summa Theologica, the nature of the world, God’s purpose in creating it and our fate in traversing it, ended his short life (short by our standards, at least) in a state of ecstasy, declaring that all that he had written was of no significance beside the beatific vision that he had been granted, and in the face of which words fail. His was perhaps the most striking example of a philosopher who comes to believe that the real meaning of the world is ineffable. Having got to this point, Aquinas obeyed the injunction of Wittgenstein, whose Tractatus Logico-Philosophicus concludes with the proposition: “that whereof we cannot speak we must consign to silence.”
But Aquinas was exceptional. The history of philosophy abounds in thinkers who, having concluded that the truth is ineffable, have gone on to write page upon page about it. One of the worst offenders is Kierkegaard, who argues in a hundred ways that the ultimate is inexpressible, that truth is “subjectivity,” that the meaning of life can be given by no formula, no proposition, no abstraction, but only by the concrete experience of surrender whose content can never be given in words.
The temptation to take refuge in the ineffable is not confined to philosophers. Every inquiry into first principles, original causes and fundamental laws, will at some stage come up against an unanswerable question: what makes those first principles true or those fundamental laws valid? What explains those original causes or initial conditions? And the answer is that there is no answer — or no answer that can be expressed in terms of the science for which those laws, principles and causes are bedrock. And yet we want an answer. So how should we proceed?
There is nothing wrong with referring at this point to the ineffable. The mistake is to describe it.
This is the text of a skillshare I delivered at The Sourcery, an awesome commission-less recruiting service. Not only is their model nifty, but they care about actually knowing what they’re recruiting for—so hopefully there isn’t anything too wrong below.
- Event-based: when programming a web browser, most of the actions you want to wire up are “when the user clicks on this, perform a different action than when they click on that.” You declare what event(s) you want to listen for and the resulting action/function you want to take place / be called when you “hear” that event take place.
- Asynchronous: when you’re interacting with a web page, you’re downloading new data (or images or video) that might take a second or two (or more for video!) to arrive over the wire. You don’t want to just freeze the browser while you wait for it to load, instead you want it to load in the background, then kick off some action when its complete (“completed loading” is another example of an event).
- Event-based: a server will constantly be getting requests (at different urls, on different ports) that need to be responded to in different ways when those events take place
- Asynchronous: in dealing with a request, a server will need to load other data (from a file, a database, another server like Twitter), and you don’t want your entire server to just lock up while it waits for that external service (we call this I/O for “Input/Output”) to finish. For a typical web request, the majority of time will be spent waiting on I/O.
So how are other languages that aren’t (for the most part) event-based/asynchronous used on web servers? Like Ruby, Python and PHP. The synchronous code runs on top of an asynchronous web server (Apache, Rack, WSGI) that creates a synchronous ”thread” to run your Ruby/Python/PHP code. If another request comes in while that first thread is still processing, the web server creates a new thread. Unfortunately, those threads are resource-intensive which means you can only create a limited number of them (based on how powerful your server is). If all your threads are in-use, a new request will have to wait until a previous request is finished and the thread becomes available again.
So what can you do when you’re able to quickly serve hundreds/thousands of simultaneous connections?
- Proxy servers / Service “Glue” / Data Plumbing: routing/piping data between different servers, services or processes
- Telemetry / Analytics: catch and analyze data as events take place in your system
- Real-time connections to the web-browser: Traditional/Threaded systems try to keep their connections brief (because they can only handle a few at a time). If you don’t have that few-at-a-time constraint, you can leave the connection open much longer and thus easily send data back and forth in real time. Socket.io is a library for doing this.
Example of all 3: visualizing traffic going through a Node.js load-balancer by geolocating the requesting IP address and sending them to a map in the web browser in real-time via Socket.io: http://vimeo.com/48470307
So if Node.js is so badass, why not use it for everything?
- CPU Blocking: because Node.js runs on only a single thread, any local processing you do (e.g. NOT database/service calls) locks the thread. For example, processing heavy numerical/algorithmic processing, or generating complicated HTML templates. Node.js works best when you do that data processing somewhere else (for example, in SQL or map/reduce database), and just sending along raw data (like JSON). This is why you’ll often see Node.js powering a thin API (calling out to a database and serving up some JSON) rather than a full-stack MVC implementation (like Rails/Django). This is why Node.js (backend server API) and single-page web apps like Backbone (frontend client-generated UI) are a powerful combination.
Still, the opportunities that Node.js creates are worth it. Other fun stuff/opportunities for Node:
- Pre-rendering browser content on the server: Typically you don’t want to do heavy CPU processing on the Node server, but maybe you’re working with really lightweight clients and you want to “emulate a web-browser” on your more powerful server. Example: Famo.us pre-renders DOM translations in their tech-demo so it will run on devices like the Apple TV
So if Node.js + Backbone is a “powerful combination”, why don’t we just ditch Rails entirely?
The Rails ecosystem is more mature than Node’s: there are more engineers, more libraries, stronger conventions and a more complete development and deployment pipeline (scaffolding, testing, continuous integration, monitoring, etc.). If you have a startup with a limited development window and a typical product design (“users create accounts, post content, favorite other user’s content, see what was most favorited content”) that you need to quickly/agilely iterate, Rails has that problem solved (this is the strength of Rails over pretty much everything, not a weakness of Node specifically). If you’re looking at a cost curve, the starting cost for Rails will be way lower for a vanilla product. Now if you’re talking about doing something non-typical (realtime interaction) or are operating at a huge scale (where you can swap infrastructure costs for engineering costs), Node is enabling (there are some things you just can’t/don’t want to do without Node) if not affordable. Also, you can use Node in a heterogenous environment (running the load balancer, or analytics server) or integrate a Node-powered service into a more traditional product (for example, real-time chat is powered by Node, but user accounts, relationships and chat history is managed by Rails).
Boring explanation: Using data from Chicago’s Open311 API the app lists, in near-realtime, service requests within the City of Chicago’s 311 system that have recently been created, updated or closed.
Awesome explanation: The mayor runs through the streets of Chicago, leaping over newly-reported civic issues and collecting coins as those civic problems are fixed.
I really like this application, not only because of its visual style, but because it lets you engage with the 311 data in a completely novel way: aurally. Turn up those speakers and let the website run in the background for about 30 minutes. Spinies (new requests) come in waves, and coin blocks (updated/closed requests) are disappointingly rare. Sure, I could have just created a chart of statistics, but I think actually experiencing requests as they come in makes you think differently about both the people who submitted a request and the 311 operators and city staff who are receiving them (just think about what caused those restaurant complaints… or maybe don’t).
The application is built with Node.js, which fetches and caches 311 requests, and a Backbone-based web-app, communicating via socket.io, which manages all of interface and animation. The source is on Github.
The core assumption of engagement applications is that people will do an activity consistently and repeatedly if you just structure the experience and incentivize it correctly — even if it’s asinine. The justification for civic engagement apps can be similarly foolish: people will perform a potentially beneficial activity that they aren’t currently doing if we give them the ability to do it on the internet (or via SMS, or iPhone, etc.). That’s why I built Civics Garden.
A few months ago a Code for America email thread came around asking how “If you could tell the story of how government works, what would you say?” I pushed back with the idea that one cannot know government without participating in it, and since we are a government of, by, and for the people, the best place to start would be reflecting on one’s own civic life and civic actions.
A tried and true method of reflection is journaling. Hundreds of millions of people keep a journal called Twitter, reflecting and writing on their days experiences, tribulations, meals, and cat sightings (this is not a comprehensive list). If people naturally do this, why not ask them to reflect specifically on their civic actions—voting, volunteering, checking out a library book, picking up a piece of trash, smiling at a stranger—and write down that reflection (however brief) on a regular basis?
Projecting from my own nature, people are fickle, lazy, unreliable creatures. That’s why gamification is the hotness. “How can I get you to look at my ads every day? I’ll make it a game.” Sure, this is no different than historical incentives (“How can I get you to work in my coal mine? I’ll pay you money.”), but now it’s on the internet where advertising is easier to place than coal mines. One form of gamification I really enjoy are virtual pets: like tomagatchi’s you feed with Unit Tests, or flowers you water with foreign language vocabulary. They encourage you to perform an activity because you instinctively (unless you’re a sociopath) feel good when they’re healthy and bad when they’re sickly… despite the fact you know they aren’t actually alive.
Civics Garden combines these concepts: by signing up, users adopt a virtual plant that they keep healthy by regularly “watering” it by writing down their civic deeds. If they go too long without a journal entry, the plant will wither and eventually die. Just like owning real houseplants, it has exhilarating potential.
I built Civics Garden with Node.js and MongoDB using Twitter for authentication. Each new user receives a healthy bamboo plant to caretake by writing short journal entries. To keep people reflecting regularly, one’s plant will wither after two days, and die after four–though users can replant it as many times as necessary. To keep them coming back to the site, users receive a tweet from @civicsgarden to let them know when their plant needs refreshing. As an minimum viable product (MVP), it’s high-on-concept/low-on-looks, but Diana and Emily helped with the graphics.
I’ve tested it internally with Code for America fellows and it should be a rousing success. All of us, as active civic participants performing important civic deeds should be able to briefly but consistently reflect upon and record our actions, right? See for yourself on Civics Garden and test your ability to reflect on a healthy civic commitment.
…or maybe people just don’t like bamboo.
Open311Status—still a work in progress–polls Open311 servers in 30 cities and aggregates configuration, performance and utilization statistics about those Open311 endpoints. I built Open311Status for two reasons: first, to provide a high-level view of how different endpoints are performing as a developer sanity check (“is my app broken or the server?”); and second, to provide some hard data to the anecdotes and experiences I’ve used in advising the City of Chicago and others in how to improve Open311′s potential for positive impact in their cities and beyond.
In designing Open311Status I took advantage of the huge benefit of the Open311: interoperability. By adopting the Open311 API for exposing civic data, cities are enabling civic developers like myself to build reusable tools and apps. To access data from 30 cities, Open311Status doesn’t need an adapter for each individual city, only a server URL that it can expect to interact and deliver data in the same way as described by the Open311 API documentation. Sure, there are some minor interoperability issues (for example, Toronto doesn’t like datetimes submitted with milliseconds), but these have been minor speed bumps for development, not major deal breakers.
A major limiting factor in the utility of Open311 isn’t these minor technical issues, but in how the Open311 servers are configured. If Open311 is supposed to provide a rich API for developers to interact with there should be a broad number of categories (“types”) of requests that can be submitted, as well as a comprehensive listing of previously submitted requests that can be analyzed or dashboarded. Compare the Boston and Washington, D.C. Open311 implementations: Washington, D.C. currently provides 83 categories of service requests (from “Tree Inspection” to “Roadway Marking Maintenance”) while Boston only provides six. Washington, D.C. provides access to all requests regardless of whether they were entered via Open311, the 311 call center, or city work crews or departments; Boston only displays service requests made via the Open311 API. These configuration decisions have a huge impact on the comprehensiveness of the data and thus the potential applications third party developers can build on top of the Open311.
If server configuration determines the potential for an innovative application, server performance determines whether such an app will be useable. Because a potential Open311 application will be heavily reliant upon a city’s Open311 server, the speed and robustness of that server will determine the responsiveness of the application. At the obvious extreme, if the server goes down, that app will probably be useless. If the server is functioning yet takes several seconds to respond to every request, the application and its users will experience significant slow-downs and frustration. As I’ve witnessed through Open311Status, usually about 10 percent of the servers take longer than one second to respond at any given time. That’s not a stable infrastructure for third party developers to build upon.
Open311Status helps measure the distance between the potential of Open311 and the current reality. It’s cause to celebrate that here are 30 cities (and more to add to Open311Status) who have adopted both open data and open standards; this is hugely important progress! But there is still a lot of work to be done beyond the specification for Open311 to enable a rich and robust third party application developer ecosystem.
Some technical details: Open311Status is built on Node.js with the Express web framework and aggregates data in MongoDB through the Mongoose ORM. The application is single process and uses cron and async modules to poll the cities’ Open311 servers every five minutes. A fun/dumb feature is that individual service requests are streamed to the browser using Socket.io, but because many servers publish service requests an hour (or more!) after they’ve been submitted, Open311Status streams the previous day’s service requests in “real time” as if they were today’s (or rather, “real time minus one day”). Tests are done with Mocha in BDD style with Should using Sinon for mocks and stubs (though coverage could–always–be better). Open311Status is hosted on Heroku.
This post is cross-posted from the Code for America blog.
From Steven Johnson’s Where Good Ideas Come From: The Natural History of Innovation:
Darwin’s notebooks lie at the tail end of a long and fruitful tradition that peaked in Enlightenment-era Europe, particularly in England: the practice of maintaining a “commonplace” book. Schol ars, amateur scientists, aspiring men of letters—just about anyone with intellectual ambition in the seventeenth and eighteenth cen turies was likely to keep a commonplace book. The great minds of the period—Milton, Bacon, Locke—were zealous believers in the memory-enhancing powers of the commonplace book. In its most customary form, “commonplacing,” as it was called, involved tran scribing interesting or inspirational passages from one’s reading, assembling a personalized encyclopedia of quotations. There is a distinct self-help quality to the early descriptions of commonplac ing’s virtues: maintaining the books enabled one to “lay up a fund of knowledge, from which we may at all times select what is useful in the several pursuits of life.”
John Locke first began maintaining a commonplace book in 1652, during his first year at Oxford. Over the next decade he de veloped and refined an elaborate system for indexing the book’s content. Locke thought his method important enough that he ap pended it to a printing of his canonical work, An Essay Concerning Human Understanding. Locke’s approach seems almost comical in its intricacy, but it was a response to a specific set of design con straints: creating a functional index in only two pages that could be expanded as the commonplace book accumulated more quotes and observations:
When I meet with any thing, that I think fit to put into my common-place-book, I first find a proper head. Suppose for example that the head be EPISTOLA, I look unto the index for the first letter and the following vowel which in this in stance are E. i. if in the space marked E. i. there is any number that directs me to the page designed for words that begin with an E and whose first vowel after the initial letter is I, I must then write under the word Epistola in that page what I have to remark.
Locke’s method proved so popular that a century later, an en terprising publisher named John Bell printed a notebook entitled “Bell’s Common-Place Book, Formed generally upon the Principles Recommended and Practised by Mr Locke.” The book included eight pages of instructions on Locke’s indexing method, a system which not only made it easier to find passages, but also served the higher purpose of “facilitat[ing] reflexive thought.” Bell’s volume would be the basis for one of the most famous commonplace books of the late eighteenth century, maintained from 1776 to 1787 by Erasmus Darwin, Charles’s grandfather. At the very end of his life, while working on a biography of his grandfather, Charles obtained what he called “the great book” from his cousin Reginald. In the biography, the younger Darwin captures the book’s marvelous di versity: “There are schemes and sketches for an improved lamp, like our present moderators; candlesticks with telescope stands so as to be raised at pleasure to any required height; a manifold writer; a knitting loom for stockings; a weighing machine; a surveying ma chine; a flying bird, with an ingenious escapement for the move ment of the wings, and he suggests gunpowder or compressed air as the motive power.”
The tradition of the commonplace book contains a central ten sion between order and chaos, between the desire for methodical arrangement, and the desire for surprising new links of associa tion. For some Enlightenment-era advocates, the systematic index ing of the commonplace book became an aspirational metaphor for one’s own mental life. The dissenting preacher John Mason wrote in 1745:
Think it not enough to furnish this Store-house of the Mind with good Thoughts, but lay them up there in Order, digested or ranged under proper Subjects or Classes. That whatever Sub ject you have Occasion to think or talk upon you may have re course immediately to a good Thought, which you heretofore laid up there under that Subject. So that the very Mention of the Subject may bring the Thought to hand; by which means you will carry a regular Common Place-Book in your Memory.
Others, including Priestley and both Darwins, used their com monplace books as a repository for a vast miscellany of hunches. The historian Robert Darnton describes this tangled mix of writing and reading:
Unlike modern readers, who follow the flow of a narrative from beginning to end, early modern Englishmen read in fits and starts and jumped from book to book. They broke texts into fragments and assembled them into new patterns by transcrib ing them in different sections of their notebooks. Then they reread the copies and rearranged the patterns while adding more excerpts. Reading and writing were therefore inseparable activities. They belonged to a continuous effort to make sense of things, for the world was full of signs: you could read your way through it; and by keeping an account of your readings, you made a book of your own, one stamped with your personality.
Each rereading of the commonplace book becomes a new kind of revelation. You see the evolutionary paths of all your past hunches: the ones that turned out to be red herrings; the ones that turned out to be too obvious to write; even the ones that turned into entire books. But each encounter holds the promise that some long forgotten hunch will connect in a new way with some emerging obsession. The beauty of Locke’s scheme was that it provided just enough order to find snippets when you were looking for them, but at the same time it allowed the main body of the commonplace book to have its own unruly, unplanned meanderings. Imposing too much order runs the risk of orphaning a promising hunch in a larger project that has died, and it makes it difficult for those ideas to mingle and breed when you revisit them. You need a system for capturing hunches, but not necessarily categorizing them, because categories can build barriers between disparate ideas, restrict them to their own conceptual islands. This is one way in which the human history of innovation deviates from the natural history. New ideas do not thrive on archipelagos.