There are many different flavors of web cache poisoning discovered by Security Researcher James Kettle. Read on for an explanation of one I’ve run across…

What is a Fat GET/HEAD Request? A GET or HEAD request is “fat” when it has a request body. It’s unexpected! Typically one sees a request body with a POST or PUT request because the body contains form data. The HTTP specification says that including a request body with GET or HEAD requests is undefined. You can do it, and it’s up to the application to figure out what that means. Sometimes it’s bad!

You can get a sense of the applications that intentionally support Fat Requests (and how grumpy it makes some people) by reading through this Postman issue.

Fat Requests can lead to CDN and cache poisoning in Rails. CDNs and caching web proxies (like Varnish) are frequently configured to cache the response from a GET or HEAD request based solely on the request’s URL and not the contents of the request body (they don’t cache POSTs or PUTs at all). If an application isn’t deliberately handling the request body, it may cause unexpected content to be cached and served.

For example, you have a /search endpoint:

• GET /search shows a landing page with some explanatory content
• GET /search?q=foo shows the search results for “foo”.

• Here’s what a Fat Request looks like:

  GET /search     <== the url for the landing page
  
  q=verybadstuff  <== oh, but with a request body
  

In a Rails Controller, parameters (alias params) merges query parameters (that’s the URL values) with request parameters (that’s the body values) into a single data structure. If your controller uses the presence of params[:q] to determine whether to show the landing page or the search results, it’s possible that when someone sends that Fat Request, your CDN may cache and subsequently serve the results for verybadstuff every time someone visits the /search landing page. That’s bad!

Here’s how to Curl it:

curl -XGET -H "Content-Type: application/x-www-form-urlencoded" -d "q=verybadstuff" http://localhost:3000/search

Here are 3 ways to fix it…

Solution #1: Fix at the CDN

The most straightforward place to fix this should be at the caching layer, but it’s not always easy.

With Cloudflare, you could rewrite the GET request’s Content-Type header if it is application/x-www-form-urlencoded or multipart/form-data. Or use a Cloudflare Worker to drop the request body.

Varnish makes it easy to drop the request body for any GET request.

Other CDNs or proxies may be easier or more difficult. It depends!

Update via Mr0grog: AWS Cloudfront returns a 403 by default.

Solution #2: Deliberately use query_parameters

Rails provides three different methods for accessing parameters:

• query_parameters for the values in the request URL
• request_parameters ) for the values in the request body
• parameters (alias params) for the problematic combination of them both. Values in query_parameters take precedence over values in request_parameters when they are merged together.

Developers could be diligent and make sure to only use query_parameters in #index or #show , or get routed actions. Here’s an example from the git-scm project.

Solution #3: Patch Rails

Changes were proposed in Rails to not have parameters merge in the body values for GET and HEAD requests; it was rejected because it’s more a problem with the upstream cache than it is with Rails.

You can patch your own version of Rails. Here’s an example that patches the method in ActionDispatch::Request:

# config/initializers/sanitize_fat_requests.rb
module SanitizeFatRequests
  def parameters
    params = get_header("action_dispatch.request.parameters")
    return params if params

    if get? || head?
      params = query_parameters.dup
      params.merge!(path_parameters)
      set_header("action_dispatch.request.parameters", params)
      params
    else
      super
    end
  end
  alias :params :parameters
end

ActionDispatch::Request.include(SanitizeFatRequests)

# Some RSpec tests to verify this
require 'rails_helper'

RSpec.describe SanitizeFatRequests, type: :request do
  it 'does not merge body params in GET requests' do
    get "/search", headers: {'CONTENT_TYPE' => 'application/x-www-form-urlencoded'}, env: {'rack.input': StringIO.new('q=verybadstuff') }

    # verify that the request is correctly shaped because
    # the test helpers don't expect this kind of request
    expect(request.request_parameters).to eq("q" => "verybadstuff")
    expect(request.parameters).to eq({"action"=>"panlexicon", "controller"=>"search"})

    # the behavioral expectation
    expect(response.body).not_to include "verybadstuff"
  end
end

GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails. I recently released two new features:

• GoodJob::Bulk to optimize enqueuing large numbers of jobs (released in GoodJob v3.9)
• GoodJob::Batch to coordinate parallelized sets of jobs (released in GoodJob v3.10)

Big thanks to @julik, @mollerhoj, @v2kovac, @danielwestendorf, @jrochkind, @mperham and others for your help and counsel!

Bulk enqueue

GoodJob’s Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can be more performant when enqueuing a large number of jobs.

I was inspired by a discussion within a Rails pull request to implement perform_all_later within Active Job. I wanted to both support the way most people enqueue Active Job jobs with perform_later and also encourage people to work directly with Active Job instances too.

# perform_later within a block
active_jobs = GoodJob::Bulk.enqueue do
  MyJob.perform_later
  AnotherJob.perform_later
end
# or with Active Job instances
active_jobs = [MyJob.new, AnotherJob.new]
GoodJob::Bulk.enqueue(active_jobs)

Releasing Bulk functionality was a two-step: I initially implemented it while working on Batch functionality, and then with @julik’s initiative and help, we extracted and polished it to be used on its own.

Batches

GoodJob’s Batch functionality coordinates parallelized sets of jobs. The ability to coordinate a set of jobs, and run callbacks during lifecycle events, has been a highly demanded feature. Most people who talked to me about job batches were familiar with Sidekiq Pro ‘s batch functionality, which I didn’t want to simply recreate (Sidekiq Pro is excellent!). So I’ve been collecting use cases and thinking about what’s most in the spirit of Rails, Active Job, and Postgres:

• Batches are mutable, database-backed objects with foreign-key relationships to sets of job records.
• Batches have properties which use Active Job’s serializer, so they can contain and rehydrate any GlobalID object, like Active Record models.
• Batches have callbacks, which are themselves Active Job jobs

Here’s a simple example:

GoodJob::Batch.enqueue(on_finish: MyBatchCallbackJob, user: current_user) do
  MyJob.perform_later
  OtherJob.perform_later
end

# When these jobs have finished, it will enqueue your `MyBatchCallbackJob.perform_later(batch, options)`
class MyBatchCallbackJob < ApplicationJob
  # Callback jobs must accept a `batch` and `params` argument
  def perform(batch, params)
    # The batch object will contain the Batch's properties, which are mutable
    batch.properties[:user] # => <User id: 1, ...>
    # Params is a hash containing additional context (more may be added in the future)
    params[:event] # => :finish, :success, :discard
  end
end

There’s more depth and examples in the GoodJob Batch documentation.

Please help!

Batches are definitely a work in progress, and I’d love your feedback:

• What is the Batch functionality missing? Tell me your use cases.
• Help improve the Web Dashboard UI (it’s rough but functional!)
• Find bugs! I’m sure there are some edge cases I overlooked.

Excerpts from the excellent RailsConf 2022 keynote: The Success of Ruby on Rails by Eileen Uchitelle [reformatted from the transcript]:

Upgrading is one of the easiest ways to find an area of Rails that can benefit from your contributions. Fixing an issue in a recent release has a high likelihood of being merged.

Running off Rails Main is another way to find contributions to Rails. If you don’t want to run your Main in production, you could run it in a separate CI build. Shopify, GitHub and Basecamp run it.

Running off Main may be harder than running off a release because features and bug fixes are a little in flux sometimes. If you are running off of Main, a feature added to the Main branch could be removed without deprecation. This is a worthwhile risk to take on because it lowers the overall risk of an upgrade. When you run off Main, you’re less likely to fall behind upgrading because it becomes part of your weekly or monthly maintenance. Upgrading becomes routine, second nature rather than novel and scary. Changes are easy to isolate. It’s just slightly less polished. Like I said, I still think it’s pretty stable.

Another way to find places to contribute is look at your own applications.

• Do you have monkey patches on Rails code that are fixes bugs or changing behavior? Instead of leaving them there, upstream the fix and delete the monkey patch.
• Is there infrastructure level code that doesn’t really pertain to your product? It’s possible this could be a great addition to Rails. When I wrote the database in Rails, it came from GitHub’s monolith. It made perfect sense because it was getting in the way of upgrades, didn’t expose any intellectual property, had nothing to do with your product features and
  something many applications could benefit from.
• Lastly and most importantly, keep showing up.

… Ultimately, if more companies treated the framework as an extension of the application, it would result in higher resilience and stability. Investment in Rails ensures your foundation will not crumble under the weight of your application. Treating it as an unimportant part of your application is a mistake and many, many leaders make this mistake.

…leaders see funding open source risky is because they don’t actually understand the work. … Often, leaders worry if there’s a team working in open source, other teams are going to be jealous or resentful that that team is doing “fun” work. …

Maintainers need to make changes, deal with security incidents and also handle criticism from many people. Maintaining and contributing to open source requires a different skill set than product work. That doesn’t make it any less essential.

…Many product companies don’t like words like “research” and “experimental.” They can imply work without goals. Use words like “investment.” And demonstrate the direct value will bring. Make sure it is measurable and will make the application and product more successful. A great example of measurable work is a change that improves performance. If you can tie contributions to direct customer improvements, it’s easier to show leadership.

…As I started contributing more and more and pealing back the layers of Rails, the impact is limitless. I started looking at how applications stretched the boundaries of what Rails was meant to do.

…Ultimately, I want you to contribute to Rails because it’s going to enable you to build a better company and product. The benefits of investing in Rails go far beyond improving the framework.

Investing in Rails will build up the skills of your engineering team. They will developer better communication skills, learn to navigate criticism, debugging skills and how the framework functions. It will teach engineers about the inner-workings and catch bugs.

Monkey patching is far more dangerous than I think most realize. They break with upgrades and cause security incidents. When you write a monkey patch, you maintain a portion of Rails code. Wouldn’t it have been better to patch it upstream rather than taking on that risk and responsibility.

It will give your engineering team the skills they need to make better technical decisions. You’re ensuring that Rails benefits your application and the company for the long-term.

…Contributing to Rails is only not essential if you don’t care about the direction the framework is headed in. We should be contributing because we care about the changes.

We want to ensure our apps are upgradeable, performant and stable.

Investment in Rails also means you won’t have to rewrite your application in a few years because Rails no longer supports what you need. When you fail to invest in your tools, you end up being unable to upgrade. Your engineering team is miserable. The codebase is a mess and writing features is impossible. You’re forced into a rewrite, your engineers want to write Rails and you can no longer let them do that. You have to build a bunch of features before you site falls over.

It’s not Rails’ fault you made the decision to invest elsewhere.

If you build contributing into your culture, the benefits are clear:

• Your engineering teams’ skills will improve.
• Rails will evolve with your application because you’re helping decide how it needs to change.
• Your application will be more resilient because there’s low tech debt and your foundation is stable. Active investment prevents your application from degrading.

Building a team to invest in Rails is proactive. Rewriting an application is reactive. Which one do you think is better for business in the long run?

GoodJob is a multithreaded, Postgres-based, Active Job backend for Ruby on Rails. GoodJob has many features that take it beyond Active Job. One such feature is cron-like functionality that allows scheduling repeated jobs on a fixed schedule.

This post is a brief technical story of how GoodJob prevents duplicated cron jobs from running in a multi-process, distributed environment.

This is all true as of GoodJob’s current version, 3.7.4.

Briefly, how GoodJob’s cron works

GoodJob heavily leans on Concurrent::Ruby high-level primitives, and the cron implementation is no different. GoodJob::CronManager accepts a fixed hash of schedule configuration and feeds them into Concurrent::ScheduledTasks, which then trigger perform_later on the job classes at the prescribed times.

A locking strategy is necessary. GoodJob can be running across multiple processes, across numerous isolated servers or containers, in one application. GoodJob should guarantee that at the scheduled time, only a single scheduled job is enqueued.

Initially, advisory locks

When GoodJob’s cron feature was first introduced in version 1.12, Cron used an existing feature of GoodJob: Concurrency Control. Concurrency Control places limits on how many jobs can be enqueued or performed at the same time.

Concurrency Control works by assigning jobs a “key” which is simply a queryable string. Before enqueuing jobs, GoodJob will count how many job records already exist with that same key and prevent the action if the count exceeds the configured limit. GoodJob uses advisory locks to avoid race conditions during this accounting phase.

There were some downsides to using Concurrency Control for Cron.

• It was a burden on developers. Concurrency Control extends ActiveJob::Base and required the developer to configure Concurrency Control rules separately from the Cron configuration.
• It wasn’t very performant. Concurrency Control’s design is pessimistic and works best when collisions are rare or infrequent. But a large, clock-synchronized formation of GoodJob processes will frequently produce collisions and it could take several seconds of advisory locking and unlocking across all the processes to insert a single job.

Then, a unique index

GoodJob v2.5.0 changed the cron locking strategy. Instead of using Concurrency Control’s advisory locks, GoodJob uses a unique compound index to prevent the same cron job from being enqueued/INSERTed into the database multiple times.

• In addition to the existing cron_key column in job records, the change added a new timestamp column, cron_at to store when the cron job is enqueued.
• Added a unique index on [cron_key, cron_at] to ensure that only one job is inserted for the given key and time.
• Handled the expected ActiveRecord::RecordNotUnique when multiple cron processes try to enqueue the same cron job simultaneously and the unique index prevents the INSERT from taking place

Now, when a thundering herd of GoodJob processes tries to enqueue the same cron job at the same time, the database uses its unique index constraint to prevent multiple job records from being created. Great!

Does it do a good job?

Yes! I’ve received lots of positive feedback in the year+ since GoodJob’s cron moved to a unique index locking strategy. From the application perspective, there’s much less enqueueing latency using a unique index than when using advisory locks. And from the developer’s perspective, it does just work without additional configuration beyond the schedule.

The main benefit is that the strategy is optimistic. GoodJob just goes for it and lets the database’s unique indexing and built-in locks sort it out. That allows removing the application-level locking and querying and branching logic in the GoodJob client; only handling the potential ActiveRecord::RecordNotUnique exception is necessary.

Using a unique index does require preserving the job records for a bit after the jobs have been performed. Otherwise, poor clock synchronization across processes could lead to a duplicate job being inserted again if the job has already been performed and removed from the table/index. Fortunately, preserving job records should not be too burdensome because GoodJob will automatically clean them up too.

Lastly, one goal of writing this is the hope/fear that a Database Administrator will tell me this is a terrible strategy and provide a better one. Until that happens, I have confidence GoodJob’s cron is good. I’d love your feedback!

Edit: In an earlier version of this post, I mixed up “optimistic” and “pessimistic” locking; that has been corrected.

• I had a conversation with a coworker that reminded me (again!) of this Foreign Affairs essay. It was about receiving feedback that seemed to focus on minimizing interpersonal conflict over things like achieving goals and strategy and impact; it felt familiar.
• I’ve had coworkers and executives in other jobs that were like “people need to be more comfortable with conflict”. I’ve also been screamed at in a postmortem at a different job. So like, my feelings are mixed. Like are we talking about the absence of psychological safety, or too much of it? I dunno, I could never draw out specifics. I’m now thinking it was actually about the previous bullet point.
• Schulman makes the point that you’re either in a relationship with someone, or you aren’t.

• I read this Harper’s essay about working at Wired UK by Hari Kunzru with the line:

  The political economist Albert O. Hirschman famously characterized the choice that is faced by people within declining institutions as being between “voice” and “exit.” Either you speak up to change things, or you leave and look for something better.’

• Which led me to find this essay about the book which pulls this lovely quote from Hirschman:

  The ultimate in unhappiness and paradoxical loyalist behavior occurs when the public evil produced by the organization promises to accelerate or to reach some intolerable level as the organization deteriorates; then…the decision to exit will become ever more difficult the longer one fails to exit. The conviction that one has to stay on to prevent the worst grows all the time.

• But this is maybe now more about disfunction than exit. But I’ve also been in the position of being in difficult conflict when explaining that a certain set of strategies will lead to difficulties in attracting and retaining talent. And then most of my favorite people were gone. And then so was I.

• Family: My mom passed away; first parent to go. Of a plus, we spent a lot of time with my brother and his young family. We had to cancel a big family trip to Europe, but hope to make up with a trip in 2023.
• Community: Turned 40 this year; celebrated 11 years in SF; coming up on 3 years on Nob Hill. Things feel good. We were approved as a foster/adopt Resource Family, but with everything that happened with my mom, we haven’t yet begun hosting children yet. I’m going into year three of strategic planning committee at St. Francis.
• Work: Started at GitHub after five years at Code for America. Still feels a bit like a dream that I get to continue to do what I want technically (Ruby and Rails) while also working with good people. I’ve continued to work with my leadership coach, which has been nice continuity.
• Projects & Consulting: It’s now been 10 years since I registered as a small business in SF, which I did when I first started earning money with Day of the Shirt and doing various consulting jobs. GoodJob is floating along, and become a small source of GitHub Sponsors funds. The other constellation of projects continued in maintenance: Brompt, Panlexicon, etc.

From Lee Shevek’s “Is Punishment ‘Carceral Logic’?”:

…the difference between carceral logic and liberatory accountability is not the presence/lack of punishment. Rather, the difference lies in how much power the person who has done harm has. Carceral logic aims to strip them of their personal power, while liberatory accountability processes require that they take ownership of that power. That is, ultimately, what accountability is: taking responsibility for your power as well as for the consequences of your use of it. Recognizing your own agency in having made a choice that resulted in harm, facing the people you hurt, giving them answers and apologies, and claiming your ability to do differently. This is what the carceral system does not allow. It strips people entirely of their agency, requires of them no meaningful repair process, and locks them in a cell where they are ritualistically abused by the State. This is a process that heals no one, nor was it ever even intended for healing or repair. It is a system only of control.

Liberatory accountability processes, on the other hand, demand something incredibly difficult for people who do harm: acknowledgement of their own power, their own responsibility to the harm they do with that power and their obligation to use that same power to make amends. Taking that responsibility also means acknowledging and respecting the consequences for the harm they do. If I truly take a harm I’ve done seriously, if I genuinely see it as harm, then I also will respect that the person I harmed may need to put more boundaries up between us to feel safe again. If the harm is more extreme, I will see the steps the surrounding community takes (closing my access to certain spaces, demanding my participation in ongoing accountability processes, etc.) as important responses to re-establish safety where my actions ruptured it, even if those responses are painful or uncomfortable to me. Absent of these consequences, the people most adept at doing harm while maintaining community support have free reign to continue perpetuating cycles of harm that will reverberate through years (often generations) to come, and survivors flee into solitude because there are no communal norms in place to provide them any real or trustworthy sense of safety. This is, in fact, the status quo of the world we live in now.

The real distinction between carceral logic and liberatory accountability is that one process violently strips someone of their humanity and agency, while the other demands that people who do harm take full command of their humanity and agency to atone for that harm and become better members of the community in the process. The carceral system says: “You are a criminal and you deserve to be subject to constant harm and control because of it.” Liberatory accountability says: “You are a person who chose to do harm, we believe in your capacity to choose to face the consequences of that harm and do what you can to repair it.”

This reminded me of SorryWatch’s “How to apologize: a short checklist”:

APOLOGIZE – Say “I’m sorry” or “I apologize.” Take responsibility. Talk about what you did, not just “what happened.” Avoid “if,” “regret,” and “it’s unfortunate.” Try “I shouldn’t have done that,” “That was rude of me,” or “It was wrong.”

TO THEM – Not just to the twitmosphere, but to the person harmed.

FOR WHAT YOU DID – Be specific. Not “hurting you” but, for example, “calling you a slimy swivel-eyed creep.”

ACKNOWLEDGE THE EFFECT – If you know it. “I embarrassed you by calling you a slimy swivel-eyed creep in front of everybody at our dinner table, and at the nearby tables.”

EXPLAIN, BUT DON’T EXCUSE – “I called you a slimy swivel-eyed creep to try to make you be quiet because I didn’t want to be thrown out before dessert came. I was a jerk.”

STOP TALKING AND LET THEM HAVE THEIR SAY – “I wasn’t upset that you called me a slimy swivel-eyed creep. I was upset that you interrupted my song. It made me feel like you don’t respect me as an artist.”

And accountability (giving an account) fits into themes in Sarah Schulman’s Conflict is Not Abuse, which I’ll requote:

… everyone deserves help when they reach out for it. …the collapse of Conflict and Abuse is partly the result of a punitive standard in which people are made desperate, yet ineligible, for compassion. … people who have suffered in the past, or find themselves implicated in situations in which they are afraid to be accountable, fear that within their group acknowledging some responsibility will mean being denied their need to be heard and cared for.

Gaia

I played a lot of Sim Earth as a kid. It had a mode called Daisyworld based on the “Gaia Hypothesis”:

proposes that living organisms interact with their inorganic surroundings on Earth to form a synergistic and self-regulating, complex system that helps to maintain and perpetuate the conditions for life on the planet.

In Sim Earth’s Daisyworld, that was simulated by populating the Earth with different shades of flower. Darker flowers survived at lower temperatures and absorbed sunlight, warming their immediate environment; lighter flowers survived at warmer temperatures and reflecting sunlight, cooling their immediate environment. Running the simulation would, eventually, usually, lead to a dynamically changing, but still steady-state environment; in Conway terminoloy: still lifes and oscillators.

Usually your Earth found a steady state, unless:

• geographical barriers prevented regulating changes from spreading, like mountain ranges or island archipelagos; or
• disasters like volcanoes or meteors significantly disrupted the environment before the flowers could adapt and moderate it; or
• you plopped some bunnies or herbivores that ate the flowers, because the Gaia mode didn’t lock-out the other Sim Earth tools. Sandboxes!

Usually your Earth found and could recover a steady state! That’s homeostasis:

Homeostasis is brought about by a natural resistance to change when already in the optimal conditions, and equilibrium is maintained by many regulatory mechanisms: it is thought to be the central motivation for all organic action.

Terroir

From Vicki Boykis’s “The Art of the Long Goodbye” :

A few years ago, I read the Southern Reach trilogy, by Jeff Van Der Meer….

One of the main concepts of the books is the idea of a terroir, a self-contained natural environment that shapes everything inside of it. Usually when we talk about a terroir we’re referring to wine: a wine from a given region tastes like wine from that given region should taste because of the grapes and the soil and the way the sunlight hits that particular spot. Since I left my last job, I’ve been thinking a lot about the idea of terroir as it relates to the workplace.

Every workplace, like every Tolstoyan family, is unique in its own way. When we start a job, we enter that terroir with the intent to shape it. But in turn, we are also shaped by it.

Landrace

A landrace is:

a domesticated, locally adapted, traditional variety of a species of animal or plant that has developed over time, through adaptation to its natural and cultural environment of agriculture and pastoralism, and due to isolation from other populations of the species.

As I’m now in the land of big companies, it’s very common to hear someone described as “from Amazon” or “from Microsoft” and like, I know what they mean!

One of my challenges as an engineering leader during the pandemic was this: how much do we adapt our values and practices to be inclusive to the current fucked-all-over situation while preserving what made us, as an engineering organization, us. I think my results were mixed.

An example: pair programming. We were a lot of people from Pivotal labs (there it is!) with a strong belief in Extreme Programming: shared ownership, an active-closeness to the users of our software, and closeness to each other through frequent pair programming. And we went from largely in-person pairing, to remote pairing. I find pair programming to be exhausting in the best of circumstances, and the ongoing pandemic didn’t help. So we adapted our values and practices, and de-emphasized pairing and went from an expectation of “most of the time” to “a tool that’s available some of the time”. It was a stretch (aide: it wasn’t just pairing that got stretched either, the whole XP thing)

A team is never static. People were leaving, we were hiring, teams were forming and reforming. It was tenuous for everyone, to test out the limits of inclusion and identity. And as an engineering leader, it led to a lot of tough conversations about how… we… work… together. It was hard; I think we lost that cultivar.

Oasis

Will Larson writes about this idea of a “Values Oasis”:

A few years ago, I heard an apocryphal story about Sheryl Sandberg’s departure from Google to Facebook. In the story she apologizes to her team at Google because she’d sheltered them too much from Google’s politics and hadn’t prepared them to succeed once she stopped running interference. The story ends with her entire team struggling and eventually leaving after her departure. I don’t know if the story is true, but it’s an excellent summary of the Values Oasis trap, where a leader uses their personal capital to create a non-conforming environment within an wider organization.

Garden

The relationship between people and environment is a problematic metaphor.

I was a member of the Boston Pubic Garden’s Rose Brigade for number of years, taking care of the roses. The Boston Public Garden’s style is Western, specimen, ornamental: large, well-spaced bushes with well-defined blooms. They’re gorgeous. And.

There’s a lot of waste. Trimming, dead-heading, opening up, clearing leaf drops. A lot of waste.

There is a trope, Cincinnatus, in which the tyrant/general/magnate retires (or desires to retire, after the present crisis, of course) to tend garden. I think it usually appears to soften them: see they can be gentle too. But I dunno, gardens aren’t gentle. There’s a lot of centering the gardener, and a lot of will to be exercised in a garden. Other people aren’t plants, don’t take it too far.

I’m reminded of Nirgal’s (spoiler!) fated ecopoetic basin in KSR’s Blue Mars:

Nirgal wandered the basin after storms, looking to see what had blown in. Usually it was only a load of icy dust, but once he found an unplanted clutch of pale blue Jacob’s ladders, tucked between the splits in a breadloaf rock. Check the botanicals to see how it might interact with what was already there. Ten percent of introduced species survived, then ten percent of those became pests; that was invasion biology’s ten-ten rule, Yoshi said, almost the first rule of the discipline. “Ten meaning five to twenty of course.”

And close on it out with a quote from Seeing like a State:

Contemporary development schemes… require the creation of state spaces where the government can reconfigure the society and economy of those who are to be “developed.” The transformation of peripheral nonstate spaces into state spaces by the modern, developmentalist nation-state is ubiquitous and, for the inhabitants of such spaces, frequently traumatic.

Let’s go!

Universe

This past week was GitHub Universe. Local SF Hubbers were invited to volunteer and I did: I worked the info booth and the swag shop. It was nice meeting lots of other local Hubbers that don’t regularly frequent the office.

Performance Review Season

It’s here again. As a manager, I have a template for how I like to receive self-reflections that I’ve been dragging around with me for the past 15 (?!) years.

1. Laundry list all of your projects, contributions, and accomplishments during the time period [if this seems difficult, have some sympathy for your manager]
2. What future growth areas (lined up in phrases from the Career Ladder) would you like to focus on developing?
3. What specific project/work opportunities exist (or you’d like to create) that would help you develop the growth areas you described in the previous question? (or you would really, really want to work on).

And then it’s a matter of trying to shoe-horn those questions into the invariably slightly different organizational template. I wrote up a big document for my reports.

Here’s what I drafted for myself on the opportunities question:

Improve predictability of team output. I would like to better develop a proactive ability to predict team output. Two focuses:

• Tactical Planning: Improving my short/medium-term project management skills to improve how work is defined, broken down, and scheduled with target dates.
• Strategic Planning: Planning and championing high-impact projects that would be engaging for both the team and engineering leadership.

The end result of this, in addition to expanding impact, is growing the Ruby Architecture team and ensuring there is sufficient capacity (and buffers) for our core Areas of Responsibility.

Further develop “out”-ward communication. Improve my ability to brief and influence across teams and leadership at GitHub. Particularly focused on reinforcing the position and importance of Ruby and Rails within GitHub.

What is “Platform”

In my new department, UI & Monolith Platform, all of my sibling teams are platform, but look very different. We’ve been working on a document to share with the rest of the Engineering team about what we would say we do here. As part of that, I’ve been thinking about Maturity Models as an explanation for why our teams look so different. Some teams are early, where they’re still focused on adoption and “fit”, and other teams, like mine, that are so mature that I think our internal consumers are not always fully aware that they’re using the platform we’ve built with huge intention.

A universal playbook

The past week at Universe, continually responding to the question “oh, what do you do?”, has helped me reflect on what I’m working towards. And realizing it feels like the same playbook I’ve run at other jobs. Lots more to it than this, but man, write your end-to-end tests people.

This book has big ’60s energy, believing in the creative potential of a group of men openly jawboning in a room (but the prompts!) to solve problems. I’ve also read The Practice of Creativity and The Innovator’s Handbook too; the former probably being the best of them. They all are mainly persuasive theory and leave me trying to figure out the actual facilitation plans between the lines.

The book as a whole is engaging and has a lot of transcripts of people talking over eachother to solve a problem, as good as any pulp hard/heroic science fiction of the period.

What is synectics?

The word Synectics, from the Greek, means the joining together of different and apparently irrelevant elements. Synectics theory applies to the integration of diverse individuals into a problem-stating problem-solving group. It is an operational theory for the conscious use of the preconscious psychological mechanisms present in man’s creative activity. The purpose of developing such a theory is to increase the probability of success in problem-stating, problem-solving situations. This increase depends on awareness of the mechanisms which must be worked through to arrive at solutions of fundamental novelty.

And play:

Three general types of mechanisms for play have emerged from these studies: (1) play with words, with meanings and definitions; (2) play in pushing a fundamental law or a basic scientific concept “out of phase”; and (3) play with metaphor.

Demo or die:

The most important (and most underrated) single aspect of a Synectics project is the implementation (in the form of working models) of those concepts developed as solutions. Such model building is vital to the success of a new product or invention program. Moreover, we have observed that unless a teaching program includes the experience of “getting the hands dirty” by actively implementing conceptions, the program is threatened with incompletion and impotence precisely because it is limited to over-abstract discussion. …[30 pages]…Exciting as it is for Synectics groups to succeed in making the familiar strange, it is hardly an end in itself. The end is a functioning, working model of the invention product, just as the end result of a narrative idea is not the idea but the novel into which it is transformed, or as the end result of a visual insight is the painting into which it is evolved.”

The general idea of a Synectics group (“in the Industrial Model”) is having the company psychologist pick a small group of [men] to go live and cook together and solve problems for the firm. This is interesting:

In the selection of personnel for Synectics activity, if we are faced with a choice between two individuals of different intellectual backgrounds, but of similar emotional orientations, our tendency would be to choose only one. On the other hand, two individuals having the same intellectual background but different emotional patterns of response could be included in an integrated group which is designed to reflect extreme diversity.

How do you choose? Lots of bias here:

1. Metaphoric Capacity: The candidate’s language is carefully watched for signs of metaphor and analogy as described by the operational mechanisms: Personal Analogy, Symbolic Analogy, Direct Analogy, and Fantasy Analogy. The candidate is encouraged to speak metaphorically…

1. Attitude of Assistance: Because some characteristics of a candidate can’t be identified from conversation alone, the interview includes taking a walk in the woods, inviting the candidate to participate in a project on the property (pipe is being laid or a little bridge is being built over a stream, or a new timber is being fitted to the bam); in the evening the candidate meets with the Synectors for a cook-out. If his attitude of assistance has not revealed itself before, the cook-out preparation is used as the ultimate test. In general there are three kinds of response to this activity:

   1. The candidate sits and watches the fire being laid.
   2. The candidate asks if he can help.
   3. The candidate observes what is needed and supplies it.

   Building a fire is such a universal function, known to everyone, that it serves as an excellent test of assistance attitude. Obviously, for the purpose of Synectics, a man who sees how he can help and does it is the most acceptable. Part of the fire making technique is to make sure that there is not enough kindling on the spot so as to guarantee ample opportunity for the candidate to reveal himself.

3. Kinesthetic Coordination: Although clumsiness is not inconsistent with creative potential, Synectors guard against selecting a man whose extreme lack of coordination implies a lack of self-confidence. [jesus christ]…

1. Risk: In the course of talking to the candidate, Synectors must determine whether he enjoys taking risks and, if so, what kind of risks. Is the candidate a self-destructive gambler who is unconsciously trying to injure himself? Is he willing to risk because he knows it’s the only way to accomplish certain tasks? Does he enjoy risking or does it frighten him?…

5. Emotional Maturity: Creative people tend to have a childlike quality about them, but this childishness is not necessarily a sign of emotional immaturity. The emotionally immature childlike person does not use his childlike surprise, wonder, and infinite curiosity about the world as a psychological basis for creative acts. It is the capacity to integrate childishness into constructive acts which the Synectors look for in their interviews with candidates….

1. The Capacity to Generalize: What are the candidate’s thinking habits? Can he take three or four facts and construct from them a straightforward, conversational, coherent generalization? Can he oscillate from particular facts to theories which embrace and integrate the facts? A generalization is a hypothesis describing and including diverse and sometimes conflicting data. Can he tolerate the ambiguity with which he must live until a soothing, all-ordering generalization explains the data? And then can he act on the basis of his generalization?…

2. Commitment: If the candidate believes in something—product or a concept—can he commit himself to bringing it to life? Or is he self-protectively analytical and falsely sophisticated? The personality characteristic of enthusiasm is not by itself a guarantee. The candidate must identify with a project so that its success is crucial to him…

3. Non-status Oriented: There are traditional symbols of status in American industrial society: carpet on the floor, large clean desks, prints on the wall, name on the door, in charge of a large number of men, conservatively natty clothes, new car—these symbolize the position of a man. Minor variations are permitted within the conventions of this framework to describe the residual differences in personalities. People chosen for Synectics activity must be beyond status as defined by the traditional symbolism because their group will develop another kind of status based on contribution and independence…

4. Complementary Aspect: No candidate can get a perfect score in all the criteria but the group as a whole should make up 100% of the characteristics implied by the list of criteria. Also, there are certain special personality traits within the candidates which must be balanced. Assume two men who score high, one from research, one from sales. The researcher, introverted and studious, rates the salesman as flamboyant and loud. The salesman calls the researcher mousey and secretive. Each man distrusts the other, yet both are necessary to the success of a Synectics operation. [o_O] Someone must be found who can bring together the researcher and salesman. The Synectors, on the lookout, decide on a man who had been in research but now, for instance, is in sales engineering. This “integrator” is interviewed, not only on the basis of the criteria, but also to learn whether his personality can resolve the conflict between the researcher and the salesman.

But what do they do?!

The Synectic process involves:

1. making the strange familiar;
2. making the familiar strange.

…Synectics has identified four mechanisms for making the familiar strange, each metaphorical in character:

1. Personal Analogy [how would you live/feel if you were the subject under study?]
2. Direct Analogy [“the actual comparison of parallel facts, knowledge, or technology”]
3. Symbolic Analogy [“views the problem qualitatively with the condensed suddenness of a poetic phrase.”]
4. Fantasy Analogy [“How do we in our wildest fantasies desire the subject to operate?”]

Oh brother.

Every mature institution or organization—industrial, governmental or educational—functions according to laws which make up its policy. For example, every company has its own “laws” about personnel, procurement, borrowing money, sales, production, salaries, quality control, customer relations, product line, budgets, advertising, etc. If the laws represent a true response to reality, they will be useful. But often, particularly in the case of an old established company, the laws are habits carried forward from a glittering past. Long-term employees from a middle management level up constitute an intra-organizational community which derives comfort from these familiar laws—obsolete or not. This community is the keeper of the scrolls. The laws are the anthropological shibboleths by which the industrial tribe conducts its ritual of business. A Synectics group is a fifth column acting from within the organization community to make the familiar “laws” strange.