I'm prototyping a new approach to drawing dynamic pictures with direct manipulation. So far, I've been calling it "Apparatus with Magnets". We'll see if the name... sticks. 😉

This document lays out some background, motivation, and rough plans. Please take a look, and get in touch if it spurs any thoughts you'd like to share or discuss. Thanks!

Table of Contents

Drawing dynamic pictures

My goal for this project is to try out some techniques which I hope can be used to draw a broad range of dynamic pictures. The best write-up of why drawing dynamic pictures is important is Bret Victor's essay Dynamic Pictures, but I'll give a quick summary here. (Feel free to skip this if you're already in the know.)

Dynamic pictures are pictures that change when you change some input. This is a broad category, encompassing:

data graphics

David A. Hathaway, via Tufte

David A. Hathaway, via Tufte

Japanese National Railroad, via Tufte

Japanese National Railroad, via Tufte

generative art

Vera Molnár

Vera Molnár

Jared Tarbell

Jared Tarbell

interactive diagrams

Red Blob Games

Red Blob Games

PhET

PhET

parametric CAD

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/8e851510-00d6-4220-aa8a-5abc5f83101a/FilthyFakeBlackwidowspider-max-1mb.gif

dynamic brushes

Jennifer Jacobs et al.

Jennifer Jacobs et al.

software UI... page layout... (...)

What these examples all have in common is:

  1. they are visual, but also
  2. they change when something else changes (a data source, a user's interaction, a parameter, etc.).

Existing drawing tools (like Illustrator) can't create the dynamic, responsive behaviors described in #2. Currently, the only real option for open-ended creation of dynamic pictures is traditional textual code. But moving into the world of code pulls the creator out of the world of the pictures they are creating. This is "coding" dynamic pictures, not "drawing" them. (Processing may call its programs "sketches", but it is a yearning, bittersweet metaphor.)

As Bret has compellingly argued (in Dynamic Pictures, Drawing Dynamic Visualizations, and elsewhere), this situation severely limits our society's ability to think, communicate, and express ourselves in the medium of dynamic pictures. To advance this medium, we need to find ways to usefully represent and modify dynamic structures as part of our pictures. It's not obvious how to do this!

Apparatus

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/ca2382e5-9806-43fc-8c22-d5dc9dba5368/Screen_Shot_2020-06-28_at_2.13.06_PM.png

Toby Schachman's Apparatus is a particularly successful prototype in the direction of drawing dynamic pictures. It is "a hybrid graphics editor and programming environment for creating interactive diagrams". Apparatus combines a vector-graphics editor (like Illustrator) with a reactive spreadsheet (like Excel). The vector-graphics editor shows shapes you can drag around. The spreadsheet has cells representing parameters of these shapes. Linking these spreadsheet parameters together with formulas creates dynamic relationships in a diagram.

Apparatus is based on three additional innovations which mesh wonderfully with this hybrid structure, taking it far beyond where one might expect: