Sunday, October 9, 2016

Augmenting reality without augmenting vision

A common narrative that people tell about virtual and augmented reality (VR and AR) goes something like this: "VR means total immersion in an environment, allowing a game designer to involve you directly in their completely hand-fabricated version of reality. It does this by completely supplanting your field of vision with a simulated 3D environment. AR, on the other hand, only supplants part of your field of vision, allowing overlays of simulated objects and information atop what is otherwise seen normally in the world."

The attentive reader will notice that one sense in particular was heavily emphasized in this explanation: vision. It seems like many people almost take it as a given that supplanting or augmenting reality means changing what we see in a very literal way, and sometimes this idea becomes almost a magic bullet, as though manipulating vision is all it takes to create compelling experiences, as if a more convincing simulation of vision is the main missing piece for telling better stories that center a human interactor. It's for this reason that I've taken to somewhat tongue-in-cheekedly referring to 3D virtual environments as "eye simulators" (thanks @zarawesome) to distinguish them from all the myriad other ways that one could consider rendering, or communicating, a simulation of space, even 3D space (I mean, you can "go up and down" between levels in games where the interface uses conventions from purely textual IF or ASCII-rendered Roguelikes).

Despite my occasional frustration with the hype surrounding VR and "immersive (visual) realism," I believe that constructed, visual virtualities have an awesome potential beyond their current use in games. Recently, Tale of Tales pointed their followers to the Real Time Art Manifesto that they published 10 years ago, and the most interesting part to me is the bit on storytelling, where they actually explicitly reject the idea of using this medium for telling constructed, drama-managed stories:

Embrace non-linearity.
Let go of the idea of plot.
Realtime is non-linear.
Tell the story through interaction.
Do not use in-game movies or other non-realtime devices to tell the story.
Do not create a “drama manager”: let go of plot!
Plot is not compatible with realtime.
Think “poetry”, not “prose”.
The ancient Greek philosopher Aristotle recognized six elements in Drama.
what happens in a play, the order of events,
is only one of them.
Next to plot we have
or the main idea in the work
or the personality or role played by an actor
the choice and delivery of words
the sound, rhythm and melody of what is being said
the visual elements of the work.
All of these can be useful in non-linear realtime experiences. Except plot.
But the realtime medium offers additional elements that easily augment or replace plot.
the direct influence of the viewer on the work
the presence of the viewer in the work
every staging of the work is done for an audience of a single person in the privacy of his
or her
Perhaps this issue is not limited to visual realtime art, as it were; perhaps it's simply a reflection of the new-at-the-time, but by now well-established, idea that indeed there is a tension between allowing full manipulation of an environment (visually realized or not) by an interactor and conveying a structured plot with mandatory authorial beats. But I do think it underscores the main theme of this post: visual and narrative suspension of disbelief are not one and the same.

Perhaps that statement seems obvious, but since interactive narrative researchers who grew up on Star Trek positioned the Holodeck as the holy grail for games, it has been surprisingly difficult to disentangle these two things. Let's call it the Holodeck Fallacy.

When you presume the Holodeck Fallacy, it's all to easy to draw the conclusion that if VR aspires to be the Holodeck, then augmented reality should aspire to be the interface from Minority Report:

Microsoft certainly seems to have made this assumption with the Hololens, and to a lesser extent so has Nintendo.

But recently, I've been much more interested in the ways that our reality can and has already been meaningfully augmented without manipulating one's visual field: specifically, through audio.

Audio as an alternative augmenter

Whenever we talk about augmenting reality, we need to answer two questions:

1) Which part of reality? What is the "default" thing that we expect a human to be doing, in which we are going to intervene with some computational process?

2) What are we augmenting it with?

A really interesting and increasingly well-explored answer to (1) is physical location, as in "location-based games," and to take a really specific example, running for physical exercise. Well, okay, it's interesting to me because it's an activity I happen to enjoy and engage in regularly, but here are some augmented running experiences I have had that fundamentally changed the way I experience that activity:

- Listening to a handcrafted, or algorithmically-generated, playlist of music.
- Using a GPS tracker that occasionally (on which occasion depends on app settings) reports my distance, pace, time, heart rate, and/or other information it can access.
- Playing Zombies, Run!

While the answer to (2) is "audio" for all of these, only the last example also has the answer "story." Largely, until recently, augmentation has been used primarily for adding more factual information to one's day-to-day experience; heck, even a wristwatch could be seen in this light. Collecting data about your geographical trail and repeating it back to you, which in turn may affect your behavior, seems to logically follow from other informational gadgets, but using the same thing for storytelling feels like it's treading new ground.

The way Zombies, Run! works is that, essentially, you are listening to a radio play about a zombie post-apocalypse while you run, which is narrated to you as though you are a character in the story receiving communication from a base through a headset. The story interspersed with lengths of silence in order to space it out to take the amount of time specified by you to match the length of your run.

If that were all there were to it, Zombies, Run! would be nothing but an amusing second-person podcast, which itself does do interesting things as augmented reality: it allows your imagination to connect the visual channels, as well as other bodily senses associated with what you feel while you run, to map onto narrative events. This in itself is interesting.

There are a few more tricks, however: 1. as you run, you (at random, I think) collect items such as med packs, clothing, and books that, after your run, can be used to build up an in-game base to which you have an interface through the app. This mechanism cleverly separates the "staring at a screen" part of the game from the running part. 2. At your own requested frequency, your run will be peppered by zombie chases: you get a warning that they are a certain distance behind you, and if you pick up your pace, you outrun them. If you don't get your pace up quickly enough, they catch you, and you lose one or more pieces of collected inventory.

This last mechanism is interesting mainly because of how it lets reality affect virtuality, not just the other way around. It's almost a cheap form of "biofeedback" that circumvents sensors by using your GPS position plotted over time as an indirect measure of your physical actions. It's one of the few ways you can actually "interact" with the game as you run, since otherwise your location doesn't really matter.

Of course, one could go further with the idea of adding a feedback line from the player's current situation into a narrative fiction. A story told to you while running but where your own location information played into the narration, and where turning in a particular cardinal direction could affect the course of the story, could be interesting. Such things are already sometimes hand-crafted; one thinks of old museum "audio tours," their extension into self-guided city tours, and more recent projects like Improv Everywhere's flashmob project, "The MP3 experiment."

What taking this idea further means, then, is coming up with new enumerations of augmentable activities (walking and running, yes -- but what else?), new means of augmenting them, and, to inform their pairings, new ways that these two things might influence one another. How might an audio story change the way that someone traverses a space, and vice versa? How could we use the data available through a mobile device's sensors -- voice, accelerometer, location, elevation -- to influence a response from a helpful guide or a cunning adversary? Could one make an AI version of the narrator from The Stanley Parable that crafts routes for you to follow in any given (well-mapped) location and reprimandingly adapts to your diversions?

In general, I love the idea of a voice speaking in my ear as I move about a space otherwise in solitude -- telling me things about what I am seeing, suggesting avenues for exploration, or augmenting my visual perception with fiction. The last has the power to transform the ordinary or the mundane, perhaps environments that I see every day, into magical objects and spaces, to imbue them with new meaning and appreciate them in a new light. That, to me, is the real appeal of augmented reality, and it's possible -- perhaps even better -- to do it all without a heads-up display.

Augmenting virtuality with people

One of the ways I've been augmenting my runs recently is by listening to podcasts, and this morning I discovered Imaginary Worlds, a podcast about science fiction and fantasy across different media. The first one I listened to was about Then She Fell, a recent immersive theater project by New York's Third Rail Productions.

I've thought and written before about immersive theater, but thinking about it anew in the context of augmented realities made me see connections I hadn't previously. Imaginary Worlds narrator Eric Molinsky's comments with regard to Then She Fell that what felt compelling to him was the intimacy, the experience of having an actor delivering lines inches from your face, making excruciating amounts of eye contact. Not only that, but also listening and responding to everything you say and do with the attention and improvisational cleverness that only humans, so far, really know how to do.

This made me think that people working on augmented reality experiences are really doing a kind of similar thing to what AR designers are trying to do, but they are approaching the objective from opposite directions: where one takes "reality" as primary and then augments it with meaning that comes from something imaginary, a virtuality, the other takes the fiction, the "virtuality," as primary and substitutes a standard literary figure -- a character in the story -- with something from "reality," namely a guest to the production who doesn't know how the story will play out. Immersive theater substitutes for stage makeup and exaggerated drama, the intensity our brains generate when a real live person in front of us is expecting interaction.

Molinsky notes that one thing he didn't like about Then She Fell was the ambiguity, or perhaps even under-thought quality, of the "audience character:" "I didn't know who I was supposed to be," he says. In other words, what is the audience member's role in the story? In the framing of augmented virtuality, that this was experienced as a failure mode makes perfect sense: while the flow of information from story to interactor is well-established, because humanity has plenty of examples to follow for how that direction works, the other direction of taking unpredictable interactions and reflecting them into story meaning has only video games as a guide. And in that case, linguistic and performative interfaces have been purposely limited, because otherwise, a process wouldn't know how to handle them -- unlike a human actor.

Incidentally, just last night I finally got around to playing Jason Rohrer's Sleep is Death.

Sleep is Death also plays this "augmented virtuality" trick, but with video games as a starting point rather than theater, such that the "typically automated" function substituted with human choice, rather than a character reading lines in a play, is the game itself. Sleep is Death substitutes the uniformity of pre-programmed game responses with on-the-fly, human responses to player-typed dialogue or actions. The game is networked two-player: one of you is the player and one the controller. After each time the player does some action (which the controller can see), the controller has a limited time window in which they can swap out scenery or sprites, type a response in a speech bubble, or provide some other game-like response. It's possible to, like a producer of a play, spend a good long while constructing your scenes before allowing an audience, but then all the dynamic action (including dialogue and switching scenes) is up to improvisation.

If you, like me, find yourself too impatient with the controller interface to experience the game first-hand, you can look at some of the "flipbooks" generated from play on the game's website, which offer some insight.

To reiterate: the main point of this post is that visual suspension of disbelief is neither necessary nor sufficient for narrative suspension of disbelief, and I worry that in the (well-deserved!) river of attention being poured into visual augmentation and VR, we are in some danger of conflating breakthroughs in these technologies with breakthroughs in storytelling. I would like to see attention paid also to the ways that other senses (audio, haptic, olfactory, gustatory, proprioceptive...?) can augment fictional experiences, as well as to the role of social play, i.e. the potentially transformative role that other real-live humans have to play in shaping these experiences, whether at a safe internet distance, inches from your face, or some virtually-distorted mediation between the two.

Sunday, August 28, 2016

Arguing for your research

Everything from paper abstracts to grant proposals to fellowship applications, at every level from an undergraduate independent study to a full grant proposal as a faculty member, requires one key task: convincing the reader that your research project is any good. Usually "good" more specifically means: does it solve an important problem? Does it address an important issue? Does it explore important unexplored territory? And, if you haven't done it yet, do you have the right tools to solve/address/explore it?

In general, I'm not a huge believer in "formulaic" writing -- the idea that every body of writing ought to be formatted the same way for best results. Especially in creative domains, so much power can be wielded in breaking traditional structures. But for scientific writing, especially project proposals or article submissions, I do find that it really helps to not have to think about how to structure something and instead just plop down a default outline. If it does happen to make sense for the writing in question, it's great -- heavy scaffolding laid down can save time when later editing the details. Philip Guo talks about this kind of scaffolding in terms of the hierarchical structure (tree, outline) of written text that mediates between the messy undirected web of concepts in our head and the linear string of text that communicates to the reader. I like this framing in general; I want to talk more specifically about the contents of a certain kind of top-level outline.

A formula that I have found especially helpful for everything from abstracts to grant proposals is something I learned from CMU's Global Communications Center, the so-called novelty moves. This strategy proposes three steps:

  1. Establish the territory;
  2. Identify a gap;
  3. Fill the gap with your research.

I have been presenting this structure to the undergraduate students I'm advising this semester in a little bit more detail; my version goes (operative words bolded):

  1. Motivate the research area.
  2. Provide context of what has been achieved in that area.
  3. Identify a gap or a compelling research question.
  4. Describe the approach we're going to take to fill or explore it.
  5. Describe the impact our work will have on (1) if we're successful.

The interesting thing to me about "structures" like these is that they're always given in sequential (list) order, the same way the final writing product will be, but what they really pertain to is an argument structure. Each step of this plan serves a communicative purpose, and the sequence as a whole satisfies a communicative goal.

A bunch of my prior research has focused on formalizing narrative structure for written text or games that are designed to entertain, to tell stories leading to rich emergent interactions between characters. With so much thinking about the structure of scientific arguments, I've instead been thinking of the structure found in those. In fact, my postdoc project at UCSC involved a formalization of proceduralist readings, which are effectively arguments about what a game means. We realized that we could use logic programming techniques to construct these arguments from a set of hand-authored rules (paper coming out soon!).

Each line of the novelty moves serves a purpose -- in conjunction with some axiomatic assumptions (e.g. "My reader believes field X has value"), the line in the argument serves to satisfy the goal "convince the reader that my research solves an important problem related to field X (which they believe has value)." If it doesn't work toward that purpose (or if the reader can't infer its purpose to that end), it will confuse the reader; if one of the assumed inferences or axioms doesn't hold, the reader will fail to be convinced. Of course, formal logic was originally invented for the purpose of formalizing arguments, so it's no surprise that their structure winds up looking very proof-like. (Then again, the inference rules that occur in human cognition are pretty different from those used in formal logics.) It seems like a perfect opportunity to unify narrative discourse generation and formal logic.

So there you go. I've been so fixated on research that now I want to do research on research writing.

Thursday, August 18, 2016

First semester as a tenure-track professor begins!

In the wake of the busiest Spring and Summer of my life, it appears that I have neglected to inform this blog of a major change to my academic life: I have accepted a position as a tenure-track Assistant Professor in the Computer Science Department at North Carolina State University! The semester just started yesterday; I taught my first Ph.D. seminar class on Generative Methods.

At NCSU I am joining an active Games research community with colleagues Dave RobertsTiffany Barnes, and -- as a result of his move this summer from UCSC -- Arnav Jhala. These folks also have very active research programmes in in learning technology (Barnes), visual narrative (Jhala), and dog cognitive science (Roberts) (yes, you read that right) (dognitive science?). I also have a number of new colleagues doing research in Software Engineering, HCI, and AI topics that overlap with my PL, tool design, and logic interests, so I'm excited about learning more!

Once I get my research priorities established and a webpage set up, I'm going to be actively recruiting Ph.D. students, so keep an eye out for that post.

Saturday, May 14, 2016

Learning to love conferences

Jean Yang, Nadia Polikarpova, and Shachar Itzhaky recently put together a post giving advice to a first-time academic conference-goer. Looking at this advice from where I am now, as a semi-experienced academic with several conference communities I really love to be around, the advice resonates with me, to an extent: if I prepare well, a conference can be a great opportunity to extend the breadth of my network and find new collaboration opportunities.

But careful preparation and bold approaches of strangers were not, by a long shot, how I learned to enjoy conferences. And enjoyment was, for me, a crucial step to being good at them. I don't want to say you have to enjoy conferences in order to be good at them, but, since they are primarily social & emotional labor, sincerely enjoying yourself will make the job easier.

Personally, when I first received advice like Jean et al.'s as a new grad student, I had very bad experiences. I went in believing that I was all but disallowed from talking to people I already knew, and that the path to success was to go up to strangers I wanted to meet and try my hardest to impress them with my research. Here's the problem: I've struggled with social anxiety my entire life, and only started treating it several years in to grad school. So... the first several times I mentally prepared myself to approach someone and introduce myself at a conference, I had panic attacks -- shaking uncontrollably, difficulty breathing, nausea, &c. I stopped being functional at all and had to retreat to my hotel room to recover. Not the best way to make friends at a conference!

The few times I happened into conversations with new folks, as soon as my research came up, I felt so much pressure to impress that I did a terrible job of explaining it, and utterly failed to communicate excitement or interest in my work (or theirs), because there were so many negative feelings behind the activity I felt like I was trying to take part in. I came away from my first few conferences thinking, "Networking is just one of those necessary evils of academia, where everyone has to be fake to get ahead, and I'm never going to like it or be good at it."

Reflecting on the difference between this attitude and my current one wherein I look forward to conferences and find them energizing and inspiring, I can list a few things that seem like they played key roles. Of course, combating social anxiety should definitely include professional therapy if at all possible, which I've discussed in a prior post. But allow me to stick to the conference-specific points for this post.

  1. Acknowledging that conference anxiety is common was important for me -- hearing from my peers that they, too, find conferences exhausting and socially draining, helped me feel less alone. Realize that most folks who have a great deal of facility in a given conference environment have probably built up relationships with that community over several years, and don't expect yourself to have the same facility immediately. It's okay to be a newcomer, and it's okay not to be neurotypical.
  2. Going in with a cohort and/or a buddy that I could treat as "home base." Counter to the "don't talk to people you know" advice, I'd say it can actually be really important and helpful to do so -- not constantly, but simply as an anchor point. Have someone you can check in with to say "Hey, I just met [really famous researcher] and she was super nice to me! Do you want an introduction?" or "Wow, that person was kind of rude in Q&A, huh?" or whatever. Help each other out!

    As a personal concrete example: when I was first getting to know the games research crowd at FDG last year, being in a cohort of UCSC folks, who are well known as a group, and being able to introduce myself in relationship to them, was really enormously helpful in getting a foothold with strangers.
  3. Not being afraid to sacrifice breadth for depth. Sometimes it feels like you're not winning the conference game unless you're meeting as many people as possible, and during a great in-depth research discussion, it might be tempting to say "let's continue later, I need to circulate around the room." I've actually found it useful not to do this. As someone who tends to get more mileage out of few close connections than loads of shallow ones, allowing myself to really spend time with one or two new people has had a lot better future payoff.
  4. Posting to social media (in my case Twitter) has been a surprisingly effective way to make friends at conferences. Use the conference hashtag; post your notes or thoughts about a talk; check the hashtag yourself and reply to other people conversing. Often what happens is someone will ask to say hi to you in person if they'd like to talk in more depth about a comment you've made. I'm a lot more comfortable speaking my mind online, so this has been a great way to take advantage of my current comfort zone in order to expand it.
  5. On the hallway track: a common adage about conferences is that they're "mostly about the hallway track" and "the talks are less important than the people." I agree, but it took me a long time to get there, and when I first started out, I wasn't really sure how to apply this advice, because I'd often waste the hallway tracks standing around awkwardly with no one to talk to. Then, at lunchtime or whatever, someone would ask a question about that one great talk and I'd have nothing to say because I had skipped it. As a newer member of a community, I've found splitting talk/hallway track more like 70/30 (rather than my typical 30/70 these days) to be beneficial. And be sure to ask any senior members of the community you know which talks are not to be missed. (As a general rule, don't skip keynotes).
  6. Finding my people in sometimes-arbitrary ways can be helpful. For example, if the catered lunch isn't vegetarian friendly, asking a group (online or otherwise) who wants to go visit the nearest Indian restaurant/Chipotle/fancy local vegan place can be a great way to carve a subset of people with whom you immediately have a common discussion topic. Doing this by research interest -- e.g. at FDG, my roommate and I organized a "computational narrative" informal birds-of-a-feather meetup -- can be even better. (See point 2: having a "partner in crime" to help you spearhead things like this can be super useful!)
  7. Giving myself room to acknowledge and care for my needs has probably been the most critical change in behavior for being happy at conferences. At literally every conference, even now, I pick 1 or more sessions to skip just to do whatever feels most comforting to myself: lying in my hotel room with a book, sitting in a park, working out, grabbing tea with a close local friend, even just camping somewhere inconspicuously adjacent to the conference with my laptop and working on some research code or reading Twitter. It's really, really okay to disengage to whatever extent you need to, and it's been crucial for helping me conserve energy to keep being socially involved throughout longer conferences.

On the conference organization side, here are some things your committee can do to help guests with (my particular flavor of) social anxiety feel more comfortable and welcomed:

  1. Host the conference either in, or very close to, the hotel(s) where folks are staying so that it's easy to get back to your room.
  2. Host the conference somewhere in an environment where it's easy to get away for short durations: an urban environment, a campus with public spaces, or somewhere close to good transit options.
  3. Have a quiet room.
  4. To the extend that budget allows, do catered or otherwise on-site lunches (taking care to accommodate a wide range of diets!). It avoids awkward "will the team captain pick me last" situations where people have to self-organize into meal groups.
  5. Do some (optional) group off-site activities like hikes or board game nights, which allows folks to interact in a more casual, low-stakes environment if they wish.
  6. Hold post-talk discussions in their own room rather than doing Q&As, so that even those who are nervous about talking in front of large groups of people can have their thoughts about a talk heard. (Bonus: this saves time in the talk schedule!)
  7. Have a mentoring workshop or doctoral consortium to help acclimate junior researchers to the conference environment and provide a space for asking "beginner questions."
  8. Especially for the mentoring activities noted in (7), but also just in general, do all the usual things for ensuring you have a diverse line-up of conference speakers and leadership. There's nothing like being not only a brand-new member of a community with few connections but also feeling like a total outsider in terms of your core identity.

Thursday, April 21, 2016

Two new drafts

I'm pleased to report that two new papers have been accepted to upcoming conferences!

A short abstract "Towards Computational Support for Experimental Theater" (2 pages) has been accepted for presentation at FDG/DiGRA.

And a new collaboration with Rogelio Cardona-Rivera on "Discourse-driven Comic Generation" (8 pages), building off work previously featured on this blog, has been accepted for poster presentation (and inclusion in proceedings) at ICCC.

These papers both represent very preliminary work, so I'm looking forward to conversations about these ideas with the broader community.

Monday, April 18, 2016

Call for Papers: International Conference on Interactive Digital Storytelling (ICIDS 2016)

I'm on the program committee for ICIDS 2016! CFP copied below.


ICIDS 2016
November 15-18, 2016

The Institute for Creative Technologies
The University of Southern California
Los Angeles, USA

Tuesday, February 9, 2016

Call for Papers: Trends in Functional Programming 2016

I'm on the program committee for TFP, the Trends in Functional Programming Symposium! TFP is an unconventional conference: papers are very lightly reviewed in order to be accepted at the symposium, then after they are presented, another phase of refereeing happens to select a subset of the articles for formal publication.

The Symposium is June 8-10 at the University of Maryland, and draft paper submissions are due April 8!

Quoting the scope from the official CFP, TFP is interested in the following kinds of articles:
  • Research Articles: leading-edge, previously unpublished research work
  • Position Articles: on what new trends should or should not be
  • Project Articles: descriptions of recently started new projects
  • Evaluation Articles: what lessons can be drawn from a finished project
  • Overview Articles: summarizing work with respect to a trendy subject
Topics of interest include (but are not limited to):
  • Functional programming and multicore/manycore computing
  • Functional programming in the cloud
  • High performance functional computing
  • Extra-functional (behavioural) properties of functional programs
  • Dependently typed functional programming
  • Validation and verification of functional programs
  • Debugging and profiling for functional languages
  • Functional programming in different application areas: security, mobility, telecommunications applications, embedded systems, global computing, grids, etc.
  • Interoperability with imperative programming languages
  • Novel memory management techniques
  • Program analysis and transformation techniques
  • Empirical performance studies
  • Abstract/virtual machines and compilers for functional languages
  • (Embedded) domain specific languages
  • New implementation strategies
  • Any new emerging trend in the functional programming area
More information can be found on TFP's website.