Spring 2015 courses

Interactive Music (Yotam Mann)

The idea of Interactive Music (empowering listeners/players to personalize, perform and explore composed music in greater depth) has been expanding in popularity in recent years with big name artists such as Brian Eno, Radiohead and Bjork taking advantage of the musical possibilities that interactive technologies afford. This is convergent with the democratization of music-making softwares and the rise of DJ’ing which has expanded the audience for interactive music to a growing group of people who don’t see music as something to enjoy passively, but to actively participate in. Interactive music is not necessarily about generative music or audio synthesis (though it does includes these) but about realizing a musical idea as a collaboration between the composer and the listener. This course will guide students to make their own interactive music and musical projects while considering how interaction enriches and augments the experience of the music.

The course will be structured around 1 final assignment in which students create an interactive music project/performance. They will be encouraged to use Javascript, but ultimately, the language or platform is up to the students. The initial assignment will be a low-tech / no-tech interactive music project followed by intermediary assignment in which students explore methods and dimensions of musical interaction which they can build off of for their final assignment. Intermediate projects will also give students a chance to learn and apply the lessons on Javascript and the Web Audio API (specifically a framework called Tone.js).

Possible projects might include: adaptive-length songs, music-based games, reactive/responsive compositions, interactive performances, collaborative jamming platforms, and interactive music boxes.

The format of the course will balance discussion, instruction and jamming. Topics will include exploring the spectrum of interactive music from playback to full-fledged instruments as well as relevant artistic questions such as “how much control should composers give to their players/listeners”. Students will be exposed to prior-art interactives in contemporary music, game pieces, and video game music.

The technical part of the course will focus on Javascript, Web Audio (an HTML5 specification for audio synthesis, processing and playback) and Tone.js. Javascript is well suited for the course both for its ease-of-use and its ability to create multiplayer, mobile, location-based and click-based applications, among many other possibilities. Additionally, the browser provides a built-in means of distributing musical works.

Making Sense of Wearables (Despina Papadopoulos)

After almost 2 decades of experimentation in wearable technologies we are seeing a renewed interest in the area. Both the apparel and technology industry are edging the market and yet, nothing seems to stick. Wearable environments are complex – their intimate relationship to the human body, their expressive potential, their ambivalent relationship to fashion, well-being and notions of selfhood, all bring into focus core questions of interaction design.

Making Sense of Wearables will review the current and past landscape of wearable environments, outline possible criteria of success and experiment with developing prototypes that account for their personal, social and material implications (and opportunities). The class will be structured around themes of personal (human senses, qualified self, well being, intimacy and communication) and social space (time and location, gestures, fashion and material culture): each week a new theme will be introduced, along with examples, historical evolution in the area, readings, and related technologies and their implications.

Students will be asked to respond to these themes with weekly assignments and fuse emerging technological possibilities with design considerations of the embodied human experience. Students will be encouraged to explore and expand their aptitude in physical computing and rapid prototyping techniques but deep experience in either is not a pre-requisite for the class.

Thesis (Kathy Wilson)

An ITP thesis is a demonstration of mastery of a particular project or problem of your design. It can take different forms—a physical prototype, a research paper—but the key is that there is a central concept, with a clear statement of purpose: in other words, 1) what it is, and 2) so what? Why are you interested in it? What need does it fulfill, what does it add to the world, a field, the literature? Your thesis does not have to be a finished product—that would be impossible in 14 weeks—but it should be a proof of concept. That is, you should show us how it could be achieved and we’d have confidence that you could. You will find a list of what is considered a proof-of-concept for different media on the blog, when it is up and running.

Unlike a dissertation at a more traditionally academic program, you do not have to defend your thesis to a committee. You do have to 'defend' it to your thesis class and teacher! They will help you hone your idea and get it in to shape. Many people think of their thesis as a platform—something from which you could write a grant, a business plan, a conference proposal or just show as part of your portfolio. It is the one thing that you do here at ITP that’s all yours: you generate the idea, you create the path towards implementation. The ITP thesis reflects our belief that you will leave here feeling comfortable as a creator, inventor, maker and thinker. And you will also be able to express your ideas to others in a variety of ways—in writing, in model, a prototype, in conversation, in a presentation.

Towers of Power (Benedetta Piantella)

In this two-credit course we will learn how the GSM network works and will explore cellphone technologies and how to get your projects communicating with a cell carrier. This course will mainly be structured in two parts, the first will be dedicated to learning about the current GSM infrastructure and how the communication between cell phones and the network happens; we will be introducing open source alternatives, both hardware and software, for setting up and deploying your own GSM network as a way to better understand the different components of the system. The second part will focus on getting a hands on experience with off the shelf GSM modules, modems and mobile hardware (possibly to build your own cellphone!) and on getting those devices communicating with cellphone networks via text message (SMS) and via data transfer (GPRS). We will have open workshop sessions for further developing your mini-project and presenting it to the rest of the class as well as guest speakers. Pcomp and basic programming skills are required.

First test of passing data from XBee to Arduino to Node.js to client


Today I successfully mapped data from three different XBee modules onto a web page hosted on a local server. This is a very big accomplishment for me since it is truly a proof of concept for the entire Sonome project.

Just a few days ago I finally hooked up the receiver XBee to the Arduino so that I could see the packets coming in through the serial monitor. Today I took that one step further, by incorporating that Arduino code into the du.ino code used to connect the Arduino to the Node.js server.

I don't have an XBee shield so I had to wire the pins myself. It's very easy though; as indicated in the code, Arduino digital pins 8 and 9 connect to the XBee DOUT and DIN pins, respectively.

AltSoftSerial allows you an additional serial port simulated through software, and is necessary since both the XBee and Arduino use serial communication. In my testing I found that the hardware serial baud rate must be significantly higher than the XBee rate in order to minimize mixed-up packets. I found this using only 3 XBees, but it will become even more critical when all 16 are blasting away simultaneously.

I then had to incorporate the code above into the du.ino stock code. There was some sort of conflict with the Servo library used by du.ino, so I removed all servo-related code; I don't need to control servos for this project.

As you can probably see from the code, I am printing an array of 16 bytes to the serial port - one byte for each sensor. This is what gets passed through to Node.js, and is how a client knows which value is which. The indices correspond to the transmitter XBee addresses so I can correctly map their locations.

I'm in very good shape now to finish the project on time. I need to connect the microphones to the XBees, and start testing ranges and levels in the real space.

Heatmap color schemes and XBee update

I have been spending a lot of time trying out different color schemes for the Sonome heatmap. Even if all of the sensors are working and successfully transmitting to the server, the visualization could easily be ruined by using the wrong palette.

I decided to focus on a single color because I think it will stand out beautifully on a minimal, sparse background. The final floor plan image will use even thinner lines than the one shown here so there will be plenty of negative space for everything to breathe.

Below are some possible options, the last being the stock heatmap colors for comparison (gradient from blue to green to red). I am favoring #1 but am curious about the grayscale versions because I think they are visually compelling as well.

In addition to the graphic design, I have pressed onward with creating an XBee network. Right now I'm borrowing 4 XBee modules and can successfully send packets from each transmitter to the master receiver. However, I am still hard at work figuring out the best way to send the data from the receiver XBee to the Arduino, and through to the web server.

The program XCTU makes programming the modules so much easier. As shown above, it even can visually display the network topography. This has proven helpful in testing multiple modules at once. I have researched further techniques for managing the barrage of data received by the single receiver. One technique is to stagger the transmit rate of each module by a few milliseconds in order to reduce overlap.

I configured the XBees using custom parameters, including the use of API mode, which allows them to send packets instead of simply serial data. This makes it possible to identify the source transmitter from the data alone. Also I am much more confident that I won't run into RF interference problems from other XBees because of the steps I've taken to isolate them in one part of the XBee wireless spectrum.

Trying out my own pyramid scheme

To explore an "equilibrium of an ecosystem in which players’ selfish behavior collapses the system," I decided on the classic pyramid scheme, in which a person's success depends on their recruiting of additional people, ad infinitum. The inherent problem is easily illustrated in the image below, which proves that a system of this nature is quite literally unsustainable.

I designed an ambiguous pen-and-paper "game" to see if I could embroil my colleagues in a similar scheme. I told two random people that the goal was to create a team-network and they needed to each recruit two people, and tell them the exact same thing I told them. They would fill out the small papers I provided and continue until the paper ran out. Each recruit was to return their papers to the person who recruited them, in effect creating a depth-first search, since I would not get any papers back until all of the people below me had submitted them to their higher-ups.

I also mentioned that their score would be determined by the number of people they managed to recruit underneath them, so that they should only choose people who they thought were likely to carry out the instructions. The lack of real incentive didn't seem to hinder people's eagerness to participate, but then again, this is ITP.

After my own recruiting, I receded into the shadows to watch the chaos unfold. It was hilarious to watch as people starting asking questions about who the higher-ups were.
"Who do I give this back to?"
"Who do you give it back to?"
"The person who gave it to me."
"You mean that guy that was standing over here?"
It only had taken me five minutes to become an anonymous figure of lore, just like the criminals who I read about while researching the basis of this experiment. I'm sure these same questions were asked during the downfall of Bernie Madoff's investment scandal.

The game also evolved on the fly, which is not something I had anticipated. Due to the half-baked nature of the rules I explained, it wasn't clear to the participants what to do if they only had one sheet of paper. I heard at least one person suggest that they should make their own paper in order to keep it going, which was not my original intention, as I wanted to minimize the amount of effort required by each individual participant. However, I did not intervene, as the mark of a good pyramid manager is to stay laissez-faire all the way to the end.

Unfortunately, I did not get back any of the papers yet so I cannot say with any certainty what happened. It appears though that there was not enough of an incentive to encourage their timely return. I will update this when all of the papers are returned to me.

Update on Sonome

A couple weeks ago I decided that the original scope of my project was too ambitious. Creating a wireless sensor module that could exist outdoors is a fairly daunting task (and solar-powered? what was I thinking...), and when combined with the front-end map interface it is entirely too much for one semester.

However, an indoor version removes a lot of constraints. The modules don't necessarily have to be wireless, power is readily available, there are plenty of mounting options, and the physical area covered by the sensor array is much smaller.

Therefore, Sonome ITP is the new name of the project. I still want Sonome to exist as it's own concept so that's why ITP is secondary in the name (small distinction but I think it's worth mentioning as I think about the future of what an urban Sonome would look like).

Sonome ITP will be a visualization of noise on the 4th Floor of 721 Broadway, premiering at the 2014 ITP Winter Show (pending acceptance of course). Visitors will see a display in the ITP lobby that will look something like this rough version below (using random data):

I may experiment with a Google Maps overlay but this depends on if and when they install the indoor map which I submitted to them two weeks ago. The idea would be to place the Sonome visualization on top of an integrated Google Maps floor plan so that mobile users could see their current location within the space. Unfortunately as I said before this feature may not come to fruition, and so my focus for now is on this simplified image version of the ITP floor.

The sensor is coming along, too. I think now that "module" is a bit too strong of a word, since what may happen is that several microphones will be connected to one transmitter instead of one microphone per transmitter. Wireless is now optional since I have the space and infrastructure to run wires, but Despina brought up a valid point that a wireless solution may still be the smartest choice, based on the description of the microphone wiring so far.

I bought a MEMS microphone breakout and it works really well. It is incredibly sensitive so I have been trying to come up with various ways of normalizing the values coming off of it. I have also successfully connected an Arduino to a Node.js server (via USB) using duino and am able to see the microphone live on the noise map, as well as access the map on other devices on the same network.

Some major things that need to be done:

  • Decide on the exact physical system layout and connectivity
  • Normalize the noise data
  • Set up a public server (instead of a local server)
More to come soon.

Two color map + current event map

Two color map

Being able to use only two colors on a map might at first seems like a limitation. But it makes things much less cluttered and easier to understand, if you choose the right data to visualize.

This is a map of every building in Manhattan. Each blue bubble is a building, and the size of the bubble is determined by how many floors that building has. The background of the map is white, and the building bubbles are blue. The attribute that gives a meaning to the map layout is the bubble size.

Notice how easy it is to pick out the areas where skyscrapers exist. Better yet, zoom in a bit and the city begins to deconstruct. The space between buildings increases and you can see the relative footprints of a high-rise or a walk-up.

Current event map

I had been interested in the number of Americans who joined/tried to join ISIS after reading a couple articles about them. When we read about historical rebel groups, we tend to ignore the fact that that kind of thing can happen in our present day. At least I do. This was an attempt to humanize some of these people.

Using Odyssey.js I constructed a simple list of people and the places of their origin, along with short blurbs taken from the original article (dutifully cited in the first slide). It shows that potential jihadists can be found all over the country, and some are even that guy who "was a goofball in high school."

Foosball front-end

Link: Foosion

Partially for class, partially for fun, I created the front-end framework for some of my Cloudcommuting colleagues' foosball-centric RFID projects. The systems are not connected as of yet but by the end of the semester we should have a fully-functional identification system that updates the front-end in real-time.

There are several data points that we are recording:

  • Date of match
  • Winning player names
  • Winning team score
  • Losing player names
  • Losing team score

In future iterations we will be recording the goals scored by each player, as well as the time of the goals with respect to the start time of the game.

For this project I used AngularJS as the framework. Eventually it will be integrated with Node.js and MongoDB so that the front-end and the Yún(s) can communicate with the server and with each other. With this setup it will be feasible to display a foosball game live on the internet (similar to ESPN's Gamecast).

Since seeing the web site is helpful to understanding how the system works, please check it out at the link at the top of this post.