Jeremy Stewart, MASARY Technical Director & Caleb Hawkins, MASARY Design Director

Memory / Diffusion, a monumental artwork permanently installed in 2022 at Boston Arts Academy, is a pivotal work in MASARY’s portfolio. The artwork spans two floors, and is situated at the main stairwell of the school. It features a two-story LED wall and L-shaped lighting elements that extends into the open stairwell space.

*Memory / Diffusion - Photo by Aram Bohgosian*

As we approach its 3-year anniversary in early Fall 2025, MASARY’s Technical Director, Jeremy Stewart and Design Director, Caleb Hawkins reflect on the development of Memory / Diffusion:

Considering memory plays such a big part of this project, do you have any strong memories from (high)-school that you believe influenced the way that you worked on this artwork?

CH: Given the site being at the Boston Arts Academy, one of the first things that kind of came up was recalling memories of our own high school experiences. Memories from when I was a kid. I think empathizing with a younger childlike-self to imagine and create something exciting was one of the first things my mind went to. I tried to put myself in the shoes of a student who might be going to the school at the Boston Arts Academy. That fueled my excitement, you know, to imagine creating an artwork for my younger self in a way, out of hopes to inspire the students at the school.

DRJ: The strongest memories I have of high school, as it relates to this project, are the pieces of high school that kept me engaged and interested and even served as an escape from the parts of high school that I disliked. Really the parts of high school that filled that need for me were being in band and playing my instrument, playing saxophone, and having band rehearsal and private lessons and finding moments to practice when I probably should have been in class. My band director was looking out for me, and if there was a class and on a particular day nothing was happening in that class or, I don't know, I just didn't want to be in that class, she would write me a note that I was needed to help with band related things, and then I would sit in her office and practice for an hour. Music and the arts, then, served as an escape or as a positive thing for me to focus my energy on and focus my efforts on and made, you know, some of the more grinding classes more survivable.

That kind of place of music and the arts early on in my history just made me excited for this project: being in an arts school where so much of what goes in this building is related to that arts. And I can really appreciate how the arts can be a positive focal point for energy and effort for these students.

How did Henri Bergson’s theory on representation-memory shape the conceptual framework of Memory / Diffusion?

DRJ: Representation memory, for Bergson, was really like being able to place yourself back in a moment where you had an experience or saw something or heard or smelled something, being able to have an awareness of where in your own personal history that memory comes from, or the kind of connected adjacent events and conditions and circumstances for that memory are all kind of a part of it. These are memories that you can actively recall through effort or will or that naturally come to us through our experience of the world. And so Memory / Diffusion is really about capturing these little representations of the setting of the artwork and then being able to recall them and being able to transform them or look at them in different ways.

In what ways does the installation embody Bergson's idea that memories can be re-experienced in nonlinear ways, free from the constraints of time?

DRJ: Bergson's ideas of memories are that our continued existence in the world and traveling through the world results in a constant recall of relevant memories from our personal history. And that is one of the primary ways that we understand the world and that we make sense of our own place and our own set of possibilities within it.

As an example, he would say when you see an artwork that you've seen before, you don't just see the artwork, but you're also experiencing your previous experience of it, that previous experience or that memory is a part of your contemporary experience of it. And that previous experience would've been colored and shaded by everything that was going on during that moment, when that memory was created: your own emotional disposition, what happened throughout the rest of that day or leading up to that experience, who you were with, the weather outside, what you were wearing, who else was there in the room with you, was it crowded, was it quiet, was it empty. All of those things would subtly impact that memory of this artwork. And so now if you're re-experiencing it, at a later date, say a year later. Naturally, your consciousness will recall that previous experience and may bring forth particular elements of that previous experience to help you understand and make sense of the current moment.

*A student walks past Memory / Diffusion on the upper floor of Boston Arts Academy.*

Could you elaborate on how the concepts of memory "translation" and "rotation," as discussed by Gilles Deleuze in relation to Bergson's work, are manifested within the artwork?

DRJ: So in Deleuze's interpretation or explanation of Bergson and representation memory, he uses these two terms, "translation" and "rotation". And by translation, he simply means that, again, to use the artwork story or metaphor, if I'm standing in front of an artwork that I had previously seen a year ago, then that memory of a year ago, that experience of a year ago will be translated to the present, by the fact that I'm experiencing the artwork again, standing in the same place or in front of the same artwork. That memory is translated to be brought forth to the present moment where I can kind of examine this memory again, or experience parts of it again.

But then Deleuze also uses the word "rotation" to refer to the way that that memory comes forth, but perhaps certain parts of that memory are stronger or amplified and other parts are attenuated based on my current experience. And so the memory itself kind of changes or our perspective of it changes based on current conditions.

These are manifested through the artwork by the fact that when somebody stands in front of the artwork, the cameras in the artwork are going to see that person and are going to look in its memory bank - in its database - for what previously stored memories are most similar to this current moment, to what it's seeing now. And it will recall those memories that are most similar to it and recall, or what Deleuze would call translation. But then the memory is brought forth, but it's also processed or transformed in a way that it may bring forth certain elements or aspects of that previous memory, but may occlude or hide or obscure other aspects of that recalled memory. That's what Deleuze would refer to as rotation.

Obviously we're doing it kind of in a very abstract way or taking artistic liberties with this concept here in the artwork, but that's sort of the underlying idea or motivation for this connection to memory.

Bergson posits that representation-memories are recalled in non-linear ways. How have you shaped the installation’s behavior or aesthetics to mirror this temporal fluidity?

DRJ: Once memories are stored within the artwork, they have no concept of order, and they're not recalled with any concept of order that they were formed.

And so when the artwork recalls a memory as a product of something occurring in front of it, in its view, it will select from all the available memories that it has previously stored. And this could result in one memory being recalled at moment one and that memory recalled was only created or captured or formed an hour ago. But then the next memory that is recalled might be a memory that was formed three years ago. So there is no linearity necessarily to the way that the artwork recalls memory.

The artwork recalls the memories as literal videos, but then it transforms it through an image processing pipeline that serves as the metaphorical "rotation" of the memory, to use Deleuze's term. The output of the pipeline is what viewers see on the screen.

How does the system determine which moments are significant enough to be stored as "memories," and what parameters influence this selection?

DRJ: Essentially, the cameras are constantly feeding their images through a VGG net for feature extraction.

And then that feature vector that comes out of that is compared against all the feature vectors that have previously been computed through a little bit of a, there's a little sidestep in there and that is that the clustering algorithm performs some dimensionality reduction where we can put in the raw feature vector that's from the VGG net through the current clustering model, and it'll output a smaller vector that better describes the comparison of that feature set against the current clustering model of all of the previous memories.

Then we look at how far away the current feature vector is from previous memories, and if it's further than some threshold value, which is a dynamically changing value, that changes over time and changes as a product of the number of times that the cluster model is rebuilt. But if the current feature vector is further away than some threshold value from the nearest similar member of the clustering model, then it is identified as being quite different from anything else, and therefore we want to save it. So then we save it and we re-build the cluster model resulting in a different organization of all of those memories and continue on.

Given the AI’s role in curating memories, do you consider the system itself a creative collaborator? How much “authorship” do you grant the machine versus yourselves?

DRJ: Uh, none. It's not a creative collaborator, and I don't grant it any authorship. It's a pipeline that is using a very clear cut set of algorithms to perform functions.

The fact that there's some pseudo randomness built into it is by design and is controlled through those algorithms. But that pseudo randomness actually is some of the space where the creative, the visual, the artistic license comes into play and the variance that exists within the artwork is allowed to come through.

How do you see the role of technology, particularly AI and machine learning, in expanding the boundaries of interactive art installations?

DRJ: Yeah, this is something I've talked about previously quite a bit with my colleagues and friends. I think the most interesting applications of AI and machine learning are probably going to be the least obvious ones.

Certainly I'm interested in, um, the kind of big image generating models and video generators and chat GPT and stable diffusion and all that. That's all interesting to some degree, but it hardly qualifies as creative or artistic in my mind because it's a machine carrying out an algorithm and then the machine has no agency, has no conscious agency. So those all just feel like tools to me.

The areas that are most interesting are when, like in Memory / Diffusion, it actually uses quite a bit of AI under the hood, but the artwork itself isn't about AI. It's making clever use of tools and algorithms in non-obvious ways to create an artwork.

And it's okay if somebody observing the artwork has no idea that there's any machine learning in it, because the artwork is not really about that. That's some of the most interesting ways or it's certainly the most interesting way that I think about machine learning in art, on a regular basis.

And that's more in the direction of "what possibilities do these algorithms and tools open up that we couldn't have done before with other methods." And in this instance, it's, you know, rapidly doing abstract feature extraction from images and video and clustering those features against many, many, many hundreds and hundreds and hundreds of other feature vectors. Of course, parts of that could have been done very simply, without machine learning, but other parts would've been impossible to do without machine learning.

*MASARY Design Director, Caleb Hawkins and Technical Director, Jeremy Stewart*

How did the environment and community of the Boston Arts Academy influence the design and functionality of Memory / Diffusion?

CH: When thinking about the environment for the project, I kinda zoomed out. You know, it's not just the inside of the school or the school itself. Part of it is recognizing a more urban area in the context of which the school's located.

The area around Ipswich street is quite exciting. It's right near Fenway Park, near Lansdowne Street and some other great venues that are located right nearby for shows and evening nights out in Boston. But these are just some of the conditions around the immediate environment of the school. From the Boston Arts Academy community itself, we learned about their past school, past locations, and what the classes they offer the students, the learning experience, school values. The school past and the experiences of the faculty and students were critical influences.

What inspired the physical form and material choices of the artwork? Was there a particular visual metaphor or design philosophy guiding the design?

CH: The commission was for a media wall and at the time, there was a real kind of question around what a media wall is or how people think of media walls, or if a media wall could be something more than what you typically expect. Some of it became a bit of a critique towards thinking that media walls were really just screens. You know, “media” encompasses many different things outside of just what could be seen on a rectangular screen.

So, we created a design challenge for ourselves - how can we break out of this presumed format for digital media? How do we get it away from just a rectangle that's so familiar and so normal in many different scenarios and scenes from billboards to phones, to TVs, and expand it or grow it or push it into something more? Spatial or formal…these philosophies between a 2D into a 3D or from a virtual into a physical space became core questions for how this artwork manifested. We were attempting to make something that was typically seen as 2D into something that was three-dimensional and spatial, and potentially physical. Something that could be touched, felt, more so with the body than just with the eyes.

*Elevation drawing of Memory / Diffusion*

Did the design change in response to site conditions (light levels, structural support, viewing angles)? How did the physical context shape the final outcome?

CH: There was also kind of the space of envisionment for how big and how long and how much space the artwork could actually take up. We were able to look at some early views and perspectives of what the interior experience was gonna be like before the building was completed. We could start to envision where people might stand, how they’d enter the space from various areas, how the artwork might be viewed from both the ground and second floors, how someone might be able to catch a peak between two stairs. We were interested too to see how the building architecture might frame particular views of the artwork that we could kind of lean into.

There were also some views that became impactful into the design, mainly around certain constraints and blocking light. As much as we wanted the light to illuminate the sculpture, there was still some need to block harsh light coming off the artwork from visitors walking by on the second floor, for instance.

The installation incorporates camera feed, LED screen, and staircase lighting. How did you design the spatial choreography so the architecture itself becomes a part of the memory flow?

CH: When starting this project, one of the things that we typically go through is a form of site analysis. In addition to recognizing the environment that the school was in, from an urban scale to some of the community themes and values of the students and teachers, there's the other kind of architectural, more immediate setting to where the project is situated.

Looking at the school, there were a couple main elements that stood out - the stage at the bottom of the bleacher stair, the stair itself and the general functioning of the entryway of the school as they had programmed this staircase / stage area adjacent to some other open programming in the school. Other architectural principles were identified in this space - one being the axis of the stairwell running along the large concrete wall that was intended to be the media wall. These areas of gathering where people might sit, stay, or produce a performance or other activity, these areas of circulation became more of the major part. But there are additional themes of repetition as well as certain axes and datums within the staircase, railings and other architectural features that became a reference to some of the design of the artwork.

How did you choose the materials for the structural elements, diffusion surfaces, and lighting enclosures?

CH: The choice for materials was a mix between recognizing some of the architectural inside and outside the building, as well as some desirable kinds of aesthetics - thinking about the combination of all three. The inside, the outside, and the ideal.

In and outside the building, we really recognized the glass, some of the lined lighting in the bleacher stairs, the raw materials like exposed concrete and stainless steel - these became really attractive in contrast to the wood features. The material choice was a means of situating the artwork into the context of the architecture immediately in the space that the artwork would be in. The materials also help situate it somewhat conceptually, as it references some of the rain screens and the facade and incorporates them in a way for diffusing light in the artwork. Some of the lighting fixtures themselves include forms of plastics that are able to spread and kind of hold light when illuminated. The idea was that some of these fixtures would be able to kind of allow some of the emitted light to become more volumetric - holding a space within this fixture. And we had worked on thinking about some raw materials and additional films that might also be able to provide this effect of diffusing light within one of the fixtures.

So it feels like there's a combination between materials that are of the site that were referenced conceptually or kind of like poetically into how they met the artwork. And then another area of materiality became more practical in regards to how it was playing with and affecting the light that was inside of the fixtures.

*Material test and discussions at DCL Boston headquarters.*

How did you approach integrating lighting with the form of the artwork, via the lighting enclosures?

CH: When designing the lights, there was this kind of core idea of it being based off of an L structure, kind of like a spine or a frame that was an L with an extrusion or kind of like a sandwiched surface between two pains that were transparent, translucent or varying opacity. The idea was there, but we needed to test. So during the project, we had worked with our fabricator to kind of prototype and test a couple different assemblies, which looked at this relationship between the spine and the sandwiched materials that kind of worked with the light. We then looked at varieties of distances and dimensions, and which could potentially give the best effect into seeing most of the fixture illuminated in color. We explored different plastics, but realized that there was a use of some diffusive films that were able to spread and hold more of the illuminance from the LEDs that were chosen in the form of the fixture itself.

How did you decide on placement for cameras and sensors?

DRJ: We decided that two cameras was interesting and would allow for much more area to be observed. And then from there we just said, "well, what, what are possible locations?". And pretty quickly we identified the one attached to the screen of the artwork, kind of facing at shoulder height, was very interesting just because it was a very direct view. Like if a person is standing in front of the artwork looking at the screen, then the artwork is looking right back at them. That's a pretty interesting perspective.

And then the other one, the other camera is at the top of the stairwell at the very last lighting element, installed at the base of it, looking down the stairwell. That stairwell is both a stairwell and kind of like stadium style seating. And then right on the floor, right in front of the artwork is a stage area, and beyond the stage is like the cafeteria area where a lot of activity happens. So having this camera at the top of the stairwell looking down, there's always students sitting along the stairwell, students eating lunch there, there's always stuff happening on the stage that's interesting. And then off into the like cafeteria where there's like more of a throughway of students moving and gathering.

You implemented a convolutional neural network as a deep feature extractor, layering clustering & dimensionality reduction. Could you walk through this pipeline—what specific models/pipelines you used and why?

DRJ: We used a VGG net, which is one of the long running image classification models that had great successes early on and outperformed many other convolutional neural networks in the task of image classification. It was a pre-trained model, so it means that we don't necessarily need to do any training or really any fine tuning.

In the original architecture for the model, you push your image through the front and then the output of the model, it would simply identify which of a number of classes that image would fall into. And those classes would relate to the image content, so one class might be a person or a car, or a chair or a dog or a bicycle or something like this. And the output from the model would be a score for each of those classes. And then you would go down the list and you would pick which class was scored the highest and you would make the assumption that that object was probably appearing in the image.

So in our case, what we did is we chopped the model in half. The majority of the model is tasked with feature extraction and then the last couple layers of the VGG model actually makes use of all of those features and turns it into classification output. So we just sliced off the classifier at the end so that what it spits out is just a series of completely abstract values that relate to the feature content of the image.

Then we treated those as, uh, vectors that, again, incredibly abstractly and non-linearly described the content of the image. We used those large vectors - I believe they were 4,000 value vectors - we used those with a clustering algorithm. So we treated those vectors as the identifiers for a given memory and each one would be more or less unique.

We would compare that very large feature vector against a previously stored database of similar feature vectors and look for the most similar ones using a K dimensional tree search. That's how we would pick which memories to recall, which ones were most similar or least similar from this current occurrence. And this approach also is how we decide when to store a new memory.

If the memory gets pushed through the convolutional neural network and we get that feature vector and we do search against the database of previous feature vectors and we find that, well, the closest one is actually pretty far away, then we say, "okay, well this feature vector, this new feature vector is different enough from everything that we've seen before, that we're going to call this occurrence that just happened - this memory or this activity that just happened that caused this feature vector - we're gonna treat that as a novel memory, something worth remembering," and we're going to store it. And then once that memory is stored, that means both the video clip of the actual activity in front of the artwork is stored in a file system, but also this deep feature vector is stored in the database along with all the other ones, along with some other metadata about it.

Once this happens the artwork uses a clustering algorithm to re-tinker all of the clustering for the artwork. So it'll just recompute everything and kind of recompute its own dimensionality reduction for the look up. What that means is that the artwork is constantly reconfiguring the way that it maps its own memories as it's gaining new memories. So that can result in some pretty interesting things like, you know, two previously distantly related memories might pinch a little closer together or something like that, or large clusters of similar memories may dissipate or stretch or incorporate other memories as time goes on because there's so much variation in the total set.

What has been the response from students interacting with the installation, and how do you envision it impacting their perception of memory and art?

CH: Shortly after artwork and building were completed and students were finally able to get into the space, our team was able to visit the school and see the students in real time around the artwork during the day, and it seemed like there was a lot of excitement for both the new building and artwork. Even after the time that has passed, students still hang out on the bleachers in front of the artwork during lunch, in between class periods, etc., so it continues to be a frequently -visited gathering place. And I'm also recalling seeing students use the stage in front of the artwork - there's lots of formal performances as part of classes, but the stage in front of the artwork seemed a little bit more informal with students really engaging in TikTok and Instagram dances.

Just to see it in use, my ideal vision and hope is that every student has or will be able to develop some sort of relationship with the artwork, whether it be everyday in casual passing or a significant memory that they might be able to hold with them once they're no longer a student at the school. That these core memories of experience in the school and with the artwork could influence or inspire future creatives - I think is definitely a hope.

Given that the piece is designed to evolve over time, how do you anticipate it will change in response to the dynamic activities within the school?

CH: It's hard to predict. I think there's a split between behaviors and content. I think that given the parameters of the artwork, the behaviors might stay and remain the same, but the content, my hope at least, is that it will change and evolve significantly based on the memories that it gathers. The excitement around the exchange of memories is definitely something I'm looking forward to as the artwork witnesses the programming at the school and types of events, performances, and happenings that might be going on in the entryway on the stage or in the stairs.

I hope that some of these memories might be more impactful or more novel and remain as a part of the artwork – it'd be great to see them embedded in the artwork over time.

Remembering Memory / Diffusion as we near its 3-year anniversary

Evolutions of Spatial Audio