Avatar is not a simple duplication of the body, but an extension of mind. The physical body is reconstructed by data. However, once we are in the space behind the screen, what kind of shape should we be transformed into, and how should we perceive the world on the screen? What form the cyber body should take? Michael Heim, the philosopher of VR, once raised such doubts.¹

The representation of avatars can vary from a great range, which may or may not even resemble a person’s body shape. We can be Mario, Pac-Man, we can be any realistic figure or cartoon image, we can have any figurative or abstract appearance. Through gestures, movements, and appearance, people feel more accessible to express themselves. Rich expressions trigger a richer emotional cognition, from which people get a whole new resonance.

In our project, we mainly consider the avatar design from the perspective of abstractness and dimensions.

Nowak and Biocca’s experiment² in 2003 showed that the higher the degree of anthropomorphism would trigger higher expectations, resulting in a reduction in resonance when these expectations were not met. Therefore participants who interacted with fewer anthropomorphic images would gain a greater sense of coexistence. In addition, according to Lugrin, Latt, and Latoschik (2015)³, the realistic avatars could make the participants pay too much attention to the details of the virtual body, which would lower the degree of Illusion of Virtual Body Ownership (IVBO).

In summary, when designing avatars, it should be the best choice to use the avatar which is similar to the overall outline of the human body but not too simulative.

2D avatar, which is more likely to be regarded as the “mirrored self”, is the projection of the physical body in a 2D dimension. In the project Me and My Shadow⁴, people from different regions are represented by abstract 2D avatars of different colors, gathering somewhere in the virtual space. Here, people express themselves through the avatar’s gestures, understand each other, and form a vibrant virtual community. In the 2D perspective which is less figurative than 3D, people are more accessible to capture the general shape of avatars and easier to focus on the information transferred by movements.

3D avatar is mainly used in VR, AR, and holographic projection. Compared with the more abstract emotional feelings brought by a 2D avatar, 3D avatar works in a more realistic way. In the Microsoft project Holoportation⁵, through 3D capture and stereo imaging, high-quality 3D human models can be reconstructed, compressed, and transmitted anywhere in the world in real-time. When combined with mixed reality technologies such as HoloLens, users can see and interact with 3D remote participants as if they were actually in the same physical space.

Based on the above theory and case studies, in Project Dysphasia, we mainly developed the following three stages of avatar image changes.

The very first one is the most basic skeleton, composed of dot and lines, clearly expressing the structure of the joints and limbs of the human body. The background is pure black, which mainly highlights the human body language.

In the second stage, we decided to use IP Camera videos from all over the world as research objects, and the focus shifted from body language to the connectedness between people. Therefore, the second version of the avatar design mainly increased the density of the lines while connected the bodies together.

In the third stage, which is the current stage, the design concept is further improved. In front of the screen, we gradually change from a pure observer to a participant. Using PoseNet to analyze the human body images captured by the webcam, a virtual avatar that changes in real-time with the participants’ movement could be generated, thus be projected to live streaming video on the other side of the world. Taking Shibuya crossroads as an example, users at miles away can “appear” on the crossing in Japan as soon as they open their webcams. Therefore, we chose to highlight the ethereal virtual feeling formed between the conversion of the physical distance and the network distance. With the “ghost-liked” image, we are able to wander on the streets of another city as the digital selves.

In the next stage, we are trying to focus more on creating a response from the other side of the screen and establish an interactive on-site system. Physical installation is expected to be present at the virtually-joined site. The sound effects of the web-users’ interactions with the bubble can be heard there, and when a bubble is smashed virtually, the installation will blow a real one. In this way, those who show up at the place, in reality, are also involved in the interaction.

Reference

[1] Heim, M. 1998. Virtual Realism. New York, NY: Oxford University Press
[2] Kristine L Nowak and Frank Biocca. 2003. The Effect of the Agency and Anthropomorphism on Users’ Sense of Telepresence, Copresence, and Social Presence in Virtual Environments. Presence Teleoperators & Virtual Environments 12(5):481-494
[3] Jean-Luc Lugrin, Johanna Latt and Marc Erich Latoschik. 2015. Avatar anthropomorphism and illusion of body ownership in VR. IEEE VR 2015.
[4] East London collective body>data>space and National Theatre in London. 2012. Me and My Shadow from
[5] Microsoft. 2016. Holoportation

List of Figures

Feature Image: Different Forms of Avatar Design. Eliane Schlemmer, Daiana Trein and Cristoffer Oliveira. 2009. The Metaverse: Telepresence in 3D Avatar-Driven Digital-Virtual Worlds. Deposito legal: V-5051- 2008. ISSN: 1989 – 3477.
Figure 1: The high anthropomorphism and low anthropomorphism avatar used in the experiment by Nowak and Biocca, 2003
Figure 2: The different forms of avatars (from realistic to abstract), and the IVBO effect graph by Lugrin, Latt, and Latoschik, 2015
Figure 3: Me and My Shadow, 2012
Figure 4: Holoportation by Microsoft, 2016
Figure 5: The First Stage – Simple Skeleton
Figure 6: The Second Stage – Connectedness of People
Figure 7: The Third Stage – Wandering in Shibuya
Figure 8: Future Plan – On-site Installation

Results were so simple and I thought we need more complex geometries. As part of the Web-GL class in semester one, we were introduced to 2D and 3D noises and created lots of shapes with them. So started working on 2D noise in grasshopper (4D noise add-on) and created a simple vector field by adding noise to it:

And then created the 4D noise and added it to the sphere. By adding colour to it, We are able to see what part of the surface is convex (Positive) or concave (Negative) so we can remap them if we need in order to prevent petal deformation later in planarization part.

And then started baking them in the Rh viewport so later we can choose which one is appropriate for our final hive:

And even more variations for better understanding of 4D noise behaviour:

One point is that each sequence saves its data with it (e.g. random numbers) so later if we decide to do reverse engineering and recreate the surface, we will not have any problem.

Now it is the time to tesselate couple of surfaces to see what method of planarization which we discussed in the last blog post is appropriate.

At the end of term two, we confronted the COV ID 19 crisis, our access in workshop and building facilities was restricted, and our everyday lives were distracted. All these lead both of us to search for ways to work remotely, collaborate, be inspired, and motivated. We are still interested in creating a performance in space, that would connect new technologies with the whole human body in a cross-sensory way.

In the past few months, we were mainly focused on theoretical research and readings that will help us support our ideas and use our design project as a means of interpreting theoretical explorations subjectively. The common ground in our research interests is the various ways that new technologies and especially Virtual Reality can offer multiple ways of changing and re-defining the human experience in performances. We are interested in the complementary relationship between Body and VR, mainly because the correlation between these mediums enhance approaches of investigating VR not as an alienating technological tool but as a conduit of expanding embodied experiences. What do we lose when we are wearing Vr headset, how we can use it not to replace human senses but to expand them?

Left: Ocean of Air – Marshmallow Laser Feast, 2018 Right: Cosmos within Us- Satore Studio, 2019

Ideas around Narrative

“We structure our lives by narratives and we understand events in the world in terms of narratives — events of all kinds, in science, in politics, in every facet of life.” (George Lakoff, Srini Narayanan, 2010).

Narratives and stories always have been fundamental means, in the way that the human brain understands and constructs the world that surrounds him. In the context of technological progress and constant innovation that we are experiencing, the role of narration is re-defined and altered. For instance, in technical industries, especially in game development, new ways of creating immersive storytelling experiences based on Virtual reality technology are being explored. As well as in the direction of Arts and Performance, mainly, in ways of using this technology in designing immersive, participatory, or interactive narratives in performative spaces.

“Some of these narratives are conscious, others are very much unconscious. In life-narratives, each of us is the protagonist, living out the narratives as best we can.” (George Lakoff, Srini Narayanan, 2010).

We would like to think of designers as potential storytellers. They design these narratives for multiple protagonists; themselves and others. Designers are always trying to connect worlds, between this conscious and unconscious sphere, between human imagination and realization in physical space. Now Virtual reality tools, and “technical media” (Elleström, 2010) have challenged the very conception of narration and performance, by giving the chance to design in multiple realities and latent spaces (Mixed Reality, Virtual Reality, Physical Space, Digital Space, Data Space, etc).

Driven by theories related to the process of constructing and interpreting narratives (communication theories, psychology), the main goal of my research is to investigate ways that new technologies, especially VR, as mediums, can construct or break barriers in audience engagement with real-time stories. Through analyzing design examples, I would like to understand the relations between media: transmediality, intermediality, and their contribution to the narration.

Left: Thinking Things – George Aperghis, 2018. Right: The Encounter – Symon McBurney, 2016

“if all media were fundamentally different, it would be hard to find any interrelations at all; if they were fundamentally similar, it would be hard to find something that is not already interrelated. Media, however, are both different and similar, and intermediality must be understood as a bridge between medial differences that is founded on medial similarities (Ellestrom 2010, 12).”

Following this thought, I believe that the findings from analyzing ways of transmedial narration across various media in performances (see photos) will be helpful for making further thoughts about the use of VR, integrating them into our design proposal, and creating our own virtual narrative. I would like to focus on the mediums of VR and Body (Mine or the Other), especially in hybrid performances that relate them, in cross-sensory modes. Also, due to the peculiarities both of VR and the human body, the transformation from one medium to another offers different effects in the audience’s participation and various levels of immersion or engagement with physical space.

What are the disadvantages or the advantages of experiencing stories in multiple realities, during the processes of transforming from one medium to another, concerning immersion or sensory engagement with physical space? Are there ways to narrate across mediums, and across multiple bodies?

Ideas around materiality, data and experience

Moving away from the 20^th century thinking of material that serves as a representational tool. We moved towards the object, material processes and expressions. We understood the capacity of matter to self-organize and play an active role in its own formation. These theories also affected the art scene. Robert Morris as part of the Process art movement manifested an equally specific relationship between action, site, and material. Eva Hesse creates her own materials in order to express her forms and ideas. In her list of materials, she identified the human body as one of them. Janine Antoni with her piece called Gnaw, transformed an everyday activity (bitting) to a tremendous repeated gesture. In her piece what the viewer encounters is not the action itself but a form that carries evidence of her bodily presence. All these artworks introduce the concept of a material that is inseparable from the body’s actions that form it. In other words, the material has embedded information about the process of its own formation, it’s an augmentation of matter.

Left:Eva Hesse- Anti -Illusion: Procedures/Materials, Whitney Museum NYC 1979. Right top: Robet Morris- Anti -Illusion: Procedures/Materials, Whitney Museum NYC 1979.Right bottom: Gnaw-Janine Antoni, 1992

We are interested in the augmentation of matter in the physical and digital world. In the digital world, the raw matter is substituted by raw data.

According to Floridi “We are not stand-alone and unique entities, but rather informationally embodied organisms (inforgs), mutually connected and embedded in an informational environment, the infosphere, which we share with both natural and artificial agents similar to us in many respects”(Floridi, 2012).

Massumi has also described the infosphere as a homogeneous space of aimless, instantaneous delivery of everything in all directions simultaneously. The human body risks drowning in the unformed sea of information and it must, order and organize, it must form the formless, but to do so, it must shield itself from immersion in it(Massumi, 2005) Right now, we are so overwhelmed of all the raw data and we are unable to actually understand them. The most common way to process data is to visualize them in two dimensions, however, in literature, it is suggested that we should actually try to physicalize them in order to get a better understanding of them.

As Kevin Kelly suggests: “Right now all we can do is see the data, the charts, the curves – but in the long term, we want to able to feel them”. We want to further explore how can we embody data and translate them into an immersive multisensory experience using VR as a main tool.

Future Day Spa- Lucy McRae,2015

Design Scenarios – Possible Worlds

We want to further understand how we can explore the space through our tactile sense. The sense of touch is not limited only in the way we touch something, external from our bodies, but what we feel when we are being touched. How our emotional state changes when are touched by other people, especially now that this relates to concerns of the global pandemic? Under this spatial confinement that we have encountered, how can we reinterpret the space and reconnect with our bodies?

We would like to create a performance that will involve the whole body, of multiple participants and different mediums. We propose a scenographic space, with at least two participants: Participant A that explores the space, and Participant B with Vr Headset. We would like to encourage Participant A to explore the space through engaging (touching with the whole skin) the space that will surround him. We would like to understand his emotional state (collect data from biosensors) and map his experience into the Other participant. Participant B is experiencing an abstract virtual world in the VR headset, the emotional world of the participant A.

References:

Elleström, Lars, “Media borders, multimodality and intermediality“. Springer, 2010.

Floridi, L., 2012. “Turing’s three philosophical lessons and the philosophy of information.” Philosophical Transactions of the Royal Society A, 370(1971), pp.3536–3542.

Massumi, B., 1996. “Interface and Active Space: Human–Machine Design.” In Proceedings of the Sixth International Symposium on Electronic Art, pp. 188–92. Montreal: ISEA

Lakoff, George, and Srini Narayanan. “Toward a computational model of narrative.” 2010 AAAI Fall Symposium Series. 2010.

Overview:

1. Artistic directing role in collaboration with Sun who takes up the technical directing role
2. Concept developing
3. C4D and Blender (GLTF files)

Artistic directing role in collaboration with Sun who takes up the technical directing role

After project faire we realised that since we are in two different timezone and each have our strength, it is more efficient to have clearer roles between us.

We decided that Sun Yu will be taking the role of a technical director and I will be in charge of the artistic development. We meet weekly to discuss our design and share progress.

My role includes,

1. developing the artistic style – gather reference and inspirations, colour scheme, font etc.
2. realising these objects in 3D modeling software to be imput onto website that Sun is developing.
3. learn basic html to insert items and have understanding of how they work in principle

Concept developing

Our virtual world consists of three parts:

• Reality
• the self
• incentive to post

Reality

Our world is a place where reality is made of both the virtual and the actual. So we are creating a virtual environment that represents our world and focuses on the fact that our media is contructed with these mixed mediums. It is vibrant and loaded with mass information.

The media is consisted of news, social media, ads. These will make up the background of our virtual environment. For the start I looked a photo collage style to emphasise the realness of these information in contrast to the distortion of the self that later appears.

The contrast between clear and confusing, showing to the user the journey of distortion of the self.

The Self

The self goes on a journey from start to finish in our interactive website. Firstly the user enters the website and is able to contract an avatar, exploring this virtual world. Shown below (the elongated figures shown below).

Then as the get closer to the virtual walls (reality stated above), information breaks in pieces and is attracted onto these figures. They are information you learn, attentions you gained with followers, likes and subcribes, and ads that left an impression that changes and constructs your self.

Eventually the human shape avatar becomes more and more abstract and much like crystals with legs and arms.

Incentives to post

Twittering machine states that the incentives that drives us addictive to the online world are:

• celebrity effect – quantified attention
• sematic impact – emotionally triggering to raise opinions
• survalent aspect – the algorithm that targets your interest

We are making a metaphor of these factors into saturated, eye catching graphic that users ends up attracting and carrying with them to become a part of their identity, and because of the sheer amount and sources of these information, we want to critique that they create a deficit in the user’s self.

This is why the figurative avatar is becoming more and more distorted.

art style reference:

https://davidlinchenstudio.com/

Cinema 4D and Blender

After settling down with the concept I started to work on some 3D elements – the notification ‘crystals’ for Sun to place in the web environment and to test.

I started with the 3D modeling tool I am familiar with, C4D, and created some astroid/floating rock 3D objects as a base for the crystals.

I followed a tutorial here.

Choosing the crayon and colourful style to represent information online that grasps users’ attention. I develop the following patterns. Which I mapped onto the crystals shown below. Also Added some lighting to attract attention and create contrast to background.

Later I learn from Sun that to become an interactive element to be placed into a VR/AR environment these object need to be exported as GLTF. Currently only Cinema4D S22, which is not available for educational license, support the export of GLTF.

So I switched onto Blender to try to replicate the same effect. Below are some of my exploration. I manage to create a rock shape but realised that as soon as it is exported as a GLTF into A-Frame, the lighting, material texture and displacemnet is lost.

I will look into what properties can be transfered to an GLTF and develop our notification crystals accordingly.

One day of middle May in my thesis tutorial, I tried to illustrate my understanding of ‘intelligent behaviours’ with a diagram which was downloaded online featured by a brain controlling hands, eyes and neural system to grab a glass of water. After seeing that, my tutor Vasilija suggested that Rolf Pfeifer’s book ‘How the body shapes the way we think’ may help me to understand the relationship between brain and body from a new perspective.  

Before reading his book, ‘The body is controlled by the brain’ was like a common sense which had already embedded in my mind. It was so naturally inhabited there that I had never questioned about. Pfeifer’s explanation of ‘intelligence requires a body’⁴ pointed out that intelligence can be enabled only by ‘tangible interaction’ between physical systems and its environment. Moreover, he illustrated how the sensors and materials helps the neural system to outsource its computing and control tasks, which greatly challenged my perspective on material that I used to think material is a trivial content which should be considered till the last step in design practice. From his point of view, I started to look for ‘intelligent behaviors’ in materials and structures, which may offer more interactive possibilities in my own design work.  

Abramovich¹ reviewed the decades of research in intelligent structures and materials and summarized the characteristic of intelligent structure as ‘a system containing multifunctional parts that can perform sensing, control, and actuation…’. As a part being incorporated into this system, smart or intelligent material is defined as designed materials that can change their geometry shape or functional property according to external stimuli, such as humidity, temperature, pH, and electric, or brightness of light.

Apart from practitioners in aerospace engineering, medical industry, new resource development and industrial design, architects have also drawn their attention at the development and implement of intelligent structures and materials for some time, and such researches facilitate them to exempt from heavy duty mechanical structures, expensive cost of construction and complex control systems.

Some projects smartly control the surface material by simple actuators, and such manipulation is realized by the designed structure or property of the surface materials. In Güvenç Özel’s interactive installation ‘cerebral hut’ (fig. 01)³, the property of origami structure alleviated the workload of control system and allow the whole surface to contract or expand smoothly just by the control of simple linear actuators. Studio INI designed their kinetic ceiling and floor ‘Urban Imprint’ (fig. 02) which interact with audience’s movement. The material they developed is a mixture of rubber and concrete, and the property of this mixture allow this material to have certain elasticity and ideal weight to balance the tension from the top control system.

Many architects are keen on discovery of materials’ physical properties. The project ‘Bloom’ (fig. 03) from DO|SU studio⁵ architecture harnessed the thermobimetal’s heat reaction to control the open and close of the architecture’s surface form without any energy consumption. This ‘two layers with different stress ratio’ approach can also be found in 4D print technology, allowing the flat printed structure to be self-forming in Post isothermal treatment process. Another project ‘Hygroskin Pavilion’ (fig. 04)² is a collaboration of Achim Menges Architect, Oliver David Krieg, and Steffen Reichert. The 3D printed veneer parts not only have same texture same with plywood but also share the same reactive property as natural materials. It exposed that novel computation and manufacture can imbue man-made material with some ‘natural intelligence’.

References:

1. Abramovich, Haim. Intelligent materials and structures. Walter de Gruyter GmbH & Co KG, 2016.

• Correa, D., and Achim Menges. “3D printed hygroscopic programmable material systems.” In MRS Proceedings, vol. 1800. 2015.

• Ozel, Guvenc. “Case for an architectural singularity: synchronization of robotically actuated motion, sense-based interaction and computational interface.” (2014).

• Pfeifer, Rolf, and Josh Bongard. How the body shapes the way we think: a new view of intelligence. MIT press, 2006.

• Sung, Doris. “Smart geometries for smart materials: taming thermobimetals to behave.” Journal of Architectural Education 70, no. 1 (2016): 96-106.

TUNG

Reflecting on everything that is going on in the world currently (week of June 1, 2020), it feels like humanity is barely holding itself together at the seams – and how could it? With our world still coming to terms with the future that Covid-19 holds, the murder of George Floyd, and the thousands of people protesting around the world due to the demoralizing injustice that our society has been built upon, I can’t help but question our role in this somber future.

It makes me question ourselves not just as individuals but also as a collective of young designers rethinking human interaction. As we talk about inclusive futures in speculative design, it would be hypocritical to not consider the implications of our current context. Clearly, the current anthropomorphic world that we live in shaped by centuries of colonial powers distributing thinking and implementing the ‘society’ as we know it is not sustainable nor is there a future for it. For a long time, I have distanced myself from being ‘political’ in my art practice because the concept of ‘political’ is so intertwined with an image in our current society but what I’m discovering with time is that ‘political’ can exist in any shape or form, and to be blind to it means to be content in the anthropomorphic world that we are so urgently trying to move away from.

What is important to note as I delve into the future of the Planthropocene,[1] a term first used by Natasha Myers, in thesis and in practice, I must clarify that this is not an attempt to run away from the realities and truths that have shaped our Anthropocene. Nor is it to turn a blind eye to the bigger problems that shape our current days. When we think of the plant and nature itself, we do not necessarily connect it with history when in fact, the majority of trees, flowers, and shrubbery seen along the roads, private gardens, and public parks all around London are silent remnants that speak to a colonial past.[2] I would recommend reading Claire Ratinon’s essay: I Don’t Belong Here if you’re interested in reading more about this topic.

Joseph Banks, an 18th century English botanist, believed that by transplanting cotton plants taken from India to the West Indies would allow for cotton to become the nation’s fastest growing industry – which it did. The British cotton goods consumption from from 5.3 million pounds (1781) to 32.5 million pounds (1788).

Source: Wulf, Andrea. The Brother Gardeners: Botany, Empire and the Birth of an Obsession. London: Cornerstone Digital, 2011, p.210)

I believe through knowing history like this, we can begin to see the colonial narrative of the plant and perhaps imagine a Planthropocene that moves away from this narrative.

It is thus acknowledging history like this that I turn towards the Planthropocene. As I begin to create narrative and imagine a different world where human beings are not the only agents for a universe in motion,[3] I am both excited and wary of how to approach this process but perhaps the best approach is to look at the lineage of speculative fiction written throughout history. The way we perceive plants in contemporary society is a result of Aristotelian thinking but time and time again, individuals have attempted to explore otherwise.

In a recent online exhibition curated by the Camden Arts Centre, The Botanical Mind is a reflection of the various views humans across the world have attempted to understand the world through vegetation. More recently, the British Library has also curated a series of fictitious tales of the Botanical Gothic – the genre of Plant Horror. Through this collection of stories, it is interesting to start perceiving the familiarity of plants as something otherworldly and even alien when we begin to imagine ourselves as the other. In Little Joe (2019), despite a somewhat sub-par but visually-beautiful cinematic exploration of an interesting idea, there is a stark reminder that synthetic biology in its rising popularity, requires speculative thinking more than ever. Baum and Leahy, in their project Microbiocene: Ancient Ooze to Future Myths presents a very compelling argument for seeing the world through a microbial lens. It is with these very resources and inspirations that I begin to write my speculative fiction of algae.

References

[1] Myers, Natasha. “Photosynthesis.” Society for Cultural Anthropology. Accessed June 2, 2020. https://culanth.org/fieldsights/photosynthesis.

[2] Wulf, Andrea. The Brother Gardeners: Botany, Empire and the Birth of an Obsession. London: Cornerstone Digital, 2011.

[3] Natania Meeker and Szabari Antónia, Radical Botany: Plants and Speculative Fiction (New York: Fordham University Press, 2020), p.39)