CARBOFLORA
This virtual environment is populated by plants that echo the Earth’s flora from hundreds of millions of years ago, specifically, the plants of the Carboniferous period that now constitute coal fields.
As is widely recognised, our age, aptly named “capitaloscene” by Donna Haraway, is detrimental to the environment and health of all living organisms.
Use of fossil fuels has been repeatedly proven detrimental to the Earth as a whole, yet hope persists that if we completely end our use of fossil fuels, we might reverse some of the effects of global warming and try to restore some of the damage we created over the last two centuries of industrial progress.
Forests of the Carboniferous age consisted of many relatives of contemporary plants – conifers, horsetail, and ferns. Some of the plants, like the early relatives of ferns, could grow to forty meters high. Lepidodendron trees had bark that resembles scales. Fossils of this plant sparked the imagination of our ancestors and might even be responsible for the imaginary construction of dragons.
Although declining, coal is still widely used in industry, not only for direct energy production but also for numerous industrial applications and derivatives; it remains a significant source of carbon dioxide emissions in the atmosphere.
Carboflora environment is connected to tracking the quantities of harmful particles in the atmosphere. Its levels are reflected in the way plants inhabit the virtual system. Properties of virtual plants are connected to a database that tracks air quality in almost real time. More than 10,000 stations throughout the world constantly send data about various pollutants like PM2.5, PM10 (small and big particulate matter), O3 (Ozone), NO2 (Nitrogen dioxide), SO2 (Sulphur dioxide) and CO (Carbon monoxide), as well as the AQI (air quality index). Upon opening, the application chooses the closest physical location and maintains the various properties of plants according to the numbers being sent from the database.
Plants as a sort of timeless ur-forms echo the past and possible future within which we might curb our polluting emissions.
Technical description
Carboflora was developed in the Unity 3D game engine using the free version of the software over a period of five months. Research was followed by modifications of digital objects, construction of the virtual space, programming, testing, and debugging. Some of the objects being used in the virtual world were developed using a variety of other commercial and open source software applications dedicated to modelling, generation, and manipulation of 3D computer graphics. A few of these also included the possibility of developing custom patches for generative development of particular forms, of which some became physical objects through the technique of 3D printing. One class of virtual plants consists of the reconstruction of Carboniferous plants made by scientific illustrator Dariusz Andrulonis which he generously donated to the Carboflora project. Dariusz’s plants are used in Carboflora without the complex original materials he made: because of the high polygon count, they had to be optimised and simplified in order to be placed in the game engine. All objects within Carboflora were homogenised through using the same uniform white material, just as are the physical objects within other installations of Sphere2, in order to bring forth several conceptual elements that constitute the platform “past-future tense of futurism” and “technological optimism” with references to ancient Greek gods of medicine, medical equipment, and biomimetic patterns of natural plants.
There are five scripts for five separate categories of air pollutants. These scripts were assigned to separate three-dimensional objects. At the initial testing phase, five objects were set to react to incoming data, with the possibility of expanding the number of objects that react to data for larger sets of virtual plants.
In addition to the main script that connects to the World Air Quality Index database and separate scripts for each pollutant, a system of tagging is also used. Tagging of particular objects enabled the general script to find and control the properties of trees-objects that should reflect the air quality.
Five virtual trees are growing, depending on the values within each category of particles measured at a particular physical location. When the application is opened, the IP address should choose the nearest physical location provided by the World Air Quality Index database.
Each object-tree has an attached script that controls the size of the tree as well as a tag that enables the main script to find it.
Trees assigned to track data are programmed to change their size every 0.5 seconds. Their size is also recalculated in relation to the average daily amount of particular pollutant. This script is invoked every hour. If there are more particles than average, the trees remain lower, and if there are fewer particles than average, the trees grow larger.
The terrain of Carboflora containing all objects is covered with the Navigation Mesh. A first person controller (FPC) with attached camera is devised as an instance of a simple artificial intelligence agent, so it can move through the terrain, avoid particular objects, and enable the generation of sound textures in real time. The script attached to this navigation agent is optimised, so it can find the best appropriate path for moving among the objects and reaching a particular target. These targets are pre-assigned, so the agent needs approximately one hour to move through all currently assigned spots while creating the sound composition in real time. The radius of sounds spreading from objects is predefined and differs in each case. The movement around Carboflora’s terrain is also limited with implementation of contact colliders.
FULL DESCRIPTION, IMAGES, VIDEO IS HERE
CARBOFLORA, Installation (generative digital environment)
Project by Tanja Vujinović
Production: Ultramono and SciArtLab, 2019
3D objects, generative modelling: Tanja Vujinović
Unity3D programming: Gaja Boc, Sara Bertoncelj Čadež, Tanja Vujinović
Sound: Alessandro Cardinale, Tanja Vujinović
3D objects of carboniferous plants: Dariusz Andrulonis for edukator
Consulting: Vid Podpečan, Jan Kušej, Jelena Guga
English language editing: Derek Snyder
Project is supported by The Department of Culture of Municipality of Ljubljana
ARBORA
Placed within the MetaGarden Sphere2, Arbora is one of the objects that takes care of our health and the health of our environments.
As a wise, old tree rising from the MetaGarden, Arbora is infused with a neural network that understands and responds to human emotions.
Arbora and three protector objects that accompany it are all inspired by the plants of the Carboniferous era. Outer surfaces of objects are covered with bark that resembles scales, much like the Lepidodendron tree that existed approximately 300 million years ago. Fossils of this plant sparked the imagination of our ancestors and might even be responsible for the imaginary conception of dragons.
Emerging from the cloud of mythology, three protector objects are synthetic young trees grown in software. They resemble sprouts and like three ancient Greek gods of medicine, Telosphoros, Hygieia and Asclepius, monitor and reflect the overall environment of MetaGarden.
Arbora senses the emotions expressed in the voice. Our voices can give clues about both the physiological and emotional state we are in. A specially developed and trained deep neural network deciphers the emotional components encoded in the captured voice in order to model a soothing binaural sound. By doing so in synergy with its environment, Arbora, together with its helpers, works towards improving our well-being.
Technical description
Arbora is constructed as a three-dimensional object and printed using 3D printing technology, with the addition of a hardware metal stand. Custom-made electronics placed inside the object are Raspberry Pi, a module with four microphones, and with headphones placed on the stand and connected to the Raspberry Pi.
Arbora is devised as a sentient device, with all of the references already mentioned in their general description: human interacting with the tree, the ancient Greek gods of medicine, biomimetic forms of plants and futuristic medical devices.
Arbora should react to the voices of people in their vicinity who wish to interact with them. For this purpose, we developed a deep neural network model that analyses sound captured from space. Software is written in the Python programming language using several libraries like Keras, librosa, numpy and others. A convolutional deep neural network was constructed using Keras which runs on the Tensor Flow backend. The network was trained on the RAVDESS database which contains professionally recorded audio samples produced by actors in different emotional states.
The mechanism of this action is that the sound is captured and stored temporarily in the sound buffer. Another part of this custom-made software filters the sound and eliminates everything that it does not detect as human speech. One-second-long pieces of sound are analysed, and the average value of determined emotion is extracted. After this, the sound properties pre-assigned to each detected emotion are generated and played through the headphones. Since we wanted to create the situation within which the object would subtly react to a human voice, binaural beats were chosen as one possible type of sound interaction.
The brain is a central powerhouse of the human body, and it is no wonder that the studying of it is of pivotal importance. Brain waves are studied in a variety of academic disciplines, like neurology and its subfield epileptology, within psychology, psychoacoustics, and cognitive neuroscience, among many other fields. While we know that different brain waves are associated with diverse states of mind, for example, delta waves are associated with deep sleep, beta with an alert mind, and alpha waves with a mind that is relaxed, there are numerous research attempts that tend to dive deeper into the fine differences among such waves and associated stimulations that might impact brain functions. Such studies are conducted on both humans and animals. One such potential technique is the implementation of binaural beat (BB) stimulation.
Binaural beats (defined as dual broadcasting of two sinusoids with a small difference in frequency) were implemented for the development of the second part of the software for the Arbora object. Since it is necessary for this particular type of binaural sound to be experienced as two sound streams simultaneously played into both ears, we needed to employ headphones.
Sound is continuously captured by the device. When the visitor speaks near the object, the voice is processed. If there is no new speech detected within the captured sound, the software emits the a pre-determined frequency. Binaural beat, usually consisting of two sinusoidal sounds that differ in frequency might, for example, emit 100 HZ to one ear and 105 Hz to another, and the final setup would provide the following constants: carrier 100, difference 5, and duration 30. Each predefined emotion (neutral, calm, happy, sad, angry, fearful, disgust, surprise) is coupled with the beat that could potentionally restore the emotional state back to calm.
The Arbora software is based on state-of-the-art deep neural networks. It is implemented as a real time audio capture, analysis and synthesis loop which continuously records and analyzes live audio data, classifies the extracted speech using a deep convolutional neural network and generates high quality binaural sound waves designed to stimulate electrical brain activity to thus amplify, soften or transform the detected prevailing emotion. This is possible because of the brain phenomenon known as the frequency following response, which reflects the activity of brainstem neurons and is characterized by a waveform that follows the auditory stimulus wave, which is in our case generated by the Arborasoftware as a precisely designed binaural beat. The parameters used for generating the binaural beat are chosen according to the currently available knowledge about brain waves and their relation to mood and emotions. In addition, a small amount of randomness employed during the selection of the carrier and beat frequency of binaural sounds from the allowed ranges ensures both non-repeatability and continuity.
Arbora protectors are devised as objects that subtly monitor the environment and echo the changes within it through the sound they generate. The custom-made device where all processing happens consists of several connected modules with analog circuits. Input signals come from three sensors that monitor light and temperature within the space. These signals are being routed into a voltage controlled oscillator (VCO), followed by the voltage controlled filter (VCF). One part of the signal travels directly into the mixer, and one pipeline goes into a delay/echo module. There is also a low frequency oscillator (LFO) that joins the others within the sound composition. Signals can be changed from sinusoidal, triangular or square within the VCO and LFO modules. Output sound is subtle, presenting an organic texture that occasionally changes, and, although minimal, it never feels repetitive. /
FULL DESCRIPTION, IMAGES, VIDEO IS HERE
ARBORA
Objects (3D printing, custom-made electronics, custom-made software, sound)
Project by Tanja Vujinović
Programming and custom-made electronics for Arbora object: Dr. Vid Podpečan, Department of Knowledge Technologies, Institute Jozef Stefan
Custom-made electronics for Arbora Protectors objects: Gregor Krpič
3D printing: RogLab
Consulting: Vid Podpečan, Jan Kušej, Jelena Guga, Gregor Krpič, Tomo Per
English language editing: Derek Snyder
Production: Ultramono and SciArtLab, 2019
Project is supported by The Department of Culture of Municipality of Ljubljana
Other significant previous and ongoing projects
Vid Podpečan: Research involving NAO humanoid robot
Tanja Vujinović:
Fontana ****
Genera ****
