Echoes of WHALE FALL
Machine learning and emotional analysis
audio-visual system
Echoes of Whale Fall
Echoes of Whale Fall is an emotion-driven audiovisual system that uses machine learning to analyze emotional signals, transforming deep-sea sounds and imagery into a generative narrative of loss, memory, and rebirth.
Team
Lilith Ren
Location
Harvard GSD
Dates
25 Spring
Machine Learning
Dataset & Source Material
A curated dataset of deep-sea whale footage and acoustic recordings forms the foundation of the system.
These clips are segmented into narrative moments such as emergence, movement, and collective behavior, providing structured input for emotional annotation.
Framework
Narrative Framework to Emotion Driven Visualization System to Generative Audio-Visual Environment
Machine Learning Process
Signal Processing & Abstraction
Raw audiovisual data is decomposed into multiple layers of information, including motion patterns, intensity shifts, and temporal rhythms. This abstraction process translates natural phenomena into analyzable signals that can be computationally interpreted.
Storyboard of Whale Fall
Emotional Mapping (ML Layer)
Each segment is annotated using a three-dimensional affective model—arousal, dominance, and valence—creating an emotional dataset. This structured mapping enables machine learning to operate as a bridge between sensory input and generative output.
Emotional Analysis and Sound Pipeline
The emotional data drives a non-linear sequencing system, where transitions between states (e.g., silence, emergence, loss) are dynamically arranged. Rather than a fixed timeline, the narrative evolves through emotional continuity and variation.
Generative Visual Output
Machine learning outputs are translated into visual forms, where parameters such as density, motion, and color respond to emotional states. This results in an evolving visual language that
reflects the underlying affective structure of the data.
System Implementation
The system is implemented in Python, integrating audio processing and rule-based generation.
Machine learning here functions not as prediction, but as a mediating framework that continuously maps emotional inputs to audiovisual behaviors.