MEDIA ART LIVE ARCHIVE
Intelligent interface for interactive mediation of cultural heritage
A project developing the possibilities of artificial intelligence, especially in the field of art archives and curation.
Vasulka Kitchen Brno, together with Brno Art House, became the application guarantor of the three-body project, which was supported by the Technology Agency of the Czech Republic. Researchers from Masaryk University and Brno University of Technology, led by Jana Horáková, are collaborating on the project. The aim is to process the author's videos of Vašulkovi, both at the level of basic identification and content analysis, in which artificial neural networks are used. This is a unique project in the international context that tests the potential of information technology in the services of humanities, the digital humanities.
The Media Art Live Archive project: Intelligent Interface for Interactive Mediation of Cultural Heritage (hereinafter MALA) responds to dynamic changes in archival and museum practice due to the global trend of digitization of cultural heritage, especially the ongoing digitization of archives of works of art and their online access. However, it is not limited to organizing the content of the archive with the help of traditional library tools, but also explores the potential of digital technologies for innovative ways of working with archives and interactive mediation of cultural heritage to the public. Specifically, the use of artificial intelligence in the form of machine or deep learning of artificial neural networks in the analysis of audiovisual works (AV works) and the development of interfaces for interactive mediation of archive content in the online environment and in the form of installation in the physical space of the Brno House of Arts. As part of the project, an archive of video art pioneers Woody and Steina Vašulka that is owned by Vasulka Kitchen is being processed in this way.
Realized at the moment:
1) Software for content-based searching in image works (completed 12/2020)
The software is implemented by using Python. This allows the use of the Keras framework, which is set to obtain predictions of the convolutional neural network. To work with the video itself, the OpenCV library was used, both to load the processed video and to render and generate the resulting video. Because the OpenCV library does not allow you to work with audio, the original audio is added to the generated video using the moviepy library. In addition, the software includes the ability to generate occurring tags and their probabilities into a graph. The matplotlib library was applied here.
Link to depository: github.com/mediaArtLiveArchive/video
2) Software for content-oriented searching in audio works (completed 12/2020)
Software for content-oriented search in audio works was implemented in Python, as was the case with software that works with visual material. Neural network models were designed and trained in the Keras module for Python. In order to be able to train neural networks on audio material, it was also necessary to use the Kapre module, thanks to which special layers of neural networks can be created and allow individual audio segments to be transformed into spectrograms. The software works with the audio track of audiovisual works and assigns annotations (tags) in predetermined time periods of the audio recording. The Librosa module is also used to work with audio tracks. The resulting annotation of the audio track is available to the user in the form of a JSON file, and the annotation is stored in a graph using the Matplotlib module for Python. Using the software, it is also possible to generate a copy of the source video file with the option of inserting an annotation directly into the image. Most of the software's features are identical to image content analysis software so that it can be controlled in a similar way.
Link to software: github.com/mediaArtLiveArchive/audio
3) Organization of an online exhibition of art projects dealing with the COVID-19 pandemic (completed 12/2020)
Realized exhibition project Black Box, see below.
Link to online exhibition: https://cerna-skrinka.cz/
The exhibition project Black Box thematically responds to the experience of the coronavirus pandemic and tests the possibilities of using artificial intelligence, specifically artificial neural networks programmed by unsupervised learning. This tool was used so that the global experience of a viral pandemic could be viewed from the outside, from the position of an inhuman actor.
Black Box as a curatorial project
The Black Box is a curatorial project that was established in the spring of 2020. Its aim was to point out the problems and challenges that Czech culture faced during the coronavirus pandemic. On March 27 2020 the curators of the TIC Gallery published an open invitation to artists to reflect on this extraordinary situation or to participate in it in the form of civic initiatives. Over the next few weeks, input was collected based on thirty applicants' contributions, from which eight projects were selected. They were Polina Davydenko, Markéta Filipová, Ondřej Homola, Juliana Höschlová, Vojtěch Kundrát, Johana Merta, Ladislav Mirvald, Tomáš Moravanský. The main evaluation criteria for the selection of projects were urgency, authenticity, diversity of chosen means of expression and artistic statements. At this stage, the Gallery TIC team joined forces with the research team working on the project Media Art Live Archive: Intelligent Interface for Interactive Mediation of Cultural Heritage (TA CR) and agreed to cooperate in continuing the project in the form of a curatorial experiment using artificial neural networks curator and on the creation of a web platform that will allow to present the collected documentation of art projects online to innovative way, in an author's web environment.
The name of the Black Box exhibition project refers to lockers stored in vehicles that record the course of a flight or ride and, in the event of an accident or disaster, serve as a source of information in the reconstruction of what actually happened. However, the black box is also a term used in sociology, psychology and cybernetics to refer to systems for which inputs and outputs are known, but it is not clear how the output values were achieved. Due to the complex recurrent processes performed by artificial neural networks, these intelligent tools are often called black boxes. Therefore, we consider the involvement of artificial intelligence in the exhibition project as an organic development of the basic motive of the black box.
Action plan for 2021
A conference on a topic expressed by the working title Live & amp; Real-time: The Art of COVID-19 is planned for 2021. As part of mapping current approaches to curating new media art, both in terms of caring for works of art in order to preserve them as a part of cultural heritage, we organized a two-day online international symposium Digital Curator, at which leading experts from the USA, Great Britain, Germany and Austria spoke. We have therefore established new contacts and obtained the latest information in this area.
The Role of Vasulka Kitchen
It was an arrangement of Vašulka's videos based on defined categories, which will serve as metadata, thanks to which the images will be searchable both on the Vasulka Live Archive website and in the Vašulka's online archive that is also in preparation. Furthermore, the graphic design, structure and functional properties of the Vasulka Live Archive website were consulted, as well as the choice of server and programming language with regard to the long-term sustainability of the website and its connection with the Vašulka Kitchen Brno website and other online projects implemented by VKB members. Based on the assignment, a website proposal was created, which was then submitted to the application guarantors for review, and their comments were incorporated into the proposal. Further consultations are planned in 2021.
Last year, Jana Horákova's team collaborated with Vašulka Kitchen Brno and the House of Arts of the City of Brno on the preparation of a collection of texts reflecting the issue of media archives and strategies for preserving media art called Vašulka's Kitchen Book # 1. Lyrics to media first & last / Vasulka Kitchen Cooking Reader # 1. Beyond Media Texts: Primal & Final.
The author of the text is Jana Horáková.
Media Art Live Archive is realized with the financial help of Technology Agency of the Czech Republic. The Centre of New Media Art – Vasulka Kitchen Brno is a guarantor of the project. Other partners are Brno Art House and TIC Gallery in Brno.