OER·harvester

← Back to the library
Zenodo PDF resource

The Ghost in the Machine - AI's Impact on Cultural Heritage (Research)

The Ghost in the Machine - AI's Impact on Cultural Heritage (Research) Over the past decade, deep learning methods have made remarkable advancements. This progress can be attributed to various factors such as massive parallelization through the utilization of Graphics Processing Units (GPUs) for massive parallelization. This shift in hardware has significantly accelerated the training of deep neural networks, allowi…

Licence
OPEN CC-BY-4.0
Authors
Sack, Harald
Published
2023-11-30 · Zenodo
Language
eng
Length
1976 words
Type
slide
Open ↗ Download Open original ↗

Source: The Ghost in the Machine - AI's Impact on Cultural Heritage (Research) · Zenodo Authors: Sack, Harald Licence: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0/

The Ghost in the Machine-AI’s Impact on Cultural

Heritage (Research)

Prof. Dr. Harald Sack IM/MATERIALITIES 2023 Torino, Nov 2023 Prof. Dr. Harald Sack: “The Ghost in the Machine-AI’s Impact on Cultural Heritage (Research)”, IMATERIALITIES 2023, 29.11.2023

The Ghost in the Machine AI’s Impact on Cultural Heritage (Research)
1. 2. 3. Outline: ○ The Ghost in the Machine-AI’s Impact on Cultural Heritage (Research) Over the past decade, deep learning methods have made remarkable advancements. This progress can be attributed to various factors such as massive parallelization through the utilization of Graphics Processing Units (GPUs) for massive The Impact of Shifting this Presentation by a few Years parallelization. This shift in hardware has significantly accelerated the training of deep neural networks, allowing researchers to tackle increasingly complex problems. Another critical factor contributing to the success of deep learning is the acquisition of vast training datasets sourced from the World Wide Web, which has become a treasure trove of AI’s Quantum Leap Forward information. As a result, these models have become adept at capturing intricate patterns and representations in various Linked Stage Graph and Visual Analysis domains. Furthermore, the development of efficient and reusable neural network architectures has also played a crucial role in the advancement of deep learning. Putting everything together, these evolutions have paved the way for the Large Language Models and the Art of Creative achievement of human-like or even superhuman performance in specific domains. Notably, the emergence of pre-trained large language models has demonstrated the capability to grasp the intricate semantics of natural languages, Hallucination yielding exceptional outcomes in classification, prediction, and generation tasks. Similarly, in the realm of image generation, models such as Stable Diffusion and Dall-E have showcased their prowess.
4. ○ Hybrid AI Iconclass Image Search and Image Classification
2 [https://chat.openai.com/chat https://beta.openai.com/playground](https://chat.openai.com/chat

The Impact of Shifting this Presentation by a few Years

AI’s Impact on Cultural Heritage (Research)

https://www.ml6.eu/resources/large-language-models

60+ Years of Machine Learning

The Road to Large Language Models

Emergence of … “How” (from examples) “Features” (used for prediction) (advanced) “functionalities”

Homogenization of … Learning Algorithms Model Architectures Models

(as e.g. logistic regression) (as e.g. CNNs) (as e.g. GPT-3/GPT-4)

Bommasani, Rishi, et al., On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.

The Advent of Foundation Models

Bommasani, Rishi, et al., On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.

From Deep Learning to Foundation Models

The Essential Factors for Success
● Transfer Learning
(Deep Learning) Pretraining the model for a surrogate task, fine-tuning of the model for a specific downstream task
● Scale ○ ○ ○ Improvement of computational power (GPU throughput and memory) Development of transformer model architecture leveraging GPU parallelism enabling more expressive models Availability of much more training data
● Self-supervised Learning

Distributional semantics, autoregressive language models, Transformer based architectures, multimodality

From Basic to Bold in a few Years! The Ghost in the Machine

Example for AI's Quantum Leap Forward AI’s Impact on Cultural Heritage (Research)

The Ghost in the Machine-AI’s Impact on Cultural Heritage (Research)

Over the past decade, deep learning methods have made remarkable advancements. This progress can be attributed to various factors such as massive parallelization through the utilization of Graphics Processing Units (GPUs) for massive parallelization. This shift in hardware has significantly accelerated the training of deep neural networks, allowing researchers to tackle increasingly complex problems. Another critical factor contributing to the success of deep learning is the acquisition of vast training datasets sourced from the World Wide Web, which has become a treasure trove of information. As a result, these models have become adept at capturing intricate patterns and representations in various domains. Furthermore, the development of efficient and reusable neural network architectures has also played a crucial role in the advancement of deep learning. Putting everything together, these evolutions have paved the way for the achievement of human-like or even superhuman performance in specific domains. Notably, the emergence of pre-trained large language models has demonstrated the capability to grasp the intricate semantics of natural languages, yielding exceptional outcomes in classification, prediction, and generation tasks. Similarly, in the realm of image generation, models such as Stable Diffusion and Dall-E have showcased their prowess.

https://chat.openai.com/chat https://beta.openai.com/playground

Example Linked Stage Graph

Example: Linked Stage Graph Image Annotation

  • Knowledge Graph of Archival Documents and Photographies from Stuttgart State theatres 1890 - 1940
  • Web page: https://slod.fiz-karlsruhe.de/ T. Tietz et al., Linked Stage Graph, in Proc. of the 15th Int. Conf. on Semantic Systems, 2019. T. Tietz et al., A Data Model for Linked Stage Graph and the Historical Performing Arts Domain, In Proc. of the Int. Workshop on Semantic Web and Ontology Design for Cultural Heritage (SWODCH), 2023. https://chat.openai.com/chat

Example Linked Stage Graph

PROBLEM:

  • No content-related Metadata or image descriptions available SOLUTION:

  • Visual Analysis:

○ Object Identification
○ Image Captioning

Example Linked Stage Graph

Visual Analysis:

  • Object Identification via denseNet-101 (2018) Huang, G. et al, Densely Connected Convolutional Networks. arXiv 2018, arXiv:1608.06993.

Example Linked Stage Graph

Visual Analysis:

Example Linked Stage Graph

Visual Analysis:

The Ghost in the Machine

Large Language Models and the Art of Creative Hallucination

Taylor, R., et al., Galactica: A large language model for science, arXiv preprint arXiv:2211.09085, 2022. https://galactica.org/paper/

The Advent of Foundation Models

https://twitter.com/ylecun/status/1592619400024428544

https://cs.nyu.edu/~davise/papers/ExperimentWithGalactica.html

https://en.wikipedia.org/wiki/Hanlon%27s_razor

https://www.aleph-alpha.com/

Semantics from Stochastics

https://www.aleph-alpha.com/

  • Language Domain

  • Based on probability and statistics it is possible to create syntactically and semantically correct texts.

  • With larger training data and larger models also contectually and might be pragmatically fitting texts can be created. or Evaluative questions

  • Factual questions can be correctly answered. (of the training data). bias

  • Interpretative questions Interpretative Questions or Evaluative questions might be subject subject of inherent of inherent bias (of the training data)

The Ghost in the Machine

Symbolic Knowledge Representation to the Rescue
Symbolic AI
Subsymbolic AI

Hybrid AI

  • Neural Networks, Deep Learning &

Models

Foundation Models

Hybrid AI-Combining Symbolic and Subsymbolic AI

Knowledge Graph Embeddings
  • Knowledge Graph Completion ● Ontology Mapping
  • KGE for Classification Tasks ● Entity/Knowledge Graph Alignment R. Biswas et al.: MADLINK: Attentive Multihop and Entity Descriptions for Link Prediction in Knowledge Graphs, Semantic Web Journal, 202 G. A. Gesese et al.: RAILD: Towards Leveraging Relation Features for Inductive Link Prediction, IJKGC 2022 G. A. Gesese et al. A Survey on Knowledge Graph Embeddings with Literals: Which model links better Literal-ly?, Semantic Web Journal, 12(4), 2020 R. Biswas et al.: It's All in the Name: Entity Typing Using Multilingual Language Models, ESWC 2022 https://peerj.com/articles/cs-341/

Hybrid AI-Combining Symbolic and Subsymbolic AI

Explainability and Fact Checking

The Ghost in the Machine

Another Example for AI's Progress AI’s Impact on Cultural Heritage (Research)

The Ghost in the Machine-AI’s Impact on Cultural Heritage (Research)

Over the past decade, deep learning methods have made remarkable advancements. This progress can be attributed to various factors such as massive parallelization through the utilization of Graphics Processing Units (GPUs) for massive parallelization. This shift in hardware has significantly accelerated the training of deep neural networks, allowing researchers to tackle increasingly complex problems. Another critical factor contributing to the success of deep learning is the acquisition of vast training datasets sourced from the World Wide Web, which has become a treasure trove of information. As a result, these models have become adept at capturing intricate patterns and representations in various domains. Furthermore, the development of efficient and reusable neural network architectures has also played a crucial role in the advancement of deep learning. Putting everything together, these evolutions have paved the way for the achievement of human-like or even superhuman performance in specific domains. Notably, the emergence of pre-trained large language models has demonstrated the capability to grasp the intricate semantics of natural languages, yielding exceptional outcomes in classification, prediction, and generation tasks. Similarly, in the realm of image generation, models such as Stable Diffusion and Dall-E have showcased their prowess.

https://chat.openai.com/chat https://beta.openai.com/playground

Example Iconclass

Example: Iconclass-based Image Classification and Multimodal Image Search

  • Classification system for art and iconography with 28.000+ concepts
  • Web page: https://iconclass.org/ C. Santini et al.,Multimodal Search on Iconclass using Vision-Language Pre-Trained Models. JCDL 2023, pp. 285-287 E. Posthumus, et al.: The Art Historian’s Bicycle Becomes an E-Bike. VISART @ ECCV 2022

Syntactic Text-based Search

Iconclass Image Search and Classification

Iconclass Multi-Label Multi-Class Classification

Iconclass Image Search and Classification

Iconclass Multi-Label Multi-Class Classification

Iconclass Image Search and Classification

Similarity-based Image Search

Iconclass Image Search and Classification

Similarity-based Image Search

The Ghost in the Machine -AI’s Impact on Cultural Heritage (Research)

Prof. Dr. Harald Sack, FIZ Karlsruhe – Leibniz Institute for Information Infrastructure, IMATERIALITIES 2023

Tha�� y�� �e�y ���h ��� yo�� At�e�t���!

Prof. Dr. Harald Sack FIZ Karlsruhe – Leibniz Institute for Information Infrastructure harald.sack@fiz-karlsruhe.de Fediverse: @lysander07@sigmoid.social Created via Midjourney

The Ghost in the Machine -AI’s Impact on Cultural Heritage (Research)

Prof. Dr. Harald Sack, FIZ Karlsruhe – Leibniz Institute for Information Infrastructure, IMATERIALITIES 2023

Bibliography:

[1] Bommasani, Rishi, et al., On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.

[2] Zhou, Ce, et al., A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT. arXiv preprint arXiv:2302.09419, 2023.

[3] T. Tietz et al., Linked Stage Graph, in Proc. of the 15th Int. Conf. on Semantic Systems, 2019.

[4] T. Tietz et al., A Data Model for Linked Stage Graph and the Historical Performing Arts Domain, SWODCH2023, 2023.

[5] Huang, G. et al, Densely Connected Convolutional Networks. arXiv 2018, arXiv:1608.06993.

[6] Taylor, R., et al., Galactica: A large language model for science, arXiv preprint arXiv:2211.09085, 2022.

[7] C. Santini et al., Multimodal Search on Iconclass using Vision-Language Pre-Trained Models. JCDL 2023, pp. 285-287.

[8] E. Posthumus, et al., The Art Historian’s Bicycle Becomes an E-Bike. VISART @ ECCV 2022.