II Generative AI
Generative AI refers to a class of algorithms and models within AI and NLP that are designed to generate new, previously unseen data that is similar to existing examples by employing a variety of techniques [21]. These models learn the underlying patterns and structures present in the training data and use that knowledge to create novel instances that resemble the original data. It has the potential to revolutionize many industries and creative fields. Generative AI models are trained on large datasets of existing content. Generative models aim to capture the underlying distribution of data, enabling them to generate new samples that are statistically similar to the training data. To achieve this, generative models employ a latent space, denoted as $Z$, which represents a hidden or underlying representation of the data. This latent space is then mapped to the actual data space, denoted as $X$, through a generator function, represented by $G_{\theta}(Z)$. The parameter $\theta$ represents the adjustable parameters of the generative model, which are optimized during the training process. The goal of training a generative model is to make the generated samples, $G_{\theta}(Z)$, virtually indistinguishable from real data samples by focusing on maximizing the probability of generating the observed data samples. The objective function for training a generative model, without specifying a particular architecture, is expressed in Equation 1, where $N$ is the number of training samples, $x^{(i)}$ represents the $i^{th}$ training sample, and $p_{\text{model }}\left(x^{(i)};\theta\right)$ denotes the probability assigned by the generative model to the $i^{th}$ training sample.
$$ \max_{\theta}\sum_{i=1}^{N}\log p_{\text{model }}\left(x^{(i)};\theta\right) $$
II-A GANs (GANs)
GANs are a type of generative AI model that consists of two neural networks: a generator and a discriminator [27]. The generator is responsible for creating new realistic and high-quality data, including images, text, and music, by learning the underlying distribution of the data [28]. The discriminator, on the other hand, is responsible for distinguishing whether the new data is real or fake [28]. The fundamental principle behind GANs involves a generator network creating realistic data, such as images, and a discriminator network evaluating the generated data by distinguishing between real and fake data [28]. Over time, the generator improves its ability to create realistic data by attempting to deceive the discriminator, which enhances its ability to distinguish between real and generated data [28]. The training process of a GAN network, as shown in Equation 2, involves optimizing the parameters of both the generator (represented by $G$) and discriminator (represented by $D$) networks [28]. Here, $p_{{data}}(x)$ denotes the distribution of real data, $p_{z}(z)$ represents the distribution of random noise in the latent space, $x$ denotes a real data point, $G(z)$ is a data point generated from random noise $z$, $D(x)$ is the discriminator’s output indicating the probability that $x$ is real, and log refers to the natural logarithm. The objective is to minimize the log-probability of the discriminator correctly identifying whether a sample is real or generated, while simultaneously maximizing the log-probability of the generator producing data that the discriminator perceives as real.
$$ \min_{G}\max_{D}V(D,G)=\\ \mathbb{E}_{x\sim p_{\text{data }}(x)}[\log D(x)]+\mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z)))] $$
Within the adversarial setting, various classes of GANs have emerged over the years, each tailored to specific tasks in the generative modeling space. For example, the work of Radford et al. [29] presents a DCGANs (DCGANs) by extending the GANs architecture, an extension of the original GAN architecture proposed by Goodfellow et al. [28]. DCGANs employ CNNs in both the generator and discriminator, enabling the generation of high-quality images. CNNs are known to perform well at capturing spatial relationships in data [29], making them well-suited for image generation tasks. Addressing the training instability issues of [28], Arjovsky et al. introduced the WGANs (WGANs) algorithm [30]. WGANs replace the binary cross-entropy loss with the Wasserstein distance, leading to improved stability and convergence during training [30]. In the context of GANs, the Wasserstein distance defines the objective function between two distributions, denoted as $A$ and $B$, as shown in Equation 3. Here, $W(A,B)$ represents the Wasserstein distance between distributions $A$ and $B$, $\inf$ denotes the infimum, which represents the minimum value, $\gamma$ refers to a joint distribution defined on the product space of $A$ and $B$, $\Pi(A,B)$ is the set of all joint distributions with marginals $A$ and $B$. The terms $(x,y$ represent samples from the joint distribution $\gamma$, and $d(x,y)$ denotes the distance between $x$ and $y$ in the metric space.
$$ W(A,B)=\inf_{\gamma\in\Pi(A,B)}\mathbb{E}_{(x,y)\sim\gamma}[d(x,y)] $$
To tackle the challenges associated with training high-resolution GANs, Karras et al. [31] proposed a progressive growth of GANs. This algorithm employs a progressive training strategy that gradually increases the resolution of the generated images throughout the training process. This approach allows the algorithm to capture finer details and produce high-resolution images with enhanced stability and scalability [31].
II-B Variational Autoencoder Models
VAEs are generative models that learn a probabilistic mapping from the data space to a latent space, a lower-dimensional representation of the data that captures its essential features, enabling the generation of new samples through sampling from the learned latent space [23]. This process involves two key components: encoders and decoders. In the VAEs framework, encoders and decoders, play important roles in the process of learning and generating data. The encoder is implemented using a neural network and it is responsible for mapping the input data $x$ to a probability distribution in the latent space $z$, as shown in Equation 4. Similar to the encoder, the decoder is also implemented using a neural network, and it reconstructs the original data from this latent representation, $z$, as illustrated in Equation 5. The encoder and decoder are trained jointly using a technique called variational inference [23, 32]. Variational inference minimizes two losses: a reconstruction loss and a regularization loss. In Equation 4 $\mu_{\phi}(x)$ and $\sigma_{\phi}(x)$ represent the mean and standard deviation of the distribution, respectively. In Equation 5, the parameters $\mu_{\theta}(z)$ and $\sigma_{\theta}(z)$ represent the mean and standard deviation of the latent space distribution, which are learned by the decoder neural network during training.
$$ z=q_{\phi}(z\mid x)=\mathcal{N}\left(\mu_{\phi}(x),\sigma_{\phi}(x)^{2}\right) $$
$$ p_{\theta}(x\mid z)=\mathcal{N}\left(\mu_{\theta}(z),\sigma_{\theta}(z)^{2}\right) $$
The reparameterization trick, introduced in VAEs to facilitate backpropagation through the sampling process [23], addresses the challenge of applying backpropagation to inherently random sampling operations. While backpropagation is a fundamental algorithm for training neural networks, its direct application to sampling is problematic due to the randomness involved. The reparameterization trick provides an alternative approach to sample from a distribution while maintaining the necessary connections for backpropagation [23]. In VAEs, this technique is employed to sample the latent variable, $z$, from a simple distribution, typically a standard normal distribution. These samples are then transformed to match the distribution produced by the encoder, as described in Equation 6. This transformation ensures that the sampled latent variables remain consistent with the encoder’s understanding of the data while preserving the randomness required for generating new samples. In Equation 6, the $\epsilon$ represents a random noise vector sampled from a standard normal distribution, $\odot$ represents the element-wise product operation, $\sigma_{\theta}(x)$ represents the standard deviation of the distribution produced by the encoder, and $\mu_{\theta}(x)$ represents the mean of the distribution produced by the encoder.
$$ z=\mu_{\theta}(x)+\sigma_{\theta}(x)\odot\epsilon,\text{~where}~\epsilon\sim\mathcal{N}(0,1) $$
The main objective for training a VAE is to maximize the ELBO (ELBO) [23, 33]. Maximizing the ELBO during training encourages the VAE to learn a meaningful and smooth latent space representation for the input data [23, 33]. By maximizing the ELBO, the VAE is trained to learn a latent space that captures the underlying structure of the data while also allowing for the efficient generation of new samples [23, 33]. The ELBO, as shown in Equation 7, comprises two terms: the reconstruction loss of the data given the latent variable ($logp_{\theta}(x\mid z)$), which measures the expected log-likelihood of the data given the latent variable, and the KL (KL) divergence between the approximate posterior (encoder) and the prior distribution ($D_{\mathrm{KL}}\left(q_{\phi}(z\mid x)\|p(z)\right)$). The KL divergence encourages the latent distribution learned by the encoder to be similar to the prior distribution, which is typically a standard normal distribution. This constraint helps prevent the encoder from learning overly complex or entangled latent representations. In Equation 7, the $\mathcal{L}$ denotes the overall loss function.
$$ \mathcal{L}(\theta,\phi;x)=\mathbb{E}_{q_{\phi}(z\mid x)}\left[\log p_{\theta}(x\mid z)\right]-D_{\mathrm{KL}}\left(q_{\phi}(z\mid x)\|p(z)\right) $$
II-C Autoregressive Models
In the context of Generative AI, autoregressive models are a class of likelihood models that generate new sequential data by predicting the next value in a sequence based on the previous values. These models involve modeling the probability distribution of each element in a sequence given the entire history of previous elements, $P(x_{t}|x_{t1},x_{t2},\ldots,x_{1})$. This ability makes autoregressive models well-suited for a variety of NLP tasks where the ability to understand and generate coherent sequences is essential [34]. They are also widely used in capturing the dynamics of time series data [34]. An autoregressive model of order $p$ can be generally represented as shown in Equation 8 where $X_{t}$ denotes the value of the time series at time $t$, $c$ is a constant term, $\phi_{i}$ are the autoregressive coefficients, representing the influence of the previous $i^{th}$ observations on the current observation, and $\epsilon_{t}$ is an error term, which represents the random noise in the data. The parameters of the model ($c,\phi_{i}$) are typically estimated from the observed data using methods like maximum likelihood estimation.
$$ X_{t}=c+\sum_{i=1}^{p}\phi_{i}X_{t-i}+\epsilon_{t} $$
II-D Mixture of Expert Models
A MoE (MoE) model represents a neural network architecture that combines the strengths of specialized expert networks with a gating mechanism to perform complex tasks [35, 36]. In the context of NLP architectures, MoE models are applied to enhance the capabilities and efficiency of the underlying language generation architecture [35, 37]. Within the realm of MoE models, these architectures optimize resource utilization by selectively activating relevant experts for specific tasks, demonstrating adaptability to different domains through the integration of domain-specific expert models [38]. Moreover, MoE architectures offer scalability, allowing the addition of more expert networks to handle a broader range of tasks [39]. The advantages of MoE models extend beyond their architectural complexities. Recent studies, such as the work presented in [39], emphasize their scalability, enabling the addition of more expert networks to handle a broader range of tasks. Furthermore, these models have demonstrated the ability to achieve superior model quality compared to their dense counterparts, even with significantly reduced training costs. However, despite these advantages, MoE models pose some critical challenges. MoE models are sensitive to small changes in the gating network weights. Since the gating network determines the contribution of each expert to the final prediction, even slight changes in these weights can lead to significant shifts in the model’s training stability and cause unpredictable behavior [35]. This sensitivity can make training and optimization of the model more challenging. To mitigate this, techniques such as sparse routing have been proposed [40, 41]. Regularization techniques such as weight decay or dropout also help mitigate sensitivity to small changes in gating network weights by preventing overfitting and promoting smoother decision boundaries [36]. Additionally, training MoE models can be computationally intensive, especially when dealing with a large number of experts or complex gating functions. Each forward pass through the network involves evaluating the outputs of multiple experts and updating the parameters of both the expert and gating networks. This computational overhead can make training slower and require more resources compared to simpler neural network architectures. Developing more efficient training algorithms specifically tailored for MoE models can help reduce computational intensity. The overall MoE model architecture can be broken down into several key components including the following.
Expert Networks. One of the main features of the MoE model is the presence of multiple expert networks. These expert networks play a critical role in learning specific patterns or features within the input data and serve as the core models of the MoE system. Each expert network is tailored to specialize in a particular aspect or subset of the input problem space.
Gating Network. The gating network mechanism a crucial component that analyzes the input data and decides which expert network is most suitable for a given instance [40]. It assigns weights to each expert, indicating their relevance or contribution to the current input. The gating network typically outputs a probability distribution over available experts, reflecting the relevance of each expert to the current input [40]. There are two main types of MoE routing strategies in MoE systems: dense routing and sparse routing. In dense routing, every input is directed to all experts, and the final output is a weighted combination of all expert predictions based on the gating network’s output. On the other hand, sparse routing is a more efficient approach where the gating network selects only a subset of experts for each input, reducing computational cost [35, 42]. The MoE model dynamically combines the predictions of multiple experts based on learned gating coefficients, allowing it to adaptively switch between different experts depending on the input data. This mechanism enables the model to capture complex patterns and improve performance compared to a single expert model. The gating network is generally represented as shown in Equation 9 where $g_{k}(x)$ denotes the gating function for gate $k$, $\sigma$ is an activation function (usually sigmoid or softmax), and $W_{g}$ represents the parameters of the gating network.
$$ g_{k}(x)=\sigma\left(W_{gk}^{T}x\right) $$
Output Computation. When the experts are activated, they process input data and generate individual predictions. These predictions are then combined to form the final output of the MoE model. The specific method of combining predictions depends on the task and MoE architecture. In the weighted averaging approach, predictions from each expert are weighted based on the output of the gating network, and the weighted average is taken as the final output. In classification tasks, experts can vote for the most likely class, and the majority vote becomes the final prediction [43]. The output of a MoE model, denoted as $y(x)$, is computed using Equation 10, representing a weighted sum of the expert outputs. The final output, $y(x)$, is computed by aggregating the contributions of all experts. It sums up the weighted outputs of all experts based on the gating values, resulting in the MoE’s prediction. This output is often passed through additional layers, such as fully connected layers or activation functions, depending on the specific task. Here, $E_{i}(x)$ denotes the output of expert $i$, $x$ represents an input to the model, and $N$ is the number of experts [35]. Gating weights $g_{i}(x)$, detailed in Equation 11, are computed using a softmax function, with $a_{i}(x)$ representing the activation for an expert $i$ given the input $x$. The gating network uses the input data to determine which expert is best suited for the task.
$$ y(x)=\sum_{i=1}^{N}g_{i}(x)\cdot E_{i}(x) $$
$$ g_{i}(x)=\frac{\exp\left(a_{i}(x)\right)}{\sum_{j=1}^{N}\exp\left(a_{j}(x)\right)},\quad i=1,2,\ldots,N $$
II-E Model Merging
Model merging is a technique used to combine the parameters of multiple task-specific pre-trained LLMs to create a new and improved language model [44]. Initially, this involves the process of selecting base models and aligning the architectures of chosen models to ensure compatibility. Techniques such as parameter averaging [45] or knowledge distillation [46, 47] are then employed to integrate the knowledge from these models. Additionally, various algorithms, including task vector arithmetic [48], TIES [44], and DARE [49] can be used for parameter merging, each with its own advantages and considerations, such as computational complexity and the ability to handle models trained on different tasks. Following integration, the merged model undergoes fine-tuning on task-specific data to refine its representations and potentially optimize overall performance. The resulting merged model retains the knowledge and capabilities of its constituent models, leading to enhanced performance and capabilities across tasks compared to traditional methods of training a single model from scratch, as well as improved robustness and resource efficiency [50]. However, challenges such as ensuring compatibility between models, managing computational complexity, and avoiding performance degradation must be addressed [50, 51].
II-F Diffusion Models
Diffusion models are specifically designed for generating images and data samples [52]. These models are trained to generate realistic samples by modeling the diffusion process of a data distribution. Different approaches like NCE (NCE) [53] and score-based generative modeling [54] exist within the domain of diffusion models in Generative AI. They operate by iteratively adding noise to a given initial image and subsequently learning to reverse this process to generate new, realistic, and high-quality images of varying styles and complexities [55, 56]. As shown in Equation 12, the general idea is to model the data distribution as a diffusion process, where the data is transformed from a simple distribution to the target distribution through a series of steps. Here, $x_{t}$ represents the data at time step $t$, $f$ denotes a diffusion process that transforms the data from $x_{t-1}$ to $x_{t}$, $\theta_{t}$ represents the parameters of the diffusion process at time step $t$, and $\epsilon_{t}$ represents a sample from a noise distribution $t$. This approach has led to the development of generative models such as DSM (DSM) and diffusion probabilistic models. The underlying idea is to transform a simple distribution through a series of steps to match the target distribution of real data. The generative process involves reversing these steps to generate new samples. Diffusion-based generative models, such as DALL-E 2 [57, 58], Imagen [59], stable diffusion [60], and others, are a class of probabilistic models that describe the evolution of an image from a simple initial distribution to the desired complex distribution [61].
$$ x_{t}=f(x_{t-1},\theta_{t},\epsilon_{t}) $$
Stable Diffusion. Text-to-image generation involves creating visual content based on textual descriptions [62]. Stable diffusion is an open-source text-to-image diffusion model that generates diverse and high-quality images based on textual prompts[^2]. This model operates by taking a noisy image as input and gradually denoising it to generate the desired output. The denoising process is guided by a text prompt, providing information about the desired content and style of the image.
Midjourney. Midjourney is a text-to-image diffusion model that, like Stable Diffusion [60], leverages prompts to generate unique and artistic images [63]. However, it is a closed-source Generative AI project requiring a paid subscription. This setup consequently may discourage community collaboration and development, leaving some users with less control over the underlying model compared to open-source alternatives like stable diffusion [60].
II-G Multimodal Generative Models
Multimodal generative models represent a significant advancement in AI. These models possess the capability to understand and create content by leveraging various data types, such as text, images, and audio [64, 65]. This integration of different data modalities enables these models to capture a more comprehensive understanding of concepts [66]. By utilizing information from these diverse sources, multimodal generative models aim to overcome the limitations inherent in traditional models that focus solely on a single data type [65]. Unimodal methods, traditional approaches that primarily focus on a single modality, such as text or images, have limitations in capturing the full complexity of real-world data [65]. For example, text-based models may lack the ability to incorporate visual or emotional context into their understanding, while image-based models might lack textual or semantic understanding [65]. Multimodal generative models address these limitations by integrating information from different modalities, such as text, images, and audio. This allows them to achieve a better understanding of the data and subsequently generate content that reflects the richness of human expression and experience. However, training multimodal models comes with its own set of challenges. These models can be computationally expensive to train and require large amounts of labeled data for each modality [65]. Additionally, finding effective techniques to seamlessly integrate information from different modalities remains an active area of research [67]. There are two main architectures used for multimodal learning: early fusion and late fusion [68]. Early fusion combines data from all modalities at the beginning of the model, while late fusion processes each modality independently before combining the results. The ability of multimodal generative models to understand and create content across different data types makes them invaluable for a wide range of tasks requiring a deep understanding of multimodal data [69]. Some real-world applications include generating realistic product descriptions with images for e-commerce platforms or creating personalized music recommendations based on a user’s audio preferences and listening history. In addition to this, these models have demonstrated remarkable capabilities in various tasks, including medical imaging analysis, image captioning, text-to-image synthesis, video understanding, and audio-visual storytelling [69]. By overcoming the limitations of unimodal models and offering new possibilities for creative content generation, multimodal generative models will play a significant role in the future of AI.
II-H Applications of Generative AI
Generative AI models are powerful tools for understanding and generating data with applications in various domains, including the following.
Image Generation and Analysis. Advanced Generative AI models have demonstrated remarkable capabilities in generating high-quality images, such as photorealistic faces and scenes [21]. Generative AI models have been employed in developing complex systems capable of generating and understanding multimodal data such as text and images. For example, the work in [70] proposes a large-scale autoregressive model that generates high-quality and content-rich images from text descriptions. Additionally, DALL-E is a generative model introduced by Ramesh et al. [57, 58], which produces images from textual descriptions. Unlike traditional image generation models that rely on pixel-level manipulations or predefined templates, DALL-E operates at a semantic level, understanding textual prompts and synthesizing corresponding images. The work in [71] introduces a novel architecture specifically designed for generating high-quality facial images. This architecture utilizes a style-based generator, demonstrating advancements in synthesizing diverse and realistic images. Furthermore, Generative AI models can also be employed in image-to-image translation [72], which involves converting images from one domain to another, such as enabling the conversion of satellite images into maps or black-and-white photos into color. The work by Zhu et al. [73] presents a model designed for unpaired image-to-image translation. This model utilizes cycle-consistent adversarial networks to learn mappings between two image domains without requiring paired training examples, making it versatile for various applications [73]. Unlike DALL-E [58], which primarily focuses on generating images, CLIP (CLIP) learns to understand the relationships between text and images in a paired manner [69]. Through contrastive learning, CLIP pre-trains on vast amounts of image-text pairs, enabling it to encode both modalities into a shared embedding space [69]. CLIP’s cross-modal understanding enables a wide range of applications beyond traditional image analysis tasks. By associating images with their textual descriptions, CLIP can perform tasks such as image classification, object detection, and even zero-shot learning, where it recognizes objects or concepts not seen during training [69]. CLIP is built upon a dual-encoder architecture, featuring separate encoders for processing images and text. This architectural design allows CLIP to independently encode visual and textual inputs into distinct feature spaces, facilitating effective cross-modal understanding. For image processing, CLIP often employs CNNs or ViT (ViT) to extract visual features [74]. The image encoder within CLIP processes visual inputs, such as images, using CNNs. Through pre-training on large-scale image datasets, the image encoder learns to extract hierarchical visual features that capture important characteristics of the input images. These features are then encoded into a high-dimensional representation space. On the other hand, the text encoder in CLIP processes textual inputs, such as captions or descriptions, using transformer architectures [18, 20]. Transformers are capable of modeling sequential data like text, allowing the text encoder to capture semantic information and contextual relationships within textual inputs. Through pre-training on large-scale text corpora, the text encoder learns to encode textual inputs into a corresponding feature space. Despite having separate encoders for images and text, CLIP achieves cross-modal understanding by mapping both image and text embeddings into a shared embedding space. This shared space facilitates direct comparisons between visual and textual representations, enabling CLIP to determine the semantic similarity between them [69]. During pre-training, CLIP leverages contrastive learning objectives to align similar pairs of image-text embeddings while maximizing the distance between dissimilar pairs, thereby enhancing its ability to understand and relate visual and textual inputs effectively [69].
Video Generation. Advanced Generative AI models have not only demonstrated remarkable capabilities in generating high-quality images but have also begun to tackle the challenge of video generation. Recent advancements in AI, such as Sora developed by OpenAI [75, 76], have enabled the generation of realistic and dynamic video content from textual descriptions. Similar to its image counterpart DALL-E [57], Sora operates at a semantic level, understanding textual prompts and synthesizing corresponding video sequences [75, 76]. Video generation involves creating coherent and visually appealing sequences of frames that align with the provided textual instructions [76]. These models typically employ architectures designed to capture temporal dependencies (i.e., relationships between frames over time) and spatial relationships (i.e., relationships between objects within a single frame). By understanding the semantic context of the text, these models generate videos that accurately reflect described scenes while exhibiting smooth transitions and realistic motion. In addition to video generation, as explained above, AI models are capable of multimodal generation, where textual prompts can result in the synthesis of both images and videos. This capability enhances the quality of generated content, enabling diverse applications in storytelling, content creation, and multimedia production. Video generation has the potential to revolutionize various domains, including the entertainment industry, education and training, augmented reality and virtual reality applications, automation of video editing tasks, and etc.
Text Generation. Advances in Generative AI models can generate human-quality text, including translations, and responses to natural language questions [4, 77]. Text generation models learn patterns and relationships in language from vast amounts of text data [4, 77].
Code Generation. Widely adopted AI tools utilize generative AI techniques to analyze the context of the code being written and suggest relevant code completions that can significantly improve programmers’ and engineers’ productivity by reducing the time spent manually typing codes [6].
Drug Discovery. Generative AI models are increasingly being utilized in various aspects of drug discovery, providing innovative approaches to developing new drugs and accelerating the identification and design of novel therapeutic agents [78, 79]. Furthermore, Generative AI models have demonstrated the capability to identify new applications for drug repurposing [80].
Material Discovery. Advanced ML and DL techniques, particularly Generative models, are being employed to explore and predict novel materials with desirable properties [81]. The application of Generative AI models in material science [82] can significantly accelerate the material discovery process by guiding experimental efforts, predicting new materials, and optimizing existing materials [83].
Fraud Detection. Generative AI models have proven effective in detecting fraud by identifying patterns indicative of fraudulent activity [84]. Furthermore, these models can also be employed in identifying anomalies in data [85, 86].
Personalization. Generative AI models can be used in personalization to tailor content, recommendations, or user experiences based on individual preferences [87, 88]. This customization can involve generating personalized recommendations or creating personalized user experiences. For example, Netflix uses Generative AI to recommend movies and TV shows to its users, while Spotify leverages Generative AI to create custom playlists.