Generative deep computer vision models

OpenPhenom-S/16 is the first in a potential series of OpenPhenom foundation models for external use (both non-commercial and commercial). OpenPhenom-S/16 is hosted in Google Cloud’s Vertex AI Model Garden and on Hugging Face, and is only available for non-commercial use. 

OpenPhenom-S/16 flexibly processes microscopy images into general-purpose embeddings. In other words, OpenPhenom-S/16 can take a series of microscopy channels and create a meaningful vector representation of the input image. This enables robust comparison of images, and other data science techniques to decode any biology or chemistry within such images. 

This enables scientists to systematically relate genetic and chemical perturbations to one another in a high-dimensional space, helping determine critical mechanistic pathways and identify potential targets and drugs.

What is OpenPhenom?

OpenPhenom-S/16 is a channel-agnostic vision transformer (ViT) model that can be used on a variable number of imaging channels in any order, enabling it to be applied to various microscopy protocols. Use-cases for this model could involve transfer-learning to other fluorescent microscopy datasets with various numbers of channels, or even histology images, which have entirely different channel characteristics than the phenomics (RxRx3) and Cell Painting (JUMP-CP) assays on which the model was trained. Embeddings computed by OpenPhenom-S/16 can be used as features for downstream modeling including classification tasks or to directly compare similarities between images and the perturbations they represent using metrics like cosine similarity.

In the visual above, observe that the model was trained on 6-channel Cell Painting images, but is flexibly being applied to perform inference on a 3-channel image. The model is agnostic to the order of the channels; i.e., the embeddings will be the same regardless of if you pass channels ordered as [0,1, 2] vs [2,1,0].

Image reconstruction is the pretraining task used to train the ViT. Below are illustrative examples of what these ViTs are capable of. ViT-L/8 (RIP-95M) is trained with ~2x the amount of data and over 3x the number of parameters. Note how the texture in the new reconstructions are much more realistic making the input block artifacts far less noticeable.

Model specs - ViT-S/16 backbone

OpenPhenom-S/16 is a small Vision Transformer (ViT-S) with 16x16 patching (/16). It has roughly 22 million parameters and always produces 384-dimensional embeddings as the output (larger ViTs make larger embeddings).

Regardless of the number of channels, C, OpenPhenom-S/16 will make a single 384 dimensional embedding for a single C x 256 x 256 input crop. Optionally, OpenPhenom-S/16 can return a 384 dimensional embedding for every channel in the input image, to give researchers more control over how they use the generated embeddings. This is by design, as more channels allows more context for the model to produce a single joint representation of the input image.

Example use-case: Recapitulating known biological relationship in public Cell Painting gene knock out screens

A valuable use-case of large-scale HCS experiments is to perform large-scale inference of biological relationships between genetic perturbations. To benchmark Phenom models on this task we evaluate each model’s ability to recall known relationships by using the biological relationship recall benchmark described in Celik et al. (2024). First, we correct for batch effects using Typical Variation Normalization (TVN) or PCA + center-scale, and also correct for possible chromosome arm biases known to exist in CRISPR-Cas9 HCS data (Lazar et al., 2023). To infer biological relationships, we compute the aggregate embedding of each perturbation by taking the spherical mean over its replicate embeddings across experiments. We use the cosine similarity of a pair of perturbation representations as the relationship metric, setting the origin of the space to the mean of negative controls. We compare these similarities with the relationships found in the following public databases: CORUM, hu.MAP, Reactome, and StringDB.

We use the code base from Celik et al. (2024) to evaluate OpenPhenom-S/16, Phenom-1, Phenom-2, and other baselines on the biological relationship recall benchmark using both RxRx3-core and JUMP-CP gene knockout datasets.

Additionally, we evaluate the ability of cosine similarities between embeddings to predict compound-gene interactions in a zero-shot context. We investigate whether compounds' embeddings are more similar to their known target genes than to other genes in the dataset. We compute cosine similarities between each compound and all genes across various concentrations, selecting the maximum similarity for each pair. This captures the strongest interaction signal, even if the highest similarity occurs at different concentrations for different pairs, and allows negatives to be drawn from concentrations different from positives. We consider the absolute cosine similarity as a proxy for the model's confidence and calculate AUC and average precision metrics for each compound. The final benchmark results present the median AUC and average precision over all compounds.