University of California

Leland Stanford Junior University

Aydogan Ozcan

Gordon S Kino

Michel J F Digonnet

Yair Rivenson

Yibo Zhang

Yichen Wu

Dino Di Carlo

Hongda Wang

Deniz Mengu

Wei Luo

Yi Luo

Steve Wei Feng

Omai Garner

Waheb Bishara

Derek Tseng

Hatice Ceylan Koydemir

Alon Greenbaum

Zhensong Wei

Ting-Wei Su

Harun Gunaydin

Xing Lin

Kevin De Haan

Alborz Feizi

Michel Jf Digonnet

Zoltan Gorocs

Bingen Cortazar

Michel J F Diggonet

Gordon S Kingo

Onur Mudanyali

Aniruddha Ray

Stoyan Dimitrov

Uzair Sikora

Serhan Omer Isikman

Swati Padmanabhan

Isa Navrus

Luzhe Huang

Qingshan Wei

Zachary S Ballard

Bahram Jalali

Michel J Digonnet

Henry T K Tse

Euan McLeod

Soojung C Hur

Meng Dai Yu

Michael J F Digonnet

Yuye Ling

Artem Goncharov

Calvin Brown

Hanlong Chen

Brandon Berg

Ahmet F Coskun

Ikbal Sencan

Hongying Zhu

Chetin Oztoprak

Anthony F Erlinger

Alon Grinbaum

Hector E Munoz

Carson Riche

Hyouarm Joung

Isidro B Salusky

Haydar Ozkan

Muhammed Veli

Jingxi Li

Yilin Luo

Yunzhe Qiu

Yijie Zhang

Bijie Bai

Hyou-Arm Joung

Michel JF Digonnet

Isa Navruz

Michael JF Digonnet

Michel F Digonnet

A deep learning framework, termed Fourier Imager Network (FIN) is disclosed that can perform end-to-end phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting success in external generalization. The FIN architecture is based on spatial Fourier transform modules with the deep neural network that process the spatial frequencies of its inputs using learnable filters and a global receptive field. FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ˜0.04 s per 1 mmof the sample area. Beyond holographic microscopy and quantitative phase imaging applications, FIN and the underlying neural network architecture may open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.

Deep neural network for hologram reconstruction with superior external generalization

A diffractive network is disclosed that utilizes, in some embodiments, diffractive elements, which are used to shape an arbitrary broadband pulse into a desired optical waveform, forming a compact and passive pulse engineering system. The diffractive network was experimentally shown to generate various different pulses by designing passive diffractive layers that collectively engineer the temporal waveform of an input terahertz pulse. The results constitute the first demonstration of direct pulse shaping in terahertz spectrum, where the amplitude and phase of the input wavelengths are independently controlled through a passive diffractive device, without the need for an external pump. Furthermore, a modular physical transfer learning approach is presented to illustrate pulse-width tunability by replacing part of an existing diffractive network with newly trained diffractive layers, demonstrating its modularity. This learning-based diffractive pulse engineering framework can find broad applications in e.g., communications, ultra-fast imaging and spectroscopy.

Pulse shaping using diffractive network device with modular diffractive layers

A diffractive optical neural network device includes a plurality of diffractive substrate layers arranged in an optical path. The substrate layers are formed with physical features across surfaces thereof that collectively define a trained mapping function between an optical input and an optical output. A plurality of groups of optical sensors are configured to sense and detect the optical output, wherein each group of optical sensors has at least one optical sensor configured to capture a positive signal from the optical output and at least one optical sensor configured to capture a negative signal from the optical output. Circuitry and/or computer software receives signals or data from the optical sensors and identifies a group of optical sensors in which a normalized differential signal calculated from the positive and negative optical sensors within each group is the largest or the smallest of among all the groups.

Diffractive deep neural networks with differential and class-specific detection

A deep learning-based spectral analysis device and method are disclosed that employs a spectral encoder chip containing a plurality of nanohole array tiles, each with a unique geometry and, thus, a unique optical transmission spectrum. Illumination impinges upon the encoder chip and a CMOS image sensor captures the transmitted light, without any lenses, gratings, or other optical components. A spectral reconstruction neural network uses the transmitted intensities from the image to faithfully reconstruct the input spectrum. In one embodiment that used a spectral encoder chip with 252 nanohole array tiles, the network was trained on 50,352 spectra randomly generated by a supercontinuum laser and blindly tested on 14,648 unseen spectra. The system identified 96.86% of spectral peaks, with a peak localization error of 0.19 nm, peak height error of 7.60%, and peak bandwidth error of 0.18 nm.

Device and method for neural-network based on-chip spectroscopy using a plasmonic encoder

A deep learning-based digital staining method and system are disclosed that enables the creation of digitally/virtually-stained microscopic images from label or stain-free samples based on autofluorescence images acquired using a fluorescent microscope. The system and method have particular applicability for the creation of digitally/virtually-stained whole slide images (WSIs) of unlabeled/unstained tissue samples that are analyzes by a histopathologist. The methods bypass the standard histochemical staining process, saving time and cost. This method is based on deep learning, and uses, in one embodiment, a convolutional neural network trained using a generative adversarial network model to transform fluorescence images of an unlabeled sample into an image that is equivalent to the brightfield image of the chemically stained-version of the same sample. This label-free digital staining method eliminates cumbersome and costly histochemical staining procedures and significantly simplifies tissue preparation in pathology and histology fields.

Method and system for digital staining of label-free fluorescence images using deep learning

A deep learning-based digital/virtual staining method and system enables the creation of digitally/virtually-stained microscopic images from label or stain-free samples. In one embodiment, the method of generates digitally/virtually-stained microscope images of label-free or unstained samples using fluorescence lifetime (FLIM) image(s) of the sample(s) using a fluorescence microscope. In another embodiment, a digital/virtual autofocusing method is provided that uses machine learning to generate a microscope image with improved focus using a trained, deep neural network. In another embodiment, a trained deep neural network generates digitally/virtually stained microscopic images of a label-free or unstained sample obtained with a microscope having multiple different stains. The multiple stains in the output image or sub-regions thereof are substantially equivalent to the corresponding microscopic images or image sub-regions of the same sample that has been histochemically stained.

Method and system for digital staining of microscopy images using deep learning

A system for the detection and classification of live microorganisms in a sample includes a light source and an incubator holding one or more sample-containing growth plates. A translation stage moves the image sensor and/or the growth plate(s) along one or more dimensions to capture time-lapse holographic images of microorganisms. Image processing software executed by a computing device captures time-lapse holographic images of the microorganisms or clusters of microorganisms on the one or more growth plates. The image processing software is configured to detect candidate microorganism colonies in reconstructed, time-lapse holographic images based on differential image analysis. The image processing software includes one or more trained deep neural networks that process the time-lapsed image(s) of candidate microorganism colonies to detect true microorganism colonies and/or output a species associated with each true microorganism colony.

Systems and methods for the early detection and classification of live microorganisms using time-lapse coherent imaging and deep learning

A microscopy method includes a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of images comprising co-registered pairs of high-resolution microscopy images or image patches of a sample and their corresponding low-resolution microscopy images or image patches of the same sample. A microscopy input image of a sample to be imaged is input to the trained deep neural network which rapidly outputs an output image of the sample, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast.

Systems and methods for deep learning microscopy

A deep learning-based digital staining method and system are disclosed that provides a label-free approach to create a virtually-stained microscopic images from quantitative phase images (QPI) of label-free samples. The methods bypass the standard histochemical staining process, saving time and cost. This method is based on deep learning, and uses a convolutional neural network trained using a generative adversarial network model to transform QPI images of an unlabeled sample into an image that is equivalent to the brightfield image of the chemically stained-version of the same sample. This label-free digital staining method eliminates cumbersome and costly histochemical staining procedures, and would significantly simplify tissue preparation in pathology and histology fields.

Method and system for digital staining of label-free phase images using deep learning

An all-optical Diffractive Deep Neural Network (DNN) architecture learns to implement various functions or tasks after deep learning-based design of the passive diffractive or reflective substrate layers that work collectively to perform the desired function or task. This architecture was successfully confirmed experimentally by creating 3D-printed DNNs that learned to implement handwritten classifications and lens function at the terahertz spectrum. This all-optical deep learning framework can perform, at the speed of light, various complex functions and tasks that computer-based neural networks can implement, and will find applications in all-optical image analysis, feature detection and object classification, also enabling new camera designs and optical components that can learn to perform unique tasks using DNNs. In alternative embodiments, the all-optical DNN is used as a front-end in conjunction with a trained, digital neural network back-end.

Devices and methods employing optical-based machine learning using diffractive deep neural networks

A computational cytometer operates using magnetically modulated lensless speckle imaging, which introduces oscillatory motion to magnetic bead-conjugated rare cells of interest through a periodic magnetic force and uses lensless time-resolved holographic speckle imaging to rapidly detect the target cells in three-dimensions (3D). Detection specificity is further enhanced through a deep learning-based classifier that is based on a densely connected pseudo-3D convolutional neural network (P3D CNN), which automatically detects rare cells of interest based on their spatio-temporal features under a controlled magnetic force. This compact, cost-effective and high-throughput computational cytometer can be used for rare cell detection and quantification in bodily fluids for a variety of biomedical applications.

Magnetically modulated computational cytometer and methods of use

A trained deep neural network transforms an image of a sample obtained with a holographic microscope to an image that substantially resembles a microscopy image obtained with a microscope having a different microscopy image modality. Examples of different imaging modalities include bright-field, fluorescence, and dark-field. For bright-field applications, deep learning brings bright-field microscopy contrast to holographic images of a sample, bridging the volumetric imaging capability of holography with the speckle-free and artifact-free image contrast of bright-field microscopy. Holographic microscopy images obtained with a holographic microscope are input into a trained deep neural network to perform cross-modality image transformation from a digitally back-propagated hologram corresponding to a particular depth within a sample volume into an image that substantially resembles a microscopy image of the sample obtained at the same particular depth with a microscope having the different microscopy image modality.

System and method for transforming holographic microscopy images to microscopy images of various modalities

A multiplexed vertical flow serodiagnostic testing device for diseases such as Lyme disease includes one or more multi-piece cassettes that include vertical stacks of functionalized porous layers therein. A bottom piece of the cassette includes a sensing membrane with a plurality of spatially multiplexed immunoreaction spots or locations. Top pieces are used to deliver sample and/or buffer solutions along with antibody-conjugated nanoparticles for binding with the immunoreaction spots or locations. A colorimetric signal is generated by the nanoparticles captured on the sensing membrane containing disease-specific antigens. The sensing membrane is imaged by a cost-effective portable reader device. The images captured by the reader device are subject to image processing and analysis to generate positive (+) or negative (−) indication for the sample. A concentration of one or more biomarkers may also be generated. The testing device is rapid, simple, inexpensive, and allows for simultaneous measurement of multiple antibodies and/or antigens making it an ideal point-of-care platform for disease diagnosis.

Serodiagnostic testing device and system for early-stage Lyme disease using a multiplexed immunoassay

A fluorescence microscopy method includes a trained deep neural network. At least one 2D fluorescence microscopy image of a sample is input to the trained deep neural network, wherein the input image(s) is appended with a digital propagation matrix (DPM) that represents, pixel-by-pixel, an axial distance of a user-defined or automatically generated surface within the sample from a plane of the input image. The trained deep neural network outputs fluorescence output image(s) of the sample that is digitally propagated or refocused to the user-defined surface or automatically generated. The method and system cross-connects different imaging modalities, permitting 3D propagation of wide-field fluorescence image(s) to match confocal microscopy images at different sample planes. The method may be used to output a time sequence of images (e.g., time-lapse video) of a 2D or 3D surface within a sample.

Systems and methods for two-dimensional fluorescence wave propagation onto surfaces using deep learning

A deep learning-based volumetric image inference system and method are disclosed that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network (RNN) (referred to herein as Recurrent-MZ), 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over an extended depth-of-field. Using experiments onand nanobead samples, Recurrent-MZ is demonstrated to increase the depth-of-field of a 63×/1.4 NA objective lens by approximately 50-fold, also providing a 30-fold reduction in the number of axial scans required to image the same sample volume. The generalization of this recurrent network for 3D imaging is further demonstrated by showing its resilience to varying imaging conditions, including e.g., different sequences of input images, covering various axial permutations and unknown axial positioning errors.

Volumetric microscopy methods and systems using recurrent neural networks

Growing community of inventors

Los Angeles, CA, United States of America

Aydogan Ozcan