Shopify Inc.

Stephan Leroux

Daniel Beauchamp

Byron Leonel Delgado

Juho Mikko Haapoja

Maas Mansoor Ali Lalani

Jonathan Wade

Bryon Leonel Delgado

Andrew Ngo

Maas Monsoor Ali Lalani

Disclosed herein are methods and systems for building and updating a virtual store based on a physical store. A computing system is structured to receive captured images of a physical space having a set of products. The physical space is simulated by a three-dimensional (3D) model of the physical space. A corresponding representation of the set of products is positioned relative to the 3D model. The computing system is structured to detect, based on the captured images, product information for products in the physical space and generate, based on the product information, an updated representation of the set of products based on the product information. The updated representation of the set of products corresponds to the products in the physical space as shown in the captured images.

Build and update a virtual store based on a physical store

A system and method are provided for generating 2D electronic images of virtual 3D models using imaging parameters determined from 2D images of other objects. The method includes determining, based on one or more 2D images of a first object arranged in a particular manner, a viewpoint capable of generating the one or more 2D images of the first object arranged in the particular manner. The method also includes generating a set of one or more 2D images of a second object, the generating including rendering a virtual 3D model of the second object according to the viewpoint. Related computer-readable media are also disclosed.

System and method for generating 2D images of virtual 3D models according to imaging parameters determined from 2D images of other objects

A computer-implemented is disclosed. The method includes: obtaining video data for a video stream; detecting a first object in at least one video frame of the video stream, the first object being one of a first set of defined objects associated with the video stream; and in response to detecting the first object: identifying the first object as a current target object associated with the video stream; and causing display of the current target object as an object of interest in at least one subsequent video frame of the video stream.

Systems and methods for real-time processing of live media streams

Systems and methods for generating AR content to virtually alter a real-world item are provided. The real-world item may be a deformable item including one or more flexible surfaces. According to an embodiment, a method includes determining the relative positions of a plurality of points on the real-world item. Based on the relative positions of the plurality of points, a 3D model of a virtual item is distorted to produce a distorted 3D model of the virtual item. The distorted 3D model may have substantially the same shape as the real-world item. AR content is then generated based on the distorted three-dimensional model.

Systems and methods for generating augmented reality content based on distorted three-dimensional models

An image processing system accesses an image depicting a particular object. The system identifies a location of the particular object within the image and defines, based on the identified location of the particular object within the image, a focal region of the image. The focal region can be used to modify the image for delivery and presentation on a webpage or software application.

Defining focal regions of images based on image metadata

The present disclosure relates, in part, to spatially aware media that includes three-dimensional (3D) spatial information pertaining to a real-world space. The spatially aware media may map this 3D spatial information to media such as an image, for example, to provide 3D spatial context for the media. This may allow users to more flexibly and efficiently interact with virtual content in real-world spaces that are relevant to them. According to one embodiment, spatially aware media is augmented to provide an image of a real-world space overlaid with a render of a 3D model defined relative to the 3D spatial features of the real-world space. Before augmenting the spatially aware media, a recommended position for the 3D model relative to the 3D spatial features of the real-world space may be determined based on the 3D model and/or on the spatially aware media.

Systems and methods for providing augmented media

A merchant may operate a retail store that users are able to visit in person in order to view and purchase products. When a user visits the store, the user might not know where a desired product ('target product') is located. Computer technology may help direct the user to the target product in real-time. In some embodiments, a model of passable areas and the location of products in the retail store may be determined, e.g. by a merchant device. In some embodiments, when the user visits the retail store, a computer generates a product recommendation, e.g. based on user-specific information, and a route in the retail store is determined for the user. In some embodiments, the route in the retail store may be determined using the model based on the target product, the user's location in the store, and one or more recommended products.

Computer-implemented systems and methods for in-store route recommendations

The present disclosure provides a system and method for object verification. The method comprises obtaining sensor data, from one or more sensors, the sensor data including measurements of one or more physical properties of a given three-dimensional (3D) object, wherein the sensor data comprises a first measurement of the one or more physical properties from a first perspective and a second measurement of the one or more physical properties from a second perspective, the second perspective being different from the first perspective. The method further comprises comparing the sensor data with reference data, the reference data including measurements of one or more corresponding physical properties of a reference 3D object, and generating, based on the comparing, a verification signal indicating a determined match between the given 3D object and the reference 3D object.

3D object verification system and method

Systems and methods are provided for the generation of augmented reality (AR) content that provides a shared AR experience involving multiple users. Shared AR experiences can improve the communication and collaboration between multiple simultaneous users. According to an embodiment, AR content is generated for a first user in a shared AR experience. The AR content includes at least one of a render of a model, a virtual representation of a second user in the shared AR experience, a virtual representation of a user interaction in the shared AR experience, and spatial audio content. Modifications to a shared AR experience are also provided. These modifications may be initiated based on instructions from one user and be reflected in the AR content generated for multiple users.

Systems and methods for representing user interactions in multi-user augmented reality

Generating and storing digital media can be resource intensive processes. Some systems and methods disclosed herein relate to generating digital media using a pre-existing three-dimensional (3D) model of an object and feature points of the object. According to an embodiment, a method includes an e-commerce platform receiving a request for digital media depicting an object. In response to the request, the e-commerce platform may obtain a 3D model corresponding to the object and data pertaining to one or more feature points of the object. The one or more feature points may correspond to respective views of the 3D model. The e-commerce platform may then generate the digital media based on the 3D model and the one or more feature points, where the digital media could include renders of the 3D model depicting the one or more feature points.

Systems and methods for generating digital media based on object feature points

When a person visits a physical retail store, the merchant often does not have enough information about the person to make meaningful product recommendations. Also, a physical retail store typically has products physically distributed throughout the store. It may be desirable to have some sort of relationship between where the customer is and the location of the product being recommended. In some embodiments, when a person visits the store, a computer determines an identity of the person and generates a product recommendation based on user-specific information for that person. In some embodiments, generating the product recommendation includes detecting that a field of view of a camera of the device has changed, and in response determining a plurality of products within or proximate to a current field of view of the camera. At least one of the plurality of products is then identified as the recommended product.

Computer-implemented systems and methods for in-store product recommendations

Systems and methods for generating multi-user augmented reality content

The consistent use of lighting in different instances of digital media may help ensure that objects are depicted in a similar manner in the digital media. However, in some cases, a three-dimensional (3D) model may depict an object under lighting conditions that differ from the lighting conditions depicted in other digital media. The present disclosure provides systems and methods for generating 3D models to include lighting that is consistent with the lighting used in other digital media. According to an embodiment, a lighting template is determined based on digital media depicting a first physical object. A modified 3D model of a second physical object is then generated based on the lighting template to light the second physical object according to the lighting template.

Growing community of inventors

East Gwillimbury, Canada

Stephan Leroux