U.S. Philips Corporation

Koninklijke Philips Corporation N.V.

Richard D Gallery

Dale R Heron

Timothy I P Trew

Jain K Shah

Octavius John Morris

David E Penna

Timothy I Trew

Edward S Eilley

Richard J Allen

Nathan D Bliss

An image encoder or other image processing apparatus includes a segmentation apparatus for assigning image pixels to regions, in accordance with predetermined criteria. The segmentation apparatus provides a temporally consistent segmentation throughout a motion picture sequence, and comprises: a motion compensation processor for predicting the segmentation for a current frame (F ) upon the basis of the segmentation (S ) of the preceding frame and estimated motion vectors (M); a pixel identifier and reallocator for modifying the predicted segmentation in dependence upon the accuracy of pixel values (F &prime;) predicted for the current frame on the basis of the motion vectors (M). In particular, for portions of the image where pixel values are not accurately predicted by the motion vectors, it is assumed that the predicted segmentation is invalid, and such portions are re-segmented and merged into the predicted segmentation to obtain the temporally consistent segmentation. In the embodiments, pixels on the borders of regions in the predicted segmentation are also considered and re-assigned to neighbouring regions according to various criteria.

Image segmentation

A virtual body modelling apparatus and method models a users body in a virtual environment and animated to follow physical movements of the user. Stores (,) hold data defining the virtual environment as well as features of the virtual body representation, including sequences of virtual body motions (for example a walking sequence for the legs of the virtual body) which sequences are initiated in response to one or more predetermined physical movements by the user, and animated by a display generator (,) providing images of the virtual environment to the user. Various forms of feedback to the user are provided, including force feedback where the users physical movements are constrained in dependence on conditions within the virtual environment, visual feedback where the users viewpoint of the virtual world is modified as a sequence of body motions is executed, and audio feedback (,) such as the sound of footsteps triggered at predetermined points in the sequence of motions.

Virtual body control device

An image encoder or other image processing apparatus includes a segmentation apparatus for assigning image pixels to regions, in accordance with predetermined criteria. The segmentation apparatus provides a temporally consistent segmentation throughout a motion picture sequence, and comprises: a motion compensation processor for predicting the segmentation for a current frame (F,) upon the basis of the segmentation (S,) of the preceding frame and estimated motion vectors (M); and a pixel identifier and reallocator for modifying the predicted segmentation in dependence upon the accuracy of pixel values (F,&prime;) predicted for the current frame on the basis of the motion vectors (M). In particular, for portions of the image where pixel values are not accurately predicted by the motion vectors, it is assumed that the predicted segmentation is invalid, and such portions are re-segmented and merged into the predicted segmentation to obtain the temporally consistent segmentation. In the embodiments, pixels on the borders of regions in the predicted segmentation are also considered and re-assigned to neighboring regions according to various criteria.

A method and apparatus are provided for providing user-directed operation of a virtual manipulator, such as a simulated hand or claw (30) within a computer-generated virtual environment. The environment comprises one or more objects (32) which may be held, released or impelled by the manipulator under user control. Movement of the virtual manipulator is directed by a suitable hand held device operated by a user, with a simple control directing opening or closing of the claw (30). In operation, actions such as throwing, catching or punching an object (32) are partially automated, with the decision as to the action initiated being determined by the separation (D) between object and manipulator, and the current and previous states of each as recorded in the form of respective state variables. A method for the derivation of an applied velocity for the object acted upon is also described.

Virtual environment manipulation device modelling and control

A method and apparatus are described for controlling a computer-generated animation suite (32), where the suite generates sequences of image frames from stored libraries of image components and component motion parameters. An interface (42) includes a store (44) of function specifications each of which, when called and applied to the animation suite, initiates one or a predetermined sequence of image component movements. A rule-based processor (10) comprises means for modelling an asynchronous logic circuit as a plurality of circuit elements the functions of which are governed by a stored set (14,22) of rules each defining a response to a given condition with the processor generating a response when the associated condition is satisfied. Predetermined conjunctions of image frame features are specified as the condition portion of some of the rules, and the calling of each of the interface function specifications is the response portion of a respective rule. A method and apparatus for the control of a motion simulation suite are also described.

Animation control apparatus

A method and apparatus are described for post-processing a computer-generated image of a three-dimensional object for display as a pixel image. A focus depth is specified either by a user or from within the system for each image frame. The image pixels are grouped into non-overlapping blocks of adjoining image pixels with a single respective depth value pre-specified or calculated at run-time for each block. Each block is extracted in turn from the image and the pixels within a block are convolution filtered with a common set of filter coefficients. Padding may be applied to fill areas where a full filter kernel cannot be formed. The technique has particular suitability for MPEG-coded image streams where the existing MPEG macroblock specification may be taken advantage of.

Post-processing generation of focus/defocus effects for computer

An interactive entertainment apparatus is provided having means (10,14) for modelling a virtual environment populated by modelled characters, with each of the characters being controlled by respective rule-based agents. A camera control function (58) within the apparatus processor periodically monitors at least one compiled behavior per character agent, together with the respective locations within the virtual environment for each of the characters. The processor (10) generates clusters of adjacent characters within the virtual environment in accordance with predetermined clustering criteria such as relatively proximity and commonality of behavioral characteristics, and generates a respective cluster value derived from the current settings of the monitored behaviors within that cluster. In operation, one cluster is selected in dependence on the derived cluster values and, following determination of the location of that cluster within the virtual environment, the viewpoint from which the image of the virtual environment is rendered is shifted to a further viewpoint overlooking the selected cluster.

Virtual environment navigation

In an image source for multimedia applications such as networked computer games, a graphics engine (16) generates pixel images one line at a time using a scan-line algorithm and supplies the generated scan lines to an encoder (18) where they are buffered. The encoder codes the buffered pixel data as macroblocks of, for example 16.times.16 pixels according to MPEG or similar standards. When the graphics engine has sent sufficient scan lines for a first macroblock to the encoder, it sends a signal (FLAG) on receipt of which the encoder begins coding the pixel data as a macroblock in a line of macroblocks whilst continuing to receive scan lines from the graphics engine (16). To increase encoder efficiency, the graphics engine specifies to the encoder global (GMV) and macroblock (BMV) motion vectors for substantially all or selected ones of the macroblocks of an image respectively. Completed coded macroblock lines may be sent to a remote user over a data network, with user commands for modifing the composition of subsequent image frames being received on a back-channel (26) of the network.

Graphics image manipulation

An apparatus is provided for modelling and interacting with a virtual environment. An image of the environment, and a representation of the user's presence therein, is generated by a CPU (12) based on data from memories (14,16) and supplied to the user by a stereoscopic display (10). The user is provided with a hand-held unit (18) which has a single control button (20). When the button (20) is not pressed, the user's movements of the control (18) are replicated by a modelled icon (such as a virtual hand) within the virtual environment to enable interaction. When the button (20) is actuated, the user's movements of control device (18) are instead converted to navigational commands shifting the location of the user's virtual presence around the virtual environment.

Virtual body modeling apparatus having dual-mode motion processing

A feedback means is provided in conjunction with a cursor control mechanism for a two- or three-dimensional graphic user interface. Feedback levels are determined by tonal variation (grey scale or color component) at the pixel (P) currently indicated by the cursor. Suitably, the levels are set by generating differentials across a patch of pixels (A) surrounding the indicated pixel, with differential generation in mutually perpendicular directions (X and Y) providing for independent feedback in two or three directions. The technique enables interface features such as window boundaries (36) or textured patches to be identified to the user without specially authored software being required for those features to trigger feedback when indicated.

Cursor control with user feedback mechanism

A collision warning system is for a head mounted display (HMD) (10) worn by a user, for example as the interface to a virtual reality system, to warn of nearby objects in the physical environment. A motion detector (30,32) detects positional changes of the HMD, which changes are reflected in changes of a displayed image viewpoint, and a comparator stage (38,40) determines whether the movement takes the user into a 'prohibited' area. If so, a visual and/or audible warning is provided to the user via the HMD (10). In an embodiment, the display screens (18) of the HMD include liquid crystal display shutters (20) which are switched to a transparent state in the event of an imminent collision such that the user is not only warned of the danger but also able to see it without having to remove or adjust the HMD.

Display headset

A face classification system which receives video data representative of a scene and extracts therefrom data representative of a face which is present in the scene. The facial video data is applied to a homomorphic filter to eliminate the effects of lighting changes in the scene. The positions of the eyes and mouth are located to form two sub-divisions, one in relation to a first line joining the eyes and the other in relation to a second line perpendicular to the first line and passing through the nose. A feature vector which is representative of the face and is rotation, scaling, translation and grey level intensity invariant is then produced by applying a recursive second order sub-division of moments to the filtered data. Such sub-division of moments is constrained to act first on the sub-divisions formed by the aforesaid lines, thereby reducing the effect of noise. The feature vector may be applied to a classifier which employs multi-layer perception to compare it with stored feature vectors of faces in one or more previous scenes to determine if the faces match. The system may be used, for example, for allowing access to secure areas or to secure computer work stations.

Growing community of inventors

Horley, United Kingdom

Richard D Gallery