International Business Machines Corporation

International Business Machines Corporiation

Alexander Zadorojniy

Sergey Zeltyn

Segev Eliezer Wasserkrug

Nir Mashkif

Lior Limonad

Asaf Adi

David Breitgand

Michael Masin

Ari Volcoff

Alexey Tsitkin

Yishai Abraham Feldman

Oded Dubovsky

Eliezer Segev Wasserkrug

Vladimir Lipets

Lan Ngoc Hoang

Sergey Voldman

Takayuki Osogami

Michael Sambur

A method for improving a machine operation are provided. The method includes receiving a plurality of domain specific heuristics and a set of states and a set of actions, where an immediate cost and/or reward is associated with a pair of state and action. The method also includes generating at least one of: a graph of state transitions for the actions, and a transition probability matrix. The method also includes executing a Markov Decision Process (MDP) model for solving an MDP problem, and outputting an MDP optimal policy of an optimal mapping of a given state to an action. The method also includes selecting one of the plurality of domain specific heuristics and heuristic input parameters thereof. The method also includes controlling the machine for solving a predefined optimization problem in a plurality of execution iterations.

Optimizing a machine for solving offline optimization problems

A control system, computer program product, and method for generating a logically-represented policy for a control system operating based on a CMDP model are provided. The control system directs the operation of a controlled application system that is subject to a constraint. The method includes receiving, at the control system, data corresponding to control action variables and system state variables relating to the controlled application system, data corresponding to a cost/reward, and data corresponding to the constraint, and automatically training a CMDP model for the operation of the controlled application system based on the received data, where the CMDP model is formulated using dual linear programming, and where the CMDP model includes a policy corresponding to occupation measures that are decision variables of the dual linear programming formulation. The method also includes automatically generating a logically-represented policy for the control system based on the policy of the CMDP model.

Generating logically-represented policy for control systems operating based on constrained Markov decision process (CMDP) models

A computer-implemented method comprising: receiving data associated with an operational control problem; formulating the operation control problem as an optimization problem; recursively generating a sequence of policies of operational control associated with the operational control problem, wherein each subsequent policy in the sequence is constructed by modifying one or more actions at a single state in a preceding policy in the sequence, and wherein the modifying monotonically changes a risk associated with the subsequent policy; constructing, from the sequence of policies, an optimal solution path, wherein each vertex on the optimal solution path represents an optimal solution to the operational control problem; calculating a ratio of reward to risk for each of the vertices on the path; and selecting one of the policies in the sequence to apply to the operational control problem, based, at least in part, on the calculated ratios.

Reward to risk ratio maximization in operational control problems

Method, apparatus and product for utilizing stochastic controller to provide user-controlled notification rate of wearable-based events. The method comprises obtaining events issued by a module based on analysis of multiple sensor readings of one or more sensors of a wearable device. The method further comprises determining by a stochastic controller whether to provide an alert to a user based on the events and based on a user preference, wherein the user preference is indicative of a desired notification rate of the user, wherein the stochastic controller comprises a stochastic model of an environment. Based on such determination, alerts are outputted to the user.

Utilizing stochastic controller to provide user-controlled notification rate of wearable-based events

A method, apparatus and a product for generating a dataset for a reinforcement model. The method comprises obtaining a plurality of different subsets of the set of features; for each subset of features, determining a policy using a Markov Decision Process; obtaining a state comprises a valuation of each feature of the set of features; applying the plurality of policies on the state, whereby obtaining a plurality of suggested actions for the state, based on different projections of the state onto different subsets of features; determining, for the state, one or more actions and corresponding scores thereof based on the plurality of suggested actions; and training a reinforcement learning model using the state and the one or more actions and corresponding scores thereof.

States simulator for reinforcement learning models

A method for automatically reducing the dimensionality of a mathematical representation of a controlled application system is provided. The method includes receiving, at a control system, data corresponding to control action and system state variables relating to the controlled application system, fitting a constrained reinforcement learning (CRL) model to the controlled application system based on the data, and automatically identifying a subset of the system state variables by selecting control action variables of interest and identifying system state variables that drive the CRL model to recommend each control action variable of interest. The method also includes automatically performing state space dimensionality reduction of the CRL model using the subset of system state variables, estimating a transition probability matrix for a constrained Markov decision process (CMDP) model of the controlled application system, and formulating the CMDP model as a linear programming (LP) problem using the transition probability matrix and several costs.

Optimizing control actions of a control system via automatic dimensionality reduction of a mathematical representation of the control system

Constructing a MARS prediction model using predictor variables at a first point in time within a time horizon, including directly-controllable variables of first physical characteristics of a system and that are associated with adjustable operational control settings for directly controlling the first physical characteristic, and including controllable variables of second physical characteristics that are affected by the first physical characteristics, recursively using the prediction model to define an optimization problem for later point in time within the time horizon, transforming the optimization problem into a MILP problem, and solving the MILP problem using an optimization engine to determine, for any given one of the directly-controllable variables and corresponding to at least one of the points in time, for adjusting, using the optimized value, the adjustable operational control setting corresponding to the given directly-controllable variable and thereby control the physical characteristic associated with the directly-controllable variable.

Control system with optimized periodic adjustments of system control settings using MARS-based MILP optimization

In some examples, a system for generating personalized recommendation includes a processor that can perform an initial training for a deep reinforcement learning (DRL) model using domain knowledge, available users data, and an items list. The processor also inputs users data and an items list to the trained DRL model to generate an initial list of recommended items. The processor also inputs the initial list of recommended items and a user profile to a content-based filter to generate a final list of recommendations for a target user.

Personalized recommender with limited data availability

A computer implemented method of detecting complex user activities, comprising using processor(s) in each of a plurality of consecutive time intervals for: obtaining sensory data from wearable inertial sensor(s) worn by a user, computing an action score for continuous physical action(s) performed by the user, the continuous physical action(s) extending over multiple time intervals are indicated by repetitive motion pattern(s) identified by analyzing the sensory data, computing a gesture score for brief gesture(s) performed by the user, the brief gesture(s) bounded in a single basic time interval is identified by analyzing the sensory data, aggregating the action and gesture scores to produce an interval activity score of predefined activity(s) for a current time interval, adding the interval activity score to a cumulative activity score accumulated during a predefined number of preceding time intervals and identifying the predefined activity(s) when the cumulative activity score exceeds a predefined threshold.

Detecting complex user activities using ensemble machine learning over inertial sensors data

Embodiments of the present invention may provide the capability to detect complex events while providing improved detection and performance. In an embodiment of the present invention, a method for detecting an event may comprise receiving data representing measurement or detection of physical parameters, conditions, or actions, quantizing the received data and selecting a number of samples from the quantized data, generating a hidden Markov model representing events to be detected using initial model values based on ideal conditions, wherein a desired output is defined as a sequence of states, and wherein a number of states of the hidden Markov model is less than or equal to the number of samples of the quantized data, adjusting the quantized data and the initial model values to improve accuracy of the model, determining a state sequence of the hidden Markov model, and outputting an indication of a detected event.

Method for complex events detection using hidden markov models

According to some embodiments of the present invention there is provided a method for determining a control action in a control system using a Markov decision process. The method comprises an action of receiving two or more predefined transition probability values of a Markov decision process (MDP) of a control system, where each of the predefined transition probability values is associated with a transition between a first state and a second state, both from two or more system states, resulting from execution of one or more control actions of the control system. The method comprises an action of computing one or more new transition probability values by an analysis of the predefined transition probability values, the system states and the control actions. The method comprises an action of determining one or more recommended control actions for the respective system state based on the new transition probability value.

Interpolation of transition probability values in Markov decision processes

According to some embodiments of the present invention there is provided a method for determining a control action in a control system using a Markov decision process. The method comprises an action of receiving measured transition probability values of a Markov decision process (MDP) and receiving simulated transition probability values generated by performing a control system simulation. New transition probability values are computed by calculating a measured data count of some of the sensor measurements and a simulated data count of some of the simulated transition data. New transition probability values are computed from a weighted average between the measured transition probability values and the simulated transition probability values using the measured data count and the simulated data count. A new control action is determined based on the one or more new transition probability value.

Hybrid estimation of transition probability values in markov decision processes

A computer-implemented method, computerized apparatus and computer program product, the method comprising: obtaining data measured by one or more sensors; segmenting the data into a plurality of sliding windows; extracting one or more features from each of the plurality of sliding windows; analyzing, by a machine learning process, the extracted features to determine, for each sliding window, an activity detection in the sliding window; and determining an activity detection result in the data to be positive responsive to activity detection by the machine learning process in at least a number M of sliding windows out of a number N of consecutive sliding windows, wherein M&#x3e;1.

Optimizing performance of event detection by sensor data analytics

A computer implemented method, a computerized system and a computer program product for anomaly classification. The computer implemented method comprises obtaining a data set, wherein the data set comprises a plurality of data points. The method further comprises filtering the data set based on an absolute distance criterion and performing anomaly classification on a test data point of the data set, wherein the anomaly classification is based on a relative density criterion. The method further comprises outputting an outcome of the anomaly classification.

Hybrid method for anomaly classification

A method for wastewater treatment that comprises receiving influent readings from sensors located along influent stream(s) of a wastewater treatment unit, effluent readings from sensors located along effluent stream(s) of the wastewater treatment unit, a feedback flow variable calculated according to a state of a feedback flow channel between an effluent output and an influent input, analyzing the influent readings and the effluent readings to extract an influent flow variable, a total nitrogen at effluent variable and a total phosphorus at effluent variable, and calculating control instructions to control the wastewater treatment unit by assigning a combination of a cost variable reflecting a treatment cost for treating the influent stream(s), a time period, the influent flow variable, the total nitrogen at effluent variable, the total phosphorus at effluent variable, and the feedback flow variable in a state space of the wastewater treatment unit.

Growing community of inventors

Haifa, Israel

Alexander Zadorojniy