Facebook, Inc.

Meta Platforms, Inc.

Emanuel Alexandre Strauss

Daniel Olmedilla De La Calle

Enming Luo

Nicolas Scheffer

Joseph Rhyu

Michelle Ruby Hwang

Yang Mu

Muhammad Saif Farooqui

Taiyuan Zhang

Rehman Mehdi Muhammad

Hongda Ma

Zixiao Chang

Chang Huang

Lijun Tang

Jiun-Ren Lin

Abhay Kumar Jha

Yiqiu Liu

Siqi Nie

John Spencer Beecher-Deighan

Katherine Ruolin Yu

Akshita Rajendra Jain

Aswin Gigi Mampilly

Dilan Chaturanga Edirisinghe

Yue Shi

Techniques for scam detection and prevention are described. In one embodiment, an apparatus may comprise an interaction processing component operative to generate a scam message example repository; submit the scam message example repository to a natural-language machine learning component; and receive a scam message model from the natural-language machine learning component in response to submitting the scam message example repository; an interaction monitoring component operative to monitor a plurality of messaging interactions with a messaging system based on the scam message model; and determine a suspected scam messaging interaction of the plurality of messaging interactions; and a scam action component operative to perform a suspected scam messaging action with the messaging system in response to determining the suspected scam messaging interaction. Other embodiments are described and claimed.

Techniques for scam detection and prevention

An online system receives digital content and determines whether the digital content includes malicious content, such as obfuscated text, before presenting the digital content to a user. To determine whether the digital content contains malicious content, the online system renders the digital content. The online system performs optical character recognition on the content. The online system uses an obfuscation machine learning model to identify obfuscated text. The online system may deobfuscate the obfuscated text. The online system may prevent presentation of the digital content in response to detecting obfuscated text.

Identification and deobfuscation of obfuscated text in digital content

A content review system for an online system automatically determines if received content items to be displayed to users violate any policies of the online system. The content review system generates a semantic vector representing the semantic features of a content item, for example, using a neural network. By comparing the semantic vector for the content item with semantic vectors of content items previously determined to violate one or more policies, the content review system determines whether the content item also violates one or more policies. The content review system may also maintain templates corresponding to portions of semantic vectors shared by multiple content items. An analysis of historical content items that conform to the template is performed to determine a probability that received content items that conform to the template violate a policy.

Detecting content items in violation of an online system policy using semantic vectors

An online system maintains machine learning models that determine risk scores for content items indicating likelihoods of content items violating content policies associated with the machine learning models. When the online system obtains an additional content policy, the online system applies a maintained machine learning model to a set including content items previously identified as violating or not violating the additional content policy. The online system maps the risk scores determined for content items of the set to likelihoods of violating the additional content policy based on the identifications of content times in the set violating or not violating the additional content policy. Subsequently, the online system applies the maintained machine learning model to content items and determines likelihoods of the content items violating the additional content policy based on the mapping of risk scores to likelihood of violating the additional content policy.

Evaluating content for compliance with a content policy enforced by an online system using a machine learning model determining compliance with another content policy

An online system enforces policies to content items that are distributed on its platform and blocks content items that violate one or more of those policies. To identify content items that are slightly varied from each other, the online system generates an embedding for each of the known content items that have already been determined to be noncompliant with one or more policies. The online system then groups the known noncompliant content items that are clustered together in the embedding space. The texts of the group of known noncompliant content items are converted to finite state automata and are merged to generate a common automaton. The common automaton is used to generate a common regular expression that is used to screen new content items. When a new content item matches the textual pattern defined by the common regular expression, the system may block the new content item.

Regular expression generation and screening of textual items

Detecting content items in violation of an online system policy using templates based on semantic vectors representing content items

An online system uses a model to detect violations of policies enforced by the online system for content uploaded to the online system by users for viewing by other users. The online system trains the model in multiple stages. To train the model, the online system obtains a set of training content items, with each content item of the set labeled with both a policy violated by the content item and a source of the content item, which acts as a proxy for a sub-category identifying a way in which the content item violated the policy. In the first stage, the online system trains the model using the set of training content items. In a second stage, the model of trained to predict policy violations from content items that are not labeled with a source. For example, the second stage is performed by freezing earlier layers in the model.

Evaluation of content items against policies regulating content presentation by an online system using machine learning

An online system determines a quality of content provided by third party systems for distribution to users. The online system analyzes URL's posted within the online system by content providers to determine the quality of content of the webpages obtained by accessing the URLs. For each URL, the online system receives an original markup language document and a copy of the markup document obtained by applying a content filter. The online system extracts features from both markup language documents. The online system provides the extracted features to a machine learning based model to generate a content quality score. The online system categorizes the URL as having high quality content or low quality content. The online system restricts distribution of content items including URLs to websites with low quality content.

Managing distribution of content items including URLs to external websites

For various content campaigns (or content), an online system predicts a likelihood score of context violations (e.g., account term violations) of a content campaign. The online system derives a plurality of feature vectors of the content campaign. The online system predicts a likelihood score of context violation of the content campaign using a memorization model based on the plurality of feature vectors. The memorization model comprises a plurality of categories and a plurality of items of each category. Each of the plurality of categories has a category weight, and each of the plurality of items of each category has an item weight. The predicted likelihood score is based on a combination of a plurality of category weights and a plurality of item weights associated with the plurality of feature vectors. The online system performs an action affecting the content campaign based in part on the predicted likelihood score.

Memorization model for context violations

An online system determines whether a test content item violates a policy of the online system. The online system extracts a semantic from the test content item and determines a distance between the extracted semantic vector and the stored semantic vectors for content items that have been labeled to indicate whether they violate a policy. Using a nearest neighbor search, the online system selects a set of the stored semantic vectors and assigns a weight to the selected semantic vectors that is inversely related to the distances. The online system then determines whether the test content item violates a policy using a weighed voting scheme, where the labels of the stored semantic vectors are aggregated based on their associated weights. The online system may first attempt to match the test content with known bad content and terminate the more complex nearest neighbor search if such a match is found.

Policy compliance verification using semantic distance and nearest neighbor search of labeled content

A mobile device generates a first representation, based on a model, of a first content received by the mobile device from a website. An online system receives the representation of the content from the mobile device. The online system receives a second content from the website and generates a representation of the second content using the model. The online system compares the representation of the first content with the representation of the second content to determine a distance between the two representations. The distance indicates a level of similarity between the first content and the second content. The online system compares the distance between the representation of the first content and representation of the second content to determine if the distance exceeds a threshold distance. If the distance exceeds the threshold distance, the online system prevents other mobile devices from accessing the website.

Detecting cloaking of websites using content model executing on a mobile device

An online system transmits to a mobile device a feed story that includes a uniform resource locator (URL) of a feed story website. The online system receives from the mobile device a URL log that includes URLs that the mobile device accessed in response to requesting content from the feed story website. The online system extracts a feature of at least one URL in the log, and inputs the extracted feature into a model that was trained, using machine learning, to identify websites that perform cloaking. The model generates a score indicating a likelihood that the feed story website performs cloaking based the extracted feature. The online system compares the score to a threshold to determine whether the feed story website performs cloaking. If the online system determines that the feed story website performs cloaking, the online system limits delivery of content including the URL of the feed story website.

Detecting cloaking of websites using model for analyzing URL redirects

A social networking system validates content items based on policies regarding use of content items. The social networking system balances the validation process of content items with at least two validation queues, each having a priority value and some number of content items. A validation queue is selected based on the priority values. From the selected validation queue, a content item is determined to be valid or invalid. A validation efficiency for each queue is calculated based on the validated content items from each queue. The social networking system dynamically adjusts the priority value of each queue based on the validation efficiencies. The dynamic adjustment may be periodic. The social networking system may withhold invalid content items from users.

Dynamic adjustment in priority values of validation queues

An online system reviews content for violation of one or more policies of the system. The system may evaluate various content policies to determine how usage of the policy affects user experience and what content is shown to each user. The system can, for example, run an A/B validation for certain policies, such as before launching a new policy. To enable the validation, when content is determined to be violating a policy, it is labeled with the specific policy violated as a shadow tag that is not visible to the user viewing the content. Then, the system may track user interactions with newsfeeds of content that include no policy violating content and newsfeeds that include some policy-violating content, and detect at a policy-by-policy level how each policy affects the newsfeed and user experience.

Growing community of inventors

San Mateo, CA, United States of America

Emanuel Alexandre Strauss