Vmware, Inc.

Apple Inc.

Oleg Zaydman

Preeti Kota

Clay Maeckel

Christopher L Crim

Hui Wang

Stephen Iremonger

Galt Johnson

Jeffrey W Sheldon

Derek William Beard

Arunachalam Ramanathan

Li Zheng

Yanping Cao

Pavan Narasimhaprasad

Ashok Srinivasan

Anupama Chandwani

Pratik Kapadia

Lyndley Crumly

Bhavana Raj Kanthur Murnad

Roman Zhirin

Techniques for implementing application-assisted VM provisioning operations, and in particular application-assisted live migration, are provided. In one set of embodiments, a hypervisor of a source host system can notify a guest application that the VM within which the guest application runs will be imminently live migrated from the source host system to a destination host system, prior to actually carrying out the live migration. In response, the guest application can execute one or more remedial actions that mitigate or avoid issues which may arise with respect to its runtime operation when the VM is stunned and switched over to the destination host system.

Application-assisted live migration

Techniques for interactive code editing are described. A system can provide for display a code editing environment that resembles a text editor. Upon detecting various user inputs, the system can display, in place of text, widgets in the code editing environment. The widgets can have the appearance of text, and have functions to interact with the user to provide various conveniences including, for example, line management, step completion, calculation completion, parameter management, and code folding.

Interactive code editing

The disclosure provides for a reactive cache coherence protocol that has efficiencies over proactive approaches. Rather than proactively performing remediation when a data item is invalidated, a destination endpoint checks cache coherence upon receiving an indication of a cache hit, and based at least on detecting a lack of coherence, performs a reactive remediation process. For example, the incoherence may be fixed by replacing, as a cached data item, a data block indicated by the cache hit with a replacement data block that triggered the cache hit.

Coherence protocol for distributed caches

Virtual-machine images (VMIs) can be compressed by identifying common cluster sequences shared across VMIs. To identify these sequences, hashes are generated for each cluster in each VMI, resulting in hash files for respective VMIs. The hashes are partitioned to address memory constraints. For each partition, its hashes are entered into buckets of a hash map according to their respective hash values. Each (non-empty) bucket associates a key hash value with one or more pointers to locations in the hash files. Clusters of hashes are fetched from the hash files referenced by multi-pointer buckets. The hash clusters are scanned across clusters to identify common hash sequences. Common cluster sequences are then identified based on the common hash sequences. This process works with files other than VMIs and with segment sizes other than clusters.

Identifying common file-segment sequences

A synchronization process for virtual-machine images (and other segmented files) provides for generating a 'delta' bitmap indicating which segments (e.g., clusters) of a first virtual-machine image were changed to obtain a second (e.g., updated) virtual-machine image on a source node. The delta bitmap can be applied to the second-virtual-machine image to generate a delta file. The delta file can be sent along with the delta bitmap to a target node that already has a copy of the first virtual-machine image. The transferred delta bitmap and delta file can then be used on the target node to generate a replica of the second virtual-machine image, thereby effecting synchronization. In variations, different bitmaps and delta files can be transferred to optimize the synchronization process.

Virtual machine image file synchronization using delta bitmaps and delta files

A cuckoo cache has plural buckets of plural cells each. The cells within a bucket are ranked to approximate relative usage recency. New items can be inserted into empty cells; when a bucket is full, room for a new item can be made by laterally transferring an older item to an alternative bucket. When empty cells and lateral transfers are unavailable, an item is selected for eviction based on the usage recency rank of the containing cell. When a match is found, depending on the embodiment, the hit item can be promoted within its bucket, to its alternative bucket, or to a separate tier of the cuckoo cache. The items can be key-value pairs. No metadata is required to track usage recency so that the cuckoo cache can be a very space efficient tool for finding cached values by their keys.

Cuckoo caching

The subject matter described herein analyzes an item that is a candidate for admission into or eviction from a cache to determine characteristics of the item. The characteristics include at least one of item content and item source. A score associated with the item is calculated based on the determined characteristics. The item is admitted into, or evicted from, the cache based on the calculated score.

Score-based cache admission and eviction

A computer system performs a technique for reducing memory usage when a key-value store is being implemented. A first key associated with data is received. A block address of a block of keys is obtained from memory. The block of keys is stored on disk storage, and the keys in the block of keys correspond to respective values stored on the disk storage. The block of keys is obtained from the disk storage using the block address. A second key in the block of keys is located. Locating the second key includes determining that the second key matches the first key. A value of the respective values is obtained using the second key.

Memory efficient key-value store

The disclosure enables management of block storage more efficiently than with traditional free space bitmap approaches. An exemplary method includes segregating disk gap indices by differentiated gap sizes; maintaining a set of lists for segregated sizes, such that each list identifies gaps of a common size; comparing a length of a list with trigger criteria, and based at least on the length of the list meeting the criteria, writing at least a portion of the list into a disk gap. Writing gap locations into gaps in disk storage reduces memory burdens, and the gap data can later be extracted when the list becomes short. These processes can be performed iteratively. The prior need for traversing a free space bitmap to find a gap of a particular size is eliminated; the new method permits more rapid location of a particular size gap by selecting an element of the proper list.

Gap management in block storage

A 'grain' is a unit of space allocation in a virtual disk. Grains are represented in physical storage only when used, that is, when they contain data. Grains may be located in storage using an in-memory grain-range table. The grain-range table is derived from a grain table in storage. A grain-range table includes entries for the starting grains of incrementing and or decrementing ranges; grain-table entries that do not start a range are omitted in the grain-range table. Accordingly, a grain-range table can serve as a compressed form of a grain table. This compression makes it feasible to store large numbers of grain-range tables in memory in cases where it would not be feasible to store the corresponding grain tables in memory. As a result, one rather than two storage accesses are require per storage access request, resulting in a substantial performance improvement.

Locating grains in storage using grain table to grain-range table compression

Files, e.g., disk-image files can be teleported from a source node of a network to a target node in that a copy of file can be assembled at least in part using file parts found on the target node. Source hashes can be generated based on segments of the source file. The source hashes can be sent by the source node and received by the target node. The target node compares each source hash with target hashes of segments of files on the target node. When a comparison results in a match, the file copy can include a copy of the matching target segment or include a reference to the matching segment. For higher performance, fingerprints of the source hash and the target hashes can be compared, with hash comparisons being performed in the event of a fingerprint match. The target fingerprints can be arranged in a cuckoo filter or other probabilistic filter.

Teleporting content over network using hash matches

Deduplication of virtual-machine disk images and other disk images can involve identifying the first clusters in a file. The clusters are hashed. The first-in-file hashes (generated from first-in-file clusters) are stored in an in-memory index, while the full set of hashes is streamed in order to find matches with the hashes stored in the in-memory index. First-in-file hashes in the stream are compared, while other hashes in the stream are compared only if the immediately preceding hash resulted in a match. Comparing non-first-in-file hashes requires disk accesses, but since such comparisons are conditioned on first-in-file matches, there are relatively likely to result in sequences of matches. The net effect is a relatively fast deduplication with compression approaching that resulting from a full comparison of all hashes.

Growing community of inventors

San Jose, CA, United States of America

Oleg Zaydman