Barrington-Gamepoweredmachinelearning-2012

Prévia do material em texto

<p>Game-powered machine learning</p><p>Author(s): Luke Barrington, Douglas Turnbull and Gert Lanckriet</p><p>Source: Proceedings of the National Academy of Sciences of the United States of</p><p>America , April 24, 2012, Vol. 109, No. 17 (April 24, 2012), pp. 6411-6416</p><p>Published by: National Academy of Sciences</p><p>Stable URL: https://www.jstor.org/stable/41588543</p><p>JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide</p><p>range of content in a trusted digital archive. We use information technology and tools to increase productivity and</p><p>facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.</p><p>Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at</p><p>https://about.jstor.org/terms</p><p>National Academy of Sciences is collaborating with JSTOR to digitize, preserve and extend</p><p>access to Proceedings of the National Academy of Sciences of the United States of America</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>https://www.jstor.org/stable/41588543</p><p>Game-powered machine learning</p><p>Luke Barrington3,1, Douglas Turnbullb, and Gert Lanckriet3</p><p>aElectrical and Computer Engineering Department, University of California, La Jolla, CA 92093; and bComputer Science Department, Ithaca College,</p><p>Ithaca, NY 14850</p><p>Edited* by Grace Wahba, University of Wisconsin-Madison, Madison, Wl, and approved December 27, 2011 (received for review October 6, 2010)</p><p>Searching for relevant content in a massive amount of multimedia</p><p>information is facilitated by accurately annotating each image,</p><p>video, or song with a large number of relevant semantic keywords,</p><p>or tags. We introduce game-powered machine learning, an inte-</p><p>grated approach to annotating multimedia content that combines</p><p>the effectiveness of human computation, through online games,</p><p>with the scalability of machine learning. We investigate this frame-</p><p>work for labeling music. First, a socially-oriented music annotation</p><p>game called Herd It collects reliable music annotations based on</p><p>the "wisdom of the crowds." Second, these annotated examples</p><p>are used to train a supervised machine learning system. Third, the</p><p>machine learning system actively directs the annotation games to</p><p>collect new data that will most benefit future model iterations.</p><p>Once trained, the system can automatically annotate a corpus of</p><p>music much larger than what could be labeled using human com-</p><p>putation alone. Automatically annotated songs can be retrieved</p><p>based on their semantic relevance to text-based queries (e.g., "fun-</p><p>ky jazz with saxophone," "spooky electronica," etc.). Based on</p><p>the results presented in this paper, we find that actively coupling</p><p>annotation games with machine learning provides a reliable and</p><p>scalable approach to making searchable massive amounts of multi-</p><p>media data.</p><p>The media last content decade has available seen an online; explosion oyer 7 in billion the amount images of are multi- up- media content available online; oyer 7 billion images are up-</p><p>loaded to Facebook each month (1), YouTube users upload 24 h</p><p>of video content per minute (2), and iTunes, the world's largest</p><p>music retailer, offers a growing catalog of more than 20 million</p><p>songs (3). Developing a semantic multimedia search engine - that</p><p>enables simple discovery of relevant multimedia content as easily</p><p>as Internet search engines [e.g., Google (4)] help us find relevant</p><p>web pages - presents a challenge because the domain of the</p><p>query (text) differs from the range of the search results (images,</p><p>video, music).</p><p>To enable semantic search of nontextual content requires</p><p>a mapping between multimedia data and a wide vocabulary of</p><p>descriptive tags. Describing multimedia content with relevant</p><p>semantics necessitates intervention from humans who can under-</p><p>stand and interpret the images, video, or music. However, manual</p><p>tagging by human experts is too costly and time-consuming to be</p><p>applied to billions of data items. For example, Pandora, a popular</p><p>Internet radio service, employs musicologists to annotate songs</p><p>with a fixed vocabulary of about five hundred tags. Pandora then</p><p>creates personalized music playlists by finding songs that share a</p><p>large number of tags with a user-specified seed song. After 10 y of</p><p>effort by up to 50 full time musicologists, less than 1 million songs</p><p>have been manually annotated (5), representing less than 5% of</p><p>the current iTunes catalog.</p><p>Crowdsourcing has emerged as an affordable and scalable</p><p>alternative to expert annotation by engaging many nonexpert</p><p>contributors to label content online. Participants are motivated</p><p>through small monetary rewards (6), or, even better, to contribute</p><p>for free by disguising tasks as fun games, appealing to scientific</p><p>altruism, or requiring it to access a service of interest. This dis-</p><p>tributed human computation has been applied; e.g., to categorize</p><p>galaxies (7), fold proteins (8), transcribe old books (9), classify</p><p>smiles (10) and apply descriptive tags to images (11), web pages</p><p>(12) and music (13) (see SI Text for a review). Despite the promise</p><p>of recruiting vast amounts of free labor, human computation</p><p>games have had limited success in tagging the vast amount of</p><p>multimedia content on the web: in 5 y, the ESPgame (11) has</p><p>collected labels for up to 100 million images - roughly the same</p><p>number that are uploaded to Facebook every 10 h - and TagA-</p><p>Tune (13) has labeled 30,000 song clips, or about 0.15% of iTunes'</p><p>catalog.</p><p>Instead of requiring that humans manually label every image,</p><p>video, or song, tagging can be partially automated using super-</p><p>vised machine learning algorithms that learn how semantics relate</p><p>to multimedia. Machine learning approaches discover consistent</p><p>patterns among a modest number of prelabeled training examples</p><p>and then generalize this learned knowledge to label new, unla-</p><p>beled data. The scalability of computer automation offers the</p><p>potential to categorize massive amounts of multimedia informa-</p><p>tion but reliability hinges on the quality of training data used. For</p><p>example, by learning from millions of example images of faces in</p><p>all possible poses, angles, and lighting conditions, machine learn-</p><p>ing algorithms (14) rapidly and reliably detect faces to automate</p><p>focus in consumer digital cameras.</p><p>This paper proposes and investigates game-powered machine</p><p>learning as a reliable and viable solution to annotating large</p><p>amounts of multimedia content for semantic search, by lever-</p><p>aging the effectiveness of human computation through online</p><p>games with the scalability of supervised machine learning. The</p><p>main idea, illustrated for music search in Fig. 1, is to use an online</p><p>annotation game to collect reliable, human-labeled examples that</p><p>are tailored for training a supervised machine learning system.</p><p>Once trained, this system can automatically annotate new content</p><p>with the same tags used in the game, rapidly propagating seman-</p><p>tic knowledge to lots of multimedia content. Through an active</p><p>learning feedback loop, the game focuses on collecting data that</p><p>most effectively improves future machine learning updates.</p><p>To validate the effectiveness of game-powered machine learn-</p><p>ing for music search, we designed and developed "Herd It," an</p><p>online music annotation game that motivates players to contri-</p><p>bute tags for songs. In contrast to previous "games with a pur-</p><p>pose" which have aimed to annotate every image (11) or song</p><p>(13, 15) on the web, Herd It was designed with a different, unique,</p><p>and more realistic goal in mind: to enable the active machine</p><p>learning approach presented in Fig. 1. To this end, Herd It was</p><p>designed to integrate with a machine learning system that actively</p><p>suggests songs and tags to be presented to users. As a result, the</p><p>game can collect the most effective data for training the machine</p><p>learning algorithm</p><p>that then automates large-scale music tagging.</p><p>Besides focusing human effort on efficient data collection, active</p><p>song and tag suggestion also made gameplay more appealing.</p><p>/ Author contributions: L.B., D.T., and G.L. designed research; L.B. and D.T. performed</p><p>research; L.B. contributed new reagents/analytic tools; L.B. analyzed data; and L.B., D.T.,</p><p>and G.L. wrote the paper.</p><p>The authors declare no conflict of interest.</p><p>This Direct Submission article had a prearranged editor.</p><p>Data deposition: Text dataset listing collected associations between popular music songs</p><p>and descriptive text tags. Electronic datafiles containing signal processing features</p><p>extracted from musical audio files.</p><p>1To whom correspondence should be addressed. E-mail: lukeinusa@gmail.com.</p><p>This article contains supporting information online at www.pnas.org/lookup/suppl/</p><p>doi: 1 0. 1 073/pnas.1 0147481 09/-/DCSupplemental.</p><p>www.pnas.org/cgi/doi/10.1073/pnas.1014748109 PNAS | April 24, 2012 | vol.109 | no. 17 | 6411-6416</p><p>i/i</p><p>LLI</p><p>U</p><p>z</p><p>LLI</p><p>U</p><p>v/»</p><p>C£</p><p>LU</p><p>I-</p><p>3</p><p>a.</p><p>2</p><p>O</p><p>U</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>Fig. 1. Game-powered machine learning framework for music annotation.</p><p>We deploy this game-based machine learning system to inves-</p><p>tigate and answer two important questions. First, we demonstrate</p><p>that the collective wisdom of Herd It's crowd of nonexperts can</p><p>train machine learning algorithms as well as expert annotations by</p><p>paid musicologists. In addition, our approach offers distinct ad-</p><p>vantages over training based on static expert annotations: it is</p><p>cost-effective, scalable, and has the flexibility to model demo-</p><p>graphic and temporal changes in the semantics of music. Second ,</p><p>we show that integrating Herd It in an active learning loop trains</p><p>accurate tag models more effectively; i.e., with less human effortj</p><p>compared to a passive approach.</p><p>Herd It-- A Social Music Annotation Game</p><p>A player arriving at Herd It (www.HerdIt.org) is connected with</p><p>"the Herd" - all other players currently online - and the game</p><p>begins. Each round of Herd It begins by playing the same piece</p><p>of music to all members of the Herd. A variety of fun, simple mini-</p><p>games prompt players to choose from suggested tags that describe</p><p>different aspects of the music they hear (Fig. 2 illustrates an ex-</p><p>ample of Herd It's gameplay with further examples in SI Text). In</p><p>every minigame, players earn points based on their agreement</p><p>with the tags chosen by the rest of the Herd, encouraging players</p><p>to contribute tags that are likely to achieve consensus.</p><p>Herd It's goal is to collect training data that primes and im-</p><p>proves the machine learning system through an active learning</p><p>loop by motivating human players to provide reliable descriptions</p><p>of a large number of example songs using a dynamic vocabulary</p><p>of tags. To achieve this goal, Herd It's development followed a</p><p>user-centered design process (16) that aimed to create an intuitive,</p><p>viral game experience. A series of rapid prototypes were released</p><p>every month and tested on focus groups of 5-50 new players,</p><p>both in person at our lab and in a controlled online environment.</p><p>During each test, we evaluated factors including playability and</p><p>appeal, user-interface intuitiveness, viral potential, and stability.</p><p>Interviews and questionnaires were used to evaluate the extent to</p><p>which players were able to focus on the music (ensuring reliable</p><p>data collection), their awareness of the other players (Herd It is a</p><p>social game and a player's score depends on the Herd), and over-</p><p>all enjoyment (indicating likelihood of large-scale participation).</p><p>Iterative user feedback led to improvements in the design and the</p><p>process continued until key gameplay and social evaluation me-</p><p>trics were satisfied (e.g., 94% of players understood the scoring</p><p>metric within five games, 82% said they would recommend Herd</p><p>It to their friends; further results in SI Text). The user-centered</p><p>design process was instrumental in determining crucial gameplay</p><p>mechanics, described below, that differentiate Herd It from other</p><p>music annotation games [e.g., (13, 15)].</p><p>In particular our user tests discovered that, while free-text tag-</p><p>ging works well when annotating images (which tend to feature</p><p>many obvious, easily named objects; see; e.g., ref. 11), many lis-</p><p>teners found it difficult to produce and agree on a variety of tags</p><p>for music in a game environment without some priming. Asking</p><p>players to type their own descriptions of the music meant that</p><p>the vast majority of tags were confined to a limited vocabulary</p><p>of generic tags; e.g., "rock," "guitar," "drums," "male/female voc-</p><p>alist" [MajorMirier (15) suffers from this problem]. As a result,</p><p>Fig. 2. Illustration of Herd It gameplay. (A) Six bubbles float around the play area, each suggesting a mood that might be evoked by the music that is playing.</p><p>The player clicks the bubble they feel is most appropriate and all other bubbles disappear with a pop. After 15 s, the minigame ends. (B) Choices made by the</p><p>rest of the Herd are revealed. During this feedback period, an "agree-O-meter" fills up as other members of the Herd agree with the player's choice. Players</p><p>earn points equal to the percentage of the Herd in agreement with them, rewarding consensus and implicitly collecting reliable music tags.</p><p>6412 | www.pnas.org/cgi/doi/l0.1073/pnas.1014748109 Barrington et al.</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>the independent inputs of multiple players rarely converged on</p><p>more interesting tags [TagATune (13) avoids this problem by ask-</p><p>ing players to guess whether they are listening to the same song,</p><p>based on the free-text tags other players entered, rather than re-</p><p>quiring agreement on the exact tags]. To achieve both variety and</p><p>consensus, Herd It's unique solution is to suggest tags for player</p><p>confirmation, thereby controlling the vocabulary used to describe</p><p>music while maintaining simple and compelling gameplay. In ad-</p><p>dition, tag suggestion addresses another, important design objec-</p><p>tive: it facilitates the active learning paradigm depicted in Fig. 1</p><p>which requires precise control over the data collected from the</p><p>Herd. Specifically? an active learning approach leverages machine</p><p>learning models tb suggest {song, tag} combinations that, if con-</p><p>firmed by human players, are most likely to produce useful train-</p><p>ing examples and optimize future model training. Herd It's tag</p><p>suggestion mechanism enables active learning by focusing human</p><p>labeling on specific {song, tag} combinations. Vice versa, Herd</p><p>It's new tag suggestion design benefits from it being powered with</p><p>machine intelligence. Indeed, suggesting tags randomly, rather</p><p>than intelligently, was found to result in many minigames that</p><p>have no relevant choices and are not fun.</p><p>To achieve widespread player engagement and thus maximize</p><p>training data collection, we found that Herd It should target the</p><p>"casual" gamer. Unlike the traditional computer gaming demo-</p><p>graphic (i.e., teenage boys) who enjoy long-lasting games with</p><p>complicated gameplay mechanics, casual games appeal to a much</p><p>wider demographic (e.g., skewed towards middle-aged women),</p><p>are played in short time increments (5-20 min) and feature simple</p><p>but addicting gameplay (17). Herd It's simple, single-click game-</p><p>play, cartoon-ish minigame design, and intuitive scoring metric</p><p>were designed to attract a broad audience of casual gamers.</p><p>Based on the choices offered in a given minigame, different</p><p>users may end up describing a song differently, using either com-</p><p>patible tags (e.g., a "romantic" song that is also described as</p><p>"carefree") or opposite tags (e.g., what sounds "exciting" to one</p><p>listener may be "boring" to another). Given this subjectivity in-</p><p>herent</p><p>in music appreciation, our design process revealed that it</p><p>is important to evaluate agreement in minigames in a (larger)</p><p>group setting, as this enables clusters of consensus to develop</p><p>between the players, around multiple "right" answers. This obser-</p><p>vation inspired us to make "the Herd" a central feature of the</p><p>game, rather than the player-vs-player mechanic used by other</p><p>games [e.g., (11, 13)]. In addition, our user tests determined that</p><p>realtime , social interaction produced more compelling gameplay</p><p>than off-line group feedback [e.g., (15)]. The group dynamic also</p><p>makes it more difficult for a few players to cheat and gain lots of</p><p>points by coordinating poor labeling (other measures to prevent</p><p>cheating include randomizing tag order in minigames and pre-</p><p>venting a single player from entering multiple games).</p><p>Finally, because individuals use music preference to commu-</p><p>nicate information about their personality (18), players desired</p><p>Herd It to be embedded in a larger social music experience. For</p><p>example, players wanted to choose preferred genres, share music,</p><p>create personal profiles, and challenge and compare scores with</p><p>friends, leading us to integrate the game within the players'</p><p>existing social network by releasing Herd It as an application</p><p>on Facebook. Integrating Herd It with Facebook offers many</p><p>avenues to engage players (e.g., easy login, personalized mes-</p><p>sages, player photos) and promote the game to a wide audience</p><p>(e.g., invites, challenges, see SI Text). Facebook also provides de-</p><p>mographic and psychographic information about players (e.g.,</p><p>gender, age, location, friends, favorite music), offering a hitherto</p><p>unavailable level of insight into how different people experience</p><p>and describe music.</p><p>Automatic Music Tagging</p><p>Statistical pattern recognition methods for tagging music begin</p><p>by extracting features that summarize properties of the acoustic</p><p>waveforms, essentially "listening" to the musical signal. By consid-</p><p>ering a training set of reliably labeled songs, supervised machine</p><p>learning algorithms identify statistical regularities in these acous-</p><p>tic features that are predictive of descriptive tags like "bluegrass,"</p><p>"banjo," or "mellow." Machines can then generalize this knowl-</p><p>edge by detecting the presence of similar patterns in vast catalogs</p><p>of new, untagged music, thereby leveraging the accuracy of human</p><p>labeling (to ojbtain the training set) with the scalability of auto-</p><p>mated analysis to tag this new music content. Machine learning</p><p>methods for music tagging continue to improve and, given training</p><p>data of sufficient quality, their accuracy approaches the ceiling set</p><p>by the inherent subjectivity in describing music with tags (19).</p><p>To thoroughly evaluate the efficacy of the game-powered ma-</p><p>chine learning paradigm depicted in Fig. 1, we consider various</p><p>state-of-the-art autotagging algorithms for its machine learning</p><p>component, including generative (19, 20) and discriminative</p><p>(21, 22) approaches. Generative methods focus on estimating</p><p>the (class-conditional) distribution [e.g., with a Gaussian mixture</p><p>model (GMM), dynamic texture mixtures (DTM), etc.] of acous-</p><p>tic features that are common among songs that human "trainers"</p><p>have labeled with a given tag. By evaluating the likelihood of</p><p>features from a new song under the learned distribution, the</p><p>model determines the probability that the tag is a relevant de-</p><p>scription of the song (19). Discriminative methods, on the other</p><p>hand, directly optimize a decision rule to discriminate between a</p><p>tag being present or absent for a given audio clip. Evaluating the</p><p>decision rule for a new song allows to obtain tag probabilities.</p><p>Just as Internet search engines rank web pages by their relevance</p><p>to a text query, the tag probabilities output by a model can be</p><p>used to rank songs by their relevance to the tag.</p><p>Traditional machine learning approaches use a single, fixed</p><p>training set to learn models that, once trained, remain static.</p><p>In our game-powered machine learning framework however,</p><p>new data is constantly being contributed by Herd It players. That</p><p>data can be used to update our tag models. Even more, because</p><p>Herd It's design permits actively focusing players' efforts on spe-</p><p>cific songs and tags, it is possible to collect specifically that data</p><p>that is expected to improve tag models most effectively, achieved</p><p>through an active learning approach (see ref. 23 for a review), that</p><p>leverages the current tag models to identify the most effective</p><p>song-tag pairs for future model updates. Active learning fully in-</p><p>tegrates the autotagging algorithm with the data collection pro-</p><p>cess, to optimize model training. To investigate the benefits of</p><p>deploying Herd It in an active learning loop compared to updat-</p><p>ing with randomly collected data, we develop a unique active</p><p>learning algorithm to suggest data for training the generative</p><p>GMM-based autotagger, a top performer in the 2008 MIREX</p><p>evaluation of automatic music tagging algorithms (24).</p><p>Various active learning algorithms have been proposed for dis-</p><p>criminative machine learning methods*, where both positive and</p><p>negative examples are used to learn a decision boundary between</p><p>classes. Generative approaches, on the other hand, require only</p><p>positively labeled examples for training (i.e., songs-that exemplify</p><p>a certain tag) and negatively labeled training examples offer no</p><p>improvement to the model*. To collect positively labeled training</p><p>examples and improve a generative model through active learn-</p><p>ing may suggest sampling unlabeled examples that have high</p><p>likelihood under the current model and procuring labels for them</p><p>[e.g., by presenting {song, tag} pairs in Herd It minigames]. How-</p><p>ever, this "certainty" sampling approach suffers from two draw-</p><p>backs: early in training, when the model is not yet well learned,</p><p>Strategies for actively learning discriminative models include uncertainty sampling (25)</p><p>where points are chosen that are least certain (or have highest entropy) under the</p><p>current model (e.g., points closest to the decision boundary) and variance reduction</p><p>(26) where samples are chosen to reduce the model's output variance.</p><p>*For example, for generative models, uncertainty sampling faces the problem that</p><p>unlabeled songs which have low certainty under the current model are likely to result</p><p>in negative labels.</p><p>Barrington et all PNAS | April 24, 2012 | vol.109 | no. 17 | 6413</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>the most likely samples may not in fact be positive examples and</p><p>thus will not contribute to the training set. Later in the learning</p><p>process, sampling from the most likely areas results in many con-</p><p>firmed positive examples that conform to the model's current</p><p>training set and lack the diversity required to generalize the cur-</p><p>rent model to uncertain areas of the feature space. Exploration of</p><p>these uncertain areas advocates for a more random sampling of</p><p>unlabeled examples. Rather than a complete rancfom sampling,</p><p>we can actively increase the efficiency of the data collection and,</p><p>thus, the learning rate, by reducing the likelihood of sampling un-</p><p>labeled examples that are eventually labeled as negatives (which</p><p>are of no use to train the generative model) and thereby avoiding</p><p>points that most disagree with the current model. More specifi-</p><p>cally, we rank all of the unlabeled examples by their likelihood</p><p>under the current model, remove the 10% of examples with lowest</p><p>likelihood, and query labels randomly from the remaining 90% of</p><p>examples. By removing the least likely points and sampling ran-</p><p>domly elsewhere, we aim to avoid querying labels for negative ex-</p><p>amples and achieve rapid confirmation of a diverse training set for</p><p>our generative model. Our active GMM experiments show that</p><p>removing the 10% least likely songs finds a good balance between</p><p>exploring poorly</p><p>modeled areas of the feature space while avoid-</p><p>ing points that are unlikely to produce positive examples.®</p><p>Game-Powered Machine Learning</p><p>Our game-powered machine learning approach aims to collect</p><p>sufficient human labels, through game play, to train an automatic</p><p>music annotation system that can reliably generalize semantics to</p><p>unlimited music. Qualitatively, we argue that this crowdsourced</p><p>approach is superior to requiring expert annotators as it is less</p><p>costly, more scalable, and collects a dynamic dataset that can be</p><p>adapted over time to focus on the most relevant or important</p><p>tags. To quantify the efficacy of our game-powered machine</p><p>learning framework, we conduct experiments designed to answer</p><p>the following two questions: (/) Can machine learning algorithms</p><p>be trained with data collected from Herd It's crowd of nonexperts</p><p>as accurately as with data collected from paid expert musicolo-</p><p>gists? (//) Can accurate tag models be learned with less human</p><p>effort by encapsulating Herd It in an active learning framework?</p><p>To answer thege questions, we deployed Herd It online, engaging</p><p>7,947 people to provide over 140,000 clicks that associate songs *</p><p>with tags through five different types of minigames.</p><p>To generate minigames in a passive system, without active</p><p>learning, we begin with 10-20 candidate {song, tag} pairs, chosen</p><p>randomly from the authors' personal music collection of over</p><p>6,000 popular songs from the past 70 y, and a vocabulary of 1,269</p><p>tags, including subgenres, emotions, instruments, usages, colors,</p><p>and more categories. To generate a minigame, one {song, tag} pair</p><p>is selected from the list of candidate pairs, biased by associations</p><p>(determined using the online music service http://last.fm/api/) with</p><p>the musical genre selected by the player at the start of the game</p><p>(pop, rock, hip-hop, blues, electronical or "everything") and by the</p><p>particular minigame (e.g., certain minigames focus on subgenres,</p><p>colors, or bipolar adjectives). Remaining minigame tags (each</p><p>minigame suggests between one and nine tags) are restricted to</p><p>the same tag category. Candidate {song, tag} pairs remain on</p><p>the list until they have been viewed by at most 50 players. At that</p><p>point the {song, tag} pair is discarded and replaced by a new, ran-</p><p>domly sampled one. Maintaining a reasonable list of candidate</p><p>pairs ensures diverse gameplay.</p><p>Consensus between players' clicks collected in Herd It mini-</p><p>games is used to "confirm" reliable {song, tag} associations. More</p><p>specifically, the generative model of labels, accuracies, and diffi-</p><p>sln a feature space of high dimension, d, the probability density of a Gaussian distribution</p><p>with variance a is focused on a small shell a distance o'fd from the mean. Thus the</p><p>majority of points tend to have very similar GMM likelihoods (27). While this fact can</p><p>make it difficult to identify positive points based on likelihood, any points that have</p><p>significantly lower than average likelihood can be excluded with confidence.</p><p>culties, or "GLAD," (10) conceives of each human input as an</p><p>estimate of the underlying true label that has been corrupted</p><p>by player inaccuracy and the difficulty of labeling the song. Using</p><p>an expectation-maximization algorithm, GLAD optimally com-</p><p>bines the votes from all Herd It players and we confirm the find-</p><p>ings of (10) that the resulting consensus is more reliable than</p><p>heuristics such as majority vote, percentage agreement, or vote</p><p>thresholds. A {song, tag} pair is presented in Herd It minigames</p><p>until GLAD "confirms" a reliable association, based on the his-</p><p>torical click data for that {song, tag} pair. If a {song, tag} pair re-</p><p>mains unconfirmed after being viewed by 50 Herd It players, it is</p><p>"rejected" and not sampled further. Overall, GLAD confirmed</p><p>8,784 {song, tag} pairs, representing song examples of 549 tags,</p><p>while 256,000 pairs were rejected. To ensure that we have enough</p><p>data to train robust machine learning models and answer the first</p><p>question, we reduce the dataset to the 127 tags for which Herd It</p><p>has identified at least 10 reliable example songs. Data was col-</p><p>lected passively (i.e., no active learning) and this provides the</p><p>baseline against which to compare an active learning strategy</p><p>and evaluate the second question.</p><p>To answer the first question, we quantify the efficacy of our</p><p>game-powered machine learning framework and compare it to</p><p>"expert-trained" machine learning. That is, we evaluate the per-</p><p>formance of a music autotagging algorithm when trained on (/)</p><p>the Herd It game data and («) data derived from expert musicol-</p><p>ogists at Pandora.com's "Music Genome Project" ( MGP ), respec-</p><p>tively. After training, the accuracy of each autotagger is evaluated</p><p>on CAL500 : an independent evaluation set of 500 songs fully</p><p>labeled by multiple humans using a controlled survey (19) (see</p><p>SI Text for details about the CAL500 and MGP datasets). For this</p><p>comparison, we train and evaluate models of all tags that are</p><p>available in both the MGP and CAL500 vocabulary and for which</p><p>Herd It has collected at least ten confirmed example songs,</p><p>resulting in 25 tags. The models of each of these tags are used</p><p>to retrieve the 10 most relevant songs from the CAL500 corpus</p><p>for each single-tag query. These top-ten search results - automa-</p><p>tically retrieved by a machine - are evaluated by comparing to the</p><p>CAL500 ground-truth, and computing the precision (i.e., the num-</p><p>ber of songs in the machine-ranked top-ten that the ground truth</p><p>effectively associates with the tag). Finally, the precision is aver-</p><p>aged over all 25 tags. Because both Herd It and MGP models are</p><p>instances of the same machine learning algorithm, but trained on</p><p>different datasets, any significant differences in autotagging per-</p><p>formance most likely reflect differences in the quality of the re-</p><p>spective training data sources and allow us to evaluate human</p><p>computation games - Herd It, in particular - as a source of reli-</p><p>able training data. To prevent bias induced by a particular choice</p><p>of machine learning algorithm, this comparison is repeated for</p><p>multiple state-of-the-art autotagging algorithms.</p><p>In addition to showing that game-powered machine learning</p><p>can be competitive with an expert-trained system, in a second step,</p><p>we demonstrate the efficacy of actively integrating machine learn-</p><p>ing with game-based data collection. The baseline here is the pas-</p><p>sive approach outlined above, which "analyzes" (i.e., confirms or</p><p>rejects through human labeling) {song, tag} pairs in random order.</p><p>As more {song, tag} pairs are analyzed (i.e., more human effort</p><p>contributed), a tag's training set grows, tag models are updated</p><p>and autotagging performance is expected to improve. We compare</p><p>this passive approach to an active learning paradigm which aims to</p><p>improve tag models more effectively by leveraging current models</p><p>to select the next {song, tag} pairs that will be analyzed. More pre-</p><p>cisely, for each of the 25 Herd It tags that were evaluated earlier,</p><p>we collect all songs that appeared with the tag (confirmed or re-</p><p>jected) in previous Herd It minigames. We then estimate 25 GMM-</p><p>based tag models by engaging in an iterative training procedure, for</p><p>each tag, based on this list of "candidate" songs. At each iteration,</p><p>we first compute the likelihood, under the current tag model, of all</p><p>remaining candidate songs and use our active learning method for</p><p>6414 | www.pnas.org/cgi/doi/10.1073/pnas.1014748109 Barrington et al.</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>generative models to prioritize 10 candidate songs for analysis with</p><p>that tag (for the first iteration, candidate songs are chosen ran-</p><p>domly). That is, we use our active learning algorithm to resample</p><p>{song, tag} pairs that were previously presented in Herd It games.</p><p>Songs for which the {song, tag} pair was previously confirmed are</p><p>added to the</p><p>tag's training set; the remaining, rejected songs are</p><p>removed from future candidate lists. Finally, we retrain the tag</p><p>model using the updated training set and evaluate its performance</p><p>on the CAL500 test set. We once again recompute the likelihood of</p><p>all remaining candidate songs under the updated model, actively</p><p>select 10 candidate songs for analysis, retrain the tag models,</p><p>and so on. Song selection is repeated up to 200 times, analyzing</p><p>up to 2,000 songs for each tag. At each iteration, we evaluate</p><p>the average performance of the 25 tag models. We compare active</p><p>learning to the passive baseline which samples 10 songs randomly at</p><p>each iteration, for each tag.</p><p>Results</p><p>Table 1 presents the average precision of the top-ten music search</p><p>results for 25 single-tag queries on CAL500 , achieved by training</p><p>four state-of-the-art autotagging algorithms on Herd It's data. We</p><p>compare to the performance obtained by training on expert MGP</p><p>data. For each of the 25 tags common to Herd It, MGP and</p><p>CAL500 , we evaluate the top-ten precision on CAL500 and aver-</p><p>age performance over all tags. While the absolute performance</p><p>depends on the machine learning method used, the relative per-</p><p>formance between models trained using Herd It and those that</p><p>use MGP data remains consistently over 95%. These findings</p><p>answer our first question by demonstrating that a game-based</p><p>machine learning system, trained on data collected from Herd</p><p>It players, provides a competitive alternative to a system trained</p><p>on expert labeled data, across a variety of algorithms.</p><p>Fig. 3 offers a more detailed comparison of Herd It and MGP</p><p>based systems, by examining the performance of each tag model</p><p>learned by the hierarchical GMM algorithm (19). The ability of</p><p>the machine learning algorithm to model different tags varies;</p><p>e.g., "acoustic," "male lead vocals," and "hip hop" songs are more</p><p>easily identified, while "hand drums" and "funk" music are poorly</p><p>modeled. In general, model performance is independent of the</p><p>training data source (i.e., most points lie close to the diagonal</p><p>in Fig. 3, indicating comparable results for each system). Models</p><p>trained on either data source performed significantly differently</p><p>for just two tags: "synthesizer" (MGP-based model better) and</p><p>"drum set" (Herd It-based model better, 2-tailed t-test, 95% sig-</p><p>nificance level). In summary, Table 1 and Fig. 3 quantitatively de-</p><p>monstrate that training from Herd It's crowdsourced data</p><p>captures knowledge similar to training from expert annotations.</p><p>We turn now to the second question: can integrating machine</p><p>learning and Herd It's game-powered data collection in an active</p><p>learning loop train accurate models with less human effort than</p><p>a passive system? To measure human effort, we consider the num-</p><p>ber of {song, tag} pairs analyzed through Herd It gameplay,</p><p>expressed as the number of songs analyzed per tag (for each of</p><p>the 25 tags being modeled). Fig. 4 displays the improvement in</p><p>song retrieval performance of the GMM autotagging algorithm as</p><p>Table 1. Average top-ten precision of autotagging algorithms</p><p>trained on Herd It examples and tested on CAL500</p><p>Top-10 Precision Relative Precision</p><p>Autotagging algorithm Herd It training Herd It : MGP</p><p>Hierarch. GMM (19) 0.40 95.8% ± 4.6</p><p>Hierarch. DTM (20) 0.42 98.9% ± 6.0</p><p>Boosting (21) 0.38 99.7% ±4.2</p><p>SVM (22)</p><p>Also shown is relative performance (top-ten precision) of Herd</p><p>It-trained models compared to models trained on expert MGP</p><p>examples. Top-ten precision of random guessing is 0.18.</p><p>Fig. 3. Top-ten precision for machine learning models trained on Herd It's</p><p>crowdsourced data (x-axis) and models trained on data from the Music</p><p>Genome Project (/-axis). While absolute performance depends on the tag</p><p>(e.g., "acoustic" music is better modeled than "soul" music), on average</p><p>(dashed lines) Herd It's crowdsourced data trains models that are as precise,</p><p>at the tag-level, as models learned from expert-labeled data.</p><p>more songs are analyzed for each tag (and, consequently, more</p><p>training examples collected for model estimation), following both</p><p>an active learning and a random sampling strategy. The results</p><p>demonstrate an improved learning rate due to active learning:</p><p>active learning requires analyzing, on average, 450 songs per tag</p><p>to achieve no significant difference between Herd It and MGP</p><p>performance (paired, one-tailed t-test,/? = 0.1) while the passive</p><p>strategy hits this level after analyzing 940 songs for each tag. Fig. 4</p><p>highlights the improved efficiency by shading the learning curves</p><p>while performance is significantly different from the MGP: by</p><p>prioritizing the order in which {song, tag} pairs are presented</p><p>to players, our active learning approach reduces the human label-</p><p>ing effort required by half. A more detailed inspection of the</p><p>results reveals that active learning achieves expert performance</p><p>by confirming an average of 31 training songs per tag, out of 450</p><p>analyzed candidates, vs. 49 out of 940 for random sampling. Ac-</p><p>tive learning boosts the learning rate by suggesting fewer {song,</p><p>tag} pairs that are eventually rejected and not used for training</p><p>[compared to suggesting random {song, tag} pairs] while still pro-</p><p>ducing a sufficiently diverse set of confirmed training songs.</p><p>Having demonstrated that Herd It data can train automatic</p><p>music taggers that are as accurate as an expert-trained system,</p><p>we now compare the tagging efficiency of crowdsourced amateur</p><p>players with that of trained experts. Pandora's musicological ex-</p><p>perts fake 20-30 min to analyze and quantify the association be-</p><p>tween a song and 100-500 semantic dimensions, a rate of about</p><p>12 song-tag associations per expert-minute. Herd It minigames</p><p>last about 30 s and present, on average, 5.4 tags for player ana-</p><p>lysis. Thus a single Herd It player analyzes 10.8 song-tag associa-</p><p>tions per minute, a little less than the Pandora expert. To quantify</p><p>(i.e., confirm or reject) a song-tag association, the analysis of up</p><p>to 50 players is required, vs. that of one Pandora expert. So, Herd</p><p>It's game-based approach gathers reliable tags from humans for</p><p>free with about 2% the efficiency of paid, expert labeling. Of</p><p>course, Herd It's lower efficiency is multiplied by the number</p><p>of simultaneous players in the Herd, which could be significantly</p><p>larger than the number of musicological experts that can be gain-</p><p>fully employed, simultaneously.</p><p>In comparing game-based and expert annotation methods,</p><p>we recognize that, even with crowdsourced consensus, multiple</p><p>Barrington et al. PNAS | April 24, 201Z | vol. 109 | no. 17 | 6415</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>Fig. 4. Music autotagging performance as a function of human effort</p><p>(i.e., number of songs analyzed by Herd It players). For each of the 25 tags</p><p>considered, "passive" randomly selects songs for analysis while "active" le-</p><p>verages an active learning paradigm. Each {song, tag} pair appeared in Herd It</p><p>minigames until it was either confirmed by GLAD or rejected after being pre-</p><p>sented to 50 players. The y-axis plots the average precision of the top-ten</p><p>search results returned by tag models trained on all songs confirmed by Herd</p><p>It at each 10-song increment on the x-axis until the target performance re-</p><p>ported in Table 1 (for batch training) is achieved (680 songs for active, 1,330</p><p>songs for passive). Error bands show the standard error of the mean, aver-</p><p>aged over three independent trials, and remain shaded while Herd It perfor-</p><p>mance is significantly below MGP (paired t-test, p = 0*1). Integrating active</p><p>learning with Herd It's data collection improves the learning rate, achieving</p><p>significant performance with less human effort.</p><p>amateur raters are likely less reliable than experts when identify-</p><p>ing certain details pertaining to musical theory (e.g., we find Herd</p><p>It players are inconsistent in Scales</p><p>minigames that ask to distin-</p><p>guish major from minor keys) or esoteric subgenres and instru-</p><p>ments (e.g., tags for the subgenres "doom metal" and "worldbeat"</p><p>and the instruments "siren" and "spoons" were rarely chosen</p><p>when suggested in a minigame and, if they were chosen, it often</p><p>was seemingly without relation to the audio content). In design-</p><p>ing Herd It's games, we generally focused on tags that are more</p><p>relevant to our goal of building a music search engine that can</p><p>empower a wide audience to discover relevant music using sim-</p><p>ple, semantic search.</p><p>Finally, while a human-only approach requires the same label-</p><p>ing effort for the first song as for the millionth, our game-powered</p><p>machine learning, solution needs only a small, reliable training set</p><p>before all future examples can be labeled automatically, improv-</p><p>ing efficiency and cost by orders of magnitude. Tagging a new</p><p>song takes 4 s on a modern CPU: in just a week, eight parallel</p><p>processors could tag 1 million songs or annotate Pandora's com-</p><p>plete song collection, which required a decade of effort from doz-</p><p>ens of trained musicologists.</p><p>Conclusions</p><p>We proposed game-powered machine learning as an integrated,</p><p>scalable, affordable, and reliable solution for semantic search of</p><p>massive amounts of multimedia content and investigated its effi-</p><p>cacy for music search. Herd It, an online music annotation game,</p><p>collects reliable examples of how humans use semantic tags to</p><p>describe music. By itself, this human computation approach is</p><p>insufficient to label the millions of songs available on the web.</p><p>Instead, the knowledge collected by our game trains machine</p><p>learning algorithms that can generalize tags to vast amounts of</p><p>new, unlabeled music. Compared to other music games with a</p><p>purpose, Herd It was specifically .designed to be actively inte-</p><p>grated with the machine learning algorithms and provide the data</p><p>that most effectively trains them. Our results demonstrate, first,</p><p>that game-powered machine learning is as good as expert-based</p><p>machine learning - annotations collected from human computa-</p><p>tion games train autotagging models as accurately as expensive,</p><p>expert annotations - while offering some distinct advantages</p><p>(e.g., cost-effectiveness, scalability, flexibility to update the game</p><p>to focus on tags of interest). Second, we show that embedding</p><p>Herd It in an active learning paradigm trains accurate autotag-</p><p>gers more effectively; i.e., with less human effort, compared to</p><p>a passive approach. We conclude that actively integrating human</p><p>computation games and machine learning - combining targeted</p><p>data collection by annotation games with automatic prediction</p><p>by scalable machine learning algorithms - enables simple, wide-</p><p>spread multimedia search and discovery.</p><p>ACKNOWLEDGMENTS. The authors thank Damien O'Malley for assistance in the</p><p>design, development, and testing of Herd It, and Sanjoy Dasgupta and Brian</p><p>McFee for active learning advice. L.B. and D.T. received funding from a</p><p>National Science Foundation (NSF) fellowship (DGE-0333451). L.B. received</p><p>funding from the Qualcomm Innovation Fellowship. L.B. and G.L. received</p><p>funding from the Hellman Fellowship Program, the von Liebig Center,</p><p>the Committee on Research (grant RJ138G-LANCKRIET), the Alfred P. Sloan</p><p>Foundation, Yahoo! Inc., and the NSF (DMS-MSPA 0625409 and IIS-1054960).</p><p>1. Facebook (2012) http://www.facebook.com/press/info.php7statistics February.</p><p>2. YouTube (2012) http:llwww.youtube.com/tlfact_sheet February.</p><p>3. Apple iTunes (2012) http://www.apple.com/itunes/features/ February.</p><p>4. Brin S, Page L (1998) The anatomy of a large-scale hypertextual web search engine.</p><p>Comput Netw 30:107-1 17.</p><p>5. Pandora Media Inc. (201 1) Securities and Exchange Commission Form S-1 ., http://www.</p><p>sec.gov/Archives/edgar/data/1 230276/0001 1931251 1032963/ds1 .htm February.</p><p>6. Amazon Mechanical Turk (2012) http://mturk.amazon.com.</p><p>1. GalaxyZoo (2012) http://www.galaxyzoo.org.</p><p>8. Cooper S, et al. (2010) Predicting protein structures with a multiplayer online game.</p><p>Nature 466:756-760.</p><p>9. von Ahn L, Maurer B, McMillen C, Abraham Df Blum M (2008) reCAPTCHA: human-</p><p>based character recognition via web security measures. Science 321:1465-1468.</p><p>10. Whitehill J, Ruvolo P, Bergsma J, Wu T, Movellan J (2009) Whose vote should count</p><p>more: optimal integration of labels from labelers of unknown expertise. In 23rd</p><p>Conference on Neural Information Processing Systems (NIPS) (MIT Press, MA).</p><p>11. von Ahn L, Dabbish L (2004) Labeling images with a computer game. In 22nd Inter-</p><p>national Conference on Human Factors in Computing Systems (ACM SIGCHI).</p><p>12. von Ahn L (2006) Games with a purpose. IEEE Computer Magazine 39:92-94.</p><p>13. Law E, von Ahn L (2009) Input-agreement: a new mechanism for collecting data using</p><p>human computation games. In 27th International Conference on Human Factors in</p><p>Computing Systems (ACM SIGCHI).</p><p>14. Viola P, Jones M (2004) Robust real-time face detection. Int J Comput Vision</p><p>57:137-154.</p><p>1 5. Mandel M, Ellis D (2008) A web-based game for collecting music metadata. J New Mu-</p><p>sic Res 37 :1 51-1 65.</p><p>16. Gould J, Lewis C (1985) Designing for usability: key principles and what designers</p><p>think. Commun ACM 28:300-311.</p><p>17. Rohrl D, ed. (2008) Casual Games White Paper. (International Game Developers Asso-</p><p>ciation).</p><p>18. Rentfrow P, Gosling S (2006) Message in a ballad: the role of music preferences in in-</p><p>terpersonal perception. Psychol Sci 17:236-242.</p><p>19. Turnbull D, Barrington L, Torres D, Lanckriet G (2008) Semantic annotation and retrie-</p><p>val of music and sound effects. IEEE T Acoust Speech 16:467-476.</p><p>20. Coviello E, Barrington L, Lanckriet GRG, Chan AB (2010) Automatic music tagging with</p><p>time series models. In 11th International Society for Music Information Retrieval (IS-</p><p>MIR) Conference (International Society for Music Information Retrieval , Netherlands).</p><p>21 . Eck D, Lamere P, Bertin-Mahieux T, Green S (2007) Automatic generation of social tags</p><p>for music recommendation. In 21st Conference on Neural Information Processing</p><p>Systems (NIPS) (MIT Press, MA).</p><p>22. Mandel M, Ellis D (2008) Multiple-instance learning for music information retrieval.</p><p>In 9th International Society for Music Information Retrieval (ISMIR) Conference</p><p>(International Society for Music Information Retrieval , PA).</p><p>23. Settles B (2010) Active learning literature survey. University of Wisconsin- Madison:</p><p>Computer Sciences Technical Report.</p><p>24. Downie JS (2008) Audio tag classification. Music Information Retrieval Evaluation ex-</p><p>change (MIREX), http://music-ir.Org/mirex/wiki/2008:Audio_Tag_Classification_Results.</p><p>25. Lewis D, Catlett J (1994) Heterogeneous uncertainty sampling for supervised</p><p>learning. In 11th International Conference on Machine Learning (ICML) (Morgan</p><p>Kaufmann, CA).</p><p>26. Cohn DA, Ghahramani Z, Jordan Ml (1996) Active learning with statistical models.</p><p>Journal of Artificial Intelligence Results 4:129-145.</p><p>27. Dasgupta 5, bcnulman u (2007) A probabilistic analysis or tM tor mixtures ot</p><p>separated, spherical Gaussians. J Mach Learn Res 8:203-226.</p><p>6416 | www.pnas.org/cgi/doi/10.1073/pnas.1014748109 Barrington et al.</p><p>This content downloaded from</p><p>�����������177.102.18.144 on Tue, 15 Aug 2023 13:27:50 +00:00�����������</p><p>All use subject to https://about.jstor.org/terms</p><p>Contents</p><p>p. [6411]</p><p>p. 6412</p><p>p. 6413</p><p>p. 6414</p><p>p. 6415</p><p>p. 6416</p><p>Issue Table of Contents</p><p>Proceedings of the National Academy of Sciences of the United States of America, Vol. 109, No. 17 (April 24, 2012) pp. i-x, 6355-6783, xi</p><p>Front Matter</p><p>QnAs</p><p>QnAs with W. E. Moerner [pp. 6357-6357]</p><p>RETROSPECTIVE</p><p>Roy J. Britten, 1919-2012: Our early years at Caltech [pp. 6358-6359]</p><p>COMMENTARIES</p><p>Improving biodiversity conservation through modern portfolio theory [pp. 6360-6361]</p><p>Perfecting precision of predicting prion propensity [pp. 6362-6363]</p><p>Crafting a system-wide response to healthcare-associated infections [pp. 6364-6365]</p><p>PNAS PLUS (AUTHOR SUMMARIES)</p><p>䕓删獰散瑲潳捯灹⁩摥湴楦楥猠楮桩扩瑯特⁃疲⁺⁳楴敳⁩渠愠䑎䄭浯摩晹楮朠敮穹浥⁴漠牥癥慬⁤整敲浩湡湴猠潦⁣慴慬祴楣⁳灥捩晩捩瑹⁛灰⸠㘳㘶ⴶ㌶㝝</p><p>Dual functions of the Hsm3 protein in chaperoning and scaffolding regulatory particle subunits during the proteasome assembly [pp. 6368-6369]</p><p>Staphylococcal biofilm-forming protein has a contiguous rod-like structure [pp. 6370-6371]</p><p>Chemical-genetic analysis of cyclin dependent kinase 2 function reveals an important role in cellular transformation by multiple oncogenic pathways [pp. 6372-6373]</p><p>Engagement of ß-arrestin by transactivated insulin-like growth factor receptor is needed for V2 vasopressin receptor-stimulated ERK1/2 activation [pp. 6374-6375]</p><p>The activity of Gli transcription factors is essential for Kras-induced pancreatic tumorigenesis [pp. 6376-6377]</p><p>PERSPECTIVE</p><p>Anthropology of microbes [pp. 6378-6381]</p><p>Shaggy/glycogen synthase kinase 3ß and phosphorylation of Sarah/regulator of calcineurin are essential for completion of Drosophila female meiosis [pp. 6382-6389]</p><p>þÿ�þ�ÿ���S���i���n���g���l���e���-���w���a���v���e���l���e���n���g���t���h��� ���t���w���o���-���p���h���o���t���o���n��� ���e���x���c���i���t���a���t���i���o���n�������s���t���i���m���u���l���a���t���e���d��� ���e���m���i���s���s���i���o���n��� ���d���e���p���l���e���t���i���o���n��� ���(���S���W���2���P���E���-���S���T���E���D���)��� ���s���u���p���e���r���r���e���s���o���l���u���t���i���o���n��� ���i���m���a���g���i���n���g��� ���[���p���p���.��� ���6���3���9���0���-���6���3���9���3���]</p><p>Tuning DNA-amphiphile condensate architecture with strongly binding counterions [pp. 6394-6398]</p><p>Mechanistic insight into the blocking of CO diffusion in [NiFe]-hydrogenase mutants through multiscale simulation [pp. 6399-6404]</p><p>Robust self-replication of combinatorial information via crystal growth and scission [pp. 6405-6410]</p><p>Game-powered machine learning [pp. 6411-6416]</p><p>Role of the Bering Strait on the hysteresis of the ocean conveyor belt circulation and glacial climate stability [pp. 6417-6422]</p><p>Antarctic and Southern Ocean influences on Late Pliocene global cooling [pp. 6423-6428]</p><p>Interfacial electronic effects in functional biolayers integrated into organic field-effect transistors [pp. 6429-6434]</p><p>Greater focus needed on methane leakage from natural gas infrastructure [pp. 6435-6440]</p><p>Contributions of local knowledge to the physical limnology of Lake Como, Italy [pp. 6441-6445]</p><p>Active contractility in actomyosin networks [pp. 6446-6451]</p><p>Thermodynamic glass transition in a spin glass without time-reversal symmetry [pp. 6452-6456]</p><p>Dynamical quantum Hall effect in the parameter space [pp. 6457-6462]</p><p>Superconductive sodalite-like clathrate calcium hydride at high pressures [pp. 6463-6466]</p><p>Power-law decay of the spatial correlation function in exciton-polariton condensates [pp. 6467-6472]</p><p>Fire-free land use in pre-1492 Amazonian savannas [pp. 6473-6478]</p><p>Neural basis of egalitarian behavior [pp. 6479-6483]</p><p>Optimal portfolio design to reduce climate-related conservation uncertainty in the Prairie Pothole Region [pp. 6484-6489]</p><p>Social environment is associated with gene regulatory variation in the rhesus macaque immune system [pp. 6490-6495]</p><p>Crystal structure of Staphylococcus aureus transglycosylase in complex with a lipid II analog and elucidation of peptidoglycan synthesis mechanism [pp. 6496-6501]</p><p>周攠䔮⁣潬椠䍳杂畣汥慴潲映捵牬椠慳獥浢汥猠瑯 눭獨敥琠潬楧潭敲猠瑨慴⁡汴敲⁴桥⁃獧䄠晩扲楬汩穡瑩潮散桡湩獭⁛灰⸠㘵〲ⴶ㔰㝝</p><p>Transcriptional requirements of the distal heavy-strand promoter of mtDNA [pp. 6508-6512]</p><p>Transcription from the second heavy-strand promoter of human mtDNA is repressed by transcription factor A in vitro [pp. 6513-6518]</p><p>De novo design of synthetic prion domains [pp. 6519-6524]</p><p>Entropy-driven binding of opioid peptides induces a large domain motion in human dipeptidyl peptidase III [pp. 6525-6530]</p><p>Single-molecule imaging of DNA curtains reveals mechanisms of KOPS sequence targeting by the DNA translocase FtsK [pp. 6531-6536]</p><p>Introducing endo-xylanase activity into an exo-acting arabinofuranosidase that targets side chains [pp. 6537-6542]</p><p>Transition states of native and drug-resistant HIV-1 protease are the same [pp. 6543-6548]</p><p>Structure of human Mad1 C-terminal domain reveals its involvement in kinetochore targeting [pp. 6549-6554]</p><p>Trigger loop dynamics mediate the balance between the transcriptional fidelity and speed of RNA polymerase II [pp. 6555-6560]</p><p>Role of autophagy in histone deacetylase inhibitor-induced apoptotic and nonapoptotic cell death [pp. 6561-6565]</p><p>Bax regulates primary necrosis through mitochondrial dynamics [pp. 6566-6571]</p><p>Clathrin-mediated endocytosis is inhibited during mitosis [pp. 6572-6577]</p><p>Phosphodiesterases coordinate cAMP propagation induced by two stimulatory G protein-coupled receptors in hearts [pp. 6578-6583]</p><p>䕲扂⁲散数瑯爠瑹牯獩湥楮慳支乆ⴄ㩂⁳楧湡汩湧⁣潮瑲潬猠浡浭潳灨敲攠景牭慴楯渠楮⁨畭慮⁢牥慳琠捡湣敲⁛灰⸠㘵㠴ⴶ㔸㥝</p><p>Differential effects of poly(ADP-ribose) polymerase inhibition on DNA break repair in human cells are revealed with Epstein-Barr virus [pp. 6590-6595]</p><p>Members of the NODE (Nanog and Oct4-associated deacetylase) complex and SOX-2 promote the initiation of a natural cellular reprogramming event in vivo [pp. 6596-6601]</p><p>Reassessment of the 2010-2011 Haiti cholera outbreak and rainfall-driven multiseason projections [pp. 6602-6607]</p><p>Summer temperature variability and long-term survival among elderly people with chronic disease [pp. 6608-6613]</p><p>Footprints of positive selection associated with a mutation (N1575Y) in the voltage-gated sodium channel of Anopheles gambiae [pp. 6614-6619]</p><p>Ecological and evolutionary determinants for the adaptive radiation of the Madagascan vangas [pp. 6620-6625]</p><p>Using translational enhancers to increase transgene expression in Drosophila [pp. 6626-6631]</p><p>Y265C DNA polymerase beta knockin mice survive past birth and accumulate base excision repair intermediate substrates [pp. 6632-6637]</p><p>䱯獳映塌α猠⡥硴牡⵬慲来 녳⤠業灲楮瑩湧⁲敳畬瑳⁩渠敡牬礠灯獴湡瑡氠桹灯杬祣敭楡⁡湤整桡汩瑹⁩渠愠浯畳攠浯摥氠潦⁰獥畤潨祰潰慲慴桹牯楤楳洠䥢⁛灰⸠㘶㌸ⴶ㘴㍝</p><p>Mast cells orchestrate type 2 immunity to helminths through regulation of tissue-derived cytokines [pp. 6644-6649]</p><p>Demyelination reduces brain parenchymal stiffness quantified in vivo by magnetic resonance elastography [pp. 6650-6655]</p><p>Antibody therapy targeting the CD47 protein is effective in a model of aggressive metastatic leiomyosarcoma [pp. 6656-6661]</p><p>The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors [pp. 6662-6667]</p><p>Heritable polymorphism predisposes to high BAALC expression in acute myeloid leukemia [pp. 6668-6673]</p><p>Molecular MRI enables early and sensitive detection of brain metastases [pp. 6674-6679]</p><p>þÿ�þ�ÿ���þ���ÿ�������H��� �����������O��� ����������� �������a�������c�������t�������i�������v�������a�������t�������e�������s������� �������G������� �������p�������r�������o�������t�������e�������i�������n�������,������� �������±������� �������1�������2������� �������t�������o������� �������d�������i�������s�������r�������u�������p�������t������� �������t�������h�������e������� �������j�������u�������n�������c�������t�������i�������o�������n�������a�������l������� �������c�������o�������m�������p�������l�������e�������x������� �������a�������n�������d������� �������e�������n�������h�������a�������n�������c�������e������� �������i�������s�������c�������h�������e�������m�������i�������a������� �������r�������e�������p�������e�������r�������f�������u�������s�������i�������o�������n������� �������i�������n�������j�������u�������r�������y������� �������[�������p�������p�������.������� �������6�������6�������8�������0�������-�������6�������6�������8�������5�������]</p><p>Identification of functionally active, low frequency copy number variants at 15q21.3 and 12q21.31 associated with prostate cancer risk [pp. 6686-6691]</p><p>Structure of the Plasmodium 6-cysteine s48/45 domain [pp. 6692-6697]</p><p>ParA-like protein uses nonspecific chromosomal DNA binding to partition protein complexes [pp. 6698-6703]</p><p>Anticancer drug oxaliplatin induces acute cooling-aggravated neuropathy via sodium channel subtype Nav 1.6-resurgent and persistent current</p><p>[pp. 6704-6709]</p><p>Transcranial direct-current stimulation modulates synaptic mechanisms involved in associative learning in behaving rabbits [pp. 6710-6715]</p><p>Universal conditions for exact path integration in neural systems [pp. 6716-6720]</p><p>þÿ�þ�ÿ���þ���ÿ�������S�������p�������i�������n�������a�������l������� �������1�������2�������-�������l�������i�������p�������o�������x�������y�������g�������e�������n�������a�������s�������e�������-�������d�������e�������r�������i�������v�������e�������d������� �������h�������e�������p�������o�������x�������i�������l�������i�������n������� �������A��� ����������� �������c�������o�������n�������t�������r�������i�������b�������u�������t�������e�������s������� �������t�������o������� �������i�������n�������f�������l�������a�������m�������m�������a�������t�������o�������r�������y������� �������h�������y�������p�������e�������r�������a�������l�������g�������e�������s�������i�������a������� �������v�������i�������a������� �������a�������c�������t�������i�������v�������a�������t�������i�������o�������n������� �������o�������f������� �������T�������R�������P�������V�������1������� �������a�������n�������d������� �������T�������R�������P�������A�������1������� �������r�������e�������c�������e�������p�������t�������o�������r�������s������� �������[�������p�������p�������.������� �������6�������7�������2�������1�������-�������6�������7�������2�������6�������]</p><p>Disruption of the 5-lipoxygenase pathway attenuates atherogenesis consequent to COX-2 deletion in mice [pp. 6727-6732]</p><p>Structural insights into biased G protein-coupled receptor signaling revealed by fluorescence spectroscopy [pp. 6733-6738]</p><p>Kruppel-like factor 15 regulates skeletal muscle lipid flux and exercise adaptation [pp. 6739-6744]</p><p>Redox signal-mediated sensitization of transient receptor potential melastatin 2 (TRPM2) to temperature affects macrophage functions [pp. 6745-6750]</p><p>þÿ�þ�ÿ���P���l���a���n���t���-���a���c���t���i���v���a���t���e���d��� ���b���a���c���t���e���r���i���a���l��� ���r���e���c���e���p���t���o���r��� ���a���d���e���n���y���l���a���t���e��� ���c���y���c���l���a���s���e���s��� ���m���o���d���u���l���a���t���e��� ���e���p���i���d���e���r���m���a���l��� ���i���n���f���e���c���t���i���o���n��� ���i���n��� ���t���h���e��� ���S���i���n���o���r���h���i���z���o���b���i���u���m��� ���m���e���l���i���l���o���t���i�������M���e���d���i���c���a���g���o��� ���s���y���m���b���i���o���s���i���s��� ���[���p���p���.��� ���6���7���5���1���-���6���7���5���6���]</p><p>Spontaneous spatiotemporal waves of gene expression from biological clocks in the leaf [pp. 6757-6762]</p><p>Patient sharing and population genetic structure of methicillin-resistant Staphylococcus aureus [pp. 6763-6768]</p><p>Distinct neuropsychological subgroups in typically developing youth inform heterogeneity in children with ADHD [pp. 6769-6774]</p><p>Effects of self-motion on auditory scene analysis [pp. 6775-6780]</p><p>Correction: Impact of declining Arctic sea ice on winter snowfall [pp. 6781-6783]</p><p>Back Matter</p>