Conserved protein sequence motifs are short stretches of amino acid sequence patterns that potentially encode the function of proteins. Several sequence pattern searching algorithms and programs exist foridentifying candidate protein motifs at the whole genome level. However, amuch needed and importanttask is to determine the functions of the newly identified protein motifs. The Gene Ontology (GO) project is an endeavor to annotate the function of genes or protein sequences with terms from a dynamic, controlled vocabulary and these annotations serve well as a knowledge base.This paperpresents methods to mine the GO knowledge base and use the association between the GO terms assigned to a sequence and the motifs matched by the same sequence as evidence for predicting the functions of novel protein motifs automatically. The task of assigning GO terms to protein motifsis viewed as both a binary classification and information retrieval problem, where PROSITE motifs are used as samples for mode training and functional prediction. The mutual information of a motif and aGO term association isfound to be a very useful feature. We take advantageof the known motifs to train a logistic regression classifier, which allows us to combine mutual information with other frequency-based features and obtain a probability of correctassociation. The trained logistic regression model has intuitively meaningful and logically plausible parameter values, and performs very well empirically according to our evaluation criteria.In this research, different methods for automatic annotation of protein motifs have been investigated. Empirical result demonstrated that the methods have a great potential for detecting and augmenting information about thefunctions of newly discovered candidate protein motifs. With the completion of many genome sequencing projects and advances in the methods of automatic discovery of sequence patterns  project  is a conThe basic approach for determining the function of a motif is to study all the sequences that contain the motif (pattern). Intuitively, if all the functional aspects of the sequences matching a motif are known, we should be able to learn which function is most likely encoded by the motif, based on the assumption that every protein function is encoded by an underlying motif. This means that we would need a knowledge base of protein sequences, in which the functions of a sequence are annotated as detailed as possible. In addition, we would also need prediction methods that can work on a given set of protein sequences and their functional descriptions to reliably attribute one of the functions to the motif that matches these sequences. To determine the function of any novel motif, we would first search the protein knowledge base to retrieve all the functional descriptions of the proteins containing the motif, and then use such prediction methods to decide which function is encoded by the motif. In this research, we use the Gene Ontology database as our protein knowledge base and explore statistical methods that can learn to automatically assign biological functions (in the form of GO terms) to a protein motif.vice versa. However, the database usually returns more than one GO terms that may or may not describe the function of the motif in the query. Thus, we need methods to disambiguate which GO term describe the function of the motif (assign a GO term to a motif) and determine how confident we are as the assignment is concerned. We use statistical approaches to learn from known examples and cast disambiguation task into a classification problem. Furthermore, the probability output by the classifier can be used to represent its confidence for the assignment.Our approach is based on the observation that the Gene Ontology database contains protein sequences and the GO terms associated with the sequences. In addition, the database also contains information of known protein motifs, e.g. the PROSITE patterns that match the sequences. Thus, the protein sequences in the database provide a sample of potential associations of GO term with motifs, among which some are correct  and some are not. This provides us an opportunity to perform supervised learning to identify discriminative features and use these features to predict whether a new association is correct or not. Current Gene Ontology database is implemented with relational database system, which allows one to perform queries like "retrieve all GO terms associated with the sequences that matches a given motif" and p-value. They found that, in the database they worked with, most sequences only had one functional GO term. Therefore, they could assign the GO term of a sequence to the motif that matched with highest score with fairly good accuracy. However, due to restrictive assumption that each sequence has only one GO term, their approach can not address the potential problem that a sequence matching a motif has multiple associated GO terms, which is common case now, and how to resolve such ambiguity.Recently, Schug et al  publisheWe use the May 2002 release of the Gene Ontology sequence database (available online ), which Using the information stored in the Gene Ontology and PROSITE, we manually judged a set of 1,602 cases of distinct PROSITE-GO associations to determine whether the association is correct or not. The PROSITE-GO association set has been judged in two different ways. One way is to label an association as correct if and only if the definition of the GO term and the PROSITE motif match perfectly according to the annotator. Gene Ontology has the structure of a directed acyclic graph (DAG) to reflect the relations among the terms. Most terms (nodes in the graph) have parent, sibling and child terms to reflect the relation of "belonging to" or "subfamily". The second way of judging GO-PROSITE association is to label an association as correct if the GO term and the PROSITE motif are either exact match or the definitions of GO term and PROSITE motif are within one level difference in the tree, i.e., the definition of GO term and the PROSITE motif have either a parent-child relation or a sibling relation according to the GO structure. Thus we have two sets of labeled PROSITE-GO associations, the perfect match set and the relaxed match set (with neighbors). Both sets are further randomly divided into training (1128 distinct associations) and test (474 distinct associations) sets. Since the test sample size is fairly large, the variance of the prediction accuracy can be expected to be small. Thus we have not considered any alternative split of training and test sets.T, and none of the sequences that do not match the motif is assigned the term T, then it is very likely that the motif pattern is encoding the function described by term T. Of course, this is only an ideal situation; in reality, we may see that most of, but not all of the proteins matching a motif pattern would be assigned the same pattern, and also some proteins that do not match the motif may also have the same term. Thus, we want to have a quantitative measure of such correlation between GO terms and motif patterns.Intuitively, we may think of the GO terms assigned to a protein as one description of the function of a protein in one language (human understandable) while the motifs contained in the protein sequence as another description of the same function in a different language . We would like to discover the "translation rules" between these two languages. Looking at a large number of annotated sequences, we hope to find which terms tend to co-occur with a given motif pattern. Imagine that, if the sequences that match a motif are all assigned a term X and Y [p with the expected joint distribution under the hypothesis that X and Y are independent, which is given by p(X = x)p(Y = y). A larger mutual information indicates a stronger association between X and Y, and I = 0 if and only if X and Y are independent.A commonly used association measure is mutual information (M.I.), which measures the correlation between two discrete random variables X and Y . It basiT to a sequence and the matching of a sequence with a motif M as two binary random variables. The involved probabilities can then be empirically estimated based on the number of sequences matching motif M (NM), the number of sequences assigned term T (NT), the number of sequences both matching M and assigned T (NT-M), and the total number of sequences in the database. Table For our purpose, we regard the assignment of a term c to define a simple decision rule: assign term T to motif M, if and only if I ≥ c. For a given cutoff c, the precision of term assignment is defined as the ratio of the number of correct assignments to that of the total assignments according to the cutoff c. In Figure We set out to test whether we can use mutual information as a criterion to assign a GO term to a PROSITE motif. One approach is to use a mutual information cutoff value most likely to be appropriate for the motif. To address this problem, we can use a different cutoff strategy, and adopt a decision rule that assigns a GO term to a motif based on the ranking of mutual information, which is a common technique used in information retrieval text categorization [M in the annotated data set, all observed motif-term associations containing M are retrieved and ranked according to mutual information, then the term that has highest mutual information is assigned to M. Alternatively, if we use this approach to facilitate human annotation, we can relax the rule to include GO terms that have lower ranks, thus allowing multiple potential GO terms to be assigned to a motif, assuming that a human annotator would be able to further decide which is correct. In this method, the key in making a decision is to select a cutoff rank that covers as many correct associations as possible (high sensitivity) while also retrieves as fewer incorrect associations as possible (high specificity). The optimal cutoff can be determined by the desired utility function.However, a drawback of such an approach is that, given a motif, sometimes, many observed motif-term associations can have mutual information above the cutoff value, making it difficult to decide which pair is correct. While in other cases, the mutual information of the observed motif-term pairs may all be below the cutoff value, but we still would like to predict what terms are rization . More spFigure While the mutual information measure appears to give reasonable results, there are three motivations for exploring more sophisticated methods. First, the mutual information value is only meaningful when we compare two candidate terms for a given motif pattern; it is hard to interpret the absolute value. While a user can empirically tune the cutoff based on some utility preferences, it would be highly desirable to attach some kind of confidence value or probability of correctness to all the potential candidate motif-term associations. Second, there may be other features that can also help predict the function (term) for a motif. We hope that the additional features may help a classifier to further separate correct motif-term assignment from wrong ones. Third, there exist many motifs with known functions , and it is desirable to take advantage of such information to help predict the functions of unknown motifs. This means that we need methods that can learn from such information. In this section, we show that the use of logistic regression can help achieve all three goals. Specifically, we use logistic regression to combine the mutual information with other features, and produce a probability of correct assignment. The motifs with known functions serve as training examples that are needed for estimating the parameters of the regression function.T) and PROSITE motif (M) co-occur (NT-M). (2) The number of sequences in which T occurs (NT). (3) The number of sequences in which M occurs (NM). (4) The number of distinct GO terms (G) seen associated with M (NG|M). (5) The number of distinct PROSITE patterns (P) seen associated with T (NP|T). In addition, we also consider, as a feature, the similarity of the sequences that support a motif-term pair. Intuitively, if a motif is conserved among a set of diverse sequences, it is more likely that the motif is used as a building block in proteins with different functions. Thus, the average pair-wise sequence similarity of the sequence set can potentially be used as a heuristic feature in the logistic regression classifier. Given a set of sequences, we use a BLAST search engine to perform pair-wise sequence comparisons. We devised a metric AvgS to measure the averaged pair-wise sequence similarity per 100 amino acids (see methods) and use it as an input feature for classifier.We now discuss the features to be used in logistic regression, in addition to the mutual information discussed in the previous section. The goal is to identify a set of features that is helpful to determine whether association of any pair of a GO term and a motif is correct or not, without requiring specific information regarding the function of GO term and motif. For a distinct motif-term pair, we collect following frequency-based features: (1) The number of sequences in which the GO term , where k is the number of features. The seven features/variables used in our experiments are NT-M, NT, NM, NG|M, NP|T, AvgS, and M.I.. Suppose we have observed n motif-term pairs, then we have n samples of , i = 1, 2, ..., n, where, yi is the correctness label and xi is the feature vector for the corresponding motif-term pair. Our goal is to train a classifier which, when given a motif-term pair and feature vector X, would output a label Y with value 1 or 0. Alternatively, we can also consider building a classifier which outputs a probability that Y = 1 instead of a deterministic label. Thus, our task is now precisely a typical supervised learning problem; many supervised learning techniques can potentially be applied. Here, we choose to use logistic regression as our classification model because it has a sound statistical foundation, gives us a probability of correct assignment, and can combine our features naturally without any further transformation.To cast the prediction problem as a binary classification problem, we augment our data set of motif-term pairs with a class label variable G, given by 2l(D|β f)/l(D|β f-), where, l(D|β f) and l(D|β f-) are the log-likelihood of the model with feature f and the model without feature f, respectively. Since we add or drop one feature at a time, G follows χ2 distribution with degree of freedom of 1 [p-value of 0.1 as a significant threshold. Figure In order to build a model only with the truly discriminative features, it is a common practice to perform feature selection for logistic regression. We use a combined forward and backward feature selection algorithm. Starting from the intercept, we sequentially add features into the model and test if the log-likelihood increases significantly; we keep the current feature if it does. After the forward selection, we sequentially drop features from the model, to see if dropping a feature would significantly reduce the log-likelihood of the model; if it does, we exclude the feature from the model, otherwise continue. When testing the significance, we use the likelihood ratio statistic dom of 1 . We use p(Yi = 1|Xi) for each test case. Table After fitting the model using the training set, we tested the model on the test set, i.e., we used the model to compute an output Y = 1, thus, a small change in the number of cases introduces a large change in percentage. For example, when the threshold is set at 0.9, only three cases are covered by the rule and two of them are correct, thus percent correct drop to 66%.As the results from the logistic regression are the conditional probability that an association of a GO term with a given motif is correct, we need to decide the cut off threshold for making decision. We calculate the sensitivity and specificity for a different threshold from 0.1 to 0.9 with a step of 0.1 and plotted the ROC curves as shown in Figure C and D. We see that using mutual information alone performs almost as well as logistic regression with additional features. However, the area under the curve (0.816) is smaller than that of logistic regression (0.875), indicating that logistic regression does take advantage of other features and has more discriminative power than mutual information alone.To see whether the additional features are useful, we also performed ROC analysis using different mutual information cutoff threshold on the perfect match test set. The result is shown in Figure β1, β2 and β3 for the three features NT-M, NT and NM, which are also involved in the calculation of mutual information, have a very interesting interpretation – they indicate that the roles of these three variables in the logistic regression model actually are to compromise the effect of mutual information! Indeed, according to the formula of the mutual information, a strong correlation corresponds to a high NT-M, low NT, and low NM, but the coefficients shown in Table NT) with respect to any pattern matched by the sequence to which the term is assigned. But, intuitively, one occurrence is very weak evidence, and at least should be regarded as weaker than when we have a term occurring 10 times in total and co-occurring 9 times with the same motif. The key issue here is that mutual information only reflects the correlation between variables, but does not take into account the strength of evidence, therefore, tends to over-favor the situation where there is a perfect correlation but very little evidence. However, the number of sequences in which the co-occurrence happens, which is called the "support" for the association, is also very important.The coefficients NG|M and NP|T, are also meaningful. Their negative signs indicate that the more terms a motif co-occurs with or the more motifs a term co-occurs with, the less likely a particular association is correct. This also makes sense intuitively, since all those co-occurring terms can be regarded as "competing" for a candidate description of the motif's function, so the more terms a motif is associated with, the competition is stronger, and thus the chance that any particular term is a correct description of function should be smaller. Thus, the logistic regression model not only performs well in terms of prediction accuracy but also gives meaningful and logically plausible coefficient values.The coefficients for the other two parameters, In this paper, we explore the use of the Gene Ontology knowledge base to predict the functions of protein motifs. We find that the mutual information can be used as an important feature to capture the association between a motif and a GO term. Evaluation indicates that, even used alone, the mutual information could be useful for ranking terms for any given motif. We further use logistic regression to combine mutual information with several other statistical features and to learn a probabilistic classifier from a set of motifs with known functions. Our evaluation shows that, with the addition of new features and with the extra information provided by the motifs with known functions, logistic regression can perform better than using the mutual information alone. This is encouraging, as it shows that we can potentially learn from the motifs with known functions to better predict the functions of unknown motifs. This means that our prediction algorithm can be expected to further improve, as we accumulate more and more known motifs.new motifs e.g., those discovered using TEIRESIAS, SPLASH or other programs. This would further enable us to perform data mining from the Gene Ontology database in several ways. For example, we can hypothesize the functions of a large number of novel motifs probabilistically, then we will be able to answer a query, such as "finding the five patterns that are most likely associated with the GO term tyrosine kinase". This is potentially very useful because it is not uncommon that substantial knowledge about the functions and sub-cellular location of a given protein is available even though a structural explanation for the functions remains obscure. On the other hand, we believe that our methods will facilitate identifying potentially biological meaningful patterns among the millions of patterns returned by pattern searching programs. A sequence pattern that associates with certain GO term with high M.I. or probability is more like to be a meaningful pattern that that with low scores. Furthermore, our methods can also be used in automatic annotation of novel protein sequences as suggested in Schug et al and Rigoutsos et al [Although we have so far only tested our methods on the known motifs, which is necessary for the purpose of evaluation, the method is most useful for predicting the functions of new and unknown motifs. For the future work, we can build a motif function prediction system and apply our algorithm to many candidate os et al ,13,22. OHaving stated the potential uses of our approaches, we also realize that there exist some limitations for our methods. For example, in order to predict the function of a newly identified sequence pattern correctly, we would require functional annotations of the sequences of GO database be complete and accurate, which may not always be the case. In this paper, we mainly used the motifs with known function to evaluate the capability of the methods developed in this research. Our result shows that the methods work well with known sequences patterns. Currently, the annotation of motif function with GO term is carried out manually at the European Bioinformatics Institute (the GOA project). Such approach is warranted because human annotation is more accurate than automatic ones. However, as the amount of information regarding protein functions accumulates and a large number of new potential motifs are discovered, it will be very labor intensive to annotate the potential association of protein function and protein patterns. By then, the methods studied in this research will potentially prove to be useful to discover the underlying protein motifs that are responsible for the newly annotated function. For example, the methods can be used as prescreening to narrow down to the most possible associations of protein function and motifs, thus facilitate human annotation.In summary, we have developed methods that disambiguate the associations between of Gene Ontology terms and protein motifs. These methods can be used to mine the knowledge contained in the Gene Ontology database to predict the function of novel motifs, discover the basis of a molecular function at primary sequence level and automatically annotated the function of novel proteins.Mutual information is defined as followsp, p(X = x) and p(Y = y) can be empirically estimated from the data by counting occurrence/co-occurrence followed by normalization.In which the probabilities The sensitivity and specificity of the rules are calculated asTP (True Positive) is the number of associations labeled as correct among the retrieved motif-term pairs meeting the ranking cutoff criteria, FN  is the number of associations labeled as correct but not retrieved, TN (True Negative) is the number of associations labeled as incorrect and not retrieved, and FP  is the number of associations labeled incorrect but are retrieved.where AvgS) of a sequence set is as followsCalculation of the average pair-wise sequence similarity per 100 amino acids  is a delta function which equals 1 if i = j and 0 otherwise.Where Sp(Y = 1|X) and X1, ..., Xk:The logistic regression model is a conditional model that assumes the following linear relationship between β =  is the parameter vector. We can fit the logistic regression model  using the Maximum Likelihood method – essentially setting the parameters to values at which the likelihood of the observed data is maximized . In our experiments, we use iteratively reweighted least squares (IRLS) algorithm [where, lgorithm  to fit t


The Mammalian Phenotype (MP) Ontology enables robust annotation of mammalian phenotypes in the context of mutations, quantitative trait loci and strains that are used as models of human biology and disease. The Mammalian Phenotype (MP) Ontology enables robust annotation of mammalian phenotypes in the context of mutations, quantitative trait loci and strains that are used as models of human biology and disease. The MP Ontology supports different levels and richness of phenotypic knowledge and flexible annotations to individual genotypes. It continues to develop dynamically via collaborative input from research groups, mutagenesis consortia, and biological domain experts. The MP Ontology is currently used by the Mouse Genome Database and Rat Genome Database to represent phenotypic data. Mammalian phenotypes are complex and the term itself is imprecise. Generally, we use the word phenotype in referring to the appearance or manifestation of a set of traits in an individual that result from the combined action and interaction of genotype and environment.Because mouse is the premier model organism for the study of human biology and disease, the goal of comparative phenotyping and building new animal models through genetic engineering holds great promise. The mouse has distinct advantages for studies that translate to humans. It is a small, short-lived mammal with a fully sequenced genome in which all life stages can be accessed, and for which myriad tools are available for precisely experimentally manipulating its genome. Further, the large collection of inbred strains of mice and the controlled environment in which the animals live provides the ability to confirm phenotype observations and to systematically perturb environmental factors and genetic input to measure effects under defined conditions. Current international efforts to 'make a mutation' for every gene through mutagenesis  and geneMammalian phenotypes are frequently genetically complex. Mutation of even a single gene almost always produces pleiotropic effects. Conversely, non-allelic mutations can produce indistinguishable phenotypes. Modifier genes and epistatic interactions can markedly alter the phenotype. Combining different allelic combinations of different genes can produce unique phenotypes not found in the single-gene mutation genotype. Imprinting of genes can dramatically affect phenotype. Mutations expressed in different inbred strains of mice can manifest as an increase or decrease of severity or penetrance of the corresponding phenotype. Quantitative trait loci (QTL) can contribute in complex nonlinear ways to the phenotype. In addition, mutations that are 'genomic' in nature, either disrupting or deleting multiple genes or occurring in intergenic regions, can produce distinct phenotypes and challenge us to think beyond gene effects to genomic effects. The outcome of these complex interactions can be dissected and reproducibly examined by characterizing inbred strains that represent the combined phenotype of the 'whole-genome' genotype in its environmental context.The Mouse Genome Database (MGD) at the Mouse Genome Informatics website ,5 servesWritten descriptions of phenotypes in higher organisms reflect the complexity of the subject, the richness of language, and the phenomenal diversity that these data represent. While text descriptions are commonly used in publications describing phenotype, and have been the basis of electronically accessible phenotypic descriptions   and the Consider the example in Table A further detriment to database text records is their difficulty to update and maintain. As new information is learned about a phenotypic mutant, the record must be continually rewritten. Although this practice might be sustained for a small number of records, it does not scale when thousands of mutant records are considered. The alternative of simply adding on another paragraph to existing text records becomes confusing, with potentially conflicting information and different writing styles appearing in one textual description, and unwieldy, with more and more text that may no longer represent a logical synthesis.Formal nomenclatures for genes, mutant alleles and inbred strains of mice have existed since the 1940s ,9. The MBeyond nomenclatures, which are key to object identities and relationships, are vocabularies that can be used to describe broader concepts and categorizations. Vocabularies can take many forms, including simple lists of controlled terms, such as the cytogenetic band designations used to name the bands defined by chromosome staining or the classes of genetic markers, such as gene, pseudogene, expressed sequence tag (EST), and so forth.The annotation of complex biological data and concepts requires more than lists and simple vocabularies. Ontologies, or 'descriptions of what there is', contain both concepts, with precise meanings, and relationships among those concepts. As such ontologies are able to support descriptions of complex biology and are useful in making these data more amenable to computational analyses. The first widely used ontology developed and adopted in the biological domain is the Gene Ontology (GO) -13 whichAlthough the need for vocabularies as key components to consistent phenotype annotations for mammals has been recognized for some time , and manOur goal is to describe the richness of phenotypes as precisely as they are known, recognizing that phenotype data are by nature complex and usually incomplete. Taking advantage of structural properties of a DAG, we have the ability to annotate phenotypes to the level of data resolution available, whether general or very specific and the ability to query with a high-level term, returning all phenotypes containing annotations to that term or to terms more specific than the query term. Thus, one can query for 'respiratory signs/symptoms' and retrieve all phenotypes annotated to this term and its hierarchical 'children' , or specifically request annotations to any of these sub-terms.The top level terms of the MP Ontology include physiological systems, behavior, developmental phenotypes and survival/aging. Physiological systems branch into morphological and physiological phenotypes at the level immediately below. A browser to view the ontology is available at  Figure . In thisEach MP ontology term has a unique identifier, a definition and synonyms. In the term detail pages, these data and the number of hierarchical paths of the vocabulary where the terms appear are displayed. A plus sign following the term indicates that children of this term exist. In this figure, displayed next to the term, is a link indicating the number of annotation instances in MGD using this term or children of this term. This feature, due to be publicly available in early 2005, will greatly improve phenotype-centric searching in MGD.To initiate the vocabulary, we first developed a high-level categorization of phenotypes consisting of approximately 100 terms, such as heart/cardiovascular dysmorphology and skeletal axial defects. As we used this list for annotations, terms were refined and general organizing principles for the MP vocabulary were developed.An important component of our approach has been to address two practical implementation questions. From the biologist's perspective, the question is what term would be used to describe a specific phenotypic trait. From the curation perspective, we ask what terms reflect biological reality and maximize curator productivity.From a purely ontological perspective, every trait could be broken down into a core object, such as 'cornea' or 'gastrulation', defined by anatomical, behavioral or physiological terms, and a series of attribute vocabularies that describe the quality, quantity and character of a trait. For the practical reason of needing robust terms to describe phenotypes up-front to speed curation and the problem of losing biological meaning, particularly for clinical or dysmorphology terms, when terms are completely deconstructed , we have chosen to use compound terms in the MP Ontology. A few examples of terms where it is difficult to preserve the full biological meaning once they are deconstructed are shown in Table pheno@informatics.jax.org.Three major strategies are being pursued to further develop the vocabulary itself. First and most important is through the ongoing process of curating phenotype data. As new phenotypic traits are described and published, the need for new terms is recognized. New terms added in this way may be a simple addition to an existing hierarchical path or may result in the addition of entire new branches in the hierarchy. Second, collaborative efforts between the MGD phenotype curators, the mouse mutagenesis centers and the rat genetics community identify new specific terms and suggest improved organization of terms within particular hierarchical branches. Third, we are recruiting individuals with expertise in specific biological domains to review and evaluate sections of the vocabulary for accuracy, completeness and systematic arrangement. The MP Ontology is a work in progress and remains incomplete in some areas. We welcome the participation of the mammalian research community so that the most useful, definitive and universally applicable terms will be included. Information can be obtained by sending e-mail to While common pathological and clinical terms are used in the MP Ontology, considerations for term placement within the structure and for precise terminology is often derived from comparison with other open biological ontologies (OBO) . RecentlThe MP Ontology was built as a DAG using the DAG-Edit software written by John Richter and Suzanna Lewis . The MP Phenotypes are described in the MGD relative to the genotype of the individual. Genotype objects specifically consist of one or more allele pairs describing mutations or QTLs and the genetic background strain(s) where the phenotype was observed. Each phenotype annotation associates a MP Ontology term with a genotype/strain and the reference or data source supporting this assertion. Additional modifying text may be annotated to describe detail that is not easily standardized. Examples include experimental conditions, age of onset and incidence, and trait penetrance, among others. The annotation note may also include specifics of the phenotype where such details are deemed to be too case specific to be a MP term. In addition, genotypes are associated with OMIM where a particular mouse genotype is a model for human diseases and syndromes. Figure The MP Ontology and annotation schema was designed to minimize curatorial time, yet remain precise enough to describe phenotypic data. It supports robust phenotypic annotations and querying capabilities for mouse phenotype data. While this vocabulary is far from complete, we have designed strategies for its continued development as a collaborative effort for supporting the representation of existing mutations and those that continue to be created.As of 1 November 2004, over 11,150 phenotypic alleles representing mutations in 5,214 unique genes had been catalogued in MGD. For these alleles, 9,696 genotype records exist, with 21,556 phenotypic annotation instances. The MP Ontology is also used in phenotypic data annotations at the RGD . As our


The function of a novel gene product is typically predicted by transitive assignment of annotation from similar sequences. We describe a novel method, GOtcha, for predicting gene product function by annotation with Gene Ontology (GO) terms. GOtcha predicts GO term associations with term-specific probability (P-score) measures of confidence. Term-specific probabilities are a novel feature of GOtcha and allow the identification of conflicts or uncertainty in annotation.Plasmodium falciparum and six other genomes. GOtcha was compared quantitatively for retrieval of assigned GO terms against direct transitive assignment from the highest scoring annotated BLAST search hit (TOPBLAST). GOtcha exploits information deep into the 'twilight zone' of similarity search matches, making use of much information that is otherwise discarded by more simplistic approaches.The GOtcha method was applied to the recently sequenced genome for E-value cutoff of 10-4.At a P-score cutoff of 50%, GOtcha provided 60% better recovery of annotation terms and 20% higher selectivity than annotation with TOPBLAST at an Plasmodium falciparum annotation and is being adopted by many other genome sequencing projects.The GOtcha method is a useful tool for genome annotators. It has identified both errors and omissions in the original  It is now often possible to obtain the complete genome sequence of an organism in a few months, but without a directed approach, determining the function of potential gene products through biological experimentation is inefficient. Accordingly, methods for function prediction are required to direct experiments in function verification. In the context of this paper the term function is used to refer to all aspects of a gene product's behaviour. This includes the concepts described by the Gene Ontology classifications for Molecular Function, Biological Process and Cellular Component. It is explicitly stated in the text when a more specific interpretation of function is intended.A powerful tool in the annotation of novel genomes is the prediction of function by similarity to a sequence of known function. Such 'transitive function assignment' can work very well where there is a clear match to a homologue with a well established function. However, accurate functional assignment is difficult in cases where the match is less well defined, either due to lower sequence similarity or the presence of many candidates with differing functions. Gerlt and Babbitt  reviewedKeywords and restricted vocabularies do not solve the problem of conflicting assignments. Unless some computable form of relationship is present between terms, it is not possible to provide any automated form of conflict resolution between terms or to identify computationally where one term is a more specific descriptor than another.An ontology represented as a graph can provide a solution to this problem. Ontologies are restricted vocabularies, or sets of terms where each term is explicitly related to parent terms and child terms (and hence to sibling terms). The Gene Ontology (GO)  is a desThe availability of the Gene Ontology has provided for the first time, a broadly accepted classification system for function assignment that can be analysed computationally. Previous work using other classification schemes, such as restricted vocabularies based on SwissProt keywords, suffered because of the lack of a distinct relationship between terms and/or due to typographical differences ,7. SinceXie and coworkers  have comE-value of the pairwise match. GOblet [Two tools based on BLAST searches have recently been presented in the literature. OntoBlast  provided. GOblet  also appPlasmodium falciparum) [In this paper we present a novel method, GOtcha, that can be applied to any database search technique that returns scored matches. We have initially implemented this with BLAST searches and extend the analysis from the similarity match scores for a search in order to provide an empirical estimate of the confidence in each predicted function. We have applied this method to Malaria (ciparum)  and six The assessment of the global accuracy of a particular annotation method is extremely problematic in the absence of a computable annotation scheme. Gene Ontology provides such a computable scheme and we present here a quantitative measure for comparison of function annotations based on assignment to GO terms. This provides a metric for direct objective comparison of annotation methods that is independent of arbitrary cut off values. The new accuracy measure encompasses true positives, false positives and false negatives, so combining sensitivity and selectivity in one value.Two sets of annotation predictions were determined for each data set in the study. One was based on all available GO annotations and the other on a reduced set of GO annotations that excluded gene-associations with the evidence code IEA (Inferred by Electronic Annotation). IEA annotations are usually considered to be less reliable as they have not been assessed by a human curator. In contrast, ISS annotations (Inferred from Sequence Similarity) are annotations which, whilst being derived electronically, have been assessed by a human curator and can be considered sufficiently reliable. IEA annotations may however give a broader coverage than non-IEA annotations. On average, each dataset contained slightly more than 50% IEA annotations, though the vast majority of the sequences had some non-IEA annotation. The number of sequences for each dataset is listed in Table y-axis indicates the proportion of annotations provided by the genome project (given annotations) that were annotated to some degree by either GOtcha . This E-value cutoff is at the top end of the E-values between 10-4 and 10-20 typically used as a threshold for confident function assignment [E-values.Figure a Figure  or TOPBLa Figure . At a P-signment -20. The x-axis representing the minimum P-score. A low P-score represents low confidence in the annotation. A high P-score represents high confidence in the annotation. Figures x-axis representing the maximum E-value. A low E-value represents high confidence in the annotation. A high E-value represents low confidence in the annotation. In Figure x-axis approaches the maximum relatively quickly when moving from high P-score to low P-score, typically coming very close to the total number of sequences annotated well before the P-score has dropped to 50%. This represents a broad coverage of sequence space, assigning annotation at a relatively nonspecific level to most sequences. In terms of the total number of annotations, these rise steadily as the P-score cut off drops. At very low P-scores (below 10%) the total number of annotations increases rapidly, indicating an increase in the spectrum of functions matched with only weak similarity. The number of annotations per sequence increases gradually as the P-score drops until a rapid rise at low P-scores . Figure E-value below the cutoff on the x-axis. At an E-value of 10-4 TOPBLAST shows a selectivity of 53.4% (43–60% s.d. 5.7). Accordingly, GOtcha outperforms TOPBLAST with improved coverage and better selectivity for each genome examined. Both the GOtcha and the TOPBLAST analyses include gene associations that are children of obsolete (GO:0008369) and the three 'unknowns' . The obsolete terms comprise a very small proportion (1.5% mean 0 – 3.1% s.d. 1.1) of the total number of annotations  of the number of annotations recovered by GOtcha.Function assignment was repeated using the same BLAST search results but excluding the IEA coded gene-associations. Figure The number of annotations per sequence was reduced by comparison to the data shown in Figure x-axis. Figure E-value below the cutoff on the x-axis. GOtcha with a P-value cutoff of 50% shows a selectivity of 60% (35–79% s.d. 14). TOPBLAST with an E-value of 10-4 shows a selectivity of 49% (25–59% s.d. 11). In all cases except that of Arabidopsis GOtcha shows a clear improvement over TOPBLAST with a mean improvement in selectivity of 1.2 fold (0.85 – 1.4 s.d. 0.17).Figure One issue with excluding IEA annotations is that the coverage of functions in the genome is lowered. This inevitably will lead to a higher number of positives that have incorrectly been assigned as false as a result of the incomplete sequence annotations. Despite excluding terms for which there is no annotation to the ontology under examination, the results are skewed by assigning a proportion of true positives as false positives. This indicates that the method is performing more poorly than is in fact the case. We have examined the nature of the false positives in more detail below.Comparing function assignment methods is difficult. Typically the standard against which they are assessed is an incompletely annotated dataset. Both a lack of experminental data confirming potential functions and a lack of knowledge about potential functions can lead to the standard data being less perfectly annotated that would be desired. It is not realistically possible in an automated analysis to cope with unrecorded true positives that are registered in the analysis as false positives. It is therefore the case that any analysis of accuracy can only give an estimate of minimum accuracy.Accuracy can also be difficult to compare between two methods that annotate to different subsets of GO. One method may only annotate to relatively general terms, allowing for a better claimed specificity than a method that attempts to annotate to a more specific level. GOtcha predicts at all levels of the GO hierarchy. It assigns a probability to every combination of GO term – sequence association and should be compared to other function assignment algorithms using a global metric, one which can account for over-specificity and under-specificity in a set of predictions as well as incorrect assignment.Ouzounis and Karp  describeWhen applied to annotation using a DAG such as GO the number of potential categories is reduced from the eight described in TABS to three. TABS was developed to compare annotations where the terms used are not implicitly related through a computable structure such as a DAG. As we are using a DAG where ancestor nodes are implicitly associated with the gene through direct association of a child node, the prediction for a particular sequence becomes a set of GO terms (the nodeset) comprising all nodes that match the prediction rather than just the most specific terms. The accuracy of a prediction can then be assessed by observing the presence of nodes in both the node sets for annotations and for the predictions rather than assigning qualitative values. The more distant a given prediction node set is from the annotation node set, the smaller a proportion of nodes (GO terms) they will have in common.The effect of a quantitative approach on the TABS categories is as follows: TABS category 0 is unchanged. This is an exact match and is represented by the presence of the term in the node sets for both original annotation and current prediction. TABS category 1 is no longer relevant. A controlled vocabulary is being used so there is no scope for typographical errors of the type described by Ouzounis and Karp or by Tsoka  or Iliopn ∈ A ∩ B), false positives  and the false negatives . The aim of any prediction method is to maximise the number of matches (true positives) whilst minimising the errors . The number of true negatives does not need to be considered as this number is very large and essentially constant over the analysis. We can use the following relation as an error quotient to assess prediction methods.Given two sets A and B corresponding to a given annotation set and a predicted set (each node in the set comprising a sequence – GO term association) we are interested in the true matches  has been determined for both GOtcha and top BLAST hit annotation sets, both with and without the use of automated annotations (IEA evidence code) for transitive function assignment . It may be that the annotation set used as the reference in comparing these results was incomplete. This would result in some true positives being incorrectly assigned as false positives with a corresponding increase in REQ. However, this would apply similarly to GOtcha and to the top BLAST hit analysis.Minimum nt Table . When aud Figure  and 0.12P. falciparum chromosomes  were assessed by hand to give an indication of the completeness of the curated annotations. Results for selected sequences in this set are shown in additional file Samples of the false positive function predictions by GOtcha with the highest P-scores from three P. falciparum genome project if no function had been identified during the first-pass automatic annotation. However, the addition of GO terms to sequences by GOtcha prompted the original annotation to be re-evaluated. For example, PFL1875w shows a hit to the Pfam K+ tetramerisation domain  supports the GOtcha annotation although it is at a level that genome annotators may feel is marginal. In PFL1780w, stronger supporting evidence  indicates again that GOtcha can suggest GO annotations that have been previously overlooked.In some examples, genes that were annotated as encoding hypothetical proteins could be re-annotated based on GOtcha predictions. GO terms had not been assigned during the manual curation phase of the In several examples, GOtcha predicted either additional functions or more specific GO terms to describe previously annotated functions. PFC0495w encodes a putative aspartyl protease. When all evidence codes were included, a molecular function of pepsin A activity is predicted. This protein matches pepsin A domains defined by the InterPro entry IPR001461 ('Peptidase_A1 pepsin A'), thus the term from GOtcha is likely to be correct.PFL2465 encodes a thymidylate kinase, which was correctly annotated by GOtcha as being involved in dTTP biosynthesis. GOtcha also indicates 'dTDP biosynthesis' as a suitable GO process term. Thymidylate kinase catalyses the synthesis of dTDP, a necessary step in dTTP biosynthesis. However, the human annotator missed the fact that dTDP biosynthesis is not a 'part of' dTTP biosynthesis within the ontology structure and in such cases, terms describing both processes must be employed.P. falciparum genome, many of which have arisen from retrospective corrections and amendments to gene models. For instance, GOtcha provides detailed annotation for a putative ATPase synthase F1 alpha subunit (PFB0795w) almost completely lacking useful GO terms. GOtcha also suggested GO terms relating to translation elongation for PFL1710c. A highly significant hit to Pfam:00009  indicates that this GOtcha prediction may well be more accurate then the original genome annotation.Sometimes, GOtcha highlighted erroneous omissions in the GO annotation of the Annotations performed with IEA terms appeared to be more specific than those where IEA terms were excluded. In many cases, such as PFC0495w, the difference was quite pronounced. Here the protein was implicated in 'proteolysis and peptidolysis' when all annotations were included but filtering out IEA annotations resulted in the more general, and less useful, description of 'metabolism'.Out of the 20 genes inspected, PFL1825w was the only example where GO terms were incorrectly suggested for the biological process, molecular function and cellular component aspects of GO. In other cases, mis-annotations often had low I scores (predictions made with P-scores > 50% but very low associated I-scores ≪ 0.1) or were due to terms taken from slightly too far down a branch in the ontology structure. For example 'ATP-binding and phosphorylation-dependent chloride channel' was predicted for PFB0795w, an ATP synthase.The cellular component of gene products are hard to annotate – often BLAST is insufficient to recognise the targeting information encoded in signal and transit peptides and specific signal sequence detection methods such as PSORT II  must be It is hard to measure what proportion of the calculated false positives does in fact represent serious mis-annotation. Although the hand analysis may provide representative examples, it is too small to be of statistical significance. Genuine false positives (with high P- and I-scores) were fewer than would be expected from the P-score. Despite the small sample size, these results show that GOtcha performs well as a guide to the manual assignment of GO terms. Not only can it provide suggestions for more granular annotation but it can highlight terms that would otherwise be missed by a human annotator.REQ goes up. This may well indicate a degree of inaccuracy in the IEA based annotations, or incomplete coverage by the human curated annotations. GOtcha, however, makes a significantly better use of the BLAST search result in the quality and coverage of the annotation.One of the major problems facing assessment of function assignment is the separation of annotation and test datasets. In this analysis we have tackled this issue by taking individual genome datasets as the test sets and using other genome datasets for the annotation source from which to transitively assign function. The scoring mechanism used for estimating accuracy values is independent of both test and annotation datasets, since it makes use of sequences that are found in neither. Whilst the sequences are independent, the annotations associated with these sequences may not be. Many of the computationally assigned annotations are derived from analyses involving the 'independent' datasets and can therefore not be regarded as entirely independent. IEA annotations are primarily obtained from sequence similarity searches. As a consequence it is not surprising that the results obtained for both GOtcha and TOPBLAST when IEA annotations are included are so similar. Interestingly, when IEA based annotations are excluded from the TOPBLAST analysis, the REQ, the GOtcha predictions for the human genome were compared to the genome consortium annotations with weights ranging from 0.5 to 15  equally to over prediction errors . In order to examine the effect of the weighting on 5 Figure . As expeREQ as described in this paper, each term in the nodeset is weighted equally. This may not be the most appropriate measure. The granularity of terms in Gene Ontology is not constant across the ontologies, nor is it readily quantifiable. This may lead to bias in the metric, where differences in the presence or absence of closely related terms is weighted equally to presence or absence of more distantly related terms even though they have the same graph path distance between them. There is also the issue of prevalence. Some terms occur in almost every nodeset, others are less prevalent. The most appropriate form for a quantitative metric will need to be examined in future work.The metric presented here is an objective measure of method performance but has some drawbacks. Using the Transitive function assignment is limited by the sensitivity of the underlying search method and the scope of the dataset being searched. The GOtcha method of preparing a weighted composite view of the functions from a complete set of search results provides a significant improvement in the annotation of sequences when compared to a method that selects the most significant annotated hit. GOtcha also provides a confidence measure for the putative function assignments, allowing for the determination of an appropriate level of specificity for the annotation set. Hennig and co-workers examined the ability of BLAST analysis to transitively assign function from distant taxa, concluding that for the majority of cases, GO-based annotation would give a good result . In thisThe GOtcha method has several significant advantages over the transitive assignment of function by TOPBLAST. Firstly each function assignment has a directly understandable accuracy estimate that can be interpreted without any knowledge of the prediction methodology. This accuracy estimate is function-specific, unlike general rules of thumb that are applied to interpretation of BLAST search results. Secondly, the GOtcha method provides much greater coverage than a top annotated match approach, annotating more sequences with reasonable confidence. In many cases it provides annotations for sequences that otherwise would have no annotations. Finally, it provides term specific annotation accuracy estimates. This is a significant advantage over TOPBLAST where every term in the set predicted for an individual sequence has the same value and a biologist interpreting the results is given little indication of which terms can reasonably be accepted. In contrast, GOtcha provides individual P-scores for each term. This allows a rapid visual examination of the prediction as a graph or a list, indicating appropriate points at which experimental verification may best be directed.In order to assess the accuracy of annotations to tree-like ontologies we have developed an objective flexible scoring metric that provides a global analysis, including assessment of both false positives and false negatives. This metric also provides a means for comparison of methods that is not dependent on the selection of any particular parameter threshold or cutoff in the scoring method used.The underlying mapping methodology applied in GOtcha can readily incorporate other search methods that provide a more sensitive similarity search. Combining search methods should also provide a better coverage of sequence space occupied by distant homologues , and sucAll data were obtained in the same week  to provide a consistent time point at which to perform the analysis.Plasmodium falciparum) sequence data for the recently determined genomic sequence [Malaria (sequence  were obtDrosophila melanogaster) data were obtained from Flybase [Fruit fly ( Flybase  as releaSacchyromyces cerevisiae) data were obtained from the Sacchyromyces Genome Database [Yeast (Database . The setVibrio cholerae data were obtained from The Institute for Genomic Research [Cholera Research . The datHomo sapiens) data were obtained from Swiss-Prot using the conceptual complete human proteome from the Swiss-Prot/EnsEMBL collaboration dated 6 March 2003 [Human (rch 2003 . The datCaenorhabditis elegans) data were obtained from Wormbase release 97 [Worm  data were obtained from The Arabidopsis Information Resource [Thale cress . Results were stored in a relational database (PostgreSQL version 7.3) or as flat files where appropriate. BLAST result parsing was performed with the BioPerl toolkit (release 0.7) . SequencD. melanogaster was the query/subject set, blastp otherwise). Default parameters were used . Each sequence database search produces a ranked set of sequences similar to the query sequence. The search result for each genome database search is parsed and a list of pairwise matches between the query sequence and the subject database sequences obtained.The GOtcha method is illustrated by a cartoon in Figure 10(E), 0 } where E is the expectancy score for that pairwise match. In this way the whole subtree to the root node is assigned the R-score. The GOtcha method allows mappings obtained from many sequence matches to be combined. For each node , R-scores for all pairwise matches which contain annotation to that node are summed and normalised to the total R-score for the root node of that ontology . This normalisation gives an internal relative score (the I-score), producing a weighted composite subgraph of the GO. This normalisation effectively removes bias in the E-value due to database size or search program used. A confidence measure is calculated as loge of the root node score (the C-score). Accordingly, this provides two measures for an individual predicted gene-association; A score relative to the other predicted gene-associations in the node set (the I-score) and a score for the function prediction as a whole (the C-score).For each similarity match between the query sequence and a database sequence, a set of GO terms corresponding to the gene-associations for the database sequence is retrieved from the appropriate gene-association dataset. The set of GO terms and all ancestral terms (the nodeset) are assigned a score R = max { -logEach genome was searched individually and I-score and C-score for each GO term association were averaged across all genome searches that provide at least one annotated pairwise match. Averaging across genomes in this way provides some correction for individual genes with exceptionally high copy numbers in certain genomes. In this paper the term 'function prediction' relating to an individual sequence refers to a prediction of a set of GO term – sequence associations . Averaging of the individual search results avoids the over-representation of large genomes in the final annotation set and allows the final result to be weighted towards a particular taxonomic grouping should that be desired. Each gene association represents a function assignment of a gene product with a GO term and is annotated with an evidence code providing an indication of the reliability of a particular annotation. The GOtcha method allows specific classes of annotation, such as those derived exclusively from computational analyses, to be excluded from the analysis if required.Although higher C-score and I-score values correspond to greater confidence in the transitive assignment of function than lower C-score or I-score values, it is not immediately apparent how these values should be interpreted. Examination of preliminary results indicated that there was considerable variation between GO terms in the confidence that can be placed in a prediction with a given I-score and C-score (data not shown). Accordingly we have created an empirically based estimate of accuracy  that can be used to indicate confidence in the prediction of association between a GO term and a gene product.A background set of 518226 annotated sequences from the SwissProt gene associations were included in the accuracy estimate after excluding taxa corresponding to the search databases and their subspecies. All background sequences were subject to a search against all 7 species specific datasets and a set of function predictions obtained as described above. A scoring table for each GO term was prepared by segregating all predictions for that GO term on I-score and C-score. I-scores were divided into ten rows by dividing the range (0 – 1) evenly. C-scores were divided into columns by unit ranges . This gave rise to approximately one hundred cells for each GO term table. Each prediction was assigned to a cell based upon its I-score and C-score. The overall accuracy of each cell was determined by comparison of the predicted associations in that cell to the annotations provided by the GO Annotation project (GOA) and calculated as the proportion of true positives to the sum of true and false positives. The table for a specific GO term was then used to deliver the P-score based on any given I-score and C-score pair for a predicted association between that GO term and the query sequence. A similar set of tables was constructed from background analyses from which terms with IEA associations were excluded. For GO terms where there are few datapoints with which to estimate accuracy reliably, accuracy estimation falls back to a scoring table that combines results over all GO terms from that ontology with the same number of ancestors.E-value for that hit.The same BLAST searches used for function assignment with the GOtcha method were analysed. Function assignments for the nodeset corresponding to the top annotated BLAST match (TOPBLAST) for each genomic dataset were transferred to the query sequence with a score corresponding to the DAG, Directed Acyclic Graph. URI, Uniform Resource Identifier. BLAST, Basic Local Alignment Search Tool. TOPBLAST, Top annotated BLAST match. Perl, Practical Extraction and Report Language. GO, Gene Ontology. TABS, Transitive Annotation Based Score. NCBI, National Centre for Biological Information. s.d., Standard deviation.The GOtcha method was devised and implemented by DMAM who also prepared the manuscript. MB performed the manual assessment of false positives and provided feedback on the presentation of results. GJB provided essential guidance for the performance assessment and revision of the manuscript.The supplementary data contains representative examples from the manual assessment of false positives. It is portrayed in tabular format and indicates the benchmark annotation, the highest scoring predicted incorrect annotation by GOtcha and the lowest scoring predicted annotation by GOtcha.Click here for file


The current progress in sequencing projects calls for rapid, reliable and accurate function assignments of gene products. A variety of methods has been designed to annotate sequences on a large scale. However, these methods can either only be applied for specific subsets, or their results are not formalised, or they do not provide precise confidence estimates for their predictions.Xenopus laevis sequences, yielding functional annotation for more than half of the known expressed genome. Compared to the currently available annotation, we provided more than twice the number of contigs with good quality annotation, and additionally we assigned a confidence value to each predicted GO term.We have developed a large-scale annotation system that tackles all of these shortcomings. In our approach, annotation was provided through Gene Ontology terms by applying multiple Support Vector Machines (SVM) for the classification of correct and false predictions. The general performance of the system was benchmarked with a large dataset. An organism-wise cross-validation was performed to define confidence estimates, resulting in an average precision of 80% for 74% of all test sequences. The validation results show that the prediction performance was organism-independent and could reproduce the annotation of other automated systems as well as high-quality manual annotations. We applied our trained classification system to Xenopus laevis contig sequences was predicted and the results are publicly available at .We present a complete automated annotation system that overcomes many of the usual problems by applying a controlled vocabulary of Gene Ontology and an established classification method on large and well-described sequence data sets. In a case study, the function for  Accurate annotation has traditionally been maintained manually with the experience of individual experts and the experimental characterisation of sequences. However, the increasing gap between the amount of sequence data available and the time needed for their experimental characterisation demands computational function prediction in complementing manual curation [and establish a method to provide a confidence value for each annotation.Ongoing genome sequencing and recent developments in cDNA sequencing projects have led to an exponential rise in the amount of sequence information. This has increased the need for acquiring knowledge from sequences as to their biological function. Annotating a single sequence is the gateway to interpreting its biological relevance. However, the usefulness of these annotations is highly correlated with their quality. curation -4. Commocuration . Such ancuration . Howevercuration ,7,8. A ncuration , GAIA [1curation , Genotatcuration , Magpie curation , GeneQuicuration , GeneAtlcuration  and PEDAcuration . Howeveret al predicted GO terms by intersecting domain profiles [The current annotation, written in a rich, non-formalised language also complicates this automated process. We addressed this problem by applying a controlled vocabulary from Gene Ontology (GO) -17. GO pprofiles . The Swiprofiles . Text miprofiles . HoweverXenopus laevis, a widely studied model organism in developmental biology. Because many researchers are now focussing on the functional genomics of this organism, a demand exists for a quality annotation [Xenopus laevis contig sequences (from TIGR Gene Indices) yielding annotation with good confidence values for more than half of these sequences.We have developed an automated system for large-scale cDNA function assignment, designed and optimised to achieve a high-level of prediction accuracy without any manual refinement. Our system assigns molecular function GO terms to uncharacterised cDNA sequences and defines a confidence value for each prediction. The cDNA sequences were searched against GO-mapped protein databases and the GO terms were extracted from the homologues. In the training phase, these GO terms were compared to the GO annotation of the query sequences and labelled correspondingly. We applied Support Vector Machines (SVMs) as the machine learning method to classify whether the extracted GO terms were appropriate to the cDNA sequence or not. In order to classify the GO terms we used a broad variety of elaborated features (attributes) including sequence similarity measures, GO term frequency, GO term relationships between homologues, annotation quality of the homologues, and the level of annotation within the GO hierarchy. To enhance the reliability of the prediction, we used multiple SVMs for classification and applied a committee approach to combine the results with a voting scheme . The connotation . TherefoThe classifier (SVM) needs to specify attribute values (features) for a broad list of samples and a class label for each of these samples. Through the training samples it learns the feature patterns and tries to group them according to their class labels. After training, the algorithm assigns class labels to new samples according to the class that they best match.We selected GO-annotated cDNA sequences for training the SVM classifier. The nucleotide sequences were searched against GO-mapped protein databases and GO-annotations were extracted from the significant hits. Then, each GO term obtained was utilized as a sample for the feature table. The sample GO terms were then labelled as either correct "+1") or false ("-1") by comparing them to the original annotation. Note that we applied the relationships of the GO terms based on their graph structure: "Correct" was assigned not only if they were exact matches but also if the GO terms were related as either "parent" or "child" , Drosophila melanogaster (fly), Mus musculus (mouse), Arabidopsis thaliana (Arabidopsis), Caenorhabditis elegans (worm), Rattus norvegicus (rat), Danio rerio (fish), Leishmania major (Leishmania), Bacillus anthracis Ame (Bacillus), Coxiella burnetii RSA 493 (Coxiella), Shewanella oneidensis MR-1 (Shewanella), Vibrio cholerae (Vibrio) and Plasmodium falciparum (Plasmodium) . Prokaryotic bacteria  contributed 20.6% and the remaining 24.1% of the sequences came from rat, fish, worm, Plasmodium, Leishmania and yeast. Yeast and fly are purely manually annotated datasets. Where as Bacillus, Coxiella, Vibrio, Shewanella, Leishmania and Plasmodium are mostly manually, and the rest mostly automatically annotated datasets. Manual annotation tends to be conservative and sparse, since the GO terms are assigned only if the annotator is highly confident. Therefore, a GO term may be missed due to a poor definition of a false negative. To reduce this critical problems, yeast and fly annotations are accompanied by an "unknown molecular function" term for sequences with questionable further functions. To reduce false negatives, we discarded all sequences with these tags for training and testing .For training and testing the SVM, we selected 39,740 GO-annotated cDNA sequences from the following organisms: Shewanella, Coxiella and Vibrio sequences had the lowest number of GO terms per sequence .The cDNA sequences were searched across the protein databases covering a wide range of organisms from prokaryotes to eukaryotes and SWISSPROT. For 36,771 sequences we got hits with GO terms, contributing to 856,632 sample GO terms and yielding an average of 23.29 GO terms per query sequence  into 99 equal subsets. Note that, amongst these 99 subsets, 96 contained data from a single organism and the remaining 3 from two organisms each. Subsequently, we built 99 classifiers with these subsets. Since the training sets were created organism-wise, the classifiers were trained from different ranges of data, based on purely manual annotation , mostly automated annotation or a mixture of both. For training each of these classifiers, we performed a model selection , which yielded varying accuracy values ranging from 78.81% to 96.03%, with an average accuracy of 85.11%.Plasmodium and Leishmania, 98 classifiers; minimum: Arabidopsis, 74 classifiers). The quality of the predictions was estimated by comparing the predicted terms with the original annotation and the results were expressed in terms of precision and accuracy values (see Methods). The average-accuracy refers to the average of the accuracy values attained by all classifiers used for the prediction. The maximum average-accuracy was achieved for fly (81.51%), followed by yeast (80.50%), and the minimum for mouse (76.0%).To test the classifiers performance, we prepared 13 test sets (each set corresponding to a single organism) using the same 856,632 sample GO terms. The prediction quality of all 99 classifiers were assessed by an organism-wise cross-validation approach, i.e. for each organism (test set), we used all the classifiers for prediction except those that corresponded to the same organism. With this approach, we were able to simulate the annotation of a new organism. The number of classifiers used for predictions varied highly across organisms (maximum: Arabidopsis) with the manually annotated test sequences (yeast and fly). The prediction of the yeast and fly sequences with the 20 classifiers from the mouse sequences produced an average-accuracy of 79% and 80% respectively. Similar results were acquired with the 25 classifiers from Arabidopsis (79% and 80%). Likewise, the worm classifiers (11 classifiers) yielded the average-accuracy of 82% for yeast and 83% for fly. These values were comparable with the average-accuracy of 81% achieved by both, using yeast as test sequences against fly classifiers (16 classifier) and vice-versa (fly test sequences against yeast classifiers). Likewise, we classified the mouse test sequences against yeast classifiers (5 classifier) and fly classifiers yielding 69% and 71% average-accuracy respectively.Additionally, we compared the classification efficiency of the classifier derived from automatic annotation . The classification performances revealed by the ROC plots were comparable between IEA and non-IEA annotated test organisms  and non-IEA . All sequences from Xenopus laevis contig sequences from the TIGR Xenopus laevis Gene Index (XGI) [We extracted all ex (XGI)  and got Xenopus, fly, yeast and mouse  to the high-level, i.e. more generalised or high-level terms of the molecular function ontology ("GO slim" for molecular function) . These me Figure . Note thXenopus contigs (TIGR Xenopus laevis gene indices). We compared our annotation with the TIGR GO annotation for molecular function. From 35,251 contig sequences, TIGR annotated 5,444 contigs with a total of 16,432 molecular function GO terms. In contrast, our approach was able to predict function terms for 17,804 contigs, i.e. more than three times that of TIGR sequences. Our procedure did not annotate 295 contigs from the TIGR annotated contigs. For the remaining 5,149 contigs, 85% of all TIGR terms were found to be exact with those using our method; 3.2% of the TIGR terms were at a higher-level of the GO tree than our annotation, so in this case we provided annotation at a deeper level; in 0.9% of the cases our annotation was at a higher-level; 8.3% of the cases were completely different; and 0.6% of the TIGR terms were obsolete. We compared the quality of TIGR and that of our annotations by a raising stringency and found that when we applied a confidence threshold of 80% for our annotation, we lost 46.6% of the sequences. This included 1,492 sequences holding equivalent TIGR annotation or 27.4% of the total TIGR annotation. With this stringency, our system annotated 9,510 contig sequences, i.e. twice the TIGR annotation at this quality.TIGR provides a GO mapping for endopeptidase activity and more specifically serine-type peptidase activity (98% and 97% confidence respectively). 2) TC209487 and TC190605 are predicted to be aminopeptidases, however for the latter the more specific prediction of prolyl aminopeptidase activity is assigned with 86% confidence. 3) TC199713 is predicted as glutathione peroxidase at 100% confidence and TC194305 is annotated as protein kinase with the same confidence. 4) Both TC187949 and TC210151 are transmembrane receptors but the latter one is classified as frizzled receptor with 82% confidence. In most of these examples the functional assignment and associated confidence were recorded in multiple levels of granularity.We were interested in novel annotated sequences with the highest confidence values and found we could predict GO terms for 557 contigs with a confidence value of 100% . Interestingly, 192 of these lacked any GO annotation by TIGR. Out of these, 184 had got a descriptive TIGR annotation and the rest had not got any. Table In this paper, we presented an automatic annotation system that is able to cope with the expanding amount of biological sequence data. Our approach efficiently combines the ongoing efforts of Gene Ontology and the availability of GO-mapped sequences with a profound machine learning system. The GO-mapped databases provide annotation described in a controlled vocabulary and also a measure of reliability, as these GO entries are labelled with their type of origin. Furthermore, GO terms are structured hierarchically, which allow us a twofold use of the information: i) the level within the tree is taken as a classification criterion to distinguish low from high-level annotations during the learning procedure, and, ii) the hierarchical structure allows us to extend hits by slightly moving up and down within a restricted local area of the tree. This may overcome fluctuations of the annotation levels coming from varying annotation experts.Our annotation system exploits the different combinations of attributes and yields functional transitivity: SVM learning and prediction are organism-independent and comparable to manual annotation, which may be supported by the nature of the attributes we utilise. Subsets and overlaps are counted in a balanced fashion to avoid biases due to the complexity of an organism and a potentially correlated complexity of its sequences. The committee approach allows us to improve the prediction quality as well as to assign confidence values for the new predictions in a straightforward manner. Our classifiers performance is hardly limited by the varying quality of the training data, whether manual or automatic annotated. The prediction results of manually annotated test sets with the classifiers based on automated annotation as well as classifiers based on manual annotation were comparable. Regarding the outcome of the overall classifiers, we achieve consistency with existing annotation from automatic annotations. This is the less complex part of our work and shows a comparable efficiency of our system. Additionally, our system reproduces annotation of purely manually annotated datasets . However, the performance results for these datasets are low in terms of recall, i.e. 47.4% recall with 80% precision compared to 60.6% recall with the same precision of the complete test set. Note that manual annotation tends to be conservative and sparse, yielding stringent true positive definitions, whereas automatically annotated sequences may accumulate information to a greater extent.Xenopus since it is a familiar model organism. However, the sequences were not very well annotated. Our system was applied to annotate the Xenopus contig sequences from TIGR. Through our approach, we annotated 50.5% of all contig sequences available at present, and associated a confidence value for each prediction, yielding roughly three times more sequences as compared to the currently available GO annotation. However, the coverage of annotation to new organism like Xenopus is crucial. We were able to attain predictions for 50.5% of all Xenopus contig sequences (no singletons). This compares to the applied databases that contained 53% satisfactory annotation for their sequences (not regarding sequences with unknown function terms), and better than the organism specific databases (36%). Obviously, improving the quality and quantity of annotation within the available databases goes along with the coverage exploit of machine learning algorithms for new organisms. In future we want to extend our method with the information from other sources such as domain databases and protein family databases.We were interested in annotating Xenopus laevis contig sequences, we obtained a remarkably enhanced annotation coverage compared to the existing annotation.We developed an automated annotation system to assign functional GO terms to an unknown sequence. We used the well-established technique of Support Vector Machines (SVM) for the classification of correct and incorrect GO terms. Our approach benefited from the broad variety of potential attributes used for the functional transitivity and a vast amount of data used for training and validating. The committee scheme exploited in our system provided a means to assign confidence values in a straightforward manner. Our system performance was robust, organism-independent and reproduced the high-quality manual annotation. When applying it to We used the following statistical terms ,31.Accuracy was the rate of correct predictions compared to all predictions,Accuracy: = (TP + TN) / (TP + FP + TN + FN),     (1)Precision was the portion of true positives with respect to all positives,where TP denotes true positives, FP false positives, TN true negatives and FN false negatives. Precision: = TP / (TP + FP).     (2)sensitivity := TP / (TP + FN), specificity := TN / (FP + TN), and false positive rate := 1 - specificity. We defined the term "coverage-of-sequences" as the portion of query sequences for which the classifier delivers a prediction; "Precision-per-sequence" the (average) portion of correct GO terms for a single query sequence, with respect to all GO terms assigned to it. Note that these terms were defined within our model, i.e. a good "accuracy" meant good consistency with respect to our training and test sets.Also used were 1 with respect to GO2 was classified as "parent", "child", "sibling" or "different" Pi denotes the set of nodes from GOi to the rootP2 is a "child" of GO1 if their paths P2 and P1 intersect such thatGO1 ⊃ P2,     (4)P2 is a "sibling" of GO1 if a common parent exists with a distance of one to GO1 and GO2  and is henceforth referred to as "path-pairs". This method could yield a list of several relations. To select the appropriate relation from this list, we considered the parent relationship to be most relevant, followed by the child relationship, and the sibling was considered least relevant. We implemented the following order:We could apply the single path relationship for most of the GO terms (3665 out of 5391). However, for the remaining 1726 terms more than one path to the root were found. For these cases we defined multiple path relationships and each path was considered individually. The single path relationship was applied to each possible pair of these paths (path for GO1. The parent relationship was set if at least one of the path-pairs gave a (single path) parent relationship;2. The child relationship was set if at least one of the path-pairs gave a child relationship. To avoid a bias due to an overwhelming number of path-pairs that did not match, we set a threshold: we considered this relationship only, if the number of path-pairs with no child relationship was equal or less than four times the number of path-pairs with child relationship;3. The sibling relationship was set if at least one of the path-pairs gave a sibling relationship. We again set a threshold: we considered this relationship only, if the number of path-pairs with no sibling relationship was equal or less than twice the number of pairs with sibling relationship;4. If none of these criteria could be applied, the relationship "different" was set.Note that we also implemented the hierarchy of these relations by tuning the stringencies for the fractions of path-pairs that must match .Arabidopsis, worm, rat, fish, Leishmania, Bacillus, Coxiella, Shewanella, Vibrio, Plasmodium, Oryza sativa, Trypanosoma brucei, and Homo sapiens. Apart from this, the SWISS-PROT database was also included [Arabidopsis, worm, rat, fish, Leishmania, Bacillus, Coxiella, Shewanella, Vibrio and Plasmodium  GO level and path: The GO structure was exploited to derive the first two attributes,GO level: the distance of the sample GO term to the root (molecular function node);A.1.GO path: the number of paths from the sample GO term to the root.A.2. B) Alignment quality criteria: These attributes are based on the BLAST alignments. For attributes B.1 - B.4, the best value for the corresponding attribute was taken, if a GO term occurred in more than one hit,Expectation value: the expectation value  from BLASTX;B.1. Bit score: the bit score value provided by BLASTX;B.2. and the hits to offset biases due to different complexities of the query and subject organisms. Attributes B.3, B.4, C.3 and D.3 were obtained from initial trials with a small dataset  and applying parameter optimisation to distinguish the samples.We wanted to award alignment length and quality by combining features. This was done with respect to the length of the query Query coverage score (QCS): Combined measure of alignment size and quality concerning the query sequence,B.3. S := (AL / QL) × (I + S),     (5)QCL denotes the alignment length, QL the length of the query sequence, I the number of identities in the alignment, and S the number of positively contributing residues in the alignment;where ASubject coverage score (SCS): as in B.3, however only with respect to the corresponding subject sequence (database hit),B.4. S := (AL / SL) × (I + S),     (6)SCL denotes the length of the subject sequence;where SAdditionally, we decomposed these attributes into the following further six attributes (B.5 - B.10). For these attributes, we considered the hit with the best coverage score if a GO term occurred in more than one hit .Query percentage (QPC): Percentage of coverage of the alignment region in the query sequence (with respect to QCS), i.e.B.5. C := (AL / QL) × 100;     (7)QPSubject percentage (SPC) Percentage of coverage of the alignment region in the corresponding subject sequence (with respect to SCS), i.e.B.6. C := (AL / SL) × 100;     (8)SPQuery identity (QI): Percentage of identical residues in the BLASTX alignment (with respect to QCS);B.7. Subject identity (SI): Percentage of identical residues in the BLASTX alignment (with respect to SCS);B.8. Query similarity (QS): Percentage of similar or positively contributing residues in the alignment (with respect to QCS);B.9. Subject similarity (SS): Percentage of similar or positively contributing residues in the alignment (with respect to SCS).B.10.C) GO frequency related attributes: We extracted information about the frequency of GO terms in the hits by the following attributes:GO frequency (FG): the number of hits that contained the sample GO term.C.1.Number of hits (TH): the total number of hits for the query.C.2.Frequency score (FS): the number of hits that contained the sample GO term. Unlike C.1, we limited this score to emphasize differences in queries with few hits:C.3. Species frequency: The number of organisms contributing to a sample GO term for a single query sequence;C.4.Total GO (TG): total number of GO terms from all hits.C.5.Unique GO (UG): as C.5, except, that GO terms occurring more than once (in the hits) were counted only once.C.6. D) GO frequency by considering relationships: For these attributes we applied the structure of the Gene Ontology graph. Not only perfectly matching terms were considered, but also their defined parents, children or siblings:Relative frequency for all (RA): the relationships for the sample GO term with all GO terms that occurred in the hits were calculated. The sum of non-"different" relationships i.e. parent, child, or sibling was used for this attribute;D.1.Relative frequency for unique (RU): similar to attribute D.1, with the exception that GO terms occurring more than once were counted only once.D.2.Relative frequency for all (limited) (RAlim): same as attribute D.1, however this score was limited to emphasize differences of queries with few hits:D.3.Relative frequency for unique (limited) (RUlim): same as attribute D.2, however this score was limited to emphasize differences of queries with few hits:D.4.E) Annotation quality related attributes: Quality attributes were selected from the evidence codes provided by the gene association tables of the GO-mapped sequence databases. We selected 9 commonly used evidence codes , resulting in attributes E.1 to E.9. The entries of these attributes for each sample GO term were calculated by summing the occurrences of the corresponding evidence codes of all hits.Before training, normalisation was performed. We normalised the attributes by taking the logarithm (log) and log of log if necessary. We used log values for 16 attributes  and log of log for 8 attributes . Furthermore, we converted the attribute values into mean 0 and standard deviation 1 by applying the Z-transformation. The feature table contained 856,632 samples and 31 attributes. We split the dataset into 99 training subsets. Each subset comprised of approximately 1% of the samples i.e. 8,566 GO terms. This resulted in 96 organism specific subsets and 3 hybrid subsets. We applied the support vector machines in the implementation of LIBSVM , which sXenopus laevis contig sequences is downloadable at . We followed the standard GO annotation style (using Gene ontology guideline). The evidence code is always IEA. The confidence value is included for each GO term.The annotation for The main work was carried out by AV. RK and KG conceived the idea of the study. AV and RK drafted the manuscript. FS developed and JM applied the machine learning strategy. KG implemented the databases in SRS. RE and SS supervised the work. All authors participated in reading, approving and revising the manuscript.