Drosophila melanogaster.The goal of the Sequence Ontology (SO) project is to produce a structured controlled vocabulary with a common set of terms and definitions for parts of a genomic annotation, and to describe the relationships among them. Details of SO construction, design and use, particularly with regard to part-whole relationships are discussed and the practical utility of SO is demonstrated for a set of genome annotations from  The Sequence Ontology (SO) is a structured controlled vocabulary for the parts of a genomic annotation. SO provides a common set of terms and definitions that will facilitate the exchange, analysis and management of genomic data. Because SO treats part-whole relationships rigorously, data described with it can become substrates for automated reasoning, and instances of sequence features described by the SO can be subjected to a group of logical operations termed extensional mereology operators. Genomic annotations are the focal point of sequencing, bioinformatics analysis, and molecular biology. They are the means by which we attach what we know about a genome to its sequence. Unfortunately, biological terminology is notoriously ambiguous; the same word is often used to describe more than one thing and there are many dialects. For example, does a coding sequence (CDS) contain the stop codon or is the stop codon part of the 3'-untranslated region (3' UTR)? There really is no right or wrong answer to such questions, but consistency is crucial when attempting to compare annotations from different sources, or even when comparing annotations performed by the same group over an extended period of time.Arabidopsis Information Resource (TAIR) [Saccharomyces Genome Database (SGD) [At present, GenBank  houses 2e (TAIR)  and the se (SGD) . Each ofgene and intron, and the properties of these features describe an attribute of the feature; for example, a gene may be maternally_imprinted.The Goal of the SO is to provide a standardized set of terms and relationships with which to describe genomic annotations and provide the structure necessary for automated reasoning over their contents, thereby facilitating data exchange and comparative analyses of annotations. SO is a sister project to the Gene Ontology (GO)  and is pLike other ontologies, SO consists of a controlled vocabulary of terms or concepts and a restricted set of relationships between those terms. While the concepts and relationships of the sequence ontology make it possible to describe precisely the features of a genomic annotation, discussions of them can lead to much lexical confusion, as some of the terms used by SO are also common words; thus we begin our description of SO with a discussion of its naming conventions, and adhere to these rules throughout this document.five_prime_UTR, except in cases where the number is part of the accepted name. If the commonly used name begins with a number, such as 28S RNA, the stem is moved to the front - for example, RNA_28S. Symbols are spelled out in full where appropriate, for example, prime, plus, minus; as are Greek letters. Periods, points, slashes, hyphens, and brackets are not allowed. If there is a common abbreviation it is used as the term name, and case is always lower except when the term is an acronym, for example, UTR and CDS. Where there are differences in the accepted spelling between English and US usage, the US form is used.Wherever possible, the terms used by SO to describe the parts of an annotation are those commonly used in the genomics community. In some cases, however, we have altered these terms in order to render them more computer-friendly so that users can create software classes and variables named after them. Thus, term names do not include spaces; instead, underscores are used to separate the words in phrases. Numbers are spelled out in full, for example Synonyms are used to record the variant term names that have the same meaning as the term. They are used to facilitate searching of the ontology. There is no limit to the number of synonyms a term can have, nor do they adhere to SO naming conventions. They are, however, still lowercase except when they are acronyms.Throughout the remainder of this document, the terms from SO are highlighted in italics and the names of relationships between the terms are shown in bold. The terms are always depicted exactly as they appear in the ontology. The names of EM operators are underlined.To facilitate the use of SO for the markup of gene annotation data, a subset of terms from SO consisting of some of those terms that can be located onto sequence has been selected; this condensed version of SO is especially well suited for labeling the outputs of automated or semi-automated sequence annotation pipelines. This subset is known as the Sequence Ontology Feature Annotation, or SOFA.SO, like GO, is an 'open source' ontology. New terms, definitions, and their location within the ontology are proposed, debated, and approved or rejected by an open group of individuals via a mailing list. SO is maintained in OBO format and the current version can be downloaded from the CVS repository of the SO website . For devThe terms describing sequence features in SO and SOFA are richer than those of the Feature Table  of the tSO is not a database schema, nor is it a file format; it is an ontology. As such, SO transcends any particular database schema or file format. This means it can be used equally well as an external data-exchange format or internally as an integral component of a database.The Molecular Biology of the Cell [CDS is defined as a contiguous RNA sequence which begins with, and includes, a start codon and ends with, and includes, a stop codon. According to SO, the sequence of a three_prime_utr does not contain the stop_codon - and files with such sequences are SO-compliant; files of three_prime_utr containing stop_codons are not. This is a trivial example, illustrating one of the simplest use cases, but it does demonstrate the power of SO to put an end to needless negotiations between parties as to the details of a data exchange. This aspect of SO is especially well suited for use with the generic feature format (GFF) [The simplest way to use SO is to label data destined for redistribution with SO terms and to make sure that the data adhere to the SO definition of the data type. Accordingly, SO provides a human-readable definition for each term that concisely states its biological meaning. Usually the definitions are drawn from standard authoritative sources such as the Cell , and eacat (GFF) . Indeed,SO can also be employed in a much more sophisticated manner within a database. CHADO  is a modLike GFF3, Chaos-XML  is a filregion and junction, equivalent to the concepts of interiors and boundaries defined in the field of topological relationships [exon or a transposable_element. A junction is the space between two bases, such as an insertion_site. Building on these basic data types, SOFA can be used to describe a wide range of sequence features. Raw sequence features such as assembly components are captured by terms like contig and read. Analysis features, defined by the results of sequence-analysis programs such as BLAST [nucleotide_match. Gene models can be defined on the sequence using terms like gene, exon and CDS. Variation in sequence is captured by subtypes of the term sequence_variant. These terms have multiple parentages with either region or junction. SOFA (and SO) can also be used to describe many other sequence features, for example, repeat, reagent, remark. Thus, SOFA together with GFF3 or Chaos-XML provide an easy means by which parties can describe, standardize, and document the data they distribute and exchange.The basic types in SOFA, from which other types are defined, are ionships . A regioas BLAST  are captde novo annotation. Several groups including SGD and FlyBase now use either SO or SOFA terms in their annotation efforts. SO is not restricted to new annotations, however, and may be applied to existing annotations. For example, annotations from GenBank may be converted into SO-compliant formats using Bioperl [The SO and SOFA controlled vocabularies can be used for  Bioperl  , because no derives_from relationship unites these two terms in the ontology. This fact illustrates another important aspect of how SO handles relationships: children always inherit from parents but never from siblings. An ncRNA is a kind_of transcript as is an mRNA. Labeling something as a transcript implies that it could possibly produce a polypeptide; labeling that same entity with the more specific term ncRNA rules that possibility out. Thus, a file that contained ncRNAs and their polypeptides would be semantically invalid.SO uses the term part_of relationships pertain to meronomies; that is to say 'part-whole' relationships. An exon, for example, is a part_of a transcript. part_of relationships are not valid in both directions. In other words, while an exon is a part_of a transcript, a transcript is not a part_of an exon. Instead, we say a transcript has_part exon. SO does not explicitly denote whole-part relationships, as every part_of relationship logically implies the inverse has_part relationship between the two terms.part_of relationships are transitive - an exon is a part_of a gene, because an exon is a part_of a transcript, and a transcript is a part_of a gene. Not every chain of part-whole relationships, however, obeys the principle of transitivity. This is because parts can be combined to make wholes according to different organizing principles. Winston et al. [configuration, whether the parts have a structural or functional role with respect to one another or the whole they form; substance, whether the part is made of the same stuff as the whole (homomerous or heteromerous); and invariance, whether the part can be separated from the whole. These six relations and their associated part_of subclasses are detailed in Table Transitivity is a more complicated issue with regards to part-whole relationships than it is for the other relationships in SO. In general, n et al.  have deset al. [part_of relationships only if they all belong to the same subclass. In other words, an exon can only be part_of a gene, if an exon is a component_part_of a transcript, and a transcript is component_part_of a gene. If, however, the two statements contain different types of part_of relationship, then transitivity does not hold.Winston et al.  argue thet al. solve many of the problems associated with reasoning across part_of relationships; thus, we are adopting their approach with SO. The parts contained in the sequence ontology are mostly of the type component_part_of such as exon is a part_of transcript, although there are a few occurrences of member_part_of such as read is a part_of contig.By addressing the vague English term 'part of' in this way, Winston Genomic annotations are substrates for a multitude of software applications. Annotations, for example, are rendered by graphical viewers, or, as another example, their features are searched and queried for purposes of data validation and genomics research. Using an ontology for sequence annotation purposes offers many advantages over the traditional Feature Table approach. Because controlled vocabularies do not specify the relationships that obtain between their terms, using the Feature Table has meant that relationships between features have had to be hard-coded in software applications themselves; consequently, adding a new term to the Feature Table and/or changing the details of the relationships that obtain between its terms has meant revising every software application that made use of the Feature Table. Ontologies mitigate this problem as all of the knowledge about terms and their relationships to one another is contained in the ontology, not the software.tRNA is a kind_of transcript; it need merely know that kind_of relationships are transitive and hierarchical and be capable of internally navigating the network of relationships specified by the ontology  operators to ask questions about gene parts. Although new to genomics, EM operators are well known in the field of ontology, where they provide a basis for asking and answering questions pertaining to how parts are distributed within and among different wholes . Theanalysis  - may alpart_of relationship and then provides a set of operations ; second, if A is a proper part of B then the B is not a part of A; third, if A is a part of B and B is a part of C then A is a part of C.EM is a formal theory of parts: it defines the properties of the ns Table  that canpart_of relationships are transitive. Accordingly, we have restricted our analyses (see Results and discussion) to component parts  contained 13,539 genes, , 18,735 transcripts and 61,853 exons.These data afford many potential analyses, but as our motivation was primarily to demonstrate the practical utility of SO as a tool for data management, rather than comparative genomics overlapping transcripts if at least one of its exons is shared between two of its transcripts, and will have disjoint transcripts if one of its transcripts shares no exons in common with any other transcript of that gene. For the purposes of this analysis, we further classified disjoint transcripts as sequence-disjoint and parts-disjoint. We term two disjoint transcripts sequence-disjoint if none of their exons shares any sequence in common with one another; and parts-disjoint if one or more of their exons overlap on the chromosome but have different exon boundaries. Note that the three operations are pairwise, and thus not mutually exclusive. To see why this is, imagine a gene having three transcripts, A, B, and C. Obviously, transcript A can be disjoint with respect to B, but overlap with respect to C. Thus, we can speak of a gene as having both disjoint and overlapping transcripts.As we had characterized the parts of the annotations using SO, we were able to employ the EM operators over these parts. This proved to be a natural way to explore the relative complexity of alternative splicing, as the alternatively spliced transcripts have different combinations of parts: that is, exons. We grouped alternatively spliced transcripts into two classes. An alternatively spliced gene will contain disjoint and overlapping transcripts in a genome says something about the relative complexity of alternative splicing in that genome. A gene may have any combination of these types of disjoint and overlapping transcripts, so we created a labeling system consisting of the seven possible combinations. We did this by asking three EM-based questions about the relationships between pairs of a gene's transcripts: How many pairs are there of sequence-disjoint transcripts? How many pairs are there of parts-disjoint transcripts? How many pairs are there of overlapping transcripts? Doing so allowed us to place that gene into one of seven classes with regards to the properties of its alternatively spliced transcripts. We also kept track of the number of times each of the three relationships held true for each pair combination. For example, a gene having two transcripts that are parts-disjoint with respect to one another would be labeled 0:1:0. Keeping track of the number of transcript pairs falling into each class provides an easy means to prioritize them for manual review. These results are summarized in Figure The relative numbers of sequence-disjoint transcript, 275 have parts-disjoint transcripts, and 2,664 have overlapping transcripts, and 53 have both parts-disjoint and overlapping transcripts. The percentage of D. melanogaster genes in each category is shown in Table sequence-disjoint transcripts in D. melanogaster, for example, is due to annotation practice; in fact, current FlyBase annotation practices forbid their creation, the reason being that any evidence for such transcripts is evidence for a new gene [Of the alternatively spliced fly genes, none has a new gene . This isnew gene  revealedThe frequencies of genes that fall into each of the seven classes shown in Table binary product of the two transcripts; whereas those exons present in only one of the transcripts constitute their difference . In oth example . WhateveD. melanogaster are highly enriched for 5-prime untranslated exons compared with single-transcript genes. Most of these exons belong to ALWAYS_FOUND; thus, there seems to be a strong tendency in D. melanogaster for alternative transcripts to begin with a unique 5' UTR region. This fact suggests that alternative transcription in the fly may, in many cases, be a consequence of alternative-promoter usage and perhaps tissue-specific transcription start sites. The high percentage of untranslated 5-prime UNIQUE exons in D. melanogaster may also be a consequence of the large numbers of 5' ESTs that have been sequenced in the fly [To investigate these conclusions in more detail, we further examined each exon with respect to its EM-based class and its coding and untranslated portions. These results are shown Figure  the fly .D. melanogaster ALWAYS_FOUND exons are coding. This makes sense, as it seems likely that one reason for an exon's inclusion in every one of a gene's alternative transcripts is that it encodes a portion of the protein essential for its function(s).Figure As with our previous analyses of alternative transcripts, our analyses of alternatively transcribed exons also illustrate the ways in which basic biology and annotation-management issues intersect one another. The fact that most ALWAYS_FOUND exons are entirely coding, for example, may have something important to say about which parts of a protein are essential for its function(s). Whereas the over-abundance of un-translated UNIQUE exons probably has more to say about the resources available to, and the protocols used by, the annotation project than it does about biology. Such considerations make it clear that the evidence used to produce an annotation is an essential part of the annotation. In this regard SO has much to offer, as it provides a rational means by which to manage annotation evidence in the context of gene-parts and the relations between those parts.part_of relationship because SO is largely a meronomy - a particular kind of ontology concerned with the relationships of parts to wholes. Extensional mereology (EM) is an area that is largely new to bioinformatics for which there are several excellent reference works available [We have sought to provide an introduction to the SO and justify why its use to unify genomic annotations is beneficial to the model organism community. We illustrate some of the ways in which SO can be used to analyze and manage annotations. Relationships are an essential component of SO, and understanding their role within the ontology is a basic prerequisite for using SO in an intelligent fashion. Much of this paper revolves around the vailable ,27,33, aUsing all of the relationships in SO allows us to automatically draw logical conclusions about data that has been labelled with SO terms and thereby provide useful insights into the underlying annotations. We have shown how SO, together with the EM-based operations it enables, can be used to standardize, analyze, and manage genome annotations.Given any standardized set of genome annotations described with SO these annotations can then be rigorously characterized. For our pilot analyses, we focused on alternatively transcribed genes and their exons, and explored the potential of EM-operators to classify and characterize them. We believe that the results of these analyses support two principle conclusions. First, EM-based classification schemes are simple to implement, and second, they capture important trends in the data and provide a concise, natural, and meaningful overview of annotations in these genomes.part_of relationships made clear, however, reasoning across diverse types of parts is a complicated process; ad-hoc approaches will not suffice where the data are complex. The more formal approach afforded by SO means that analyses can be easily be extended beyond the domain of transcripts and exons to include many other gene parts and relationships as well - including evidence. It seems clear that over the next few years both the number and complexity of annotations will increase, especially with regard to the diversity of their parts. Drawing valid conclusions from comparisons of these annotations will prove challenging. That SO has much to offer such analyses is indisputable.One criticism that might be justifiably leveled against the SO- and EM-based analyses presented here is that they are too formal, and that simpler approaches could have accomplished the same ends. As our discussion of SO and SOFA provide the model organism community with a means to unify the semantics of sequence annotation. This facilitates communication within a group and between different model organism groups. Adopting SO terminology to type the features and properties of sequence will provide both the group and the community the advantages of a common vocabulary, to use for sharing and querying data and for automated reasoning over large amounts of sequence data.SO and SOFA have been built and are maintained using the ontology-editing tool OBO-Edit. The ontologies are available at .D. melanogaster [mRNA, or tRNA. It was therefore possible to restrict the analysis to given types of transcript. CGL tools were used to validate each of the annotations, iterate through the genes and query the features. EM-operators were applied to the part features of genes.The FlyBase nogaster  data wasnogaster  relationgenomes section of GenBank [genomes section of GenBank. The same Perl class was used to type the feature_relationships according to SO relationship types. The EM analysis was performed over the Chaos-XML annotations using the CGL suite of modules to iterate over the parts of each gene.Other organism data was derived from the  GenBank . GenBank GenBank ) and Bio GenBank . The Bio


Comparative sequence analysis is considered as the first step towards annotating new proteins in genome annotation. However, sequence comparison may lead to creation and propagation of function assignment errors. Thus, it is important to perform a thorough analysis for the quality of sequence-based function assignment using large-scale data in a systematic way.Arabidopsis thaliana, Saccharomyces cerevisiae, Caenorrhabditis elegans, and Drosophila melanogaster. Using a measure of functional similarity based on the three categories of Gene Ontology (GO) classifications , we quantified the correlation between functional similarity and sequence similarity measured by sequence identity or statistical significance of the alignment and compared such a correlation against randomly chosen protein pairs.We present an analysis of the relationship between sequence similarity and function similarity for the proteins in four model organisms, i.e., Various sequence-function relationships were identified from BLAST versus PSI-BLAST, sequence identity versus Expectation Value, GO indices versus semantic similarity approaches, and within genome versus between genome comparisons, for the three GO categories. Our study provides a benchmark to estimate the confidence in assignment of functions purely based on sequence similarity. Large-scale genome sequencing projects have discovered many new proteins. Of all the proteins whose sequences are known, functions have been experimentally determined for only a small percentage . AnnotatSaccharomyces cerevisiae and Arabidopsis thaliana acquired from the Website of Clusters of Orthologous Groups of proteins (COGs) [Despite the central role that sequence comparison programs play in functional annotation, a thorough analysis of the quality of methods based on a large-scale dataset has not been performed. Improvements in the sensitivity of sequence comparison algorithms have reached the point that proteins with previously undetectable sequence relationship, for instance with 10–15% identical residues, may be classified as similar . An estis (COGs) . The COGE. coli genome. However, this study is limited only to within genome comparisons and lacks any analysis based on inter-genome comparisons. Devos et al. [Arabidopsis thaliana, Saccharomyces cerevisiae, Caenorrhabditis elegans, and Drosophila melanogaster) and controlled vocabularies of function annotation terms in the Gene Ontology [A number of studies in sequence-function relationship have been carried out. Shah et al.  showed ts et al.  have stus et al. . HoweverOntology  from thrThe sequence comparisons within and across the four genomes provide a global view on the relationship between sequence similarity and function similarity. Figure E. coli genome [Figure i genome  and by Wi genome  who use Functional conservation measures from GO annotations based on computational techniques such as electronic annotation based on sequence similarity has a behavioral pattern completely different from Figures versus the sequence similarity in terms of E-value and percentage of sequence identity for intra-genome comparisons within four genomes. In this case the localization is measured by five types as described in Section 4.4, instead of the GO Cellular Component Annotation, a detailed level that no existing software can predict reliably. Subcellular localization conservation shows similar results when compared in terms of E-value or sequence identity. Inter-genome comparisons based on the predicted subcellular localizations also behave in a manner similar to the intra-genome comparisons (data not shown). It is interesting to note that the behavior of the curves of the four genomes is similar in respect of E-value  using BLAST. Figure We have also computed results as described above for any random pairs with known function annotation. Then, we calculated a normalized ratio of function similarity in terms of sequence identity by comparing the results in Figures Figure It has been long recognized that genome annotations using computational methods produce many false function assignments. Many of these methods have been applied to function prediction. They often provide valuable hypotheses, but none are perfect. As a result, it is known that many databases contain incorrect function assignments, and these erroneous assignments propagate from one database to another. Nevertheless, up until now there has been no systematic study for this critical issue. The question whether two proteins are functionally similar is very complex to answer. Function is a very complex notion involving many different aspects including chemical, biochemical, cellular, organism mediated, and developmental processes. Qualitatively it is expected that with higher sequence similarity, the two proteins are more likely to have related functions. However, quantitatively the relationship between function similarity at the different categories and sequence similarity has not been studied deeply. Such a quantitative study is fundamentally important, as it can provide assessment of gene function prediction quality and insights into the underlying mechanisms of new evolving functions through changes in sequence ,26.Our study confirms that sequence comparison often provides good suggestions for gene functions or related functions. These suggestions serve as useful hypotheses for further experimental work to confirm, refine or refute the predictions. Such a process can substantially increase the speed of biological knowledge discovery. On the other hand, when assigning function based purely on similarity to proteins of known function (as annotated in databases), it is important to be aware of incomplete or wrong annotations. Given the value of computational function annotation, our study also shows that a significant portion of gene annotations of biological process, molecular function, and cellular component based solely on sequence similarity, in particular, when the sequence similarity is low, are unreliable. Our study also provides a numerical benchmark for the extent to which one can trust computational annotation. It is possible that a confidence score can be derived from our study for any annotation based on sequence similarity. With this score in the annotation file, the user can have a better insight about the quality of the annotations. Furthermore, our analyses highlights the different sequence-function relationships identified from BLAST versus PSI-BLAST, sequence identity versus Expectation value, GO indices versus semantic similarity approaches and within genome versus between genome comparisons, for the three GO classification types.There are some limitations in our current study. Our study can only reflect certain aspect of protein function. Protein function variations may result from factors other than sequence, such as alternative splicing and post-translational modification, and our method does not address these factors. Another limitation is that when we assess gene function prediction, we only consider one hit at a time in a database. In many cases, sequence comparison yields multiple hits for one query protein and these hits may have different functions. In our future study, we will develop a new method to assess the function prediction for a query protein by combining the functions of multiple hits while considering the dependence among these functions and the E-values of the hits.Arabidopsis thaliana, Saccharomyces cerevisiae, Caenorrhabditis elegans and Drosophila melanogaster for the study. All four genomes are well-studied model organisms in eukaryotes. The complete set of Arabidopsis thaliana protein sequences for 27,288 ORFs was acquired from The Arabidopsis Information Resource (TAIR) [Caenorrhabditis elegans ORFs, 6350 Saccharomyces cerevisiae ORFs and 13,665 Drosophila melanogaster ORFs from NCBI [We selected the genomes of e (TAIR) . We alsorom NCBI . Table 1The Gene Ontology (GO) functional classification  has threWe assume that the functional relationship between two proteins is reflected by the number of index levels that they share. We have demonstrated the usefulness of such an assumption in our early studies for gene function prediction ,29. We aGene Ontology annotation is based on various evidences to annotate functional categories. Towards quality control, all the plots  and level 2 (1-1-3) and will have functional similarity equal to 2. The functional similarity defined this way can assume values from 1 to 12.We also calculate functional similarity in terms of semantic similarity between the GO functional annotation terms ,31. An epms is the probability of the minimum subsumer for terms t1 and t2. The minimum subsumer for terms t1 and t2 is defined as the common parent of the deepest GO Index level shared by t1 and t2.where, Saccharomyces cerevisiae, 27,288 in Arabidopsis thaliana, 21,588 in Caenorrhabditis elegans, and 18,498 in Drosophila melanogaster. It is worth mentioning that the subcellular localization predictions were not based on sequence similarity.The subcellular distribution of proteins within a proteome is useful and important to a global understanding of the molecular mechanisms of a cell. Protein localization can be seen as an indicator of its function. Localization data can be used as a means of evaluating protein information inferred from other resources. Furthermore, the subcellular localization of a protein often reveals its activity mechanism. The subcellular localization information was predicted using SubLoc ,33,41. TThe sequence similarity search was done using tools such as BLAST , FASTA ,35 and PWe compared the sequences for within as well as between genome sequence similarities. Each protein sequence was compared against the complete set of proteins for the same genome for within genome comparisons. For between genome comparisons, a pair of similar protein pair was identified using the reciprocal search method , i.e., tTo assess the significance of a sequence comparison, an expectation value or E-value can be calculated. This value represents the number of different alignments with the observed alignment score or better that are expected to occur in the database search simply by chance. The E-value is a widely accepted measure for assessing potential biological relationship, as it is an indicator of the probability for finding the match by chance. Smaller E-values represent more likelihood of having an underlying biological relationship. In this study, we will use both E-value and sequence identity as parameters to quantify sequence similarity. On the other hand, E-values depend on a number of computational factors, such as the length of the query protein and the size of search database. The issues prevent the E-value from being a reliable indicator for homology, as addressed in Fig. The data and results are publicly available at our website .TJ contributed in the data collection, sequence alignments and generation and analysis of the results. Both TJ and DX contributed in the formulation, design and writing of the study. Both authors read and approved the final manuscript.


The international FANTOM consortium aims to produce a comprehensive picture of the mammalian transcriptome, based upon an extensive cDNA collection and functional annotation of full-length enriched cDNAs. The previous dataset, FANTOM2, comprised 60,770 full-length enriched cDNAs. Functional annotation revealed that this cDNA dataset contained only about half of the estimated number of mouse protein-coding genes, indicating that a number of cDNAs still remained to be collected and identified. To pursue the complete gene catalog that covers all predicted mouse genes, cloning and sequencing of full-length enriched cDNAs has been continued since FANTOM2. In FANTOM3, 42,031 newly isolated cDNAs were subjected to functional annotation, and the annotation of 4,347 FANTOM2 cDNAs was updated. To accomplish accurate functional annotation, we improved our automated annotation pipeline by introducing new coding sequence prediction programs and developed a Web-based annotation interface for simplifying the annotation procedures to reduce manual annotation errors. Automated coding sequence and function prediction was followed with manual curation and review by expert curators. A total of 102,801 full-length enriched mouse cDNAs were annotated. Out of 102,801 transcripts, 56,722 were functionally annotated as protein coding , providing to our knowledge the greatest current coverage of the mouse proteome by full-length cDNAs. The total number of distinct non-protein-coding transcripts increased to 34,030. The FANTOM3 annotation system, consisting of automated computational prediction, manual curation, and final expert curation, facilitated the comprehensive characterization of the mouse transcriptome, and could be applied to the transcriptomes of other species. The RIKEN Mouse Gene Encyclopedia project was launched with the aim of cloning and sequencing full-length mouse cDNAs. An international annotation consortium (FANTOM) was organized to annotate the collected mouse cDNAs. In FANTOM1, the consortium annotated 21,076 cDNAs with the development of a Web-based annotation interface [FANTOM1 and FANTOM2 considerably extended our knowledge of the mouse transcriptome, but compared with the number of predicted protein-coding genes from mouse genome sequencing, the cDNA resource covered only half of all predicted genes. Therefore, cDNA collection from a number of novel cellular and tissue sources was continued. In this process, many novel cDNAs derived from distinct genomic loci were fully sequenced. In FANTOM3, these newly sequenced cDNAs were mapped to the mouse genome and subjected to functional annotation. Given the substantial increase in cDNA sequence information in mouse and other mammalian species since FANTOM2, the new annotation process provided the opportunity to update and improve the previous functional annotation of RIKEN cDNAs from FANTOM1 and FANTOM2.Here we report the development of the new annotation interface and decision pipeline, and the modification of our annotation strategy to accelerate manual annotation. And we also provide functional annotation of 102,801 mouse full-length enriched cDNAs, to our knowledge the largest such dataset.The result of this functional annotation was shared among FANTOM3 consortium members for further analyses such as protein coding analysis and noncoding RNA (ncRNA) analysis [The Web-based online annotation system from FANTOM2 was likewise implemented for FANTOM3. This system allowed all curators to annotate transcripts from remote sites around the world through the Internet and resulted in significant acceleration of the manual annotation process. Nevertheless, time remained an issue. Even 10 min spent on manual annotation of each transcript would mean that the total task would consume 15,000 h, and our aim was to complete the task within a matter of weeks. In FANTOM2, curators could enter comments when they encountered problematic cDNAs or ones that were difficult to annotate. However, it was a heavy burden for expert curators, who reviewed and corrected annotations, to read all written comments and correct annotations one by one. For these and other reasons, we introduced a precomputational pipeline in which the annotator could accept the automated decision by ticking a series of boxes. Only where there was some ambiguity, or a better alternative name, was the annotator required to assess additional data and enter alternative decisions. In general, this process reduced the annotation time for unequivocal cases down to 10–20 s.We updated the original annotation rules that were determined during the FANTOM2 meeting ,5 in ordFirstly, coding sequence (CDS) annotation items were expanded in FANTOM3. In FANTOM2, curators annotated the following four items: CDS status , completeness of 5′ and 3′ ends of CDS, maturity of transcripts, and presence of in-frame insertion/deletion errors and stop codons. In FANTOM3, three additional items were introduced: exact positions of in-frame insertion/deletion errors, and flags for selenoproteins and mitochondrial transcripts with unique codon usage. This information was used for computational translation to make a complete dataset of protein-coding transcripts, and it allowed us to avoid unwanted frameshifts and stop codons in the middle of a CDS region.Secondly, the set of CDS prediction algorithms, the outcomes of which are displayed at the top level to the annotator, was changed based upon our previous experience. ProCrest (unpublished) and NCBI CDS Predictor (unpublished), which were used in FANTOM2, were phased out because they cannot identify exact positions of in-frame insertion/deletion errors, although they are able to predict whether these errors exist or not within CDS regions . InsteadThirdly, we improved our annotation pipelines for assigning transcript descriptions (renamed from “gene names” in FANTOM2), symbols (renamed from “gene symbols”), and synonyms to transcripts A and forEscherichia coli, it is curated as “chimeric clone.” If a cDNA has evidence that implies cloning in the reverse direction, for example, having CT-AC splicing patterns rather than GT-AG ones, it is curated as “reverse clone.” These problematic entries are then automatically excluded from further curation and analyses.Fourthly, new annotation items to identify problematic clones were added. In the FANTOM3 annotation system, two buttons for these problematic clones, chimeric clones and reverse clones, were introduced to simplify the annotation process. If a cDNA is deemed to be derived from two or more mRNAs or to be a contaminant from To help curators annotate accurately, the curation interface was improved from that of FANTOM2. Information such as MGI assignment, cDNA status prediction, sequence quality, expressed sequence tag mapping, genome mapping, splicing information, predicted transmembrane regions, and protein motifs was provided on the curation screen. Some information was provided in a simple graphical display to expedite rapid decisions. Moreover, additional information such as raw alignments and hyperlinks to public databases could be accessed by clicking corresponding bars in the cDNA summary image section.In the new FANTOM3 interface, annotators were provided with an initial computational annotation that the curators were then required to accept or reject by clicking buttons. To simplify the annotation process when the computational annotation was rejected, several major reject reasons and alternative CDS predictions, CDS statuses, transcript descriptions, and GO terms were provided as a list with checkboxes, and the curators were prompted to select an appropriate one. Curators were also encouraged to add notes on each transcript, based upon their background knowledge.The computational annotation in FANTOM3 was carried out prior to manual annotation, as in FANTOM2. The FANTOM3 annotation pipelines for assigning transcript descriptions and for GO assignments are summarized in After manual curation on potential protein-coding transcripts, we next considered annotating potential non-protein-coding transcripts. To reduce human annotation errors, potential non-protein-coding transcripts were classified into several subcategories and were released stepwise depending on their coding potential. The transcripts that completely or partially matched known genes at the DNA level were open to curators first, followed by the transcripts that showed similarity to known genes at the amino acid level. Finally, the transcripts that were just covered with expressed sequence tags were subjected to manual curation. Out of 11,555 potential non-protein-coding transcripts, 7,343 (63.5%) transcripts were annotated as non-protein-coding. And 1,893 (16.4%) and 386 (3.3%) transcripts were annotated as immature and truncated forms, respectively.To improve the quality of the functional annotation dataset, a review process was carried out following the manual curation. Expert curators were selected from all registered curators based on their performance, and they reviewed the rejected entries. In FANTOM3, computational filtration was intensively performed to lighten the burden for expert curators. Several criteria are discussed below.In eukaryotes, nonsense-mediated mRNA decay is known as a mRNA surveillance mechanism . It Flanking adenine-rich sequence at the 3′ end of a transcript suggests the possibility that the cDNA could be produced by internal priming of oligo-dT primer. Therefore, we extracted the transcripts that had more than ten adenosines in the 20 flanking nucleotides by using mouse genome sequence, and these transcripts were manually reviewed by expert curators. If transcripts seemed to be produced by internal priming of coding transcripts, they were curated as coding/immature.In FANTOM3, we also developed a genomic element browser by customizing the generic genome browser ,13 to reIn FANTOM3, 42,031 transcripts were newly annotated and the functional annotation of 4,347 FANTOM2 transcripts was updated with the improved annotation system. Combining the results of FANTOM2 and FANTOM3, 102,801 cDNAs were functionally annotated by the international effort. Out of these, 47,761 and 8,961 transcripts were annotated as complete coding and truncated coding, respectively, and 34,030 transcripts were annotated as non-protein-coding .Our FANTOM3 annotation system largely contributed to the prompt and precise annotation that was accomplished, and this system could be a model for other mammalian transcriptome projects.http://fantom3.gsc.riken.jp/db and ftp://fantom3.gsc.riken.jp/fantomdb/3.0.The curated annotation data are available at We annotated 102,801 sequences derived from RIKEN mouse full-length enriched cDNA libraries . The sethttp://www.soe.ucsc.edu/~kent/exe/linux/blatSuite.zip) with options –minAli = 0.96 –nearTop = 0.005. Next, the alignments were post-processed by an algorithm designed to extend transcript-to-genome alignments by using information about exon positions from neighboring alignments . Subsequently, the highest-scoring alignment or alignments, according to the following formula, were retained for each transcript: round , where identity = number of matches/(number of matches + number of mismatches + number of non-intron gaps), coverage = number of matches/transcript sequence size, and introns are gaps of at least 20 bp in the transcript sequence only. Ties were broken in favor of assembled chromosomes over unassembled genomic sequence. If there were still two highest-scoring alignments for a transcript, both were displayed in the annotation interface. Finally, adjacent alignment blocks were connected if they appeared to belong to the same exon. The criteria for deciding that blocks belonged to the same exon were adopted from the Sim4 program [Transcript sequences were mapped to the mouse genome (assembly mm5) in several stages. In the first stage, the sequences were aligned to the genome using BLAT version 30  with opt program : (1) gap program .http://repeatmasker.org) to exclude regions containing known repetitive sequences. FANTOM3 query sequences were searched against mouse non-expressed-sequence-tag mRNA sequences in the MGI database [http://www.informatics.jax.org), against the mouse sequences in dbEST [http://www.ncbi.nih.gov/dbEST), and against known ncRNA sequences in RNAdb (http://research.imb.uq.edu.au/rnadb) [http://www.ebi.ac.uk/interpro).Assembled full-length cDNA sequences were first masked using RepeatMasker (database  (http://in dbEST  (http://u/rnadb) . DNA seau/rnadb)  with theu/rnadb)  in the Fu/rnadb) . Open reu/rnadb)   relational database management system. Other data such as similarity search alignments and clone sequences were stored in indexed flat files. The cDNA annotation (curation) interface was implemented as a Web-based application using mod_perl and the gd graphics library on a Linux system running an Apache 2.0 server. All curated annotations and annotation histories were stored in a custom database implemented in a Sybase (


In the last decade, sequencing projects have led to the development of a number of annotation systems dedicated to the structural and functional annotation of protein-coding genes. These annotation systems manage the annotation of the non-protein coding genes (ncRNAs) in a very crude way, allowing neither the edition of the secondary structures nor the clustering of ncRNA genes into families which are crucial for appropriate annotation of these molecules.LeARN is a flexible software package which handles the complete process of ncRNA annotation by integrating the layers of automatic detection and human curation.This software provides the infrastructure to deal properly with ncRNAs in the framework of any annotation project. It fills the gap between existing prediction software, that detect independent ncRNA occurrences, and public ncRNA repositories, that do not offer the flexibility and interactivity required for annotation projects. The software is freely available from the download section of the website  Science magazine selected the discovery of small RNA with a regulatory function as a scientific breakthrough of the year [C. elegans [Our knowledge of small non-protein-coding RNAs (ncRNAs) has considerably evolved during the last decade. In 2002, the year . Since,  elegans -5. BecauDetecting novel ncRNAs by experimental RNomics is not an easy task . This haab-initio methods rely on an existing bias in base composition between ncRNA and the rest of the genome in order to provide a segmentation of the genomic sequence into ncRNA regions and others [Computational methods for ncRNA prediction can be classified into four approaches; bias composition analysis, minimization of free energy, searching for homologous RNA in the context of conserved family-specific characteristics, and sequence-based homology searches. The first approach analyzes the intrinsic features of the genomic sequence in order to detect ncRNA candidates. These d others ,14. Theyd others . Candidad others . A thirdd others , or covad others . More fld others ; Milpat d others ). These d others , RNAz [2d others , MSARI [d others , ddbRNA d others  have higComplementary to the development of detection strategies, many user interfaces have been developed to modify and annotate RNA sequences and structures .The aim of LeARN, the annotation platform presented here, is to manage the complete process of annotation of ncRNA genes. It can incrementally integrate the results of arbitrary detection programs and provides life scientists with user friendly interfaces allowing both structural and functional annotations. In order to facilitate later exploitation of annotations, LeARN relies on existing standards such as the Rfam database and the RNAML data exchange format, thus providing full interoperability with existing databases and software.The general architecture of LeARN is shared by most annotation platforms Figure . The firThe package . LeARN handles this redundancy by prioritizing the detection software in the main configuration file. A second type of redundancy is caused by overlaps between sequences, and this is critical in the context of ongoing BAC-to-BAC sequencing projects. In order to manage this source of redundancy, LeARN can rely on an additional file describing pairwise overlaps to avoid artificial over-prediction of redundant ncRNA genes.n-1 before starting the greedy analysis of the release n.The software allows for incremental updates of gene and family annotations which is an essential feature for ongoing sequencing projects. Incremental updating is made possible by using the RNAML files of the release In order to analyze one or several complete genomes, it is often useful to run detection computations in parallel. To accommodate the parallelization and the greedy algorithm previously described, the program offers the possibility to execute all detection programs beforehand. In this case, a command line is generated for each execution of a detection program. Each command line stores the result of its execution in a cache directory using an unambiguous filename based on the program name and version and the MD5 checksum of the analyzed sequence and can be directly executed in parallel. When the pipeline later requires the execution of a prediction program, it may directly use the cache result instead of running the prediction itself.The pipeline can easily be customized to integrate arbitrary detection programs providing results in GFF2 format. In addition to the definition of site specific pipelines, this opens the possibility of using LeARN as a light-weighted visualisation interface for researchers willing to develop new detection software.LeARN relies on a repository of RNAML documents to store the annotations of molecules and ncRNA families. This technical choice is compatible with the limited amount of data generated to annotate ncRNA (less than 10 Mb of RNAML to describe the annotation of 130 Mb of legume genome sequences). Relying on RNAML documents (i) provides a native interoperability with the visualisation software which use this standard format; (ii) takes advantage of XSLT processors which allow both document transformations and efficient (via XPATH queries) searches in XML repositories, and (iii) provides users with a light-weight package that does not require any RDBMS skill.The Web interface of LeARN is structured by the different functionalities it offers: scanning the current database and annotating ncRNA molecules or families.The "browse" tab allows lists of ncRNAs and ncRNA families to be displayed Figure . By clicIn order to correct the unavoidable discrepancies and errors generated by an automatic process, the LeARN package provides the expert user with an annotation interface for editing both structural and functional annotations as well as merging and splitting families. This requires user authentication. After the first login, the user must create his/her own workspace, initially defined by a copy of the public database. At any time, the 'Status' page allows the user to select the database he/she wants to use: it can be either the browsable public database or its own editable private one. Editing rights are made visible by a change in the background colour. The system allows for the parallel annotation by several experts, but prevents the concurrent annotation of the same family by different experts.In LeARN, the annotation of a ncRNA molecule is a three step process Figure . The firThe annotation process for families is similar to the previous one Figure . The firPyrococcus abyssi (Pa), Pyrococcus furiosus (Pf), Pyrococcus horikoshii (Ph) and Thermococcus kodakarensis (Tk). For these genomes, the RFAM genome browser gives 11 sRNA families in Pa, 12 sRNA families in Pf, 10 sRNA in families in Ph and 10 sRNA families in Tk for a total of 13 different families. With LeARN, iteratively built with RFAM covariance models, a classification in 16 families was obtained. From the original RFAM families, only the snoR9 family disappeared in this classification.We illustrate LeARN usage with two case studies on thermoccocales genomes We chose the C/D box sRNA family as the first case study. This family is mainly characterized by the presence of four motifs: C (RUGAUGA), D' (CUGA), C' (UGAUGA) and D boxes (CUGA). The region of 9 nucleotides downstream of D and/or D' boxes generally interacts with the target of the C/D sRNA mostly forming Watson-crick interactions. For a more accurate annotation, it is usual to further classify these ncRNA genes into subfamilies with a common target. In LeARN, four families were built automatically from the RFAM covariance model associated to the C/D box sRNA family (ID: snoPyro_CD). The largest snoPyro_CD family contained 74 candidates while the other ones contained only 1 candidate. From the Family browser, it was possible to select and merge all the four families  into only one (RFL0002) making it possible to analyze all candidates together and to classify them according to the conserved boxes and the sequence similarity at the target interaction site. This was done on the merged family using the Edit family function. Using the incrementally updated alignments provided by LeARN, the clustering of sequences into subfamilies based on their targets was straightforward. At the end of the process, 23 novel families were proposed. We renamed them according to the archaea sRNA database . Three cThe second case study concerns H/ACA sRNA. In RFAM, HgcE, HgcF and HgcG have unknown function but were found to be H/ACA sRNA genes named respectively Pf3, Pf6 and Pf7 . Thanks To summarize, LeARN provides a way to complete the gap between existing databases and gene prediction tools. It offers the user a working environment for editing sequence and structure of individual sRNA, as well as RNA families, by using sRNA dedicated functionalities for automatic and manual annotation operations.The first case study showed that one of the advantages of LeARN is to manage redundancy of candidates. The case of sRNA sR9 is a good example. One of the drawbacks of the automatic process of sRNA identification was the assignment of the sR9 C/D box sRNA to the general snoRNA family instead of the more accurate snoR9 family provided by the covariance model of RFAM. This is certainly a result of the proposed iterative approach which relies in part on the energy heuristic clustering. Despite this wrong assignment and the incomplete sequence for two members of this family, it was possible to group all the sR9 candidates and to extend those for which the H/ACA region was lacking by using the various functionalities of LeARN. In both case studies, it was particularly useful to be able to edit a sequence (extension of some regions) with regard to the literature and available knowledge of known H/ACA sRNA, candidates of the family, and availability of genomic sequences. The availability of an automatic or manual alignment of sRNA of the sequence and the structure, and the graphical representation of the consensus secondary structure considerably facilitated the improvement of the structural annotation.All the annotations were done and saved in the private environment of the annotator which offers a very nice and useful working space for personal annotation. After the annotation, it appears now essential to submit these structural annotations to the administrator to replace less accurate ones. One can also imagine that these annotations could be submitted, in the context of a collaborative annotation process, to any RNA database administrator in order to share the annotations with the scientific community. For example, novel sequence and structural annotations of C/D box and H/ACA archaea sRNA could allow new, more accurate RFAM covariance models to be generated for future archaea genome annotations.The demo section of the LeARN home page provides access both to raw results generated by the automatic process applied on the four thermoccocale genomes ("demo server"), and to the database after the edition of C/D box and H/ACA sRNA families ("annotation of C/D box and H/ACA families").The LeARN package can be used for ncRNA annotation projects for any set of sequences including complete genomes. It integrates tools and web interfaces covering the three layers of the ncRNA annotation process: a flexible detection and clustering pipeline, a RNAML database and a web interface to manage the expert annotation. The software has been designed to manage the complete ncRNA annotation process in the frame of whole genome annotation projects and to fill the gap between existing detection software and public ncRNA repositories. Moreover, LeARN is also an extendible and light-weight package, which can be used as an annotation interface either by life scientists wanting to annotate a single ncRNA family or by bioinformaticists who need a simple interface to visually evaluate their results. It can also be used to build training datasets or for any other activity involving ncRNA annotation.Project name: LeARNProject home page: Operating system: UNIXProgramming language: Perl-OO, XSLTOther requirements: Infernal [Infernal , Vienna Infernal , Mfold [Infernal , Rfam [8Infernal , ClustalInfernal , ncbi-blInfernal . The AdmLicense: Free Software license CECILL2 Any restrictions to use by non-academics: NoneXML: eXtensible Markup LanguageXSLT: eXtensible Stylesheet Language TransformationsCN designed, implemented and tested the software; JG led the design. JG, CG and TS tested the programs and contributed to the preparation of the manuscript. All authors have read and approved the final manuscript.LeARN 1.0.1 tarball. Tarball with LeARN source code. For installation instructions, see .Click here for file


The Gene Ontology (GO) is a collaborative effort that provides structured vocabularies for annotating the molecular function, biological role, and cellular location of gene products in a highly systematic way and in a species-neutral manner with the aim of unifying the representation of gene function across different organisms. Each contributing member of the GO Consortium independently associates GO terms to gene products from the organism(s) they are annotating. Here we introduce the Reference Genome project, which brings together those independent efforts into a unified framework based on the evolutionary relationships between genes in these different organisms. The Reference Genome project has two primary goals: to increase the depth and breadth of annotations for genes in each of the organisms in the project, and to create data sets and tools that enable other genome annotation efforts to infer GO annotations for homologous genes in their organisms. In addition, the project has several important incidental benefits, such as increasing annotation consistency across genome databases, and providing important improvements to the GO's logical structure and biological content. Biological research is increasingly dependent on the availability of well-structured representations of biological data with detailed, accurate descriptions provided by the curators of the data repositories. The Reference Genome project's goal is to provide comprehensive functional annotation for the genomes of human as well as eleven organisms that are important models in biomedical research. To achieve this, we have developed an approach that superposes experimentally-based annotations onto the leaves of phylogenetic trees and then we manually annotate the function of the common ancestors, predicated on the assumption that the ancestors possessed the experimentally determined functions that are held in common at these leaves, and that these functions are likely to be conserved in all other descendents of each family. A curator in this context is a Ph.D. trained professional life scientist whose task is to meaningfully integrate published, and in some cases unpublished, biological data into a database The functional annotation of gene products, both proteins and RNAs, is a major endeavor that requires a judicious mix of manual analysis and computational tools. The manual aspect of this annotation task is carried out by curators, from the Latin The GO was developed within the community of the Model Organism Databases (MODs), whose goal is to annotate the genomes of organisms having important impact on biomedical research are available.The annotations based on experimental data provide a solid, dependable substrate for downstream analyses to infer the functions of related gene products. High-quality manual annotation by experts is an absolute prerequisite for seeding this system and, other than the major MOD projects and large sequence databse projects (such as UniProt and Reactome), very few research communities have the resources or trained GO curators to perform this labor-intensive task. Therefore, the functional annotation of non-manually curated genomes typically relies on automated methods that provide the core information for the transfer of annotations from related genes for which experimentally supported annotations Arabidopsis thaliana, Caenorhabditis elegans, Danio rerio, Dictyostelium discoideum, Drosophila melanogaster, Escherichia coli, Gallus gallus, Mus musculus, Rattus norvegicus, Saccharomyces cerevisiae, and Schizosaccharomyces pombe. Collectively those twelve species are referred to as the “GO Reference Genomes”. Each model organism has its own advantages for studying different aspects of gene function, ranging from basic metabolic reactions to cellular processes, development, physiology, behavior, and disease. The organisms selected to provide this gold-standard reference set have the following characteristics: they represent a wide range of the phylogenetic spectrum; they are the basis of a significant body of scientific literature; a reasonably sized community of researchers study the organism; and the organism is an important experimental system for the study of human disease, or for economically important activities such as agriculture. Importantly, all of these organisms are supported by an established database that includes GO curators who have the expertise to annotate gene products in these genomes according to shared, rigorous standards set by the groups participating in the Reference Genome project (see below).The GO Reference Genome project is committed to providing comprehensive GO annotations for the human genome, as well as that of eleven important model organisms: Although the development of the GO has been a collaborative effort since its inception, each participating group has previously worked independently in assigning GO annotations. Thus, prior to this project, specific protocols for annotation varied greatly between the different databases. Variation in annotation results from different curator decisions as to which data is appropriate to annotate and which GO terms to employ. We expect these reference annotations to have two important applications. First, they will increase the quality of the annotations provided by the GO Consortium, with a focus on providing precise annotations for each gene and the broadest possible coverage of each genome. Second, the gold-standard annotation set will greatly accelerate the annotation of new genomes where extensive experimental data on gene function or the resources and expertise to perform the annotations are unavailable.Depth refers to the amount of information about each gene that has been captured. For maximal depth, annotations should be as precise as possible; ideally, all experimentally determined information  about the gene products from each of these organisms should be curated to the deepest level in the gene ontology graph. Breadth refers to the coverage of the genome, that is, the percentage of genes annotated. For maximal breadth, the annotations would ideally cover every gene product in a genome. From a production standpoint, these dual aspects imply a dependency, that is, we must carry out curation in two passes: first literature-based annotation of to capture all information based on experiments, followed by the inference of annotations to the homologous gene products that have not yet been experimentally characterized. Finally, it is important to distinguish genes for which the function is actually unknown from genes that simply have not yet been annotated. To this end, reviewed proteins for which there is no experimental data and that do not share significant homology with experimentally characterized proteins are annotated to the root term of each ontology: biological process (GO:0008150), molecular function (GO:0003674), and cellular component (GO:0005575).There are two different aspects of comprehensive annotation: “breadth” and “depth”. This procedure maximizes both depth and breadth of annotation across all of the curated genomes. We refer to the annotations as ‘comprehensive’ rather than ‘complete’ because it is not always feasible to completely annotate every published paper for every gene with our resources. For genes with a large body of literature, the comprehensiveness of annotations is assessed by curators based on a recent review or text-mining applications.One major advantage of annotating several genomes concurrently is the ability to carry out parallel annotations on homologous genes. Annotating several genes in a single step improves annotation efficiency. Moreover, it improves breadth of annotations by allowing easy access to known function of related genes. Finally, concurrent annotation of gene families across different databases promotes annotation consistency.The organisms represented in the GO Reference Genome project span well over 1 billion years of evolutionary divergence. The premise that underpins the comparative genomics approach is that homologous genes descended from a common ancestor often have related functions. This is not, of course, to deny that genes will diverge in function, but it is generally true that at least some aspects of function are conserved . For our purposes, a critical first step is the establishment of a standard approach to determining sets of homologous genes. Ideally, the evolutionary history of each gene in all organisms would be analyzed and stored in a single resource that could be used as the definitive reference for gene family relationships and homologous gene sets. However, generating this resource is a non-trivial problem, both theoretically, as just described, as well as practically. At present no single resource offers a fully satisfactory solution. Different resources exist that provide different results in terms of specificity and coverage and have different strengths and weaknesses Data availability” below). The P-POD One central confounding problem has been the lack of a “gold standard” protein set that would be used by all databases and homology prediction tools. Because the different homology prediction tools do not use the same protein sets as inputs their results cannot be meaningfully compared. Moreover, the protein sets that are being annotated by the GO Consortium members may, and often do, differ from those used by the different homology prediction programs. The GO Consortium is now providing an index of protein sequence accession identifiers for each organism to groups who compute homology sets (see “Tree-based propagation of annotations to homologous genes” below). We are using trees generated by the PANTHER project (http://www.pantherdb.org/) based on our standardized protein-coding gene sets. The trees also include protein sequences from 34 other species to provide a more complete phylogenetic spectrum. The quality of the trees was assessed by comparing the trees to “ortholog clusters” generated by the OrthoMCL algorithm for the same protein sets. The agreement was very good overall: of the 412 OrthoMCL clusters covering the comprehensively annotated Reference Genome genes, 387 (94%) were consistent with the trees. Most of the disagreements involved a relatively distant evolutionary relationship that was difficult to resolve with certainty. Manual analysis of the trees is part of the curation process to ensure that suspicious absence of presence of proteins in the trees is supported by the genome sequence and/or the multiple sequence alignments upon which the trees are determined.Having agreed to use standardized protein sequence datasets as inputs, we next considered the existing algorithmic approaches to the determination of homology that would best meet our objectives. We chose the phylogenetic tree-based approach because it is based on an explicit evolutionary model that can be computationally evaluated. Moreover, the trees are amenable to intuitive graphical output that facilitates the rapid identification of homology sets by curators  to 27,029 in Arabidopsis thalianaWhile at present the total number of gene products in any organism is imprecisely known  there are reasonable estimates available from the MODs for the numbers of genes encoding protein products in each genome, ranging from 4,389 in protein representing every gene in each genome, this strategy still presents a large and formidable target annotation list. Nevertheless, it is clear that coordination of the Reference Genome project demands a coherent prioritization of targets for curation. Accordingly, Reference Genome curators are selecting targets using the following principles:The goal of the Reference Genome project is to provide constantly up-to-date annotations for all gene families; however this work will take time. Even by initially concentrating solely on one canonical Genes whose products are highly conserved during evolution, e.g. the gyrase/topoisomerase II gene family conserved from bacteria to human.Genes known to be implicated in human disease and their orthologs in other taxa, e.g. the MutS homolog gene family, that includes the gene MSH6, a DNA mismatch repair protein involved in a hereditary form of colorectal cancer in humans.Genes whose products are involved in known biochemical and signaling pathways, e.g. the PYGB gene (a phosphorylase) that participates in glycogen degradation.Genes identified from recently published literature as having an important or new scientific impact, e.g. POU5F1 (POU class 5 homeobox 1 gene) that is important for stem cell function.This promotes the comprehensive annotation of genes of high relevance to the current research efforts, as well as the development of the ontology to fully support those annotations.Literature curation is done by the different groups using the same method: curators read the published literature about the gene they are annotating, capturing several key pieces of information: the organism being studied, the gene product to be annotated; the type of experiment performed; the GO term(s) that best describes the gene product function/process/location; and an identifier  as the source for the information (citation). For each gene that is part of a curation target set, curators review existing annotations as well as add new annotations based on more recent information. If there is no literature, then the genes are immediately considered completely annotated with respect to the available experimental data. For genes with little literature, the curator reviews all available papers, but for genes for which hundreds of papers are available this is impractical. In these cases, curators assess the comprehensiveness of curation based upon recent reviews or text-mining applications, and curate key primary publications accordingly. When this is complete, the gene is considered comprehensively annotated based on the information available in the biomedical literature.Genes that are concurrently annotated are periodically selected for annotation consistency checks among the different curation groups. Automated tests include the verification that older annotations lacking traceable evidence are replaced with annotations that adhere to the new standards, and verifying that outlier annotations, that is, those made only in one organism, are valid and not due to annotation errors. The manual review uses a peer review system in which a curator evaluates the experimentally determined annotations provided by other curators for a selected gene family. The curation consistency review process often identifies problems with the interpretation of particular GO terms. To ensure proper use of these terms in the future, they are flagged within the GO with a comment that a curator must take extra care when using these terms. For example, certain concepts, such as “development”, “differentiation” and “morphogenesis” are used with various, overlapping meanings in the literature. In GO they are distinctively defined, and we strive to ascertain that all annotations uniformly use terms as defined by the GO. The consistency review also identifies GO annotations that may be incorrect, or do not have sufficient evidence.Generating sets of homologous genes’ above. The homology inference process has two steps: (1) inferring annotations of an ancestral gene, based on the  experimental annotations of its modern descendants, and (2) propagating those ancestral annotations to other descendants by inheritance. For the Reference Genome project, both of these steps are documented by an evidence trail that allows GO users to evaluate the inferences that were made. In the first step, a curator annotates an ancestral node in the phylogenetic tree, based on one or more experimentally annotated extant sequences. To document this step, a tree node (with a stable identifier) is associated with both a GO term identifier, and evidence for the association . In the second step, this annotation is propagated to all its descendants (by assuming inheritance as the norm), unless the curator explicitly annotates a descendant as having lost the annotation and provides a citation for this statement. To document this step, a modern-day sequence is associated with both a GO term identifier, and evidence for the association . The two documented steps allow each homology annotation to be traced through to its ancestral node (exactly what inference was made), and then to the modern-day sequences that provide experimental evidence for the annotation. This is not an automatic process, rather a curator reviews each inferred annotation with care since the function of a gene can diverge during evolution, particularly after gene duplication events that may free one of the duplicated copies from selection constraints and allow the evolution of new functionality.The GO Reference Genome project infers functions by homology using a tree-based process that has been previously described An illustration of this process is shown in Gene products selected for concurrent annotation in the course of the Reference Genome project have improved the breadth and depth of annotation coverage. As of November 2008, we have annotated approximately 4,000 gene products. These genes have a higher percentage of annotations derived from published experimental research. Moreover, the annotation of these genes is significantly more detailed relative to when we started this project. Initially, 34% of the 4,000 genes had annotations supported by experimental data. Now, there are 71%, a 2-fold increase; while a randomly selected sample with the same number of genes, has only 52%, a 1.5-fold increase.We might expect the Reference Genome project to yield annotations to more specific terms. Given some specificity metric for a term, we can calculate the average specificity of terms used in annotations for Reference Genome genes and compare these against the average specificity of annotations as a whole, and observe whether there has been an overall increase in specificity. Unfortunately, there is no single perfect measure of specificity. The depth of a term in the graph structure is often a poor proxy, as this is open to ontology structure bias. In this paper we use the Shannon Information Content (IC) as a proxy for specificity of a term. The IC of a term reflects the frequency of annotations to that term (or to descendants of that term), with frequently used terms yielding a lower score than infrequently used terms. The IC is calculated as follows:maximum IC within each branch. We then calculate the average of this maximum IC for all genes in a set to get a measure of the annotation specificity for that set. We compared this number for two sets of genes: the group of all annotated genes for all 12 gene reference genome species , and the subset of this set corresponding to those genes that have been selected for thorough annotation. We then averaged the maximum IC values for both sets of genes before being selected for annotation by the Reference Genome project (July 2006) and again with the most recent set of annotations (December 2008). The results, shown in We can measure the increase in IC on a gene set over time by measuring the average IC of the terms used to annotate the genes in that set before and after reference genome curation. Genes can have multiple annotations in each of the three branches of the GO; here we take the Another measure of the depth and breadth of GO annotations is what range of the ontology graph they cover. The graph coverage of a gene is the size of the set of terms used to annotate a gene, plus all ancestors of that term. In July 2006, the average graph coverage per reference genome gene in a reference species was 34.7, versus an average of 22.9 over all genes in all 12 species. In December 2008 this increased to 64.0 versus 27.0. This shows that the coverage of genes selected for the reference set is proportionally higher, 1.84 versus 1.18.The collaborative annotation of a group of similar gene products has also proven to be useful for the development of GO itself. For example, as a direct consequence of the Reference Genome project, 223 ontology changes or term modifications were made . Examples of requested new terms include “regulation of NAD(P)H oxidase activity”, “DNA 5′-adenosine monophosphate hydrolase activity”, “neurofilament bundle assembly”, and “quinolinate metabolic process”. We have also enhanced the ontology by adding synonyms , improving definitions, and correcting inconsistencies. Examples of terms where definitions and inconsistencies have been corrected include “electron transport” (replaced by two terms: “electron transport chain” and “oxidation reduction”), and “secretory pathway” (replaced by two terms: “exocytosis” and “vesicle-mediated transport”).http://amigo.geneontology.org/) GO annotations may be viewed using AmiGO, the GOC browser (http://geneontology.org/GO.annotation.conventions.shtml). The genes that have been targeted by the Reference Genome project have significantly improved annotation specificity as compared to their previous annotations, and the number of genes annotated by inference through homology has also increased. This increased breadth and depth of genome coverage in the annotations is one of the major goals of the project. An additional benefit has been the improvements to the GO itself, and this will consequently improve the accuracy of inferences based on these annotations. Genomes that are fully and reliably functionally annotated empower scientific research, as they are essential for use in the analysis of many high-throughput methodologies and for the automated inferential annotation of other genomes, a major motivation of the Reference Genome project's work. We encourage users to communicate with the GO Consortium (send e-mail to gohelp@geneontology.org) with questions or suggestions for improvements to better achieve this aim.The aim of the Reference Genome project is to provide a source of comprehensive and reliable GO annotations for twelve key genomes based upon rigorous standards. This endeavor faces many difficult challenges, such as: the determination and provision of reference protein sets for each genome; the establishment of gene families for curation; the application of consistent best practices for annotation; and the development of methodologies for evaluating progress towards our goal. Although this is a laborious effort, steady progress is being made in developing this resource for the research community. This initiative has propelled the GOC into the provision of standardized protein sets for these genomes that we expect to be of broad utility beyond the Reference Genome project. By engaging curators from across the MODs in joint discussions we are observing improvements in curation consistency and refinement of the GOC best practices guidelines (see http://geneontology.org/GO.refgenome.shtml. Annotations made by the databases participating in the Reference Genome project are available from the GOC website in gene_association file format (http://geneontology.org/GO.current.annotations.shtml). The protein sequence datasets are available for the community as a standardized resource from http://geneontology.org/gp2protein/, and as FASTA sequence files here: ftp://ftp.pantherdb.org/genome/pthr7.0. These sets provide a representative protein sequence for each protein-coding gene in each genome, cross-referenced to UniProt whenever possible, but augmented with RefSeq and Ensembl protein identifiers as well. The exact queries used to gather statistics for the annotation improvement reports can be found at: http://geneontology.org/GO.database.schema-with-views.shtml.Access to all GOC software and data is free and without constraints of any kind. An overview of the project as well as links to all resources described below can be found at


Individual researchers are struggling to keep up with the accelerating emergence of high-throughput biological data, and to extract information that relates to their specific questions. Integration of accumulated evidence should permit researchers to form fewer - and more accurate - hypotheses for further study through experimentation.Saccharomyces cerevisiae :S7) is applied to predict GO terms and phenotypes for 21,603 Mus musculus genes, using a diverse collection of integrated data sources . This combined 'guilt-by-profiling' and 'guilt-by-association' approach optimizes the combination of two inference methodologies. Predictions at all levels of confidence are evaluated by examining genes not used in training, and top predictions are examined manually using available literature and knowledge base resources.Here a method previously used to predict Gene Ontology (GO) terms for M. musculus.We assigned a confidence score to each gene/term combination. The results provided high prediction performance, with nearly every GO term achieving greater than 40% precision at 1% recall. Among the 36 novel predictions for GO terms and 40 for phenotypes that were studied manually, >80% and >40%, respectively, were identified as accurate. We also illustrate that a combination of 'guilt-by-profiling' and 'guilt-by-association' outperforms either approach alone in their application to  With the ever-increasing collection of high-throughput experimental techniques, data acquisition at the genomic scale has never occurred more rapidly. As the raw data continue to amass, each biologist is faced with the difficult challenge of integrating and interpreting the data that are most relevant to each specific research question. Comprehensive annotation systems are thus of paramount importance, as evidenced by the integration of a large number of data types in many model organism databases -4. Such Recognizing this problem, curation systems are becoming increasingly reliant on computational approaches to assist in the annotation process. Sequence similarity (both at the nucleotide and peptide levels) has traditionally been the primary source of automated annotation. Particular motifs found within a sequence can be used to infer a gene product's molecular activity, with increasing work being done to identify the domains that facilitate protein interactions . SimilarMus musculus: the MouseFunc project ' indicates the range from a to b, inclusive of a and b). We wished to evaluate separately the performance for terms of different types and levels of generality. To this end, terms were divided into 12 disjoint sets, each representing both a single branch in the GO  and a range in current annotation count, that is, the number of genes annotated with the term . The number of terms in each set is given in Table The 2,938 GO terms currently (as of November 2006) annotated with a number of genes in the range  were selected for training and prediction  curve (AUC-ROC) and precision at specified levels of recall . We use the same general approach here, but include an additional measure: 'mean average precision' (MAP). An ROC curve indicates the relationship between true positives and false positives as the score threshold for calling a prediction is varied. Also used are points along the precision-recall curve (as the threshold varies), with MAP being the mean precision obtained at all distinct recall levels. In the case of a tie among scores, the average precision over all permutations of the response variable (the GO or phenotype term annotations) is taken as the precision for that level of recall. The MAP has been identified as a good alternative to other precision-recall based statistics. Most notably, it has been deemed more useful than area under the precision-recall curve (AUC-PR) as a measure of comparison between classifiers . We alsoTable For GO term prediction, mean AUC-ROC for the combined classifier exceeds 0.8 for 9 out of 12 categories (with 0.5 AUC-ROC being expected for random predictions). At 1% recall, the precision ranges from 41% to 92% across the 12 categories, with MF and CC terms being easier to predict for than BP terms, and with difficulty in prediction increasing as the existing annotation count for these terms decreases . The performance levels for each category as the threshold varies can be seen in Figures Phenotype annotations prove systematically more difficult to classify than GO term annotations . The AUC-ROC for each pooled phenotype category exceeds 0.7, with precision at 1% recall ranging between 10% and 44% (again with the observation that terms with lower annotation count have lower predictive accuracy). Performance characteristics along the threshold range are depicted in Figures Cross-validation and out-of-bag  performaTo gain intuition from specific examples, we examined some of the most interesting novel predictions within the literature. A gene/term prediction is novel and was considered interesting if it was not currently annotated in the reference database and if there did not exist any current annotation involving the gene and any non-root ancestor of the term. Otherwise, we consider such predictions to be 'refinements' of existing annotations. Within each category, the top interesting gene/term combinations are scanned in decreasing order of confidence  and - to avoid over-weighting particular genes or terms - a further filtering step is employed to limit each gene and term to a single appearance in the list to be followed up in depth. This filter is described in detail in the Materials and methods (see below).Predicted GO annotations were reviewed by biologists within the MGI group who are experienced with literature curation (DPH and JAB). Three predictions were reviewed for each of the 12 GO categories based on novelty with respect to existing annotations. Each of these 36 predictions were placed in one of the following four classes: class (i), available experimental literature supports the predicted annotation (21 predictions); class (ii), prediction likely to be correct but supported only by indirect evidence in the literature (5 predictions); class (iii), veracity of prediction is unclear (4 predictions); and class (iv), the prediction is incorrect or unlikely to be correct based on current knowledge (6 predictions). We determined that 19 predicted annotations were class (i), that is, they would qualify for annotation by current curation standards but have not yet been annotated. A remaining two had been annotated since our training data were collected, and were, therefore, correct by definition. Excluding 4 'unclear' evaluations, this leads to 26/32 ≈ 81% accuracy. Additional data file Adra2, predicted for annotation with 'blood pressure regulation' (GO:0008217), could be annotated to 'baroreceptor feedback regulation of blood pressure' (GO:0001978), which is a child of 'blood pressure regulation' ), 11 were considered 'unclear' (class [iii]), and 16 were found 'unlikely' or 'very unlikely' (class [iv]). Excluding 'unclear' evaluations, the success rate was 13/29 ≈ 45%. Additional data file Nfatc2, the absence of an embryonic heart phenotype had been reported .M. musculus data manipulation given below.Training data for the machine learning algorithms were organized generally as described in , with deE0.5, E0.6, E0.7, E0.8, and E0.9; each group defined as:The Pearson product-moment correlation coefficient of expression data was computed for each gene-pair (for each of the three data sets). These continuous coefficients were then binned into five groupings, a and b are genes, and ρ1 is the correlation coefficient function. This resulted in 15 binary attributes describing each gene-pair. The protein-protein interaction data were already provided in gene-pair format, with each positive edge resulting in a positive gene-pair for a single binary matrix column representation.where A and B represent sets of annotated domains belonging to (respectively) genes a and b. The Jaccard similarity coefficient is then defined as:Protein domain pattern data were converted to gene-pair format using the Jaccard similarity coefficient. Let a, b) having ρ2 ≥ 0.9 were assigned membership in a single set. Each domain dataset (Pfam and Interpro) was processed using this method, resulting in two binary variables.Those gene-pairs , resulting in 6 additional binary variables (3 each for Biomart and Inparanoid data).Thus, a total of 26 distinct binary variables describing each gene-pair were used by the functional linkage classifier. For phenotype prediction, eight additional variables were included representing co-annotation of terms within the MF and CC annotation sets , leading to 34 binary variables. We excluded terms within the BP branch to avoid potential circularity because GO BP terms are often tightly related to phenotypes. The data were then organized into a matrix with genes as rows and the 34 predictive variables as columns.For the random forest base classifiers, binary gene-centric data were represented in a binary matrix form . When predicting GO terms, the training matrix was composed of the protein domain data, the phenotype data (top level only), the phylogenetic profile data, and the disease data . For phenotype term prediction, the training matrix consisted of the same data used for GO prediction as above except without the phenotype data .To generate functional linkage graphs, a single probabilistic decision tree  was builribed in .v/20 where v is the number of remaining variables to be sampled at a given node. For the phenotype predictions, 200 trees/forest and the same v/20 were used as parameters. Each term  corresponds to a separate random forest model.Random forests are used as described in , with paα parameter corresponding to each category is shown in Table The two base classifiers for each GO or phenotype term were combined via a logistic model, as described in . The α pt we convert gene i's score si to a new score:The combined scores go through a final calibration so that they more closely approximate posterior probabilities. The calibration method developed for this study was also applied in the companion paper describing the MouseFunc project , where iL ∈  is a free parameter chosen such that the following relation holds:t) is the number of annotations for term t .where count(AUC-PR, area under the precision-recall curve; AUC-ROC, area under the ROC curve; BP, biological process; CC, cellular component; GO, Gene Ontology; MAP, mean average precision; MF, molecular function; MGI, Mouse Genome Informatics; OMIM, Online Mendelian Inheritance in Man; ROC, receiver operator characteristic; SAGE, serial analysis of gene expression.The authors declare that they have no competing interests.FR conceived the study, and MT, WT and FR conceived of the methods. MT and WT performed all code construction and program optimization. FG aided in data pre-processing and clustering and constructed the web interface. DH performed literature evaluations of predictions with guidance from JB. MT and FR drafted the manuscript. All authors read and approved the manuscript.The following additional data are available with the online version of this paper. Additional data file Predictions, ratings, and evidence supporting novel GO term predictions.Click here for filePhenotype term predictions and supporting evidence.Click here for file