Publications
Detailed Information
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Elnaggar, Ahmed | - |
dc.contributor.author | Heinzinger, Michael | - |
dc.contributor.author | Dallago, Christian | - |
dc.contributor.author | Rehawi, Ghalia | - |
dc.contributor.author | Yu, Wang | - |
dc.contributor.author | Jones, Llion | - |
dc.contributor.author | Gibbs, Tom | - |
dc.contributor.author | Feher, Tamas | - |
dc.contributor.author | Angerer, Christoph | - |
dc.contributor.author | Steinegger, Martin | - |
dc.contributor.author | Bhowmik, Debsindhu | - |
dc.contributor.author | Rost, Burkhard | - |
dc.date.accessioned | 2024-05-16T01:26:45Z | - |
dc.date.available | 2024-05-16T01:26:45Z | - |
dc.date.created | 2021-08-09 | - |
dc.date.created | 2021-08-09 | - |
dc.date.issued | 2022-10 | - |
dc.identifier.citation | IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.44 No.10, pp.7112-7127 | - |
dc.identifier.issn | 0162-8828 | - |
dc.identifier.uri | https://hdl.handle.net/10371/202521 | - |
dc.description.abstract | Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans. | - |
dc.language | 영어 | - |
dc.publisher | Institute of Electrical and Electronics Engineers | - |
dc.title | ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning | - |
dc.type | Article | - |
dc.identifier.doi | 10.1109/TPAMI.2021.3095381 | - |
dc.citation.journaltitle | IEEE Transactions on Pattern Analysis and Machine Intelligence | - |
dc.identifier.wosid | 000853875300088 | - |
dc.identifier.scopusid | 2-s2.0-85138449494 | - |
dc.citation.endpage | 7127 | - |
dc.citation.number | 10 | - |
dc.citation.startpage | 7112 | - |
dc.citation.volume | 44 | - |
dc.description.isOpenAccess | Y | - |
dc.contributor.affiliatedAuthor | Steinegger, Martin | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.subject.keywordPlus | PROTEIN SECONDARY STRUCTURE | - |
dc.subject.keywordPlus | NEURAL-NETWORKS | - |
dc.subject.keywordPlus | PREDICTION | - |
dc.subject.keywordPlus | LOCALIZATION | - |
dc.subject.keywordAuthor | Proteins | - |
dc.subject.keywordAuthor | Training | - |
dc.subject.keywordAuthor | Amino acids | - |
dc.subject.keywordAuthor | Task analysis | - |
dc.subject.keywordAuthor | Databases | - |
dc.subject.keywordAuthor | Computational modeling | - |
dc.subject.keywordAuthor | Three-dimensional displays | - |
dc.subject.keywordAuthor | Computational biology | - |
dc.subject.keywordAuthor | high performance computing | - |
dc.subject.keywordAuthor | machine learning | - |
dc.subject.keywordAuthor | language modeling | - |
dc.subject.keywordAuthor | deep learning | - |
- Appears in Collections:
- Files in This Item:
- There are no files associated with this item.
Item View & Download Count
Items in S-Space are protected by copyright, with all rights reserved, unless otherwise indicated.