KGen: a knowledge graph generator from biomedical scientific literature – BMC Medical Informatics and Decision Making

An evaluation was conducted to assure the quality of the KGs generated through our solution. To this count, we conducted two experiments. Our objective is to guarantee that the key steps from our method acting produce the most allow outputs. We aim to ensure that press out triples from the sentences are exchangeable to triples that would be manually extracted by knowledge domain specialists, and ultimately, ensure that render KGs make sense to specialists. In addition, we assess to which extent the proposed yoke method acting improves the final examination output of our method acting. The foremost experiment ( cf. experiment I : evaluation of Triples subsection ) involved two physicians who charitable volunteered to assist us in evaluating the quality of the triples extracted from biomedical textbook. To the best of our cognition, there are no gold standards in literature to evaluate triples extracted from amorphous text from the biomedical domain. In this way, we have chosen to invite subjects that are knowledgeable in this world for this experiment. Both physicians have more than 10 years of working feel in their areas, and they attend international events regularly, therefore, are used to read scientific papers from the biomedical domain. As a disclaimer, it is authoritative to mention that, at any time preceding the experiment, neither of the physicians were told about the actual nature of this employment, or what does our method achieves. The second experiment ( cf. experiment two : Ontology links subsection ) involved a comparison of the ontology links obtained from the stream version of our method acting, in comparison with the initial adaptation, as described in the exploit by Rossanez and Dos Reis [ 14 ]. With this experiment, we assess the remainder between using a train model to recognize biomedical named entities and their UMLS CUIs, to ultimately link with a final biomedical ontology, against using NCBO ’ s REST APIs to provide the concluding biomedical ontology links directly .

Experiment I: evaluation of triples

This experiment, as previously stated, involved two physicians, and consisted of two parts. The preamble of the experiment was to have each doctor read three discrete abstracts ( i.e., six distinct abstracts in sum ), extracted from medical papers related to Alzheimer ’ randomness Disease, from the Neurology journal. The abstracts of this daybook follow the Objective, Methods, Results, and Conclusions format. We took the six foremost publications which resulted from a search for the Alzheimer’s Disease terminus, in the journal ’ s search locomotive, at hypertext transfer protocol : // ( As of Dec. 2019 ).

Extraction of triples analysis

A little introduction about RDF was given to the subjects in the beginning separate of the experiment. We presented some examples of RDF triples ( subject, predicate, and object ) extracted from small sentences. We made surely that they completely understood the serve before starting the main procedure of the experiment. We asked them to only take into bill the decision section of the abstracts to manually create their own triples ( graph ) according to their interpretation of the text. such sub-section contains, in general, one or two little sentences ( for example, This study confirms the high prevalence of poststroke cognitive impairment in diverse populations, highlights common hazard factors, in finical, diabetes mellitus, and points to ethnoracial differences that warrant care in the development of prevention strategies. ). No communication was allowed between the subjects or the subjects and the experimentalist, until the process was finished. We ran our creature, which implements our KGen method, to extract triples from the same abstracts ’ conclusions sub-sections handed to the subjects. table 1 summarizes the sum of triples extracted from the 6 abstracts, labeled from A00 to A05, by the subjects, and besides, by our method acting .Table 1 Comparison of extracted triples between experts and KGen results Full size table As shown by table 1, our method has extracted more triples than the subjects. This is achieved due to the amount of main and secondary relations that our method extracts from the sentences. The secondary relations are extracted using the colony parse proficiency, resulting in triples that relate nouns with their compounds and adjectives, a well as other nouns. This may result in a great sum of triples, depending on the sum of these parts of language in the sentence. For exemplify, consider the postdate sentence : Ethnoracial differences warrant attention in the development of prevention strategies. An exemplar of manually extracted triple by one of the experts is the adopt :

  • ( “Ethnoracial differences”, “warrant”, “attention in the development of prevention strategies” )

On the other pass, using our tool, the addiction parsing technique resulted in the follow sic of secondary triples :

  • ( “prevention strategies”, rdfs:subClassOf, “strategies” )
  • ( “ethnoracial differences”, rdfs:subClassOf, “differences” )
  • ( “development of prevention strategies”, local:of_preventionstrategies, “development” )
  • ( “development of prevention strategies”, local:development_of, “prevention strategies” )

As for the independent relations, extracted using the SRL technique, they result in at least two triples for prison term, depending on the issue of verb arguments that are retrieved by the proficiency. Considering the same prison term, our tool outputs the following set of triples from the SRL technique :

  • ( “warrant”, local:AM-LOC, “development of prevention strategies” )
  • ( “warrant”, vn.role:Agent, “ethnoracial differences” )
  • ( ”warrant”, vn.role:Theme, “attention” )

The predicate of the first triple is an URI that indicates localization. The predicates from the second and third gear triples are URIs representing the function that the object assumes in the original prison term. If we consider the SRL proficiency alone, without the proposed hypostatization shape, then we have the trace triples, that are in a format that is similar to the manually extracted by the experts :

  • ( “ethnoracial differences”, “warrant”, “attention” )
  • ( “ethnoracial differences”, “warrant attention in”, “development of prevention strategies” )

For a clean base of comparison, we ran our cock considering different configurations for triples origin : ( 1 ) Extracting entirely the main relations through the SRL proficiency ; ( 2 ) Extracting merely the main relations through the SRL technique, without considering the adopted depersonalization phase ; And ( 3 ) extracting only the secondary relations through the colony parsing proficiency. board 2 summarizes the differences in the results .Table 2 Comparison between KGen’s configurations Full size table As another set about to compare KGen ’ s generated triples with the manually excerpt ones, we used the Jaccard similarity coefficient. This coefficient measures the similarity between finite sets, and it is defined as the size of the intersection divided by the union of the sets. equality 1 shows how the Jaccard coefficient J ( A, B ) is calculated for two sets A and B.

$$\begin{aligned} J(A, B) = \frac{\mid A \cap B \mid }{\mid A \cup B \mid } = \frac{\mid A \cap B \mid }{\mid A \mid + \mid B \mid – \mid A \cap B \mid } \end{aligned}$$

( 1 ) The Jaccard coefficient ranges between 0 and 1. If both sets have the same elements, the rate is 1. If there is no intersection between such sets, the measure is 0. If both sets are empty, the Jaccard coefficient is defined as 1. To obtain the Jaccard coefficient, we considered the sets of manually generated triples and KGen ’ second. From such two sets, we identified triples that were only found in the manual of arms process, triples that were only found by KGen, and finally, triples that are found both manually and by KGen ( i.e., the intersection between both sets ). postpone 3 presents the prevail results .Table 3 Comparison between triple sets Full size table We observe that, according to the Jaccard coefficient, the sets are not very alike, having a identical little intersection. This is primarily due to the differences between the number of elements from the compared sets, i.e., KGen extracts much more triples than the specialists. We already discussed that the manually extract triples are more exchangeable to the triples extracted from the SRL technique without the proposed hypostatization class. In this manner, we foster compared the KGen triples extracted using the SRL without hypostatization configuration. such results are described in table 4 .Table 4 Comparison between triple sets (KGen on SRL without reification) Full size board The analysis concerning the extraction of triples by KGen using the SRL without hypostatization shape, the Jaccard coefficient increases. The intersection comparing the measure of triples that were extracted alone in the manual and entirely in KGen ’ s sets can be explained by some particularities that were found in the manually origin of triples. One of such is the fact that the experts derived relations that are not explicitly in the sentences. For exemplar, the first abstract contains the follow sentence : This study confirms the high prevalence of poststroke cognitive impairment in diverse populations, highlights common risk factors, in particular, diabetes mellitus, and points to ethnoracial differences that warrant attention in the development of prevention strategies. Some of the triples the experts were able to extract are :

  • ( “ diabetes mellitus ”, “ is ”, “ risk gene ” )
  • ( “ poststroke cognitive impairment ”, “ is prevailing ”, “ in divers populations ” )
  • ( “ exploitation of preventions strategies ”, “ are ”, “ needed ” )

  • ( “ preponderance of poststroke cognitive damage ”, “ involves ”, “ ethnoracial differences ” )

Although such triples make perfective sense, KGen is not able to build them given the employed techniques. This is by and large due to their predicates not being explicitly found in the text. It requires some legitimate intend to build them, and in some cases, even a previous knowledge domain cognition ( which is expected from such specialists ). Another case from the pursue sentence is in the third pilfer : Results at 3 years after unilateral transcranial magnetic resonance-guided focused ultrasound thalamotomy for essential tremor, show continued benefit, and no progressive or delayed complications. In this shell, the specialists were able to distinguish essential tremor as the condition, and the ( very large ) treatment type unilateral transcranial magnetic resonance-guided focused ultrasound thalamotomy, resulting in the adopt triple :

  • ( “ unilateral transcranial magnetic resonance-guided focused ultrasound thalamotomy ”, “ is ”, “ an choice to manage essential tremor ” )

In early cases, some triples were derived from complex sentences. This is handled by KGen in the preprocessing step, ending up on avoiding such redundancies. For case, the be prison term from the second abstract : High-convexity tight sulci may confound clinical and biomarker interpretation in Alzheimer’s Disease clinical trials. KGen extracted two triples in this sheath, whereas the adept extracted the follow three triples :

  • ( “ High-convexity compressed sulcus pattern ”, “ may confound ”, “ clinical and biomarker interpretation in Alzheimer ’ s Disease clinical trials ” )
  • ( “ High-convexity mean sulcus design ”, “ may confound ”, “ clinical interpretation in Alzheimer ’ s Disease clinical trials ” )
  • ( “ High-convexity tight sulcus form ”, “ may confound ”, “ biomarker interpretation in Alzheimer ’ s Disease clinical trials ” )

In this sense, we found that such triples represent a secondary cognition that is derived from the chief cognition obtained from the text, which is, in turn, represented by the triples extracted by KGen. Therefore, an authoritative moral learned is that, no matter what proficiency or method acting used to extract triples from textbook in the biomedical knowledge domain, it is important to allow the late addition of manually generated triples from experts to the output. such semi-automatic approach might enrich the cognition representation, both in terms of denotative cognition from the text, but besides, in terms of derived cognition that is preferably implicit in the text. Another possibility is to include a new sub-step, or post-processing sub-step, in which we could mechanically attempt to derive triples from the prevail trio set, exchangeable to the ones manually obtained from the subjects, through inference reason .

Knowledge graph analysis

once the first part of the experiment was finished, we started the formulation for the second separate. Both subjects were explained the concept of Knowledge Graphs. They were instructed about the graphic representation of RDF triples in a KG ( i.e., edges representing the predicate, while subjects and objects are represented either by ellipses in case of URIs, or by rectangles in character of literals ). once again, we made sure that the concepts were wholly understood by them, before moving foster. Each national was then presented a prison term extracted from one of the abstracts ’ conclusions subsection, along with a elementary KG generated for that prison term. The give KG was a simple interpretation of the KG generated by our method acting, as it did not give any ontology connection. This was meant to remove the extra complexity that such figure may present and become very large. figure 17 presents one of such graphs .Fig. 17figure 17 Reduced cognition graph case. Knowledge graph generated for the triples extracted from the come sentence : This study highlights common risk factors, in particular diabetes mellitus. Full size image With those KGs in handwriting, we asked the subjects to analyze them and freely provide any comments they might have. It is important to mention here that neither of the subjects were told that such KGs were generated by a tool which implements our method, to avoid any kind of bias in their judgements. Both subjects found such an concern form to visually describe the sentences from the conclusion subsection of the abstracts. They agreed that the starting point of the graph with the main verb ( californium. Fig. 17 ) is a good begin charge as it represents the principal information from the sentence. This reflects the idea of a main relative, represented by the main triples extracted from the SRL proficiency. This binds the secondary information, represented by the secondary coil triples extracted from the colony parsing technique. This binding is well represented by the local URIs that make the graph amply connected. The subjects agreed that breaking down larger objects and subjects ( for example, common risk factors into risk factors and then, into factors ) into smaller and more cohesive terms makes it easier to find a specific concept in the representation. In practical terms, this could allow a SPARQL question performed in the KG on finding if a specific concept is stage in the graph. such task would be harder to accomplish if this concept should be embedded into a graph node that represents a more specific concept ( for example, common risk factors ), rather than a more generic concept ( for example, risk factors ). still regarding the breakdown of larger nodes into smaller ones, one point of improvement was indicated by the subjects. When, for case, breaking down common risk factors into risk factors, and in turn, into factors, some terms are left aside, such as common, and risk. Representing such terms could enrich the cognition representation, specially because they could besides be considered in a SPARQL question. In practical terms, they could be linked to ontologies, as far discussed in Experiment II ( cf. “ experiment II : Ontology links ” incision ). One possible way to represent such terms, would be to add new triples to the existing adjust, using local URIs that would represent those as separate of the initial specific concept, such as the ones represented in bold below :

  • ( “ coarse risk factors ”, rdfs : subClassOf, “ hazard factors ” )
  • (“common risk factors”, local:hasAdjective, “common”)
  • ( “ risk factors ”, rdfs : subClassOf, “ factors ” )
  • (“risk factors”, local:hasCompound, “risk”)

Another point of improvement suggested by the experts concerns specifically the graph from Fig. 17. In this graph, we observe that the diabetes mellitus concept is dealt as a specific concept of the generic mellitus concept. This is not dependable, as diabetes mellitus represents a concept ( a disease ). It is not supposed to be broken down as the consequence obtained by our technique. This happened because the colony parser from Stanford CoreNLP toolkit considered both diabetes and mellitus as separate nouns. besides, diabetes is a compound linked to mellitus. In practice, this could be mitigated by incorporating a biomedical named entity recognizer ( NER ) to the proficiency. If such NER identifies diabetes mellitus as a whole entity, there would be no crack it down. Another option would be using a colony parser trained in biomedical textbook, that would prevent such an return. Most models used in NLP tools and techniques are trained in a very large, but finite fit of textbook. Due to new findings and probe works, the biomedical world evolves quickly and new entity names are introduced to the vocabulary. For this cause, the train models require seasonably updates to catch up to the state of the art. therefore, most NLP tools and techniques may constantly fail in some view, being it either recognizing name entities, identifying parts of language, or generating parse trees. This enforces the use of a semi-automatic method acting, where such limitations on tools and techniques may be overcame by manual interaction when required. furthermore, there may be minor human errors in the textbook ( for example, incorrectly punctuation, ambiguous sentences, etc. ) that may besides interfere with the output signal of NLP tools and techniques. such erroneous outputs might interfere with the generation of RDF triples, and in consequence, generate erroneous KGs. For this reason, a semi-automatic border on is valuable, as an technical might be able to review the method ’ s overall and mediator outputs, and interfere with the process, so that we may have the most appropriate outputs .

Experiment II: ontology links

In this evaluation, we compared the ontology links obtained from the same unstructured input signal textbook, between KGs generated using the ontology linking method acting from our former work [ 14 ], against the update associate method proposed and implemented in this work. The previous method consists on retrieving the annotations generated from the National Center for Biomedical Ontology ( NCBO ) bioportal, by means of their REST APIs, where we passed in as parameters, the actual text to be annotated, and a target ontology, from which the links should be obtained. The new method acting, on the other bridge player, consists in using a trained model able to recognize biomedical named entities, and a NLP Part of Speech ( PoS ) tagger to recognize verbs. The model is able to provide UMLS CUIs, for the given recognized entities and verbs. then, by using NCBO bioportal ’ sulfur SPARQL end point, we query the match matches for such concept IDs in a prey ontology. The comparison of both methods used the lapp conclusions sub-sections of the abstracts used in Experiment I, labeled from A00 to A05. The target ontology selected is NCIT, the like ontology from the examples in Section. NCIT is the most suitable ontology, as it is suggested by NCBO recommender servicing [ 59 ] for all the abstracts texts. The train model, which provides NER and UMLS CUIs, was obtained from the ScispaCy project [ 57 ]. We first compared the amount of links obtained using both methods, in terms of number of links generated. table 5 summarizes the results. We observe that for all abstracts, the modern method shows better results in obtaining links .Table 5 Comparison between ontology linking methods Full size postpone Considering the deviation between the links obtained in both methods, we have links that were added ( i.e., they did not exist in the previous method acting ), links that were maintained ( i.e., they exist in both the previous and the new method ), and links that were removed ( they exist in the previous method acting, but no longer exist in the newfangled method ). postpone 6 presents such cases .Table 6 Analysis of changes in links provided by the through ScispaCy trained model and UMLS SPARQL queries Full size table The total of links found for the modern method acting in table 5 considers the links obtained in the prey ontology. As already explained, before obtaining the final connection, the method first identifies the terminus ( a named entity or a verb ), then obtains a UMLS CUI, and last, obtains the target ontology link through a SPARQL question. consequently, there may be some cases where : ( 1 ) the term is identified and there is no UMLS match ; ( 2 ) the condition is identified, a UMLS CUI is found, but there is no match in the prey ontology ; and ( 3 ) the most successful lawsuit, where the terminus is identified, a UMLS CUI is found, and a pit in the target ontology is besides found. such cases are presented in postpone 7 .Table 7 Steps in generating links in the new method Full size table

The numbers presented in table 7 bespeak that if we choose a different aim ontology, we may find different results, as the final examination liaison is obtained through an existing map between the target ontology and UMLS. If we placid consider the same target ontology, one possible direction to improve the results would be combining the results for both methods, i.e., adding the target ontology links obtained from the previous method ( NCBO annotator ) that are not found through the UMLS method to the output signal. The results may besides be possibly further enhance if we specifically search for the identify terms, that neither have a correspond UMLS CUI, nor an gloss consequence in the target ontology. We updated our KGen tool to incorporate and evaluate this compound set about to far enhance the results. table 8 shows the update results when performing this overture. We found a positive consequence, as the number of links increased .Table 8 Further comparison between ontology linking methods Full size board

informant :
category : Education

Trả lời

Email của bạn sẽ không được hiển thị công khai.