Pellet has been moved to http://pellet.owldl.com
The information on this page is kept for historical/archival reasons. These pages are not updated any more. Please update your link.
Overview | Project Page | Download | Support | FAQ | Online demo | Performance | Pellet Widget
Pellet Performance
This page contains information about the performance evaluation of Pellet as of version 1.3 (beta2). All the experiments are done on an IBM ThinkPad T41 computer with Pentium Centrino 1.6GHz processor and 1.5GB of main memory. The Java heap space was set to 256MB (using -Xmx option).
There are four basic tests done to evaluate the performance.:
- Consistency: Consistency checking ensures that an ontology does not contain any contradictory facts. OWL Abstract Syntax & Semantics document provides a formal definition of ontology consistency that Pellet uses. In DL terminology, this is the operation to check the consistency of an ABox with respect to a TBox.
- Classification: Classification computes the subclass relation between each named class to create the class hierarchy. The class hierarchy is used to answer queries such as getting all or only the direct subclasses.
- Realization: Realization finds the most specific classes that an individual belongs to, in other words computes the direct types for each of the individuals. It should be done after classification since direct types are defined with respect to a class hierarchy.
- Query: We use conjunctive queries that query the instance data. More specifically, we run conjunctive ABox queries against the given ontology.
Ontology Tests
The following table summarizes the Pellet performance results for some generic OWL ontologies. All the times are listed in seconds. Times do not include parsing time (creating an RDF model from RDF/XML file) but displays the loading time (loading an RDF model to Pellet). The program that generated these results is included in the Pellet distribution.
| Name | OWL Species | DL Expressivity | Triples | Classes | Properties | Individuals | Loading | Consistency | Classify | Realize | Total |
| Mindswappers | DL | ALCIF(D) | 996 | 48 | 60 | 122 | 0.14 | 0.05 | 0.13 | 0.17 | 0.49 |
| AKT Portal | DL | ALCHOIF(D) | 1771 | 169 | 137 | 75 | 0.11 | 0.01 | 0.67 | 0.06 | 0.85 |
| OWL-S CongoService | DL | SHOIF(D) | 2463 | 121 | 187 | 283 | 0.13 | 0.07 | 0.54 | 0.57 | 1.31 |
| Food | Full | ALCOF | 869 | 64 | 8 | 45 | 0.09 | 0.00 | 0.82 | 0.00 | 0.91 |
| SUMO | Full | ALH(D) | 3688 | 630 | 238 | 435 | 0.12 | 0.01 | 1.31 | 0.59 | 2.03 |
| Tambis | DL | SHIN(D) | 4200 | 392 | 100 | 0 | 0.22 | 0.00 | 1.00 | 0.00 | 1.22 |
| Financial | DL | ALCIF | 65723 | 59 | 16 | 17941 | 2.04 | 2.52 | 0.03 | 2.70 | 7.31 |
| SWEET-JPL | DL | SHOIF(D) | 5894 | 1400 | 137 | 110 | 0.23 | 0.00 | 8.01 | 0.39 | 8.63 |
| Wine | Full | SHOIF(D) | 2708 | 137 | 17 | 206 | 0.13 | 0.78 | 11.18 | 1.04 | 13.13 |
| Galen | DL | SHF | 30854 | 2749 | 413 | 0 | 1.29 | 0.00 | 46.10 | 0.00 | 47.39 |
Comparisons with Other Systems
-
Classification
We have compared the classification times of different DL reasoners for some ontologies included in the standard DL benchmark test suite . The systems we have compared with Pellet are Racer Pro 1.8.2 (we used this version although it is not released yet because it has significant performance improvements compared to last release), the commercialized version of the well-known Racer system, and FaCT++ 0.99.5, a C++ reimplementation of the established FaCT reasoner. The selected ontologies do not contain instances, nominals or qualified number restrictions (Qualified cardinality restrictions are not present in OWL-DL and not supported by Pellet. Instance reasoning is not supported by FaCT++ and nominals are not available neither in FaCT++ nor RacerPro). The following figure shows the classification times for these ontologies. The results are sorted based on the size of the ontologies (number of classes). The scale is logarithmic to improve the presentation.
The results show that Pellet is not as efficient as RacerPro or FaCT++ for classification but still has acceptable performance. The difference in performance seen in these cases might not be significant for applications because the processing times are generally couple of seconds. It is also interesting to note that, even though FaCT++ significantly performs better than Pellet, for the famous Galen medical ontology Pellet is much faster (NOTE: Due to technical difficulties of running FaCT++ under Windows (more specifically the interface that allows loading KRSS files) we had to use cygwin system which has some negative effect on the performance. We will update the results soon for a better comparison).
-
Query Answering
We have evaluated the performance of conjunctive query answering using the data and queries included in the Lehigh University Benchmark (LUBM). Since FaCT++ does not provide reasoning support for instances, we have only compared Pellet with RacerPro. The data generator in the benchmark creates information about universities, departments, professors, students and courses. In order to provide a finer grained comparison, we used three different data sets which contain 1, 3 and 5 universities, respectively. The number of instances in these data sets are 17174, 55664 and 102368. In our experiments, we have evaluated three different features: time it takes the reasoner to check the consistency of the data, amount of total preparation time spent before queries can be answered (excluding parsing and loading time), and finally the time it takes to answer each query. The following figure summarizes the results we obtained on three data sets. The results show that Pellet significantly outperforms RacerPro for consistency checking (also note that RacerPro was unable to perform the consistency test for 15 universities where Pellet spent only 22 seconds). For this reason, we had to disable consistency checking for RacerPro for query answering. Even without consistency checking RacerPro has a much higher initial preparation time compared to Pellet (RacerPro builds index structures in order to perform efficient query answering). Note that, preparation time for Pellet is very close to consistency checking time because there is no other significant operation Pellet performs for initialization. The program that generated these results is included in the Pellet distribution.
