Evaluation¶
Wrapper for MKB evaluation module.
Parameters¶
-
entities
-
relations
-
batch_size
-
true_triples – defaults to
[] -
device – defaults to
cuda -
num_workers – defaults to
0
Examples¶
>>> from mkb import datasets
>>> from ckb import evaluation
>>> from ckb import models
>>> from ckb import scoring
>>> import torch
>>> _ = torch.manual_seed(42)
>>> train = [('mkb', 'is_a', 'library'), ('github', 'is_a', 'tool')]
>>> valid = [('ckb', 'is_a', 'library'), ('github', 'is_a', 'tool')]
>>> test = [('mkb', 'is_a', 'tool'), ('ckb', 'is_a', 'tool')]
>>> dataset = datasets.Dataset(
... batch_size = 1,
... train = train,
... valid = valid,
... test = test,
... seed = 42,
... )
>>> dataset
Dataset dataset
Batch size 1
Entities 5
Relations 1
Shuffle True
Train triples 2
Validation triples 2
Test triples 2
>>> model = models.DistillBert(
... entities = dataset.entities,
... relations = dataset.relations,
... gamma = 9,
... scoring = scoring.TransE(),
... device = 'cpu',
... )
>>> model.entities
{0: 'mkb', 1: 'github', 2: 'ckb', 3: 'library', 4: 'tool'}
>>> model
DistillBert model
Entities embeddings dim 768
Relations embeddings dim 768
Gamma 9.0
Number of entities 5
Number of relations 1
>>> validation = evaluation.Evaluation(
... entities = dataset.entities,
... relations = dataset.relations,
... true_triples = dataset.train + dataset.valid + dataset.test,
... batch_size = 1,
... device = 'cpu',
... )
>>> validation.eval(model = model, dataset = dataset.valid)
{'MRR': 0.3958, 'MR': 2.75, 'HITS@1': 0.0, 'HITS@3': 0.75, 'HITS@10': 1.0}
Methods¶
compute_detailled_score
compute_score
detail_eval
Divide input dataset relations into different categories (i.e. ONE-TO-ONE, ONE-TO-MANY, MANY-TO-ONE and MANY-TO-MANY) according to the mapping properties of relationships.
Parameters
- model
- dataset
- threshold – defaults to
1.5
eval
Evaluate selected model with the metrics: MRR, MR, HITS@1, HITS@3, HITS@10
Parameters
- model
- dataset
eval_relations
Evaluate selected model with the metrics: MRR, MR, HITS@1, HITS@3, HITS@10
Parameters
- model
- dataset
get_entity_stream
Get stream dedicated to link prediction.
Parameters
- dataset
get_relation_stream
Get stream dedicated to relation prediction.
Parameters
- dataset
initialize
Initialize model for evaluation
Parameters
- model
types_relations
Divide input dataset relations into different categories (i.e. ONE-TO-ONE, ONE-TO-MANY, MANY-TO-ONE and MANY-TO-MANY) according to the mapping properties of relationships.
Parameters
- model
- dataset
- threshold – defaults to
1.5