Trying to add doc.sents to rel_component example's get_instances function

[yofayed]

Hello,
I am trying to improve the rel_component by using the doc.sents from the sentencizer or parser component. I tried to follow the instruction in the "predictions from preceding components" section in https://spacy.io/usage/training, by modifying my config file to include the parser in annotating_components as noted below, but I keep getting this error when I call doc.sents in the get_instances function:

File "C:\rel-model\scripts\rel_model.py", line 79, in get_instances
for sent in doc.sents:
File "spacy\tokens\doc.pyx", line 890, in sents
ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe('sentencizer'). Alternatively, add the dependency parser or sentence recognizer, or set sentence boundaries by setting doc[i].is_sent_start.

Here are my modifications to the config file
...
pipeline = ["tok2vec", "parser","relation_extractor"]
...
[components.parser]
source = "en_core_web_lg"
...
[training]
...
annotating_components = ["parser"]

What do I have to do to include the sents from the parser or sentencizer in training this rel_component model so that I can access them in the model's get_instances function?

Thank You!

[vinbo8]

Could you paste your full config?

[yofayed]

Below is my config file. Do I have to modify the parser_data.py to include the "sent_starts"? Thanks

[paths]
train = null
dev = null
raw = null
init_tok2vec = null

[system]
seed = 342
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["tok2vec","parser","relation_extractor"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}
batch_size = 1000

[components]

[components.tok2vec]
#factory = "tok2vec"
source = "en_core_web_lg"

[components.tok2vec.model]
@architectures = "spacy.HashEmbedCNN.v1"
pretrained_vectors = "en_core_web_lg"
width = 96
depth = 2
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

[components.parser]
source = "en_core_web_lg"

[components.relation_extractor]
factory = "relation_extractor"
threshold = 0.5

[components.relation_extractor.model]
@architectures = "rel_model.v1"

[components.relation_extractor.model.create_instance_tensor]
@architectures = "rel_instance_tensor.v1"

[components.relation_extractor.model.create_instance_tensor.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = 96

[components.relation_extractor.model.create_instance_tensor.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.get_instances]
@misc = "rel_instance_generator.v1"
max_length = 15

[components.relation_extractor.model.classification_layer]
@architectures = "rel_classification_layer.v1"
nI = null
nO = null

[initialize]
vocab_data = null
vectors = null
init_tok2vec = ${paths.init_tok2vec}
after_init = {"@callbacks":"sentinel_customize_tokenizer"}

[initialize.before_init]
@callbacks = "spacy.copy_from_base_model.v1"
tokenizer = "en_core_web_lg"
vocab = "en_core_web_lg"

[initialize.components]

[corpora]

[corpora.dev]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.dev}

[corpora.train]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.train}

[training]
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600000
max_epochs = 0
max_steps = 10000
eval_frequency = 500
frozen_components = ["tok2vec","parser"]
annotating_components = ["tok2vec","parser"]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
before_to_disk = null
logger = {"@Loggers":"spacy.ConsoleLogger.v1"}

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
rel_micro_p = 0.0
rel_micro_r = 0.0
rel_micro_f = 1.0

[shadeMe]

I'd suggest doing the following:

• Use replace_listeners for the sourced parser so that it gets its own copy of the tok2vec layer.
• Remove tok2vec from the frozen and annotating components so that it's just used for relation_extractor.
• Ensure that tok2vec has include_static_vectors = true so that it's using the vectors that have been included from en_core_web_lg.

[yofayed]

Thanks for your response @shadeMe. I will try these suggestions and let you know how it turns out.
I added the tok2vec to the frozen list as an attempt to fix the following error that seems to only happen if I try to loop through the sents in the get_instences function.

ValueError: operands could not be broadcast together with shapes (1122,36) (150,36)

This happens in the "get_loss" function when calculating the gradient
gradient = scores - truths

[yofayed]

Hello @shadeMe, I modified the config as below according to your suggestions, so the model will tain fine as long as I don't try to loop through the doc.sents in the get_instences function, otherwise I get a "ValueError: operands could not be broadcast together with shapes (200,36) (196,36)"

Config:

[paths]
train = null
dev = null
raw = null
init_tok2vec = null

[system]
seed = 342
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["tok2vec","parser","relation_extractor"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
#tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}
batch_size = 1000

[components]

[components.tok2vec]
#factory = "tok2vec"
source = "en_core_web_lg"
#Ensure that tok2vec has include_static_vectors = true so that it's using the vectors that have been included from
include_static_vectors = true

[components.tok2vec.model]
@architectures = "spacy.HashEmbedCNN.v1"
pretrained_vectors = "en_core_web_lg"
width = 96
depth = 2
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

[components.parser]
source = "en_core_web_lg"
#Use replace_listeners for the sourced parser so that it gets its own copy of the tok2vec layer.
replace_listeners = ["model.tok2vec"]

[components.relation_extractor]
factory = "relation_extractor"
threshold = 0.5

[components.relation_extractor.model]
@architectures = "rel_model.v1"

[components.relation_extractor.model.create_instance_tensor]
@architectures = "rel_instance_tensor.v1"

[components.relation_extractor.model.create_instance_tensor.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = 96

[components.relation_extractor.model.create_instance_tensor.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.get_instances]
@misc = "rel_instance_generator.v1"
max_length = 15

[components.relation_extractor.model.classification_layer]
@architectures = "rel_classification_layer.v1"
nI = null
nO = null

[initialize]
vocab_data = null
vectors = null
init_tok2vec = ${paths.init_tok2vec}
after_init = {"@callbacks":"sentinel_customize_tokenizer"}

[initialize.before_init]
@callbacks = "spacy.copy_from_base_model.v1"
tokenizer = "en_core_web_lg"
vocab = "en_core_web_lg"

[initialize.components]

[corpora]

[corpora.dev]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.dev}

[corpora.train]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.train}

[training]
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600000
max_epochs = 0
max_steps = 10000
eval_frequency = 500
#Remove "tok2vec" from the frozen and annotating components so that it's just used for relation_extractor.
frozen_components = ["parser"]
annotating_components = ["tok2vec","parser"]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
rel_micro_p = 0.0
rel_micro_r = 0.0
rel_micro_f = 1.0

Modified the get_instences function to loop through doc.sents

Modified the get_instences function to loop through doc.sents
@spacy.registry.misc("rel_instance_generator.v1")
def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]:
def get_instances(doc: Doc) -> List[Tuple[Span, Span]]:
instances = []
for sent in doc.sents:
for ent1 in sent.ents:
for ent2 in sent.ents:
if ent1 != ent2:
if max_length and abs(ent2.start - ent1.start) <= max_length:
instances.append((ent1, ent2))
return instances
return get_instances

Training stops after first epoch with this error

E # LOSS TOK2VEC LOSS RELAT... DEP_UAS DEP_LAS SENTS_F REL_MICRO_P REL_MICRO_R REL_MICRO_F SCORE
0 0 0.00 11.29 0.00 0.00 51.13 0.28 34.00 0.56 0.00
�  Aborting and saving the final best model. Encountered exception:
ValueError('operands could not be broadcast together with shapes (200,36)
(196,36) ')
Traceback (most recent call last):
File "C:\software\python3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\software\python3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\software\python3\lib\site-packages\spacy_main.py", line 4, in
setup_cli()
File "C:\software\python3\lib\site-packages\spacy\cli_util.py", line 71, in setup_cli
command(prog_name=COMMAND)
File "C:\software\python3\lib\site-packages\click\core.py", line 1128, in call
return self.main(*args, **kwargs)
File "C:\software\python3\lib\site-packages\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "C:\software\python3\lib\site-packages\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\software\python3\lib\site-packages\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\software\python3\lib\site-packages\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "C:\software\python3\lib\site-packages\typer\main.py", line 532, in wrapper
return callback(**use_params) # type: ignore
File "C:\software\python3\lib\site-packages\spacy\cli\train.py", line 45, in train_cli
train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
File "C:\software\python3\lib\site-packages\spacy\cli\train.py", line 75, in train
train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
File "C:\software\python3\lib\site-packages\spacy\training\loop.py", line 122, in train
raise e
File "C:\software\python3\lib\site-packages\spacy\training\loop.py", line 105, in train
for batch, info, is_best_checkpoint in training_step_iterator:
File "C:\software\python3\lib\site-packages\spacy\training\loop.py", line 203, in train_while_improving
nlp.update(
File "C:\software\python3\lib\site-packages\spacy\language.py", line 1164, in update
proc.update(examples, sgd=None, losses=losses, **component_cfg[name]) # type: ignore
File "C:\workspace\spacy-models\rel-model\scripts\rel_pipe.py", line 133, in update
loss, gradient = self.get_loss(examples, predictions)
File "C:\workspace\spacy-models\rel-model\scripts\rel_pipe.py", line 146, in get_loss
gradient = scores - truths
ValueError: operands could not be broadcast together with shapes (200,36) (196,36)

[shadeMe]

Sorry about the late reply.

I did a quick test to reproduce the above with the example project (https://github.com/explosion/projects/tree/v3/tutorials/rel_component) which I imagine what your script is based on. The primary problem here is that the number of entities that are detected by the get_instances function differs when iterating over the entities per sentence (as opposed to iterating over all the entities in the Doc). The reference (gold-standard) Docs do not contain any sentence boundaries, so it returns a truth array with a different shape than the one returned by the model which is calculated from Docs that have annotated sentence boundaries.

[yofayed]

@shadeMe No worries. I really appreciate you taking the time to debug this error.

I have seen the error above even when using the original get_instences function that is not using sentence boundaries
and that's after I added all the components from the "en_core_web_lg" except for the "tok2vec". The data samples used in training include the "en_core_web_lg" ner results but also include manually added/corrected ner entities via the annotation tool.

I will try to adjust the golden data to ensure it aligns with the entities detected by the get_instences function and will let you know if this works.