2. Generalising over holophrases

In the next few interactions, the agents are presented with a different question. As before, the only thing the learner can do is create hollistic mappings between the observed question and the reconstructed meaning. However, most of these mappings will fail to generalise across scenes. At some point, the learner will find a form-meaning mapping that has a minimal difference with the previously acquired holophrase construction on both the form side and the meaning side. This is when the pattern finding mechanism that generalises over holophrases will come into play. The agent will capture the similarities in an item-based construction. This construction abstracts over the differences by providing open slots on both the form side and the meaning side. The agent also learns lexical constructions, for filling the item-based slots, and adds links to the categorial network. These links indicate which lexical items are suitable fillers for which item-based slots. Here, the 'substitution' variant of the generalisation learning mechanism is demonstrated, where the new form-meaning pairing differs from the holophrase construction by substituting a minimal difference. There also exists an 'addition' variant, where the new form-meaning pairing extends the holophrase constructions by a minimal difference, and the 'deletion' variant, where the new form-meaning pairing is a reduction of the holophrase constructions by a minimal difference.


Interaction 4

The tutor is the speaker.

The learner is the listener.

Current Scene:

Topic:

material-category

Question: "What material is the big cube?"


Comprehending "What material is the big cube?"


Applying
g
in comprehension


Computing next composer solution

using ontology:


Result

composition process

Composer queue:

Found 1 solutions:


Evaluating irl program

irl program
((bind color-category ?category-368 purple) (bind attribute-category ?attribute-348 material) (query ?answer-224 ?source-object-252 ?attribute-348) (unique ?source-object-252 ?source-set-559) (filter ?source-set-559 ?source-set-560 ?category-368) (get-context ?source-set-560))
gstruct0(bindcolor-category?category-368purple)struct4(filter?source-set-559?source-set-560?category-368)struct0:varCATEGORYdash3681->struct4:varCATEGORYdash3685struct1(bindattribute-category?attribute-348material)struct2(query?answer-224?source-object-252?attribute-348)struct1:varATTRIBUTEdash3482->struct2:varATTRIBUTEdash3483struct3(unique?source-object-252?source-set-559)struct2:varSOURCEdashOBJECTdash2523->struct3:varSOURCEdashOBJECTdash2524struct3:varSOURCEdashSETdash5594->struct4:varSOURCEdashSETdash5595struct5(get-context?source-set-560)struct4:varSOURCEdashSETdash5605->struct5:varSOURCEdashSETdash5606

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-224
material-category
score: 1.000
?category-368
color-category
score: 1.000
?attribute-348
attribute-category
score: 1.000
?source-set-560
clevr-scene
score: 1.000
?source-set-559
clevr-object-set
score: 1.000
?source-object-252
clevr-object
score: 1.000


initial structure
application process
applied constructions
resulting structure

Meaning:

gstruct0(bindcolor-category?category-368purple)struct4(filter?source-set-559?source-set-560?category-368)struct0:varCATEGORYdash3681->struct4:varCATEGORYdash3685struct1(bindattribute-category?attribute-348material)struct2(query?answer-224?source-object-252?attribute-348)struct1:varATTRIBUTEdash3482->struct2:varATTRIBUTEdash3483struct3(unique?source-object-252?source-set-559)struct2:varSOURCEdashOBJECTdash2523->struct3:varSOURCEdashOBJECTdash2524struct3:varSOURCEdashSETdash5594->struct4:varSOURCEdashSETdash5595struct5(get-context?source-set-560)struct4:varSOURCEdashSETdash5605->struct5:varSOURCEdashSETdash5606

The following construction(s) were learned:

Comprehension succeeded

Interpretation finished

Computed answer:

material-category

Interaction failed


Interaction 5

The tutor is the speaker.

The learner is the listener.

Current Scene:

Topic:

material-category

Question: "What material is the big cube?"


Comprehending "What material is the big cube?"


Applying
g
in comprehension


Evaluating irl program

irl program
((bind color-category ?category-369 purple) (bind attribute-category ?attribute-349 material) (query ?answer-225 ?source-object-253 ?attribute-349) (unique ?source-object-253 ?source-set-561) (filter ?source-set-561 ?source-set-562 ?category-369) (get-context ?source-set-562))
gstruct0(bindcolor-category?category-369purple)struct4(filter?source-set-561?source-set-562?category-369)struct0:varCATEGORYdash3691->struct4:varCATEGORYdash3695struct1(bindattribute-category?attribute-349material)struct2(query?answer-225?source-object-253?attribute-349)struct1:varATTRIBUTEdash3492->struct2:varATTRIBUTEdash3493struct3(unique?source-object-253?source-set-561)struct2:varSOURCEdashOBJECTdash2533->struct3:varSOURCEdashOBJECTdash2534struct3:varSOURCEdashSETdash5614->struct4:varSOURCEdashSETdash5615struct5(get-context?source-set-562)struct4:varSOURCEdashSETdash5625->struct5:varSOURCEdashSETdash5626

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-225
material-category
score: 1.000
?category-369
color-category
score: 1.000
?attribute-349
attribute-category
score: 1.000
?source-set-562
clevr-scene
score: 1.000
?source-set-561
clevr-object-set
score: 1.000
?source-object-253
clevr-object
score: 1.000


Computing next composer solution

using ontology:


Result

composition process

Composer queue:

Found 3 solutions:

Checking solution 1 against 1 past scenes

Checking solution 2 against 1 past scenes

Checking solution 3 against 1 past scenes


Computing next composer solution

using ontology:


Result

composition process

Composer queue:

... and 2 more

Found 15 solutions:

... and 12 more

All composer solutions so far

... and 15 more

Checking solution 1 against 1 past scenes

Checking solution 2 against 1 past scenes

Checking solution 3 against 1 past scenes

Checking solution 4 against 1 past scenes

Checking solution 5 against 1 past scenes

Checking solution 6 against 1 past scenes

Checking solution 7 against 1 past scenes

Checking solution 8 against 1 past scenes


Evaluating irl program

irl program
((get-context ?source-set-548-13) (filter ?source-set-563 ?source-set-548-13 ?category-363-13) (filter ?source-set-564 ?source-set-563 ?category-371) (unique ?source-object-254 ?source-set-564) (query ?answer-226 ?source-object-254 ?attribute-350) (bind attribute-category ?attribute-350 material) (bind shape-category ?category-363-13 cube) (bind size-category ?category-371 large))
gstruct0(get-context?source-set-548-13)struct1(filter?source-set-563?source-set-548-13?category-363-13)struct0:varSOURCEdashSETdash548dash131->struct1:varSOURCEdashSETdash548dash132struct2(filter?source-set-564?source-set-563?category-371)struct1:varSOURCEdashSETdash5632->struct2:varSOURCEdashSETdash5633struct6(bindshape-category?category-363-13cube)struct1:varCATEGORYdash363dash132->struct6:varCATEGORYdash363dash137struct3(unique?source-object-254?source-set-564)struct2:varSOURCEdashSETdash5643->struct3:varSOURCEdashSETdash5644struct7(bindsize-category?category-371large)struct2:varCATEGORYdash3713->struct7:varCATEGORYdash3718struct4(query?answer-226?source-object-254?attribute-350)struct3:varSOURCEdashOBJECTdash2544->struct4:varSOURCEdashOBJECTdash2545struct5(bindattribute-category?attribute-350material)struct4:varATTRIBUTEdash3505->struct5:varATTRIBUTEdash3506

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-226
material-category
score: 1.000
?attribute-350
attribute-category
score: 1.000
?category-363-13
shape-category
score: 1.000
?category-371
size-category
score: 1.000
?source-set-548-13
clevr-scene
score: 1.000
?source-set-563
clevr-object-set
score: 1.000
?source-set-564
clevr-object-set
score: 1.000
?source-object-254
clevr-object
score: 1.000


initial structure
application process
applied constructions
resulting structure

Meaning:

gstruct0(get-context?source-set-548-13)struct1(filter?source-set-563?source-set-548-13?category-363-13)struct0:varSOURCEdashSETdash548dash131->struct1:varSOURCEdashSETdash548dash132struct2(filter?source-set-564?source-set-563?category-371)struct1:varSOURCEdashSETdash5632->struct2:varSOURCEdashSETdash5633struct6(bindshape-category?category-363-13cube)struct1:varCATEGORYdash363dash132->struct6:varCATEGORYdash363dash137struct3(unique?source-object-254?source-set-564)struct2:varSOURCEdashSETdash5643->struct3:varSOURCEdashSETdash5644struct7(bindsize-category?category-371large)struct2:varCATEGORYdash3713->struct7:varCATEGORYdash3718struct4(query?answer-226?source-object-254?attribute-350)struct3:varSOURCEdashOBJECTdash2544->struct4:varSOURCEdashOBJECTdash2545struct5(bindattribute-category?attribute-350material)struct4:varATTRIBUTEdash3505->struct5:varATTRIBUTEdash3506

The following construction(s) were learned:

The following link(s) were added to the categorial network:

g0cyan-39991what-material-is-the-?x-cube-150->10.102big-3542->10.20

Comprehension succeeded

Interpretation finished

Computed answer:

material-category

The following cxns are punished:

Interaction failed


Interaction 6

The tutor is the speaker.

The learner is the listener.

Current Scene:

Topic:

material-category

Question: "What material is the big cube?"


Comprehending "What material is the big cube?"


Applying
g2big-3541what-material-is-the-?x-cube-152->10.200cyan-39991->00.10
in comprehension


Evaluating irl program

irl program
((get-context ?source-set-548-14) (filter ?source-set-565 ?source-set-548-14 ?category-363-14) (filter ?source-set-566 ?source-set-565 ?category-373) (unique ?source-object-255 ?source-set-566) (query ?answer-227 ?source-object-255 ?attribute-351) (bind attribute-category ?attribute-351 material) (bind shape-category ?category-363-14 cube) (bind size-category ?category-373 large))
gstruct0(get-context?source-set-548-14)struct1(filter?source-set-565?source-set-548-14?category-363-14)struct0:varSOURCEdashSETdash548dash141->struct1:varSOURCEdashSETdash548dash142struct2(filter?source-set-566?source-set-565?category-373)struct1:varSOURCEdashSETdash5652->struct2:varSOURCEdashSETdash5653struct6(bindshape-category?category-363-14cube)struct1:varCATEGORYdash363dash142->struct6:varCATEGORYdash363dash147struct3(unique?source-object-255?source-set-566)struct2:varSOURCEdashSETdash5663->struct3:varSOURCEdashSETdash5664struct7(bindsize-category?category-373large)struct2:varCATEGORYdash3733->struct7:varCATEGORYdash3738struct4(query?answer-227?source-object-255?attribute-351)struct3:varSOURCEdashOBJECTdash2554->struct4:varSOURCEdashOBJECTdash2555struct5(bindattribute-category?attribute-351material)struct4:varATTRIBUTEdash3515->struct5:varATTRIBUTEdash3516

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-227
material-category
score: 1.000
?attribute-351
attribute-category
score: 1.000
?category-363-14
shape-category
score: 1.000
?category-373
size-category
score: 1.000
?source-set-548-14
clevr-scene
score: 1.000
?source-set-565
clevr-object-set
score: 1.000
?source-set-566
clevr-object-set
score: 1.000
?source-object-255
clevr-object
score: 1.000


initial structure
application process
applied constructions
resulting structure

Meaning:

gstruct0(get-context?source-set-548-14)struct1(filter?source-set-565?source-set-548-14?category-363-14)struct0:varSOURCEdashSETdash548dash141->struct1:varSOURCEdashSETdash548dash142struct2(filter?source-set-566?source-set-565?category-373)struct1:varSOURCEdashSETdash5652->struct2:varSOURCEdashSETdash5653struct6(bindshape-category?category-363-14cube)struct1:varCATEGORYdash363dash142->struct6:varCATEGORYdash363dash147struct3(unique?source-object-255?source-set-566)struct2:varSOURCEdashSETdash5663->struct3:varSOURCEdashSETdash5664struct7(bindsize-category?category-373large)struct2:varCATEGORYdash3733->struct7:varCATEGORYdash3738struct4(query?answer-227?source-object-255?attribute-351)struct3:varSOURCEdashOBJECTdash2554->struct4:varSOURCEdashOBJECTdash2555struct5(bindattribute-category?attribute-351material)struct4:varATTRIBUTEdash3515->struct5:varATTRIBUTEdash3516

Comprehension succeeded

Interpretation finished

Computed answer:

material-category

Alignment started

The following cxns are rewarded:

The following cxns are punished:

Interaction succeeded


At this point, the agent has acquired an item-based construction and two lexical constructions that are sufficient to cover both of the observed questions.

Back to top of this page.

Back to the main page.