1. Learning holophrases

We start with the very first interaction of the experiment. When observing the question, the only thing the learner can do is to use intention reading to create a meaning hypothesis and to learn a hollistic mapping between the observed question and the constructed meaning hypothesis. At this point, the learner cannot know which parts of the form correspond with which parts of the meaning.

In the interactions that follow, you will see that the agents are repeatedly presented with the same question. Many meaning hypotheses generated by intention reading are not adequate representations of the meaning of that question. Hence, they fail to generalise across scenes. When the agent reuses a previously acquired holophrase construction and fails to achieve communicative success, intention reading is used to make a new meaning hypothesis. Importantly, intention reading ensures that it does not generate exactly the same meaning hypothesis as before, as it turned out to be unsuccessful.


Interaction 1

The tutor is the speaker.

The learner is the listener.

Current Scene:

Topic:

material-category

Question: "What material is the cyan cube?"


Comprehending "What material is the cyan cube?"


Applying
g
in comprehension


Computing next composer solution

using ontology:


Result

composition process

Composer queue:

Found 1 solutions:


Evaluating irl program

irl program
((bind shape-category ?category-364 cylinder) (bind attribute-category ?attribute-344 material) (query ?answer-220 ?source-object-248 ?attribute-344) (unique ?source-object-248 ?source-set-551) (filter ?source-set-551 ?source-set-552 ?category-364) (get-context ?source-set-552))
gstruct0(bindshape-category?category-364cylinder)struct4(filter?source-set-551?source-set-552?category-364)struct0:varCATEGORYdash3641->struct4:varCATEGORYdash3645struct1(bindattribute-category?attribute-344material)struct2(query?answer-220?source-object-248?attribute-344)struct1:varATTRIBUTEdash3442->struct2:varATTRIBUTEdash3443struct3(unique?source-object-248?source-set-551)struct2:varSOURCEdashOBJECTdash2483->struct3:varSOURCEdashOBJECTdash2484struct3:varSOURCEdashSETdash5514->struct4:varSOURCEdashSETdash5515struct5(get-context?source-set-552)struct4:varSOURCEdashSETdash5525->struct5:varSOURCEdashSETdash5526

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-220
material-category
score: 1.000
?category-364
shape-category
score: 1.000
?attribute-344
attribute-category
score: 1.000
?source-set-552
clevr-scene
score: 1.000
?source-set-551
clevr-object-set
score: 1.000
?source-object-248
clevr-object
score: 1.000


initial structure
application process
applied constructions
resulting structure

Meaning:

gstruct0(bindshape-category?category-364cylinder)struct4(filter?source-set-551?source-set-552?category-364)struct0:varCATEGORYdash3641->struct4:varCATEGORYdash3645struct1(bindattribute-category?attribute-344material)struct2(query?answer-220?source-object-248?attribute-344)struct1:varATTRIBUTEdash3442->struct2:varATTRIBUTEdash3443struct3(unique?source-object-248?source-set-551)struct2:varSOURCEdashOBJECTdash2483->struct3:varSOURCEdashOBJECTdash2484struct3:varSOURCEdashSETdash5514->struct4:varSOURCEdashSETdash5515struct5(get-context?source-set-552)struct4:varSOURCEdashSETdash5525->struct5:varSOURCEdashSETdash5526

The following construction(s) were learned:

Comprehension succeeded

Interpretation finished

Computed answer:

material-category

Interaction failed


Interaction 2

The tutor is the speaker.

The learner is the listener.

Current Scene:

Topic:

material-category

Question: "What material is the cyan cube?"


Comprehending "What material is the cyan cube?"


Applying
g
in comprehension


Evaluating irl program

irl program
((bind shape-category ?category-365 cylinder) (bind attribute-category ?attribute-345 material) (query ?answer-221 ?source-object-249 ?attribute-345) (unique ?source-object-249 ?source-set-553) (filter ?source-set-553 ?source-set-554 ?category-365) (get-context ?source-set-554))
gstruct0(bindshape-category?category-365cylinder)struct4(filter?source-set-553?source-set-554?category-365)struct0:varCATEGORYdash3651->struct4:varCATEGORYdash3655struct1(bindattribute-category?attribute-345material)struct2(query?answer-221?source-object-249?attribute-345)struct1:varATTRIBUTEdash3452->struct2:varATTRIBUTEdash3453struct3(unique?source-object-249?source-set-553)struct2:varSOURCEdashOBJECTdash2493->struct3:varSOURCEdashOBJECTdash2494struct3:varSOURCEdashSETdash5534->struct4:varSOURCEdashSETdash5535struct5(get-context?source-set-554)struct4:varSOURCEdashSETdash5545->struct5:varSOURCEdashSETdash5546

Applying



on the ontology


evaluation process

Solutions:

no solutions


Computing next composer solution

using ontology:


Result

composition process

Composer queue:

Found 2 solutions:

Checking solution 1 against 1 past scenes

Checking solution 2 against 1 past scenes


Computing next composer solution

using ontology:


Result

composition process

Composer queue:

... and 2 more

Found 15 solutions:

... and 12 more

All composer solutions so far

... and 14 more

Checking solution 1 against 1 past scenes

Checking solution 2 against 1 past scenes

Checking solution 3 against 1 past scenes

Checking solution 4 against 1 past scenes

Checking solution 5 against 1 past scenes

Checking solution 6 against 1 past scenes

Checking solution 7 against 1 past scenes

Checking solution 8 against 1 past scenes

Checking solution 9 against 1 past scenes


Evaluating irl program

irl program
((bind shape-category ?category-363-6 cube) (bind color-category ?category-366 cyan) (bind attribute-category ?attribute-346 material) (query ?answer-222 ?source-object-250 ?attribute-346) (unique ?source-object-250 ?source-set-555) (filter ?source-set-555 ?source-set-556 ?category-366) (filter ?source-set-556 ?source-set-548-6 ?category-363-6) (get-context ?source-set-548-6))
gstruct0(bindshape-category?category-363-6cube)struct6(filter?source-set-556?source-set-548-6?category-363-6)struct0:varCATEGORYdash363dash61->struct6:varCATEGORYdash363dash67struct1(bindcolor-category?category-366cyan)struct5(filter?source-set-555?source-set-556?category-366)struct1:varCATEGORYdash3662->struct5:varCATEGORYdash3666struct2(bindattribute-category?attribute-346material)struct3(query?answer-222?source-object-250?attribute-346)struct2:varATTRIBUTEdash3463->struct3:varATTRIBUTEdash3464struct4(unique?source-object-250?source-set-555)struct3:varSOURCEdashOBJECTdash2504->struct4:varSOURCEdashOBJECTdash2505struct4:varSOURCEdashSETdash5555->struct5:varSOURCEdashSETdash5556struct5:varSOURCEdashSETdash5566->struct6:varSOURCEdashSETdash5567struct7(get-context?source-set-548-6)struct6:varSOURCEdashSETdash548dash67->struct7:varSOURCEdashSETdash548dash68

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-222
material-category
score: 1.000
?category-363-6
shape-category
score: 1.000
?category-366
color-category
score: 1.000
?attribute-346
attribute-category
score: 1.000
?source-set-548-6
clevr-scene
score: 1.000
?source-set-556
clevr-object-set
score: 1.000
?source-set-555
clevr-object-set
score: 1.000
?source-object-250
clevr-object
score: 1.000


initial structure
application process
applied constructions
resulting structure

Meaning:

gstruct0(bindshape-category?category-363-6cube)struct6(filter?source-set-556?source-set-548-6?category-363-6)struct0:varCATEGORYdash363dash61->struct6:varCATEGORYdash363dash67struct1(bindcolor-category?category-366cyan)struct5(filter?source-set-555?source-set-556?category-366)struct1:varCATEGORYdash3662->struct5:varCATEGORYdash3666struct2(bindattribute-category?attribute-346material)struct3(query?answer-222?source-object-250?attribute-346)struct2:varATTRIBUTEdash3463->struct3:varATTRIBUTEdash3464struct4(unique?source-object-250?source-set-555)struct3:varSOURCEdashOBJECTdash2504->struct4:varSOURCEdashOBJECTdash2505struct4:varSOURCEdashSETdash5555->struct5:varSOURCEdashSETdash5556struct5:varSOURCEdashSETdash5566->struct6:varSOURCEdashSETdash5567struct7(get-context?source-set-548-6)struct6:varSOURCEdashSETdash548dash67->struct7:varSOURCEdashSETdash548dash68

The following construction(s) were learned:

Comprehension succeeded

Interpretation finished

Computed answer:

material-category

The following cxns are punished:

Interaction failed


Interaction 3

The tutor is the speaker.

The learner is the listener.

Current Scene:

Topic:

material-category

Question: "What material is the cyan cube?"


Comprehending "What material is the cyan cube?"


Applying
g
in comprehension


Evaluating irl program

irl program
((bind shape-category ?category-363-7 cube) (bind color-category ?category-367 cyan) (bind attribute-category ?attribute-347 material) (query ?answer-223 ?source-object-251 ?attribute-347) (unique ?source-object-251 ?source-set-557) (filter ?source-set-557 ?source-set-558 ?category-367) (filter ?source-set-558 ?source-set-548-7 ?category-363-7) (get-context ?source-set-548-7))
gstruct0(bindshape-category?category-363-7cube)struct6(filter?source-set-558?source-set-548-7?category-363-7)struct0:varCATEGORYdash363dash71->struct6:varCATEGORYdash363dash77struct1(bindcolor-category?category-367cyan)struct5(filter?source-set-557?source-set-558?category-367)struct1:varCATEGORYdash3672->struct5:varCATEGORYdash3676struct2(bindattribute-category?attribute-347material)struct3(query?answer-223?source-object-251?attribute-347)struct2:varATTRIBUTEdash3473->struct3:varATTRIBUTEdash3474struct4(unique?source-object-251?source-set-557)struct3:varSOURCEdashOBJECTdash2514->struct4:varSOURCEdashOBJECTdash2515struct4:varSOURCEdashSETdash5575->struct5:varSOURCEdashSETdash5576struct5:varSOURCEdashSETdash5586->struct6:varSOURCEdashSETdash5587struct7(get-context?source-set-548-7)struct6:varSOURCEdashSETdash548dash77->struct7:varSOURCEdashSETdash548dash78

Applying



on the ontology


evaluation process

Solutions:

solution 1
?answer-223
material-category
score: 1.000
?category-363-7
shape-category
score: 1.000
?category-367
color-category
score: 1.000
?attribute-347
attribute-category
score: 1.000
?source-set-548-7
clevr-scene
score: 1.000
?source-set-558
clevr-object-set
score: 1.000
?source-set-557
clevr-object-set
score: 1.000
?source-object-251
clevr-object
score: 1.000


initial structure
application process
applied constructions
resulting structure

Meaning:

gstruct0(bindshape-category?category-363-7cube)struct6(filter?source-set-558?source-set-548-7?category-363-7)struct0:varCATEGORYdash363dash71->struct6:varCATEGORYdash363dash77struct1(bindcolor-category?category-367cyan)struct5(filter?source-set-557?source-set-558?category-367)struct1:varCATEGORYdash3672->struct5:varCATEGORYdash3676struct2(bindattribute-category?attribute-347material)struct3(query?answer-223?source-object-251?attribute-347)struct2:varATTRIBUTEdash3473->struct3:varATTRIBUTEdash3474struct4(unique?source-object-251?source-set-557)struct3:varSOURCEdashOBJECTdash2514->struct4:varSOURCEdashOBJECTdash2515struct4:varSOURCEdashSETdash5575->struct5:varSOURCEdashSETdash5576struct5:varSOURCEdashSETdash5586->struct6:varSOURCEdashSETdash5587struct7(get-context?source-set-548-7)struct6:varSOURCEdashSETdash548dash77->struct7:varSOURCEdashSETdash548dash78

Comprehension succeeded

Interpretation finished

Computed answer:

material-category

Alignment started

The following cxns are rewarded:

The following cxns are punished:

Interaction succeeded


At this point, the construction inventory of the agent contains a single holophrase construction that covers the observed question. All other holophrase constructions failed to generalise across scenes. Because they were competitors of the single remaining holophrase construction, they were punished several times and eventually removed.

Back to top of this page.

Back to the main page.