Igor A. Mel’čuk and Alain Polguère, (1987) “A Formal Lexicon in Meaning-Text Theory”, Computational Linguistics, vol. 13, pp. 261-275.

portal.acm.org/citation.cfm?id=48160.48166
www.aclweb.org/anthology/J/J87/J87-3006.pdf

Within the context of that theory, the output of the Stanford parser is strictly at the SSynR or “surface syntactic representation” level, while, as a general rule Relex attempts to generate the DSynR or “Deep syntactic representation” structure.  Some of what I’ve been trying to do with opencog is towards the “SemR” structure, as described in that paper.

The more I read about MTT, the more it seems to capture some of what we are trying to do (defacto are doing) with NLP within opencog.  In particular, the MTT concept of a “lexical function” (which is not really described in that paper??) could be a particularly strong way of guaranteeing correct syntactic output for segsim, nlgen or NLGen2

– Linas Vepstas

I wrote up a few paragraphs on the most prominent differences; most of my updates were to the RelEx dependency relations page.

Here are the main bullet points:

• RelEx attempts basic entity extraction, and thus avoids generating nn noun modifier relations for named entities.
• RelEx will collapse the object and complement of a preposition into one. Stanford will do this for some, but not all relationships.
• RelEx will convert passive subjects into objects, and instead indicate passiveness by tagging the verb with a passive tense feature.
• RelEx avoids generating copulas, if at all possible, and instead indicates copular relations as predicative adjectives, or in other ways.
• RelEx extracts semantic variables from questions, with the intent of simplifying question answering. For example, “Where is the ball?” generates _pobj(_%atLocation, _$qVar) _psubj(_%atLocation, ball), which can then pattern-match a plausible answer: _pobj(under, couch).
• RelEx attempts to extract comparison variables.

Its also clear to me that I could split up the relex processing into two stages: one which generates stanford-style syntactic relations, and a second stage that generates the more abstract stuff. This might be a wise move … Since RelEx is already more than 3x faster than the Stanford parser, this could attract new users.

– Linas Vepstas

Patten matching is now used through-out all of the OpenCog NLP pipeline, although not in a unified manner. The Link Grammar parser uses patterns (called “disjuncts”) to determine how the words in a sentence can link to one-another, thus “parsing”, or pulling the grammatical structure out of a sentence (this paper provides an excellent overview). The RelEx dependency relation extractor applies patterns on the link-grammar output to extract syntactic relations. For example, the sentence “John threw a rock” becomes

_obj(throw, ball)
_subj(throw, John)

after RelEx gets done with it. And now, there are a dozen patterns inside of OpenCog that can pick out certain kinds of questions and statements from RelEx output, and pattern-match questions to find answers to them.

For example, the new OpenCog patterns convert “The capital of France is Paris” into

capital_of(France, Paris)

and similarly, “What is the capital of France?” into

capital_of(France,what)

Treating “what” as a variable, there is yet another pattern that matches up the form of the question to the form of the answer, thus deducing that “what” must be “Paris”.

Somewhat harder is using patterns to distinguish similar from dis-similar concepts, so that sentences like “John threw a green ball” aren’t used as answers to questions such as “Did John throw a red ball?”: the word “ball” with modifier “green” has to be detected as a different entity than the word “ball” with modifier “red”: these are two different entities (called “semes” in the code). In fact, out of laziness, I’ve punted on this one: the promotion of word-instances to “semes” is done by code, rather than by pattern matching. But soon, I hope, this will change. In the meanwhile, the README file provides a more detailed discussion.

Here are some patterns that work these days:

<me>         John threw a green ball.
<me>         Fred threw a red ball
<me>         Mary threw a blue rock
<me>         who threw a ball?
<cogita-bot> Syntax pattern match found: Fred John
<me>         who threw a red ball?
<cogita-bot> Syntax pattern match found: Fred

<me>         Did Fred throw a ball?
<cogita-bot> Truth query determined “yes”: throw

<me>         Did Fred throw a red ball?
<cogita-bot> Truth query determined “yes”: throw

<me>         The color of the book is red.
<me>         What is the color of the book?
<cogita-bot> Triples abstraction found: red

<me>         the cat sat on the mat
<me>         what did the cat sit on?
<cogita-bot> Triples abstraction found: mat

And here are some that don’t yet work: “Did Fred throw a green ball?” — gets no reply, because the system can’t find an answer, and doesn’t make the common-sense leap of “can’t find answer-> answer must be no”. Another common-sense problem is illustrated by: “Did Fred throw a round ball?” — the system doesn’t know that balls are round, and simply assumes that a “round ball” is some special kind of “ball”. Oh well. There’s work to be done.

You can try out the chatbot yourself (when its up, and not broken!) on the IRC chat channel #opencog on the freenode.net chat servers.

– Linas Vepstas

MVp- Js+

is a disjunct that says “there must be an MVp link from this word, going to the left, and an Js link, going to the right”. This disjunct commonly connects prepositions to a verb on their left (the MV- link) and the object of the preposition on the right (the J+ link).

A good way to think about disjuncts is to imagine them as very fine-grained part-of-speech tags. Thus, when one sees “MVp- Js+” associated to a word, one knows not only that the word is a preposition, but even a bit more: its a preposition that took a singular object.  Disjuncts classify words not just into crude part-of-speech categories, but much finer categories:  thus verbs are not just as transtivie or intransitive verbs, but mgiht be transitive verbs that take both direct and indirect objects, or participles, etc.

Siva Reddy, a GSOC 2009 summer student, prepared a table of the frequency of occurrence of different disjuncts in a large collection of text. The top six entries are

Ds+           950275.635843
Xp-           838569.90527
A+          616522.664867
AN+        566658.997313
MVp- Js+       563082.649325
MVp- Jp+      446487.310222

and these are exactly what one might expect:

• Ds+ connects the determiner “the” to nouns: and of course, “the” is the most frequent word in the English language.
• Xp- connects the period at the end of the sentence to the start of the sentence, so of course its frequently observed.
• A+ connects adjectives to nouns, AN+ connects noun modifiers to nouns.
• As noted above, MV connects verbs to modifying phrases, and J connects prepositions to objects, so that MV- J+ is the disjunct that most prepositions will get. Js connects to a singular object, Jp connects to a plural count or mass noun.

A graph of rank vs. frequency is shown below:

Disjunct rank vs. frequency of occurance

As can be seen, the distribution is more or less Zipfian, with a power-law exponent of 1.5.  The fact that the long tail appears to be linear indicates that grammatical construction in the English language appears to be more ore less scale-free: difficult and akward constructions are increasingly rare.  The fact that the graph is not purely Zipfian, but instead has a knee for the most common grammatical connections suggests that the most common grammatical constructions are “less common than they should be”: almost as if English speakers are resisting the use of formulaic sentence constructions. So, for example, since adjectives and noun-modifiers appear near the top of the rank, this suggests that English speakers “could have” used more adjectives and noun-modifiers, but didn’t. Quite why this is so is not clear.  Perhaps the use of anaphora and references in general  helps decrease the need for lots of modifiers.

The open questions are then:

1. Why a power law of 1.5?
2. Why is there a knee?
3. Does this result hold for other languages?

The corpus used here consists of approximately 1 million sentences, obtained by parsing entire Wikipedia articles, Voice of America news stories, and 10 books from Project Gutenberg, including War and Peace, Jane Austen, and some scientific or medical texts.

– Linas Vepstas

<linas> Mary ate a mango
<cogita-bot> Hello linas, parsing ...
<cogita-bot> linas, you made a statement: Mary ate a mango
<linas> what did Mary eat?
<cogita-bot> Hello linas, parsing ...
<cogita-bot> linas, you asked a question: what did Mary eat?
<cogita-bot> The answer to your question is: mango

Its meant to be a demo of the basic NLP pipeline within OpenCog. It takes input text, runs it throught the Link Grammar + RelEx parser, imports the results into the OpenCog atomspace, sucks in a small common-sense database, and waits for questions to be posed. The common-sense database is derived from MIT’s ConceptNet (OpenMind/CommonSense project), and so one can have interactions like the following:

<linas> cogita-bot: what is a saxophone?
<cogita-bot> Hello linas, parsing ...
<cogita-bot> linas, you asked a question: what is a saxophone?
<cogita-bot> No answer was found to your question.
<linas> hmm
<linas> cogita-bot: what is an instrument?
<cogita-bot> Hello linas, parsing ...
<cogita-bot> linas, you asked a question: what is an instrument?
<cogita-bot> The answer to your question is: woodwind r bass harmonica An_Oboe Oboe megaphone saxophone chronometer drum scale ukulele cymbal instrument
<linas> Heh. Complete with assorted linguistic garbage

You get the idea. Don’t ask it anything more complicated than the above examples: it will fail to find any answer. Again, it does no reasoning at all. Its as thick as a brick. You can test-drive it at the #opencog channel on the freenode.net IRC network.  Assuming its not down for development.

Next up: wire in NLGen for natural-language output, and start taking baby steps in actual reasoning. Anyway, I’m pretty excited, as this means that a lot of the basic bits&pieces are working, and I can now dive into the deep end, and start working on the hard stuff.

– Linas Vepstas

Mutual information of nearby words

Above is a graph showing the distribution of the mutual information of word pairs that occur in the same sentence. A number of texts were analyzed, including a portion of Wikipedia, some books from project Gutenberg, etc. A collection of all possible pairs of words was created, where each word in the pair occurs in the same sentence (with the left word of the pair having occurred in the sentence to the left of the right word in the pair). These were counted — about 10 million word pairs were observed — and their mutual information was calculated.

Mutual information is a measure of the likelihood of seeing two words occur together: thus, for example “Northern Ireland” will have a high mutual information, since the words “Northern” and “Ireland” are used together frequently.  By contrast, “Ireland is” will have negative mutual information, mostly because the word “is” is used with many, many other words besides “Ireland”; there is no special relationship between these words. High-mutual-information word pairs are typically noun phrases, often idioms and “collocations”, and almost always embody some concept (so, for example, “Northern Ireland” is the name of a place — the name of the conception of a particular country).

In mathematical terms, the mutual information of a word pair (x,y) is defined as:

M(x,y) = log_2  P(x,y) / P(x,*) P(*,y)

where P(x,y) is the probability of seeing the word pair (x,y), P(x,*) is the probability of seeing a word pair where the left word is x, and P(*,y) is the probability of seing a word pair where the right word is y.

The graph shows M(x,y) on the horizontal axis, and the probability of seeing such a value of M on the vertical axis. This is a bin-count of the distribution of possible values of mutual information, over all word pairs.  This is *NOT* a scatterplot of M(x,y) vs. P(x,y).

Here’s another graph: same as above, except that this time, only pairs of words that occur immediately next to one-another are considered.  The sample size is much smaller: only about 2.4M word-pairs were collected.

Mutual Information of Neighboring Word Pairs

The blue and green exponential lines are located in exactly the same place as in the previous graph. It’s humped in a different way than the previous graph. What is the shape of this hump?  Are the slopes characteristic, or do they vary from one corpus sample to another?  If anyone knows the answers to these questions, please let me know!

Notice the peaks off to the right, at high MI values, in the first graph. I think these are word pairs which are heavily used (topics/terms that are discussed) in one single contributing text, but in none of the others. That’s the hypothesis, I don’t know.

Here is a more detailed discussion, with many other additional figures.

– Linas Vepstas

The basic observation driving this work is that fairly often, one can identify the meaning of a word (or at least narrow it down) simply by observing how it is used in a sentence. To do this, one needs accurate, fine-grained syntactical information about a sentence: and Link Grammar provides an abundance of it. Link Grammar generates very fine-grained syntactic linkage information for every word in a sentence. Every dictionary word is associated with dozens, or even hundreds, of different linkage patterns, or ‘disjuncts’. These disjuncts indicate how a word in a sentence can be connected to other words to its left or right.  One may imagine that these disjuncts provide highly detailed grammatical information about a word. For example, they not only distinguish between a noun and a verb; they not only distinguish between present, past and future tenses of a verb, they not only distinguish between a transitive and an intransitive verb, but they also distinguish between a wide variety of relationships that are so fine that most are not even given formal names by linguists. This fine-grained information is a rich source of information, and is ideal for correlating with word senses.

For word-sense tagging, it turns out that simply picking the Most Frequent Sense (MFS) already gives a rather strong indication of what the correct word-sense assignment should be — it gives the correct answer about half the time– See [Mi04] below. So the idea here is that supplementing the MFS with additional data– how the word was used syntactically — will improve accuracy even further. That is, the idea is to compute the “MFS for a given syntactic context”.

Before one can perform statistical correlations between syntax and sense, one must first assign a sense to each word.  This is, of course, the really hard part to modern NLP. For now, I’m taking a fairly easy, straightforward, yet strong approach to this: I’m using  an algorithm due to Rada Mihalcea[Mi05].  This algorithm is currently, as far as I know, the most accurate algorithm known for tagging words with word senses. Unfortunately, it is also fairly slow and CPU intensive, making practical deployment tricky.  It performs this tagging by associating, with each word in a sentence, a list of its possible senses.  Then, senses of nearby words are linked together with a similarity measure. This forms a network, a graph, over the sentence, with vertices being word-senses, and edges being weighted by the similarity between senses. Such a network is formally a Markov chain, and can be solved as such. There are many ways of solving a Markov chain; Mihalcea proposes, and I’ve implemented, the Google (Page-Brin) page-rank algorithm[PB]. The result is that each vertex (each word-sense) is assigned a probability; The highest-probability senses do indeed appear to be the linguistically correct senses most of the time; the correct sense will almost always appear in the top three probabilities.

Once a word sense is identified, it can be correlated with the Link Grammar disjunct in play for the particular sentence.  This is done simply by processing a lot of sample text. A database then stores a frequency count for each (word, disjunct, word-sense) triple. After a reasonable amount of data is accumulated the unconditional probability p(w,d,s) can be calculated (w==word, d==disjunct, s==sense), and, from this, various marginal and conditional probabilities and entropies. To make use of this information in a new sentence, one first parses the sentence using the Link Grammar parser, thus obtaining (word,disjunct) pairs. It is then a straightforward (and fast) database lookup to obtain the conditional probability p(s|w,d), the probability of observing the sense s given the pair (w,d).  The exciting result of this effort is that, quite often, the conditional probability p(s|w,d) identifies one sense more or less uniquely (i.e. there is one sense for which p(s|w,d) is about 1).

Although this result is quite exciting, its based on the inspection of a small handful of nouns and verbs. I believe that the result holds well in a broad setting, but I don’t have any quantitative measure for the extent of the setting. Clearly, there will be *some* words for which the sense will be obvious from the grammatical usage. But, on average, how many of these are there per sentence?  In some cases, the sense won’t be unique, but there will be many senses that are ruled out. How often does this happen?  Is it possible that the accuracy results are equal to, or even improve, on the Mihalcea accuracy results? (They may improve on them by averaging over and eliminating false-positives, eliminating them because of the various different semantic contexts a word might appear in).  A quantitative measure of the recall and accuracy can, in principle, be done, as there is a database (the SemCor database) of text that has been hand-annotated with the correct senses.  I’ve not yet given any serious thought to performing this quantitative analysis.

Still, I’m pretty excited. It seems to work pretty well; I like that. I’ve already roughed in some basic infrastructure into the Link Grammar parser so that it will return the sense tags for each parse, assuming you have the database installed.  The tags returned are WordNet 3.0 sense keys — strings like “run%2:38:04::” which can be used to look up specific senses from WordNet.

To expose this function, database support has been added to the link-grammar parser.  This has been added to the parser itself, as opposed to a layer built on top of it, because database support is needed for other reasons — specifically, for parse ranking (Gee, I haven’t talked about parse ranking, have I?).  The database support is provided by sqllite[SQLLITE]. This was picked for two reasons: (1) its license is public domain, and is thus compatible with the link-grammar BSD license, and (2) it is an embedded database, requiring zero administration by the user. This second point is quite unlike traditional SQL databases, which typically require trained database administrator to configure and operate. One reason that zero administration is possible is because the database is used in a read-only fashion: the data it holds is static. Code integrating this database is in the link-grammar SVN repository now, and will be available in version 4.4.2.

Creating the dataset is a good bit tricker. Currently, the Mihalcea algorithm is implemented within OpenCog.  Was this a good technology choice? I dunno, but it seemed like a reasonable experiment at the time. Parsed sentences are fed to OpenCog, where word senses are assigned, and then frequency counts are updated.  I’ve been feeding it a diet consisting solely of parsed Wikipedia articles — not very healthy, but maybe OK for now.  The Markov chain network used to solve for word senses is four sentences wide, as a window sliding across an article. That is, a given word sense is influenced by other words occurring in sentences as far as four sentences away.  This should keep accuracy up, without bogging down in solving a Markov chain across an entire article. The current OpenCog implementation uses the Page-Brin PageRank algorithm; however, I’m thinking tht it might be faster simple to use the linpack subroutine library to solve for the eigenvectors directly. (The Page-Brin algorithm shows its power when the Markov chain has  billions or trillions of nodes, e.g. as used by Google.  By contrast, with a four-sentence sliding window, the Markov matrix connects at most thousands of senses, and thus should be rapidly solvable by ordinary linear equation techniques.)

So far, I’ve put a few CPU-months of data crunching into this. Its not much. It’s slow; it takes minutes per sentence — and this gives only one word-sense, disjunct pair. To build up a reasonable statistical dataset will require cpu-hours or more per word — and there’s not really all that many cpu-hours in a cpu-month. So its slow slogging. The  database coverage remains quite thin, as most disjuncts have been observed only a handful of times. There are maybe about a thousand or so words for which an ‘adequate’ amount of statistics have been collected; I’d like it better if the  database was deep enough to cover at least 20K words and 100K senses.  Since the preliminary results look so promising, I’m corssing my fingers, and am slowly been tinkering with ways to improve performance.

I’ll let you know …
References
==========
[Mi04] Rada Mihalcea, Paul Tarau, Elizabeth Figa, “PageRank on Semantic Networks, with Application to Word Sense Disambiguation” (2004) COLING ’04: Proceedings of the 20th international conference on Computational Linguistics DOI
[Mi05]  Rada Mihalcea, “Unsupervised Large-Vocabulary Word Sense Disambiguation with Graph-based Algorithms for Sequence Data  Labeling”, (2005) HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing DOI

[SQLLITE] http://www.sqlite.org/

[PB] http://en.wikipedia.org/wiki/PageRank

– Linas Vepstas

Solving these required adding new rules to the link-grammar dictionary: so, for example, adding the rule “O+ &amp; {@MV+}” to the dictionary entry for “asked” — thus allowing the word “asked” to take a direct object (“…asked Jim…”) and an indirect object (“… a qeustion”).

Patching dictionary entries by hand is tedious and time consuming. The long term goal is to get to the point where the system can learn new entries on its own. So I’ve been day-dreaming about how to do that, and some of the baby-steps in that direction.

The first step is to put all of the rules into a database where they can be easily modified, added-to, and deleted. Currently, the rules all live in a file, which can only be hand-edited.

A second step is to assign probabilities to the rules: often, multiple parses are possible, yet not each rule is equally likely to lead to a correct parse. So it would be better to have a probability assigned to each rule, indicating how often it leads to a good parse. These probabilities are already needed for the GSoC project of adding a SAT solver to link-grammar. I’ve got a database of tens of millions of word pairs and link types, ready to go for this step.

A third step is to distinguish good parses from bad ones: this is the parse-ranking step: to judge parsees for the likely-hood of being good, or bad. There are superficial things one can do for parse ranking, and quite complex ones. I am hoping to someday-soon reinforce the parse-ranking scores with the results of word-sense disambiguation. Which brings us to the next point:

Step four — some grammar parsing rules are appropriate only for some senses of a word, but not for others. Link grammar already accounts for this at a rough scale: it has distinct rules for different parts of speech. These are indicated by tacking a single extra letter to the end of a word: walk.v (I walk.v) versus walk.n (I took.v a walk.n) This basic idea can be further refined, for finer divisions than just parts of speech: some word senses can only be used in certain ways, and not others. The technical problem is that 26 letters is not enough… already, link-grammar is using some 20 or so of these suffixes. So this mechanism needs to be expanded, somehow.

Step five — where teh rubber hits the road — actually learning new rules. This is much more vague; my ideas are still swimming. There are two distinct problems: new words, and fixing rules for existing words. New words should not be *much* of a problem: try to find synonyms for a word, and assume the new word can be used like the synonym. Modifying existing rules is much much harder…

… so hard, in fact, that I’m not sure I’m ready to engage in that, just yet. Some steps can be taken in that direction, by looking at minimum-spanning-tree dependency grammars (MST grammars): that is, computing the mutual information of word pairs, and comparing them to the output of link-grammar. This should suggest where new links can be created. Then comes the question: what should the appropriate link type be, for this new link? For some words, perhaps synonyms can help, but other words are so unique, that they have no synonyms: the word “to be” is so central to the English language that trying to discover information about it is very difficult: it has no synonyms, even as it has many.

I’ve got some infrastructure set up to run this last experiment: I’ve got a fairly large collection of parsed sentences, from which I can build mutual information pairs, and compare these to link-grammar parses. In fact, I’ve already done so: I use these for parse ranking. (that is, if a link-grammar parse has a high mutual information content, then I assume its a good parse).

I could turn this around: given a sentence that is parsing badly, I can look for high mutual-info parses. Perhaps broaden the coverage by comparing to parses of approximately synonymous words, perhaps reinforced by word-sense disambiguation. But this generally leads to a combinatorial explosion — which, I guess is expected. We know that general intelligence requires a lot of CPU. The trick is to have the patience to set up an experiment, run the experiment, wait for its results, and then do it again…

Enough for now. I’m off to work on “related words” — another part of the puzzle.

To quote Ben Goertzel’s post on opencog@googlegroups.com:

The Google Summer of Code selection process is done, and 11 proposals were chosen.

It was a really painful process to go through, as we had more than 70 applications, and at least 25-30 of them were really quite good.

The accepted proposals span a fairly wide variety of areas, and the choices were ultimately made based on a number of factors including

– clarity and completeness of the proposal
– background of the student
– readiness of the OpenCog codebase for the project
– critical-ness of the project for OpenCog

Of the 11 selected, 2 were for OpenBiomind projects, and the other 9 for OpenCog proper … including a bunch of stuff for the RelEx NLP toolkit.

There was a strong bias toward proposals dealing with improvements to OpenCog-related software components that already are moderately mature, like RelEx and MOSES.

Next year when OpenCog is more mature, if we are chosen to participate in GSoC again (as we hope, and have reason to somewhat expect), you can expect to see more explicitly, broadly, AGI-related proposals.

Anyway this list of selected projects is here for all who are curious:

OpenSim for OpenCog
by Kino High Coursey, mentored by Andre Luiz de Senna

Implementing a SAT/SMT Based Link Grammar Parser
by Filip Marić, mentored by Predrag Janicic

Bayesian and Causal Networks Inference using Indefinite Probabilities
by Cesar Augusto Cavalheiro Marcondes, mentored by Cassio Pennachin

Java GUI for OpenBiomind
by Bhavesh Sanghvi, mentored by Murilo Saraiva de Queiroz

MOSES: the Pleasure Algorithm
by Alesis Novik, mentored by Nil Geisweiller

Graph Algorithms for HyperGraphDB
by Guo Junfei, mentored by Ben Goertzel

Improved MOSES
by ChenShuo, mentored by Moshe Looks

RelEx Web Crawler and HypergraphDB Manager
by Rich Jones, mentored by David Hart

RelEx: Learning Simple Grammars
by Elizabeth Dawn Alpert, mentored by Lukasz Kaiser

Distributed HipergraphDB Version
by Costa Ciprian, mentored by Borislav Iordanov

Recursive Feature Selection for Enhancing Genetic Disease Prediction
by Paul Cao, mentored by Lucio de Souza Coelho

Many thanks to all who applied, all who agreed to help mentor … and especially to David Hart for coming up with the idea of applying for SIAI to be included as a mentoring organization in GSoC, with a focus on OpenCog work.

We’d also like to thank the terrific Open Source team at Google, particularly Leslie Hawthorn, Dave Anderson and Chris DiBona, for their patience and good advice.