Knowing .NET

I used to be on top of  Artificial Intelligence — I wrote a column for and ultimately went on to be the Editor-in-Chief of AI Expert, the leading trade magazine in the AI field at the time. I’ve tried to stay, not professionally competent, but familiar with the field. That has been rather difficult because the AI field has largely put aside grand theories and adopted two pragmatic themes: statistical techniques and mixed-approaches.

Statistical techniques rely on large bodies of data that allow you to guess, for instance, that “push comes to”->”shove” not from any understanding of metaphor or causation but because the word “push” followed by “comes” followed by “to” is followed 87.3% of the time by the word “shove”. Statistics excel at extracting patterns from large input sets.

Mixed approaches are ones which use different strategies to try to tackle different aspects or stages of a problem. Imagine a blackboard around which people raise their hands, come forward, add or erase a small bit of information, and step back into the crowd. For instance, one (relatively) simple tool might know that “X comes to Y” implies temporal ordering. Another might say that temporal ordering implies escalation. And another might say “A ‘Shove’ is an escalation of a ‘Push’”.

The more I read about Watson, the more it seems that while Watson used mixed approaches, what it’s mixing are almost all statistical techniques. So while it would undoubtedly be able to answer that “shove” is what “push often comes to…” I think it would do so without any reasoning, or schema, about temporal ordering or escalation.

The problem with statistical techniques is they are not general.

If a child is shown how to win tic-tac-toe by always starting with a ‘X’ in the upper-left box, and then we asked them if they could always win by starting in another corner, we would be disappointed if they couldn’t figure it out. Maybe not at first, but if tic-tac-toe was something they enjoyed, they would eventually recognize the pattern. If they never achieved the recognition, it would be troubling.

Pattern recognition, not pattern extraction, seems to be “how” we work. If pattern extraction were at the core, we wouldn’t be troubled by sharks when entering the ocean and we wouldn’t spend money on lottery tickets.

So it seems that Watson uses a fundamentally different “how” in its achievement. Yet the capability of rapidly and accurately answering questions (ones that have been intentionally obfuscated!) is clearly epochal. Clearly Watson has a role in medicine (diagnostics), law and regulatory compliance (is there precedent? is this a restricted behavior?), and intelligence (where’s the next revolution likely?). The problems of “Big Data” are very much in the mind of the software development community and Watson is a stunning leap forward in combining big data, processing power, and specialized algorithms.

ResolverOne is one of my favorite applications in the past few years. It’s a spreadsheet powered by IronPython. Spreadsheets are among the most powerful intellectual tools ever developed: if you can solve your problem with a spreadsheet, a spreadsheet is probably the fastest way to solve it. Yet there are certain things that spreadsheets don’t do well: recursion, branching, etc.

Python is a clean, modern programming language with a large and still-growing community. It’s a language which works well for writing 10 lines of code or 1,000 lines of code. (ResolverOne itself is more than 100K of Python, so I guess it works at that level, too!)

From now (Dec 2008) to May 2009, Resolver Systems is giving away $2K per month to the best spreadsheet built in ResolverOne. The best spreadsheet received during the competition gets the grand prize of an additional $15K.

Personally, it seems to me that the great advantage of the spreadsheet paradigm is a very screen-dense way of visualizing a large amount of data and very easy access to input parameters. Meanwhile, Python can be used to create arbitrarily-complex core algorithms. The combination seems ideal for tinkering in areas such as machine learning and simulation.

I try to do some recreational programming every year between Christmas and New Year. I’m not sure I’ll have the time this year, but if I do, I may well use ResolverOne and Python to do something.

 import sys import clr sys.path.append("c:\\program files\\Microsoft Research\\Infer.NET 2.2\\bin\\debug") clr.AddReferenceToFile("Infer.Compiler.dll") clr.AddReferenceToFile("Infer.Runtime.dll") from MicrosoftResearch.Infer import * from MicrosoftResearch.Infer.Models import * from MicrosoftResearch.Infer.Distributions import *  firstCoin = Variable[bool].Bernoulli(0.5) secondCoin = Variable[bool].Bernoulli(0.5) bothHeads = firstCoin & secondCoin ie = InferenceEngine() print ie.Infer(bothHeads) --> c:\Users\Larry O'Brien\Documents\Infer.NET 2.2>ipy InferNetTest1.py Compiling model...done. Initialising...done. Iterating: .........|.........|.........|.........|.........| 50 Bernoulli(0.25) 

Sweet

The July 08 (Vol. 30, #7) IEEE Transactions on Pattern Analysis and Machine Intelligence has an incredible paper by Raykar, Duraiswami, and Krishnapuram. A Fast Algorithm for Learning a Ranking Function from Large-Scale Data Sets appears to be a game-changer for an incredibly important problem in machine learning. Basically, they use a “fast multipole method” developed for computational physics to rapidly estimate (to arbitrary precision) the conjugate gradient of an error function. (In other words, they tweak the parameters and “get a little better” the next time through the training data.)

The precise calculation of the conjugate gradient is O(m^2). This estimation algorithm is O(m)! (That’s an exclamation point, not a factorial!)

On a first reading, I don’t grok how the crucial transform necessarily moves towards an error minimum, but the algorithm looks (surprisingly) easy to implement and their benchmark results are jaw-dropping. Of course, others will have to implement it and analyze it for applicability across different types of data sets, but this is one of the most impressive algorithmic claims I’ve seen in years.

Once upon a time, I had the great fortune to write a column for a magazine on artificial intelligence and could justify spending huge amounts of time implementing AI algorithms (well, I think I was paid $450 per month for my column, so I’m not really sure that “justified” 80 hours of programming, but I was young). Man, would I love to see how this algorithm works for training a neural network…

According to InformationWeek, Umakant Sharma, seeded 2nd in a tournament in New Delhi, was caught with a Bluetooth headset stitched into a cap that he wore “pulled down over his ears” during competition. According to the All India Chess Federation, accomplices fed him moves from a chess program. He’s been banned by FIDE for 10 years.

This reminds me of something I’ve discussed before — during Gary Kasparov’s famous 1997 battle with Deep Blue, he demanded that the program’s code be escrowed because Kasparov was of the belief that no computer could generate such play and that a human or humans must be feeding the machine moves. That response — an expert in his domain asserting that computer behavior “must be from a human” – always struck me as more important than the ability of the computer to ultimately grind down the world’s best chess player. The response was the first, and to date, closest thing to a triumph in the Turing test.

I have to admit it also reminds me of my own scandalous behavior in 3rd grade, when as a Cub Scout I made a pinewood derby car into which I could slip a fishing weight after the official weighing. I was caught because my car didn’t just win the race, it ran down the ramp about twice as fast as anything else (objects of different mass might fall in a vacuum at equal speed; objects with wire axles running through a wood block, not so much). Needless to say, that was the end of my time in scouting. (Although, to be fair, they actually wanted to have some kind of disciplinary thing and then let me continue. Somehow I never made it over to the Brennan’s house for that meeting.)

The Netflix optimization challenge exemplifies a situation for which there should be a solution, but which I’ve not seen a good answer. Namely: for-profit but initially ad-hoc cooperation. For instance, let’s just say that I made the case that a Pandora-like “Movie Genome Project” was the key to winning the prize. And let’s say that you are sitting on top of a data-mining algorithm that you think will work great in conjunction with such a database. And let’s say that there are 5 people who, reading this, think “Well, I might not be able to contribute an algorithm, but for a piece of $1M, I’d be willing to ‘genotype’ some movies.”

The problem is: how do we go about working with each other? In the Open Source world, one can prototype a project by throwing it against the wall and seeing if it sticks: if people contribute or show interest, one can make a judgment about continuing or discontinuing the project. This works because all work, whether prototype or production, is given the same (free) value. However, if there’s money at stake, one cannot begin prototyping until “what if we win?” is sorted out. Even more importantly, the payoff percentages seem to necessarily be predetermined even though the relative contributions of all parties to the task won’t be known until after-the-fact.

I wonder if some derivative of a fairness-ensuring “cake cutting” algorithm can be applied to the problem.

Scott Swigart points to this article that gives a non-technical overview of the use of genetic algorithms to determine the optimal tuning characteristics of one’s Linux Kernel. This ought to work: many years ago I wrote a genetic algorithm that tuned the optimization parameters of one’s C++ compiler and it worked perfectly (well, who knows if it worked perfectly, but it did create better runtime performance than one would generally get from naive optimization options).

Netflix is offering $1M to the first person who can achieve 10% better movie recommendations than their current system. Sweet.

I have all sorts of ideas on this. Thank heavens I have copious spare time.

Infants exposed to a Mandarin-speaking adult for less than 5 hours (25-minute sessions over 4 weeks) were “able to distinguish phonetic elements of that language.” Very impressive. But infants exposed to a similar amount of speech delivered over DVD could not. Fascinating. General-audience article here (via The Old New Thing)

My guess is that there’s a difference-in-kind to the type of attention that infants pay humans to the type of attention they pay brightly flickering screens. It would be interesting to see the effect of exposure to, say, a guy in a big purple talking dinosaur suit. Or is the key ingredient perhaps, non-verbal facial communication (eyes and so forth)?

I guess I can avoid “Offtopic” by slotting this under “AI”…

Drools.NET is a port of the Drools library to .NET. I have a bg, big architectural decision coming up for a client and I am debating about whether to tackle the issue with an inference engine or a scripting language. So I’ve been looking at Drools pretty closely. It’s okay. I wouldn’t put it in the same league as ILog JRules, but the price is right and it seems to have momentum.

In the .NET front, I’ve been told there’s a Rete-based inference engine inside BizTalk (?), but I never followed up on that. Another tool is mTuitive’s xPert Authoring Environment. More to check out….