Software Development

On chat bots that understand context in discussions

Daniel — Wed, 05 Apr 2006 22:05:10 GMT

I played with the OpenNLP librabry and it is a nice port of the java nlp library. (Natural Language Processing)

I am toying with the idea of creating a chat bot that tries to extract the context of what is being discussed and formulate responses based on context instead of keywords only.

Then the next step would be to train the bot by observing other people in discussions and learn how other people respond given certain context. For example, bank & bills => sadness, stress, saving etc... bank & payday => happiness, shopping, speding etc... I know it's a very crude example however this could work.

The advantage here is that all contextual responses can be learned and do not have to be hard wired into the chat bot, even better, giving the bot the ability to learn and make meaningful transitions as context changes thoughout the discussion.

Hmmm...back to work.

Mozilla Thnderbird nightly builds

Daniel — Wed, 22 Mar 2006 14:39:35 GMT

http://www.mozilla.org/developer/#builds

Releases 3.0x alpha have some cool new features, improved new mail notification as well as RSS feeds support.

C# String Formatting

Daniel — Mon, 20 Mar 2006 10:20:50 GMT

Great discussion at regarding String.Format(...) calls, it's a good summary and referrence for dates and other types.

http://blog.stevex.net/index.php/string-formatting-in-csharp/

On data sharing, thread safety and priority queues.

Daniel — Thu, 28 Jul 2005 11:47:12 GMT

Good article for basic understanding of thread safety and static utility methods. I think the article does a good job with data sharing examples.

http://http://odetocode.com/Articles/313.aspx

I would only add that it is not only important to watch out for data sharing between threads, but also access and sharing of I/O resources like db connections, files and sockets.

One way would to use lock(obj){ ... } mechanism, where the system runtime will manage its own queue (serialized access), or another way would be to implement your own access queue mechanism.

Implementing your own queue mechanism can give you advanced features, should you need to implement them, such as priority scheduling and better micro-level locking.

By micro-level locking I mean the following. Let's say you have a function named 'SaveFile(...)' which uses lock(obj){...} mechanism to serialize access to a particulat file resource, it is possible to implement this in such a way that a unique lock object is used for each unique file name.

Better, queue can be scanned/sorted by resource objects being accessed and allow concurrent queue processing as long as each work item in the queue processed is accessing a different resource.

In essence, this turns one queue into a multi-queue mechanism where each shared resource has it's own queue. This greatly improves performance and would be similar to record-level locking mechanisms in databases, but instead could be applied to sockets, files, etc...

Lastly, in such multi-queue system priority scheduling can be implemented where queue management code can select by weight or some other calculated heuristic how much time should be dedicated to each queue.

Blog comment spam keeps on coming.

Daniel — Thu, 23 Jun 2005 12:02:48 GMT

Even with Captcha enabled, comment spam keeps appearing on blogs. Either spammers have the ability to do OCR on captcha text, which I doubt as I only found few academic papers how to do this and the required skill set and amount of tweaking for each site using captcha images, would be well beyond most spammers.

Looking at the IP logs it is clear most comment spam comes from overseas where human labour costs are low and it may be worth while to pay someone to manually defeat captcha measures. What I think is needed is some kind of classification system, like Bayesian filtering, similar to what DSPAM, BogoFilter and SpamAssassin are using.

I have most of the code developed based on Paul Graham's (http://www.paulgraham.com/spam.html) 'Plan for Spam' thesis. I think such symptoms will, or already do, affect any public online systems such as Wikis and Forums.

Ultimately, only some automated classification moderation system will do, in addition, with some human supervision to fix false positives.

Log4Net

Daniel — Fri, 10 Dec 2004 20:04:09 GMT

One of the essential .Net tools I've found is log4net logging framework. (http://logging.apache.org/log4net/)

I've use it in everything from console apps, windows forms, web forms and dll libraries. A typical use scenario would for a .Net dll library would go like this.

Let's say we have executable named “uniBlogger.Drivers.exe” which is a console application in C#.

1. Create .log4net log file named “uniBlogger.Drivers.exe.log4net' with contents:

2. Add the following code into the “AssemblyInfo.cs” file in the project:

using log4net;

...

// Load the configuration from the 'uniBlogger.Drivers.exe.log4net' file
[assembly: log4net.Config.DOMConfigurator(ConfigFileExtension="log4net", Watch=true)]

3. Create the “log-data” subdirectory, in the directory where the executable resides.

4. For every class you want to participate in logging include the following member handle:

using log4net;

...

// LOG4NET
private static readonly log4net.ILog LOG = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

5. For every method you want to participate in logging include the following code:

public bool CheckProfileGeneral(){

   string sql = "";

   MySqlParameter[] p =
   {
    BlogIDParam
   };

   if (LOG.IsInfoEnabled)
   {
    LOG.Info( " class: " + this.GetType().FullName +
     " method: " + System.Reflection.MethodBase.GetCurrentMethod().Name +
     " MySqlParameters: " + LogMySqlParams(p));
   }

...

}

6. That's it! Remember you can make generic calls the the log API via LOG.Error(...), LOG.Warn(...), LOG.Info(...) and LOG.Debug(...) function calls. You get to control which log levels get filtered down via the XML lo4net config file and the log level setting.