Full text search - status?

[Nazar]

See http://goldendict.org/forum/viewtopic.php?f=4&t=1081 as well..

[ikm]

No one's working on it right now. If someone wants to, that someone's welcome.

[Tvangeste]

No one's working on it right now. If someone wants to, that someone's welcome.

I've been looking into full text search libraries for C++ that we could use, just in case there is something that already works.

So far, it seems that Xapian is the most interesting library (GPL licensed C++): http://xapian.org/features

Looks promising. Are you aware of any other projects that could be of use?

[ikm]

Xapian indeed looks the most promising. As for the FTS implementation in GoldenDict, well, here's my proposal: I am willing to implement the dictionary crawling required to perform the FTS indexing, if someone (wink, wink!) is willing to implement the GUI for indexing and searching (using Xapian as a backend).

[Tvangeste]

Well, here's my proposal: I am willing to implement the dictionary crawling required to perform the FTS indexing, if someone (wink, wink!) is willing to implement the GUI for indexing and searching (using Xapian as a backend).

Whoa! Abgemacht! I mean, yeah, I"ll do my best. Hopefully, my abilities with Qt will be advanced enough by then!

[ikm]

Ok then. I'll post back once I'm done.

[betwee]

yay, .

[ikm]

Ok, the crawling interface is ready. Dictionary::Class objects now have the following two functions: isCrawlingSupported() and crawl(). If the former returns true, then the latter can be used to create a Dictionary::Crawler object. From that moment on, it is completely independent from the originating dictionary object, and can be used to traverse through all the articles. For each article, a list of headwords (first one is the main one, the others are alternates, if any) and a body in html is returned. A simple crawling example:

Code: Select all

    Dictionary::Class & d = .....  // Obtain a dictionary instance from somewhere

    if ( d.isCrawlingSupported() )
    {
      printf( "Gonna crawl it!\n" );

      File::Class f( "/tmp/crawled.txt", "wb" );

      sptr< Dictionary::Crawler > crw = d.crawl();

      vector< string > headwords;
      string body;
      while( crw->fetchNextArticle( headwords, body ) )
      {
        for ( int x = 0; x < headwords.size(); ++x )
        {
          string const & str = headwords[ x ];
          f.write( str.c_str(), str.size() );
          f.write( "\n", 1 );
        }
        f.write( body.c_str(), body.size() );
        f.write( "\n", 1 );
      }

      printf( "Done!\n" );
      }
    }


This interface right now is implemented for Dsl dictionaries. Support for others will follow.

All the code lives in the 'fts' branch at github. This should be sufficient to implement dictionary indexing and searching with Xapian.

[Nazar]

Well, it seems Konstantin has done his part as prompt as one could only wish. But what comes next? We want our full-text search!!! lol.

[Tvangeste]

Well, it seems Konstantin has done his part as prompt as one could only wish. But what comes next? We want our full-text search!!! lol.

Heh, Konstantin was WAAAY to fast for me. But no worry, once I'm done with the UI tweaks, this is the biggest priority.