The time for stats ended a little over week ago week. I added little new features, like per hour granularity and counting of active translators instead of edits and there is now a simple GUI to generate a code that can be included in pages for those who don’t care to remember the parameters.

My current task is wikipage translation. I have been waiting for this task and I’m very excuisited to see what will come from it. The basic system should already work and it is being tested on an example page. That means it is possible to mark content for translation, translate it, and changes to the content will invalidate the translation. But as you can see it is still missing a lot, like for example selection for language.

This is a status report again.

Possibility to translate aliases for special pages didn’t cause any mass movement. Let’s wait for a another week to see what happens. In the mean time I’ve improved the check framework. There is now own checks for FreeCol, existing checks were improved and it is easy to add checks for new types. It is also possible for translators to get list of messages tagged by the checks. The down side is that running checks is slow, so that the list doesn’t update in real time. Anyway it is better than before.

Gettext got improvements and fixes, thanks to a new testing project. Mobile Broadband Configuration Assistant is project of another Summercode Finland participant and friend of mine that is enabled for translation for testing. The same guy made me to fix one bad usability issue that was long due. It was a quick hack, but a start for better.

Xliff support is still stalled, but I managed to contact the nice pootle developer I met at FOSDEM 2008. We talked shortly in which ways we can co-operate (less in terms of code) and a little which features should be implemented. Didn’t get much out of it, as Xliff is quite a new standard, and support in other tools is also in emerging state. This confirms my thoughts that it is not worth to shoot in the dark, but do features for it when there is a need or support for those.

Had to spend some time maintaining Betawiki, so the progress has been a little slow for the past week. Aside from that I’ve been working on many things.

I have setup a test project for Gettext: a Wesnoth campaign. It is now shown to all, just to us few testers who are going to translate it using Betawiki. It already helped to find some bugs, simplify the code and the edit view got support from displaying information extracted from the pot file.

To make a project available for translation it is not enough to only add it to the list. That part is easy to do, just checking out the files and about twenty lines of code. But to really support a project, we need to work closely with the development team and with the existing translation communities around it. It would be easy if we could just get everyone to use Betawiki immediately, but often some people don’t want to use the web interface for one reason or another. We need to setup rules which languages are translated and where, to avoid conflicts, map our language codes to what the project uses and setup some kind of integration process that translation actually get to the upstream, and that upstream changes propagate in a proper way back to us.

But back to the project. I have been reading Xliff format specifications. It’s fortunately quite short and clear and has nice examples. Xliff supports all kinds of nice features, and I have been trying to decide which features we need to support. I wrote a simple implementation that can export translations in minimalistic Xliff file. It was actuall a pretty easy to do, under 100 lines of code. It would be a really good to get someone who uses programs that accept Xliff files to comment which features would be useful to implement. In any case, I will implement a parser too this week, so that we can get those translations back too :).

If the test project doesn’t bring any big surprises, I start preparing to tackle the next task in the project schedule.

All kinds of weird things show when looking different po files. While implementing support for the standard way of doing plurals, I found that for example Wesnoth has comments inside the definitions itself, which is quite unexpected, when the po format itself has designated syntax for many kinds of comments! Just look at the snippet taken from Gettext manual:

     #  translator-comments
     #. extracted-comments
     #: reference...
     #, flag...
     #| msgid previous-untranslated-string-singular
     #| msgid_plural previous-untranslated-string-plural

I’m not sure what to do to those… It doesn’t hurt to leave them as is… but it is counter-intuitive to translators… but that would make the code more complex… decisions.

Anyway, the plural support seems to kinda work. At least it parses nicely, but I still need to make sure the special handling doesn’t break comparing messages (for import) and gets exporter properly. Also, KDE3 uses some kind of own format for plurals, but doubt if it is worth making it a special case too.

Aside from improving Gettext, I had to fix somewhat broken author export. It didn’t all authors always, and while I was fixing it, I implemented a blacklist to filter our bots so that they do not appear in every author list.

I also changed the code to support having messages in different namespaces. Now that we support multiple formats, and the number of external projects is probably growing, it was needed. The message cache of MediaWiki was not designed to handle hundreds of thousands of messages. Having one namespace per project also helps filtering and reducing key conflicts. Unfortunately it broke our recentchanges hacks, which means I had to rewrite those, now in a more proper way.

I have managed to translate one string with the extension and export a file including existing translations, and have it work. This means that I’ve implemented a parser and exporter for po files.

Most of the time went into refactoring the old code to support other file formats easily. Exporting is as easy as possible:

$lang = array( 'fi', 'de' );
$target = '/tmp';
$group = MessageGroups::getGroup( 'out-freecol' );
$writer = $group->getWriter();
$writer->fileExport( $langs, $target );

The above code exports FreeCol translations to the tmp directory. Those file can then be committed to the vcs. Each project (group) has a preferred output format. It is also possible to export to other supported file formats by creating a writer manually.

As gettext messages don’t have ids, I had to create those. Currently it is just hash + snippet, which produces page titles like MediaWiki:33e5da6ddaa6edf2f1cdf8c235813747e40fc326-Disable Ethernet Wake-On-Lan w/fi. It is not pretty, but good enough for now.

Anyway, the support is “coming”, not ready. Still to do are for example gettext plural support, better formatting of authors, importing external changes and so on.

Gettext starts from quite a different perspective to i18n. Especially it differs in who should do the ugly work. The thing is, Gettext tries to hide the i18n from developers, while the system we build in Betawiki aims for minimising the work translators have to do. These two aims produce systems that are different and it needs some thought how to combine them together. Fortunately these two aims aren’t entirely incompatible. I have to say that hiding i18n from the developers has its good and bad sides, but I’m not to judge whether it has more good or more bad.

Paradigms aside the main difference seems to boil down to tracking changes to messages. Betawiki does it, and it is easy because every message is identified by a unique name. Gettext doesn’t really, it just prefills the translations for new and changed messages by guesswork.

We use MediaWiki pages, which have a concept of unique name. So obviously I need to generate some kind of unique names for the messages in Gettext files. Maybe hash of the contents and context, which is the Gettext definition of uniqueness. Not pretty, but as the developers aren’t forced to name the messages, there is probably no way to get meaningful names.

That should be a start at least. I have been fiddling with the code trying to separate file format support to own classes from other code, but I’m not yet happy with it.

I also hope I figure out some clever trick to track messages changes from .po files to keep more history in the wiki. It may be that i could use Gettext guesswork algorithm to some extend, but it may also be that it is not worth it, due the nature of Gettext. In any case we have the history for changes in the translations.