Tux

...making Linux just a little more fun!

2 cent tip: reading Freelang dictionaries

Jimmy O'Regan [joregan at gmail.com]


Sun, 5 Sep 2010 14:58:30 +0100

Freelang has a lot of (usually small) dictionaries, for Windows. They have quite a few languages that aren't easy to find dictionaries for, so though the coverage and quality are usually quite low, they're sometimes all that's there.

So, an example: http://www.freelang.net/dictionary/albanian.php

Leads to a file, dic_albanian.exe

This runs quite well in Wine (I haven't found any other way of extracting the contents). On my system, the 'C:\users\jim\Local Settings\Application Data\Freelang Dictionary' translates to '~/.wine/drive_c/users/jim/Local\ Settings/Application\ Data/Freelang\ Dictionary/'. The dictionary files are inside the 'language' directory.

Saving this as wb2dict.c:

#include <stdlib.h>
#include <stdio.h>
 
int main (int argc, char** argv)
{
	char src[31];
        char trg[53];
	FILE* f=fopen(argv[1], "r");
	if (f==NULL) {
		fprintf (stderr, "Error reading file: %s\n", argv[1]);
		exit(1);
	}
 
	while (!feof(f)) {
		fread(&src, sizeof(char), 31, f);
		fread(&trg, sizeof(char), 53, f);
		printf ("%s\n   %s\n\n", src, trg);
	}
	
	fclose(f);
	exit(0);
}

The next step depends on the contents... Albanian on Windows uses Codepage 1250, so in this case:

./wb2dict Albanian_English.wb|recode 'windows1250..utf8' |dictfmt -f --utf8 albanian-english dictzip albanian-english.dict (as root:) cp albanian-english.* /usr/share/dictd/

add these lines to /var/lib/dictd/db.list : database albanian-english { data /usr/share/dictd/albanian-english.dict.dz index /usr/share/dictd/albanian-english.index }

/etc/init.d/dictd restart

and now it's available: dict agim 1 definition found

From unknown [albanian-english]:

agim dawn

-- 
<Leftmost> jimregan, that's because deep inside you, you are evil.
<Leftmost> Also not-so-deep inside you.


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Sun, 5 Sep 2010 12:30:10 -0400

On Sun, Sep 05, 2010 at 02:58:30PM +0100, Jimmy O'Regan wrote:

> Freelang has a lot of (usually small) dictionaries, for Windows. They
> have quite a few languages that aren't easy to find dictionaries for,
> so though the coverage and quality are usually quite low, they're
> sometimes all that's there.
> 
> So, an example: http://www.freelang.net/dictionary/albanian.php
> 
> Leads to a file, dic_albanian.exe

Sweet. Thanks, Jimmy - I can use that!

> This runs quite well in Wine (I haven't found any other way of
> extracting the contents). On my system, the 'C:\users\jim\Local
> Settings\Application Data\Freelang Dictionary' translates to
> '~/.wine/drive_c/users/jim/Local\ Settings/Application\ Data/Freelang\
> Dictionary/'. The dictionary files are inside the 'language'
> directory.

Oh, right - reminds me: for stuff like this, I've got a special directory I use so I don't have to hunt through the WINE structure. I created a symlink at ".wine/drive_c/temp/to_unix" that points to my /tmp directory, so if I just install the program to that directory, it shows up in my /tmp, all ready to be played with.

> Saving this as wb2dict.c:

[snip]

Whoops - that double-prints the last entry in the dictionary. :) Not a big deal, though.

> The next step depends on the contents... Albanian on Windows uses
> Codepage 1250, so in this case:
> 
> ./wb2dict Albanian_English.wb|recode 'windows1250..utf8' |dictfmt -f
> --utf8 albanian-english
> dictzip albanian-english.dict

Or, all of the above in one step:

#!/usr/bin/perl -w
# Created by Ben Okopnik on Sun Sep  5 12:11:02 EDT 2010
use strict;
 
die "Usage: ", $0 =~ /([^\/]+)$/, " <dict_file> [encoding]\n"
    unless @ARGV;
 
use open IN => ":encoding(" . (defined $ARGV[1]?$ARGV[1]:'utf8') . ")",
    OUT => ":utf8";
 
(my $dct = $ARGV[0]) =~ s/\.wb$//;
$dct =~ tr/_ A-Z/-_a-z/;
open my $in, $ARGV[0] or die "$ARGV[0]: $!\n";
open my $out, "|/usr/bin/dictfmt -f --utf8 $dct"
    or die "Pipe failure: $!\n";
 
{
    my $ret1 = read $in, my $src, 31;
    my $ret2 = read $in, my $tgt, 53;
    last unless $ret1 & $ret2;
    s/\0.*// for $src, $tgt;
    printf $out "%s\n   %s\n\n", $src, $tgt;
    redo;
}
close $in;
system ('dictzip', "$dct.dict");
 
print <<"+EOT+"
database $dct.dict.dz
{
	data  /usr/share/dictd/$dct.dict.dz
	index /usr/share/dictd/$dct.index
}
+EOT+

Just specify the '.wb' file as the first argument and its encoding as the second.

> (as root:)
> cp albanian-english.* /usr/share/dictd/
> 
> add these lines to /var/lib/dictd/db.list :
> database albanian-english
>  {
>   data  /usr/share/dictd/albanian-english.dict.dz
>   index /usr/share/dictd/albanian-english.index
> }

For convenience, the script actually spits that out so it can be copied and pasted. :)

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *


Top    Back


Jimmy O'Regan [joregan at gmail.com]


Sun, 5 Sep 2010 17:36:45 +0100

On 5 September 2010 17:30, Ben Okopnik <ben at linuxgazette.net> wrote:

> On Sun, Sep 05, 2010 at 02:58:30PM +0100, Jimmy O'Regan wrote:
>> Freelang has a lot of (usually small) dictionaries, for Windows. They
>> have quite a few languages that aren't easy to find dictionaries for,
>> so though the coverage and quality are usually quite low, they're
>> sometimes all that's there.
>>
>> So, an example: http://www.freelang.net/dictionary/albanian.php
>>
>> Leads to a file, dic_albanian.exe
>
> Sweet. Thanks, Jimmy - I can use that!
>
>> This runs quite well in Wine (I haven't found any other way of
>> extracting the contents). On my system, the 'C:\users\jim\Local
>> Settings\Application Data\Freelang Dictionary' translates to
>> '~/.wine/drive_c/users/jim/Local\ Settings/Application\ Data/Freelang\
>> Dictionary/'. The dictionary files are inside the 'language'
>> directory.
>
> Oh, right - reminds me: for stuff like this, I've got a special
> directory I use so I don't have to hunt through the WINE structure. I
> created a symlink at ".wine/drive_c/temp/to_unix" that points to my /tmp
> directory, so if I just install the program to that directory, it shows
> up in my /tmp, all ready to be played with.
>
>> Saving this as wb2dict.c:
>
> [snip]
>
> Whoops - that double-prints the last entry in the dictionary. :) Not a
> big deal, though.
>

Ah well... I spent more time on the dict stuff than looking at the raw files/writing the C :)

It also loses the first entry (I think) because of the way dictfmt adds its initial entries.

>> The next step depends on the contents... Albanian on Windows uses
>> Codepage 1250, so in this case:
>>
>> ./wb2dict Albanian_English.wb|recode 'windows1250..utf8' |dictfmt -f
>> --utf8 albanian-english
>> dictzip albanian-english.dict
>
> Or, all of the above in one step:
>
> ```
> #!/usr/bin/perl -w
> # Created by Ben Okopnik on Sun Sep ?5 12:11:02 EDT 2010
> use strict;
>
> die "Usage: ", $0 =~ /([^\/]+)$/, " <dict_file> [encoding]\n"
> ? ?unless @ARGV;
>
> use open IN => ":encoding(" . (defined $ARGV[1]?$ARGV[1]:'utf8') . ")",
> ? ?OUT => ":utf8";
>
> (my $dct = $ARGV[0]) =~ s/\.wb$//;
> $dct =~ tr/_ A-Z/-_a-z/;
> open my $in, $ARGV[0] or die "$ARGV[0]: $!\n";
> open my $out, "|/usr/bin/dictfmt -f --utf8 $dct"
> ? ?or die "Pipe failure: $!\n";
>
> {
> ? ?my $ret1 = read $in, my $src, 31;
> ? ?my $ret2 = read $in, my $tgt, 53;
> ? ?last unless $ret1 & $ret2;
> ? ?s/\0.*// for $src, $tgt;

Not quite. The reason I used C was because the data showed some evidence of C string reuse: schmal(t)z\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 "devojka za s\0"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 factotum\0\0\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0

... so you'd at least need to split both strings on \0

> ? ?printf $out "%s\n ? %s\n\n", $src, $tgt;
> ? ?redo;
> }
> close $in;
> system ('dictzip', "$dct.dict");
>
> print <<"+EOT+"
> database $dct.dict.dz
> {
> ? ? ? ?data ?/usr/share/dictd/$dct.dict.dz
> ? ? ? ?index /usr/share/dictd/$dct.index
> }
> +EOT+
> '''
>
> Just specify the '.wb' file as the first argument and its encoding as
> the second.
>
>> (as root:)
>> cp albanian-english.* /usr/share/dictd/
>>
>> add these lines to /var/lib/dictd/db.list :
>> database albanian-english
>> ?{
>> ? data ?/usr/share/dictd/albanian-english.dict.dz
>> ? index /usr/share/dictd/albanian-english.index
>> }
>
> For convenience, the script actually spits that out so it can be copied
> and pasted. :)
>
>
> --
> * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
>                                              
> TAG mailing list
> TAG at lists.linuxgazette.net
> http://lists.linuxgazette.net/listinfo.cgi/tag-linuxgazette.net
>
-- 
<Leftmost> jimregan, that's because deep inside you, you are evil.
<Leftmost> Also not-so-deep inside you.


Top    Back


Jimmy O'Regan [joregan at gmail.com]


Sun, 5 Sep 2010 18:12:12 +0100

On 5 September 2010 17:36, Jimmy O'Regan <joregan at gmail.com> wrote:

>>> Saving this as wb2dict.c:
>>
>> [snip]
>>
>> Whoops - that double-prints the last entry in the dictionary. :) Not a
>> big deal, though.
>>
>
> Ah well... I spent more time on the dict stuff than looking at the raw
> files/writing the C :)
>
> It also loses the first entry (I think) because of the way dictfmt
> adds its initial entries.
>

This version fixes both problems:

#include <stdlib.h> #include <stdio.h>

int main (int argc, char** argv) { char src[31]; char trg[53]; int c; FILE* f=fopen(argv[1], "r"); if (f==NULL) { fprintf (stderr, "Error reading file: %s\n", argv[1]); exit(1); }

printf ("00-database-info\n Converted from %s\n\n", argv[1]); printf ("00-dummy-entry\n For dictfmt\n\n");

while ((c = (int) fgetc(f)) != EOF) { ungetc(c, f); fread(&src, sizeof(char), 31, f); fread(&trg, sizeof(char), 53, f); printf ("%s\n %s\n\n", src, trg); } fclose(f); exit(0); }

-- 
<Leftmost> jimregan, that's because deep inside you, you are evil.
<Leftmost> Also not-so-deep inside you.


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Sun, 5 Sep 2010 13:13:59 -0400

On Sun, Sep 05, 2010 at 05:36:45PM +0100, Jimmy O'Regan wrote:

> On 5 September 2010 17:30, Ben Okopnik <ben at linuxgazette.net> wrote:
> 
> > {
> > ? ?my $ret1 = read $in, my $src, 31;
> > ? ?my $ret2 = read $in, my $tgt, 53;
> > ? ?last unless $ret1 & $ret2;
> > ? ?s/\0.*// for $src, $tgt;
> 
> Not quite. The reason I used C was because the data showed some
> evidence of C string reuse:
> schmal(t)z\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
> "devojka za s\0"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
> factotum\0\0\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
> 
> ... so you'd at least need to split both strings on \0

Actually, except for the double-printed entry, it produces precisely the same output as your program - so that seems to work just fine.

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Sun, 5 Sep 2010 13:32:23 -0400

On Sun, Sep 05, 2010 at 05:36:45PM +0100, Jimmy O'Regan wrote:

> 
> Not quite. The reason I used C was because the data showed some
> evidence of C string reuse:
> schmal(t)z\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
> "devojka za s\0"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
> factotum\0\0\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
> 
> ... so you'd at least need to split both strings on \0

Just recalled: C strings are null-terminated, right? That means the assignment to the string will terminate at that first null, regardless of the content after it. I'm just doing that manually.

#include <stdlib.h>
#include <stdio.h>
 
int main()
{
    char *str = "abc\0def";
    printf("%s\n", str);
    exit(0);
}

This will only print the first three characters of the string.

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *


Top    Back


Jimmy O'Regan [joregan at gmail.com]


Sun, 5 Sep 2010 18:39:40 +0100

On 5 September 2010 18:13, Ben Okopnik <ben at linuxgazette.net> wrote:

> On Sun, Sep 05, 2010 at 05:36:45PM +0100, Jimmy O'Regan wrote:
>> On 5 September 2010 17:30, Ben Okopnik <ben at linuxgazette.net> wrote:
>>
>> > {
>> > ? ?my $ret1 = read $in, my $src, 31;
>> > ? ?my $ret2 = read $in, my $tgt, 53;
>> > ? ?last unless $ret1 & $ret2;
>> > ? ?s/\0.*// for $src, $tgt;
>>
>> Not quite. The reason I used C was because the data showed some
>> evidence of C string reuse:
>> schmal(t)z\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
>> "devojka za s\0"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
>> factotum\0\0\0ch\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
>>
>> ... so you'd at least need to split both strings on \0
>
> Actually, except for the double-printed entry, it produces precisely the
> same output as your program - so that seems to work just fine.
>

Sorry, misread "s/\0.*//". I need 1) new glasses, and 2) to clean my monitor :)

-- 
<Leftmost> jimregan, that's because deep inside you, you are evil.
<Leftmost> Also not-so-deep inside you.


Top    Back


Jimmy O'Regan [joregan at gmail.com]


Sun, 5 Sep 2010 18:55:24 +0100

On 5 September 2010 17:30, Ben Okopnik <ben at linuxgazette.net> wrote:

> ```
> #!/usr/bin/perl -w
> # Created by Ben Okopnik on Sun Sep ?5 12:11:02 EDT 2010
> use strict;
>
> die "Usage: ", $0 =~ /([^\/]+)$/, " <dict_file> [encoding]\n"
> ? ?unless @ARGV;
>
> use open IN => ":encoding(" . (defined $ARGV[1]?$ARGV[1]:'utf8') . ")",
> ? ?OUT => ":utf8";
>
> (my $dct = $ARGV[0]) =~ s/\.wb$//;
> $dct =~ tr/_ A-Z/-_a-z/;
> open my $in, $ARGV[0] or die "$ARGV[0]: $!\n";
> open my $out, "|/usr/bin/dictfmt -f --utf8 $dct"
> ? ?or die "Pipe failure: $!\n";
>

print $out "00-dummy-entry\n For dictfmt\n\n";

here will get rid of the second bug I had

-- 
<Leftmost> jimregan, that's because deep inside you, you are evil.
<Leftmost> Also not-so-deep inside you.


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Sun, 5 Sep 2010 14:28:53 -0400

On Sun, Sep 05, 2010 at 06:55:24PM +0100, Jimmy O'Regan wrote:

> 
> print $out "00-dummy-entry\n   For dictfmt\n\n";
> 
> here will get rid of the second bug I had

OK, so the "improved" version looks like this (I was trying to remember what in Perl handles C strings... 'pack/unpack', of course):

#!/usr/bin/perl -w
# Created by Ben Okopnik on Sun Sep  5 12:11:02 EDT 2010
use strict;
 
die "Usage: ", $0 =~ /([^\/]+)$/, " <dict_file> [encoding]\n"
    unless @ARGV;
 
use open IN => ":encoding(" . (defined $ARGV[1]?$ARGV[1]:'utf8') . ")",
    OUT => ":utf8";
 
(my $dct = $ARGV[0]) =~ s/\.wb$//;
$dct =~ tr/_ A-Z/-_a-z/;
open my $in, $ARGV[0] or die "$ARGV[0]: $!\n";
open my $out, "|/usr/bin/dictfmt -f --utf8 $dct" or die "Pipe failure: $!\n";
 
my $src;
print $out "00-dummy-entry\n\tFor dictfmt\n\n";
printf "%s\n\t%s\n\n", unpack("Z31 Z53", $src) while read $in, $src, 84;
close $in;
 
system ('dictzip', "$dct.dict");
print <<"+EOT+"
 
database $dct.dict.dz
{
    data  /usr/share/dictd/$dct.dict.dz
    index /usr/share/dictd/$dct.index
}
+EOT+

The amusing part is the amount of work done by that "printf" line. Real workhorse, that thing. :)

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Sun, 5 Sep 2010 14:47:04 -0400

On Sun, Sep 05, 2010 at 02:28:53PM -0400, Benjamin Okopnik wrote:

Whoops, one mistake there:

> printf "%s\n\t%s\n\n", unpack("Z31 Z53", $src) while read $in, $src, 84;

Should be

printf $out "%s\n\t%s\n\n", unpack("Z31 Z53", $src) while read $in, $src, 84;
-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *


Top    Back