Linux Gazette

Mailbag

LG Staff — Wed, 01 Jul 2009 22:07:56 -0500

By LG Staff

Still Searching
icewm question. maximized windows stick to top of screen can this be changed?
Our Mailbag
file system related question
linux command to read .odt ?
Laptop installation problem
[Mailman-Users] Virtual domain not quite working on HTTP (but fine on SMTP)
Quoting question
Fixed. Was: Virtual domain not quite working on HTTP (but fine on SMTP)
Jim, HELP needed, 5-minute solution needed by computer industry

Talkback

LG Staff — Wed, 01 Jul 2009 22:07:56 -0500

By LG Staff

Talkback:141/brownss.html
157/lg_mail.html
Talkback:133/cherian.html
Talkback:141/lazar.html

2-Cent Tips

LG Staff — Wed, 01 Jul 2009 22:07:56 -0500

By LG Staff

2-cent Tip: Wrapping a script in a timeout shell
2-cent Tip - Poor Man's Computer Books
2-cent Tip: Checking the amount of swapped out memory owned by a process
2-cent Tip: ext2 fragmentation

News Bytes

Deividson Luiz Okopnik and Howard Dyckoff — Wed, 01 Jul 2009 22:07:56 -0500

By Deividson Luiz Okopnik and Howard Dyckoff

Contents:
News in General
Power IT Down Day - August 27, 2009
Ubuntu Remix to power Intel Classmate Netbooks
Citrix Contributes Code for Virtual Network Switch
New Community Website for Growing XenServer Customer Base
Dell and Goodwill Expand Free Computer Recycling Program
Conferences and Events
Distro News
New Features listed for Forthcoming Ubuntu 9.10
A naming contest for Ubuntu 10.04 and the South African Linux community
Fedora 11: Many Virtualization Enhancements
Software and Product News
Opera 10 beta: new features, new speed
Oracle 11g Sets New Price/Performance Record with TPC-C Benchmark

Layer 8 Linux Security: OPSEC for Linux Common Users, Developers and Systems Administrators

Lisa Kachold — Wed, 01 Jul 2009 22:07:56 -0500

By Lisa Kachold

As users of Linux each of us is in a unique position with a powerful tool. Use of any tool without regard for security is dangerous. Developers likewise carry a great responsibility to the community to maintain systems in a secure way. Systems Administrators are often placed in the uncomfortable role of holding a bastion between insecurity or pwnership and uptime.

Building the GNOME Desktop from Source

Oscar Laycock — Wed, 01 Jul 2009 22:07:56 -0500

By Oscar Laycock

First, I must say that the GNOME documentation "strongly recommends" against building from source files. If you do, it suggests using the GARNOME scripted system. Even GARNOME it says, is only for the "brave and patient".

I started building programs from source as I wanted to upgrade a very old distribution and did not have enough disk space to install a Fedora or Ubuntu. The memory on my old PC is probably a bit low too, at 128 Meg Also, it is fun to run the latest versions, be on the leading edge, and get ahead of the major distributions. You could not run Windows Vista (tm) on my old PC!

I took the files from the latest GNOME release I could find, being version 2.25. This was probably a mistake. I later found out that odd numbered minor releases like 25 are development releases and may be unstable. The stable 2.26 series should have come out in March. In fact, when I built the file manager "Nautilus", I could not find a dependency, "libunique", so I went back to version 2.24. The GNOME sources are at http://ftp.gnome.org/pub/gnome.

I used the Beyond Linux from Scratch (BLFS) guide to know where to find packages and in what order to build them. You can find it at http://www.linuxfromscratch.org (LFS). Unfortunately the BLFS guide uses versions from the summer of 2007, GNOME version 2.18. Also, I installed the libraries and programs in the default location /usr/local, but looking back it would have been better to have put them in their own directory and give a --prefix option to "configure", as the BLFS project recommends.

It took me about a month to build a basic GNOME desktop in my spare time. Disappointingly, you don't see anything until the last piece, "gnome-session", is in place.

When I ran my first applications like the image-viewer "eye-of-gnome", some appeared on the desktop and then core-dumped. The core file said the last function called was in GTK, in the libgtk-x11 library. I had had to upgrade my GTK when building gnome-panel. I had chosen the latest version 2.15.2. This is probably a development releases too. When I switched back to my year-old GTK, everything was fine. I finally settled for the stable version recommended by GARNOME. It took me a few days to sort this out!

There is a free book on programming GNOME, "GTK+ / Gnome Application Development" by Havoc Pennington available at http://developer.gnome.org/doc/GGAD. This is somewhat old.

Overall, I like GNOME because it is truly free and lighter than KDE.

GNOME versus KDE

Last year (2008), I built the 4.0 release of KDE from the source files. GNOME is made up of around sixty packages, KDE about twenty. Qt (the user interface framework) and KDE are much larger than GNOME. The compressed source of Qt is about three times the size of the equivalent GTK+, GLib, Cairo and Pango put together. But, of course, KDE includes the web browser Konqueror. GNOME needs the Gecko layout engine from Mozilla.org that you see in Firefox.

Qt and KDE's C++ code is slower to compile than GNOME's C, at least on my old laptop - a Pentium III with only 128 Meg RAM! KDE uses CMake rather than "configure" to generate the makefiles. It took a while to learn how to use CMake, but its files are much clearer.

The BLFS instructions only covered KDE 3.5, so I had to adjust things for KDE 4.0. In GNOME, if you wanted to find out where to find some unusual packages, you could look at the makefiles of the GARNOME project. Its website is at http://projects.gnome.org/garnome. GARNOME automatically fetches and builds a given GNOME release.

For KDE, I had to change a little code when building the "strigi" library. (It searches through files for the semantic desktop). To be exact, I had to add definitions for "__ctype_b" and "wstring". For GNOME, it was easier. I mostly had to go off and find required libraries that were not in the BLFS notes.

KDE has trouble starting up on my laptop, although it is all right on my PC. There is a lot of swapping in and out and thrashing with my low memory. In fact, both computers have 128 Megabytes. The laptop even has a faster processor. It has a Pentium III processor whereas than the PC has a Pentium II. The disks even seem about the same speed in simple tests. It is a puzzle. Anyway, KDE takes over ten minutes to start on my laptop! GNOME takes one minute. (KDE takes just over a minute on my PC.)

Browsers

On KDE, I could get the Konqueror browser running fairly easily. But I had to add about 250 Meg of swap space to get the WebKit library to link with the low memory on my PC. WebKit is the rendering engine. I like the way Konqueror uses many of the same libraries as other parts of KDE. It makes Konqueror a bit lighter when running inside of KDE. (I have put the commands I use to add swap at the end).

With Firefox 3, I had to turn the compiler optimization off with the "-O0" switch to avoid thrashing with the small memory on my old laptop. I also had to leave out a lot of the components in "libxul". Otherwise the link of libxul ends up just swapping in and out as well. (I understand XUL is the toolkit used to build the user interface of Mozilla). Firefox does not work without a real libxul though.

I thought I would try building GNOME's own browser - Epiphany. But I had trouble configuring it to use the Gecko rendering libraries in the Mozilla Firefox 3 browser I had just built. Epiphany offers a simpler interface than Firefox and concentrates just on web browsing, taking features like mail from other GNOME applications. Unfortunately Epiphany seems to expect "pkg-config" entries for Gecko, which Firefox 3 had not generated. (When you install a library, a file in /usr/lib/pkgconfig is created, listing the compiler flags (-L, -l and -I) needed by a program using the library.) I manually added in all the Firefox shared libraries I could find. But there seemed to be some core Firefox functions that Epiphany still could not access. However, Epiphany does really recommend Firefox 2. But Firefox 2 uses a lot of deprecated functions in my more modern version of Cairo. I got lost in all the errors.

Then I noticed that Epiphany's web page mentions an experimental version using WebKit as the browser engine. But the configuration seemed to have incomplete autoconf/automake files and use the Apple xcodebuild tool. It was too difficult for me. You can begin to see there is some skill required to build a complete distribution!

I later learnt from a GNOME blog that the Mozilla version of Epiphany is not actively developed any more, because about a year ago GNOME started to switch to WebKit. But the switch has not been completed. It was originally scheduled to be delivered with GNOME 2.26, but has been postponed to GNOME 2.28. I remembered that WebKit is a fork of KDE's KHTML browser engine used by Konqueror. It was made by Apple for their Safari browser, and more recently used by Google in its Chrome browser.

In the end, I built the Dillo browser. It used to use GTK and Glib. But it now uses the FLTK (Fast Light Tool Kit) - a C++ GUI toolkit. Of course, it does not support Javascript or Java.

Strangely enough, Konqueror runs quite well within GNOME! I think I will use it when developing complicated web pages which use Javascript and AJAX.

Problems compiling packages

Would you enjoy building programs from the source tarballs? I think you probably need some knowledge of C. Not too much, as I didn't know a lot about C++ when I built KDE. There always seem to be a few problem with compiling packages. Here are a few to give you an idea.

A simple problem occurred when I was building the "dbus-glib-bindings" package:

DBUS_TOP_BUILDDIR=.. dbus-send --system --print-reply=literal --dest=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.Introspectable.Introspect > dbus-bus-introspect.xml.tmp && mv dbus-bus-introspect.xml.tmp dbus-bus-introspect.xml Failed to open connection to system message bus: Failed to connect to socket /usr/local/var/run/dbus/system_bus_socket: Connection refused

I guessed it wanted the dbus system bus running. I remembered something from building D-Bus last year, had a look around, and ran "dbus-daemon --system".

A typical problem was with gnome-terminal. It included the "dbus-glib-bindings.h" header file which itself included the glib header "glib-2.0/glib/gtypes.h". But glib did not want its header file directly included and had put a "#error" statement in to give a warning. I solved this by crudely commenting out the #error.

The strangest problem I got was when running GStreamer's configure script:

    ./configure: line 32275: syntax error near unexpected token `('     ./configure: line 32275: `  for ac_var in `(set) 2>&1 | sed -n 's/^\([a-zA-Z_][a-zA-Z0-9_]*\)=.*/\1/p'`; do'

I tried narrowing it down by putting in some simple echo statements and then I changed some back ticks to the more modern "$()". But, I had the same problem with the GStreamer Plugins package. So I decided to upgrade my version of Bash, which was a year old. The LFS project suggests using the "malloc" (memory allocation) from the libc library rather than that bundled with Bash, so I also ran configure with the option "--without-bash-malloc". All this luckily fixed the problem.

Out of interest, I have added the commands I typed (over sixty times) to build the packages, at the end of this article.

GNOME libraries

I enjoy reading about the libraries as I build them. I also wanted to draw a diagram showing which packages needed to be built first.

The libraries are, in fact, described very well in the GNOME documentation at http://library.gnome.org/devel/platform-overview.

The libraries shown below are used in various combinations by the GNOME applications you see on the desktop.

My diagrams show the dependencies between packages. For example GTK requires Pango and Cairo. A package usually generates many libraries, so it is nice to think at the package level. I believe you can generate graphs of dependencies automatically from the GNOME build systems.

I have included some notes that I found in the README file or documentation that came with the packages.

I did not build most of the audio or video libraries, as I don't play music or watch videos on my laptop! It also keeps things simple.

Base libraries

       /|\                 /|\         +-------------------+         |       pygtk       |         |      pycairo      |         |     pygobject     |                        /|\            /| |\         | (python bindings) |          +---- gail ----+  +-----------+ +------+         +-------------------+          |    (GNOME    |  | dbus-glib | |libxml| /|\                                    |accessibility)|  | bindings  | |      |  +----------------------- gtk+ -----------------------+  +-----------+ +------+  |     (GIMP Tool Kit - graphical user interfaces)    |  |   D-Bus   |  |-- pango --+--------+                +---- atk -----+  | (message  |  |(rendering)| libpng |      +---------+(accessibility|  |  passing  |  | of text)  | libjpeg|      | libtiff |  interfaces) |  |  system)  |  +----------------------- cairo ----------------------+  +-----------+  |  (2D graphics library for multiple output devices) |  +------------------+------------ pixman -------------+                     |         (low level pixel        |                     | manipulation, e.g. compositing) |                     +----------------------------------                                                      /|\                               +-------- gvfs ---------+                                |   (backends include   | /|\                           | sftp, smb, http, dav) |   +----------------------- glib -----------------------+   |(data structures, portability wrappers, event loop, |  |threads, dynamic loading, algorithms, object system,|  |GIO - the virtual file system replacing gnome-vfs)  |  +----------------------------------------------------+

Notes

The GIO library in GLib tries not to clone the POSIX API like the older gnome-vfs package, but instead provides higher-level, document-centric interfaces. The GVFS backends for GIO are run in separate processes, which minimizes dependencies and makes the system more robust. The GVFS package contains the GVFS daemon, which spawns further mount daemons for each individual connection.

The core Pango layout engine can be used with different font and drawing backends. GNOME uses FreeType, fontconfig, and Cairo. But on Microsoft Windows, Pango can use the native Uniscribe fonts and render with the Win32 API.

The Cairo documentation says it is "designed to produce consistent output on all output media while taking advantage of display hardware acceleration when available (for example, through the X Render Extension) ... The cairo API provides operations similar to the drawing operators of PostScript and PDF. Operations in cairo include stroking and filling cubic Bezier splines, transforming and compositing translucent images, and antialiased text rendering. All drawing operations can be transformed by any affine transformation (scale, rotation, shear, etc."

GTK+ was first developed as a widget set for the GIMP (GNU Image Manipulation Program). The FAQ mentions one of the original authors, Peter Mattis, saying "I originally wrote GTK which included the three libraries, libglib, libgdk and libgtk. It featured a flat widget hierarchy. That is, you couldn't derive a new widget from an existing one. And it contained a more standard callback mechanism instead of the signal mechanism now present in GTK+. The + was added to distinguish between the original version of GTK and the new version. You can think of it as being an enhancement to the original GTK that adds object oriented features." The GTK+ package contains GDK, the GTK+ Drawing Kit, an easy to use wrapper around the standard Xlib function calls for graphics and input devices. GDK provides the same API on MS Windows.

The GAIL documentation says "GAIL provides an implementation of the ATK interfaces for GTK+ and GNOME libraries, allowing accessibility tools to interact with applications written using these libraries ... ATK provides the set of accessibility interfaces that are implemented by other toolkits and applications. Using the ATK interfaces, accessibility tools have full access to view and control running applications." GAIL stands for GNOME Accessibility Implementation Layer. It has just been moved into GTK.

D-BUS is a simple inter process communication (IPC) library based on messages. The documentation says "a core concept of the D-BUS implementation is that 'libdbus' is intended to be a low-level API, similar to Xlib. Most programmers are intended to use the bindings to GLib, Qt, Python, Mono, Java, or whatever. These bindings have varying levels of completeness."

PyGObject allows you to use GLib, GObject and GIO from Python programs.

Platform libraries

                                                      gnome-icon-theme                                                             /|\                                                       icon-naming-utils                                                (xml to translate old icon names)                   libgnomeui*      (a few extra gui widgets, many now in gtk)                       /|\                        +---------------------------+                        |                           |                   libbonoboui*                gnome-keyring            (user interface controls     (password and secrets daemon              for component framework)     and library interface)                       /|\                          |                        |                        libtasn1                        |               (Abstract Syntax Notation One)                        |                        +--------------------------------------+                        |                                      |                     libgnome*                          libgnomecanvas*     (initialising applications, starting programs,    (structured graphics:      accessing configuration parameters, activating  polygons, beziers, text      files and URI's, displaying help)               pixbufs, etc. Rendered                       /|\                             by Xlib or libart)   +--------------------+----------+                           |   |                               |                          /|\   |  gnome-vfs monikers           |                           |   |   /|\       /|\               |                           |   |    |         +-------- gnome-vfs* (virtual file system)   +----------+   |    |                 (consistent API for local and        |          |   |    |               remote files, cameras, WebDAV, etc)    |          |   |    |                         /|\                          |          |   |    |             +------------+--------+               libglade    libart*   |    |             |                     |            (dynamically   (fast 2d   |    |          Gconf             gnome-mime-data      load glade    graphic   |    |      (configuration           (classify         user        primitives)  libbonobo*     database -            file contents)     interface   (component      daemon &                                description  system, e.g.    library)                                files)  multimedia         /|\  in word processor)  |   /|\                 |   +------------------+    |  ORBit2+ * (fast and light CORBA object broker)    +--- libIDL (parses interface definitions for CORBA)            Note: * indicates library is depreciated

Notes

Many of the libraries on the left hand side of the diagram are not used in newly written code and references to them have been removed in some old code. But the API's are still supported, at least for the current GNOME 2.x series. GNOME is moving to using D-Bus for communication between processes. GNOME Panel applets use Bonobo to communicate with the panel. GnomeVFS has been replaced by GIO, and the GVFS library which supplies the network and local protocols.

According to the documentation, " Libglade is an alternative to using Glade's code generation. Instead of generating code from the XML interface description, libglade loads and parses the description at runtime. It also provides functions that can be used to connect signal handlers to parts of the interface."

Libart is, in its own words, a "sophisticated, antialiased, alpha-compositing engine."

The "gnome-keyring" README says it can "manage several keyrings, each with its own master password, and there is also a session keyring which is never stored to disk, but forgotten when the session ends."

According to the Wikipedia, Abstract Syntax Notation One (ASN.1), is used "in telecoms and computer networking... [and] is a standard and flexible notation that describes data structures for representing, encoding, transmitting and decoding data".

You can read the reference documentation for libbonobo at http://library.gnome.org/devel/libbonobo/stable. Maybe it is worth a skim through to get an overview. There are similar API references for GConf, gnome-vfs, libgnome, libgnomecanvas, libbonoboui and libgnomeui.

Desktop libraries

gnome-themes   gnome-user-docs   gnome-backgrounds   gnome-terminal      |          (user, admin                        (just the shell:        /|\            guides)                           menubar, prefs dialog)      |               |                                     /|\ gtk-engines    gnome-doc-utils                              |    (themes)     (xslt for Docbook                            vte                 and translations)                    (terminal emulator                                                  widget implemented in GTK)                                     gnome-session                                          |                                     control-center                                         /|\  +----+----------------+-----------------+------------+------------+   |    |                |                 |            |            |  | gnome-applets  metacity  nautilus (file manager)   |        gstreamer  |   (for panel)  (window         |      |       |    |         plugins   |    |     |     manager)       /|\  libexif librsvg |            |  |   /|\ libgtop       |          |  (digital  (fast  |     +------+------+       |    |  (portable     zenity     |   cameras)  SVG)  |     |             |   |    | performance   (dialogs    |              |    |    gstreamer   liboil   |    |  statistics    in         |      +-------+   /|\  (sound &    (optimized  |    |  library       scripts)   | libcroco  libgsf  |    video)    inner loops /|\   |  and server)              | (CSS2     (structured                  using  |    |                           | parser)    files) |                 assembly  |    |                           |                   +--+-------+     MMX, SSE)  |    |                           |                      |       |  | gnome-panel        eel (nautilus widgets, extensions) |  libgnomekdb  |    |     |           /|\                        |     |      /|\  |    |     +-------+----+                      gnome-menus      |  |    |             |                           (engine)    libxklavier  | libwnck          |                                   (accessing & setting  | (library for window management e.g. workspace          keyboard layouts)       |                  |       pagers and tasklists)               /|\  +--------+---------+                                            |            |                                                   isocodes      gnome-desktop                                     (list of country, language (functions for background, monitor, thumbnails,           and currency names)  desktop items, and gnome-about. Icon files)

Notes

Remember the diagram shows the dependencies. For example, some of the features in the GNOME control panel are implemented in Metacity, such as changing the window border theme, or window navigation shortcuts.

Libgnome-desktop contains APIs that really belong in libgnome(ui) or GTK+ but are limited in their use. It provides support for desktop backgrounds, monitor configuration, thumbnails, and desktop items.

Libwnck is the Window Navigator Construction Kit. It uses the Extended Window Manager Hints specification (EWMH).

The documentation for "eel" says "The Eazel Extensions Library is a collection of widgets and extensions to many modules of the GNOME platform. These widgets and extensions were developed by hackers working on Nautilus."

The "gnome-menus" package contains an implementation of the draft "Desktop Menu Specification" from freedesktop.org.

The README for "libexif" says it "contains a library for parsing, editing, and saving EXIF data. Most digital cameras produce EXIF files, which are JPEG files with extra tags that contain information about the image ... It is intended to replace lots of redundant implementations in command-line utilities and programs with GUIs."

Librsvg is a high performance SVG rendering. It can draw a SVG to a Cairo surface or render SVGs into GDK pixbufs for easy use in GTK+ applications. Librsvg is not used by the very latest Nautilus code, but is part of other GNOME applications.

The README file for "libcroco" says it is a "standalone css2 parsing and manipulation library. The parser provides a low level event driven SAC (Simple Api for CSS) like api and a css object model like api. Libcroco provides a CSS2 selection engine and an experimental xml/css rendering engine."

The documentation for "libgsf" says the "GNOME Structured File Library (GSF) is an I/O abstraction for reading/writing compound files... GSF was written as part of the Gnumeric project. It started in 2002 as a replacement for libole2 which Gnumeric was using to handle the Microsoft Excel". It reads gzip, bz2, zip, XML and OASIS Open Document formats amongst others.

According to its home page, "Liboil" “is a library of simple functions that are optimized for various CPUs. These functions are generally loops implementing simple algorithms, such as converting an array of N integers to floating-point numbers or multiplying and summing an array of N numbers ... The goal of this project is to consolidate some of the code used by various multimedia projects, and also make optimizations easier to use by a broader range of applications."

The "gnome-applets" package includes the libpanel-applet library which allows you to develop small applications which can be embedded in the panel.

On systems like BSD and Solaris, "libgtop" provides a setgid/setuid server for graphical tools. On Linux, libgtop has a library that gives performance information in a generic way to the differents tools that use it such as "gtop" and panel applets.

In the Nautilus architecture document it says "Nautilus is a general-purpose shell application for browsing arbitrary content. It is implemented as `nautilus', a container application which exercises overall control and provides chrome, and a number of nautilus view components. These view components may use the `libnautilus' library to ease implementation."

Gstreamer, of course, is a framework for streaming media. According to the documentation, " It's based on plug-ins that will provide the various codec and other functionality ... Is GStreamer a media player ? No, GStreamer is a development framework for creating applications like media players, video editors, streaming media broadcasters and so on. That said, very good media players can easily be built on top of GStreamer".

VTE can also be used to embed a console/terminal in games, editors, IDEs, etc. VTE supports Unicode and character set conversion, as well as emulating any terminal known to the system's terminfo database.

The author of Metacity says it is a "boring window manager for the adult in you. Many window managers are like Marshmallow Froot Loops; Metacity is like Cheerios ... includes only the window manager; does not try to be a desktop environment. The pager, configuration, etc. are all separate and modular." According to Wikipedia, GNOME used the Enlightenment window manager and then Sawfish before choosing Metacity. Apparently, it is aimed at new computer users that do not need the extreme configurability offered by its predecessors.

Build commands

First I look at the README file in the package directory. Then I check if the INSTALL file is anything but the standard one, using "head".

set -b     # so bash tells me when jobs finish  configure --help=short 2>&1 | less    # shows options specific to package  configure >myconf01.lst 2>&1 &        # usually installs into /usr/local -                                       # unless you add a --prefix=/mydir option # have a look at the log file, - # especially the end  make >mymake01.lst 2>&1 & # at the start, "tail -f" the log file - # to make sure the compiler options are OK  make install >mymakeinst01.lst 2>&1 & # you may need to be root, of course. # look at the log file to see where the binaries are put, what are the actual # names of the libraries installed, and where to find the documentation

Commands to add swap

Adding swap lets you pretend you have more memory. I did not know how to do this when I started!

# run as root dd if=/dev/zero of=swap1 bs=1024 count=100k   # 100 Meg mkswap swap1 swapon swap1

Sometimes the program really is too big and the PC will just swap files in and out all the time. I can see the laptop is "thrashing" simply by the hard-disk light staying on! I also use the "vmstat 5" command. It reports every five seconds. The CPU will be waiting for IO to finish most of the time. The "cpu wa" column will be high, say 97%. The program will not have much time to run, with a low "user time" for my non-kernel code. The "cpu us" column might say 3%. You can see the amount of blocks read and written rising in the "IO bi bo" columns. (The first line produced by vmstat gives averages since the system booted.)

Good luck!

Joey's Notes: VSFTP FTP server on RHEL 5.x

Joey Prestia — Wed, 01 Jul 2009 22:07:56 -0500

By Joey Prestia

File Transfer Protocol (FTP) utilizes Transmission Connection Protocol (TCP) communication and listens on port 21 for incoming client connections by default. The connection to this port from the FTP client will form the control stream from which commands are passed from the FTP client to the FTP server and on occasion from the FTP server to the FTP client. FTP uses a separate connection for control and data. So for the actual file transfer to take place, a different connection is required which is called the data stream. Depending on the transfer mode, the process of setting up the data stream is different. Port 21 for control, port 20 for data.

Understanding Full Text Search in PostgreSQL

Paul Sephton — Wed, 01 Jul 2009 22:07:56 -0500

By Paul Sephton

Those who have yet to enter their first search query in Google, Yahoo, or any of the myriad of other search engines available to the web, or have no interest in the use of PostgreSQL as a database engine to implement their own text search facility will find little use in reading further.

Although PostgreSQL coverage on the topic of Free Text Search is as good as the rest of their excellent documentation, the following might serve as an introduction to those who would further persue the topic. In the interest of conciseness, this article skips quite a lot of detail and depends on the ability of the reader to infer what is not explicitly stated.

Introduction

Full Text Search (or FTS) is not a new technology. The earliest recorded patents related to the search for documents about a given topic were filed in the year 1963, more than forty-five years ago. These patents include "CONTENT ADDRESSABLE MEMORY APPARATUS" (US Pat. 3290659 - Filed Dec 30, 1963), "SCAN CONTROL, AND NORMALIZATION FOR A CHARACTER RECOGNITION SYSTEM" (US Pat. 3295105 - Filed Aug 27, 1964) and "INFORMATION RETRIEVAL SYSTEM AND METHOD" (US Pat. RE26429 - Filed Dec 8, 1964).

To this day, the ability to impose structure upon the dearth of what is known as "Unstructured Data", in order to turn this haystack into an information system, continues to challenge the finest minds.

The long list of granted patents persists through 2006 and 2007, namely "Device and system for information management" (US Pat. 7376273 - Filed Jun 1, 2007 - Silverbrook Research Pty Ltd), "Metasearching by sending a plurality of queries to a plurality of servers" (US Pat. 7277918 - Filed Jan 16, 2007) and "Distributed internet based speech recognition system with natural language ..." (US Pat. 7203646 - Filed May 22, 2006 - Phoenix Solutions, Inc.)

Of course, we know today that "Full Text Search" is far from the only way to organise documents. A close runner up, though unable to reproduce the exact features of FTS, is the Bayesean search; a statistical means of indexing documents which determines the probabability that two documents are similar. Bayesean search has an almost uncanny knack of deriving the essence of documents rather than depending on a literal text match. The most common use of Bayesean search technology today is in spam filters, although engines such as DeepDyve demonstrates a more ambitious use.

Several SQL databases already have a built-in implementation of FTS, notably servers such as Oracle, SQL Server, and the free MySQL and PostgreSQL servers.

MySQL native FTS is currently only available against the MyISAM database back-end, and is not yet available for the more popular InnoDB back-end. For Postgres, the earliest implementation of FTS was the TSearch module, replaced in version 7.4 with Tsearch2, and finally included in the core PostgreSQL engine in version 8.3.

There are many implementations of external FTS engines which may be used in conjunction with SQL database engines, or individually. These include Lucene, which is a very powerful and popular engine implemented as part of the Apache project, Swish-E, and the Sphynx search engine, which has been steadily gaining in popularity.

Lucene is really a library against which programmers may develop search products. Their site has many links to generic products already developed against the library, including interfaces for Java (the original interface), PHP and DotNet. Sphynx is particularly popular amongst MySQL developers, as this engine supports both MyISAM as well as InnoDB database back-ends.

PostgreSQL Full Text Search

In a nutshell, Full Text Search is implemented by indexing the words contained in a document, and associated the indexed words with a reference to the document. Subsequently, searches for a boolean phrase using operators and, or, not and braces may be matched against the index to locate documents containing the words in the phrase. PostgreSQL does not currently support the fuzzy logic operators near, far or strip.

Eliminating redundancy

Clearly, indexing every single word in a document would result in a very large index. This is neither required, nor is it particularly useful. For example, one might convert all text to lower case prior to indexing the words, making your searches insensitive to case, and simultaneously achieving a smaller index. Then one might eliminate words from the text which carry no real meaning (like 'and', 'or', 'the', 'for', etc.) as these words would be likely found in most documents already. The first approach is called normalisation, and the second process is called stop word elimination.

Finally, one might further reduce the size of the index by replacing words with others which have an identical or similar meaning. Thus one may replace instances of 'hungrily' and 'hungry' with 'hunger'. This process is called dictionary substitution.

Further algorithmic measures (see Snoball)[Ref. 4] may be performed to further reduce words to their essential meaning prior to the article being indexed. The replacement of colour names with their hexadecimal equivalents and the reduction of numeric values by reducing precision are other ways of normalising the text.

All of these measures are to eliminate redundancy from the indexed text, thereby reducing index size and resulting in less storage requirement, less disk I/O, faster indexing and a consequently faster search.

PostgreSQL Dictionaries

To aid in the process of normalisation and elimination of redundancy from text, PostgreSQL provides templates for several types of dictionary for use in a text search configuration. These are the Simple Dictionary, the Synonym Dictionary, the Thesaurus Dictionary, the iSpell Dictionary and the Snoball [Ref.4] Dictionary.

The Simple Dictionary eliminates stop words, and performs case normalisation. The Synonym Dictionary replaces one word with another, Thesaurus Dictionaries provide for industry specific phrase recognition, and the iSpell template may be used to embed any of the standard iSpell dictionaries available from OpenOffice.Org. The Snoball[Ref.4] dictionaries, which perform algorithmic stemming and stop word elimination are included by default for a variety of languages in the PostgreSQL installation.

With PostgreSQL, all dictionaries are functionally equivalent, with the possible exception of the thesaurus type. Effectively, they consume a word (or token), and return an array of lexemes, or NULL. When a dictionary returns NULL, the token is considered to be unrecognised. This allows dictionaries to be strung together, with the most general dictionary at the end of the list. Thus, if a dictionary earlier in the list returns a word, further dictionaries in the list are ignored. However, where an earlier dictionary returns NULL, the token is processed by subsequent dictionaries in the list.

A lexeme is the equivalent of a token which has been converted to it's base form. Before being passed to the dictionary, PostgreSQL converts the document text into an array of tokens through means of a simple parser. The parser is able to identify various types of tokens from the text, such as XML or HTML tags, integer or floating point numbers, version numbers, URL's, host names and so forth. PostgreSQL provides the ability to process different token types specifically, by mapping a given token type to different dictionary lists.

Text Search Configuration

Note that although PostgreSQL provides a lot of configurability, the base installation already provides a workable configuration. Unless there is a specific reason, it is probably not necessary to mess with it. Even so, an example text search configuration would be the following:

CREATE TEXT SEARCH CONFIGURATION public.my_config (COPY=pg_catalog.english );

Now, having a new text search configuration, we can create a new dictionary:

CREATE TEXT SEARCH DICTIONARY english_ispell (     TEMPLATE = ispell,     DictFile = english,     AffFile = english,     StopWords = english );

In the above, we created a dictionary based on the iSpell template, where DictFile, AffFile and StopWords refer to files in the `pg_config –share`\tsearch_data\ directory, called english.dict, english.affix, and english.stop respectively.

We can then add this dictionary into a configuration like this:

ALTER TEXT SEARCH CONFIGURATION my_config     ALTER MAPPING FOR asciiword, asciihword, hword_asciipart,                       word, hword, hword_part     WITH english_ispell, english_stem;

This adds our new dictionary to our new text search configuration, overriding the default lexeme types (asciiword etc.) to ensure that they are processed through both the english_ispell and english_stem dictionaries. We can then start using our new text search configuration by changing the global parameter default_text_search_config:

SET default_text_search_config = 'public.my_config';

The above parameter will apply for the duration of the session, or until it is changed. To make the configuration permanent, one would need to set it in the data/postgresql.conf file.

Search Vectors

All of this is very impressive, but how does one turn document content into an array of lexemes using the parser and dictionaries? How does one match a search criterion ti body text? PostgreSQL provides a number of functions to do this. The first one we will look at is to_tsvector().

A tsvector is an internal data type containing an array of lexemes with position information. The lexeme positions are used when searching, to rank the search result based on proximity and other information. One may control the ranking by labelling the different portions which make up the search document content, for example the title, body and abstract may be weighted differently during search by labelling these sections differently. The section labels, quite simply A,B,C & D, are associated with the tsvector at the time it is created, but the weight modifiers associated with those labels may be controlled after the fact.

We can create a tsvector for text like this:

bob=# select to_tsvector('Free text seaRCh is a wonderful Thing');                      to_tsvector                       ------------------------------------------------------  'free':1 'text':2 'thing':7 'search':3 'wonderful':6 (1 row)

Assigning Weight Labels

As may be seen, the tsvector is just a list of lexemes with associated positions. The stop words such as 'a' and 'is' have been eliminated, and we have everything in lower case. Another example, adding labels:

bob=# select setweight(to_tsvector('Free text seaRCh is a wonderful Thing'),'A');                          setweight                          -----------------------------------------------------------  'free':1A 'text':2A 'thing':7A 'search':3A 'wonderful':6A (1 row)

In essence, one may create a new tsvector with lexemes labeled for different sections of the text using setweight for the various sections:

bob=# select                                                                   bob-# setweight(to_tsvector('All about search'), 'B') || bob-# setweight(to_tsvector('Free text seaRCh is a wonderful Thing'),'A');                            ?column?                             ---------------------------------------------------------------  'free':4A 'text':5A 'thing':10A 'search':3B,6A 'wonderful':9A (1 row)

Assuming "All about search" was the title, and "Free text seaRCh is a wonderful Thing" was the body, we have now labelled the lexemes from the title, together with those from the body. Note that the word 'All' and 'About' in the title were both considered to be stop words. Subsequently, one may use the labels to rank the results depending on weight labelss associated with the title and body. We will visit this in more detail later.

Matching Boolean Search Phrases

But first, how does one go about matching a boolean search phrase to the tsvector? This is done using the @@ operator. For example:

bob=# select to_tsvector('Free text seaRCh is a wonderful Thing') @@ 'wonderful';  ?column?  ----------  t (1 row)

On the other hand 'wonderful' is not exactly a wonderful example of a boolean phrase, now is it? Nope. It's just ordinary text. If that text is in the wrong format, for example contains other words, the query will in fact fail. PostgreSQL provides two functions which may be used to turn text into a query, represented as the built in tsquery data type, for subsequent matching against tsvectors. These are the plainto_tsquery() and to_tsquery() functions.

The rather limited plainto_tsquery() function simply turns the text into a tsquery with all the lexemes in that text qualified by the '&' (or AND) operator:

bob=# select plainto_tsquery('wonderful text');   plainto_tsquery     ----------------------  'wonderful' & 'text' bob=# select to_tsvector('Free text seaRCh is a wonderful Thing') @@ plainto_tsquery('wonderful text');  ?column?  ----------  t (1 row)

Where this might be useful, plainto_tsquery() does not provide for boolean operators other than '&'. to_tsquery() goes a lot further toward providing a decent boolean phrase. Unfortunately, it is also rather finicky about the text you throw at it. It will not accept text which is not separated by '&', '|' or '!'. It does accept braces, which are used to indicate operator precidence, but two tokens which translate to different lexemes directly after one another will cause to_tsquery() to fail.

bob=# select to_tsquery('wonderful text'); ERROR:  syntax error in tsquery: "wonderful text" bob=# select to_tsquery('wonderful | text');       to_tsquery       ----------------------  'wonderful' | 'text'

Ranking Search Results

Nevertheless, we may use a tsquery boolean search phrase to match against any tsvector, or use a tsquery and a tsvector to produce a ranking by use of either the ts_rank() or ts_rank_cd() functions. These two functions behave slightly differently. ts_rank() is considered the 'standard' ranking function, whilst ts_rank_cd() uses the Cover Density Ranking algorithm[ref.6], which is more interested in phrases than in the actual terms of the query itself.

To rank a match, one would use:

bob=# select ts_rank(to_tsvector('Free text seaRCh is a wonderful Thing'), to_tsquery('wonderful | thing'));   ts_rank   -----------  0.0607927 (1 row)  bob=# select ts_rank(to_tsvector('Free text seaRCh is a wonderful Thing'), to_tsquery('wonderful & thing'));   ts_rank   -----------  0.0991032 (1 row)  bob=# select ts_rank_cd(to_tsvector('Free text seaRCh is a wonderful Thing'), to_tsquery('wonderful & thing'));  ts_rank_cd  ------------         0.1 (1 row)

Relating this ranking ability back to the subject of tsvector weights, you will recall

bob=# select                                                                   bob-# setweight(to_tsvector('All about search'), 'B') || bob-# setweight(to_tsvector('Free text seaRCh is a wonderful Thing'),'A');                            ?column?                             ---------------------------------------------------------------  'free':4A 'text':5A 'thing':10A 'search':3B,6A 'wonderful':9A (1 row)

where we labelled the text sections of our document? Now we can do the following:

bob=# select ts_rank( bob-#  array[0.1,0.1,0.9,0.1], bob-#  setweight(to_tsvector('All about search'), 'B') ||  bob-#  setweight(to_tsvector('Free text seaRCh is a wonderful Thing'),'A'), bob-#  to_tsquery('wonderful & search'));  ts_rank   ----------  0.328337 (1 row)  bob=# select ts_rank( bob-#  array[0.1,0.1,0.1,0.9], bob-#  setweight(to_tsvector('All about search'), 'B') || bob-#  setweight(to_tsvector('Free text seaRCh is a wonderful Thing'),'A'), bob-#  to_tsquery('wonderful & search'));  ts_rank   ----------  0.907899 (1 row)

The array[0.1,0.1,0.9,0.1] which is passed as the initial argument to ts_rank() takes arguments in order {D,C,B,A}. Since we labelled our sections A (for the body) and B (for the title), we first assigned B=0.9, A=0.1 and later B=0.1,A=0.9 in the statements above. Results of the ranking function differ accordingly. If not specified, the optional weights array defaults to {0.1, 0.2, 0.4, 1.0}.

Indexing TSVectors: The GIST & Gin

Until now, we have used only the select statement in our examples to demonstrate the matching between tsvector and tsquery, and subsequent ranking capabilities of PostgreSQL. Moving forward we will consider the use of indexes to speed up searches, and later tackle the real showstopper which we have glossed over until now: why is to_tsquery so pedantic, and how do we deal with that?

The two index types supported by PostgreSQL for full text search, are the GIST index, which is based on hash tables, and the Gin index which is based upon the Btree.

GIST is really fast in creating indexes. It really stores a hash table of the terms in the tsvector, and uses a hash of the terms in the tsquery to find the associated documents. Unfortunately, since the hash search result is non-deterministic, PostgreSQL has to check the results of the search, reading all of the articles located and double check the match before returning the result. For smaller sets of lexemes, and smaller databases, this works quite well. However, for really large datasets, the overheads in re-reading the data (or index) tend to make this approach quite slow.

Gin indexes are deterministic, and there is no implied overhead after a search using the index, but the downside to Gin is that index creation slows down logarithmically (thankfully not exponentially) as the number of entries grows. Whichever index method is chosen should therefore take the nature of the database and it's size into account.

There are two approaches to using an index. The first of these creates an index against a function of the data field, and the second approach is to store a tsvector in an additional field in the table; this tsvector field is then indexed. For example,

CREATE INDEX pgweb_idx ON pgweb    USING gin(to_tsvector('english', title || ' ' || body));

would create a Gin index using the combination of the title and body fields. This method is simple, but has an implied overhead: Index creation has to do additional work, and your search is more complex. The second approach uses a trigger to populate the dedicated tsvector field whenever a record is added or deleted from the table:

ALTER TABLE pgweb ADD COLUMN tsv tsvector; UPDATE pgweb SET tsv =      to_tsvector('english', coalesce(title,'') || ' ' || coalesce(body,'')); CREATE TRIGGER tsvectorupdate BEFORE INSERT OR UPDATE   ON pgweb FOR EACH ROW EXECUTE PROCEDURE   tsvector_update_trigger(tsv, 'pg_catalog.english', title, body);

Now we simply add, update or remove table rows as normal, and the trigger populates the tsvector field in the table row, which in turn updates the index appropriately. Subsequent queries may be against the tsvector field, and that simplifies the query.

Search result Markup

We should mention at this point, before proceeding any further, a rather useful feature related to text search. PostgreSQL provides the ability to mark up text based upon the result of a text search. The function here, is pg_headline(). This function returns text as a result, with all matching words in the text enclosed in <b></b> HTML tags. It is really easy to use:

bob=# select ts_headline('Free text seaRCh is a wonderful Thing',         to_tsquery('wonderful & thing'));                  ts_headline                      -----------------------------------------------------  Free text seaRCh is a <b>wonderful</b> <b>Thing</b> (1 row)

To apply highlights to a matching abstract, one might issue the command:

select ts_headline(abstract, query)   from pgweb, to_tsquery('wonderful & thing') query,    where query @@ tsv;

Phrase Search

Speaking of features, one allegedly missing feature which some people, particularly users of external text engines continuously complain about PostgreSQL not having, is phrase search. Here we refer to the apparent non-existent ability of PostgreSQL's full text search engine to support the match of indexed literal phrases in the text.

Just stop to think about it for a moment. What exactly are we expecting here? How many literal phrases are there in a document of say 10 words? Let's assume for the moment that the words are single character words: a,b,c...j. All the possible phrases in this document are:

a,ab,abc,abcd... (10 phrases)

b,bc,bcd,bcde... ( 9 phrases)

etc...

Put differently, for a document consisting of (n) words, there are (n) phrases starting with the first word, n-1 starting with the second word, n-3 starting with the third word, etc. This translates to

number of phrases = n/2 * (n+1) = 10/2*(10+1) = 55

phrases in a document with just 10 words. For a document of 100 words, we have 5050 phrases! Are we seriously suggesting a literal phrase index is a good thing? Remember, such an index would have to contain all words, and could not afford to eliminate redundancy from the text (eg. Stop words). Not really a practical approach.

How then, do external text search engines support index assisted literal phrase search? Why, in exactly the same way PostgreSQL does it! Quite simply, the algorithm goes:

Turn the text phrase into a boolean search phrase, using the '&' operator
Use the full text search engine to locate the matching documents
For only those documents found, and using the original text phrase do a case insensitive text match, eliminating false positives

For PostgreSQL, case sensitive phrase matching is supported using the LIKE operator, and case insensitive matching through the ILIKE operator. Searching for the text

free text and 'Postgres text search'

the query would go something like this:

select headline, ts_rank(tsv, query)    from pgweb, to_tsquery('free & text & postgres & search') query     where tsv @@ query and body ilike '%Postgres text search%';

... which uses the FTS index to find articles with body text matching a query, and then for those items finds the intersection matching the literal text.

Simple, eh?

Parsing Human Query Strings

Not quite so simple. How, pray, is a poor developer supposed to turn the 'human' phrase above into the SQL statement below? This is a non-trivial task and mores the pity, well beyond the abilities of some developers. Indeed, this is considered to be a 'show-stopper' for many people. It involves building a pre-parser to parse the entered search phrase, the execution of which translates the natural language search phrase into a SQL statement formatted similarly to the above.

To end off this article, we include with apologies to YACCers, the listing of a pre-parser which does precisely that. The general logic is generic enough to be translated to almost any language, but the listing shown here is in standard C++. This is a priority biassed recursive descent parser with no external dependencies. Feel free to use the code in any way you wish (hereby BSD license).

Fedora 11 on the Eee PC 1000

Anderson Silva — Wed, 01 Jul 2009 22:07:56 -0500

By Anderson Silva

Red Hat's sponsored Fedora Project just released its 11^th version of the RPM based distribution this month, and as a long time user, I thought I'd share a few things I had to do to get Fedora 11 running smoothly on my Eee PC 1000.

Sending and Receiving SMS from your Linux Computer

Suramya Tomar — Wed, 01 Jul 2009 22:07:56 -0500

By Suramya Tomar

What is SMS? SMS stands for Short Message Service and is used to send text messages from one GSM enabled cell phone to another. But sometimes you might want to send an SMS message using your computer. Now there are three ways that you can send an SMS from a computer:

Use a SMS gateway service that allows your computer to connect to it using an API and send messages. There are a lot of these services available on the web but most are not free and require subscriptions
Use a cell-phone/GPRS Modem connected to the computer.
Some service providers allow you to send SMS to a number by sending an email to a specifically formated email address.

I choose to use the second option because my provider doesn't support the last option and I think its cheaper than option one. All you need for this to work is an extra phone and a cheap phone plan that allows you to send and receive SMS's. In my case I had an old Sony Ericsson T610 lying in my desk gathering dust so I got a pre-paid SIM card and I was ready to proceed.

Pre-Installation

Ecol

Javier Malonda — Wed, 01 Jul 2009 22:07:56 -0500

By Javier Malonda

These images are scaled down to minimize horizontal scrolling.

XKCD

Randall Munroe — Wed, 01 Jul 2009 22:07:56 -0500

By Randall Munroe

Click here to see the full-sized image

More XKCD cartoons can be found here.

The Linux Launderette

LG Staff — Wed, 01 Jul 2009 22:07:56 -0500

By LG Staff

Find me good clinic, plz
I came across this to today

Linux Gazette

Mailbag

Talkback

2-Cent Tips

News Bytes

Layer 8 Linux Security: OPSEC for Linux Common Users, Developers and Systems Administrators

Building the GNOME Desktop from Source

GNOME versus KDE

Browsers

Problems compiling packages

GNOME libraries

Base libraries

Notes

Platform libraries

Notes

Desktop libraries

Notes

Build commands

Commands to add swap

Joey's Notes: VSFTP FTP server on RHEL 5.x

Understanding Full Text Search in PostgreSQL

Introduction

PostgreSQL Full Text Search

Eliminating redundancy

PostgreSQL Dictionaries

Text Search Configuration

Search Vectors

Assigning Weight Labels

Matching Boolean Search Phrases

Ranking Search Results

Indexing TSVectors: The GIST & Gin

Search result Markup

Phrase Search

Parsing Human Query Strings

Other Linux Gazette Articles about PostgreSQL

References

Fedora 11 on the Eee PC 1000

Sending and Receiving SMS from your Linux Computer

Pre-Installation

Ecol

XKCD

The Linux Launderette