August 2006 (#129):

• 2-cent Tips
• Mailbag
• Talkback
• Making XMMS Effect plugins, by Kumar Appaiah
• Away Mission -- SEMTECH - The Semantic Web Technology Conference, by Howard Dyckoff
• The Geekword Puzzle, by Ben Okopnik
• Low-Fat Linux - Now with Less Cruft!, by Ben Okopnik
• A Prisoner of Windows, by Lew Pitcher
• Issues In Concurrent Server Design on Linux Systems - Part I, by Amit Saha
• Ecol, by Javier Malonda

2-cent Tips

:%s#\(\s*\n\)\{2,\}#\r</p>&<p>\r#g
:%s#\(.*\n\)*#<p>\r&\r</p>#g

" Mark the file with HTML paragraphs
map \H :%s#\(\s*\n\)\{2,\}#\r</p>&<p>\r#g<CR>:%s#\(.*\n\)*#<p>\r&\r</p>#g<CR>

user_pref("capability.policy.policynames", "nojs");
user_pref("capability.policy.nojs.sites", "http://site1.com http://site2.com");
user_pref("capability.policy.nojs.javascript.enabled", "noAccess");

perl -we 'print sort { int( rand 2 ) * 2 - 1 } <>' list.m3u > random.m3u

  sudo apt-get install libxmu-dev
  sudo apt-get install libxaw7-dev
  wget http://www.unixuser.org/~euske/vnc2swf/vnc2swf-0.5.0.tar.gz
  tar zxvf vnc2swf-0.5.0.tar.gz
  cd vnc2swf-0.5.0
  ./configure
  make
  ./vnc2swf -startrecording outfile.swf your_vnc_server:1 > outfile.html

  sudo apt-get install python2.3-tk
  sudo apt-get install python-pygame
  wget http://www.unixuser.org/~euske/vnc2swf/pyvnc2swf-0.8.2.tar.gz
  tar zxvf pyvnc2swf-0.8.2.tar.gz
  cd pyvnc2swf-0.8.2
  ./vnc2swf.py

Talkback: Discuss this article with The Answer Gang

Mailbag

--------------------------------------------------------------------------
* * * * * * * THESE ARE NOT THE DROIDS YOU'RE LOOKING FOR * * * * * * *

The address to which you have sent your query is that of The Answer Gang 
for the Linux Gazette. We answer questions related to the Linux operating
system. Unrelated questions are likely to get anything ranging from a 
polite answer to astonished laughter, with curmudgeonly replies being 
somewhat of a norm. Good karma points are awarded to those who tell us 
where they got our address and the reason they thought that we might help 
with fish identification, spaceship repair, or nude snowboarding. 

Spammers will be whacked with a dead trout; flames will be graded for 
content, laughed at, and sent to /dev/null; amusing bits may get shared 
with our readers.

Good luck with your search.
--------------------------------------------------------------------------

  > I've always avoid the use of escape codes in favour of tput(1) -- at
  > least then it works remotely, too.

	w/o == without
	h/w == hardware
	s/w == software
	s/o == Significant Other (*not* son of)
...

It  is  generally not good practice to hard-wire terminal controls
into programs.  Linux supports a terminfo(5) database of terminal
capabili- ties.   Rather than emitting console escape sequences by hand,
you will almost always want to use a terminfo-aware screen  library  or
utility such as ncurses(3), tput(1), or reset(1).

  > > Well, the standard disclaimer applies. Any suggestions and corrections
  > > are welcome (all flames to /dev/null).
  > 
  > It may already be filled up; that's where we're sending ours...

bogo_router:
 domains = +local_domains
 no_verify
 condition = ${if !eq {$received_protocol}{bogodone} {1}{0}}
 driver = accept
 transport = bogo_transport

bogo_transport:
 driver = pipe
 command = /usr/sbin/exim4 -oMr bogodone -bS
 use_bsmtp = true
 headers_add = X-Bogofilterd: true
 transport_filter = /usr/bin/bogofilter -d /etc/bogofilter -l -p -e -u
 return_fail_output = true
 group = mail
 user = mail
 home_directory = "/tmp"
 current_directory = "/tmp"
 log_output = true
 return_path_add = false

sudo update-exim4.conf && sudo /etc/init.d/exim4 reload

2006-06-02 04:54:13 1Fm0jl-0006a4-8V ** thomas at edulinux.homeunix.org
R=bogo_router T=bogo_transport: Child process of bogo_transport transport
returned 2 from command: /usr/sbin/exim4

An error was detected while processing a file of BSMTP input.
The error message was:

 500 Command unrecognized

The SMTP transaction started in line 0.
The error was detected in line 1.
The SMTP command at fault was:

'exim -d9 -M <message-id>' - turning on max_debug and vgrepping through

  > As at log in prompt I see  <hostname> login:
  > so I tried to grep for the "login:" in .c files. but really could not
  > find the the proper location in the source code.

  > I have a 2.6.15-01.2054_FC5 source code.
  > 
  > Am very very interested to understand this code. I understand C
  > language quite well.
  > 
  > I have 3 questions.
  > 1. Is this approach to understand the linux kernel code all right?

  > 2. If any one can give me some hint where I can find the login related
  > code or where I should start .. I will be very glad.

  > 3. Is there any documentation available which I can use as reference?

-------- Original Message --------
Subject: Mail delivery failed: returning message to sender
Date: Mon, 19 Jun 2006 16:48:51 +0100
From: Mail Delivery System <Mailer-Daemon at blueyonder.co.uk>
To: martinjh at blueyonder.co.uk

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

   tchung at fedoranews.org
     SMTP error from remote mail server after MAIL 
FROM:<martinjh at blueyonder.co.uk>:
     host fedoranews.org [64.34.165.170]: 550 5.7.1 Access denied

------ This is a copy of the message, including all the headers. ------

Return-path: <martinjh at blueyonder.co.uk>
Received: from [172.23.170.146] (helo=anti-virus03-09)
	by smtp-out5.blueyonder.co.uk with smtp (Exim 4.52)
	id 1FsLze-0003tl-GW
	for tchung at fedoranews.org; Mon, 19 Jun 2006 16:48:50 +0100
Received: from [62.30.33.77] (helo=[192.168.0.2])
	by asmtp-out2.blueyonder.co.uk with esmtp (Exim 4.52)
	id 1FsLzd-00033G-Ni
	for tchung at fedoranews.org; Mon, 19 Jun 2006 16:48:50 +0100
Message-ID: <4496C760.8070803 at blueyonder.co.uk>
Date: Mon, 19 Jun 2006 16:48:48 +0100
From: Martin Hooper <martinjh at blueyonder.co.uk>
User-Agent: Thunderbird 1.5.0.4 (Wind0ws/20060516)
MIME-Version: 1.0
To:  tchung at fedoranews.org
Subject: Fedora Security Bulletins
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Tom,

Having returned to Fedora after having used Kubuntu for a while I was
wondering if there was a security update announce list I could subscribe
to??

Keep up the good work on Fedora News...

Martin

Talkback: Discuss this article with The Answer Gang

Talkback

initializing renderer...FAILED;
illegal parametre;

[Ben] - Well, first off, Unix doesn't have "Folders";
I presume that you're talking about directories.

netstat -apvn | grep -i xfs
netstat: no support for `AF IPX' on this system.
netstat: no support for `AF AX25' on this system.
netstat: no support for `AF X25' on this system.
netstat: no support for `AF NETROM' on this system.
unix  2      [ ACC ]     STREAM     LISTENING     12007  5435/xfs /tmp/.font-unix/fs-1
unix  3      [ ]         STREAM     CONNECTED     2398890 5435/xfs /tmp/.font-unix/fs-1
unix  2      [ ]         DGRAM                    11994  5435/xfs

Talkback: Discuss this article with The Answer Gang

Making XMMS Effect plugins

By Kumar Appaiah

Most of us will agree that though there are many audio players for GNU/Linux and the BSDs, the most popular by far is XMMS. Not only is it similar to Winamp, which eases things for people coming from Windows, it is also simple and has all the features one would expect in a complete multimedia player. Best of all, it is extensible with additional plugins for added functionality, file support, visualization and, of course, sound effects.

In this article, I will present an introduction to some functions provided in the XMMS plugin library for developing our own effects, along with some useful techniques. Using an example (a simple echo plugin), I shall walk through the steps, introducing the functions one by one. The only requirement is that you should know C, but you needn't be an audio engineer (I'm not one!) to be able to churn out some simple effects.

Before we begin - an effect plugin, or, for that matter, any other plugin for XMMS, is a shared object (.so) file. Make sure you have gcc, make and other development tools on you machine, along with the XMMS plugin development headers and libraries.

Prerequisites

Check if the XMMS plugin library is installed already on your system. The presence of headers, /usr/include/xmms or /usr/local/include/xmms, is a good indication of this. If it isn't, install the xmms-devel (or equivalent) package from your distribution.

Technique

Though this article should be only about writing XMMS plugins, I also thought that it would be appropriate to mention a buffering technique used all the time in audio processing, called circular buffers. In our case, let us say we want to make the echo sound, say 370 ms (0.37 seconds) later. If we are working with data sampled at 44100 Hz, then the size of the buffer required would be 0.37 * 44100 = 16317 elements each for left and right (assuming you are providing separate echoes for the two channels). Now, the obvious way to solve our problem is, as soon as we get a new sample, we can retrieve the most delayed sample from the buffer for the echo effect, then shift the contents of the buffer so that the second element goes to the first place, third to second and so on and push the new sample to the last position, just like a FIFO queue. However, this would mean about 16317 memory read and write operations have to be executed for every sample processed, which is quite expensive when it comes to CPU use.

A simple solution to this problem is to treat the buffer as circular. Here, we create an index or pointer to the element (or an index) in the buffer which you mark as the last element to be read for the delay, read that element for the echo effect, write the new sample into the same memory location, and increment the index. If the pointer or index reaches the end of the array, it just loops back to the first element. The following picture should give you an idea.

With this, we have reduced the memory operations per sample to just one read and write; many times more efficient than the previous case. For more details, see the Wikipedia article on this topic.

Writing the Plugin

Now, we start writing a minimal plugin that will create a fixed echo effect. The full source is available in exampleecho.c.

A plugin is registered with XMMS using the following function:

EffectPlugin
*get_eplugin_info(void);

First, we create the buffers for the echo effect. I have chosen a 16384-sample delay, which is about 0.37 sec. Since we will use these as circular buffers, we shall keep two integers to index the location in the buffers.

#define BUFFER_LENGTH 16384

static gfloat *buffer_left = NULL;
static gfloat *buffer_right = NULL;
static guint index_left = 0;
static guint index_right = 0;

Now, we observe the requirements of the EffectPlugin structure, defined in /usr/include/xmms/plugin.h:

typedef struct
{
	void *handle;		/* Filled in by xmms */
	char *filename;		/* Filled in by xmms */
	char *description;	/* The description that is shown in the preferences box */
	void (*init) (void);	/* Called when the plugin is loaded */
	void (*cleanup) (void);	/* Called when the plugin is unloaded */
	void (*about) (void);	/* Show the about box */
	void (*configure) (void);	/* Show the configure box */
	int (*mod_samples) (gpointer *data, gint length, AFormat fmt,
	gint srate, gint nch);	/* Modify samples */
	void (*query_format) (AFormat *fmt,gint *rate, gint *nch);
}
EffectPlugin;

Though most of the functions specified are optional and you can just make them NULL, we see that we need to write some functions for our plugin to be useful. I will call these query, init, cleanup, configure, about and mod_samples, though you can name them whatever you like. Of these, we will first write about, since it seems to be the easiest!

static char *about_text = "Kumar's test reverberation plugin!";

void
about(void)
{
	static GtkWidget *about_dialog = NULL;
	if (about_dialog != NULL)
		return;
	about_dialog = xmms_show_message("About Example Echo Plugin",
					 about_text, "Ok", FALSE,
					 NULL, NULL);
	gtk_signal_connect(GTK_OBJECT(about_dialog), "destroy",
			   GTK_SIGNAL_FUNC(gtk_widget_destroyed),
			   &about_dialog);
}

Next, we write the query function. The query function informs XMMS what type of audio data we wish to process. This is a good idea, because our plugin may not function as expected for data with different attributes, such as a different sampling rate. To make it more clear, let us say we want an echo after 0.1 seconds, that is 0.1 * 44100 samples later when 44100 Hz is the sampling rate. But it should be changed to 0.1 * 22050 for a file with 22050 Hz sampling rate. Yes, it can be done on the fly, but I leave such tricks for you to try out.

We'll make our plugin work with 16 bit little-endian stereo data, sampled at 44100 (stereo) samples per second.

void
query(AFormat *afmt, gint *s_rate, gint *n_channels) {
	*afmt = FMT_S16_LE;
	*s_rate = 44100;
	*n_channels = 2;
}

The init function is used to initialize your plugin's settings, and you may also want to initialize your audio buffers here. This function is called the moment the plugin is enabled, or whenever the plugin loads (like when XMMS is started). In our plugin, we use two audio buffers to hold past left and right samples, which need to be zeroed initially to avoid "noise". Note that g_malloc0 is a GLIB function. In general, it is convenient to use GLIB functions and data types, and there are no extra requirements because XMMS needs them and uses them anyway.

void init(void)
{
	if (buffer_left == NULL) {
		buffer_left = g_malloc0(sizeof(gfloat) * BUFFER_LENGTH);
	}
	if (buffer_right == NULL) {
		buffer_right = g_malloc0(sizeof(gfloat) *
		BUFFER_LENGTH);
	}
}

The cleanup, as the name very well suggests, is to free any resources you might have allocated and prepare for releasing the plugin. It is called the moment the plugin is disabled, or when XMMS exits, if the plugin remains enabled at that time. Here, we just free the buffers that we have allocated.

void
cleanup(void)
{
	g_free(buffer_left);
	buffer_left = NULL;
	g_free(buffer_right);
	buffer_right = NULL;
}

The configure function is used to allow the user to modify some characteristics of the plugin. It is called when the user clicks "Configure" for your plugin in the Preferences dialog box. In our plugin, I have left it blank, since I wanted to avoid adding a lot of GTK code. However, this is fairly simple, and left as an exercise for the reader. I would suggest that you refer the source of the plugins which come with XMMS for ideas.

Note that the xmms_cfg_* functions make the use of the XMMS configuration extremely easy, with XMMS doing the dirty work of reading and writing the configuration file. They are present in /usr/include/xmms/configfile.h

Finally, we come to the most important function of the effect plugin, mod_samples, in which the samples are provided and are to be appropriately modified. The declaration of this function is as follows:

gint
mod_samples(gpointer * d, gint length, AFormat afmt, gint srate, gint nch);

Here, d is the pointer to the data (samples), and length is the number of bytes of data provided to the function in a given instance. This function is called again and again with successive data; that is, in one pass, you are provided the first set of samples, in the next one, the next set of samples and so on, in order. The format of the data is given by the AFormat value. By checking the afmt value, you can dynamically adjust to changes in data type, etc.; however, we are going to just assume it is what we specified in the query function.

gint mod_samples(gpointer * d, gint length, AFormat afmt, gint srate, gint nch)
{
	/* Read data as 16 bit values */
	gint16 *data = (gint16 *) * d;
	gint i;

	/* We have "length" bytes, so "length / 2" samples */
	for (i = 0; i < length / 2; i += 2) {
		gfloat echo_val;

		/* For the left sample */
		echo_val = buffer_left[left_index]; /* Retrieve delayed value */
		buffer_left[left_index++] = data[i]; /* Store latest sample */
		data[i] = (1 - ECHO_SCALE) * data[i] + ECHO_SCALE * echo_val; /* Set the echoed value */
		left_index = left_index % BUFFER_LENGTH; /* Circular buffer shift */
		/* Do the same for the right channel samples */
		echo_val = buffer_right[right_index]; /* Retrieve delayed value */
		buffer_right[right_index++] = data[i + 1]; /* Store latest sample */
		data[i + 1] = (1 - ECHO_SCALE) * data[i + 1] + ECHO_SCALE * echo_val; /* Set the echoed value */
		right_index = right_index % BUFFER_LENGTH; /* Circular buffer shift */
	}
	return length; /* Amount of data we have processed. */
}

We are expected to process the samples and write the modified values back into the same memory locations. There are two things which you must note here. First, since we have to process samples which are 16 bits (or two bytes) each, the number of samples is actually half the number of bytes. So, length / 2 is the actual number of samples we have. That explains the cast to gint16. Also, since we are handling stereo audio, successive samples alternate between left and right. So, we actually have length / 4 number of left and an equal number of right channel samples, such that data[0] is a left sample, data[1] is a right sample, data[2] is a left sample, and so on.

Finally, we register the plugin with XMMS as follows:

EffectPlugin e_plugin = {
	NULL,
	NULL,
	"Example Echo plugin",
	init,
	cleanup,
	about,
	configure,
	mod_samples,
	query};

EffectPlugin *
get_eplugin_info(void)
{
	return &e_plugin;
}

Build, install and test

The compilation and linking is just like for any other shared object file; note that we have to take care of the GTK and GLIB dependencies. You can see the Makefile for details. I also add a line to copy the plugin to $HOME/.xmms/Plugins/ after every compile. This ensures that I don't forget to put it there manually after each build. It is very useful for testing!

I strongly recommend that you initially compile your plugin with the -g option, so that you can debug it using gdb. There are many ways to use gdb to debug the plugin. I generally start gdb as gdb /usr/bin/xmms, or find the PID of a running XMMS process and use gdb --pid <PID>.

For best performance, once you are done with debugging the plugin, recompile it with -O2 or a similar optimization to allow the compiler to optimize your code. You may also wish to consider using a profiler to optimize the code further.

Further details

Now that you are ready and raring to write your own XMMS effect plugins, enjoy yourself. I'd also recommend that you go through the source code of various effect plugins available at xmms.org to get an idea of the techniques and possibilites. Also, for serious DSP advice, comp.dsp is the place to go. dspguru also has a good collection of DSP tricks and ideas. Harmony Central is an excellent place to learn more about artificial sound effects.

Comments, corrections and cribs are always welcome. Please e-mail me at the address in my bio.

Talkback: Discuss this article with The Answer Gang

Kumar Appaiah is studying to earn a B.Tech in Electrical Engineering and M.Tech in Communication Engineering (Dual-degree) at the Indian Institute of Technology Madras. He has been using GNU/Linux for the past five years, and believes that Free Software has helped him do work more neatly, quickly and efficiently.

Away Mission -- SEMTECH - The Semantic Web Technology Conference

By Howard Dyckoff

Semantics is a new, interesting, and demanding area for web technologists; it promotes the creation of ontologies and provides tools to conserve meaning within the data. The use of semantics in the development of data taxonomies is now a $2 billion per year market, and is projected to grow to over $50 billion by the year 2010. A large part of this is driven by 3-letter government agencies [ e.g., the NSA ] - unsurprisingly, several of the attendees hailed from that sector.

IT industry analysts have estimated that 35-65% of our 'untamable' system integration costs are due to semantic issues [semantic discontinuities]; this is making the overlap of semantics and web technologies an issue of critical interest to businesses on the cutting edge.

The fusion of semantics and web technologies goes back to the late 1990s and was proposed by Tim Berners-Lee. His proposal to the WC3, "A roadmap to the Semantic Web" in 1998 inspired other contributors.

In 2004, the W3C released the Resource Description Framework (RDF) and the Ontology Web Language (OWL) as W3C Recommendations. RDF is used to represent information and to exchange knowledge on the Web; in this system, the combination of Subject/Predicate/Object constitutes an RDF "Triple" where the 'object' uses URIs to encode this information and to point to a specific web location.

OWL is used to publish and share sets of terms called ontologies, which support advanced Web searching, software agents, and knowledge management. OWL and RDF are two of the main ingredients of today's Semantic Web. For more information, see this working definition of RDF.

In simple terms, RDF provides a standard and unambiguous way to make statements about resources, while OWL overlays RDF and provides the ability to make inferences about data. OWL, in turn, is used to build complex rule systems. Prior art for OWL comes from DAML, the so-called DARPA Agent Markup Language.

Semantic Web Technology

The big idea here is extend the HTML web with the addition of "meaningful" tags such that software can help us find things - and then relate those things to other things. The Semantic Web extends the utility of the World Wide Web through the use of standards, semantic markup languages, inference engines and related processing tools. That may well be the stuff of Web 3.0...

A Sunday evening primer on the Semantic Web, "Semantics in Perspective," was given by Dave McComb, President of Semantic Arts. His presentation drew from his 2005 tutorial, which is available from his company web site.

This is, admittedly, very dense material; after the opening keynote and a tech session, I felt the early onset of brain fatigue. But the final session I attended, on the challenges of classifying Disney's vast media collection, gave me much a better sense of why and how to apply these techniques. I also learned that semantic metadata can increase the value of content, especially media. For instance, that animation cel in a frame is worth more depending on 'who' painted it and 'when' and how many there were. [See this great slide pack for more].

That opening keynote with Ora Lassila [co-author of the RDF standard and Research Fellow at the Nokia Research Center] set the tone and context for the entire conference. He shared the stage with James Hendler [Director of the Joint Institute for Knowledge Discovery, University of Maryland], with whom he and Tim Berners-Lee had co-authored the seminal 2001 Scientific American paper "The Semantic Web" which discussed the concept of Semantic Webvision.

They had also independently written two of the earlier, defining articles on the concept of the Semantic Web, setting the terminology and direction for many of the researchers and developers to follow. We are not, however, talking about a long history here: Ora wrote about pervasive computing in his 1999 article, "Why your microwave oven needs to surf the WWW." Jim published his DARPA article on "Tetherless DAML and the Mobile Semantic Web" in Oct. 2000.

In an informal and interweaving style, both Ora and Jim described a timeline of Semantic Technology history and also gave themselves a report card on their pronouncements. [The slides for this keynote are availible here.]

The prehistory of the Semantic Web starts in 1990s, emphasizing metadata and the second wave of AI. Much of the discussion at that time focused on the key changes in traditional AI, its failings and its rebirth with space probes like Mars Rover and Deep Space-1.

Ora -- "We learn from the history of AI that only embedded AI was successful. Standalone AI did not succeed." He noted that reasoning tools are a 'means' and not an end. "And toward any end, content is still very important."

Ora -- [referencing the 2001 Scientific American article where he wrote about robust agents to work on the "behalf" of people] "... Sem-Tech web services are the 'plumbing' for these agents, and some trials are now underway. These must be able to handle unexpected and inconsistant data... Semantic systems will help here."

Later, Ora said, "Another thing about agents... I work for a company that makes little mobile devices... the way we use mobile devices for browsing is not the best possible paradigm... [there are, e.g., problems when driving]. So here is another case where you need systems that work for you."

Jim -- "...the miracle is not in a new magic box... it is when we start linking things together." He went on to describe SPARQL [an RDF Query Language] as a doc and data exchange mechanism. With a picture of massively interlinked systems on screen, he added "... we have both broad tech and deep technology -- we will use these things we all are building but also we must link them all together..."

Ora - "...most of what we predicted is already happening, and its happening faster than we anticipated ... the ontolgies are there but there is little linking so far... and little progress on agents..."

The closing keynote by Mills Davis was distinctly less academic. Subtitled "The Executive Guide to Billion Dollar Markets," it was a summary of a market research paper prepared by Project 10X: Semantic Wave 2006, Part 1. The presentation addressed the business value of the new semantic wave technologies and provided perspectives to help business executives seeking to use "distributed intelligence" for strategic advantage in global markets. [This conference site link provides a digest of the 55 page report and also provides a link for the full file.]

The WSDL connection

There was also much discussion about the connection between Web Services and Semantics. During the week of the SemTech conference, the shorter name that the organizers used frequently, the World Wide Web Consortium (W3C) reinforced its commitment to Web Services with continued work on SOAP, Web Services Description Language (WSDL), and addressing specs by chartering a new group to enhance WSDL with semantics to automate the composition and discovery of Web Services.

WSDL is used to describe the interface of a Web service with inputs and outputs. But WSDL alone does not provide enough information to determine automatically whether the interfaces of two arbitrary Web services are compatible. Semantic descriptions can make it easier to automatically identify services that have compatible inputs and outputs. With a description of a service for finding books and a description of a service for purchasing them, it becomes much easier to combine these into a useful composite. Similarly, machine-readable semantic descriptions facilitate the discovery of the correct Web services.

Semantic WSDL [or WSDL-S] was discussed in the context of "Large Scale Distributed Information Systems" by Amit Sheth of the University of Georgia. This talk focused on semantics for Web Services and SOAs, and offered analysis of the underlying architectures. He noted that today "... a lot of land work is required to link processes" across different business domains. This is because different businesses and public intstitutions use different vocabularies. There needs to be agreement on processes and on data and data formats. Said Sheth, "...semantic description is the way to integrate disparate services and data." He offered a semantic-aware archtitecture that used web taxonomies to link different domains.

Semantics Enrich Web Service Descriptions

WSDL 2.0 includes the hooks to allow the specification of semantic descriptions. The W3C has chartered the Semantic Annotations for Web Services Description Language (SAWSDL) Working Group to develop annotations that takes advantage of those hooks. This group will also provide a mapping of the new properties into a form compatible with the existing RDF mapping for WSDL.

Currently, more than 400 researchers and developers participate in the continuing Semantic Web Services Interest Group (SWSIG), a public forum created in 2003 to discuss the integration of the Semantic Web and Web Services. That should go some way toward putting working semantics under the covers of Web Services.

This is the working definition of RDF from the W3C XML (the "10 Points" doc):

   W3C's Resource Description Framework (RDF) is an XML text format that supports
   resource description and metadata applications, such as music playlists, photo
   collections, and bibliographies. For example, RDF might let you identify
   people in a Web photo album using information from a personal contact list; then
   your mail client could automatically start a message to those people stating that
   their photos are on the Web. Just as HTML integrated documents, images, menu
   systems, and forms applications to launch the original Web, RDF provides
   tools to integrate even more, to make the Web a little bit more into a
   Semantic Web.

The full text is available here.

And here is the web page for the W3C on the Semantic Web: http://www.w3.org/2001/sw/.

Finally, here is the Web Service Semantics - WSDL-S link: http://www.w3.org/Submission/2005/10/

Conference followups

Sem-Tech conference blog: http://semantic-conference.blogs.com/semtech06/

The business of Semantic Web Technology: http://www.dmreview.com/article_sub.cfm?articleID=1019819

Most of the session presentations and tutorials are availible for download here. If you are new to the subject matter, allow yourself some time to get through it all [well, lots of time. It's a complex subject.]

The 5th International Semantic Web Conference will be hosted in Athens, GA, USA [dates: November 5 (Sunday) - 9 (Thursday), 2006] and the 1st Annual Asian Semantic Web Conference (ASWC) which will be held in Beijing early September of 2006. SemTech 2007 will take place May 20-24 next year [but probably in another US city].

Talkback: Discuss this article with The Answer Gang

Howard Dyckoff is a long term IT professional with primary experience at Fortune 100 and 200 firms. Before his IT career, he worked for Aviation Week and Space Technology magazine and before that used to edit SkyCom, a newsletter for astronomers and rocketeers. He hails from the Republic of Brooklyn [and Polytechnic Institute] and now, after several trips to Himalayan mountain tops, resides in the SF Bay Area with a large book collection and several pet rocks.

The Geekword Puzzle

By Ben Okopnik

Introduction

This month, in the spirit of "Making Linux a Little More Fun", I decided to try my hand at a Linux crossword puzzle. This presented a number of interesting challenges: for one thing, using a crossword compiler requires having a large list of words as well as a grid that matches that list to some degree (i.e., word length, common letters in appropriate positions, etc.) The Answer Gang kicked in several hundred Linux- and computer-related words; unfortunately, it wasn't quite enough for the hungry compiler beast - even though I spent a lot of time bending my brain around various grid designs. Perhaps that approach will pay off later; in the meantime, I decided to design a crossword manually, since I'd heard that some people consider this a fun activity.

...

The random twitching and the screaming in the middle of the night have abated somewhat by now, and the doctor assures me that, with a year or two of therapy, the other effects will decrease to a manageable level as well.

Originally, I had considered writing a bit of CGI to allow readers to solve the puzzle on-line, but rejected the thought very quickly; one of the most important guidelines here at LG is keeping the content static, and thus available to everyone. I also couldn't just crank out a nice JavaScript-enabled version, since those of our readers who use Lynx would be left out (other text-mode browsers - links, elinks, and w3m all render tables quite happily, although they will not allow puzzle completion on-line - you'd have to print the page and write your answers.) In fact, given the way that Lynx treats tables, it wouldn't even look like a crossword. On the other hand, to those who have a modern graphical browser, a simple ASCII-art layout would look - well, rather crude. Hmm, what to do?

I decided to get the best of both worlds by doing both (my wife and I share a common characteristic: when offered a choice of two good alternatives, as in "do you want to do X or Y?", we will usually respond with "Yes".) However, since I'm too... let's just call it efficient... to spend time laying out two different puzzles, I wrote a Perl script that reads a single input file which creates both. Thus, the Javascript puzzle is what you see below; the "printable version" link provides access to the (nearly) pure ASCII layout that is viewable with any browser. This solution satisfies my persnickety sense of The Right Thing, and should provide access to everyone (anyone left out for some reason I may have missed should contact me and let me know the details, and I'll do my best to include them as well.)

In any case, the result of all the cogitation, agitation, and meditation is presented here for your enjoyment; I hope you have as much fun... yeah, that's the word... solving it as I had designing it!

[ The solution to the current puzzle will be provided in the next month's issue. ]

Talkback: Discuss this article with The Answer Gang

Ben is the Editor-in-Chief for Linux Gazette and a member of The Answer Gang.

Ben was born in Moscow, Russia in 1962. He became interested in electricity at the tender age of six, promptly demonstrated it by sticking a fork into a socket and starting a fire, and has been falling down technological mineshafts ever since. He has been working with computers since the Elder Days, when they had to be built by soldering parts onto printed circuit boards and programs had to fit into 4k of memory. He would gladly pay good money to any psychologist who can cure him of the recurrent nightmares.

His subsequent experiences include creating software in nearly a dozen languages, network and database maintenance during the approach of a hurricane, and writing articles for publications ranging from sailing magazines to technological journals. After a seven-year Atlantic/Caribbean cruise under sail and passages up and down the East coast of the US, he is currently anchored in St. Augustine, Florida. He works as a technical instructor for Sun Microsystems and a private Open Source consultant/Web developer. His current set of hobbies includes flying, yoga, martial arts, motorcycles, writing, and Roman history; his Palm Pilot is crammed full of alarms, many of which contain exclamation points.

He has been working with Linux since 1997, and credits it with his complete loss of interest in waging nuclear warfare on parts of the Pacific Northwest.

Low-Fat Linux - Now with Less Cruft!

By Ben Okopnik

OK, I admit it - there's no such thing as "Low-Fat Linux" (unless somebody just came out with it in order to spite me.) These days, the average installation is pretty big - and many of the most popular programs take up huge swaths of space. Even with a 60GB hard drive like the one in my laptop, you still have to pay close attention lest it become Godzilla, The Eater Of All Available Space (and time, too - but we're not talking about Quake III or Xbubble at the moment.)

So there you are - going along, innocently doing your work (or playing your games), installing delicious little packages (Yum!) that sound interesting or that your friends recommended... and then, one day, you cast a casual glance at your system, and a slight chill runs down your spine.

ben@Fenrir:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3             958M  362M  548M  40% /
/dev/hda5              33G   25G  6.5G  79% /home
/dev/hda1             5.5G  4.8G  521M  91% /home/ben/Music
/dev/hda6              13G   12G  389M  97% /usr
/dev/hda7             1.9G  699M  1.1G  40% /var
/dev/hda8              46M  9.1M   35M  21% /boot
tmpfs                 253M  4.0K  253M   1% /dev/shm

Yikes! Just look at that '/usr' partition!

And now comes the dreadful awareness that you're going to have to delete something. In fact, given the size of the partition, and the number of megabytes that it takes to make up even one percent of it, you're going to have to delete a lot of that something. How the heck are you supposed to figure out what to delete???

Let the panic begin.

After a few calming exercises - a little yoga, a lot of deep breathing, several gin-and-tonics, etc. - you decide on a rational approach (at least if you didn't go for the gin-and-tonics. Otherwise, you'll be doing it later - and that hangover headache will not be working in your favor.) First, you clean out a few games - the ones you don't play much if at all. However, you quickly discover that the interesting games are all BIG - and the simple, boring ones don't do much to fill the trash bucket with megabytes. Kilobytes, more likely. Run "apt-get clean" to remove the cached installation packages in Debian? Oh, that helped... in the '/var' partition, that is. Not in '/usr', where we need it most desperately.

OK, well... let's give it some thought. Oh yeah... all those installed packages - right. It's going to be quite the long and tedious process, but it does indeed pay off: one of the effects of free software is that many people tend to install things "just to check them out", get distracted, and - well, cruft, like entropy, always increases. It's just a law of physics. However, although we haven't yet found a way to reverse the latter, the former can indeed be done - with a bunch of effort. So, on my Debian laptop, it all looks like this (at least post-cleanup):

# Count installed Debian packages
ben@Fenrir:~$ dlocate -l|grep '^ii'|wc -l
2133
ben@Fenrir:~$ dlocate -l|grep '^ii'|less
ii  9menu                      1.8-1.1                    Creates X menus from the shell
ii  a2ps                       4.13b.dfsg.1-1             GNU a2ps - 'Anything to PostScript' converter and pretty-printer
ii  aalib-bin                  1.4p5-25                   sample programs using aalib
ii  aalib1                     1.4p5-28                   ascii art library - transitional package
ii  abcde                      2.3.99.5-1                 A Better CD Encoder
ii  accesscontrol              1.0-2                      control panel that activates handicapped accessibility features
ii  acpi                       0.09-1                     displays information on ACPI devices
ii  acpid                      1.0.4-5                    Utilities for using ACPI power management
ii  adduser                    3.87                       Add and remove users and groups

[ the rest of long list elided ]

Over 2,000 packages... well, I could mutter some sort of a justification for most of them, anyway. The ones that I couldn't find a reason to keep went away - configuration and all. That cleared away a significant chunk - but not enough to "grow on". It was similar to giving your house a quick little once-over with a whisk broom and ignoring the dust monsters under the bed, the horrible mess in the attic ("It's not a problem - nobody can see it!"), and the biological experiment that used to be a half a pound of tuna salad in your refrigerator: just fine for a quick visit from your neighbor, but it would all collapse in a second if your snoopy aunt Marge came over. Heck, the stuff in the attic will just collapse onto your head if you let it go a couple more weeks even without Auntie crawling around up there. It really is better to take care of it now, before the ambulances, the embarrassing articles in the local newspaper ("Local Man Buried In His Own Trash: Mother Dies Of Shame"), and your friends' sarcastic laughter. Indeed, Drastic Measures are now called for.

"More drastic measures?", you say, aghast. "What could be more drastic than giving up my programs, installed by the sweat of my brow (even if I have no idea what they do and have never used them in the three years that they've been installed)? What do you want, blood???" Nah, nothing like that; I'm usually happy with fine wine and chocolate. Maybe a side of blood oranges and Rainier cherries, if you really insist.

However, my point is that system cruft accumulates almost invisibly - and costs you little (other than a little study and searching) to remove. If you've had your system installed for any significant length of time, and have done a lot of updates, there are a number of programs which don't remove their previous versions before installation - notably, programming languages. I'm a big fan of Perl, and have been installing every new version of it, as well as many of its modules, as long as I've been using Linux. Chances are that you, too - given how many programs on the average Linux systems require Perl - have done the same thing. Ditto Python. Ditto the GCC compiler. Ditto your favorite language - or perhaps some language that you've never even heard of but is necessary for compiling your favorite applications. In other words, you need to take a second, differently-informed look at that package listing: even if you don't know much about programming, you can still spot multiple installed versions of languages. Of course, if trying to uninstall one produces a message warning you of current dependencies, and you want to keep all the programs that depend on that specific version of a language, then you need to hold on to it - but usually, the latest version is enough. If you're careful, and know what you're doing, you can look at, e.g., the '/usr/{share,lib}/{perl,python,ruby}*' and '/usr/local/{share,lib}/{perl,python,ruby}*' directories and see which ones are redundant - or perhaps can become so if you reinstall the modules in the newest versions of those languages. I found, for example, that removing the old Perl, Python, and Ruby versions from those locations freed up several hundred megabytes - at no cost beyond reinstalling a few packages. Admittedly, this did take some time and careful querying of the package database.

OK, now that we're on the hunt - what's next? Well, one of the traditional ways to chomp arbitrary but huge amounts of space on your machine - not that this is specifically Linux-related - has always been the installation of fonts. Great Ghu knows why we need so many: after all, despite our best imagination, few of us are actually graphic designers. Most of us, though, end up installing lots and lots of them... and then sneaking in a few more, just to be sure. Many ([cough] several hundred) of these can all be some indistinguishably-similar version of Arial or Times New Roman or Tekton - but, hey, who knows? We just might feel like this one instead of that one someday, and then what would we do if it wasn't there?...

"Off", cried the Red Queen, "with their heads!" And rightly so. At least if they're unnecessary fonts.

ben@Fenrir:~$ xlsfonts|wc -l
18673

Oh, dear. That is just way, way too many fonts. Sure, I like them as much as anyone else... but really. If I estimate - generously - the number of fonts that I actually use, explicitly and by name, I might come up with a dozen. OK, two dozen. If I want to get really fancy with an OpenOffice document while making up a business brochure, I might like to have a field of, say, fifty or a hundred to look at... but over eighteen thousand? I think not.

# Count the number of available fonts
ben@Fenrir:~$ apt-cache search xfonts|egrep -c '^xfonts|^ttf'
86
# Check which ones are installed on my system
ben@Fenrir:~$ dlocate -l |grep '^ii  xfonts'
ii  xfonts-100dpi              1.0.0-2                    100 dpi fonts for X
ii  xfonts-75dpi               1.0.0-2                    100 dpi fonts for X
ii  xfonts-a12k12              1-8                        12-dot Kanji & ASCII fonts for X
ii  xfonts-artwiz              1.3-5                      x11 fonts created by Artwiz, TigerT, and Daniel Erat
ii  xfonts-base                1.0.0-3                    standard fonts for X
ii  xfonts-base-transcoded     6.9.0.dfsg.1-6             standard fonts for X (transcoded from ISO 10646-1)
ii  xfonts-bolkhov-cp1251-75dp 1.1.20001007-6             75 dpi CP1251 encoded Cyrillic fonts for X (Cyr-RFX collection)
ii  xfonts-bolkhov-cp1251-misc 1.1.20001007-6             Character-cell CP1251 encoded Cyrillic fonts for X (Cyr-RFX collection)
ii  xfonts-bolkhov-koi8r-75dpi 1.1.20001007-6             75 dpi KOI8-R encoded Cyrillic fonts for X (Cyr-RFX collection)
ii  xfonts-bolkhov-koi8r-misc  1.1.20001007-6             Character-cell KOI8-R encoded Cyrillic fonts for X (Cyr-RFX collection)
ii  xfonts-bolkhov-koi8u-75dpi 1.1.20001007-6             75 dpi KOI8-U encoded Cyrillic fonts for X (Cyr-RFX collection)
ii  xfonts-bolkhov-koi8u-misc  1.1.20001007-6             Character-cell KOI8-U encoded Cyrillic fonts for X (Cyr-RFX collection)
ii  xfonts-cronyx-cp1251-100dp 2.3.8-6                    100 dpi CP1251 encoded Cyrillic fonts for X (Cronyx collection)
ii  xfonts-cronyx-cp1251-75dpi 2.3.8-6                    75 dpi CP1251 encoded Cyrillic fonts for X (Cronyx collection)
ii  xfonts-cronyx-cp1251-misc  2.3.8-6                    Character-cell CP1251 encoded Cyrillic fonts for X (Cronyx collection)
ii  xfonts-cronyx-koi8r-100dpi 2.3.8-6                    100 dpi KOI8-R encoded Cyrillic fonts for X (Cronyx collection)
ii  xfonts-cronyx-koi8r-75dpi  2.3.8-6                    75 dpi KOI8-R encoded Cyrillic fonts for X (Cronyx collection)
ii  xfonts-cronyx-koi8r-misc   2.3.8-6                    Character-cell KOI8-R encoded Cyrillic fonts for X (Cronyx collection)

[ Actual complete list snipped in embarassment ]

Well, lots of them, anyway.

A careful review and removal of these - paying attention to the dependency warnings, of course - helped quite a lot, another couple of hundred megabytes or so. What helped even more was a careful examination of the actual font directories - it turns out that X.org (as contrasted against XFree86) decided to rationalize their font structure by combining it with the traditional system location for fonts, '/usr/share/fonts' instead of the old '/usr/X11R6/lib/X11/fonts'. However, the old font directory was still there - and still packed with a hundred-plus MB worth of files! A careful comparison of what was there (in fact, I backed up the entire directory temporarily to see if there were any bad effects) with the new '/usr/share/fonts/X11' structure helped recover a huge bunch of space. I did end up keeping quite a lot more than a hundred fonts, by the way - I depend on Unicode fonts for quite a lot of my work, and I need a good selection of Russian and Japanese fonts in addition to the base font package that comes with X - but at this point, I'm down to a much more reasonable 5000+ fonts (around 3000 unique ones), including all the TrueType fonts that I like.

A quick digression into details may be useful here: I suggest taking a good look at Thomas Adam's "How Fonts Interact with the X Server and X Clients" article to gain a bit of understanding of how fonts work. Taking him at his word ("...font servers: you don't need them."), I carefully checked all my font directories for the presence of "fonts.alias", "fonts.dir", and, in the case of TrueType fonts, "fonts.scale" files - all easily created with the help of the 'ttmkfdir' and 'mkfontdir' utilities. I then confirmed that all my font directories were listed in my '/etc/X11/xorg.conf', in the "Files" section, and - Voila! - I had all my fonts available without running the 'xfs' or 'xfstt' servers, or having 'unix/:7100' or 'unix/:7101' (the server ports) listed in that section or in the 'xset q' report. Not that those servers took up a huge amount of space, but X now seems to start a bit faster than it used to, at least in my perception.

So, was all that worth it? You might say so.

ben@Fenrir:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3             958M  362M  548M  40% /
/dev/hda5              33G   25G  6.5G  79% /home
/dev/hda1             5.5G  4.8G  521M  91% /home/ben/Music
/dev/hda6              13G  9.9G  2.3G  82% /usr
/dev/hda7             1.9G  700M  1.1G  40% /var
/dev/hda8              46M  9.1M   35M  21% /boot
tmpfs                 253M  4.0K  253M   1% /dev/shm

Sure, that took a couple of hours of work - but at this point, I know the state of my system, I've got a couple of gigabytes worth of space in '/usr' again, and I'm more aware of what kinds of things eat up space (and what should be checked once in a while.) In my opinion, that's definitely a gain - in knowledge and understanding, as well as in available disk space.

Talkback: Discuss this article with The Answer Gang

Ben is the Editor-in-Chief for Linux Gazette and a member of The Answer Gang.

Ben was born in Moscow, Russia in 1962. He became interested in electricity at the tender age of six, promptly demonstrated it by sticking a fork into a socket and starting a fire, and has been falling down technological mineshafts ever since. He has been working with computers since the Elder Days, when they had to be built by soldering parts onto printed circuit boards and programs had to fit into 4k of memory. He would gladly pay good money to any psychologist who can cure him of the recurrent nightmares.

His subsequent experiences include creating software in nearly a dozen languages, network and database maintenance during the approach of a hurricane, and writing articles for publications ranging from sailing magazines to technological journals. After a seven-year Atlantic/Caribbean cruise under sail and passages up and down the East coast of the US, he is currently anchored in St. Augustine, Florida. He works as a technical instructor for Sun Microsystems and a private Open Source consultant/Web developer. His current set of hobbies includes flying, yoga, martial arts, motorcycles, writing, and Roman history; his Palm Pilot is crammed full of alarms, many of which contain exclamation points.

He has been working with Linux since 1997, and credits it with his complete loss of interest in waging nuclear warfare on parts of the Pacific Northwest.

A Prisoner of Windows

By Lew Pitcher

Those of us who use Linux desktops sometimes forget that most of our peers do not have our luxuries. At times, we find ourselves forced to devolve, and use a Windows desktop to do our work. When we do, we pine for the vast variety of tools we had under Linux and curse the clumsiness of the software prison that we are locked into.

But there is hope for Prisoners of Windows. If our Microsoft Windows environment is network-enabled, and if we can connect to a suitably configured network-enabled Linux box, we can still work on our platform of choice. So, how do we free ourselves? By giving ourselves secure remote access to Linux, both at the command line and at the GUI level with a combination of standard Linux tools and a couple of Open Source applications for Windows.

My goal is to show you how to let a Windows desktop user log into and run both command-line and GUI applications on a remote Linux system, with Windows handling the presentation (and only the presentation). I will show you how to set up OpenSSH on a Linux system in order to provide secure access and facilitate GUI interaction, and how to configure and use an Open Source SSH client (PuTTY) and an Open Source X server (Xming) on Windows. With these tools, Windows captives can securely and easily escape to a Linux system and work on it remotely as if it were the only system they were on.

The escape plan: Setting up the OpenSSH Server

Before you can escape, you have to prepare a "safe haven" on a Linux system. You will need to set up an OpenSSH server on a Linux system and make it reachable from the outside world.

As most Linux distributions come with (or make available) a native version of OpenSSH, I will not cover the distribution-specific details of installation from native packages, or the tedious technical details of installation from source. Instead, I will assume that you already have located and installed the appropriate package, and are ready to configure it.

Configuring the OpenSSH Server

You can find the default configuration file for the OpenSSH server in /etc/ssh/sshd_config. This text file contains three types of text: blank lines, comments (which start with a #) and lines of keyword/argument pairs which control the behaviour of the SSH server. You only need concern yourself with the keyword/argument lines, since 'sshd' ignores comments and blank lines.

Even though the default values for the 'sshd' parameters (the values assumed when no options are specified) are usually adequate for general SSH use, I'd like you to explicitly set the values of four of the parameters to ensure proper access.

Specifically, set:

• AllowTcpForwarding to yes
• X11Forwarding to yes
• X11DisplayOffset to 10
• X11UseLocalHost to yes

AllowTcpForwarding

controls whether or not the SSH daemon will permit generic TCP forwarding. This feature permits SSH clients to direct selected network traffic through the SSH tunnel so that it moves securely from client to server (or vice versa). While this parameter defaults to "yes", you may find that the default sshd_config explicitly sets it to "no", and since we wish to ensure that TCP forwarding is permitted, you need to change it.

X11Forwarding

controls whether or not the SSH daemon will explicitly handle SSH forwarding of X11 data streams. This parameter defaults to "no" and you should explicitly set it to "yes" in sshd_config in order to permit your Windows detainee to easily interact with Linux GUI applications. Note that, with AllowTcpForwarding enabled, X11Forwarding is a convenience, and is not absolutely necessary for X11 tunnelling.

X11DisplayOffset

specifies the starting DISPLAY number for SSH-transported X11 datastreams. This parameter should be set to a value higher than the highest 'real' (locally attached) X11 display on the Linux system in order to avoid interfering with the local GUI users. The default value for this parameter is 10, but you may wish to assign a larger value if your Linux system supports a large number of X users.

X11UseLocalHost

controls whether SSH should "bind" X11 forwarding to the local loopback address or to the less secure wildcard address. If bound to the local loopback address, some older X11 clients may not function properly. The default for this parameter is "yes" (which is the value I'd like you to explicitly set it to), but your initial configuration file may have a different value.

Of the above four parameters, two are typically unnecessary (since their default values are sane anyway), and one of the other two may be redundant. Specifically, X11DisplayOffset and X11UseLocalhost default to sane values, and you don't need to alter them in sshd_config. Also, AllowTcpForwarding and X11Forwarding offer clients similar abilities; if AllowTcpForwarding is already specified, then it is simple (but not obvious) for a client to perform X11 forwarding without the X11Forwarding option. If X11Forwarding alone has been set, our incoming clients can forward X11 sessions, but not other types of data streams. For your prisoner of Windows, you want to allow access to all network data streams (hence the explicit AllowTcpForwarding setting), and painlessly permit him to forward X11 specifically (hence the explicit X11Forwarding setting).

Once you've made the appropriate changes to your /etc/ssh/sshd_conf file (see Figure 1), save your editing efforts and continue to the next step: generating your SSH server keys.

Generate your Server Keys

Before you can use the SSH server, you must generate encryption keys for it. Some Linux distributions arrange for the first execution of the SSH server to build these keys, but some do not. In the next step, you will explicitly build three sets of keys (one set for each method of conversation that SSH understands), and place these key sets in the standard files.

As root, first execute the ssh-keygen command to build your rsa1 key into the /etc/ssh/ssh_host_key file:

/usr/bin/ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_key -N ''

Next, again execute the ssh-keygen command, but this time to build your dsa key into file /etc/ssh/ssh_host_dsa_key:

/usr/bin/ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ''

Finally, execute the ssh-keygen command for a third time, this time to build your rsa key into file /etc/ssh/ssh_host_rsa_key:

/usr/bin/ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''

With these commands, you have given your SSH daemon both ears and a voice through which your Windows captive can communicate with the Linux environment.

Start the OpenSSH server

Finally, it is time to start your OpenSSH server, which will use the configuration and key files you've prepared to conduct conversations with the outside world. If your system starts the SSH server automatically, you can just "signal" it to have it reread its configuration. To do this, as root, you

/bin/kill SIGHUP `cat /var/run/sshd.pid`

/usr/bin/sshd

[ In most distributions, launching standard system daemons such as SSH is best done via the scripts in '/etc/init.d'. The above two tasks, for example, would be handled by '/etc/init.d/ssh restart' and '/etc/init.d/ssh start' executed as root. -- Ben ]

Now you have completed the extra Linux configuration, created a secure door to your Linux "safe haven" (did you remember to adjust your firewall to permit SSH connections?), and have taken the first step to freeing that Prisoner of Windows from his oppressive confines. Of course, he will still require an account on the Linux system, which you will need to create with the standard tools and methods of your distribution. Initially, he will log into the Linux system using his assigned username and password; after that, he will have the option of logging in through the secure facilities of SSH.

A foot out the door - Linux Commandline with PuTTY

Fully locked-down Windows systems, where you can't even run unapproved programs from floppy disk or USB "thumb drive" memory stick, are the worst prisons of all. You can't do anything about these dungeons without the co-operation of the Windows system administrator. Let's hope that your Windows captive is not locked in one of these "solitary confinement" cells, and instead has a little latitude in his actions.

The first step to freedom, from the Windows side, is to install the "PuTTY" SSH client application. "PuTTY" is a Windows GUI application written and maintained by Simon Tatham, and released (both in executable form, and in source code) under a BSD-like Open Source license. When installed, it takes about 2800K of disk space, and creates a couple of (not absolutely required) Windows Registry entries. The 6 programs in the PuTTY suite provide the client tools necessary for our captive to escape to our Linux "safe haven".

PUTTY.EXE	This is the SSH client program used for "command-line" access to outside systems, and is a secure replacement for the telnet client command.
PUTTYGEN.EXE	This program builds client "public key" keys, for use in identifying the client to the ssh server.
PAGEANT.EXE	This tool is a "key agent" that allows the Windows user to "make available" selected keys on an ongoing basis to PuTTY applications. This is a convenience (to reduce the number of times a user must enter key information), and not a requirement.
PSCP.EXE	An SSH replacement for the rcp client command, and allows secure copying.
PSFTP.EXE	An SSH replacement for the ftp client command.
PLINK.EXE	This tool is an SSH client program (like PUTTY.EXE) suitable for use in automated processes (like .BAT and .CMD files). Other GUI programs (like Xming) also use PLINK.EXE to establish their own SSH connections to the target systems.

Your first task will be to configure PuTTY to connect to the Linux system. This will be a simple connection that depends on Linux userid and password confirmation; it will not (yet) use the power of SSH public key authorization to give you immediate access to the Linux system. We'll save that little flourish for later.

So, from the Windows Start -> All Programs -> Putty menu, you now run PuTTY. The PUTTY.EXE program will start, and present you with a PuTTY Configuration window. Because this is your first time running PuTTY, and you have not yet set any configuration information, the window will be mostly empty. On the left hand side of the window, you will see a tree of PuTTY options, and on the right, the settings for the "Session" option. It is here that you provide the basic information necessary to get out of Windows and into your Linux commandline.

In the Host Name field, you enter the DNS name or IP address of your Linux system. If you intend to return to this system (and you likely do), you can save the session information by entering a meaningful name in the Saved Sessions entry box and selecting the Save button (see Figures 2a-2c). From this point on, you can launch PuTTY, select and Load the Linux system session information from the list of Saved Sessions, and then select the Open button to immediately connect to the Linux system.

Because PuTTY has never conversed with your Linux system before, it will show you the SSH key that the "Safe Haven" OpenSSH server provided (see Figure 3a), and ask you if you trust it. Answer "Yes" to complete the SSH handshake with the Linux system; if you answer "No" instead, then PuTTY will continue with the handshake, but will not remember that you trust the Linux system, and if you answer "Cancel", Putty will abandon the connection altogether. Once you've completed the handshake, it is now just a simple matter of completing the text-based login with your Linux userid and password (Figures 3b-3d), and you are done; the world of your Linux system awaits you, from a character-based interface (like in Figure 3e).

Free as a Bird - Linux GUI with Xming

But a character-based interface is just "a shadow on the wall" to our GUI-acquainted eyes. While it is possible to use Linux this way, you likely prefer a less limited interaction, and that means X. Your Windows captive will better enjoy his freedom if you install the "Xming" X server, and that is your next step.

Xming is a port of the X.org X11 server to the Windows environment, currently maintained by Colin Harrison who took over from Alexander Gottwald. The latest release of Xming (at the writing of this article) is based on the new X.org X11R6.9 X server, and (like PuTTY) is released under an Open Source licence.

Unlike PuTTY, however, the Xming install comes in multiple parts. There are two different X servers (one that uses OpenGL for rendering, and one that uses the much slower Mesa renderer) and one font package. You will need to install one of the two X servers, and some of the fonts from the font package. The installations are very straightforward, consisting of a familiar graphical installation wizard that takes care of all the mechanics of the install. Configuration (at this point) is not an issue; the basic Xming services that the install builds will suit our needs perfectly. With the OpenGL server and all the fonts (both obligatory and optional) installed, Xming takes about 45Mb of disk space (and a few optional registry entries) to install.

Once you have the X server and fonts installed, you will start a "rootless" Xming server by traversing the Windows Start -> All Programs -> Xming menu, and selecting the Xming option. This will start the X server, and leave you with an Xming icon (see Figure 4) in your toolbar. Xming does not come with the xauth or xhost commands (although they are available through an add-on package), and the X server does not, by default, permit unauthenticated remote applications to connect to and use the display. You could add in some of the add-on toolset in order to get the X authority tools, but instead, I'll show you how to use the X11 forwarding capabilities of your OpenSSH server and PuTTY client to cause your remote X11 applications to appear as local applications to Xming. So, back to PuTTY we go.

When you launch PUTTY.EXE, you are always first presented with a configuration window. When you first connected to your Linux "safe haven", you (hopefully) saved the PuTTY configuration for that connection. You now want to Load that Saved Session, and alter it a bit. If you scroll down the "Category" list on the left hand side of the configuration window, you will come to a leaf titled "X11". Select this leaf and alter its settings (see Figure 5a) to Enable X11 Forwarding. Then, scroll back to the Sessions leaf, and Save the altered definition before you Open the session to your Linux system.

When you do open the session, you will find that, on the Linux side of the connection, the $DISPLAY environment variable is now set (see Figure 5b) to a display on the Linux system. This display doesn't actually physically exist; it is one of the X11 tunnels that you permitted with your OpenSSH X11Forwarding and X11DisplayOffset settings. Your Linux X client applications will interact with OpenSSH instead of a real X server, and OpenSSH (on Linux) will relay the X protocol messages to PuTTY (on Windows), which will then pass them on to your Xming X server running on Windows. The X server will respond in reverse, passing data to your PuTTY client, which passes it back to OpenSSH to be finally handed over to the X client application. It is a round-about route to be certain, but it does provide several advantages:

1. it provides a layer of security around your GUI traffic,
2. it gives your X client applications an assured route to your X server, guaranteeing that they will not be stopped by firewall rules or routing problems that may inadvertently block the X protocol traffic, and
3. it eliminates the need to use the X authorization tools (like xauth or xhost) because, to your X server, all the X protocol traffic is local traffic.

As Figure 5c shows, from this PuTTY and Xming setup, you can run even the most sophisticated X application. Truly, almost any X application that is available to you in Linux can be run this way. Only those applications that require services not implemented in the Xming server (such as OpenGL when you use the Mesa version of Xming) cannot be run this way.

A Person In Danger

With this, I've met my goal and given you the tools to free your Prisoner of Windows. Now, all the facilities of Linux are open to him, command-line and GUI tools alike. He no longer needs to curse the confines of his local working environment, and can luxuriate in the freedom of the remote Linux system.

The I Ching says "A person in danger should not try to escape at one stroke. He should first calmly hold his own, then be satisfied with small gains, which will come by creative adaptations." We've given our Prisoner of Windows a small gain achieved by a creative adaptation. The next steps you take will lead to his complete escape.

Resources

OpenSSH SSH server (Linux)
http://www.openssh.com/

PuTTY SSH client (Windows)
http://www.chiark.greenend.org.uk/~sgtatham/putty/

Xming X server (Windows)
http://www.straightrunning.com/XmingNotes/

X Foundation (X server reference site)
http://www.x.org/

Remote X Apps mini-HOWTO (Linux Documentation Project)
http://www.tldp.org/HOWTO/text/Remote-X-Apps

Usenet Newsgroup for SSH matters (the authors of PuTTY sometimes hang out here)
news:comp.security.ssh

Usenet Newsgroup for general X Window system matters
news:comp.windows.x

Usenet Newsgroup for Linux X Window system matters
news:comp.os.linux.x

Talkback: Discuss this article with The Answer Gang

Canadian by birth, and living in Brampton, Ontario, I am a career techie working at a major Canadian bank. For over 25 years, I've programmed on all sorts of systems, from Z80 CP/M up to OS/390. Primarily, I develop OS/390 MVS applications for banking services, and have incorporated Linux into my development environment.

Issues In Concurrent Server Design on Linux Systems - Part I

By Amit Saha

What is a Concurrent Server

In the simplest terms, a server is a process which waits for incoming client connections. The server can be a simple echo server, echoing back what the client sends to it - or it can be a complex program such as a FTP or web server handling thousands of connection requests from clients. A concurrent server is one that can process more than one client connection at the same time.

Implementing a Concurrent Server

Stevens suggests the following choices for the type of process control to use while designing concurrent Linux servers:

• One Fork Per Client Request
• Prefork with child calling accept
• Prefork with file locking to protect accept
• Prefork with thread mutex locking to protect accept
• Prefork with parent passing socket descriptor to child
• Create one thread per client request
• Prethreaded with mutex locking to protect accept
• Prethreaded with main thread calling accept

We shall now take a brief look at each of the above techniques by implementing the ideas to design TCP-based concurrent servers.

A. TCP Concurrent Server, One Child per Client

This is by far the simplest method of concurrent server design. The server process forks a child for each client connecting to it. The child forked then takes care of the corresponding client connection while the server waits for other client connections. In this issue, we shall stress on this particular method using it in to design an "enhanced" file transfer application.

B. TCP Preforked Server, No Locking Around Accept

This technique of server design uses a technique called preforking. The server launches a number of child processes when it starts, and the children service the clients as each client connection arrives. A few things worth noticing here are:

• New clients can be handled without the cost of a fork by the server
• The server has to guess the number of children to prefork when it starts

If the number of clients being serviced at any time exceeds the number of children, the additional clients will experience a service time delay (they will not be ignored as long as the backlog in listen() is less than or equal to the additional clients).

The preforking mechanism in a preforked server has been reproduced from Stevens for your reference:

pid_t child_make(int i,int listenfd,int addrlen)
{
	pid_t pid;
	if((pid=fork())>0)
		rteurn(pid);	//parent
	child_main(i,listenfd,addrlen); //never returns

}

void child_main(int i,int listenfd,int addrlen)
{
	int connfd;
	socklen_t clilen;
	struct sockaddr *cliaddr;
	cliaddr=malloc(addrlen);
	printf("Child %ld starting\n",(long)getpid());
	for(;;)
		{
		clilen=addrlen;
		connfd=accept(listenfd,cliaddr,&clilen);
		/*
		perform
		processing jobs
		*/
		}
}

A point of interest in this regard is that multiple processes are calling accept on the same listening descriptor. How this is done and other associated issues are going to be dealt in subsequent parts of this series.

C. TCP Preforked Server, File Locking Around Accept

The implementation just described works fine with Berkeley-derived kernels. However, System V kernels, which implement accept() as a library function may not allow this. The solution lies in placing some kind of a "lock" around the call to accept, so that only one process at a time is blocked in the call to accept. The remaining children will be blocked trying to obtain the lock.

There are various ways to provide this locking mechanism (we will address these in subsequent articles.) The only change to the code previously decsribed in section B is to call the locking function before the children are created, and to release the lock in the child_main() function before performing the processing tasks.

D. TCP Preforked Server, Descriptor Passing

Another possible modification to the preforking type servers is having only the parent call accept and the "pass" the connected socket to one child. A few points worth noting here are:

• This gets around the possible need of file locking around the call to accept in all the children, but requires a form of descriptor passing to the children
• This complicates the code somewhat as the parent has to keep track of which children are busy and which are free to pass a new socket to a free child

There are two more design mechanisms possible - TCP Prethreaded Server, per-Thread Accept and TCP Prethreaded Server, Main Thread Accept - where instead of forking new child processes, we use threads to achieve concurrency. The rest is similar to the design mechanism just described.

We shall now introduce ourselves to two famous problems associated with scalability of Linux servers.

The Thundering Herd Problem

Network servers that use TCP/IP to communicate with their clients are rapidly growing in capability. A service may elect to create multiple threads or processes to wait for increasing numbers of concurrent incoming connections. By pre-creating these multiple threads, a network server can handle connections and requests at a faster rate than with a single thread.

In Linux, when multiple threads call accept() on the same TCP socket, they get put on the same wait queue, waiting for an incoming connection to wake them up.

In the Linux 2.2.9 kernel and earlier, when an incoming TCP connection is accepted, the wake_up_interruptible() function is invoked to awaken waiting threads. This function walks the socket's wait queue and awakens everybody. All but one of the threads, however, will put themselves back on the wait queue to wait for the next connection. This unnecessary awakening is commonly referred to as a "thundering herd" problem and creates scalability problems for network server applications.

For more about the "Thundering Herd problem", see this document.

The C10K Problem

'C' stands for 'Client' and 'K' stands for 1000. Dan Kegel's contention is that the web is a big place now, and web servers have to be able to handle ten thousand clients simultaneously; the solution to this problem involves finding the optimal configuration, both at Operating System level and network application level, which helps minimize the latency while serving on the order of tens of thousands of clients. In his 'C10K' paper, Dan suggests the following optimizations:

• Serve many clients with each thread, and use nonblocking I/O and level-triggered readiness notification
• Serve many clients with each thread, and use nonblocking I/O and readiness change notification
• Serve many clients with each server thread, and use asynchronous I/O
• serve one client with each server thread, and use blocking I/O
• Build the server code into the kernel

For more details, see the above link.

Now that we've acquainted ourselves briefly with the various design alternatives, we are going to take up the simplest among the design techniques - that is, TCP Concurrent Server, One Child per client. Then we are going to see how we can design an "enhanced" file transfer program using this principle. The other design techniques shall be described with an effective application for each in subsequent issues.

Enhanced Server Design Method I

This is by far the simplest method of concurrent server design; the server process forks a child for each client connecting to it. The child then takes care of the corresponding client connection while the server waits for other client connections. The complete source code can be found here. In this example, the server just displays any message sent by the client. Let us now go through a part of the server code where the client connection is accepted:

1. while(1){
2. if ((newfd = accept(listener, (struct sockaddr *)&remoteaddr,&addrlen)) == -1)
3.          {
4. 	  perror("accept");
5.          }
6. else
7.          {
8.	//fork a new child whenever we have a new client connection

9.	pid=fork();
10.     if(pid==0){

	//handle the client

11.	printf("new connection from %s on socket %d\n", inet_ntoa(remoteaddr.sin_addr),newfd);
12.     printf("Recieved %d bytes from the client",read(newfd,buf,sizeof(buf)));
13.     printf("%s",buf);

	//do any other tasks as needed
14.	exit(0);
        }
}

Examination of the server code reveals that if a client connection is succesfully accepted (line 6), then the server forks a new child (line 9). The child process then does the other jobs using the client connection (lines 11-13). After this the child exits.

Meanwhile the parent process goes back into the infinite loop (line 1) and waits for incoming client connections (line 2).

Let us now use this server design method to do something useful. First, we'll outline our goal - i.e., building an Enhanced File Transfer Application - clearly:

• Design a simple File Transfer client & Server Program. This effectively creates a single connection between the client and the server for each file transfer operation. It can be very easily observed that the file transfer operation consumes more time as the file size increases even if the client and the server is on the same machine.
• Enhance the application to be less time consuming. The enhancement method that we implement here is that instead of opening a single connection, we are going to open multiple connections from the sender to the receiver. From the implementation point of view, we are creating multiple sockets (endpoints). This automatically calls for management issues like server concurrency (our issue here), since we are opening up multiple connections to the server, though from the same client. This will also introduce us to the issue of concurrency in client programs.

In this "simple" file transfer program, the server binds itself to a port and waits for the file that the receiver is sending. The client then connects to the server on the port opened and proceeds with the transmission. For simplicity, only the relevant code snippet is given here; you'll need to add the standard server and client program procedures like filling up the socket data structures, binding your server, etc. to test it if desired.

// The client
sock_descr=socket(AF_INET,SOCK_STREAM,6);  //create a TCP socket
c_status=connect(sock_descr,(struct sockaddr*)&remote_addr,sizeof(remote_addr));

if(c_status<0){
	printf("Can't connect");
	exit(0);
}
else {
	fd=fopen(argv[1],"r");   //the file to be transferred is supplied as a command line argument
	if(fd==NULL){
		perror("Error reading file");
	}

	while(!feof(fd){
		ch=fgetc(fd);
		write(sock_descr,&ch,1),sock_descr);
	}
}

// The server

//listener is the descriptor returned by listen()
if ((newfd = accept(listener, (struct sockaddr *)&remoteaddr,&addrlen)) == -1){
	perror("accept");
}
else {
	printf("Recieving file...\n");
	if(read(newfd,&ch,1) <= 0){
		fputc(ch,fp);
        fseek(fp,1,SEEK_CUR);
	}
}

The file transfer method just described suffers from a drawback: the time involved in the whole process increases drastically with file size. We will now enhance the process by introducing a degree of concurrency at the sender as well as the receiver. The terms sender/receiver and client/server, by the way, are used interchangeably in this text. What we seek to do at the sender's end is to logically divide the file using the algorithm mentioned below. The splitting is done at two levels: level 1 is the logical division of the file horizontally into Num_chunks. Then, each of the Num_chunks is divided into Num_packets. The number of simultaneous connections that the client will open up with the server is num_sockets which is equal to Num_packets. The various parameter values in the following pseudo-code are arbitrary, and can be replaced as appropriate for your application.

Check file size, f_size(in MB)
if f_size > 5  then call module SPLIT else proceed with normal("sequential transmission")
	Module SPLIT
		For 5 < f_size < 10 
		Level 1 split
			Num_chunks=[f_size/ sizeof-1-chunk]
			Sizeof-1-chunk=2mb
		Level 2 split
			Num_packets=[sizeof-level-1-chunk/ sizeof-1-level2 packet]
			Sizeof-1-level2 packet=0.25 mb

		For 10 < f_size < 15 
		Level 1 split
			Num_chunks=[f_size/ sizeof-1-chunk]
			Sizeof-1-chunk=4mb
		Level 2 split
			Num_packets=[sizeof-level-1-chunk/ sizeof-1-level2 packet]
			Sizeof-1-level2 packet=0.25 mb

		For 15 < f_size  < 20 
		Level 1 split
			Num_chunks=[f_size/ sizeof-1-chunk]
			Sizeof-1-chunk=5mb
		Level 2 split
			Num_packets=[sizeof-level-1-chunk/ sizeof-1-level2 packet]
			Sizeof-1-level2 packet=0.25 mb

		No of open sockets=Num_packets

	End Module SPLIT

Proceed with transmission

That was the algorithm we intend to implement. This will be covered in the next issue, which will introduce us to concurrent clients and Interprocess Communication (IPC) mechanisms. Till then - happy coding and experimenting!

References:

1. Unix Network Programming(Vol I & II) - W.Richard Stevens
2. Dan Kegel's C10K paper
3. Accept() scalability on Linux, Steve Molloy, CITI - University of Michigan

Talkback: Discuss this article with The Answer Gang

The author is a 3rd year Computer Engineering Undergraduate at Haldia Institute of Technology, Haldia. His interests include Network Protocols, Network Security, Operating systems, and Microprocessors. He is a Linux fan and loves to hack the Linux kernel.

Ecol

By Javier Malonda

The Ecol comic strip is written for escomposlinux.org (ECOL), the web site that supports es.comp.os.linux, the Spanish USENET newsgroup for Linux. The strips are drawn in Spanish and then translated to English by the author.

These images are scaled down to minimize horizontal scrolling.

Click here to see the full-sized image.

Click here to see the full-sized image.

These cartoons are copyright Javier Malonda. They may be copied, linked or distributed by any means. However, you may not distribute modifications. If you link to a cartoon, please notify Javier, who would appreciate hearing from you.

Talkback: Discuss this article with The Answer Gang