This diagramming tool is called Lucidchart and it’s based on open web standards, so it’s completely device-independent. It’s a far cry from Microsoft Visio and runs blazing fast on Chrome. Plus, site updates and improvements require nary a click from the end user. If you’re curious, check out this recent blog post from the Lucidchart team that compares the performance of Lucidchart on various web browsers.

Here are a few diagram types that I’ve made with Lucidchart:

• Network diagrams (complete with the latest AWS 2.0 shapes)
• ERD diagrams
• UML diagrams

I’ve also heard good things about Lucidchart’s capabilities for mocking up UI prototypes and wireframes.  Lucidchart lets you add interactivity with hot spots, links, states, and demo mode. These features create dynamic action between objects, pages, and external links, and are particularly helpful for demonstrating how a live website or mobile app will respond to user interaction.

Part and parcel of this cloud-based service is the ability to collaborate with any user, from any device, on any browser.  Changes from any user are reflected instantly, which saves me tons of time on team projects.

Once you sign up, simply log in to the editor and start diagramming. To get started, drag out shapes and place them on the canvas. When you need to connect shapes with a line, they’ll automatically pop out from natural connection points on any given shape. This small detail, along with the snap-to-grid feature, really makes your diagrams look neat and professional without any extra effort on your end.

As I mentioned, it’s easy to share Lucidchart with another viewer or collaborator. You can invite a collaborator to work on a diagram with you, as well as publish your diagram to all sorts of places, including:

- Traditional printed page
- Unique webpage hosted on the Lucidchart site
- Permanent online PDF link
- Popular image formats: PDF, PNG, and JPEG
- Embedded in a blog or wiki
- Shared on Twitter, Facebook, or the Lucidchart Community page

If you’d like to give Lucidchart a whirl, try out the free demo and be sure to let me know what you think in the comments below!

The critical aspect of the command prompt is to ensure that an automated system is in sync with the remote process it is controlling. A command prompt is meant to indicate that there is a process waiting for commands or other inputs. Duh. Accurate prompt detection becomes very important to the reliability of an automated system because without knowing whether or not you have captured the most recent prompt and the remote process is actually waiting for input, your system might wind up sending data to a system in an unknown state, a condition likely to cause failures and errors and possible even exceptional exits. Typically the cause of losing sync with the current prompt is that the prompt captured is the one before the command was issued – this results in an automated system thinking it has received the output of a completed command and the remote session is ready for the next command.

This article will thoroughly address the issue of prompt detection regardless of the factors that may confound ordinary prompt detection and verification, and allow automation developers to focus on other tasks instead of debugging what can be an intermittent error.

For those of you who intend on working with the code, I can’t emphasize this enough: always wear eye protection.

Can You Control The Prompt?

Once common mistake developers make is to use a generic regular expression using common characters used as the last non-space character of the prompt, and anchoring the expression. For example, consider this prompt:

“\[\r\n]?\[^\r\n]+\[%#>\$] $”

This defines a string where there may be a line break, then a string which terminates in a common prompt character and a space, and that space must be the last character. This is a reasonable definition and works in most cases, until it stops working.

Besides the fact that prompts may be set to something that does not adhere to the standard prompt expression, there are also instances in which you may thing you got the prompt, but instead matched something within the incoming data. Even on very fast connections, if you enable diagnostics and watch Expect’s internal matching attempts, you may see that it is processing one character at a time, and you can watch the input buffer grow until a match is made. When dealing with cell modems and other slow connections, it is possible (I know from personal experience) that the data may appear to stop and the last part if it may match the above expression. Your expect clause will exit, thinking it has captured the next command prompt, when in fact your system should still be iterating the expect clause and waiting for the real prompt.

The above condition is even possible no matter how much you can control the prompt, and you may also have to deal with systems that don’t allow you to set the prompt.
In situations where the prompt itself is part of the returned data (such as if you printenv on the remote system), and you are able to control the prompt output, there is one trick that will help many of the subtle mistakes that prompt expressions cause. By setting the command prompt to include the command number of the command you just executed, you can expect a prompt with a number one higher than that number. For example, in csh, adding the parameter “%h” will output the command history number. You can then set your expected clause to include a number two higher (see below) as the next command prompt, and avoid the confusion of seeing the last prompt. The actual arguments you would use in your prompt settings, and how you would choose to get this number (either by regular expression to capture it or using the history command) is up to you.

The following section applies to handling basic command shells like bash, sh, csh, etc. The section after that applies to any prompting at all.

A Solution for Spawned Shells

For sessions which invoke something beyond just shell commands, we can’t rely on this solution unless we want to specify the prompt we expect. For example, the history number would not show if you were to invoke any program that prompts (like telnet).
I use Csh when I spawn sessions, so the following example will only work for Csh/Tcsh users, and I’ve added other interesting data (the %h in the prompt setting is the command number in the shell’s history, and it increments every time a command is executed but not when blank lines or carriage returns are sent):

set prompt = “\[%h\]$USER@${HOST}:`pwd`> “

This will result in a prompt that looks something like this:

[223]root@cyborg:/usr/local/etc>

Before we issue the command that we’re expecting some output from, we should know what the history number will be of the next prompt. A simple shell command will suffice:

set number = `history | tail -1 | awk '{print $1}'`

This is the command number of the command we are going to send, and incrementing it by one will give us the command number that we expect to see in the next command prompt

“\[\r\n]?\\[$number\\]\[^\r\n]\+[%#>\$] $”

The above expression assumes that “number” is the calculated value of the next command prompt to expect.. This solution should be elegant enough to stand the test of time, but only when dealing with systems where you can set the prompt to include history numbers. An alternative would be to use unique numbers and set the prompt every time before sending the command, and then using that unique number to determine whether it is the same as the one you sent the command to – if so, your expect clause should continue executing. In TCL8.4, you could use “clock clicks -milliseconds” to get the number, but this command has been deprecated and in TCL8.5 beta the command would be “clock -milliseconds”.
Here is an example in TCL to send a command and make sure your expect clause doesn’t exit earlier than it should.
, to accomplish the above example, using the sample prompt:

Is This Really The Prompt I’ve Been Expecting?

Suppose you are unable to create a unique expression for each time you are waiting for the next command prompt. This is where it gets annoying and when automated systems get derailed.
If we’re reasonably sure we’ve captured the most recent prompt and the remote process is actually prompting for more commands, we may want to make sure. If we couldn’t do enough with the prompt to be reasonably sure it’s the most recent, then we really want to make sure. If we don’t even know what the prompt is, then we have to find out and verify it. Like most philosophies about handling multiple remote command sessions, we often hit the return key once or twice to make sure the system is still responsive – after we’ve determined that the system isn’t waiting for an answer to a question. If it’s waiting for an answer, and especially if it provides a default if you simply hit the enter key, then this strategy won’t work. Therefore, besides just detecting command prompts, we must have something to determine whether we’re being asked a question instead.

The following example is for optionally discovering and definitely determining what the current prompt is. It is a re-entrant procedure for discovery, and only runs once for verification.

proc check_prompt { id {prompt ""} {wait 0} {timeout 1} } {

    ## we have to determine the prompt, and we may have to wait first
    after $wait;

    ## find out if the session is responsive
    exp_send -i $id "\n";

    ## process what we get back as a response
    if { "$prompt" != "" } {
       ## we are looking for a particular prompt to verify
       ## this can be changed to allow flexibility if needed
       expect {
           -i $id
           -t $timeout
           -exact "($prompt)$" {
               ## partial or complete prompt matched with anchors
               return $expect_out(1,string);
           }
           timeout {
               error "Timeout checking prompt against $prompt";
           }
       }
    } else {
       ## use a fairly well-crafted regular expression to find a prompt, and
       ## then re-enter this procedure with a verification request
       expect {
           -i $id
           -t $timeout
           -re {your-(well-crafted-expression)} {
               ## you may not feel the need to verify this one
               return $expect_out(1,string);
           }
	    -re "\[\r\n](\[^\r\n\]+\[%#>>\$] ?)$" {
		## verify the suspected prompt
		return [check_prompt $id $n $expect_out(1,string)];
	    }
	    timeout {
		error "Timeout checking prompt";
		return "UNKNOWN";
        }
    }
}

Note the first argument is the spawn id – Expect programmers should get used to passing this variable if they are handling multiple spawned sessions in the same procedures.
If this procedure is called with just the spawn id, it will attempt to determine the current prompt and then call itself with the prompt on the command line to verify it, and then return the prompt or throw an exception. If this procedure is called with both the spawn id and the n parameter (the next history command number), the we can more precisely define an expression that should exactly match the command prompt we’re expecting. f this procedure is called with a prompt argument, it will either return the very same prompt or throw an exception.

But Wait – There’s More

The solutions presented above have proven to be invaluable in keeping an automated system in sync with a remote command prompt. There are still some instances where even more is required. Sometimes a threaded program will return to the command prompt while one or more of the threads have not yet completed, and when they do, they will report to the same stdout/stderr that the original program had.

At Optimation, I had a Solaris workstation, mostly SPARC, for a short time X86. With Solaris came Calendar Manager. I soon came to depend on it. Other people had personal organisers, or diaries. I used CM. Again my personal and corporate personae were interlinked.

When Optimation and I parted ways, I needed to extract my personal bits. By that time, I had cable Internet at home and a little-used email address. This is now my main email presence.

How would I replace Calendar Manager? I suppose I could have run Solaris X86 at home. These days I could run OpenSolaris in a VM. Back then, I searched around and came across Remind.

Remind is my idea of a Killer App. I cannot sing its praises highly enough. Remind comes from Roaring Penguin and is written by David Skoll.

Why Remind ticks all my boxes

Basically, in my opinion, Remind exemplifies the right way to implement a GUI. In fact, one could argue that there are two Reminds: a command-line version, and a GUI. I suspect that David started by writing the command-line version; he added the GUI front-end, TkRemind, later.

	It provides a friendly graphical interface which allows you to
	view your calendar and add reminders without learning the
	syntax of Remind.

		- TKREMIND(1)

But the two are beautifully integrated. The term “front-end” does it an injustice.

Remind has a very rich and powerful language to specify reminders. More commonly used reminders can be created simply. But there is also a bewildering array of features which are more complicated to specify. So, if you want to do the tricky stuff, you can; you’re just going to have to work a bit harder. If you have simple requirements, you can stick to the GUI.

I believe that it is hard to find applications that can be used easily and quickly by beginners, but can still provide all the features for users who are there for the long haul. In this, Remind suits me very well. I am often reluctant to spend a long time learning an application before I can see any results.

With Remind, I was able to see useful and usable results in a matter of minutes. In the last 4 years, I have learnt more about Remind and can do more with it. It’s called ROI (Return On Investment). In this case, my investment is my time. I want to be able to make a small investment and recoup it before making bigger investments. I was comfortable investing the extra time in learning more, because Remind kept giving back. It has never disappointed me. It’s analogous to a satisfying relationship.

A Tour of Remind

TkRemind has a bit of CM about it. Typically, it displays a view of the current month. Click on a day and you can add a reminder. These can be quite complex. I don’t think the details are important. You can find out more at the URL above. You can also go to Remind: The Ultimate Personal Calendar which was the article that turned me on to Remind.

To me, what makes Remind so special is the brilliant (and very simple) integration of Remind, TkRemind and the supporting database. That and it’s near-infinite flexibility; I have found nothing set in concrete.

By default, TkRemind reads $HOME/.reminders. That’s what I started with. Eventually, I decided my requirements were better served another way. Instead of a single file, I now have a REMINDERS directory. In that directory, I have a “main” file (.reminders) and perhaps a dozen include files. The names of these files tell you the story:

.reminders			The "main" file
.reminders_birthdays
.reminders_custom		personal customisations (see below)
.reminders_daylight_savings
.reminders_public_holidays	public holidays in my neck of the woods
.reminders_utility		useful definitions
.reminders_year_by_year		a container for the following

.reminders_ad_hoc_2011		uncategorised reminders
.reminders_footy_2011
.reminders_movies_2011
.reminders_theatre_2011

.tkreminders			reminders inserted via tkremind (see below)

.reminders_custom needs a little explanation. One of the nice things that come with Remind is a highly sophisticated capability for calculating astronomical events eg sunrise, sunset. However, for this to work, your location (latitude and longitude) are essential. This is the file in which I set these parameters.

I discovered that TkRemind permits the user to have a separate file for entries added through the GUI. I use .tkreminders for this. It’s a little feature that demonstrates why David is brilliant.

At first, I had a single reminder file based on a sample provided with the software. Then I decided I wanted to separate different parts. One of the obvious separations revolves around things that happen in a calendar year. Once the year is over, there’s no point keeping them in the database (archive perhaps).

All these files can be kept under revision control eg RCS. I guess it’s more useful for the first group of files above; these constitute the infrastructure as it were. The group with the year (2011) in the name are only relevant for a single year.

Integration

How does it all hang together? In a word, superbly. It’s a classic case of horses for courses. If I want to add a single event, like a dentist appointment, I do that through the GUI. Anything ad hoc but easy to describe can also be added through the GUI.

Each year, when the football fixture comes out, I edit .reminders_footy_YYYY – because I find that sort of entry easier to generate using an editor. The same is true for the other year-by-year activities.

If you think about it, you realise that adding one-off reminders through the GUI makes sense; but deleting them is often best done in bulk. From time to time, I can edit .tkreminders pruning old entries.

The whole thing integrates seamlessly. If I edit the underlying text files, I don’t need to perform any magic to get the changes reflected in the GUI. At worst, I only need to go forward a month and then back to the current month to see the effect of my changes.

Unix/Linux Principles

Remind does not try to be an operating system:

	You  can  use  any  text editor capable of creating plain
	ASCII files ...

It just does the one thing extremely well.

How can you possibly manage change in a GUI-only environment?

Consider Microsoft Word. It has a mode which lets you compare one rev with another. But this could be seen as the bane of Linux purists. An app should do only one task, be as small as feasible and do that one task well.

I already have diff and grep. I don’t want the next app I adopt to provide another flavour for me. For all I know Remind has a search function. I don’t care. I know that all the data is stored in flat text files. Easy to search – grep. Easy to manage revisions – rcs. Easy to determine what’s changed – diff.

And for those things that are best done through a GUI – TkRemind: quick, clean, easy to learn, easy to use. Dare I say intuitive?

Recommendation

If you already have something which handles your calendar needs, I doubt that this article will get you to change. That’s fine. To each his own.

On the other hand, if you’re dissatisfied with what you have, or you’re looking around, Remind has my unequivocal recommendation.

Footnote

While I was writing this article, I was struck by the possibility of improving the way I handle the year-by-year events. I now have several files which achieve the equivalent of a couple of loops:

#       for year in lastyear thisyear nextyear; do
#               for activity in theatre_ footy_ movies_ ad_hoc_; do

The nub of the loop is to conditionally include only those files which exist.

It means that I can now add or delete files of the form

	.reminders_activity_yyyy

without ever changing the infrastructure. If the file exists, it will be included; if not, not. In December, I can look ahead to see what’s coming up next year. In January, I can see what I did the previous year.

Try that with your favourite GUI.

DreamForce 2011
Aug 29-Sept 2, Moscone Center, San Francisco, CA
http://www.salesforce.com/Dreamforce

Intel Developer Forum – IDF 2011
Sept 13-15th. Moscone Center, San Francisco, CA

http://www.intel.com/IDF

Storage Decisions New York
September 19, 2011 – New York, NY

(ISC)2 Security Congress
September 19-22, Convention Center, Orlando, FL
https://www.isc2.org/congress2011

Semantic Technology Conference – LONDON
September 26-27, 2011, Hotel Russell, London
http://semtechuk2011.semanticweb.com/index.cfm
,
Intel Elements 2011
Sept 28-29, Seattle, WA
http://appdeveloper.intel.com

Oracle OpenWorld 2011
Oct 2-6, Moscone Center, San Francisco, CA

http://www.oracle.com/openworld

Mobile and Smart Device Security
Oct 4-6, Intercontinental, Atlanta, GA
http://www.misti.com/mobile

Silicon Valley Code Camp 2011 – Free
Oct 8-9, Foothill College, Los Altos, CA
http://www.SiliconValley-CodeCamp.com/Register.aspx

DISKCON USA 2011

October 18-20, Convention Center, Santa Clara, CA.
http://www.idema.org/?p=2054

LinuxCon Europe/Embedded Linux Conference Europe
October 26-28, 2011
Clarion Congress Hotel/Prague, Czech Republic
http://events.linuxfoundation.org/events/linuxcon-europe

AnDevCon II
Nov 6-9, 2011, Hyatt Regency, Burlingame, CA
http://www.andevcon.com/

ApacheCon North America 2011
Nov 7-11, Vancouver BC
http://us.apachecon.com/c/acna2010/articles/2011/02/08/coming-soon-apachecon-north-america-2011/

Enterprise 2.0 Conference
Nov 15-17, 2011, Convention Center, Santa Clara, CA
http://www.e2conf.com/santaclara/

Cloud Security Alliance Congress
Nov 16-17, 2011, Orlando, FL
http://www.misti.com/cloud

LinuxCon Brazil
November 17 & 18, 2011
Expo Center Norte/Sao Paolo, Brazil
http://events.linuxfoundation.org/events/linuxcon-brazil

Semantic Technology Conference – EAST
Nov 29-Dec 1, 2011, Kellogg Conference Hotel, Washington, DC
http://semtecheast2011.semanticweb.com/index.cfm

Distros

Network Security Toolkit (NST) Version 2.15.0 Released

The latest NST release: “v2.15.0″ is based on Fedora 15 using Linux Kernel 2.6.40 rebased from latest Linux Kernel: 3.0.

Added NST RPM packages include the Open Vulnerability Assessment Scanner (OpenVAS) and Greenbone Security tools. Also added is an NST RPM package of the THC IPv6 ATTACK TOOLKIT.

Here are some of the highlights for this release:

* Developed a new interactive dynamic SVG/AJAX enabled Network Interface Bandwidth Monitor application that is integrated into the NST WUI for monitoring pseudo real-time network bandwidth rates.

* Added NST RPM packages and NST WUI integration for the Open Vulnerability Assessment Scanner (OpenVAS) and Greenbone Security tools.

* Added an NST RPM package for IPv6 network security investigation: The THC IPv6 ATTACK TOOLKIT.

* Now supporting the GNOME 3.0 desktop for NST integrated network security applications.

* Refactored all NST integrated services for transistion to the new systemd System and Service Manager for service control.

* By default, now supporting the new Fedora Network Device Naming convention (e.g., eth0 => p2p1 or eth0 => em1).

* Many new NST WUI enhancements and refinements.

As always, the networking and security applications included have been updated to their latest version.

Lucid Puppy 5.2.8 released

Lucid Puppy is the edition of Puppy Linux that is built from Ubuntu binary packages and has compatibility with, and access to, the vast Ubuntu package repositories.

According to Puppy coordinator Larry Short:

“Lucid Puppy 5.2.8 is the fastest and friendliest Lucid yet. It is the fastest because it is the first Lucid to use the C and ffmpeg libraries optimized for i686 computers rather than the older i386 computers. Lucid 5.2.8 has also received a thorough going-over under the hood. There is new and updated firmware and drivers for many devices, courtesy of forum member tempestuous, and the hardware detection and configuration routines have been extensively tested and enhanced by forum member rerwin. The overall look and feel has been influenced by experiments forum member pemasu, to make the user interface fun for Linux newbies and experienced users alike.”

Lucid 5.25 is also a major update from Lucid 5.2.  It has Bash 4.1.0, an upgrade from Bash 3, Syslinux 4.03, an upgrade from Syslinux 3, and e2fsprogs 1.41.14, the latest from Ubuntu Natty. It also now has JWM 500, up from 493 and Gnumeric 1.10.13 (up from 1.10.12 in 5.2).

For full release notes go to :
http://distro.ibiblio.org/puppylinux/puppy-5.2.8/release-Lucid-528.htm

Products

Reflex Announces Integrated Capacity and Performance Analytics

In August, Reflex Systems, a leader in virtualization management solutions, announced availability of a vCapacity module for Reflex Virtualization Management Center (VMC). With vCapacity, Reflex VMC integrates monitoring, performance, capacity, configuration management and security into a single solution, providing real-time visibility, correlation and control across your entire virtual or private cloud environment. VMC with vCapacity allows administrators to spend more time solving infrastructure issues, instead of manually integrating, correlating and reporting activity across multiple products and consoles.

From a single interface and a single logical database, Reflex VMC provides the ability to show historical performance, change tracking, monitoring, capacity, configuration, inventory, chargeback/showback, and security intelligence. The new integrated capacity planning and enhanced performance analytics in vCapacity enable integrated functionality that isn’t possible with VMware vCenter, VMware management products, or point solutions. vCapacity is highly customizable in terms of both presentation of data within the interface, as well as online and offline reporting. This enables organizations to customize capacity planning and performance management to truly meet the individual demands of their organization.

“Capacity planning is a big issue for organizations as they mature in their use of virtualization, especially as they evolve to a private cloud model,” said Rachel Chalmers, Research Director, Infrastructure Management for The 451 Group. “Users are searching for capabilities that make capacity planning more accurate, straightforward and integrated with their existing systems and best practices. Reflex offers them broad functionality that spans multiple management disciplines within the IT organization.”

vCapacity, when combined with the integrated feature sets of the VMC, provides a broad view of capacity and performance analytics in context of the entire virtualized data center. Operations teams may be using resources efficiently based on initial capacity analysis but ongoing management of resources within the broader management of the virtual infrastructure is critical to improve both CAPEX and OPEX efficiency.

• Reflex provides a high-level summary dashboard as well as granular targeted dashboards highlighting capacity, contention, latency, inventory, allocation, availability and efficiency. vCapacity allows administrators to quickly visualize capacity planning issues, such as:
• Resources (CPU, memory & storage) that are being wasted
• Where resource contention is highest across the entire environment
• Physical resource overcommitment usage and trends
• Which VMs are consuming the most resources
• Capacity usage across time, including future forecasting

For further information on the new vCapacity capabilities, visit  www.reflexsystems.com and follow Reflex at http://twitter.com/ReflexSystems and the vCurve Blog: blog.reflexsystems.com.

In August, LinuxCon celebrated 20 years of Linux. That’s right, its been 2 rapid decades since Linus Torvolds sent our his historic post to the MINIX news group that he was working on a small open source version of a Unix-workalike for personal computers.

LinuxCon North America in Vancouver will be ground zero for the festivities, and a major Linux conference as well. Besides the “Roaring 20′s” 20th Anniversary Gala, there will be hot topic panels and presentations on the Kernel, device drivers, embed Linux, IPv6, benchmarks and tracing. LinuxCon brings together the Linux community including core developers, administrators, end users, business executives and operations experts. In short, all of us.

As part of the festivities, the Linux Foundation is hosting a Facebook Quiz: “Could you do Linus Torvalds’ Job?” Linus has made it clear he’s not going anywhere, but, here is a cool way to find out if you’ve got what it takes to manage the kernel development community. But here is something even cooler: everyone who participates in the quiz gets a a custom code for 25% off the LinuxCon registration fee. You can also take the quiz as many times as you like. Take the quiz now: http://www.facebook.com/TheLinuxFoundation?sk=app_209551195757882

You also have a chance to experience IBM’s Watson by testing your skills against the Jeopardy! champion. The Watson game and video interface will be on display at LinuxCon and is at the IBM booth in the Technology Showcase.

Besides the panels, parties, and the special 20th aniversary t-shirt, there is also an
historical gallery at LinuxCon North America to help remember Linux’s last 20 years. This gallery is a walk down memory lane and should be fun for everyone, but LinuxCon needs your help – send or bring more memorabilia. Aready collected: the original books Linus used to learn programming, CDs and boxes of early Linux distributions, computers used to do early hacking, memorabilia from IBM’s Peace/Love/Linux campaign and much more.

For more information on the Linux History Gallery and how to donate, please visit Amanda McPherson’s blog:

http://www.linux.com/news/featured-blogs/167-amanda-mcpherson/466227-do-you-have-linux-memorabilia-to-donate-to-our-linuxcon-gallery

USENIX Security Symposium and co-located events

Preceeding LinuxCon in the second week of August will be the 20th USENIX Security Symposium which is co-located with specialty conferences such as WOOT (Workshop on Offensive Technologies) and HotSec (Hot Topics in Security).

Here is the full list of co-located events:

EVT/WOTE ’11: 2011 Electronic Voting Technology/Workshop on Trustworthy Elections
CSET ’11: 4th Workshop on Cyber Security Experimentation and Test
FOCI ’11: USENIX Workshop on Free and Open Communications on the Internet
WOOT ’11: 5th USENIX Workshop on Offensive Technologies
HealthSec ’11: 2nd USENIX Workshop on Health Security and Privacy
HotSec ’11: 6th USENIX Workshop on Hot Topics in Security
MetriCon 6.0: Sixth Workshop on Security Metrics

I have previously been to HotSec and WOOT events and can say these are research-oriented and very cutting edge. Having all of these sessions under one roof will be very interesting and a great opportunity to interact with many people at the frontiers of computer and network security. Here are links to more recent USENIX security conferences, which have slides and white papers available:

http://www.usenix.org/events/sec10/tech

http://www.usenix.org/events/hotsec10/tech/

http://www.usenix.org/events/woot10/tech/

http://www.usenix.org/events/cset10/tech/

The program begins with the two-day tutorial TCP/IP Weapons School 3.0 taught by Richard Bejtlich and continues with its 3-day technical program. All events will take place at the Westin St. Francis near Union Square.

Black Hat North American

In the first week of August is the somewhat infamous Black Hat Conference for hackers and security professionals. This is the event where alpha geeks demonstrate successful attacks on smartphones, ATMs, and DOD facilities.

The real heavy duty hacking happens at the even more infamous DefCon event which overlaps the main Black Hat event, but you need an exclusive invitation to attend to DevCon. And if you do go to DefCon, or even Black Hat, be prepared to reformat your internet access device. Everyone is a target.

But this year, you can see some of the event online. Via the Black Hat Uplink program, thousands of security professionals from around the world will be a part of Black Hat USA 2011, but not in Las Vegas at the live event. With Black Hat Uplink, you can experience the content that shapes the security industry from the comfort of your home or office machine. There will be 2 tracks of content available – perhaps vendor-sponsored sessions – and the keynotes and award ceremonies, all for free. The hitch here is that you can’t review any sessions online without paying the access fees – only the keynotes will be available for replay. Of course, onsite registraton will set you
back $2495.

Black Hat Uplink registration is here:

http://www.blackhat.com/html/bh-us-11/bh-us-11-uplink.html

NoSQL Now

NoSQL Now is a new conference but its being shepherded by Wilshire which also organizes MDM and BI events (and the Semantic Technology conference until this year) Its also a very hot trend right now and should attract a large and varied group of attendees. Like other Wilshire events, lunch is NOT provided except for a one day lunch and learn session. Pack your own or carry cash for for eateries in San Jose.

Topics so far include items such as “NewSQL vs NoSQL for New OLTP” and “Putting Business Intelligence to Work on Hadoop Data Stores” . Although this is the first one, there is reason to expect this to be a good one.

VMworld 2011

Leading up the rear, but not the least by any criteria, is the annual VMworld conference. This is where all things VMware will be discussed and it keeps getting bigger. So its in Vegas this year instead of San Francisco. Hot days and nights. Carry lots of water.

Here are some stats from last year in San Francisco: VMworld 2010 attracted more than 17,000 customers, partners, and analysts, and 233 sponsors and exhibitors signifying a major shift in the IT industry towards cloud computing. VMworld 2010 featured more than 170 unique breakout sessions across 10 tracks. Expect an even bigger event this year, which is probably the reason its being held in Vegas.

This event could be ground-zero for for the newest innovations in virtualization and cloud computing. VMware and EMC continue to be the virtualization juggernaut, although the Linux KVM technology is gaining marketshare.

While VMworld isn’t streamed over the net (not yet), many sessions are available at the VMware Community site, where content is basically free., You can pre-purchase access to 2011 content for $699, which may include addtitional materials not on the community site. Many sessions from 2010 are currently available here: http://www.vmworld.com/community/sessions

For 2011, the first keynotes will be streamed and available for replay later. However, the Thursday keynote will NOT be available for replay. I assume it will be streamed, however.

Here is the August event list:

Black Hat North America 2011
Aug 3-4, Caesar’s Palace, Las Vegas, NV
http://www.blackhat.com/html/bh-us-11/bh-us-11-home.html

20th USENIX Security Symposium
Aug 10-12, 2011, San Francisco, CA
http://www.usenix.org/sec11/

LinuxCon North America 2011
Aug 17-19, Hyatt Regency, Vancouver, Canada
http://events.linuxfoundation.org/events/linuxcon

NoSQL NOW!
Aug 23-25, 2011, San Jose, California
http://nosql2011.wilshireconferences.com/

VMworld 2011
Aug 29-Sept 1, Venetian, Las Vegas, NV
http://www.vmworld.com/index.jspa

Contents:

• News in General
• Conferences and Events
• Distro News
• Software and Product News

Please submit your News Bytes items in plain text, as a one- or two-paragraph summary plus a URL; we do not publish entire press releases. Submit items to bytes@linuxgazette.net. Deividson can also be reached via twitter.

• • Access over 1,000 live and on-demand technical training sessions, keynotes, and Super Sessions.  About 70 conference video and 200 audio sessions will be on-demand, after the event
• • Download 2011 session PDFs prior to the conference
• • Build a virtual “briefcase” to manage and organize the content that interests you
• • Join the conference conversation in the blog center and social media
• • Download Cisco booth content and partner information in the exhibitor resource centers
• • Participate in games and win prizes

I was very happy with Cisco Live 2010, especially as almost all session PDFs were available before the day of the session.   You just can’t get slides in advance at most virtual conferences.     But not all sessions were available to the free attendees last year.   If that limitation is removed, this is an excellent opportunity. Since free registration to Cisco Live Virtual is available only for 6 months, last year’s content is no longer available as a virtual conference.   But some sessions are available. One of the sessions I downloaded in 2010 was entitled “Highly Available Wide Area Network Design” (session BRKCRS-2042) and it weighed in at 102 slides.  So many of these presentations are both detailed and quite good. There may be a lot of Cisco partners at the solution pavilion, but the collateral will be on-line.  However, attending has other incentives: the lab experience – Cisco Live’s four- and eight-hour labs give direct experience installing and troubleshooting network technologies and practicing network operations in a simulated environment moderated by Cisco experts. Many labs also qualify as preparation for specific Cisco Certifications exams.  But you have to go to be there, and the labs cost $600-1000 each over the cost of the conference. Collaboration User Group at Cisco Live 2011 If you are actually attending Cisco Live in Vegas, here are also special opportunities for Cisco Collaboration User Group members to meet with each other and Cisco executives and get other members-only perks at the conference. Collaboration User Group members get extra benefits at Cisco Live:

• • Access to a welcome reception following the Collaboration Welcome Session
• • Exclusive NDA product direction sessions
• • Preferred access to the Walk-in Self Paced Labs
• • Preferred seating at the Collaboration Town Hall
• • Collaboration User Group polo shirt and badge ribbon

To learn more about Cisco Collaboration User Group activities at Cisco Live 2011, go to the usergroups web site. Conference Hotels include the Luxor, the MGM and Mandalay Bay, so the event is at the end of the Strip nearest the airport.   That make leaving easier and taxi fares less.  That must be Cisco taking care of us. Of course, this is an expensive event to attend physically.   We are now in the ‘late’ registration period, with costs raised $100 to $2195  for June 4 -July 8, and an extra $100 the week of the conference.  Airfare and hotel and expenses would easily be another $1000-2000.  So do consider that virtual conference option.   Its good value for no money.

August preview:
– LinuxCon North America 2011 in Vancouver, Canada
– The first NoSQL NOW! event in  San Jose, California
– VMworld 2011 at  the Venetian, Las Vegas, NV

Here is a list of the full set of articles:

• Henry’s Techno-Musings: My Help System
• What Really Matters or The Counting Article
• HAL: Part 1 Preliminaries
• HAL: Part 2 My Favourite Aliases
• HAL: Part 3 Functions
• HAL: Part 4 Scripts

In HAL: Part 1 Preliminaries, there was a link which would allow the reader to download my HAL. There you will find about 123 scripts.

Here are the most interesting ones.

Aliases Which Invoke Scripts

I’m going to start with scripts that I mentioned in ‘HAL – My Favourite Aliases’ in the section ‘Aliases Which Invoke Scripts’.

alias cleanup $HOME/scripts/cleanup.sh

This is only relevant to emacs users. By default, when you edit a file, emacs keeps a copy of the original file in a file whose name has a trailing tilde (~). They’re worth hanging on to while you are developing your program, script or article. But once you have finished, they become just clutter, especially if you maintain some sort of revision control (which I do).

This script finds all such emacs backup files in the tree rooted at the current working directory, lists them and offers to delete them.

alias diff $HOME/scripts/diff.sh

This script could be seen as related to the previous script. With more than one arg, it behaves like a regular diff. If given a single arg, it compares that file against its emacs backup. So, for example, ‘diff HAL.Scripts’ (the file I’m currently creating) would become ‘diff HAL.Scripts~ HAL.Scripts’.

Looking at the script now, I wince: it seems so badly written. And it seems it would be better cast as a function. But it has evolved over time, so perhaps I should be more forgiving.

alias lns '$HOME/scripts/lns.sh'

Like “ln -s”, but, if the target is a symbolic link, delete it first. If it’s a file, offer to delete it.

Ideal for replacing a symlink.

alias tcg='/home/henryg/scripts/tarcompressuuencode.sh +u -g'
alias tcu='/home/henryg/scripts/tarcompressuuencode.sh'
alias uut='/home/henryg/scripts/uudecodeuncompresstar.sh'

The name of the script provides clues to its origins. I’ve used email since at least 1993. Back in the early days, the typical way to send directory trees was tar-compress-uuencode. That’s what this script does. Over the years, gzip replaced compress and often I don’t uuencode (except when sending to myself).

‘tcu’ was what I used at first; ‘tcg’ is what I typically use now.

‘uut’ is for use in the other direction. I guess, these days, I would only use it on mail from myself. Other related capabilities are provided by functions ‘gttv’ and ‘gtx’ (see ‘HAL: Part 3 Functions’).

alias x='/home/henryg/scripts/expr.sh'

The ‘x’ alias (nice and short) invokes expr.sh, my Mickey Mouse calculator. For most of my calculations, this suffices. I can enter asterisk (‘*’) or lower-case ‘x’ for multiply. I can use parentheses for grouping as long as I quote the expression.

For example:

x '(2 + 3)/ 7'
0.714286

Under the covers, it invokes awk.

alias xr='/home/henryg/scripts/xrenamewindow.sh'
alias xt='/home/henryg/scripts/xterm.sh'

I’m currently running 34 xterms spread across 6 virtual desktops. By default, they’d all have the text “xterm” in the title bar. When iconised they’d all be identical and all have the text “xterm” near the bottom of the icon. How would I find the one I want?

When I start an xterm, I use the ‘xt’ alias followed by a string specifying the text I want to see in the title bar and on the icon. (I can choose to have different text for the icon, but usually I don’t.)

This text is meaningful to me and helps me to keep track of all my xterms. The one I’m working in at the moment has “WRITING” in the title bar. There’s another right near it with “WRITING2″. This is my main window for creating what you are now reading. The other one is a “helper” window; for instance, I use it to look up what I wrote in a previous article.

These two windows are here for the long haul. I opened “WRITING” last Christmas Day, soon after I rebooted 23 December 2010 (as I write, today is 28 February 2011). I’m not sure why it should be this way, but I only opened “WRITING2″ 3 Jan 2011.

Often I only work on something for a short time. When I’ve finished, if I want to recycle the xterm, I might decide to give it a different text string in the title bar. For that I use ‘xr’.

Scripts that stood the test of time

Some of these display my ignorance or naivete at the time that I wrote them – proving, yet again, that a script does not have to be brilliant or perfect to be effective and useful.

If someone provides me with a better way of achieving the same result, I’ll consider replacing the corresponding script.

The dates below refer to the first time I wrote the script. Many have been modified over time.

1991	expire.sh - move a file to the expire directory

Nowadays it would just be a function.

The basic idea is that some things (files, directories) look like you’ve finished with them, but you’re not quite sure. Move them to the expire directory. From time to time, when you visit the expire directory, if they are still there, delete them. I guess it’s a little like Trash on a Microsoft Windows system.

I use this infrequently.

1992	howWide - displays the number of chars in the widest line in a file

I cannot tell you how many times I’ve used this script. It often dictates whether I choose to print a file in landscape as opposed to portrait.

1993	rcsdiffp - do an rcsdiff on this file and the previous
                   revision in RCS

rcsdiff is usually used to compare the working copy with the last checked-in version. There are many options to perform other sorts of comparisons. Over the years, the second most common comparison for me has been with the penultimate revision because, after I have checked a file in, I may want to check again what I changed.

1995	add.sh

Adds numbers. Example usage:

ls -la | cutf 5 | add.sh

cutf, a function, is my response to some non-uniformity between awk and cut. cut only knows about single-char field separators; by default, awk treats white space as the field separator. cutf behaves more “intuitively” than cut on many sorts of (unstructured output) data.

The above pipeline lists the files, selects the 5th field (file size) and adds the sizes.

In many cases, even if there is text intermingled with numbers, add.sh produces the sum.

1995	ps_who_owns_it*.sh - provide a list of parent process ids for a pid

Solaris has ptree which does a better job than this script. Linux has pstree, but I’ve never found it satisfactory.

Give it a pid and this script will provide a full PS list entry for the given pid and all of its parents up to 1. I’ve never managed to get this script to work upwards and downwards. Each process has a single parent, but can have more than one child, so the full script would be more difficult.

I use this often.

1996	emacs-nw - interlude for commands in .mailrc

This is an interesting script. It has about 12 functional characters in the entire script. Any explanation will be bigger than the script itself. If you don’t run emacs, you won’t care.

emacs comes in 2 flavours. If you run emacs from the command line, it will try to run using its special interface to X in a separate emacs window. If you want it to run in an xterm you have to specify something special. I never want to run the GUI version.

There are times when applications offer to run your favourite editor. Sometimes you can add arguments, sometimes you can’t. This script is for those times when you can’t.

Note that, although it could be cast as a function for interactive use, a function may not be visible to the application.

This is one of the few scripts I have written that does not have .sh at the end of the name. I cannot recall what my thinking was at the time.

1996	chmod_g-o.sh - chmod copying rx bits from group to other

This is a script that I don’t use often, but every time I do I’m glad I wrote it. It’s such a simple function to state, and not hard to write, but not the sort of thing one can do easily on the command line. This script does not treewalk. Use a construct like chmod_g-o.sh `find …`

It’s particularly nice because, quite often, when you are working on a tree, you want to set read permission on files and rx on directories. This saves you dealing with that sort of detail. If the group permissions are right, then the other permissions will be right after the script is run.

1997	rename.sh - rename files

Ostensibly written in 1997, a comment near the top suggests that I had already written something like this previously. I thought I based it on a similar command from VMS, but I can find no confirmation for that proposition.

When I wrote it, I was concerned that I might get it wrong, so rather than having it do a rename, it simply displays the commands which it would have executed. I check that the commands look right and then copy and paste them to achieve the rename.

I guess because I use it infrequently, I left in the usage so that it is always displayed. Here’s an example:

rename.sh 'part*' part=.doc
Usage:  /home/henryg/scripts/rename.sh  <from> <to>

        where <from> is filename pattern to match

        <from> is of the form <string1>*<string2>
        <to>   is of the form <string3>=<string4>

either of <string1> or <string2> may be null
either or both of <string3> or <string4> may be null

 mv part1 part1.doc
 mv part2 part2.doc
 mv part3 part3.doc

Best of the Rest

dircmp - compare 2 directories

There may be other ways to achieve the same ends, but I’ve used dircmp since 1993 and still find it useful. It reports which files are different and which files in the first directory are absent from the second directory (it’s not symmetrical).

disp_args.sh - display args

I use this to determine what a shell script sees as its arguments. This can be helpful if there is a lot of quoting of different sorts going on.

du-forte.sh - industrial-strength du
du-root.sh - industrial-strength du for the root partition

The main purpose of these script is to avoid NFS-mounted filesystems. If I want to know where the space is being used on a disk, especially if it’s the root filesystem, I don’t want to wait while large remote disks are searched inefficiently. If the space is remote, I’ll go to the remote machine to do the analysis.

Lately, I’ve found that NFS-mounted filesystems are not the only pain. I also have a 1TB USB disk attached to my machine. It’s pretty slow. Usually I don’t want to search that. I’m not sure there’s an easy way to achieve this objective.

Until I started writing this, I’d forgotten that I had du-root.sh. Damn, it would have been very useful the other day!

I tried it just now:

du-root.sh
This script performs
        du -csk /*
for directories (not files) on the root partition only.
2       /.snap
Skipping non-root /dev
Skipping non-root /tmp
Skipping non-root /usr
Skipping non-root /var
1902    /etc
Skipping non-root /cdrom
Skipping non-root /cdrom1
2       /dist
1026    /bin
160510  /boot
6842    /lib
242     /libexec
2       /media
14      /mnt
2       /proc
4128    /rescue
66      /root
5398    /sbin
Skipping non-root /p6
1320    /History_dir
75398   /sw
2       /Big
Skipping non-root /WD5
Skipping non-root /DISKA
Skipping non-root /p7
Skipping non-root /p8
2       /p5
2       /p1
Total (excludes /proc) 256858

That was quick. That’s so clever.

openwin*

There’s a whole family of scripts that I use to fire up the same window environment on Solaris, Linux and FreeBSD. I use this family constantly.

treecopy.sh - like Prime's tree copy (trecpy)

This script uses back-to-back tar to copy a directory tree. I used this extensively until I discovered rsync.

That’s It

That’s all the scripts that I’m going to discuss in any detail. There are many more, but I don’t think they are of interest to a wider audience. If you’ve downloaded them, feel free to explore, ignore, adapt, reuse and recycle.

As I said at the start, these entities (aliases, functions, scripts) were intended to make life more comfortable for me. They suit the way I want to operate. I do not expect I have clones. Every reader will be different and will have different notions of comfort.

I anticipate that an individual reader will completely ignore perhaps 90% of these entities. Of the remaining 10%, perhaps 2 or 3 will be deemed worthy of further consideration.

On the other hand, I hope there are enough readers that at least one person finds some of these entities worthwhile, even if it’s only to act as a launching pad for an entirely different entity.

Enjoy.

Click here to see the full-sized version

How to deal with spawned processes in hostile waters, and managing ever-changing spawn ids

One of the problems Expect programmers have with spawn ids is trying to manage them when remote systems continually require a re-establishment of the connection. There are numerous effective ways of doing that and many lead to spawn id parametrization, but I’ve found that regarding each spawned process the same way I regard windows in a windowing operating system – each one is a command shell, and if I need telnet or ftp or whatever, I can simply type that command into the command shell, perhaps make sure I didn’t make any mistakes, and then hit the return key.

Here’s the sample from the book “Exploring Expect” written by Don Libes, and also found in the manual page, using telnet as the session we are creating. We’ve all followed the simple example and done it this way and it worked just fine (until it didn’t work):

catch {spawn telnet 1.2.3.4} pid; 

Technically, in order to detect the failure of the process to spawn, the above command should really be implemented this way:

if { [catch {spawn telnet 1.2.3.4} pid] } { ## handle error, which is in the pid variable } 

Assuming success, we now have a spawn id variable that is our handle to the telnet session, and the spawn command returned the process ID of the new session in the pid variable, not the spawn ID. If there is an error in spawning the process, the pid variable will contain the error message. The process ID may be useful for identifying the process for various reasons, but it is the spawn_id variable that is required to communicate with the session.

There is an inherent and unavoidable problem with directly spawning processes following the above example – if the spawn command itself hangs for whatever reason, it will hang the script indefinitely. Many programmers find themselves digging deep into the source code of Expect to find ways of determining when such an anomaly occurs and how to handle it, but this should be their first clue that they are looking in the wrong place for a solution. Sometimes problems don’t need solutions, but simple circumventions.

I’m using telnet as an example, because it is widely used in corporate intranets and, like other communications protocols, it sometimes get closed remotely, physical network cables are sometimes accidentally disconnected or suffer electrical or failure, and the spawn id of the associated process becomes invalid because the spawned process exited. While catching exceptions during communication with these sessions will prevent them from propagating up the call-stack, and potentially causing the script to exit, caution must be exercised with this approach, especially when the expect clause continues to execute until the final criteria are met for the clause to exit and the following commands to execute.

Spawn ids are never re-used during the execution of an Expect program – if you spawn the same command again, you will get a new spawn id, but with a 32-bit system, you won’t run out of new spawn ids until you’ve done over four billion spawns, and if you have, you’re probably not doing things the way you should (even in run-forever systems).

Many Expect developers use the spawn id of spawned processes as an index or value in an array, or have other reasons why they prefer to reference the session based on other data, such as the IP address or hostname of a remote system, the name of the program spawned, etc. The spawn id will always be necessary to send data to, and listen to, a particular session. Indirect lookup tables may sometimes seem easier to work with, but they add an unnecessary layer and complexity to the management of these sessions.

Given the number of times I’ve had to fix this issue in existing code, and the number of times I’ve had to explain this simple solution in interviews and in courses I’ve taught in automation, it seems beneficial to share it with the entire community. My favorite solution is so simple that it seems too obvious, too easy, and therefore not something the average programmer would consider the answer they seek.

Again, I mention that the easiest way to regard spawned sessions is the same way to regard separate windows in a windowing system – each has a unique “identity” to the user, and it is intuitive when dealing with them. In the same way we would keep track of which window represented a unique session, spawn ids should he considered the same. Each window is a command shell with a CLI, and as long as we haven’t lost track of all of the sessions we have going, we know instinctively how to manage each of them.

Here is how to not only maintain the association between processes and their spawn ids, but also to avoid exceptions

catch {spawn /your/favorite/shell} pid; exp_send "telnet 1.2.3.4\r"; 

Technically, to detect failure of the process to spawn or of the communication with the process after successfully spawning, the above commands should really be implemented this way:

if { [catch {spawn /your/favorite/shell} pid] } { ## handle error, which is in the pid variable } elseif { [catch {exp_send "telnet 1.2.3.4\r"} err] } { ## handle error, which is in the err variable } 

Granted, this does increase the resource usage as twice as many processes are started (the host shell and the telnet session that runs within it), but the benefits far outweigh the price. Most notably, unless there is a catastrophic failure in the local system running the host shell (in which case you probably have much larger problems than being unable to communicate with the remote system), you still have an active spawn id with which to communicate. Even if the remote system is still connected, the communication program (in this example, telnet) can be interrupted with a terminate or suspend signal, and the host shell is still available.

The advantageous byproduct of this approach is that spawning a local shell should never hang, and a timeout for expecting a response from the subsequent exp_send command can easily handle situations in which a direct spawn might hang the entire script.

In much the same way as you would handle various shells, login sessions, and other windows that you would use, identification of the system each one represents is important. With each, you would want a prompt that uniquely identifies the session. It is simple enough to define a command prompt that identifies the session, and include it in the expect clause for determination of the process that sent the output.

Once setting the appropriate command prompts, you can use the indirect variable reference “any_spawn_id” as opposed to the direct variable reference “$any_spawn_id” in your expect_before and expect_after statements to detect and identify each session’s output. It then becomes an easy matter to determine whether you have lost a connection to a remote system without also losing the associated spawn id, and you can query the host shell for any information you please, beyond the variables returned from the exp_wait command. All of the same information can be found in the shell and environment variables, and the same spawn id can be used to restart the session.

The difference between using the indirect and direct reference to “any_spawn_id” is that the direct reference is evaluated once, at the entrance to the expect clause. This list is then used throughout the iterations of the clause until the exit criteria are met, and during the iterations, a spawned session may be lost and subsequent iterations will cause an exception because the spawn id no longer exists. The indirect reference is evaluated on each iteration, so when a spawn id is lost (or a new one is created in one of the action branches), the list will be updated to reflect the new active spawn ids. (A more thorough discussion addressing active control of this list will be the topic of a future article).