Doodads and Whatnot

The following is a letter I wrote to the Minister for Broadband, Communications and the Digital Economy which I penned in response to the Australian Government’s unworkable internet filtering scheme.

Senator the Hon. Stephen Conroy
Minister for Broadband, Communications and the Digital Economy
Level 4, 4 Treasury Place
Melbourne
Victoria, 3002

Dear Minister,

As an employee of an Australian company which provides secure internet based communications and transaction exchange software, and an internet user I am deeply troubled by plans to introduce mandatory internet filtering.

Such filtering has been shown in countries where it is used to be largely ineffective. The massive and ever growing amount of information available means that it will never be possible to eliminate inappropriate content from the internet. It would be better to focus on efforts such as law enforcement and education which would help reduce the instances of this content being created or sought in the first place.

The size of the problem will mean that only known and high profile items will make it to the list of inappropriate content in a timely fashion. The ability to effectively filter only these targets will make us a target of ridicule. It has become a not uncommon activity to show a ‘Chinese view’ of a website beside a ‘normal view’ of the same site. I doubt it will do Australia’s reputation good to be added as the third party in these comparisons.

The technical realities of the problem will adversely affect the cost and performance of internet services in Australia. As well as being a nuisance to Australian internet users it will also force internet based services operating out of Australia to relocate hosting and technical operations to foreign countries.

Furthermore, as an employee of an Australian company which provides internet based secure transaction exchange services, I have serious security concerns with the implementation of this plan. The sales of our communications products rely in large part on the level of privacy and security we can offer our clients. The use of ‘man in the middle’ attacks, required by the tested filtering software for Secure Sockets Layer connections, will remove our ability to assure customers that their security has not been compromised. The correct operation of the proposed filters will be indistinguishable from a hostile party performing ‘man in the middle’ attacks.

While Australian customers may in time come to accept this as unavoidable (since online banking and other operations will suffer the same problem) it will make our products unmarketable to foreign customers without moving our hosting and server infrastructure overseas. This will make a number of our Australian staff redundant.

I regret that the nature of the plans became clear to me too late for it to have been a factor in my voting choices for the recent election. I do not doubt that these plans are ultimately motivated by good intentions but I fear that the originators have bought snake oil and wish to continue applying it to gain the favour of a few ‘key’ politicians.

Please consider how the money for the filtering scheme may be better spent. I would suggest that increasing funding to education and law enforcement would better meet its objectives and the requirements of all Australians.

Sincerely,

Name Removed.

I received the same form letter reply as everyone else who wrote to the minister.

Ruby-serialport doesn’t play well with RubyGems. Trying to require ’serialport’ when using RubyGems results in:

NameError: (eval):1:in `private_class_method': undefined method `create' for class `Class'
        from (eval):1
        from (eval):1
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
        from (irb):2

The quick fix to get around this is by doing:

Kernel::require 'serialport'

It would be nicer if we could avoid the conflict with RubyGems. It would be even nicer if the serialport library was itself packaged as a gem. Turns out this has been done by aaronp but it doesn’t take advantage of GitHub’s automatic gem builder and doesn’t build correctly using GCC.

I’ve cloned and tidied the git repository. This gem version should build the needed native extension and work on both POSIX compatible and Windows operating systems.

Installing the serialport gem

If you haven’t configured GitHub as a gem source yet you will need to run the following command:

gem sources -a http://gems.github.com

Then, you can install the gem as normal:

sudo gem install toholio-serialport

Unlike the official ruby-serialport extension, this gem works with RubyGems (duh!). When you want to use it you should only need to do the following:

require 'rubygems'
require 'serialport'

On a side note: git and GitHub rock. Forking, contributing to and merging projects should be this simple.

I’m falling in love with GitHub. To see if this love will last I’ve moved my LocoNet server gem to GitHub from RubyForge. The RubyForge gem still exists (and will continue to do so) but changes will be happening at GitHub (probably permanently).

To get the new version of the gem you will need to have GitHub configured as a gem source. So if you haven’t already, you must run:

gem sources -a http://gems.github.com

To get the latest version of the gem run:

sudo gem install toholio-nickel-silver-server

It’s been hot here in Melbourne so rather than actually leaving the house to do anything I’ve been sitting in front of a fan mucking about with my computer. Yesterday afternoon I created a Ruby implementation of a LocoNetOverTCP server. I’ve decided to call it Nickel-Silver because that’s the common name of the alloy most model railroad track is made of and some trifling project has already used the name “Ruby on Rails”. Here’s a quick list of Nickel-Silver’s features:

• Implements all of LocoNetOverTCP protocol version 1
• Multithreaded (inherits from GServer)
• Easily extendable interface for LocoNet hardware devices
• Only about 100 lines of code (according to egrep -c -v “^[ ]*$|^[ ]*#”)
• Base implementation uses only the standard library
• Included “driver” for the LocoBuffer-USB hardware uses a single extra package, ruby-serialport
• It’s written in Ruby

Hardware setup

Currently the only “driver” provided connects to a LocoNet network via a LocoBuffer-USB. So a complete hardware setup looks something like the following diagram.

LocoNetOverTCP clients connect to the server running on the computer which acts as a proxy to the LocoNet. This removes the need for each computer to have its own LocoNet hardware interface. It also removes the need for the clients to be physically located near layout.

Server setup

Assuming we’re going to use the included LocoBuffer-USB “driver” we need to do the following to start a server.

1. Install ruby-serialport if you haven’t already.
2. Install the nickel-silver-server gem
3. Open a new ruby script in your favourite editor or start IRB.
4. Determine the serial port your LocoBuffer-USB is connected to.
5. Create an instance of the LocoBufferUSB “driver” class.
6. Create an instance of the server with the “driver” as a parameter.
7. Run the script

To install the server’s gem open a terminal and run:

gem install nickel-silver-server

That should get everything that’s required with the exception of ruby-serialport which you must install manually.

My LocoBuffer-USB is connected to my computer via the virtual serial port /dev/cu.usbserial-FTQ1P4JC so a complete script for my server would look like the following.

#!/usr/bin/ruby
 
require 'rubygems'
require 'nickel-silver-server'
 
# connect to a LocoBufferUSB on the virtual serial port /dev/cu.usbserial-FTQ1P4JC'
interface = NickelSilver::Server::Interface::LocoBufferUSB.new( '/dev/cu.usbserial-FTQ1P4JC' )
 
# create a server using the default port (i.e. 5626, 'loco' spelt on a phone keypad)
# using our freshly connected LocoBuffer-USB
server = NickelSilver::Server::LocoNetServer.new( interface )
 
# start the server
server.start
 
# wait for the server to stop before exiting
server.join

Documentation

Currently this post is the only documentation not contained within the gem. You can view the gem’s rdoc documentation at http://nickel-silver.rubyforge.org/nickel-silver-server/rdoc/

Sending trains down the track

Until boredom strikes again there isn’t a Ruby throttle program that can interface with Nickel-Silver. Instead we can use the excellent Java program JMRI to talk to our trains.

JMRI already knows about LocoNetOverTCP servers right out of the box. Starting JMRI and looking at the preferences we can wrangle things to look like the following.

Now we’re ready to go. Selecting Tools > Throttles > New Throttle gives us a throttle window where we can enter a locomotive address and make it move.

Implementing a new “driver”

The only LocoNet computer interface I have access to is a LocoBuffer-USB. I’d highly recommend one to anyone who is interested but acknowledge that there are many people with other hardware.

As long as you can send and receive LocoNet packets via your hardware in Ruby (or in any language that may be used for Ruby extensions) you can write a “driver”.

The only things a “driver” needs to be able to do are:

• Store incoming packets as FixNums representing bytes in a buffer array
• Send outgoing bytes (represented as FixNums in a buffer array) to LocoNet
• Use a Mutex to lock access to the buffers when in use (remember Nickel-Silver is multithreaded)
• Provide a method that causes your interface to start collecting packets

The interface is simple. Only the following public methods are needed:

• Accessors for input_buffer, output_buffer and io_mutex
• run() which starts buffering

How you do this is up to you but you can take a look at LocoBufferUSB.rb to get an idea of how it might be done. A stub driver might look like the following…

class SomeLocoNetInterface
  attr_accessor :input_buffer, :output_buffer, :io_mutex
 
  def initialize
    # these may be modified at any time by the server
    @input_buffer = []
    @output_buffer = []
 
    # only make changes when locked using @io_mutex
    @io_mutex = Mutex.new
  end
 
  def run
    loop do
      # get incoming bytes
      if byte_waiting?
        @io_mutex.synchronize do
          # byte getting code here
          @input_buffer << get_byte
        end
      end
 
      # send outgoing bytes
      until @output_buffer.empty? do
        @io_mutex.synchronize do
          # send a byte
          send_byte( @output_buffer.shift )
        end
      end
    end
  end
 
end

Remember that either buffer may be empty or already containing data and that you must lock the mutex for the minimum amount of time possible.

Isn’t there already a LocoNetOverTCP server project?

Yes there is! It’s called LbServer and is hosted over at SourceForge.

If you’re looking for a mature LocoNetOverTCP server that’s probably the place you should start. Unlike my implementation there are more people than just me using it and they probably have more bugs fixed than I’m likely to notice any time soon.

Has the boss just discovered wikis? Does he now want to be able to create links between records in your intranet applications? Need to quickly add wiki functionality to your models and views so you can get back on with your life?

That’s what happened to me recently. Since the wiki code was almost exactly the same for all of the applications that needed changing, I’ve put the code into a plugin. Hopefully someone else will find it useful.

Installing the plugin

The plugin is installed in the same manner as any other. Fire up your favourite terminal and navigate to the root of your rails application. Then run the following command:

script/plugin install http://hazy.stupor.org/svn/has_wiki_field/trunk/has_wiki_field/

That should be all that is required.

Using the plugin

The following example should get you up and running quickly but make sure you read the efficiency note below.

Lets say you have a Person model that has name and bio attributes. We would like to be able to link to other people from within a bio. So in Allan’s record we might want the bio to read “This man is friends with [[Betty]]” and have [[Betty]] automatically change into a link to her record when it is displayed.

We, start by adding has_wiki_field to the Person model:

class Person < ActiveRecord::Base
  has_wiki_field :field => "name", :sql_match => "LIKE"
end

The :field option specifies what field should be matched by the text in [[links]]. The :sql_match option specifies how the records should be matched. :sql_match defaults to “=” but using “LIKE” will allow case insensitive links.

Now we need to change show.html.erb for the people controller so that links are generated from the bio.

Where previously we might have displayed a bio using:

<%= h @person.bio %>

We will now use the wikify method, passing it the name of the attribute to wikify and a code block which generates the html for links. A code block is used to allow flexibility in link generation. The following block simply calls link_to but you could, for example, write a block which looks up an image associated with your model and displays that instead.

Anyway, for our Person model we add the following to the view:

<%= @person.wikify(:bio) do |name, object|
    if object.nil?
      h name
    else
      link_to name, object
    end
  end
%>

In a real application this should be put into a method in people_helper to help keep things tidy.

That’s it! Where [[Some Name]] appears in a Person’s bio it will be turned into a link to the person with the name ‘Some Name’.

Remember to read the important efficiency note section below before using this in a production environment.

An important note about efficiency

The default method used for finding which object a [[link]] points to is not efficient. For sites that receive more than a tiny amount of traffic it should be replaced by your own model specific method.

The default method takes the contents of each [[link]] and queries the database for the first record that has ‘link’ as the value for the field specified in the model, I.e.

find(:first, :conditions => ["somefieldname = ?", key])

This will obviously require a database look-up for every link which could slow your site down tremendously for pages containing many links.

To fix this you must provide your own method to match objects with the keys extracted from [[links]]. The class method to override for this is object_for_key(key).

For example, if we have a class called Person we might replace the object_for_key method like so:

class Person < ActiveRecord::Base
  has_wiki_field
 
  def self.object_for_key( key )
    # some super fast method for matching the key
    # from a link to a person object, goes here...
    #
    # return the Person object that matches or nil if there is no match
  end
end

Naturally, it’s up to you to manage the data used by your object_for_key method. wiki_options[:field] and wiki_options[:sql_match] will be available to you in this method if they were set as options to has_wiki_field.

Future plans

The plugin doesn’t do much currently. If I’d only needed it for a single application there’s a good chance it wouldn’t have made it into a plugin at all. It has provided me with a reasonable starting point for my applications and hopefully it’ll get better as time goes on.

Some ideas for future versions:

• Add a controller action which takes a key and redirects to the appropriate record or creates a new record with the needed field already filled.
• Using the new controller method above, allow links of the form [[Some Key#Model]] where “Model” specifies what type of thing links are pointing to. Ta-dah! Now we can link between all the models in an application.
• Implement some basic object_for_key methods for common situations. The plugin should attempt to gently direct programers towards these methods.
• Implement a basic link caching mechanism. When a has_wiki_field model is saved links should be extracted and stored in a table. Then all current links for a model can be loaded in a single database transaction.

Recently I needed to set-up an application to keep track of regular safety inspections for power tools and other equipment. The application is very trivial but my client was quite pleased with one particular feature which wasn’t actually requested.

The “bonus” feature was an iCalendar feed which allows the client to have his computers subscribe to the feed and keep up to date with what equipment needs inspecting and when. Very swish and, thanks to Rails and a nice gem, very easy.

So how’s it done? Easy!

Lets start by installing the icalendar gem. Open a terminal and install it as normal:

user@host railsapp$ gem install icalendar

Change to the vendor directory of your Rails application and unpack the gem:

user@host railsapp$ cd vendor/
user@host vendor$ gem unpack icalendar

Make sure it will be loaded by adding it to the end of config/environment.rb in your Rails application:

require 'icalendar-1.0.2/lib/icalendar'

Now we’re ready to add a calendar feed to our application. Lets say we have a table that contains the names and scheduled service dates of some tools. The client wants, or is going to get, a feed which marks each tool needing service in their calendar when their servicing is due.

Generating the iCalendar data is simple. The following method from tool_controller creates an event for each tool and inserts it into a calendar. The to_ical call at the end of the method get the calendar as a string which can be served to the user. Note that this method assumes the @tools variable has already been set (you could always add a call to Tool.find(:all) at the start of the method if needed).

def generate_ical
  cal = Icalendar::Calendar.new
  @tools.each do |tool|
    # create the event for this tool
    event = Icalendar::Event.new
    event.start = tool.inspection_date
    event.end = tool.inspection_date
    event.summary = "Service of " + tool.name + " is due."
 
    # insert the event into the calendar
    cal.add event
  end
 
  # return the calendar as a string
  cal.to_ical
end

Next, we add the icalendar feed to respond_to in the tools_controller’s index method. We’ll make this call the generate_ical method to get the calendar data.

def index
  @tools = Tool.find(:all)
 
  respond_to do |format|
    format.html # index.html.erb
    format.xml  { render :xml => @tools }
    format.ics  { render :text => self.generate_ical }
  end
end

And now you should be able to open your calendar application and subscribe to the feed straight from the tools controller using a URL like http://example.com/tools.ics

I’ve frequently needed a very simple search method that allows partial matches for any combination of fields in a table. The following snippet shows how an ActiveRecord derived object might have a filter to allow for selection of records given a set of partial values.

class YourClass < ActiveRecord::Base
   def self.filter( partial_values )
    # don't bother at all if there is no search object
    return find(:all) unless partial_values
 
    con_string = ""
    con_array = []
 
    # build collection of conditions
    partial_values.each do |key,value|
      if value != "" then
        con_string += " and " if con_array.size > 0
        con_string += "#{key} LIKE ?"
        con_array << "%#{value}%"
      end
    end
 
    # construct the actual conditions array
    conditions = [con_string]
    con_array.each { |item| conditions << item }
 
    find(:all, :conditions => conditions)
  end
end

To use this you would obtain a set of search parameters, one for each filterable column, and pass it to YourClass.filter as a hash to get the matching rows.

So if you had a table with title and category columns you might create a page containing a form to collect partial value for filtering. When creating the form, assuming you will use Rails’ form helpers, the fields_for :collection function is nice as it will allow for easy collection of a hash for the field values.

<% form_tag your_object_path, :method =>"get" do %>
	<% fields_for :partial_values do |f| %>
		Title contains: <%= f.text_field :title %><br />
		Category contains: <%= f.text_field :category %><br />
	<% end %>
	<%= submit_tag "Filter items", :name => nil %>
<% end %>

Then once the form is submitted you would get the appropriate results in your controller using:

@your_objects = YourClass.filter( params[:partial_values] )

I recently needed to install all the PIC programming tools I use on several Macs. Since people often ask for help installing these kinds of things I’ve decided to put together an installer package containg all the tools I use for the majority of my PIC’n needs. Hopefully someone else will find it useful.

The installer contains universal binaries for:

• Jeff Post’s PK2 v2.01 - for operating the PICkit2 programmer hardware
• GPUTILS v0.13.4 - assembler, linker, etc
• sdcc v2.6.0 - C compiler

I have only tested the installer and resulting binaries on Macs running 10.4.7 (x86 and PPC) so if you try it under a different version of OS X, please let me know how it works.

The following installer contains all three packages (you can choose which ones to install).

PIC toolbox (8.1MB)

IMPORTANT:The installer wont reconfigure your PATH, MANPATH or SDCC_HOME environment variables automatically. The things to add are:

• /usr/local/bin to PATH
• /usr/local/man to MANPATH
• SDCC_HOME=/usr/local

For most people these can be configured by opening a terminal and running the following commands:

echo 'export PATH=/usr/local/bin:$PATH' >> ~/.bash_profile
echo 'export MANPATH=/usr/local/man:$MANPATH' >> ~/.bash_profile
echo 'export SDCC_HOME=/usr/local' >> ~/.bash_profile

For a long time I’ve been frustrated by the lack of tools for programming micro-controllers (PICs specifically) using MacOS X. Most bit banging serial programmers wont work with a USB-to-Serial adapter, though I’m told some commercial ones do. There are some USB programmers available but until recently I wasn’t able to find one that was compatible with the 16F88 (PDF, datasheet) and had MacOS drivers available.

Enter the PICkit 2

After finally getting tired of SSHing into a Linux box in another room every-time I wanted to write to a PIC I started looking for a USB based programmer with support for the 16F88. I was pleasantly surprised to see that the firmware for Microchip’s PICkit2 had been updated to work with the 16F88 (as of version 1.20). The PICkit is a nicely made unit (about 10cm x 4cm x 1cm) that can be bought, with a low pin count demo board included, directly from Microchip for under AU$100.

Using PK2 with MacOS X

Version 2.00 of PK2 works well well with MacOS. Older versions may work too but I haven’t done much testing with them. The program is easy to build and there is a native HID report which means that there is no need for MacOS users to muck about with libusb. Very nice indeed. When building make sure you use the osxhid Makefile target (unless you really want to use libusb). Once built and installed PK2 can be used to update the programmer’s firmware. Everything should be good to go and the programmer should be able to see the PIC it is connected to like so:

tobie@Tea-Chest ~ $ pk2 -device

PK2 version 2.00 - 2006/07/02
 pk2 -device
Communication established. PICkit2 firmware version is 1.21.0

Device ID 0x0760
PIC16F88 Rev 5 found
  Family:         Midrange
  Program size:   0x1000 (4096) words, width 0x3fff
  Eeprom size:    0x100 (256) bytes
  ID location:    0x0
  ID size:        0x4 (4) bytes
  Device ID       0x0760
  Write burst     4
  Program command O
  Program mode    n
  Program timing  n
  Data timing     d
  Erase mode      0
  CP mask         0x0834
  Bandgap mask    0x0000 0x0000
  Config mask     0x3fff 0x0003
  Save osccal     0
  Save bandgap    0
  Command table   63 00 02 03 04 05 06 08 18 17 1f 1f ff ff ff ff

Handy resources

The PICkit 2 comes with a CD-ROM which contains lots of examples and code for the supplied PIC16F690 including several lessons which should help someone not familiar with PIC programming to get a good grasp on the basics. For other PICs there are many forums and mailing lists on the internet including Microchip’s own. A quick Google search should point you in the right direction.

For updates and discussion of PK2 there is the pickit-devel Google group.