Instead of using convert, one can use the command mogrify:

mogrify -format jpg *.gif

These commands are from the fantastic suite of software known as ImageMagick.

I’m using this to convert the fantastic icons from famfamfam to gif as IE6 (running as the browser of choice on NHS computers - my target browser sadly) has problems with PNG format graphics. This is for my all singing all dancing WebObjects based medical research / patient record online system that is being deployed this month for the first time.

Please check out Apache Lucene….


while (1) {
System.out.println("Internet explorer 6 must die");
}


Why is internet explorer so ubiquitous across all NHS organisations?

I’m currently fighting CSS, javascript and graphics transparency problems…. ARRRRGHHHHHHH!

To do this, PostgreSQL supports set operations. For instance, I have two tables:

rsdb=# \d t_concept
                                      Table "public.t_concept"
        Column        |          Type          |                      Modifiers
----------------------+------------------------+-----------------------------------------------------
 concept_id           | bigint                 | not null default nextval('t_concept_seq'::regclass)
 concept_status_code  | integer                | not null
 ctv_id               | character varying(10)  | not null
 fully_specified_name | character varying(255) | not null
 is_primitive         | integer                | not null
 snomed_id            | character varying(10)  | not null

rsdb=# \d t_cached_parent_concepts
Table "public.t_cached_parent_concepts"
      Column       |  Type  | Modifiers
-------------------+--------+-----------
 child_concept_id  | bigint | not null
 parent_concept_id | bigint | not null

I want to identify rows in t_concept without a corresponding row in t_cached_parent_concepts using SQL:

Try this!

select concept_id from t_concept except select child_concept_id from t_cached_parent_concepts;

Now this doesn’t explain why my sophisticated caching system is not so sophisticated…..

rsdb=# select count(*) from t_concept;
 count
--------
 388289
(1 row)

rsdb=# select count(*) from (select concept_id from t_concept except select child_concept_id from t_cached_parent_concepts) as tmp;
 count
--------
 142960
(1 row)

Hmm.. that’s a lot of concepts without a cache….

Update 8th March 2010

rsdb=# select count(*) from t_concept;
 count
--------
 2
(1 row)

Now that’s better. But why isn’t there only a single root node with no parent concepts? Further investigation identifies a UK-only SNOMED term that hasn’t been classified correctly. A bug report will be filed with Connecting for Health!

I’m currently writing an introduction to WebObjects and I hope to post these introductory documents both here and to the WebObjects wiki

To execute arbitrary SQL without getting any results back (e.g. executing non-Select SQL statements) use ERXEOAccessUtilities.evaluateRawSqlForEntity()

For example:

	protected static void deleteCache() {
		ERXEOAccessUtilities.evaluateSQLWithEntityNamed(ERXEC.newEditingContext(), "CachedParentConcepts", "truncate t_cached_parent_concepts;");
	}

Deletes all the rows in the table t_cached_parent_concepts

If you wish to get the results back from a custom SQL query and convert the raw rows into enterprise objects, use EOUtilities.rawRowsForSQL

For example:

	/**
	 * Helper function to execute raw SQL and convert the results to enterprise objects
	 * @param ec
	 * @param entityName
	 * @param sql
	 * @param columns
	 * @return
	 */
	public static NSArray executeRawSqlToEO(EOEditingContext ec, String entityName, String sql, NSArray columns) {
		NSArray gids = executeRawSql(ec, entityName, sql, columns);
		return ERXEOGlobalIDUtilities.fetchObjectsWithGlobalIDs(ec, gids);
	}

	/**
	 * Executes a raw SQL statement returning an array of EOGlobalIDs.
	 * The raw SQL should return sufficient columns to allow the raw row to be converted into an
	 * enterprise object.
	 * You will usually need to convert column names to attribute names to allow the raw rows
	 * to be converted properly.
	 *
	 * @param ec - editing context
	 * @param entityName - name of the entity that the raw rows will be converted to
	 * @param sql - the raw SQL statement to be executed
	 * @param columns - array of attribute names to replace the column names that will arrive from the database
	 * @return
	 */
	public static NSArray executeRawSql(EOEditingContext ec, String entityName, String sql, NSArray columns) {
		NSMutableArray gids = new NSMutableArray();
		EOModelGroup modelGroup = ERXEOAccessUtilities.modelGroup(ec);
		EOEntity entity = modelGroup.entityNamed(entityName);
		String modelName = entity.model().name();
		NSArray> rawRows = EOUtilities.rawRowsForSQL(ec, modelName, sql, columns);
		for(NSDictionary row : rawRows) {
			EOGlobalID gid = entity.globalIDForRow(row);
			if (gid==null) throw new NullPointerException("Could not fetch global ID for raw row: " + row);
			gids.add(gid);
		}
		return gids;
	}

Those who work in the NHS will be more familiar with Read codes. These are primarily used in primary care information systems. The origins of terminology systems worldwide is a subject for another article, but Read codes are limited to a fixed hierarchy. Such a hierarchy results in duplicate terms as semantically some terms may be logically related to different terms in the hierarchy.

The structure of SNOMED CT is complex and again, beyond the scope of this article. Those who download the distribution will find a large number of explanatory technical and implementation guidelines. There are essentially three tables representing a Concept, a Description and a Relationship. There is a one to many relationship between Concept and Description - as such, one concept can have multiple descriptions. There is also a more complex relationship between Concept and Relationship with a relationship defined as a join table between related concepts. Each relationship links a “source concept” to a “target concept” and as such has an implicit direction - from parent to child. A relationship can have different types - themselves represented as a concept - the most commonly used concept reflects a “IS A” type of relationship. Thus, each Relationship has three Concept foreign keys - “source Concept”, “target Concept” and “type Concept”.

The greatest difficulty in implementing SNOMED CT in an application is its sheer size. I debated using only a small (manually managed) subset but instead decided to include all SNOMED CT terms available - including those in designated UK-specific subpacks such as that provided to document pharmaceutical products available in the UK. As such, careful optimisation of full-text searches and other queries is essential, with applications decided which subsets to use.

My web-based application supports automatic updating from multiple SNOMED CT distributions and as such grows very large. Even with only the core SNOMED CT distribution there are 388,289 concepts, 1,149,406 descriptions and 1,387,930 relationships.

How does this relate to the use of recursive SQL in PostgreSQL? My naive approach to implementing SNOMED CT was to iterate manually through an object graph of SNOMED CT as provided by a sophisticated object-relational mapping (ORM) - in this case the Enterprise Object Framework (EOF) from WebObjects. This works well for many use cases, but iterating recursively to find all child or parent concepts uses memory and processor time. There are not many use cases when you need to do this - certainly fetching all child concepts of a high-level SNOMED CT concept even as raw SQL is inappropriate. However, there is a need to find all parent concepts of a particular concept to understand its position in a semantic sense. For instance, understanding algorithmically than “Friedreich’s ataxia” is a “Degenerative disease”.

It certainly is possible to generate a cache of all parent concepts and keep this up to date which can sometimes be required if you clearly demonstrate that this is the rate-limiting step using profiling. However, I try to avoid early optimisation and as such, use recursive SQL to rapidly identify all parent concepts.

Recursive queries appeared in PostgreSQL only with the release of version 8.4. The documentation is here.

Here is my table structure:

rsdb=# \d t_concept
Table "public.t_concept"
Column        |          Type          |                      Modifiers
----------------------+------------------------+-----------------------------------------------------
concept_id           | bigint                 | not null default nextval('t_concept_seq'::regclass)
concept_status_code  | integer                | not null
ctv_id               | character varying(10)  | not null
fully_specified_name | character varying(255) | not null
is_primitive         | integer                | not null
snomed_id            | character varying(10)  | not null

rsdb=# \d t_description
Table "public.t_description"
Column          |          Type          |                        Modifiers
-------------------------+------------------------+---------------------------------------------------------
concept_id              | bigint                 | not null
description_id          | bigint                 | not null default nextval('t_description_seq'::regclass)
description_status_code | integer                | not null
description_type_code   | integer                | not null
initial_capital_status  | character varying(10)  | not null
language_code           | character varying(10)  | not null
term                    | character varying(255) | not null

rsdb=# \d t_relationship
Table "public.t_relationship"
Column            |         Type          |                        Modifiers
------------------------------+-----------------------+----------------------------------------------------------
characteristic_type          | integer               | not null
refinability                 | integer               | not null
relationship_group           | character varying(10) | not null
relationship_id              | bigint                | not null default nextval('t_relationship_seq'::regclass)
relationship_type_concept_id | bigint                | not null
source_concept_id            | bigint                | not null
target_concept_id            | bigint                | not null

And this is the SQL to fetch the concept_id of all parent concepts of the concept Friedreich’s ataxia (concept ID 10394003).

-- SHOW ALL PARENT CONCEPTS FOR FRIEDREICH'S ATAXIA (10394003)
with recursive parent_concepts(concept_id) as (
select t0.concept_id from t_concept t0 inner join t_relationship t1 on t0.concept_id = t1.target_concept_id
where (t1.relationship_type_concept_id = 116680003 and t1.source_concept_id = 10394003)
union all
select t0.concept_id from t_concept t0,t_relationship t1, parent_concepts pc where t0.concept_id = t1.target_concept_id and
t1.source_concept_id = pc.concept_id and t1.relationship_type_concept_id = 116680003
)
select distinct(concept_id) from parent_concepts;

You might ask, why am I limiting the parent concepts to those linked by relationships with the concept type “116680003″? This is because that reflects the IS A relationship type concept.

A quick hack to get the fully specified names of the concepts…

rsdb=# select fully_specified_name from t_concept, (with recursive parent_concepts(concept_id) as (
rsdb(# select t0.concept_id from t_concept t0 inner join t_relationship t1 on t0.concept_id = t1.target_concept_id
rsdb(# where (t1.relationship_type_concept_id = 116680003 and t1.source_concept_id = 10394003)
rsdb(# union all
rsdb(# select t0.concept_id from t_concept t0,t_relationship t1, parent_concepts pc where t0.concept_id = t1.target_concept_id and
rsdb(# t1.source_concept_id = pc.concept_id and t1.relationship_type_concept_id = 116680003
rsdb(# )
rsdb(# select distinct(concept_id) from parent_concepts) as temp where t_concept.concept_id = temp.concept_id;

gives these results:

                    fully_specified_name
---------------------------------------------------------------
 Disorder of head (disorder)
 Clinical finding (finding)
 Cerebellar disorder (disorder)
 Disorder of body system (disorder)
 SNOMED CT Concept (SNOMED RT+CTV3)
 Spinal cord disorder (disorder)
 Disorder of nervous system (disorder)
 Disorder of back (disorder)
 Encephalomyelopathy (disorder)
 Head finding (finding)
 Degenerative disorder (disorder)
 Disorder of body cavity (disorder)
 Disease (disorder)
 Disorder of spine (disorder)
 Finding of back (finding)
 Finding of head and neck region (finding)
 Finding by site (finding)
 Disorder of spinal region (disorder)
 Disorder of the central nervous system (disorder)
 Degenerative disease of the central nervous system (disorder)
 Spinocerebellar disease (disorder)
 Finding of body region (finding)
 Degenerative brain disorder (disorder)
 Disorder by body site (disorder)
 Finding of spinal region (finding)
 Disorder of brain (disorder)

Note: I am not limiting concepts by whether they are “active” or not, as this is something I switch on and off at the application level.

Welcome back!

The Problem

You need to take care of PID (patient identifiable data) but you don’t have the time to learn the ins and outs of encryption schemata. Even if you use your USB key to transfer the data and then delete it, if you were to lose the key and it fell into unscrupulous hands it possible that someone could read even the deleted files.

The Solution

TrueCrypt is an open source, free application that allows you to use tried-and-tested NHS approved encryption methods. It can be set up to automatically load when you plug the USB key into the PC.

How Does it work?

When you plug a USB stick into a PC it usually mounts the drive as a drive letter (F in this example).  When TrueCrypt is running and you have entered your password you will be greeted by a second drive letter, this is the encrypted one. The files on this new drive will only be decrypted and visible to you once you have typed in your password. Existing files on your USB key will not encrypted, only ones added to the new drive letter. You have to store the encrypted files in a container I have called ’secretfile’, but you could call it anything you wish. If someone found your USB key the encrypted data will all be enclosed safely within your file container ’secretfile’  which someone could see exists but they would not be able to view the actual contents of the file without obtaining your password.

The How-to

1. Download the newest version of TrueCrypt from here.

3. Double-click on the downloaded file and it will run. You will be warned that the file has come from the internet, this is OK.

4. Accept the default terms and conditions and then select extract rather than install. You do not want to install the program on your computer as this would probably be in breach of your local IT guidelines.

5. Extract the files to your desktop, leave the box ticked that allows you to open the folder once extracted.

6. This new folder will pop-up. Plug your USB key in and note what drive it comes up as. (eg F:)

6. Run the program called “TrueCryptFormat.exe” by double-clicking on it.

7. Select “Create File Container”, then “Standard TrueCrypt Volume”

8. Where it says Volume Location, enter “F:secretfile” (substituting the drive letter of your USB key if it is not F.) This is the container in which your encrypted files will be stored.

9. Keep the standard options on the next screen, that of AES and RIPEMD-160.

10. You need to tell the program the size of the encrypted drive you want to create. The container produced will use up that amount of space, so if for instance you created a 100MB encrypted drive, this would take up 100MB of un-encrypted space on your USB key, even if the encrypted area was ‘empty’. You may wish to leave room for un-encrypted data. The amount of free space should be displayed to help guide you.

11. Next enter your password, the strength of your password is critical, see below for details on this.

12. Click next and your encrypted drive should now be created, this may take a few minutes depending on the size of your key and speed of your computer.

13. Once complete close the window.

14. Now open the file called “TrueCrypt.exe”, go to the “Tools” option on the menu, then “Traveller Disk Setup”.

15. Under file settings specify the location F:\truecrypt, (changing the drive letter F to whatever the current drive letter is for your USB key). Where it asks for autorun configuration select “Auto-mount”. In the box where it says “file to mount” enter secretfile .

16. Click create, when done, close all windows that you have opened and remove the USB key.

How to use

1. Insert USB key. You should get the following popup, select “Mount Truecrypt Volume” and then enter your password at the next prompt. A window should pop up with a new drive letter but there will be no files, yet.

2. To copy files to the new encrypted area just drag and drop as if it was a USB key.

3. To remove the USB key first dismount the key by clicking on the TrueCrypt logo on the right hand side of the status bar and select “Dismount all devices”

What is Encryption?

When you use your password to get into a website it is used to compare it with a list of usernames and paswords to confirm your identity and let you in, that means if you lose your password your supervisor can look it up and remind you of it.

Encryption is different. Think of encryption as your data being scrambled using your password, and not even your supervisor or boss can access it. This means if you lose your password then you can’t get your data back.

How secure is AES-256 Encryption?

It is secure… secure enough for the US government for use with  “Top Secret” information. Your encryption, however,  is only as strong as your password.

Password Strength

The easiest way to “crack” or hack your encrypted USB key would be to try common passwords or run through the dictionary. The longer and more complicated your password, or pass phrase the safer your data. Try using a password with at least 8 letters, use upper and lower case and numbers. Use more than one word or use a combination of your favourite non-English language words. This is an example of a strong password. “Love”, “Password”, “Arsenal”  or your name are all examples of weak passwords.

eg Ouvert1971NerdBoy

Potential Problems

Your local IT department may not allow you to use programmes that run off the USB key because you may not be running as an “administrator”

A short introduction to Encryption

Encryption is a way of scrambling the data within your files to prevent a third party eavesdropping. Encryption can be symmetric, where the same password is used for encrypting and decrypting the data. Asymmetric encryption is the method using the concept of public and private keys eliminating the need to transfer a password between the 2 parties.

Encryption within Windows

Windows supports the use of ZIP or compressed folders, however it is hampered by the use of weak encryption, that can be brute-forced using a myriad of programs in minutes to hours with a modern PC. Winzip, WinRAR, 7-Zip and others offer the more secure AES standard. AES, like any encryption scheme, can be broken given time, but this is likely to run into hundreds if not thousands of years.

Installing 7-Zip

1. Download the most modern non-beta version from here.

2. Install using default options.

Using Z-Zip

As default 7-Zip installs itself to with “explorer extensions” that allow you to right click on items on the desktop or in windows explorer to compress files. Z-Zip has its own file format 7z which is more efficient at compressing files than the standard zip extension, but this will mean the person you are sending the file to will also have to use 7zip. Using the zip format will enable people using other programs to de-compress the file.

1. Right click on the files or folder you wish to compress and encrypt.

2.Firstly change the Archive format to Zip (or use 7z if both you and your intended recipient use 7zip), then change the encryption method to the robust AES-256, thirdly enter your password. Then click OK. The rest of the options can be left as default.

Decryption

Simply right-click on the file, select extract then enter the password when requested.

Important Notes

Your data will still be vulnerable to a “dictionary attack” where an attacker cycles through common passwords such as “Love”, “Password” etc. To protect from this simply make your password a mixture of numbers and letters rather than simply a single English word. One way to protect from this would be to use a random generated password or use asymmetric encryption.

For a comparison of software for compressing files see here.

The technique I will describe can be used to create professional looking job application forms whether the form is printed or in PDF format.

Adobe Acrobat (not Acrobat-Reader) is the professional product that allows you to edit/fill in forms from Adobe, it currently retails at around 270US$.

Foxit Reader is free to download and use and relatively lightweight at 2.3MB download compared to 33.5MB download with Acrobat Reader. Foxit produce a PDF Editor which allows editing text and images within PDF files, but to fill in forms you only really need the “Plus pack” which is currently 39 US$. You can use all Foxit versions in evaluation mode on a trial basis but this will leave a watermark on your form.

Create PDF from Printed Form

1. In Windows XP the required software is PDFCreator, details of which can be found here. This is only needed if you have a paper form, my technique is to scan in the pages in 300dpi format. Then select the pages in the correct order and right click to “Print using Photo Printing Wizard.” Select PDF Creator as the printer and you will have the required pages as a PDF file.

2. When you have Foxit Reader installed click

Tools>>Typewriter>>Typewriter

This will allow you to type directly onto the form. The advantages of this method over using a image editor and typing over the jpeg scan is that Foxit will allow you to save the form and come back and edit the text at a later date as well as offering a spell checker.

Problems with Foxit and Typewriter: Landscape Forms

There is an annoying bug in Foxit,  acknowledged here and here but not fixed by the software house. When you try and use the typewriter in a landscape document the text comes out 90 degrees out.

This makes filling all those landscape forms impossible without resorting to Acrobat Professional.

The Hack

1. Select Print from the File Menu in Foxit.
2. Select PDFCreator as your Printer
3. Click on Properties
4. Then Advanced
5. Select Papersize “Postscript Custom”
6. Select Edit Custom Paper Size
7. Assuming you are using A4 paper enter 297 * 210mm
8. Change paper feed to “Short Edge First”
9. Print to PDF.

You should now have an identical looking PDF but the typewriter will work correctly. If the typewriter function is still 90 degrees out then change paper feed to “Long Edge first”