Jim's Blog Ramblings about novels, comics, programming, and other geek topics

5Oct/0715

How to convert a street address to longitude and latitude (geocoding) via web services

Google AdSense

Several years ago, I had a school project that required converting some addresses to longitude/latitude coordinates. We then took the coordinates and placed them on a street map for a mobile device application. The addresses came from a "Yellow Pages" like system lookup service which provided a phone number, street address, name, and business category. But the Yellow Pages service didn't provide a geographic coordinate.

First, I tracked down a free and very useful web service that provides this functionality. The web site GeoCoder.us provides a few different free web services including geocoding on street address, ZIP code, or city and state. GeoCoder.us offers SOAP, XML-RPC, REST-ful RDF, and REST-ful CSV formats.

We'll use the web service the requires a street address, so we can get a more accurate coordinate.

string street = "1060 west addison street";
string city = "chicago";
string state = "il";

The GeoCoder.us URL requires a specific parameter format. Here we append the street, city, and state to the base URL.  Then we create and load the URL into an XML document. By passing the URL to the XmlDocument Load method, it will handle calling and reading the web response.

string geocoderUri = string.Format(
    "http://rpc.geocoder.us/service/rest?address={0},{1},{2}",
    street, city, state);
XmlDocument geocoderXmlDoc = new XmlDocument();
geocoderXmlDoc.Load(geocoderUri);

Next, we need to add a namespace manager since GeoCoder.us uses the W3.org Basic Geo (WGS84 lat/long) vocabulary.  Below is an example of this vocabulary and the actual response that was loaded into our XmlDocument.

<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" 
         xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" 
         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<geo:Point rdf:nodeID="aid62658379">
<dc:description>1060 W Addison St, Chicago IL 60613</dc:description>
<geo:long>-87.655788</geo:long>
<geo:lat>41.947372</geo:lat>
</geo:Point>
</rdf:RDF>

The namespace manager is needed so that we can select the long and lat XML nodes. Below we select the two XML nodes and parse them.

XmlNamespaceManager nsMgr =
    new XmlNamespaceManager(geocoderXmlDoc.NameTable);
nsMgr.AddNamespace("geo",
    @"http://www.w3.org/2003/01/geo/wgs84_pos#");

string sLong = geocoderXmlDoc.DocumentElement.SelectSingleNode(
        @"//geo:long", nsMgr).InnerText;
string sLat = geocoderXmlDoc.DocumentElement.SelectSingleNode(
        @"//geo:lat", nsMgr).InnerText;

double latitude = Double.Parse(sLat);
double longitude = Double.Parse(sLong);

Console.WriteLine("Lat: " + latitude + " Lon: " + longitude);

Finally, the coordinates are printed to the console and appear as:

Lat: 41.947372 Lon: -87.655788

Update: 10/6/2007 Once you have the coordinates, you can perform radius searching. For example, you can query for all records within 5 miles of each other or find the nearest record to another record. Troy DeMonbreun provides an example of how to calculate the distance between two geographic coordinates using a SQL Server UDF (User Defined Function). The GeoCoder.us blog also discusses how to calculate distances using two geographic coordinates.

Update 2: 10/19/2007 Yahoo also provides a very nice Web Service for geocoding at Yahoo! Maps Web Services - GeoCoding API. Yahoo uses the same REST method for the accessing the Web Service, but with separate URL parameters for city, address, and state. Yahoo's XML response doesn't use namespaces and won't require the use of a NamespaceManager. You'll be able to just use SelectSingleNode(@"//Longitude") and SelectSingleNode(@"//Latitude") to collect the geographic coordinates.

 

kick it on DotNetKicks.com

4Oct/076

How to resize and/or convert large image files

Google AdSense

If you happen to work in the GIS industry, you'll routinely come across large image data sets. Some TIFF files might be several hundred megabytes each.  Although, these files look great when printed and should be kept as the source imagery. They are a bit too large for daily use and especially if you are zoomed out to a scale where multiple images are tiled onto the same screen.

This is where you can make lower resolution and lower quality images and I'll demonstrate how simple it is to do using .NET System.Drawing.Imaging classes. First, we'll make a console application so we can run it unattended at night or use batch files to call and execute the application.

The application entry point will begin as:

static void Main(string[] args)

Our main function will contain four (4) arguments and be called such as "imgcnvt.exe [drive:][path] [drive:][path] [JPEG|TIFF] [scaleFactor]." For this example, we'll support converting to JPEG and TIFF formats and use a scale factor to determine if any resampling will occur.

ImageFormat outFormat = ImageFormat.Jpeg;            
switch (args[2].ToUpper())
{
     case "JPEG": case "JPG":
         outFormat = ImageFormat.Jpeg; break;
     case "TIFF": case "TIF":
         outFormat = ImageFormat.Tiff; break;
     default:
         Console.WriteLine("ERROR!");
         return;
}
float scaleFact;
if (!float.TryParse(args[3], out scaleFact))
{
      Console.WriteLine("ERROR! The scale factor is invalid.");

return;
} if (scaleFact <= 0f || scaleFact > 1f) { Console.WriteLine("ERROR! Please enter a value >= 0 and <= 1."); return; } string currentDir = args[0]; string outputDir = args[1]; if (!Directory.Exists(outputDir)) Directory.CreateDirectory(outputDir);

Now we have some basic error handling and our four variables (outFormat, scaleFact, currentDir, and outputDir) defined and ready for processing. Passing a scale factor value of 1 would result in no scaling and a scale factor of 0.5 would change a 5000x5000 pixel image to 2500x2500 pixels.

Next, we'll get a list of the images files in the current directory and loop through each file and call a Process TIF function using the filename and other settings as arguments.

 foreach(string filename in Directory.GetFiles(currentDir, "*.tif"))
     ProcessTif(filename, outputDir, outFormat, scaleFact);

In the ProcessTif function, we will first create a string that will hold the output file name. This file name will be the concatenation of the outputDir and the filename without extension (we will append the extension associated with the outFormat). For example, if the output is a ImageFormat.Tiff the output file name will be:

string outImage = Path.Combine(outputDir, 
               Path.GetFileName(filename));

Some of the images that I was processing were over five hundred megabytes and loading those into memory resulted in a few memory and processing problems, so I needed to specifically call the garbage collector before I began to process each image to clear out the old image. Calling the garbage collector fixed my memory problems, but you shouldn't ever need to do this.

System.GC.Collect();
System.GC.WaitForPendingFinalizers();

Okay, here's the important pieces of the code. We will now begin the processing of the image and change its format and/or scale.

using (Image fullSizeImg = Image.FromFile(filename))
{
     Image.GetThumbnailImageAbort dummyCallBack = 
          new Image.GetThumbnailImageAbort(ThumbnailCallback);
     int width = (int)(fullSizeImg.Width * scaleFactor);
     int height = (int)(fullSizeImg.Height * scaleFactor);
     using (Image lowResImg  = fullSizeImg.GetThumbnailImage(
             width, height, dummyCallBack, System.IntPtr.Zero))
     {
          lowResImg.Save(outImage, outFormat);
     }
}

Here we are loading the source image from file (filename) to fullSizeImg variable. Next, we setup a ThumbnailImageAbort delegate and calculate the output image's width and height. Then we call the GetThumbnailImage function using the output image dimensions, delegate, and callback data. Lastly, we call Save and provide the filename and image format.

For the ThumbnailImageAbort delegate, I just used a dummy function that always returns false such as:

private static bool ThumbnailCallback()
{ return false; }

Of course, there's much more error trapping and exception handling that you should add to these code snippets, but this should point you in the right direction.

By processing our image data using this technique we were able to create several hundred JPG images (with JGW files) that were between 5 and 15 megabytes. That was a huge performance gain and helped offload the network from processing the several hundred TIFF images that were about 500 megabytes each. Using a scale factor of 1, left us with great images that were only slightly lossy. Also,  our TIFF images were 32 bit and the resulting JPG images were 24 bit, but there weren't any loss in color.

World Files

You can also use a similar technique using StreamReader and StreamWriter to include processing of the world files (TWF) . Within a TFW file (or JGW), there are six lines:

  1. Line 1 - Cell size in the "X" direction
  2. Line 2 - "Rotation" term for the row
  3. Line 3 - "Rotation" term for the column
  4. Line 4 - Cell size in the "Y" direction
  5. Line 5 - Easting value of insertion point "X"
  6. Line 6 - Northing value of insertion point "Y"

To update the world file just divide the value in Line 1 and Line 4 by the scale factor used to process the image. All of the other lines in the world file will remain the same.

My Related Posts: , , ,

3Oct/070

Microsoft .NET goes open source! Oh noes!

Oh noes! Microsoft .NET goes open source!

Well, not exactly. There's a lot of buzz going around the web today that started with an post by Scott Guthrie "Releasing the Source Code for the .NET Framework Libraries" (via DotNetKicks.com).

We'll begin by offering the source code (with source file comments included) for the .NET Base Class Libraries (System, System.IO, System.Collections, System.Configuration, System.Threading, System.Net, System.Security, System.Runtime, System.Text, etc), ASP.NET (System.Web), Windows Forms (System.Windows.Forms), ADO.NET (System.Data), XML (System.Xml), and WPF (System.Windows).  We'll then be adding more libraries in the months ahead (including WCF, Workflow, and LINQ).  The source code will be released under the Microsoft Reference License (MS-RL).

This generated a lot of buzz including:

What does this really mean for .NET Developers? Well, at first glance I'm agreeing with Aaron's (AjaxNinja) thoughts. It won't mean much to the average developer. And as I commented at over at DNK, the above average developer already uses Reflector to peek inside the .NET library.

I truly think this is much more of a PR initiative than a technical initiative. It's the "cool thing to do" and will (did) generate a lot of buzz and talking. I'm sure there will be blogs and articles abound in the next few weeks or months talking about this. Maybe in a few weeks, there will be some new information that might provide some insight into how this will be helpful to developers.

Update: JudahGabriel (via DNK) pointed out three reasons why this will benefit developer beyond what we already get from using Reflector:

First, the source comments are intact, which means it becomes easier to understand .NET library source code.
Second, the code structure and local variable names are preserved.
Third, we can use the debugger to step in to .NET libraries and use the threads, locals, watch, etc. windows.

Shawn Burke also provides a good reason why this might benefit the average developer saying:

Even with one of the decompilers (e.g. Reflector) that you can easily get out there, [an exception from deep in the code] can be tricky to solve.  With the source it's much, much easier.

 

 

My Related Posts: , ,
2Oct/077

How to convert a Visual Studio 2005 solution to Visual Studio 2005 Express

Not everyone is fortunate enough to own a copy of Visual Studio. Although, if you are a student, I highly recommend that you look into the academic license for Visual Studio. With the Academic version, you get everything, but you are not licensed to sell any products produced from it (there's more rules, but that's the major issue besides needing a student id card).

I've needed to convert some solution files to Visual Studio Express because sometimes the computer that I'm working at (such as a computer lab) might not have Visual Studio installed. So I was able to download and install the Express edition, edit the solution files and then open/compile/debug, etc.

For the most part, this information is a complete //HACK, so be warned that you should make a backup of any files that you edit.

I will be using the DotNetKicks.com source code as our demo files.

Step 1 - Make a backup of your files

Step 2 - Create an empty Express solution and project files to use as the template. Save these files anywhere such as a temp directory. These files will be deleted in the end.

Step 3 - Open the VS Express template (step 2) solution file using Notepad. (Notepad2 is a great alternative).

Step 4  - Open the source (Visual Studio Professional) solution file using Notepad (in another instance of Notepad).

Step 5 - Update the source to use the template's header.

The first few lines of the VS Express template solution will say:

Microsoft Visual Studio Solution File, Format Version 9.00
# Visual C# Express 2005

The first few lines of of the VS Pro solution will say:

Microsoft Visual Studio Solution File, Format Version 9.00
# Visual Studio 2005

Copy the VS Express lines and paste them over the VS Pro lines.

Step 6 - Finish Solution edits. Save the file and close the instances of Notepad applications.

Your solution has now been converted to Visual Studio Express compatible. However, there are still a few gotchas...

If you are performing this on a solution that contains a Web Project. Then you will need Visual Studio Web Developer 2005 Express (since VS Express C# cannot open web projects) and perform the same action as indicated above. This requires that you break out your Web project from any class library projects. You will need to create a second solution file for your web project and just open the directory where the web files are located at.

In the DotNetKicks solution there are 5 projects that include:

  1. Incremental.Kick.Tests.csproj (Class Library)
  2. Incremental.Kick.Web.UI.Tests.csproj (Class Library)
  3. Incremental.Kick.StoryPublisher.Console.csproj (Console)
  4. Incremental.Kick.csproj (Class Library)
  5. Incremental.Kick.Web.UI.csproj (Web)

Additionally, solution folders are not supported in Visual Studio Express. So if you just converted the DNK solution file, you should see 1 Web Project (Incremental.Kick.Web.UI) and 3 Solution Folders (Solution Items, Tests, Utilities) marked as "(unavailable)". Just right click and choose "remove" for these items.

At this point, you should have 2 solution files. One Web solution and one for the Application/Class Library projects. The class library solution should look like:

DotNetKicks.com Solution in Visual Studio 2005 Express

Additionally, you'll have another Web project in VS Web Developer 2005 Express. You'll just need to update any paths for the project references to make sure the web project can automatically update the .DLL files from the Application/Class Library solution files.

One thing that I did notice was that when I went to build the solution, I had lots of missing library references. I don't understand this part since the path in the reference properties is correct. I fixed this by just copying the files from projectdir/bin/debug to projectdir/bin/release (there's 4 projects). After copying the reference files (x4), I was able to get the "build succeeded" message.

This was a rough and quick walk through of converting a VS solution file. It uses the current code base for DNK which might be different from the DNK download. I'll try to revisit this and clean it up a bit within the week.

 

My Related Posts: , ,

kick it on DotNetKicks.com