Benford Does Stuff

The past few sessions at my internship have involved me gaining a crash course introduction to AJAX, otherwise known as Asynchronous Java and Xml. The Asynchronous part is what distinguishes AJAX from normal javascript that we’ve had for years. Instead of having the entire page update and refresh each time an action is taken, such as pressing a button, you can set only a portion of the page to refresh, such as a few objects that are relevant to the button’s action.

This technology can be paired with several scripting languages, including asp.net and php. It is one of the driving forces behind every “Web 2.0″ application such as Google Docs.

So far I’ve got a pinging animation going so that you can ping a specified hostname from your web browser. Though this is relatively “Hello World” level I’m going to be working on some more sophisticated applications next.

Since finishing my previous data reporting project I’ve been setting up my workstation with the newest versions of various Microsoft tools that we have aquired in the new quarter. Though the install time was lengthy on my somewhat aging workstation (p4, 1 gb ram) at least everything was able to work without too much trouble. Running all of the installers over the network was priceless. Microsoft SQL client was not so easy to get working however. The main hurdle was that to install only the client, it was necessary to start the installer on the second disk, then choose a very specific set of components from there. I was able to get everything up and running on the second attempt.

Read the rest of this entry »

My last project for this quarter was to write a Graphical User Interface which would allow users to generate an XML based quiz, as opposed to manually creating one with a text editor. Though relatively simple, this task was a bit of a departure from my previous experiences which largely centered around command line programs. The goal was to support generation of a quiz which could have multiple questions,  each of which could have multiple responses. The UI allows users to enter in a question text and solution. Once that is written, it is possible to add responses which have a response text field and a checkbox to indicate the correctness of the response. It is possible to have multiple correct responses in a single question. There are buttons to add and delete questions and responses. Updates to existing fields happen automatically when focus leaves the field, so there is not need for an explicit update widget. The greatest challenge here is that with a GUI, the user can do lots of things at different times, which necessitated that I make the interface robust even when odd input is given, such as attempting to remove a question  when there are no questions to remove. Pressing cancel at a save file dialogue box also triggered an exception which needed to be caught. We’ll be updating our tools next quarter. Visual studio 2005 will be upgraded to the 2008 version,  and we’ll toss Visual Source Safe in favor of a flavor of subversion designed for the .net environment.

I spent some more time developing the equipment and work order report. Apparently some of my problems stemmed from the fact that the meaning of some rows changed over time, and that there were many nulls in fields in which there should not be nulls. There were also some serious normalization issues. After seeing the depths of my problem one of the other programmers spent some time and developed a query which eliminated all duplication that I could detect. Now armed with a proper query, I was able to generate a report with Discoverer Plus. I also added the ability to allow the user select two dates and only display records falling between them. This report was put onto one of our testing servers. I showed it to the customer who could view it using discoverer viewer, a web-browser based front end to an Oracle database. This project is mostly done now. I’m mostly waiting on some user input and for some help with some optional features from the oracle programers. My primary correspondent at Cleanwater does not work on Fridays, and the next quarter is starting soon, so he had to find some project in which we could work together but asynchronously. There is a application that generates XML based quizzes which needs a UI. We are going to develop the application together using Test Driven Design, otherwise known as Extreme Programming. Most of the API was already in place, so he set up a few sample tests for me, and I had to code the remaining tests for the API. After that I will be working on a GUI for the quiz generator. 

Work continued on the reporting project. I learned a bit about how the report would be deployed to end users. Apparently there is a component of Oracle Discoverer which allows a report to be exported into html format so that anyone can read it, even without the discoverer program installed locally.

I had was able to arrange taking off early on Friday so that I could continue work on the Operating Systems shell. The project was near completion by the time I left.

I continued my work with the report generation project. I am now able to generate some meaningful results with the Oracle Discoverer Plus software. The only column that I am having trouble with at this point is of long data type. This is similar to a blob in that it cannot be sorted, but it differs in that a long contains characters instead of binary data. Longs can store up to 2 GB of characters per record under this version of Oracle SQL server.

I encountered some trouble with displaying float typed data in the Administrator Plus. After seeking advice from a coworker, we discovered that the data were simply displayed incorrectly. Adjusting the display type to floating point fixed that problem.

There was a meeting on Friday with our Customer. It went well overall and we were able to better define the project, avoiding several expensive pitfalls. He was planning to change certain aspects of the database in the near future, so it was best that we knew about it and did not generate reports for those parts of the database. The customer also had a request for another column to add into the report. I’ll need to research how to derive that one when I come in next time.

I continued with my work in developing the search queries which would give our customer the data he wanted. Running time for the original all-inclusive query was around 4:30. I spent most of my time attempting to improve this. After dropping some of the tables from the search and using some Oracle specific tricks, I reduced the runtime for the query down to less than 15 seconds. I learned during this that oracle has a “Distinct” keyword that it can apply to a select statement, which allows the db to automatically filter out duplicate entries.

The next step will be to integrate these queries with Oracle Discoverer Administrator, which is sort of a GUI frontend to database code. It creates a layer of metadata that allows the user to reorganize database tables into useful views without actually affecting the database. A business area generated here can be used by Oracle Discoverer Plus to generate reports. I’ve already done some of the work here, which I’ll continue on next Wednesday.

Well, it’s another Friday and that means another internship day at Cleanwater services. Today, since my previous task was completed, I was assigned to do a new project. One piece of software we use to compile reports about equipment and water services is Tabware. Unfortunately this solution was not working for one of our customers at all.  We use some Oracle database software called “Discoverer” to build front ends of our databases for customer use. It is somewhat like a highly sophisticated tool for generating views. I spent a good portion of the day working through example code in the manual. By the end I had a working “Business area” which could be used by another Oracle program to generate end-user queries. I also acquired experience with Toad, a program similar to MySqlAdmin, but for tailored for Oracle databases. Before I could generate any detailed reports I had to bring together the relevant information from at least a dozen different tables, typically with ambiguous relationships. The end goal would be to get all information about work orders paired up with equipment used during those work orders.  Fortunately for both myself and everyone else, a virtual machine was set up for testing purposes, so regardless of how taxing my queries are, they will not affect anyone else’s work. This was a Very Good Thing as I found out that a select * across half a dozen joined tables, totaling several hundred thousand records each can bring a modern desktop system to a standstill.Oracle 10g was deployed on my virtual server. I connected to the server using Microsoft Remote Desktop connection.

I started up my internship at Cleanwater Services again for the spring term. Cleanwater Services deals with maintaining the quality of the local water supply, which includes finding pipe breaks and sending crews out to fix them. They also do a significant amount of software development which is where I am involved.

After the fall term ended and before I went on my trip to Japan I started work at Cleanwater Services. Under the guidance of my managers I was assigned to update and improve a piece of software. The software in question was a program to record still frames from webcameras that were watching a construction site for some new water treatment equipment. The old program was unweildy and when one aspect of the picture taking process failed, the entire program failed, sending email every minute.

My task was to split the program into three discrete modules: An image capturer, a compiler and and archive cleaner. The image capture module is meant to grab images from the webcams and save them in an archival location. Snapshots are saved for different time periods. The compiler takes the existing still images in the archive and converts them into a movie (avi format) using mencoder, an open source video encoding program. The archive cleaner will clean out old still images from the archive, typically images that have already been compiled into a movie. I’ve put it through most of the stages of testing. It will be deployed to a testing server first to see how the programs run in the field. After that they will be transferred to an actual production server for use within the organization.

The program itself was written in C# using Visual Studio.NET 2005.

I found that programming in a corporate environment was quite different than programming for academic goals. Most notably, I was encouraged to ask for advice from higher-ups, rather than working entirely alone. When I started working on the program I was given some project specs which were quite similar in detail to the specs that I was required to write for my senior project modules.