Breeding Ideas

According the Merriam Webster online dictionary one possible definition of the word ‘idea’ is a “formulated thought or opinion”. So if you are looking for new ideas you need to think about it because they do not just come by themselves. Even if sometimes you think you got an idea spontaneously that’s not true, it is only an illusion. In reality your subconscious was busy working on it and was waiting for some sort of event or trigger to send it to your conscious.

So how ideas are created? Usually ideas are issued from unsolved problems or unsatisfying solutions. People who are having trouble with certain matters on a daily basis are more likely to think about new ways to improve the way to perform their day to day tasks.

If they succeed in finding a solution then the idea is born and might appear genius to some people and irrelevant to others. Not every idea will be a success at the business level, it all depends how the market reacts towards it. However, if you have solved a problem that you were facing, there are big chances that some others out there are facing the same issue and might be looking for the solution that you have invented.

The above is one approach for breeding ideas. But that’s not all! There is at least one other way to create new ones and innovate. Needless to say every approach requires intellectual efforts and brainstorming.

So another way to breed more ideas is to combine several existing one to create a new hybrid one. At start, there might not be an immediate need for this new idea, but as the time goes on and the idea is put to use, the public might discover that it is serving them in a way they did not expect.

Combining several ideas into one is straightforward. All what is needed is to draw a table in which you put current inventions in rows and their features or uses in columns.

Consider the following example: A mobile is used for making calls and send SMS at a long range; a remote control is used to control a device such a TV or Stereo at a small distance. One possible combination is to merge them into one instrument that allows people to make calls and control their home machinery over a long distance. This way when they get home they have everything in place as desired; like music is on, plate heated in the oven, the AC is working etc…

Is it useful or will it gain acceptance? This is a different story. The important is that you created the idea and now you can work on it to make it appeal to people.

You can also read an interesting blog “How to unlock your creative genius” written by Don Dodge in which he exposes his opinion about how genius ideas are born.

IPad, Just Another Gadget

Recently I have been tinkering with IPad for some time, in an attempt to convince myself to use it on day to day basis. However I felt it is was not built for working people who use the computers to produce something of value to their companies or customers.

This might be hard said, but that was my reaction. For me IPad is a gadget and not a tool for work. Before going further, I would like to note that this does not diminish its value as an amazing technological device.

On the other hand for people who use computers as a productivity tool, there are some big issues related to it. The minute you lay your hands on one of them you will instantly notice the following:

• Your fingers are not as precise as a mouse pointer, so if you are doing something like paintings or graphic design you will have hard times going through it.
  
• Also it is not suitable for real work on spreadsheets such as Excel, due to the endless manipulations you need to do with your fingers, instead of a mouse.
  
• Certainly it is not suitable for programming, but I guess this is the least of concerns for many people.
  
• Another annoying feature is the continuous switching back and forth of keyboards as well as jumping from alphabetic mode to numerical mode. This will make any writings take at least 1.5 times the writing on a real keyboard.
  
• Now the real big issue is in the design concept itself. To really use the IPad for writing you will have to bend over it to use its keyboard causing you a pain in the neck and shoulders on the long run, and reducing your ability to easily view the content of the screen. If you place it vertically to read or view the content you won’t be able to write without twisting your fingers and wrists.
  

So with all those drawbacks, what is IPad for?

Well it suffices to look at some of the advertisement to know that IPad is only for sharing photos, doing social networking, reading books or articles, viewing videos, and web browsing in general. But if you are doing serious work it won’t help you.

When I asked a friend if she is going to buy an IPad she replied: “what for? For music I have my IPod, for work I have my laptop and for roaming I have my mobile”.

So it is just fair to think that IPad is more cumbersome than a mobile, and less effective than a laptop.

Based on this assertion I don’t think that all of us agree with Mr. Steve Jobs that the era of PC is over.

The People, The Place and The Subject

There are plenty of chatting and communication services out there. Each one has its own concept and tries to solve a specific problem. They adopt different approaches towards serving their users:

- Instant Messengers are centered on detecting when a friend or a contact goes online in order to chat with him/her.

- Chat rooms are about bringing people together in one place to discuss a certain subject. Usually rooms are labeled by topics. However there is rarely specified material to discuss and people engage into free discussions and speeches.

- Forums are usually more focused and members are required to stick to the subject of the thread otherwise their posts will be deleted. The problem of the forums is the lack of real time interaction, and of course people have to search for the forums which fit their needs. Believe me that's not an easy task.

However a brand new approach is emerging. This new concept takes a different approach to tackle the problem and offers a solution by bringing together the people, the place and the subject.

The place is simply any web page on any website.
The people are the users who are viewing that web page.
The subject is the content of that web page.

So if a number of people are seeing the same web page, they are most probably interested in the subject offered there. If they are interested, chances are they either agree or disagree and probably they would like to say something about it and discuss it with others who are viewing the same web page as well.

Social Networking Anywhere: Make Every Website a Chat Room

When you are browsing the web you are most likely searching for something like a product, service, info, or may be just for fun.

Suppose you have found what you are looking for, or probably something similar. What would you do next?

Most probably you will not venture into taking what is given to you for granted, or at least this is what you should do, but instead you will look to concur it with a different source, such as an article, a review, a blog etc…

So common sense tells you to make a little research little research, otherwise you would be taking a risk that will affect your wallet, your credibility and may be your position!

Chances are that you google again the product/service/article to get a feedback about it, especially if your credit card is on the stake. You check the forums to see what people are saying about this stuff, and may be you would join to discuss the subject and weigh the advantages and disadvantages.

In short once you have found your item, you will enter another circle of indecision (unless you are 100% sure of what you are doing of course).

However, there is something very obvious that has not been very well implemented so far. What about meeting people on the same website and discuss with them on spot about the content of the page being visited! It would be great if there are some people around to ask them about this item or that; a discussion is always a way to enlighten both parties.

This is a real issue! It is wonderful to feel that someone is not alone!
It is just amazing to know that the web is alive and there is someone over there with whom you can instantly share your opinion and experience.

SimpleConnexion.com is trying to do just that.

Selective Chat vs Random Chat

Random chat is a new trend for chatting. It started in mid 2009 with Omegle and continued in 2010 with Chatroulette. Although it is popular, there are some severe problems concerning these services.

These issues were detailed in a previous blog “Random Chat Review” and in short they are: a person can’t easily chat because users are always hitting next button, there are lots of perversions and people abusing the service by showing sexual acts, most of the time guys are looking for girls who are rarely found.

Although the idea is appealing, however these services are not delivering real value. What is meant by real value is to be able to have a nice chat with interesting people with whom some would share a minimum of common stuff.

Selective Chat might be the solution! This is fairly a new term that does not appear a lot on Google Search or Google Trends. It consists of finding the right person to chat with. A person who displays a minimum profile that allows others to find him or her.

As opposed to social networking services like Facebook or others the aim of a “Selective Chat” is to bring together people who want to chat and express a preference for a specific topic. While on social networking people share videos, images and activities with contacts and friends; they are reluctant to accept people they don’t know. Besides they are not necessarily online at the same time and might not be interested in chatting at all.

Conversely in “Selective Chat” people go there in order to find someone to chat with. However to avoid lengthy introduction and discovery of interests of the other person, “Selective Chat” gives people the option to build a mini profile in which they can tell about themselves, their interests, their personal website or YouTube.

Because each person might have his/her own set of preferences and interests it will be easier to find others who share the those interests, thus resulting in a nice constructive chat that deliver value to the chatters and might pave the way for some more evolved types of relations.

This concept is being implemented by http://www.simpleconnexion.com/ in which this service is trying to offer chatters an appropriate context for people to meet each other and discuss about common topics in real time.

When File Beats Database

It might seem odd or unusual to compare system files to databases. You might think that the comparison is by far in favor of databases that present a huge advantage in manipulating data in every way one would image. While is this completely true, there might be cases when you really discover that system files are needed to assist databases due to better performance.

In the financial field there are market data providers with whom you can connect to get information about financial instruments, their prices and values. The data received are a stream in the form of tickers. You have the TickerID, the DateTime and the values which can be Ask, Bid, and Mean.

You might receive thousands and thousands of data in a very short time. Certainly you will have to store them in a database in order to use them later in your computation. However, if you have a standard database, you will experience very poor performance trying to insert those data as they arrive.

Consider this MySql database example in which there is Tickers table like the following:


CREATE TABLE `tickers` (
`TIK_ID` varchar(50) NOT NULL,
`TIK_DATE` datetime NOT NULL,
`TIK_BID` double NOT NULL,
`TIK_ASK` double NOT NULL,
`TIK_MEAN` double NOT NULL,
PRIMARY KEY (`TIK_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Inserting market data into this table leads to execute an SQL statement like “INSERT INTO tickers (TIK_ID, TIK_TIME, TIK_ASK, TIK_BID, TIK_MEAN) VALUES (@id, @time, @ask, @bid, @mean);” .

Suppose you have 1000 tickers to store in the table. You have to execute the previous SQL statement 1000 times. Whether you use a StoredProcedure or not the execution time varies between 30sec and 40sec.

Certainly you don’t want your database to be locked down by a continuous, resource consuming process like that.

Luckily there is workaround for this problem. The solution consists of storing this stream of data into a text file and periodically imports it into the database. Let’s say you have incoming tickers stream, instead of executing SQL statement for each one, what you do is append the data to the end of a text file. When the file grows big enough, you import it into the database table using a special import tool that usually ships with most databases.

I have rewritten the same example but this time using text file. The performance was clearly far better. The time needed to write 1000 tickers into the text file and import them into the database is only 500ms. This constitutes a gain of 600%.

Here is the code snippet:

private void SaveToFile() {
    DateTime t0 = DateTime.Now;
    DateTime t1 = DateTime.Now;
    StreamWriter wrt = null;
    MySqlConnection con = null;
    try {
        Random rand = new Random();
        wrt = File.AppendText("dbvsfile.txt");
        for (int i = 0; i < 1000; i++) {
            String id = "ticker" + i;
            DateTime dt = DateTime.Now;
            double ask = rand.NextDouble() * 100;
            double bid = rand.NextDouble() * 100;
            double avg = (ask + bid) / 2;
            wrt.WriteLine(String.Format("{0};{1};{2};{3};{4}", id, dt, ask, bid, avg));
        }
        if (wrt != null) {
            wrt.Close();
        }

        con = new MySqlConnection(Properties.Settings.Default["dbvsfileConnectionString"] as string);
        con.Open();
        MySqlCommand cmd = con.CreateCommand();
        cmd.CommandText = @"LOAD DATA infile 'dbvsfile.txt' into table tickers fields terminated by ';' lines terminated by '\r'";
        cmd.ExecuteNonQuery();
    } catch (Exception x) {
        MessageBox.Show(x.Message);
    } finally {

        if (con != null) {
            con.Close();
        }
        t1 = DateTime.Now;
        TimeSpan ts = t1 - t0;
    }
}

There is no doubt that the second method is by far more advantageous than the first, however everything comes at a price. The file solution has the following issues:

• Data are not stored in real-time into the database: there is a laps of time (estimated in minutes) in which the data reside in a text file before being imported into the table. During this time those data are not exploitable.
  
• Management overhead: An extra code must be developed to verify proper import and to handle errors that might occur during the process.

In case you are asking yourself why it is so, the response is crystal clear. It is similar to the previous blog “Why Excel COM API is a Bad Choice”. Every time you are executing an SQL command, this order travels through layers of software framework to arrive to the database server to be executed over there. While the import tool avoids this entire complex trip and writes directly to the database.

Once again, it is better to avoid (when possible) the use of APIs when there is another clean solution.

ASP.Net HttpApplicationState and IIS

If you are ASP.net developer you certainly know about the HttpApplicationState object in which you store application wide data. As opposed to HttpSessionState the HttpApplicationState object will store those data as long as your application is running, while those stored in the HttpSessionState object are destroyed when the session ends i.e. the user signed out or has been inactive for some time.

However, this is the theory. The reality is not as simple as that. In case you plan to use the HttpApplicationState object to store information be careful because what is successfully tested on your PC might not work as expected when deployed on the server.
Let’s consider a simple example in which you want to count the number of hits on your website. You will do something like this:

int cnt = (int)Application[“hits”];
Application[“hits”] = cnt +1;

You test the code in your Visual Studio and everything works perfect. But when you deploy it to your server you begin to notice weird behavior. Sometimes you get the right number of hits the next time you will find a zero or a number that is far less than the expected one.

In order to understand what is going on, you should know how IIS deals with your web application.
As a matter of fact, you do not control the life cycle of your application, IIS does. At start your application will not be running until the first web page request of your application reaches the server. Then IIS loads the application into memory and runs it inside a private process. At this time the Application[“hits”] will be zero. After the first request it will become 1. As long as requests reach this same process the Application[“hits”] continues to increase. However at a certain period of time the IIS decides to create another process for your application. Now you have two processes running, each with its own copy of Application[“hits”]. So if the value of Application[“hits”] in the first process is 100 the value Application[“hits”] in the second process will be zero!

Furthermore, if a process appears to be idle for a certain amount of time, IIS will kill it and destroy all the information it contain, which means that you will lose the count in Application[“hits”].

What you have to learn is that the HttpApplicationState object is unique per process provided that this process is still alive. You can be sure that your process will not be alive (running) if it is going to be idle for a long time.

To solve this issue you have two options:

• Force the IIS to maintain only one process for your application, which means the same process will handle all the http requests. This way IIS will not create extra processes to handle new requests. This solution works only for transient data that do not have to live beyond the life of your application. But performance will suffer in case the number of hits increases significantly, because IIS will not be permitted to create another process to handle them. To configure IIS, open the IIS console, expand Application Pool and open your application property page. On the Home tab click configuration and select the performance tab. There you can set the “maximum number of worker processes to 1”.
  
  
  
  

• Use a database to store all important information. This solution is better when the information must be stored and outlive the application life cycle. It is also useful even if multiple processes of your application are created at the same time.

As you can see HttpApplicationState definition can be misleading if considered without the IIS configuration. It is better not to rely on it to store application wide data but use database instead to store them effectively and permanently.

Best Ways to Learn from Books

Since the start of my career I have been reading books, I have read hundreds of technical books of all types. My problem was always how to keep this knowledge and avoid losing it by the time. This was hard to accomplish.

Throughout the years I have devised a technique that helped me keep a minimum of the information I gather from books, and in case I forgot some parts, I was able to quickly recover them with a relatively small time.

This technique is not big secret, however it needs devotion and commitment and seriousness at work. All you need to do is to summarize the book!

If you think you can go that far, here are the details of how to do it:

Open your book whether a hardcopy or an eBook and grab a pen and a notebook. Read one chapter comprehensively and then do another pass while stopping at end of each paragraph; write down in your notebook what have you understood from this paragraph and move to the next one. I know the task is daunting and time consuming in comparison to just reading one time. But remember, if you are reading a technical book, it means that you are determined to learn from it and not to read it for fun as it was a novel.

The summary must not exceed 15% of the original book size. So a 400 pages book can be reduced to 60 pages only. These 60 pages are stripped from all the fat content that usually comes with each book. They constitute what you have understood from your reading, and are written in your own style. For this reason, you can get back to them and refresh your memory at any time, in a very quick manner. It looks like that you have reread the whole original book in just one hour.

Here is a list of Do’s and DON’Ts:

• Don’t copy/paste from an eBook to a Word document because you won’t be writing in your own style.
• Read while sitting in front of your desk because it will be easier for you to take notes. Don’t read in bed or in the train.
• Don't use markers to highlight paragraphs on the book itself, because you won't be able to locate them rapidly in the future.
• Write in a notebook and not on separate sheets because you will lose them very easily.
• Don’t take notes directly on your PC because you will not be able to resist copy/paste for a long time.
• Read one chapter a day or two if their size is not big.
  

This technique has helped me a lot in keeping important information from fading away over the time. But it should be noted that those that will be instantly available in your head are those that you are applying in practice.

Why Excel COM API is a Bad Choice

In some projects you might be asked to generate reports as Excel Worksheets. This is especially true when your customer is in the financial domain. Financial guys are keen of Excel; it is Omni-present in every aspect of their job.
Disregarding if their addiction to Excel is justified or not, there is something you as a software engineer should know. To generate an Excel report your first reflex is to use the COM API.
However this might not always be the right choice.

I am going to share my experience with Microsoft.Office.Interop.Excel which is the official .Net assembly that uses Excel COM API underneath.
Two years ago one of my customers asked me to include in my project the generation of reports as Excel file. Naturally my reflex went directly into using Excel APIs. I spent over a week creating all the details needed to add into the report. At the end I launched a test on real data and it took almost 10 minutes to complete! This was way unacceptable performance.
It was really one of the most stressful moments during that project.

To solve the problem I had two choices:

1. Search on the internet for readymade packages that generate Excel files directly and without passing through the COM APIs
2. Or use VBA script to generate the reports.

The first option was too risky for me because that involves asking the customer to buy an unknown library, which needs to be tested and approved and it is programming model learnt. If it turned out not suitable we need to claim our money back and start searching for another one. I kept this option as the last painful resort.

The second option could be tested in just one day. I replaced the APIs calls by a code that generates VBA then loaded that VBA into Excel and executed it. Everything worked fine and the generation time dropped to 3 minutes, which was far acceptable than 10.
Everything went fine and the project ended successfully.

Still, it is not the ultimate solution. So if one day you are confronted to a similar situation, first thing you do is to include in your plan to search for and test libraries that generate Excel files. Discard and resist any attempt to use COM APIs.
The following code compares the generation and savings of 100k strings into an Excel file. The first one using Excel COM API and the second one using ExcelLibrary found on Google code.

Excel COM API
Excel._Application app = new Excel.ApplicationClass();
Excel.Workbook wb = app.Workbooks.Add(Type.Missing);
Excel.Worksheet wrk = (Excel.Worksheet)wb.Sheets.Add(Type.Missing, Type.Missing, 1, Excel.XlSheetType.xlWorksheet);
for (int r = 1; r <= 1000; r++) {
for (int c = 1; c <= 100; c++) {
wrk.Cells[r, c] = String.Format("row #{0}, col#{1}", r, c);
}
}
wb.SaveAs("c:\\testExcelApi.xls", Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Microsoft.Office.Interop.Excel.XlSaveAsAccessMode.xlExclusive, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing); wb.Close(Type.Missing, Type.Missing, Type.Missing);
app.Quit();
Marshal.ReleaseComObject(wrk);
Marshal.ReleaseComObject(wb);
Marshal.ReleaseComObject(app); app = null;
wb = null;
wrk = null;

ExcelLibrary
string file = "C:\\testExcelLib.xls";
Workbook workbook = new Workbook();
Worksheet wrk = new Worksheet("First Sheet");
for (int r = 1; r <= 1000; r++) {
for (int c = 1; c <= 100; c++) {
wrk.Cells[r, c] = new Cell(String.Format("row #{0}, col#{1}", r, c));
}
}
workbook.Worksheets.Add(wrk); workbook.Save(file);

You will be shocked to know that the first code takes 3 minutes to execute, while the second only 2 seconds. This is simply 80 times faster.
The explanation is pretty obvious; the poor performance of Excel COM API comes from the fact each API call travels through several layers of codes and frameworks which makes it time consuming. Conversely ExcelLibrary writes directly into the file without any intermediary layers.

As conclusion, it is better to keep your options wide open and check/test available libraries before venturing into the unknown.

PS. Some of the available open source libraries are ExcelLibrary, NPOI, ExML…But of course you might find others.