Improving software development productivity

Cutting down software development time and improving productivity are the most debated topics in the software industry for some time now. The industry has responded to the challenge with new technologies, different software development paradigms, different design methodologies, promoting re-use, use of COTS components rather than developing them in house etc.

The availability of new programming languages like Java, .NET etc have helped improve developer productivity significantly by providing a rich set of libraries and by providing enhanced language constructs. Availability of specialized COTS libraries have further improved productivity and they have also resulted in improved quality of products. Availability of modeling tools, Domain Specific Language(DSL) Tools, Generative Programming Techniques, Software Product Line development methodologies, Software Factory approach, availability of domain reference architectures, Design and Architectural Patterns, application frameworks etc, are some of the techniques used today for building better software systems as well as for improving development productivity.

There are lots of modeling tools available in market today. Many of them support modeling in UML. While these modeling tools allow the designer model the solution using the constructs available in UML, they have many shortcomings. The tools generate only the template code and the developer still needs to write source code manually. Another problem is, generally, the quality of generated source is not good. UML is too vast and this deters designers, programmers from using it. While the experts are of the opinion that 80% of the modeling can be accomplished by 20% of the UML constructs, there is no universal consensus on what those 20% constructs are!. Another related problem with UML is its poor semantics. Because of these and other problems, Microsoft is betting big on the software factory approach.

A software factory approach combines different concepts listed above and provides an integrated environment for software development. The concept is based on the Economy of Scope. The concept is based on the fact that- unlike in the automotive industry where millions of identical products are manufactured(aka economy of scale), the software industry needs to develop ’similar’ but distinct products( aka economy of scope). It combines various approaches – component re-use, generative programming, models, patterns etc to achieve this. Microsoft provides Domain Specific Language(DSL) Toolkit to achieve automatic artifact generation including – source code, configuration files etc. DSLs are domain specific constructs( can be built for each domain using a tool that supports meta-modeling) and do not suffer from the deficiencies of UML tools. The generative programming technique is based on the idea of software generating software i.e. source code and other artifacts from graphical models, textual specifications etc. Microsoft is providing lots of add ins to the Visual Studio for supporting the factory approach. There are also lot of commercial add-ins available from many different vendors. Eclipse is another popular platform that can be exploited for Factory based solutions. INRIA, for example, is developing add-ins called ‘Smart Tools’ that is based on the Factory approach. Eclipse is easily customizable. It is highly flexible and its superior architecture makes it a very attractive tool for Factory based development.

The software product line approach is based on the same assumptions as the factory approach. It recognizes that the products that belong to the same domain have lot of things in common. Main idea behind this is the concept of commonality and variability among different products. The development of common and variable features of products are done by disparate teams. The common features are developed by a Core team(aka Domain development team) and variable features are developed by Application development teams. The software is designed such that it behaves little differently based on different configurations.  Some of the techniques used  by the product line development projects are – configuration files, templates(aka generics), file substitutions, #ifdefs,  reflection etc. By using the same common functionality in different products helps achieve significant savings in development time and also during the maintenance.

In the future, the architect needs only to know different components, patterns etc available. He will use these predefined artifacts and mix and match artifacts and arrive at a solution without much effort. Hope that day is not too far!

Simulation:Keep them simple, but no simpler…

It is surprising that in the software industry, lot of managers believe that availability of tools can magically solve some of their pressing problems. They do not think about the competency needed to use the tool for solving those problems. They forget that – ‘fool with a tool is still a fool’!. This applies to many engineering tools, especially to modeling and simulation tools. Lot of organizations do think that these tools are a panacea. They invest heavily on these tools only to realize later that they have not made any progress!

One of the few things to be aware of simulations are – the results of simulation are only as good as the simulation model itself. One of the important things to be aware of is the correctness of input data used for the simulation. Put it another way – ‘garbage in, garbage out’!. The first thing to do while building a simulation model is to analyze the input data. Many times, data needs to collected from the real-world processes. These data are called are stochastic data. The data can be modeled as probability models. Once you collect data from the real-world processes, you need to come up with probability models for generating these data. There are many input analysis tools that give you the probability models and their parameters from the real-word or experimental data. The trick is to decide when you have enough data that can be fed to input data analysis tools! Once you have probability models for inputs, you can verify that they indeed have the same distribution as the real-world data by carrying out Goodness of Fit(GOF) test. Again, this is tricky. You need to know that you have enough data so that the test correctly tells you if they are drawn from the same distribution. Another way to verify this is – if after feeding the input data the simulation result should resemble the real-world process behavior. If not, you will have to suspect your input data. If no real-world data is available, then you can consult experts in the field about the correctness of your input data. Another factor that affects simulation output is the correctness of model itself.

There are so many input distributions possible: random data, uniform distribution, normal distribution, exponential distribution etc. One of the ways these plethora of distributions are handled is by using the Gamma distribution. Other distributions can be derived from this distribution as special cases.

It makes sense to keep the simulation model simple to start with. Once the basic model is running and the behavior is acceptable, start adding more and more details so that it is closer to the real-world process. The trick here is to model only the relevant information that affects the simulation output and omit irrelevant or less important details. The correctness of the simulation can be done via verification and validation. To quote Einstein – Keep the model simple, but no simpler. So, the philosophy here is – Seek simplicity Then distrust it. [Alfred North Whitehead].

Many simulation tools provide – hierarchical modeling capability. This feature allows the modeler to model the system at different levels of abstraction. The models themselves can act as basic modeling blocks for higher levels of abstraction. This feature not only makes models simpler to understand, but also makes them scalable and re-usable.

In many of the real-world problems we need to create models using many different paradigms for eg: continuous time, discrete time, discrete event etc. One of the few tools that allows ‘embedding’ of different modeling paradigms is Berkeley University’s: Ptolemy. OMnet++ is a popular open source network simulation tool. Visit here for a list of simulation tools.

We saw it. Yet, we din’t see..

This is so true!. It happens to everyone of us. Sometimes, the world around us would have changed so much that we realize and wake up to this fact after a long time. One of the reasons for this is that – our beliefs, goals etc act as blinders and keep us moving in a particular direction without paying much attention to what is happening elsewhere. The other reason is because of the information overload. We can not spot and identify these patterns in this huge amount of data. Or, the trend may be too small a thing to get noticed and we ignore it as irrelevant.
The same thing happens to businesses also. They need to spot the trends early and capitalize on it. Many times they are so much obsessed with their strategies, tactics and new technologies that they don’t realize the change in current trends. There is little point in sharpening your saw, when your opponent is possessing a gun!. These kinds of surprises are seen many times in the history of wars, businesses( the Iridium project for e.g) etc. It is not easy to spot changes in the trends either. It requires an open mind and willingness to abandon the current path and pursue a new strategy. This is easier said than done. Businesses are so emotionally attached to their strategies, technologies that they don’t ’see it coming’!. Another reason could be because of the sunk cost and loss aversion biases. By the time they realize, it is too late. So, they must be ready to renunciate before it becomes too late.

Many times we do recognize that it is coming. However, we wish it may not really come. This is kind of wishful thinking. The kind of thinking that suits our needs, wishes rather than accepting the reality which is uncomfortable. We tend to search for, collect more information and interpret information that conforms to our beliefs and tend to avoid information that contradicts our beliefs. This is called – conformation bias. Psychologists say that this kind of thinking is hardwired in our brain!. So, it is very hard to escape from these biases. All we can do is to be on alert, always!!.

(I have taken the heading line from Robert Pirsig’s “Zen and the Art of Motorcycle Maintenence”. There is a similar shloka by Sankaracharya in ‘Bhajagovindam’ which says – “Pashyannapi na cha pashyati mudho..” which means- fools see, but don’t see!. However, Sankaracharya wrote this in a different context!).