Extreme Prototyping is an architectural process for developing applications, especially web applications, in terms of increasingly functional prototypes. At a high level it breaks down the web development into three distinct phases. The first phase is the Static Prototype consisting of html pages and possibly a logical data model supporting those pages. The second phase is a coding process in your chosen web framework where by the screens are fully functional using a simulated services layer. The third phase is where the services are implemented. The process is called "Extreme Prototyping" to draw attention to the second phase of the process where fully functional UI is developed with very little regard to the services other than their contract. This deliverable much like the static html prototype can be considered a prototype, albeit a "dynamic one".
Using Extreme Prototyping you will be able to substantially shorten the requirment cycle and also the development cycle. Extreme Prototyping will allow you to consistently estimate and deliver your projects faster, cheaper, and better. Extreme Prototyping achieves these results by primarily breaking down the dependencies between multiple teams and there by allowing parallel development. Extreme Prototyping continues to put extra emphasis on executable deliverables.
If you imagine system development as a set of concrete deliverables, you can imagine an "Extreme Prototype" to be yielding 3 bricks as depicted in the following diagram.
The first brick is the Static Prototype. Based on this static protoype the UI brick and the Service brick can be developed fairly independently and brought togehter towards the end of the project for an integration alignment. The service contracts (or blades) should allow for a perfect alignment in any typed language.
Extreme Prototyping has some unexpected not so obvious advantages. Before I attempt to answer that in the rest of the article I want to clarify first the resemblance of Extreme Prototyping to other iterative approaches including the much respected extreme programming that focuses on frequent executable builds.
iterative approaches recommend to build and deliver your working programs in small steps and often. The key is the frequency of releases of executable code. These iterations or releases should at least be every week. When an iteration becomes 6 months it might as well be waterfall. But there are assumptions for this to work well. The team has to be small enough for the continuous feedback. If projects have a tendency to be cancelled this approach is not the best. In other words this works best when the team is fully commited to carry out the project. In IT this is not always the case. When the iterative progress is not satisfactory projects tend to be cancelled. Business could be impatient for the final outcome.
Extreme programming is really kind of orthogonal yet symbiotic to the iterative processes. The daily output and builds in each of the phases are crucial. But the fact is that there are three distinct phases. These phases bring some distinct advantages. This article is about the advantages these phases bring and also how to architect at a code level to realize the advantages.
It is not untrendy that a good number of IT projects start with a formal requirement phase. Usually this involves a certain allegiance to usecases. The idea is one can hand over the usecases to business teams for verification. Business teams will then read the usecases and sign off at which time the development team will take the usecases and develop screens.
What is more likley to happen though is Business team is going to be lost in the Sticky figures, Activity diagrams and so on and so forth. As an artifact, Usecases leave much to be desired for the Business teams to imagine a completed system. To prove otherwise much more time is spent and more usecases are written more clearly. The intent of completeness is so much emphasized that all i's and all t's need be dotted and crossed. Special tools are bought to make usecases much more effective. More consultants are hired for elaboration. In the end all the allocated budget, and some more, is all spent before writing a single line of code.
The fundamental problem is not with with the Usecases. The key is to wrap up the usecases very quickly and use them to derive an html prototype that is as complete as possible. It is easier for Business teams to validate and agree on a prototype and agree on it. The clarity of prototypes will work like a catalyst to complete the usecases as well.
Let me bring some numbers to bear on the influence of Static Prototypes. Imagine a form based web system with 30 screens. One should be able to wrap up the complete prototype along with usecases in about a month. Coming to the same level of clarity and agreement with only usecases will probably take a quarter with 3 resources. I see Static Prototype approach being 3 to 4 times more effective monitorily and time wise for closing the requirements phase.
Very often Business starts with a vision statement. They would draft an initial requirements document that is basically an enlarged wish list. It could be a powerpoint identifying modules and sub-systems. IT team is generally asked to draw an estimate of cost and time to accomplish the system. The estimates based on this document are quite subjective and may have a variability of 10 times, usually on the upper side. This estimate usually involves all the phases of a project including requirements gathering. The resulting confusion and anticipated cost will close the project down.
I suggest something different in Extreme Prototyping. The Business Team instead of putting an RFP (Request for proposal) for the whole project scope should put an RFP just for the Static Prototype. This will yield an estimate that is mostly fixed cost and the deliverable is readily reused in an eventual estimation.
In this approach hire or allocate an experienced html prototyper and an experienced documenter and come up with a fully functional html prototype. As I pointed out for a 30 screen system this might take about a month. That might cost about 50,000 dollars. At the end of it the completed prototype will give business a concrete set of requirements. The finished look will also give the business an extra resolve to go through the project.
Subsequent estimates for money and resources will be much more realistic and less subjective.
An important thing to do in this phase is not involve any other resources so that the burn rate is not large.
Static Prototypes are developed with known and less risky technologies such as html, css, and javascript. As these technologies are proven and low risk, it is unlikely that the Static Prototype will be delayed or cost more than expected.
In contrast directly moving into an iterative phase with the end technology from the start could pose learning issues or even substantial road blocks. Dealing with these challenges during the delicate, uncertain, and crucial phase of requirements does not sound wise.
Frequent demos and frequent executables are important steps in a development process. Especially the completion of a Static Prototype is a good logical step at which time the system can be unveiled for the end users as a demo.
So far I have quoted some of the benefits of Static Prototypes. Some of these benefits are well understood, but some benefits are more subtle. For example although it is fairly well recognized that Static Prototype is a good tool for requirement gathering it is usually not seen as a budgeting tool or as a tool that will increase the probability of a system going into production.
Before talking about the "dynamic prototype" let me introduce the overall development process of Extreme Prototyping.
Static Prototype phase Static Prototype master or background pages css, javascript business rules, usecases Extended Static Prototype phase All of the above Logical data model to support the screens Dynamic prototype (Or extreme prototype phase) UI recoded/adjusted for the chosen web framework working executable code Field validations work Navigation of screens will work Service signatures solidified A complete working ui with no implementation behind Service Implementation Phase API document Each service implemented by calling databases or other resources Integration
Extreme Prototyping starts out like any other well planned project with a Static Prototype, but quickly distinguishes itself in the coding of UI. Extreme Prototyping strives to see the UI deliverables as another prototype with a well promised service layer. In this sense the UI layer is defining an API for the implementers. This is similar to how various JSRs are being developed. One body defines the standard and the API. A number of companies will implement that standard using the commong API. The UI implementation will then be followed by the service implementation. But in reality, in Extreme Prototyping, both UI and Service implementations can start at the same time.
The details of each of these phases covered below will further clarify Extreme Prototyping.
Static Prototype will start with a vision first. From the vision you recognize and write down a series of users that would be using the system. You will write down goals for the system. The analysis can take the form of actors and usecases or a word document outlining each need.
The key to Extreme Prototyping is to minimize the details in these documents so that most of the requirements are captured primarily in the prototype. The documents will tell a clear high level story about how the system works and what are the essentials.
A minimalistic and essential approach to documentation is essential to Extreme Prototyping.
Extreme Prototyping recognizes that prototypes, working screens, well written source code, meeting minutes, email communication all forms a united core of documentation.
html prototyping resource is a key resource. You want this prototyper to be thorough in all of the following
Money spent in this phase is well spent. Typically a single prototyper should be able to handle a system with 30 or so screens at a time. If there are more, potentially more prototypers can be used. You want to hire a prototyper that can grasp your vision for the project after a few sittings with the users and can lead them to a final solution ably and quickly.
The prototyper will have to frame a mental picture of the following before embarking on a prototype
Armed with these, the prototyper can start framing the skeleton and continue to work the details with constant feedback from users.
The key for the prototyper is to understand the system well enough either by talking to the users or reading documents which ever is faster and efficient. It is important that he/she would be able to answer "why"and not "what"?
If they do, the prototyper has much more material to depend on. The more detailed the usecases are the more easier the job of the prototyper is. Well there is a price to pay for this. Detailed usecases take time. Users getting discouraged about a system increases exponentially with the time it takes for the users to see some output other than documents.
So usecases are kind of optional for the prototyper. Nevertheless as the prototyping is taking shape, a usecase specialist can capture the details and document them along with the prototyper.
Nevertheless if one can modify the view of usecases so that they are simple and minimalistic then the users can put these things together very quickly in an informal document to minimize the dependency of prototyper on the users.
I expect web based tools to emerge rapidly in the next few years. I am especially fond of project portals where a project is started in a company by allocating a web site to it. Everything that happens to the project is captured with ease on the project web site. I have used this very effectively for a couple of projects.
All project related communication and assets are portrayed via the web presence. For instance users can enter their wish lists for the system as a series of documents on line with out any additional tools. Discussions and elaborations can take place for each posted item. The items can classified into modules and searched upon if required.
Prototyper can take notes on the website while in meetings and there by capturing the tacit knowledge.
All emails related to the project can be tagged and made available as conversations on the web site.
This early knowledge aggregation will help the developers down stream to quickly decide what needs to be done on a particular screen or in a given situation.
Prototypers typically use Dreamweaver, or Frontpage, or just plain html enabled editors for accomplishing this. Which ever tool you choose that tool should support the following
I expect online web tools to emerge in this space as well. Recently I have been experimenting with knowledgefolders to do the following in this space
The conclusion of this phase will be a clickable web site that will reproduce the user experience as close to the real thing as possible. Usually the following would have gone into this effort
A Static Prototype that is fairly complete has hidden assets that can be mined. The primary one is the logical data model that is necessary to support the prototype.
Screens much like use cases will give away the main topics or logical entities or tables in a database. The summary and detail screens will give away the relationships between tables. Various form elements and display elements will clarify field level attributes of a table. In short it is possible to derive the logical data model from the screens.
The logical data model can further be converted into a physical data model. At the end of this it is possible to give an instantiated data base ready for development by the developers.
I call this Static Prototype, when accompanied with an instantiated database, an "Extended Static Prototype". The extended Static Prototype is a logical unit that can play a roll in division of duties or just out sourcing.
Companies often evaluate to see if a certain software is best developed outside due to expertise or time and resource constraints. The main drawback to outsourced development is how the delivered software will fit into the internal architectures and resources, not to mention to the coding standards and the rigor expected internally.
An Extended Static Prototype can solve this problem. A company can chose to outsource an Extended Static Prototype. The variability of an Extended Static Prototype is small. This means there are very few places that the outsourcing company can go wrong. The technologies are known. What is delivered is conceptual. On the other hand source code for a web site has a substantial variability. So by outsourcing elements of least variability you will come out more satisfied.
Under this scheme an outsourced company that specializes in web sites and their design can give a working prototype along with the logical data model to the in house development team. Development team can then develop the concept using any of the chosen frameworks. It is like a tag team. Ready to run from the start.
Another asset that can be directly derived from the prototype is the set of test cases for each screen.
The primary goal of a dynamic prototype, abstractly speaking, is to separate the "known" and "unknown" so that progress can be made in the "known" space.
A dynamic prototype divides the software production into a UI layer and a services layer. UI Programmers, on the receipt of a Static Prototype, will proceed to create a UI layer that is completely functional. UI layer will talk to the service implementation through one or more of service interfaces. It is possible to complete this working UI layer with out ever implementing a single line of the real implementation. The finished UI layer will be complete in the following respects
Despite the seemingly hard exterior of typed languages, they provide the basics needed to successfully execute a dynamic prototype. The key is enforcing the contracts interms of well defined objects and the service API.
The primary reason is to break down the dependencies so that the service implementation team and the UI team can be productive simultaneously. It is very common for the UI team to sit paralyzed until a database instance is in place and domain objects are in place and a service layer is in place. They ask "How can we code unless we know all these details?". So this will make the production process serialized with the UI team depending on the back end implementation.
By having conceptual domain models for screens and screen level objects it is possible to envision the system interms of these objects and then let the service layer contracts translate these objects into proper implementations.
By taking this approach both UI and Services teams can start their work simultaneously.
Because a Static Prototype already exists and the business rules are written up interms of usecases or similar documents the UI programmers can independently code up the working screens and also in the process generate the signatures for the service layer.
Following this process UI programmers often raise the question of what goes into the UI layer and what goes into the services layer? Or how far should they go in the UI layer? Or how do you draw the boundaries?
Some of this is clear enough where data access, storage, and business logic that is devoid of screen influence is relegated to the service layers.
Some times there is anothe way of looking at this that can help you. If you know how to execute or perform a certain function clearly with the information you have in the UI layer then do it so. This is what is called "known" solely based on UI. The moment you run into a road block where you can not proceed in the UI with out knowing this bit of information, make that "unknown" a service and simulate it like everything else allowing you to postpone that unknown while you proceed with the rest of the UI.
In this sense the interfaces are treated like the boundaries between "known" and "unknown".
If you examine closely the dynamic prototype that is being suggested here you will see that it is advantageous to represent an html page as a Page object in memory. This page object parallels the physical html page. It will have data on it. It will have actions on it etc.
Consider the following code
public class MyWebPage
{
//ui controls
uiControl1;
uiControl2;
...
//Population etc
onload(){}
postback(){}
//Actions on the page
action1(){}
action2(){}
}
Here is a web page in memory that is a collection of ui controls and a number of actions. The methods will adjust the ui controls in a certain manner with data and let the page paint itself. In general this is the method employed by modern OO based web frameworks. Some frameworks are more friendlier for data binding than the others.
It is possible to rewrite the above page as follows
public class MyWebPage
{
//Concentrate your data in one place
MyWebPageData;
//ui controls
uiControl1;
uiControl2;
...
//Population etc
onload(){}
postback(){}
//Actions on the page
action1(){}
action2(){}
}
public class MuWebPageData
{
//data only
field1;
field2;
list1;
list2;
object1;
}
In this approach all the data on a web page is encapsulated into a page object. This approach has the following benefits
This is just another tool that can be used to reduce the dependenices between teams to shear contracts and not implementations.
Consider the following architecture where an action on a web page is directly accessing the DAO objects and manipulating them for a certain business purpose.
In this scenario Action class is responsible for two things. On one side it interacts with the UI of which it is part of, and also interacts with business logic. There is a distinct advantage in separating these two concerns. If the action class can postpone the "business concern" through a stub or a service then the UI can be built independent of the business service. This will allow division of labor and hence quick deliverables by throwing more people at it.
With this thought in mind let me rearchitect the picture a little bit
The key features of this design are
This can be further enhanced by introducing a static wrapper for the services as below
The key concept in dynamic prototyping is to build the system as if (as much as possible) there is a services layer implementation from the start fulfilling the contract. Initially the implementation is going to be using some simulation mechanism to simulate the contract.
The static wrapper or an equivalent scheme can nicely package the interfaces into some sort of a distributor where multiple implementations can co-exist. For exampel the article I wrote on "server-side typed event distributors" points towards an architecture where each method of an interface can gradually refined for a final implementation.
Under this architecture the static wrapper will look more like the following.
Finer points of this architecture are
Hopefullly the aspect of transactions are implemented in the "static wrapper" and not in the "implementation". This means the static wrapper is the one that puts the transaction on the thread local. In a sense the "static wrapper" is acting as a simplified or a poor man's container for service implementations.
The implication is that the implementation of a service will not (except under very extraordinary circumstances) deal with transactions at all
Under this scheme your method will look like this
public class ServiceImpl
{
public void saveSomething(parm1, parm2, parm3)
throws ServiceException
{
try
{
FirstDaoX.save(?);
SecondDaoX.save(?);
ThirdDaoX.save(?);
}
catch(DaoXException e)
{
throw new ServiceException(e);
}
}
}
Both single saves and multiple saves work the same. The above code is portable to all "service" oriented architectures which will include session beans, spring, web services etc.
It is quite possible as long as the intent is kept the same. Especially the ease with which you can define new service contracts and their implementations should be kept in mind in chosing a services framework. Because the main goal of a "dynamic prototype" is to quickly break down the dependencies, the suggested static wrapper approach will work quite well for medium size projects.
The idea of this phase is to bring the tools and architecture necessary to manage information and dependencies so that progress can be made in parallel. The chosen architecture should guarantee this basic need of "neutralizing dependencies".
In this phase or layer you will accomplish the following
The Static Prototype will directly lead to a logical data model. The initial logical data model can be worked out by a development lead based on these screens. The approach is not that different from coming up with domain objects based on use cases in the rational process.
The logical data model can be further refined as the service contracts take shape.
Once the logical data model is in place it makes sense to describe the relationships of tables in an annotated document called "Annotated Logical Data Model". Each page will have 4 to 5 tables and how they are related. This is in contrast to a single logical diagram that spans a wall in an office.
There are no specific surprises here. If there is a DBA on hand this could handed over to the DBA. Otherwise a developer can do this as well. It might be better to manage this model as an ER diagram in some tool of your choice. This will allow for consistent updates to the table structures etc.
There is one document that I have found most useful in the execution of these projects. I Call this the API doc. This document essentially documents the service contracts. The table of contents of this document may look like this
Using this document service implementers can write the apis to satisfy the contracts. This document can also be used by the UI developers to know what the screen looks like and what needs to be done slightly better than the use case document. Or UI developers can help write this document as they start coding each ui screen.
Each screen should have a set of test scripts that the developers can pull up any time to see how much of their screen is complete and if they fulfilled their contract.
Access to these test scripts for the developers will not only improve the quality but will expedite the delivery of the product.
Towards the end of the project it will also be useful to have 4 or 5 sanity test cases for the system. These are high level system test cases that test a complete path through the system.
Once the system goes into production, it is important to monitor the health of the system, especially during initial months. Although it is possible to rely on external web tools to do this, we can also build availability into the architecture itself by following some of the rules below.
I hear a number of startups ask me this questio. They have this great idea for a web site. They want to develop this whole thing in a couple of months. They already hired a bunch of developers and ready to code. What platform or technology should they choose so that they can really develop this really fast. Most of the times the developers are quite novice and the startup is assuming that the chosen framework is so easy that they can just churn out the web sites.
Extreme Prototyping brings both good news and bad news, but mostly sober news to this scenario.
Extreme Prototyping doesn't look at the whole development space as a uniform process or a single deliverable from start to finsih. Instead it wants you to hedge your resources, talent, and your bets in general across multiple deliverables.
Extreme Prototyping says, hire a really good prototyper that is well versed in html and css and get your prototype fully functional. This is your phase where you spend your best talent and vision and reduce the number of resources that work to reduce the cost.
once the prototype is fully ready then you can engage the relatively newer developers to complete the job in a framework of your choice.
By focussing on the Static Prototype you are not solving the problems related to the framework of your choice and see if it works or not etc. At least your focussed battle is going to the usability and look and feel of the api.
The whole idea of Extreme Prototyping is the separation of concerns so that you find each battle one at a time. For example don't fight the framework battles during prototyping. Don't fight the data implementation battles during ui development.
Extreme Prototyping suggests a prototype based mechanism that is stripped to the essentials of delivering a software system. I see the following benefits in taking this approach