Waterfall Architects

"After three years of hard work, we have signed off on the system design."
(gestures towards a wall of black 3 ring binders)
"We are so excited; now, how long will it take to build it? 6, 8 months?"

--A real conversation between a client and a software consultant in the late 90's after spending a small fortune on management consulting fees. 


This waterfall type of thinking was rampant in the late 90's and early 2000's before agile started taking hold in software development in the mid to late 2000s.

Waterfall model By Peter Kemp / Paul Smith - Adapted from Paul Smith's work

Around the same time, the term "architect" and "architecture" was used to define the specific "design" role on software projects.  For the immature IT industry in search of an identity, borrowing metaphors from the construction industry ( a more "mature" art that has been around for 1000s of years) was on the rise. Other metaphors like "Design Patterns" (copied from famous (building) architect Christopher Alexander) were adopted and embraced in an effort to make the industry appear more "professional" and less like the cutting edge "wild west" .

Many businesses adopted "waterfall" because, as a process, it was easy to understand.  In the .com boom of the late 90's traditional business felt pressure to get online or risk being left behind by young upstarts.  Traditional "brick and mortar" companies would hire one of the big 3 consulting organizations, and find themselves on the hook for years of expensive consulting work to produce "requirements, documentation, and UML diagrams". And just like the construction industry, developers were just like the "carpenters, laborers and other grunts" with the thankless job of taking these designs and making them a reality. 

Building software the "waterfall" way was simple, you check off the first two boxes of "Requirements" and "Design", and then "Implementation" was simply the translation from the design documents to code, more akin to "data entry" than "engineering", and the "Maintenance" would be some fixed price process that'd be roughly equivalent to one of the earlier phases.  

The outcome of many large products using waterfall was failure and the industry learned and adapted to a more agile approach.

Post Waterfall "Architecture"
With the industry heading away from "waterfall" in the direction of being more agile, one concept / title survived and became more sophisticated, that of the "Software Architect". Wikipedia defines the Software Architect position as "necessary" because: 

"OOP allowed ever-larger and more complex applications to be built, which in turn required increased high-level application and system oversight."

The Software Architect title itself has many variants (Solution Architect, Enterprise Architect, Application Architect, Chief Architect). What a person in one of these roles "does" (as described by Wikipedia):

"Software architecture is about making fundamental structural choices which are costly to change once implemented. "  --Wikipedia

In short the "architect" is different from the "coder", in that the "architect" is empowered to make the "big decisions that effect the project".  

Although I've spent most of my career with the title "Architect" I've always felt a bit uncomfortable with the title, it has a definite air of superiority to it... And then after visiting other "architects" in the profession, or helping others obtain "architecture" certifications, it occurs to me why the title of "Architect" as a role on a software project is a bit of an albatross.

The reality is that most software projects desperately need is (in order):

1. accountability
2. expertise 
3. communication
4. leadership 
5. agility
6. decision-making 

And unfortunately the role of "architecture" is more akin to a "liason" and generally involves only these two of the (6) aspects.  

My personal definition of the role that an architect fills on software projects is more in line with the definition of a "practitioner". What differentiates someone who is an "architect" from a "great coder" is:

"A software architect is responsible for communicating the design and direction of a software project to the team and anticipates, identifies and solves roadblocks while developing software." --Eric DeFazio

The reason I had to come up with my own definition verses Wikipedia one is to handle the realities of software development today.  :

• Architecture is not "just" making decisions it is also living with the decisions (architects need to have Skin in the Game)
• The architect should be empowered to change the process (to shift priorities, reorganize the team if need be based) 
• Architects need design and communicate the vision and direction within the development team in whatever means necessary (design documents, mentoring sessions) 
• The core job of the architect should developer, else they are disconnected from the actual feedback loop of development (they might focus on nice to have ethereal solutions that might happen down the road completely overlook the fact that it takes 5 minutes to recompile the app for each developer, killing programmer productivity)  

Wasteland Fog of War -Mapped, NoMapped

Concept of navigating the wastes (Party at location A) Destination at location B (+Fog of war)

Concept for Navigating the Wastes No Map (Party at Location A, Destination at B)

Image Transparency & Performance (Frame Rate) Solved

Moving from a Android Phone to an Android Tablet, I have been having a hell of a time trying to keep my frame-rate stable and about 60FPS.  Search as I may on the internet, no one seems to have asked or answered this question so let me post it here:

"How can I draw many Sprites/Bitmaps that contain transparency on a (SurfaceView) canvas  in Android without having my framerate (FPS) go to hell?"

Things worked fine on my phone (58FPS), but deploying the same app to my tablet, the framerate chugs to (>30FPS) I assumed it had something to do with the larger screen.  (I however assumed the video card would be "beefer" on the Tablet, so I couldn't quite understand what was going on.)

Searching the internet I found this article from Romain Guy about the performance of rendering to give you more information. in general the performance for rendering an RGB_565 image verses others can 1.5 to 2x faster (Depending on the screen you are rendering to.)

That made some sense, but then I had read many times that transparencies are not possible in RGB_565 format (Which is not true by the way, it just does not contain an Alpha channel, big difference).  the ARGB_4444 format means literally 4 bits for each Alpha, Red, Green, Blue
RGB_565 means 5 bytes Red, 6 bytes Green, 5 bytes Blue

RGB_565 does have a "transparent" pixel value the int (0) so images stored as PNGs and read in (with transparency) or you can create a bitmap:


Bitmap bitmap = Bitmap.createBitmap(width, height, Config.RGB_565);

anyways, this has made a HUGE difference, too bad it's taken this many days of struggling trying other workarounds... hopefully someone else out there will find this info and it will be a help to them...

Added Note:
For gingerbread... all windows are 32 bits, OpenGL is 16 bits for compatibility
http://www.curious-creature.org/2010/12/04/gingerbread-and-32-bits-windows/

Side note: playing Plants verses Zombies on android tablet and I think they are suffering from this problem (using the wrong assets to draw on the screen), the framerate practically kills this stellar experience
for me.

Optimizing the Development Process for Android

Developing Applications for Android (using the .apk Android Package deployable) is optimized for a smaller memory footprint and for runtime performance.  Heres a good video of explaining an APK
http://sites.google.com/site/io/inside-the-android-application-framework In the presentation, an APK is called "an island" which is an appropriate metaphor.  In this way assets and code are constructed and linked/bound together to allow developers can quickly access resources (layouts, images, etc.) using a main (precompiled) registry (i.e. R.strings.name) this convenience does much for things like making sure your get things correct at compile time so you don't push out a half baked application which is missing a string, image, sound, or other asset.  (And that's a good thing)

Because the apk structure of an Android application is optimal for storage footprint and runtime performance, this means it is unfortunately sub-optimal for another aspect, and that would be for flexibility/quick iterations.
Let me explain (through a use case).

Designing Layouts in Android provides convenient ways of accessing resources (images, text, etc. in the res directory) directly from the Layout.xml file.  In eclipse, there is even a "real time" editor and viewer which allows you to layout components (buttons, text, images) in and see what they "should" look like on the android device.  This is no doubt a nice feature and time saver... Assuming what you are looking at in the preview pane is what you get on the device (which in my experience is not the case).  In practice, what ends up happening is that  the UI will:

• look one way in the Layout preview pane, 
• a different way in the emulator, 
• and a third way running on a specific target device.

Going through the development process where I had an emulator, the Layout preview, and a target phone (HTC Droid Incredible) and trying to get the layout to look right on all of these devices was tedious...
1) make one change to the layout.xml,
2) redeploying to phone.
3) enter information in the application until I get to the page I am testing (frequently the last one in the app).
4) realize the page doesn't look quite right and go back to step 1)

I must have spent a good 1/3 of my time just tweaking the UI to try to make things look right (and this was in development.)

Then when the application was in QA this continued.  The QA leads were using 1st gen Android phones and regardless of my emulator settings, those screens were impossible to emulate, so I ended up receiving feedback such as ("the character needs to move up more", and "the guys hand is getting cut off to the right") I'm not blaming QA, they were stellar (They would even take pictures of what the screens looked like), but as a architect/developer, it was death by 1000 paper cuts.


The sad part of this was the fact that I was making the most minor changes (i.e. tweaking the number of dips (device independent pixels) between the header image and the content, etc.) in some xml file ... this was EJB deployment descriptors all over again... 
In addition, we had internationalization issues, and the text (for things like buttons and screen verbiage) changed frequently, and for each (simple) change, we had the following process:
1) (QA) identified problem
2) (QA) JIRA writeup made (screenshots, etc)
3) (QA) assigned JIRA
4) (Management) scheduled and assigned JIRA
4) (Developer) a single XML file was changed
5) (Developer) redeployed app to dev phone
6) (Developer) check in change
7) (Developer) update JIRA status
8) (Developer) cross fingers/ pray it fixed problem (if not goto 1))

What I would love is to be able to make all of these changes (to things like layouts, strings, etc.) push out a debug version of the application, and allow changes to be made at runtime. (Until things are refined to the point where a production version is available)

Alternatively
Deploy a "debug" version of the application  which resolves assets (layouts, strings, etc.) locally or from a Server, so effectively you can change (simple assets) at runtime.  To accomplish this, we are looking at a combination of Google App Engine and C2DM http://www.youtube.com/watch?v=51F5LWzJqjg&feature=related
... where C2DM just notifies your android device that it should go out and reload some asset from the Server.)--- More later

The Editor != The Game, The Game != The Editor

A design goal for real time editing is that the game (in it's finished, running form) does not contain the infrastructure and overhead from the editor.  Likewise, the editor should not be "the game+some debug code" running within a "debug" container.  Therefore one can be in a "broken" state without effecting the other, (decoupled).  The reason being is that it is difficult for the game engine to handle not only loading the assets, but having a solid framerate and operating the debug operations...And you don't want the most tested version of the game to be the version that it not being shipped. Also, I don't want to build a image/sound/ map editor inside the game, I want the state of the game to be exported, then I can edit the game map with an appropriate tool, update game map changes to a server, and have it be reloaded into the game)

One approach I was considering was to having 2 separate Projects each with using separate Application classes and each of these projects would reference a "library" project which contains the main code(i.e. Activity and Views, etc.) and assets.  It's not a bad idea, other than there is the potential that the code diverges (from one version to the other) and then you have a perfectly good running debug version (which gets the majority of iterations) and a relatively "untested" production version.

The approach I am taking now is to have the "editors" introduce themselves into the code at bootstrap and this will take place if the Development project is built in the classpath.  So there is only 1 place in which the Production (running) version of the code differs from the debug version of the code... here is the (current) wiring code for the "production" version.


public class BeaconDependencyProvider implements IBeaconDependencyProvider {

GameRenderer gr;
GameTiltListenerConfig tiltConfig;
GameTiltListener tilt;
GameMapLoader mapLoader;
GameState gameState;
Simulation sim;


@Override
public void setUp(Application application) {
gr = new GameRenderer (new GameMapRenderer(new SolidColorTileRenderer(), new BallRenderer()));
tiltConfig = new GameTiltListenerConfig();
tilt = new GameTiltListener(tiltConfig);
mapLoader = new GameMapLoader(application);
gameState = new GameState();
sim = new Simulation (mapLoader, tilt, gameState);
}


The Debug version is "wired" together differently:



/** Provides the debug dependencies for Beacon */
public class BeaconDebugDependencyProvider implements IBeaconDependencyProvider {


/** All Editors for the Game */
public CompositeEditor ce;

IGameRenderer gr;
GameTiltListenerConfig tiltConfig;
GameTiltListener tilt;
IGameMapLoader mapLoader;
GameState gameState;
Simulation sim;

public void setUp(Application application) {
Log.e("BeaconDebugModule", "Configuring Dependencies");

TileColorPalette tcp = new TileColorPalette();
BallRenderer br = new BallRenderer();
SolidColorTileRenderer sctr = new SolidColorTileRenderer(tcp);
GameMapRenderer gmr = new GameMapRenderer(sctr, br);
GameRenderer sr = new GameRenderer (gmr);
this.gr = sr;
GameTiltListenerConfig tiltConfig = new GameTiltListenerConfig();
tilt = new GameTiltListener(tiltConfig);
GameMapEditor gme = new GameMapEditor();
mapLoader = new EditableGameMapLoader(application, gme);
gameState = new GameState();
sim = new Simulation (mapLoader, tilt, gameState);

ce = new CompositeEditor();

FieldEditor tcpe = new FieldEditor(tcp);
GameTiltConfigEditor gte = new GameTiltConfigEditor(tiltConfig);
FieldEditor gse = new FieldEditor(gameState);

ce.addEditor ("tiltConfig",gte);
ce.addEditor("tileColorPalette", tcpe);
ce.addEditor("gameState", gse);
ce.addEditor("gameMap", gme);
}

All the Editor code does not exist in the production distro, in addition, none of the overhead of creating and wiring the editors is incurred by the main game while running.

One aspect I need to explore is forcing a refresh of the screen, for instance, at the moment my gameMap class is created and stored, and each time the frame is drawn, the cached bitmap is presented on the screen.  I need a "generic" avenue for forcing the game to realize that a recently loaded asset is dirty, and it needs to trigger a refresh to cause the screen to re-render.   

Runtime State Editing Android

I was happy to find some more supporting evidence that rapid iterations is the key to developing good software.  Here's a great article from Gamasutra about the development of tools for Dead Rising 2)

"Typically a tool will be some sort of viewer that allows a designer to tweak their content data. That data is then compiled, built, or baked, and somehow makes its way into the game. Often this process is long -- minutes if not hours -- involves restarting the game, and usually involves having a programmer enter some secret code.
It seemed much better to have a designer move objects around at run time from the comfort of their PC, so we developed a communications protocol that could talk between a game console and the PC.  Tools could now make the game do things (like spawn objects, move their locations, change an items attributes, etc.) at runtime."

I think games are a prime example of why designing tools for real time are so powerful.  Developers for games spend a good deal of time developing the foundation (physics, rules, etc.) then folks (who aren't programmers, sometimes artists or level designers) populate the game with content.  If those iterations are slow (as described in the article) it's painful, and you really make the process tedious.  The easier it is for people to make changes, the more likely you will end up with a refined product in the end.

Normally (and especially with Android) the Software are optimizing for the finished/deployed application (i.e. they are concerned with performance, load times, memory footprint, etc.)  And those are good things to focus on for a finished product.  However, having all of the resources local inside a single distro requires a recompile, repackage and redeploy each time, and therefore is not efficient for creation of software.  What I want to offer is a way to optimize quick iterations, in a way that can produce a refined finished product.

Originally I was looking to create some simple telemetry for determining what areas of the game are too difficult and return information about the game as it is being played.  Taking this a step further, it seemed logical that if I could Monitor the state of the game, why not allow real time editing as well. (If you can identify an issue using telemetry, why couldn't you "fix" and retest it in real time?) 


The system I have in mind is a client server based system.  The Android device will have a Service (call the RealTimeChangeService) that can be bound to by the application (only in development/debug mode).  During bootstrap the application class will resolve all of the dependencies including all of the application "state" (it will not only wire components (code) together, but it will also resolve variables (i.e. touchSensitivity), images (headerleft.gif), sounds (alarm.au), etc.

For example, I have a "state" class which contains the variables for dealing with the accelerometer:


/** The configuration properties of a GameTiltListener */ 
public class GameTiltListenerConfig {


/** threshold to determine if the x tilt encountered should cause movement (i.e. dead zone) should be > 0 */
public float xTiltDampener = 0.3f;

/** threshold to determine if the y tilt encountered should cause movement (i.e. dead zone)  should be > 0 */
public float yTiltDampener = 0.3f;

/** factor applied to the xTilt (assuming the xTilt > xTiltDampener)...(> 1.0 = speedup) (< 1.0 = slowdown)  should be > 0 */ 
public float xTiltSpeedFactor = 1.0f;
}


A GameTiltListenerConfig is created at bootstrap and used within the game to modify the character movement.  In real-time development mode, in addition to creating and setting these values, the application will also create and register an "Editor" which has a reference to the GameTiltListenerConfig, and this editor will be capable of : exporting the "specification" (i.e. the Editor will be aware of all of the properties available in the GameTiltListenerConfig object), and the editor will be able to export the state of the object (at the moment I'm using JSON as a data format) as well as accept changes to the state (i.e. allow someone to say editor.set("yTiltDampener", 0.5);


This is where Java Reflection comes in handy... I create a simple class (FieldEditor) that takes any JavaBean class in it's constructor, it introspects the class (in this case a GameTiltListenerConfig to find all of it's fields (the names and types).  The FieldEditor can accomplish all of the tasks (export the specification of the object, the state of the object, and also accept changes to update the value of the fields).

So on bootstrap
1) The state is created and the game entities are wired together
2) Editors are created with references to the game state
3) The application binds to the "RealTimeChange" service
4) The specification(s) are exported (there are many specifications for the many stateful game objects)
5) The state of the game is exported
6) The game listens for changes coming from the "RealTimeChange" Service
---On change: The appropriate Editor will process the change

The Service will perform all of the required Server connectivity, it will upload the specifications and state to the server, and the state can be changed on the server (through any client that is appropriate).  The Service (on the Android Device) will poll the server at set increments and determine if changes are required, if so, the client requests the updates and the appropriate editor will mutate the state of the program at runtime.

..Later I'll describe more on the Specifications, etc., and how the specifications are tied to the data and the server-side editors (a dynamic UI editor that that is created at runtime).

RoboGuice (Gone, baby, Gone)

So after fiddling around a bit with RoboGuice, I looked at how to design the application and not incur the cost of using "extends" and remove extraneous jars and annotations.  What I ended up with was (actually) simple, just have all of the components wired together in the "Application" class:

public class BeaconApplication extends Application {
public IGameMapLoader mapLoader;

public BeaconApplication () {
mapLoader = new GameMapLoader(this);
}
}


...then in my Activity:

public class BeaconActivity extends Activity {
IGameMapLoader gameMapLoader;

public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
BeaconApplication ba = (BeaconApplication)getApplication();
this.gameMapLoader = ba.mapLoader;
}


Generally speaking, I don't like the fact that the Activity now needs to know about the Application, but I sure like it a helluva lot more than:


public class BeaconApplication extends RoboApplication {
   protected void addApplicationModules(List modules) {
       modules.add(new BeaconModule()); 
   }
}


public class BeaconModule extends AbstractAndroidModule{
  private Application application;

public BeaconModule(Application application) {
this.application = application;
}

protected void configure() {
IGameMapLoader mapLoader = new GameMapLoader(this.application);
bind(IGameMapLoader.class).toInstance(mapLoader);
    }
}


public class BeaconActivity extends RoboActivity {
@Inject IGameMapLoader gameMapLoader;
} 


I could go back and utilize a "Module" to construct and swap out depending on the environment, but for the time being I'm not coupled to all this extension and annotation nonsense. Also, I could use "get()" methods rather than instance variables on the Application class which could provide me with a way to lazyily create things (although in practice thats usually not a good idea), but anyways for normal operation, the tradeoff to use Guice/RoboGuice in an Android application just seems like a bad idea. (You pay alot for not so much).