Some Thoughts on MVVM – Part 2

Relational database vendors stepped up to the plate

DEC RDB (Relational DataBase, I know, creative) had long since provided SQLMOD. These were source modules you compiled into object form which could then be added to your site specific library and used by any application. This is one of the reasons service oriented architecture was our first solution. It was very easy to whip up a server that accepted data streams in known formats from raw TCP/IP ports and message queues then chewed on it and returned a result.

The Java and Web developers began decrying, not unjustifiably, that the remote services were getting overloaded and taking too long to respond. Part of this was the volume being fed to them and the other part was “just add it here” syndrome. Many players are guilty of allowing “just add it here” syndrome to overtake SOA. Upper management did not want to pay for all of the time and people it would take to create, in a large corporate environment, a new service for each and every thing the Web needed. A systems analyst had to modify system start and stop procedures. The networking team had to assign and make open a network port. If you were using a major messaging handler like Websphere/MQ Series/insert-your-favorite-here as your external interface, you were almost forced to “just add it here” because it became a hassle to set up a completely new message route for each and every message. If there were only a handful of backend services receiving messages it was a simple matter to setup the message translation and just send it to one of the previously defined services.

Remember what I said in part 1?

The correct place to validate data is the last step/process which actually writes it to the database. Once it gets into the database it is too late.

Commercial database vendors and several OpenSource projects, created the stored procedure concept. Most of the database and business world started to realize that quote of mine was right. Their solution, however, was hideously flawed.

A stored procedure is much like an SQLMOD. It is some database specific code callable by any language able to access the database and invoke said procedure. Unlike SQLMOD it is not stored in some external library, but the database itself. In order to execute it your application has to connect directly to the database. This bypasses any and all safeguards. The database has to protect itself. In the SOA world if someone sends in data that your service cannot handle you either dump it to a log or your service crashes. While Denial of Service (DOS) attacks can cripple a business’ cash flow, they do not expose sensitive customer data to evil doers.

There are many different SQL injection techniques out there. If anybody actually knew all of them databases would be able to defend against all of them. Boiling it down to a thimble’s worth of content, here are the general classes I know about:

  • A massive amount of data is sent in for a single, poorly handled field/column causing some form of overflow error giving the following text the ability to act directly on the database.
  • A series of highly specialized “escape” or “control” characters are included in the data stream which exploit a back door to admin or other user level allowing the following text to operate directly on the database.
  • A stored procedure is poorly written and tested only for the “happy path” of good data so when bad data is sent the procedure itself gives access to the database in ways not intended.

Directly connecting a Web or external service to a database is always bad.

That was my belief when stored procedures were first introduced. If Java or any other Web language was allowed to directly access a database not owned by the Web service itself and outside of the firewall demons aren’t just at the gate, they are inside of your kitchen eating all of your food. Just ask Yahoo or any of the other Web companies having to fess up to breaches and identity theft. If you think that is old news read up on the HBO hack.

Your relational database, at least at the enterprise level, is supposed to be the repository of low level business logic and all of your data. It must be protected as sacred and access must be restricted. Stored procedures were a good idea. Making them callable from the Web was not.

I hear some of you asking “low level?” This is where it gets interesting and where a System Architect either shines or buries a company. Let’s take a look at some of the low level stuff.

Referential Integrity – Relational databases have provided this for decades. It is what stops you from deleting a customer master record when there are invoices/orders/whatever tied to that customer.

Column Level Validation – Say you are on a Web page about to order a pair of blue jeans. The jeans only come in certain waist sizes. Far too many Web pages code that logic in the page when it belongs in the database. Column level validation means you have a reference table and only values found in that reference table can be placed in the column. Referential integrity stops someone from deleting an entry in the reference table if there are any rows in this table containing that value.

Range and Duration rules – Many times range and duration rules get handled via column level validation, but other times they do not. Can a person weigh -256 pounds? Can blood pressure be less than zero? Some databases allow you to set numeric ranges directly on the column and others only let you restrict a column to positive numeric values and others allow you to assign a list or range of legal values. For those which don’t you have to create a stored procedure. If you are a wholesaler allowing stores to purchase soda by the 6, 12 or 24 bottle packages, you order system had best not allow someone to enter 3.

Duration rules are a bit more interesting. If you are a retailer who has sales, an item’s price is good from a certain time on a certain day to a different day and potentially different time. Trying to hack entry screens for this each and every week would be insane. Instead you have an item pricing table and a stored procedure. The procedure accepts an item code and timestamp then looks through your pricing table returning either an error code or a the price which was active for that item at that date and time. Some procedures get really complex involving customer specific discounts and quantity break pricing, which, of course, means additional parameters.

Oh come on, you’ve all been to a Web site and seen something like:

0-5   $12.95

6+ $9.95

That pricing logic needs to reside within the database so every interface shows the same values. Forget that Trivago commercial about the same hotel room having different prices on different sites. That is a completely different situation.

Some Thoughts on MVVM – Part 1

There is currently a lot of chatter in the IT world involving MVVM (Model-View-ViewModel). While there are merits, there are also drawbacks. One of the biggest drawbacks I’ve noticed is in the tutorials and documentation found on-line. Take a quick look at this article. Below is a quote from it.

The key to remember with the model is that it holds the information, but not behaviors or services that manipulate the information. It is not responsible for formatting text to look pretty on the screen, or fetching a list of items from a remote server (in fact, in that list, each item would most likely be a model of its own). Business logic is typically kept separate from the model, and encapsulated in other classes that act on the model. This is not always true: for example, some models may contain validation.

 

I made the last two sentences bold because they are what triggered this post. Much of what you find about MVVM comes from a PC or tiny system background. It views all data as an object and believes business logic should be placed in other classes. The writers of such things have never worked in an enterprise level environment.

Enterprise level applications all started out using some form of indexed file system. RMS, ISAM, VSAM were the predominant file types of the day during the late 1980s to mid 1990s. These were all we had. All business logic had to be coded within the application. This pushed systems analysts to force developers to create modular programs. Many of you have heard the phrases before:

  • source/copy libraries
  • external subroutines
  • site specific object libraries
  • company specific reference manuals

External source libraries started rather early on. COBOL provided the Copy Lib concept where you could pull in source code from external directories and text libraries. BASIC and other languages followed suit with various forms of %INCLUDE. While you _could_ include source anywhere in these modules, with BASIC, FORTRAN and a few other languages where variable declarations weren’t required you had to worry about variable collision. If you used a variable, say X to store a value you needed and the included source used X as a local scratch variable, interesting bugs occurred.

COBOL was a bit safer because each Copy Lib containing source paragraphs had a corresponding Copy Lib for the WORKING-STORAGE section. The compiler would alert you to a duplicate declaration, usually. You could still have the same variable name occurring at different levels in both the I-O SECTION and WORKING-STORAGE. If you tried to use X without saying X IN PAYROLL-RECORD or X-MY-SCRATCH-VARIABLES the compiler would usually flag it so you knew in advance you had a problem.

External subroutines came about rather quickly. When programming in BASIC with a version which required each line to have a unique line number and a language imposed line number range of 1-32767 you simply couldn’t fit all of the required logic into a single source file. Eventually BASIC standards changed to the point you needed only the first and last line number and finally the line number requirement was pretty much removed. Even so we had compiler limitations. More than once I encountered a source file which exceeded the compiler input capacity. For languages which did not require variable declarations, external subroutines also got around the collision problems. Each individually compiled routine had its own variable scope. Any values it needed from the caller were passed in.

After a few years, maintaining all of these external subroutines and functions within build procedures for applications became a real hassle. We started creating site specific object libraries so if some compilation parameter or such needed to change because of a change to a specific routine, we didn’t have to go update the build procedures for hundreds or thousands of other applications.

It shouldn’t take a big leap of faith to believe all of that site/company specific business logic became really hard to keep track of. It should be an even smaller leap of faith to realize many site specific libraries had multiple routines which did the exact same thing or quite close to it. This lead to semi-official librarian positions. One or more persons in charge of maintaining the site specific libraries and updating a company specific reference manual. Various code review steps were put in place to ensure programmers weren’t spending their time re-inventing the wheel.

All of this evolution is why companies became single language shops. COBOL, BASIC, FORTRAN or whatever the first major application was written in became the language of the shop. Once you started creating your business logic in a language and incorporating it into various libraries you wanted to keep it all in one place. OpenVMS had the OpenVMS calling standard. If you adhered to the calling standard when declaring parameters, you could, quite literally, have your business logic library written in many programming languages and use them from many others. Other platforms strictly enforced language specific calling standards making it difficult, if not impossible, to call a COBOL module from FORTRAN or vice versa.

Shops started needing more languages. The Internet came along after all. Now we needed Web pages and Web services to also incorporate 20+ years of business logic nobody remembered. The first major attempt at solving this problem was Service Oriented Architecture. You may have heard the term “Data Silos” bandied about. By and large this meant each division of each company had its own set of data and applications containing the business logic. Sadly, data entry screens contained a large portion of the business logic. Applications which didn’t use forms managers like DECForms and CICS tended to code portions of the business logic directly in the form rather than making each field validation an external subroutine in the company library.

Even if programmers tried some of the validation was provided by the forms packages themselves. Few thought about writing external date/numeric/integer/limited-character-string validation routines because they were so easily enabled in the forms managers. I’ve been doing a lot of Qt application development for embedded systems over the past years and most of the applications I see/work on have these same validations being provided by the Qt library. In the defense of these systems, they have almost all been one-off embedded systems having touch screens. The data they collect gets sent off to back end services which also have to provide their own validation because they receive data from many different sources.

The correct place to validate data is the last step/process which actually writes it to the database. Once it gets into the database it is too late.

It should not come as a surprise to anyone that, first Java, then many other Web language developers ended up recreating various forms of the same business logic which had been incorporated into the original “green screens.” For Java, at least initially this meant they had to roll their own validators. Later, various libraries such as SWING provided some rudimentary validation routines.

The major fly in the ointment was and still is date validation. You cannot validate a date string until you know the format. You cannot convert a date string to some internal binary representation if it is invalid. True, you can write a routine which accepts both the string and a format string such as:

"01-01-2017", "MM-DD-YYYY"

But you cannot write a routine which just validates the date string itself without the format because 01-01 could be MM-DD or DD-MM.

 

What is SOA?

Oddly enough, this question came up recently. Actually it came up more in the form of “Can you define SOA for me?” When you work in IT every day and design solutions to problems, not just code, your first reaction is “That’s like asking an auto mechanic to define a half inch wrench.” Later, you see the reasoning behind the question.

To some people SOA is the exposing of heritage data silos to new applications and dashboards used by upper management when making decisions. While this is a tidy little definition, and where a lot of SOA efforts start, it certainly isn’t SOA.

To some people, it means “putting your business on the Web and getting rich!” This really has nothing to do with SOA, but, it plays well in MBA circles.

To some people it means using XML for all communications. While XML can be used for all external communications, it certainly should not be used for internal communications, nor is it a requirement of SOA.

To some people, it means a chance to sell massive quantities of overpriced middleware to customers that have no idea about what they are buying. While marketing always thinks with the darkest of intentions, this really isn’t SOA either. Oddly enough, on extremely rare occasions, something resembling SOA can result.

SOA is the salvaging of business knowledge by exposing pieces of that knowledge to newer and trendier platforms. This allows the continued use of a highly stable heritage system while new services are created hooking its services and the services of other heritage systems together in business meaningful ways. Some, not all, probably not even the majority, will be exposed to the outside world via the Web or a VPN. While many view it as the retirement of all Heritage systems, it’s really the continuance of their never ending life. If your Heritage system was around to be updated during Y2K, it will most likely outlive your company. That trendy thing you are using today, it won’t be here next year. Doubt me? Search the job boards and tell me just how many postings you find for Windows 3.1, OS/2, or Wang developers. At one point in time, each of these were very trendy skills to have.

2009 Finalist

Logikal Solutions is proud to announce that “The Minimum You Need to Know About Service Oriented Architecture” ISBN-13 978-0-9770866-6-5 has been declared a finalist in the 2009 Eric Hoffer Awards. This award competition is much more general that there is no category for computer related books, they must compete with all other business and reference books.

Many of you reading this may also be aware that “ The Minimum You Need to Know About Service Oriented Architecture” also won a Best Book Award from USA Book News, which does have a category for computer books.

 

SOA Book Wins National Award

The Minimum You Need to Know About Service Oriented Architecture by Roland Hughes

ISBN 0­9770866­6­6

ISBN­13 978­0­9770866­6­5

Award-Winner in the Business: Technology/Computers/Internet category of the National Best Books 2008 Awards, sponsored by USA Book News