Date: 29-01-99
Source: InformationWeek
Subject: True Test Of The Web

As the Web moves to the core of business, testing of applications and
infrastructure is more important than ever. Vendors are supplying the tools.

By Alan Earls

Many companies doing business on the World Wide Web are one click away from
disaster. With Web sites playing an increasingly important business role,
it's essential that these systems deliver--whether their job is handling
product sales, distributing information, or linking up with partners' systems.

Of course, gaining that assurance has been anything but easy. The task of
testing
the performance of a Web application involves more than just running a
functional test on the specific application, though that's important. It's
also critical to ensure that all the supporting infrastructure works under
varying pressures and conditions.

A growing number of vendors are addressing that difficult task with testing
tools that range from freeware to complex bundles of products and services
that can cost $250,000 and up.

The high stakes of E-business mean it's no longer a question of whether to
test but how. Consider the experiences of those companies that have
suffered high-profile site failures. In October 1997, with a market rally
in process, E-Trade Securities Inc., the Palo Alto, Calif., pioneer in
online stock trading, suffered a lengthy and embarrassing service outage
when trading volumes got too high for its system to handle. The company
survived the crisis and made several important technical and procedural
changes--but not before seriously damaging the confidence of its customers.

Similarly, a glitch during unscheduled maintenance at Amazon.com forced the
site offline for several hours last year, prompting rumors that the online
bookseller had gone out of business. Although Amazon.com wouldn't reveal
the cost of the site shutdown, outside analysts point out that daily
business on the Amazon.com site can be as high as $400,000. And in order to
repair its customer relations, the company felt compelled to issue
discounts to those who claimed to have been unable to access the site
during the shutdown.

Troubled Debut
Worst of all might have been what happened to Egghead Software Inc. The
very weekend that the retailer closed its last store as part of a much
publicized move to Web-based electronic product distribution, its
Egghead.com site went down for two days. Again, the company won't say what
the glitch cost but outsiders note that Egghead probably has about $1
million a week in online sales, making a botched debut a matter of at least
a few hundred thousand dollars in lost revenue.

These three examples of Web-site failures are indicative of what companies
can expect as they put their most critical operations on the Web. Customers
have little tolerance for failure. So the questions for these Web-based
businesses are, who should test and how often?

New Venture, Or Old Rules?
Sandy Taylor, director of analyst services at the Software Productivity
Group in Natick, Mass., divides these risky new Web ventures into two
categories: companies that are developing a brand-new E-commerce Web site
with built-from-the-ground-up functionality, and companies that are simply
building Web functionality on top of their existing business rules and
infrastructures. Taylor says companies in the first group are the ones
taking the biggest risks and therefore are most in need of testing.
Companies in the second category also must test their applications, but
they at least have some existing metrics and experience to build upon.

Industry experts agree that Web testing is a good investment. Testing in
general has achieved a lot more visibility lately, aided in part by the
focus on the year 2000 remediation efforts. And, given the visibility of
Web sites, it's fairly easy to argue for testing, says Liz Barnett, a VP at
Giga Information Group. Barnett is especially bullish on the need to test
sites involved in E-commerce. Although large intranets and extranets can
face similar problems, she says, it's even more difficult for site
developers to anticipate load once an E-commerce site is opened to the
general public. Doing business on the Web is like opening a traditional
store but not knowing whether it will be located in the middle of an Iowa
cornfield or on New York's Fifth Avenue.

Furthermore, in the world of the Web, functional changes happen much faster
and more frequently than in traditional mainframe environments, which tend
to implement functional changes annually Client-server environments tend to
implement functional changes on a monthly basis. But Web sites sometime
need to make daily changes, which can impact back-end infrastructure and
system functionality.

Fortunately, the problems faced in the testing of Web applications are not
entirely new. Indeed, the history of most site-testing tools goes back to
the traditions of client-server computing. In fact, from a conceptual
standpoint, Web testing is much like client-server testing--but with more
potential tiers and greater load variability. With a Web site, there is no
way to predict with certainty how many users will hit a site at any given
time. The Web environment has also introduced extreme heterogeneity to the
support equation. Users may be accessing resources deep in a company's
infrastructure, and the IT department can't control what browser they will
be using. The potential interactions between technologies are almost
limitless.

That's why some analysts prefer the term Web application testing to the
more traditional functional and load testing. What matters is the
application seen by the users, not whether the Web site is having trouble
or the server is broken.

Three Functions Of App Testing
Within application testing there are three functional areas. The first is
site testing: a structural test that looks to see whether there are pages
that are inaccessible or whether popular features are unnecessarily hard to
access. The second and third areas deal with whether the application
functions as designed, and whether that functionality survives at various
load levels and under a variety of scenarios.

Vendors offer each of these capabilities in a variety of flavors and levels
of complexity. Some also offer integrated suites that provide life-cycle
testing.

The Web-testing market is young and fragmented. Prominent vendors include
Mercury Interactive and Segue Software, whose roots are in client-server
testing. RSW Software is a new subsidiary of hardware test giant Teradyne
and focuses exclusively on Web testing.

Mercury Interactive is generally regarded as the 800-pound gorilla of the
business, with a mature and well-respected product line that includes
LoadRunner and Astra. Segue is an aggressive company offering tools such as
SilkPerformer and a sophisticated scenario testing product called
LiveQuality (see story, p. 6A) that permits the generation of complex
interaction with large numbers of virtual users. Other players include
Compuware, Mindcraft, Rational Software, and RadView Software.

Web Watching
All of these products are currently being put through their paces in the
real world. E-Trade for example, is using Mercury Interactive's LoadRunner.
After its aforementioned meltdown, E-Trade vowed to never go through that
again, says Gary Kattge, director of quality assurance. E-Trade looks back
at that experience as positive because the company learned a great deal
from it. "Since that time, we've had much higher peaks, which we've handled
without any problems," he says.

The meltdown forced E-Trade to adopt a proprietary stateless architecture,
which enables the E-Trade Web site to provide secure, reliable, and almost
instantaneous service simultaneously to hundreds of thousands of users.

E-Trade, which rolled out its new and improved Web site this past summer to
all 500,000 active accounts, experiences about 2.2 million page views per
day, and expects further traffic increases as more noncustomers visit the
site. During the quarter ended June 30, E-Trade processed an average of
$1.5 billion in customer transactions per week.

In preparing for its public launch, E-Trade developed and installed a
technology infrastructure able to handle massive traffic volumes. E-Trade's
Stateless Architecture system allows more than 1 million visitors to use
the Web site simultaneously and permits as many as 150,000 customers to
place orders at the same time. E-Trade says the system can be scaled up to
handle even heavier loads.

Getting the system to that level of reliability was Kattge's job, and
testing was a big part of the project. Several testing-tool vendors were in
the running initially, Kattge says, but based on functionality and
extensibility, E-Trade reduced the field quickly to one: Mercury Interactive.

Mercury Interactive's LoadRunner product lets E-Trade put hooks in the test
code. This allows E-Trade personnel look in-depth at application response
time--right down to the SQL database-query level. The testing tool has
given E-Trade the ability to calibrate and verify new system designs and to
detect areas that might need additional tuning.

But LoadRunner's most critical role was in helping to measure response
times between various tiers, where the potential for serious errors exists.
LoadRunner is used on an hourly basis as part of an alert system to quickly
escalate any performance issues likely to require intervention.

Commonplace Tests
Even in the slightly slower-paced world of mutual funds, testing is
becoming commonplace. Mutual funds giant T. Rowe Price in Baltimore is a
strong proponent of Web testing. T. Rowe Price recently implemented its
first major Web site redesign. Time to market and availability were
critical to IT manager Tom Dietz, who has been looking at Web testing
issues at T. Rowe Price for more than two years. "We recognized Web-site
testing as a very important area," he says.

After evaluating testing tools, the company settled on Segue's
SilkPerformer, due in part to its regression testing, Web functional
testing, and load-testing capabilities, Dietz says.

First Call Corp. in Boston is another organization with a Web site that
gets put to the test. First Call keeps broker notes, research documents,
earnings, and buy-sell recommendations flowing to its Wall Street
professionals via its Web site. The company's roots in online services
predate the Web, but when it came time to embrace Internet delivery, they
selected Segue's SilkPerformer. "We were interested in testing costs, the
ability to simulate user loads, and memory footprints per virtual user,"
says First Call IT manager James O'Keefe.

FirstCall also wanted a good scripting language. While it was important to
demonstrate site viability through load and stress testing, it was equally
critical that the scripting tool be able to show that real information was
being delivered. SilkPerformer let the First Call team monitor specific
deliverables like HTML code, or a copyright symbol that could verify that
the site was delivering as intended.

Of course, not every mission-critical site is seen by the public. Paul
Roach, a senior quality assurance specialist in the Richardson, Texas,
office of Northern Telecom Ltd., got his exposure to Web-site testing
during the requirements phase of Nortel's global intranet development project.

Nortel's intranet consists of many applications deployed across a global
intranet--an environment that demands frequent testing of the Web servers
and applications. Scalability, robustness, and a rich feature set were on
Nortel's shopping list.

In one instance, Roach helped deploy an employee-satisfaction survey that
would be accessible by 70,000 employees worldwide. Other sites required
testing of functions as diverse as financial accounting, sales, and
marketing. Dropping the ball with employees, even on a satisfaction survey,
can be just as serious as offending paying customers, he says.

Cost Conscious
Choosing a product and determining what to spend are perhaps the greatest
challenges to the newcomer to Web testing. CBS.com has opted for a freeware
program. Compared with the tasks of supporting real-time stock trading or
other "heavy duty" E-commerce activities, Michael Skorski's job might seem
easy. Director of technology for CBS Net Media, Skorski makes sure
www.cbs.com is up and running when the network's viewers come looking for
program information and news. And the old adage about the short shelf life
of news applies to Web news sites--specially when there's scandal in the
air. "We did four times our normal traffic and an order of magnitude more
hits and page requests when Kenneth Starr's report hit," says Skorski.

Skorski and CBS have been testing their site extensively, which may account
for the fact that it survived the onslaught of Starr gazers with only minor
problems.

That's attributed in part to CBS's use of Webstone, a freeware product now
maintained and distributed by Mindcraft. "There are limitations on what it
can do," Skorski says, "but the price is right and the product is now part
of our testing regime."

Commercial testing tools have more bells and whistles, Skorski says, but do
not necessarily produce more useful information. His biggest challenge with
testing is to simulate the complex infrastructure of www.cbs.com.

Karl Van Neste, a software system architect at the College Board in Reston,
Va., warns buyers to beware. "Test managers need to continually review the
products and new capabilities available on the market." Van Neste has been
using a product recently introduced by a small company called RadView
Software. Van Neste says he first looked at Mercury Interactive's
LoadRunner but that RadView's product had what he felt was comparable
functionality for a small fraction of the price.

Indeed, new entrants keep cropping up almost daily, making it likely that
increased capability and decreased price will be the order of the day. IT
managers will also see a variety of service offerings for those shops that
don't want to experience the cost and learning curve of Web testing tools.

Illustration by Dennis Harms