Soapui Vs Soatest Comparison b1v6z

Automation and Performance Tool Evaluation SoapUI Pro vs Parasoft Soatest

SOA Testing Tool Evaluation The objective of this exercise was to evaluate automation and load tools for SOA implementations. We went through a similar exercise 9 months ago in which SoapUI Pro was not considered. This revaluation of our automation and load tool is driven by three factors: 1. Recent additions to SoapUI Pro functionality (inluding a load tool) which we felt now made SoapUI Pro competitive due to SoapUI’s relatively low price. 2. Our contract with Soatest is subscription based, we thought it best to justify the cost of our previous our tool choice before renewal.

2

Feature Comparison # Feature 1 Transport Protocol: Ability to test through different protocols (HTTP,HTTPS, JMS, MQ, RMI)

soapUI Pro This tool s HTTP,REST and HTTPS. MQ and JMS not ed

Parasoft SOAtest This tool s only HTTP,HTTPS,JMS ,REST, RMI, SMTP, TIBCO, .Net WCF HTTP & T and MQ (Sonic and WebSphere)

2 XML Schema Validation: Able to validate the data characteristics for each xml element 3 Request and Response SOAP messages - Able to interpret and monitor data - Ability to check for correctness in the response of the service 4 Response message comparison: - Raw String - XPATH Logical expression

Automatic schema validation

Automatic schema validation

The SOAP request can be parameterized using properties. Different types of assertions are available for checking the SOAP response.This feature is very easy to use in SoapUI. Response message comparision can be done effectively using basic assertions provided in the tool.

Request formation can be done through easy methods and UI availabe. Data parameterization can be done using various sources like, FileSyste, Excel, DB, etc.

5 Asynchronous Testing 6 Data Driven Testing

ed s data driven testing through scripting s adding check points in test cases s adding of check points in test cases to break down timings. to break down timings. s WSDL inspection and Invoking s WSDL Inspection, WSDL Validations, validations against BP 1.1

7 Ability to add check points 8 WSDL Inspection and Invoking

XML Assertor and Diff tool helps to do various kind of assertions and regression validation. for EXamXML tool is also available for XML Comparison. ed s data driven testing by default

3

Feature Comparison #

Feature

soapUI Pro

Parasoft SOAtest

9 Ability to create test cases without WSDL 10 Types of Test cases

Not ed

ed

13 WS-I, WS-Addressing, WSSecurity

The for WS-Security and tests are It s all managed from the WS-Security tab in the Project View and then applied to their desired targets at corresponding locations It is possibile to create Mock Stubs can be created Implementations, or Web Service Simulations, (called "MockServices") of any Web Service from its WSDL contract s Attachment testing s attachement testing

SOAP, REST, Browser Testing (UI), EJB, ISO 8583, WebMethods 11 Read Data from various input Using Groovy Scripts it is possible to read Data parameterization is ed sources (DB, XLS, text files) data from an external data source and write in easier steps it to another steps' properties. 12 s SOAP 1.1, 1.2 Soap UI s both SOAP versions 1.1 s both SOAP 1.1, 1.2 and 1.2

14 Web Service Simulation

SOAP Test case & REST test cases only

15 for Attachment Testing 16 for Regression Testing Regression test suites cannot be created but s for regression testing by retesting of the test suites can be performed creating regression test suites 4

Feature Comparison Feature

soapUI Pro

# 17 18

WSDL Scoring Assertions or Success Criteria

19 20

Baseline Regression Testing Automated Reporting

21

Reporting Types:

22

Reports Coverage:

23

Scenario based Reporting: Able to generate execution results reports based on given scenarios Validate WSDL WS standards: ed Able to WSDL schema to WS standards(Basic Profile 1.0,1.1,1.2)

24

Parasoft SOAtest

Not ed Few assertions are available such as: SOAP validation, X-path assertion, SOAP contains etc.

WSDL Tests does this XML Assertor and Diff tool helps to do various kind of assertions and regression validation. for EXamXML tool is also available for XML Comparison. Not Possible in SoapUI. Regression can be created For Automated Reporting, SoapUI Automated report is available includes a "Launch Test Runner"

Reports can be generated in XML, HTML, Text PlainText only Reports Generated are moderate Detailed reports are available as they do not give sufficient information and not easily interpreted by the end s Not ed ed

ed

5

Feature Comparison # Feature 25 Ability to test the design time interoperabilty of the service 26 Authentication & Authorization 27 Time Stamp 28 Signatures 29 Encryption & Decryption 30 Secure Socket Layer (SSL) 31 Additional Security Features 32 Installation and setup 33 Good interface 34 Short learning curve required 35 Minimal manual coding effort required to use 36 Minimal supervision during test execution 37 Easy to create test suites 38 Ability to group (add or remove) test cases in an existing test case suites 39 Easy portability for test suites and test cases for different groups to use 40 Ability to store tests and test suites for reuse

soapUI Pro

Parasoft SOAtest

ed

ed

ed ed ed ed ed Not ed Easy Moderate Moderate needs to know scripting Moderate Easy Easy

ed ed ed ed ed ed Moderate Good Good Less scripting required Tutorials and Help guide available. Easy Easy

ed

ed

ed

ed

41 Ability to update or sync up tests and test suites on Not ed ed different client machines 42 for integration with Rally Not ed ed through API 43 Easy integration with other components such as By Scripting we can integrate the Can be done easily Excel, Notepad, ODBC,Access other components like Excel, File Data Sources & Database 44 Compatible with Operating systems like windows Soap UI is compatible with windows, Yes, separate installers available XP,Linux,Mac,Solaris etc. Linux & Mac 6

Cost Comparison Tool Automation Load Implementation Total

Soap UI Pro $11,250.00 $360,000.00 $32,000.00 $403,250.00

Soatest $50,000.00 $36,000 (unlimited) $0.00 $86,000.00

1. These costs are based upon (50) 1 year subscription licenses for automation and 7500 vs for load 2. Soap UI Pro’s Load UI is not an acceptable tool for performance/load. A choice of Soap UI Pro requires an investment in Loadrunner vs. Where we have an enterprise version of Performance Center , we only have 2500 vs. The load cost for Soap Ui Pro reflects the purchase of an additional 5000 vs. 3. The cost to switch to rewrite regression scripts in Soap UI Pro reflects 4 offshore developers for 2 months. 4. Costs do not reflect any training expense or professional Services expense.

7

Scorecard Features Functional Testing Regression Testing Integration/End-toEnd Testing Database Testing Ease of Use Ease of Modifying the Scripts Reporting Security/Penetration Testing Performance Testing Active Tool In House Tool Expertise Cost Total

SOAPUI Pro 4 2 1 2 3 2 3 3 1 3 5 31

Soatest 4 5 5 5 4 4 4 5 4 3 4 2 49

8

Recommendation While SOAPUI Pro can be used for smoke testing and basic functional testing for simple request and responses, there is a quite considerable effort required to create and maintain regression suites for complex requests and responses. The recent addition of features to SoapUI Pro make it a more attractive option, but it still falls well short of the mark. The adoption of the tool by the developers in the scrum will help to push testing forward. Parasoft Soatest for Services testing serves all the pre-requisites and its ease to create and maintain automated regression and load scripts makes it by far the right choice for Enterprise Services

9