Testing an App with the Jasmine Test Framework

Jasmine is a behavior driven development test framework for JavaScript. With App Factory, you can use the Jasmine framework to run test cases for the Native and Desktop Web platforms. For more information, refer to the following sections.

NOTE:
If you want to run Jasmine tests for the Native channel, or if you are using version V9 ServicePack 2 or earlier, you can run Jasmine tests only during the build process (buildVisualizerApp).

Prerequisites

Custom Test Environment

Jasmine tests always run in a Custom Test Environment. App Factory uses the default Appium version to launch the app on the test device. But, you can provide a version number in the Build Parameters to override the Appium version. You can also provide the name of the Test Runner file that you want to run on the test device.

Building and running a test

To build and run a test, follow these steps:

1. From the Visualizer folder of your project, navigate to the Tests sub-folder.
2. For the runTest job, click the Build with Parameters icon on the right side of the row.
   Alternatively, select the runTest job, and then from the left panel of the details page, select Build with Parameters.
3. Configure the build parameters for the runTest job. For more information about the parameters, refer to the following sections.
   
   Source Control

   Parameter	Description
   PROJECT_SOURCE_CODE_BRANCH	Specifies the branch, release tag, or the commit ID of the repository that contains the test files. The tests must be placed at the following location: <Repository-Root>/<Project-Root>/test/Jasmine/ The <Repository_Root> must be configured in the Project Settings. The <Project-Root>is the name of the project that is configured while creating the Visualizer project in App Factory.

   Testing

   IMPORTANT: To run tests by using the Jasmine Test Framework, make sure that your app is built in the test mode.

   Parameter	Description
   TEST_FRAMEWORK	Specifies the framework that you want to use for testing the app. The test framework can be TestNG or Jasmine.
   RUN_NATIVE_TESTS	Specifies whether App Factory must run tests for the Native platform. If you enable this parameter, the console displays additional parameters for Native tests. For more information, refer to Native build parameters.
   RUN_DESKTOPWEB_TESTS	Specifies whether App Factory must run tests for the Desktop Web platform. If you enable this parameter, the console displays additional parameters for Desktop Web tests. For more information, refer to Desktop Web build parameters.
   OVERRIDE_TEST_DATA	Specifies whether custom data must be used during the test run. If you enable this parameter, App Factory displays the following parameters: WEB_CUSTOM_DATA_PATH NATIVE_CUSTOM_DATA_PATH NOTE: Custom data is only applicable for Jasmine tests.
   RERUN_FAILED_TESTS	Specifies whether App Factory tracks the test suites that have at least one failed test case and reruns the test suites after the first test run is complete. To use this feature, the test suite must be written such that it can be run independent of other test suites. The test results for the rerun are included in a separate section of the notification email. For more information, refer to Build Results Notification. IMPORTANT: This parameter is displayed only if Jasmine is selected as the test framework and either RUN_NATIVE_TESTS or RUN_DESKTOPWEB_TESTS is enabled.

   Native build parameters

   The following parameters are applicable only if you want to run automated tests on a native app that is generated by the protected build.

   IMPORTANT:

   • To run the tests individually, make sure that you provide public URLs in the Native parameters.

   • You can use the universal application binary and run the tests on mobile and tablet devices based on the availability in the device pool. You can trigger a runTest job either with a universal binary or with a mobile or a tablet binary.

   

   Parameter	Description
   ANDROID_UNIVERSAL_NATIVE_BINARY_URL	Specifies the URL of the Android universal binaries that are generated by a build job or generated locally.
   ANDROID_MOBILE_NATIVE_BINARY_URL	Specifies the URL of the Android Mobile binary that is generated by a build job or generated locally.
   ANDROID_TABLET_NATIVE_BINARY_URL	Specifies the URL of the Android Tablet binary that is generated by a build job or generated locally.
   IOS_UNIVERSAL_NATIVE_BINARY_URL	Specifies the URL of the iOS universal binaries that are generated by a build job or generated locally.
   IOS_MOBILE_NATIVE_BINARY_URL	Specifies the URL of the iPhone binary that is generated by a build job or generated locally.
   IOS_TABLET_NATIVE_BINARY_URL	Specifies the URL of the iPad binary that is generated by a build job or generated locally.
   NATIVE_TESTS_URL	Specifies the URL of the binaries for the compiled test project. If you provide Git parameters, you can skip this parameter.
   AVAILABLE_TEST_POOLS	Specifies the device pool that must be used for the tests. If a device pool is empty, the build fails. For more information, refer to Configuring Device Pools.

   NOTE: The URLs that you provide must be open and accessible on the internet. For example, a URL to an S3 bucket from the email notification that is sent by the buildVisualizerApp job.

   Custom Test Environment related parameters

   In a custom test environment, you can select the Appium version on which you want to run your tests. Make sure that you provide an appropriate Appium version. For more information, refer to AWS Device Farm.

   Parameter	Description
   APPIUM_VERSION	Specifies the Appium version that you want to run for the tests. For more information, refer to Custom Test Environment Run. For information about the artifacts that are available in the notification mail for AWS Custom Environment Run, refer to Building an App in AWS Custom Environment.
   TEST_PLAN	Specifies the test plan that you want to execute. The test plan files must be located at <Project-Root>/testresources/Jasmine/Mobile/TestPlans
   NATIVE_CUSTOM_DATA_PATH	Specifies the path of the custom data that you want to use for the native tests. The path is relative to the channel directory in the testresources/Jasmine folder of the Visualizer project source. For example, if your custom data files are located at /testresources/Jasmine/Mobile/customTestData/DevData/DataFile.js, then the relative path is customTestData/DevData.
   NATIVE_TEST_PLAN	Specifies the relative path of the test plan (such as testRunner.js) that you want to execute. The path is relative to the following folders: /testresources/Jasmine/Mobile/Test Plans /testresources/Jasmine/Tablet/Test Plans NOTE: If the test plan is not provided, the default test plan (testPlan.js) is executed.

   Desktop Web build parameters

   

   Parameter	Description
   WEB_APP_URL	Specifies the URL of the Fabric app with which the Desktop Web WAR file is published.
   WEB_CUSTOM_DATA_PATH	Specifies the path of the custom data that you want to use for the web tests. The path is relative to the channel directory in the testresources/Jasmine folder of the Visualizer project source. For example, if your custom data files are located at /testresources/Jasmine/Web/customTestData/DevData/DataFile.js, then the relative path is customTestData/DevData.
   WEB_TEST_PLAN	Specifies the relative path of the test plan (such as testRunner.js) that you want to execute. The path is relative to the following folder: /testresources/Jasmine/Desktop/Test Plans NOTE: If the test plan is not provided, the default test plan (testPlan.js) is executed.
   AVAILABLE_BROWSERS	Specifies the browser on which you want to run the Desktop Web tests.
   SCREEN_RESOLUTIONS	Specifies the resolution at which the browser must launch and run the tests.
   RUN_WEB_TESTS_ARGUMENTS	Specifies additional parameters that must be passed in a Maven command to the Web tests.

   Custom Hooks

   

   Parameter	Description
   RUN_CUSTOM_HOOKS	Specifies whether Custom Hooks must be run in the after the tests are executed (the Post-Test stage). For more information, refer to Custom Hooks.

Running Tests from the buildVisualizerApp job

You can trigger the tests from the buildVisualizerApp job available at Project Name > Visualizer > Builds folder. It is the simplest way to validate your code by launching tests right after Building the application. There is no need to specify all parameters as they are provided while running the tests manually. The results can be viewed in line to the Test Results of Manual Tests.

Test Results

Native test Results

When the project test build is complete and the test results are published at S3, e-mail notifications are sent to recipients:

On an unsuccessful build for the tests run, the following e-mail is sent to the recipients:

Sample email for Native tests

In the final stage, the consolidated test run status is notified via e-mail. Refer the following screenshots for the sample email for consolidated status.

To download test data that is fetched from the AWS Device Farm, click the Download File link. The data is stored at S3 and available for review without any time limitations.

To view the detailed test report, click the Test Report link.

Desktop Web test results

When the project test build is complete and the test results are published at S3, e-mail notifications are sent to recipients:

On an unsuccessful build for the tests run, the following e-mail is sent to the recipients:

Sample email for Desktop Web tests

In the final stage, the consolidated test run status is notified via e-mail. Refer the following screenshots for the sample email for consolidated status.

The summary of the test results are shown based on the tests executed on specified browser that includes passed, failed, skipped. Usually the tests are skipped when the driver initialization fails.

To view the detailed test report, click the Detailed Test Report link.

To view the details on the browser log, click the Browser Console Log link.

Test Status

Test Status is available in the test result email which is sent to the recipient after you execute the test cases. Following are different statuses in the test result email:

• PASSED: The status of the test result is PASSED when the test case is executed without any issues, and when the assertion is successful.
• FAILED: The status of the test result is FAILED when the expected result does not match with the actual result of the test case, and the assertion fails.

The SKIPPED, WARNED, STOPPED, ERRORED statuses are not applicable for Jasmine Tests.

Troubleshooting Issues

The following errors can help you during troubleshooting if the runTest job fails:

• Error: Jasmine Tests can only be executed when the app is built in Debug mode.
  Solution: Make sure that you have selected the build mode as debug when the app is built before running the tests.
• Error: Please provide pool to test on.
  Solution: Make sure that you provide a value to AVAILABLE_TEST_POOLS parameter to run the native tests.
• Error: Sorry, You can't run test for Android Universal binary along with Android Mobile/Tablet.
  Solution: The universal binary tests both the Mobile and Tablet binaries are based on the availability of the devices in the device pool. So it is not required to provide mobile/tablet binary when you give universal binary.
• Error: Build parameter <parameter name> value is not valid URL!
  Solution: Make sure that the URL you provide for the following parameters is valid:
  • ANDROID_MOBILE_NATIVE_BINARY_URL
  • ANDROID_TABLET_NATIVE_BINARY_URL
  • IOS_MOBILE_NATIVE_BINARY_URL
  • IOS_TABLET_NATIVE_BINARY_URL

  • NATIVE_TESTS_URL

  • ANDROID_UNIVERSAL_NATIVE_BINARY_URL

  • DESKTOPWEB_TESTS_URL

  • IOS_UNIVERSAL_NATIVE_BINARY_URL

The builds that failed during the test can be tracked for the issues that caused the failure. For example, consider build #4 from the following screenshots of runTests job. To troubleshoot the issue, follow these steps.

1. From your project folder, navigate to Visualizer > Tests.



2. Select the runTests job, and then find the failed build.
   For example, find and click the link for build #4.



3. From the details page of the failed build, select the Console Output on the left panel. The Console Output Log opens in the right pane.



4. Scroll to find errors in the log that caused the build to fail. For example, you can trace an error that is related to invalid Android Tablet URL.



5. Check if the ANDROID_TABLET_NATIVE_BINARY_URL parameter is valid.

Alternatively you can troubleshoot the issue from the stages of Manual Testing.

To troubleshoot the issue from the Stage View, follow these steps:

1. From the Stage View, point to the failed stage, and then click Logs.
   
2. From log messages in the Stage Logs window, identify the error message that corresponds to the failure of the build.
   For example, you can trace an error that is related to invalid Android Tablet URL.



3. Check whether the ANDROID_TABLET_NATIVE_BINARY_URL parameter is valid.

Auto-Triggering Tests

App Factory supports two types of auto-triggered test jobs: scm-triggered (Watchers) and cron-triggered (Schedulers). Every project contains a folder called Tests in the Visualizer sub-folder, which contains a job for creation of auto-triggered jobs (createTest) and two sub-folders where created jobs are stored.

To setup an auto-triggered test job, follow these steps:

1. From the project folder, navigate to the Tests sub-folder.
2. Select the createTest job.



3. Configure the CRON_SCHEDULE parameter based on your requirement.
   • If you want to create a Scheduler, type a Cron expression that corresponds to the time at which the build job must be triggered.
     For information about cron expressions, refer to Cron.
   • If you want to create a Watcher, leave the CRON_SCHEDULE parameter blank.
4. Configure the remaining parameters, which are the same as the runTest job.



5. After you configure all the parameters, click BUILD.