New Features:

A new feature for Data Cleansing is introduced to the Data Preparation plugin. The Data Cleansing option can allow users even with a non-technical background to view, profile, and cleanse data in a user-friendly interface.

Any input added to the data preparation pane will have a new tab ‘Data Preparation’ with the ‘Data Cleansing’ option by clicking which the user-friendly UI for data preparation will be launched as a new tab.

The ‘Data Cleansing’ tool comes with the following pre-loaded features:

1. View data in familiar excel like tabular form with pagination and resizable grid
2. The following actions can be performed on Data:
1. Data Profiling
2. Check for Data Type Accuracy
3. Identification of mismatched Data Types
4. Detection of anomalies Patterns
3. The ‘Data Quality’ bar visually flags the presence of invalid or blank data in columns
4. The ‘Info’ tab displays a common summary of the statistical quantities (like min, max, average) for all the columns
5. Filter data anomalies and perform deeper investigation through one-click on the displayed bar chart
6. Suitable Transformation actions for each column appears automatically
7. The list of available transforms:
1. Delete Columns
2. Delete Rows with Invalid Cells
3. Delete Rows with Empty Cells
4. Change to Upper Case
5. Change to Lower Case
6. Change to Title Case
7. Clear Cells with Negative Values
8. Remove Fractional Part
9. Clear Cells on the Matching Value
10. Delete Rows on the Matching Value
11. Fill Empty Cells with Text
12. Concatenate Columns
13. Round Value using Ceil Mode
14. Round Value using Floor Mode
15. Round Value using Down Mode
16. Round Value using half-up Mode
17. Remove Part of the Text
18. Remove Trailing and Leading Characters
19. Remove Consecutive Characters
20. Search and Replace
21. Add, multiply, subtract or divide
22. Fill Cells with a value
23. Format Numbers
24. Rename Column
25. Convert Duration
8. All the performed transformations will be saved and displayed in order on the ‘Data Cleansing’ page
9. Identifies the datatype of every column
10. An ‘Export’ option is provided to export the performed transformations/cleansing back to the Data Preparation UI

Enhancements:

1. Input component: Excel Sheet is integrated as a data source

New Features:

1. The Neural Network workspace has been integrated to the Predictive Workflow with the following features:
   1. Saved Workflows – All the saved NN workflows will be added to this node
   2. Data Sources
      1. CSV
      2. Data Service
      3. Data Store
   3. Prepackage Models
      1. Sentiment Analysis Models with related predefined scripts (other models can be added as per the requirements)
   4. NN Models
      1. Create New Model
      2. Saved NN Models
         1. Tensor-board for NN Model training visualization
   5. Custom Python Scripts
      1. Create New Script- The NN Workspace provides two options to be chosen for the Custom Python Scripts:
         1. Normal Python Script
         2. NN Model Object File Script
      2. Saved Script
   6. Model Training (and Retraining)
      1. Additional support for creating NumPy file creation script is provided
   7. Apply Model
      1. It supports NN Apply Model
   8. Data Writer
      1. Data Store Writer
      2. File Writer
         1. CSV
         2. JSON
      3. Data Base Writers
         1. Internal Data Writer (RDBMS Data Writer)
2. The ‘Predefined Scripts’ node is provided for existing users and new users in the R and Python Workspaces
3. The following predefined R scripts are added to the ‘Predefined Scripts’ node:
   1. Performance Metrics
   2. Prophet Forecasting
   3. Boruta Model Select
   4. Anomaly Detection
   5. ETS Forecast Model
   6. Weekly Forecasting
   7. SVM Models
   8. Random Forest
   9. Weekly Forecasting
   10. SVM Models
   11. Random Forest
4. The ‘Predefined Scripts’ node in the R Workspace
   1. The predefined script node lists while creating a new space
   2. The script summary includes both model and performance accuracy summary
   3. Proper visualization is provided for the following scripts:
      1. The Boruta Script
      2. The Random Forest Script
5. Process-based logging is provided in the R Workspace
6. UI Modularization has been completed for all the Predictive Workspaces

Enhancements:

1. The following enhancements are provided in the R and Data Preparation Workflows:
   1. Mechanism to auto kill the enqueued process has been provided to avoid chocking of PA Queue after a configured ideal timeout period
   2. Centralized WebSocket to Support Distributed deployment
   3. Centralized storage for Distributed deployment
   4. Improved Concurrency for workflows and scheduled jobs
2. R Regression analysis (R Linear and R Multiple Linear) is provided with the ‘Intercept’ option
3. The ‘XG Boosting’ option is provided for the R CNR Tree algorithm
4. Spark Workspace:
   1. Spark components now support the JDBC reader and writer
   2. Spark version has been upgraded to 2.3
5. NN Workspace:
   1. A new queue has been introduced in the ‘Celery Queue’ for NN Model training
   2. A saved NN Model is provided with ‘Delete’ and ‘View’ options
6. Python Workspace:
   1. Python MySQL Library is updated to PyMySQL-0.8.1 for performance improvement
   2. Python server can utilize HTTP/HTTPS for services
7. Data Source:
   1. The ‘Properties’ tab for CSV files is provided with a proper header.
   2. A backend validation has been added to check whether the Number of Headers and Columns are matching in the data extracted from a CSV file
8. Master Settings will provide ‘Max Process Allowed Count,’ ‘Clear Cache Time,’ and ‘Process Timeout’ information.
9. A centralized location has been provided for File upload, Cached files, Cached models, and Saved models
10. Environment variables are provided for PA properties
11. Clear Cache- Relevant modifications are done for a distributed environment

Notes:

1. Users must provide the model variable name as ‘model’ for NN Model Structure Script. The users should create a Model Structure to compile parameters.
2. Users should use ‘SHARED_PATH’ variable in their script for NN Model Script to use Pre-Package Models’ file. Moreover, to use the user-defined models’ supporting files, they should use ‘FAKE_PATH’ in their scripts.
3. Users can access the supporting files for Normal Python Script by using, ‘FAKE_PATH’+ + ‘/’ +
4. Connect an NN Model before the test data into ‘NN Apply Model’ while creating a Predictive Workflow.