Monday, February 18, 2019

Loading large flat files via SAP GUI into BW

Recently I needed to upload a large flat file into BW object. A size of the file was above 100MB. I was using classical data flow for uploads of the flat files. Means the file was uploaded via infopackage to PSA. From there there was a DTP to upload it further.

When I ran the upload to the PSA I got following error in the upload monitor:

Not enough memory for Data Provider
Message No. FES013

SAP GUI file functions have a limitations on the size of file to be uploaded / downloaded. That was clear to me. However my impression was that the issue can occur with the files a way too larger than just 100MB. However when I did a split of the file into smaller chunks I realized that ideal file size that work within any issues is below 20MB. Which is quite small portion of data. I used SAP GUI version 740 which is quite new. Therefore this finding was a surprise to me. 

Within SAP Note 875871 they mention that this limit is tied to user workstation where SAP GUI is running. They specifically mention that issues are present when there is only 1 or 2 GB of RAM available at user workstation. This wasn’t really my case as I worked with laptop having 8GB of RAM.

All in all it is quite surprising that in today’s age of big data there is such issue. Only solution is to either do split of files to smaller chunks (ideally below 20MB) or upload the whole file into application server of BW and upload it from there.

More information:
875871 - DP: Memory bottleneck when you upload very large files

Monday, January 21, 2019

SAP IDES SAP ERP 6.0 EhP8 installation

Here is my latest experience with installing IDES system. I installed the latest available IDES system to date which is ERP 6.0 EhP8 aka SAP Business Suite 7i 2016. Installation is WINDOWS OS based on MS SQL as database. VirtualBox was used as virtualization environment.

1. OS installation:
I used Windows Server 2012 R2 as an OS. After the OS was installed I did following setup:
1.1 install Guest Additions in VirtualBox to enable folder sharing: in VM go to disk D - VirtualBox Guest Additions and run installer (e.g. C:\Program Files\VirtualBox\VBoxGuestAdditions.iso)
1.2 change host name of VM to 6 char only
1.3 enable .NET 3.5.1 manually for the Windows Server via Service Manager -> Local Server -> Manage -> Add Roles and Features Wizard -> Features
1.4 modifying host file with following entry:
#SAP ECC 6 EhP8      ides ides.dummy.nodomain
1.5 Switch off firewall in Windows from PowerShell in elevated mode by entering the following command:
Set-NetFirewallProfile -enabled false
1.6 make sure you have at least 350GB available space on hard drive where the system will be installed.

2. DB installation:
From installation medias choose one marked as SQL Server 2014 for SAP and run installation via file SQL4SAP.bat
Script takes approximately 20 minutes to complete.

3. SAP system installation:
3.1 Preparation check
Run SAP installer from command line located at media marked as SWPM. I used SWPM of version SP24 (Software Provisioning Manager 1.0 SP 24). The SAP installed itself is executable file sapinst.exe. The SWPM used to be regular Win app not web based before. Not sure if the web app is the best but it looks working for me.

The installer open default web browser and points to URL: https://:port/sapinst/docs/index.html

which is UI of the installer. Here I choose SAP Business Suite 7i 2016 -> EHP8 for SAP ERP 6.0 ABAP -> MS SQL Server -> Preparations -> Prerequisites Check.

On next screen there is a possibility to choose what shall be checked. Safe option is to leave it as it is to run just default checks.

Next screen is to browse for installation media of Kernel.

Once the path to the Kernel is checked it open the files in there and checks them. After the successful checks a Status is set to available. If it is not available you need to provide path to proper Kernel installation media.

On next screen the installer provides results of Prerequisites checks.

Some things listed I just ignored and went further by selecting No button to do not check them again.

Almost final screen of prerequisite check screen.

And finally the final screen of prerequisite check screen.

3.2 Installation it self

Start sapinst.exe again for next phase. From very first screen from installation UI based in web browser select: SAP Business Suite 7i 2016 -> EHP8 for SAP ERP 6.0 ABAP -> MS SQL Server -> Installation -> Application Server ABAP -> Standard System -> Standard System.

Here I said Custom type of the installation:

Restart of OS due to user switch:

Installation resumes automatically afterwards.

Specifying SID for the SAP system:

Specifying domain:

Specifying master password:

 Settings related to WIN domain:

Specifying passwords for SAP users on OS level (by default passwords are copied from master one):

Specifying connection to the database:

Confirmation of and DB creation:

Browse for installation media. Starting with SAP Kernel.

Once the Kernel media are okay screen looks like this:

Again run of prerequisite checks:

Announcement that not all pre-req checks were successful. This can be ignored by opting to do not repeat checks.

Software package browser this time for SAP Host Agent component:

More options for SAP Host Agent related to Windows Domain:

Passwords of user under which of SAP Host Agent will run:

Specifying path to media for Data export CD1:

Specifying path to media for Data export CD2:

Specifying passwords to the database:

Specifying size of system. Off course I went with the smallest option:

Location of the database file can be adjusted on this screen:

Database temporary DB files configuration:

SQL Server memory config related settings:

Declustering / decoupling options. As this is NetWeaver 75 based system this is mandatory – it can’t be really skipped. More on this topic can be found here.

This operation takes huge amount of time during installation. All cluster and pool tables are being reorganized.

By settings related to DB import you can speed up installation if there is a powerful hardware available. 

Primary Application Server Instance and ABAP Central Services Instance, I just left it as it came up by default.

ABAP Message Server Ports and Transport Host related settings.

ICM User Management for the SAP Web Dispatcher related settings.
 SLD Destination for the SAP System OS Level related settings. 

Message Server Access Control List related settings, not applicable in my case.

Additional Components to be included in the ASCS Instance

SAP Web Dispatcher Parameters

Secure Storage Key Generation

Generated secure key (needs to backup):

Review parameter list:

Installation now runs:

Importing data to database phase:

Most of installation time was spent on declustering / decoupling operations. In my case it was almost 24+ hours. That runtime I had with intel i7 processer on just 1 core and 16GB of RAM.
Progress of import DB phase can be seen in log file located in:

"c:\Program Files\sapinst_instdir\BS2016\ERP608\MSS\INSTALL\STD\ABAP\import_monitor.log"

When progress reached Service Completion at 57% there was following error that can be ignored:

Package 'SAPVER_CLUSTR' not loaded.
Not all packages are loaded.
DIAGNOSIS: For details see output file with missing packages invalid_packages.txt and log file package_checker.log.

Normally it indicates the data load error but in some special cases (for example, if some packages were processed externally) you can choose OK to continue.

Afterwards install continues with re-writing phase:

Running of procedures phase:

File actualization:

Attempt to start SAP instances:

During start of SAP instances phase I got following error:

An error occurred while processing option SAP Business Suite 7i 2016 > EHP8 for SAP ERP 6.0 ABAP > MS SQL Server > Installation > Application Server ABAP > Standard System > Standard System (Last error reported by the step: ABAP processes of instance PCT/D00 [ABAP: STARTING] did not start after 10:10 minutes. Giving up). You can now:
• Choose Retry to repeat the current step.
• Choose Log Files to get more information about the error.
• Stop the option and continue later.
Log files are written to C:/Program Files/sapinst_instdir/BS2016/ERP608/MSS/INSTALL/STD/ABAP.

It turned out that issue was related to paging file of my WINDOWS OS. I had to manually increase the file if the page file via - Click right to This PC -> Properties -> Advanced System Settings -> Advanced -> Performance -> Settings -> Advanced -> tab Advanced -> Virtual Memory. Change page file size to minimum 10240 and maximum 20480.

In case you are stuck in this step with similar error, here’s’ what you can try:
- Check logs files like dev_w0 and dev_disp.

- Kill all other programs running in parallel to your SAP installation. The install may need all computer resources to complete successfully.

- Try to start the instance manually and proceed with the installation.

- See SAP Note 2535340 - ABAP processes of instance [ABAP: UNKNOWN] did not start after 10:10 minutes. with 28: No space left on device in dispatcher trace file while installing a additional dialog instance.

Analyzing of log files ended up with following error:

An error occurred while processing option SAP Business Suite 7i 2016 > EHP8 for SAP ERP 6.0 ABAP > MS SQL Server > Installation > Application Server ABAP > Standard System > Standard System(Last error reported by the step: The JVM reports an exception during execution of class ( ) and function executeStepClass. DETAILS: The reported error message from JVM is: java.lang.Exception: Immediate start not currently possible at ( at at at at ). You can now:
• Choose Retry
to repeat the current step.
• Choose Log Files
to get more information about the error.
• Stop the option and continue later.
Log files are written to C:/Program Files/sapinst_instdir/BS2016/ERP608/MSS/INSTALL/STD/ABAP.

To solve this error again (as I did in case of instance not started error) I just tried to cleanup up memory as much as possible by removing not needed programs. And retrying to execute the step.

Execution of report RUTTTYPSET – this step is also time consuming due to heavy memory requirements. It might be that this reports is scheduled in the system in many parallel instances and they causing system overload. Purpose of the report is to correct some entries in tables.

Final message – installation is successfully over!

And very last message:

In closing words I’d like to say following.

Reason why SAP moved SWPM to be pure web app is not very clear to me. There are normally issues like: timing out of web page when the installation progress is not refreshed, crashing of web browser, consuming too much memory by the web browser that can’t be used for installation etc. Probably to have it web based is helping SAP to deploy cloud based solutions however to have a thick client would serve better for on-premise installs.

Necessity of performing declustering / decoupling of DB tables. This operation is now mandatory as of NetWeaver 7.5. I understand that this is needed in case of HANA DB. By I do not see that much benefit of it in case of other DBs. I’d prefer to have this optional and do not enforce it by the installation. The operation itself very time consuming and overall installation time ballooned just because of that.

Hard drive space is needed for MSSQL based IDES system to be allocated at least for 350GB. Even better is to have approximately 380GB. Otherwise the installation will crash and in many cases you need to start over with it.

This is also one of my last installation of IDES systems. As SAP is moving everything to could and to HANA DB there won’t be a lot of chances to do this again in future. This is mostly by decreasing demand. It is much easier to deploy it in cloud. Also IDES on HANA DB is much difficult to setup as HANA needs to be installed first and that is also not an easy task to complete.