little tips and tricks, which i stumbled upon randomly

Thursday, November 12, 2015

How to import/export workbooks in Datameer

To export workbook
 curl -u <user>:<pass> -X GET http://<datameer_server>:8080/rest/workbook/<workbookid> > <fil_name>.json

To import workbook:
 curl -u <user>:<pass> -X POST -d @<file_name>.json 'http://<datameer_server>:8080/rest/workbook/'


  • It is better to remove "uuid" lien to give new uuid to workbook.

Friday, August 7, 2015

How to find which process is holding file cache (disk space) in aix

fuser -dV <path>

Tuesday, August 4, 2015

How to move data out of Informatica File Archive to Hadoop using sqoop

Although it seem hard and complicated. Once you discovered correct connection string and driver name getting data out of Informatica file archive with sqoop is pretty simple and straightforward.

Unfortunately due to structure there is no way to specify schema name other than getting data out with --query command


sqoop import --driver com.informatica.fas.jdbc.Driver --connect jdbc:infafas://<server_name>:<port: Default=8500>/<database_name> --username xxxx --password xxxxx -m 1 -delete-target-dir --target-dir <target_dir> --query "SELECT * FROM <schema_name>.<table_name> where \$CONDITIONS" --fields-terminated-by \| --lines-terminated-by \\n  --hive-drop-import-delims

You have to keep "where \$CONDITIONS" even though you do not specify one.

You also need to copy "infafas.jar" to shared library.

Please feel free to ask any questions.

Monday, July 27, 2015

Pig: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader - [Solved] - CDH 5.4.X+

Hi All;

With new CDH versions class name for HCatalog loader changed. Unfortunately this is not documented clearly.

Just change
org.apache.hcatalog.pig.HCatLoader to org.apache.hive.hcatalog.pig.HCatLoader
in your pig scripts to solve problem.

I hope, you wont waste too much time like me.

PS: You still need to specify hive-site.xml

Monday, July 13, 2015

Datameer upgrade guide - Step by step instructions

Here is a list of instructions to upgrade Datameer. 
It is basically unzip the new Datameer version into a new folder and copy over several files to the new version. Then you run a script that upgrades HSQLDB. 

Step 0: Save all Datameer setting (Administration --> Hadoop Cluster 
Save all configurations, connection strings and especially the YARN settings. 

Step 1: Stop Datameer
/<datameer installation path>/<Datameer Application Folder>/bin/conductor.sh stop

Step 2: Backup Datameer directory just in case

Step 3: Unzip new Datameer version
/<datameer installation path>/<New Datameer Application Folder>

Step 4: Give groups and user rights to new datameer folder similar to old one

Step 5: Adjust Xmx memory settings in das-env.sh to match old settings
/<datameer installation path>/<New Datameer Application Folder>/etc/das-env.sh

Step 6: Copy the files from the das-data folder of the old distribution to the new location.
cp -r /<old-location>/das-data /<new-location>/


Step 7: Copy over the native libraries that you have added to Datameer (if this applies)
cp -r /<old-location>/lib/native/* /<new-location>/lib/native/

Step 8: Copy over files from etc/custom-jars (these files could be database drivers or 3rd party libraries).
cp -r <old-location>/etc/custom-jars /<new-location>/

Step 9: Copy Datameer Plugins
cp -r <old-location>/etc/custom-pluginss /<new-location>/

Step 10: Upgrade HSQLDB:
<new location>/bin/upgrade_hsql_db.sh --old-das-data /<old-location>/das-data --new-das-data /<new-location>/das-data

For official instructions you can also visit 

Monday, March 2, 2015

Solution to Failed to execute command Create Sqoop Database on service Sqoop 2 error

Depending on java configuration Sqoop2 database creation may fail to update derby drivers and may get "Failed to execute command Create Sqoop Database on service Sqoop 2" error.

To solve this apply following steps:

1. Download derby derby client from db-derby-10.11.1.1-bin.zip
2. Extract and copy derby.jar and derbyclient.jar from zip file to /var/lib/sqoop2
3. Copy derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/ as well
4. Delete /opt/cloudera/parcels/CDH-<version>/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby​-<version>.jar soft link.
5. Make /opt/cloudera/parcels/CDH-<version>/l​ib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/derby.jar


Wednesday, December 17, 2014

UC4 Automic "Automatically deactivate when finished"

You can use following SQL to query "Automatically deactivate when finished" status from UC4 repository.



select OH_IDNR, OH_NAME,
CASE 
    WHEN JBA_AUTODEACT = 'A' THEN 'Always'
    WHEN JBA_AUTODEACT = 'J' THEN 'After error-free execution'
    WHEN JBA_AUTODEACT = 'N' THEN 'No'
    WHEN JBA_AUTODEACT = 'F' THEN 'After an error-free restart'
end as "Automatically deactivate"  
from uc4.oh t1,  uc4.JBA  t2
where OH_IDNR = JBA_OH_IDNR