Thursday, November 12, 2015
How to import/export workbooks in Datameer
To export workbook
curl -u <user>:<pass> -X GET http://<datameer_server>:8080/rest/workbook/<workbookid> > <fil_name>.json
To import workbook:
curl -u <user>:<pass> -X POST -d @<file_name>.json 'http://<datameer_server>:8080/rest/workbook/'
curl -u <user>:<pass> -X GET http://<datameer_server>:8080/rest/workbook/<workbookid> > <fil_name>.json
To import workbook:
curl -u <user>:<pass> -X POST -d @<file_name>.json 'http://<datameer_server>:8080/rest/workbook/'
- It is better to remove "uuid" lien to give new uuid to workbook.
Friday, August 7, 2015
Tuesday, August 4, 2015
How to move data out of Informatica File Archive to Hadoop using sqoop
Although it seem hard and complicated. Once you discovered correct connection string and driver name getting data out of Informatica file archive with sqoop is pretty simple and straightforward.
Unfortunately due to structure there is no way to specify schema name other than getting data out with --query command
sqoop import --driver com.informatica.fas.jdbc.Driver --connect jdbc:infafas://<server_name>:<port: Default=8500>/<database_name> --username xxxx --password xxxxx -m 1 -delete-target-dir --target-dir <target_dir> --query "SELECT * FROM <schema_name>.<table_name> where \$CONDITIONS" --fields-terminated-by \| --lines-terminated-by \\n --hive-drop-import-delims
You have to keep "where \$CONDITIONS" even though you do not specify one.
You also need to copy "infafas.jar" to shared library.
Please feel free to ask any questions.
Unfortunately due to structure there is no way to specify schema name other than getting data out with --query command
sqoop import --driver com.informatica.fas.jdbc.Driver --connect jdbc:infafas://<server_name>:<port: Default=8500>/<database_name> --username xxxx --password xxxxx -m 1 -delete-target-dir --target-dir <target_dir> --query "SELECT * FROM <schema_name>.<table_name> where \$CONDITIONS" --fields-terminated-by \| --lines-terminated-by \\n --hive-drop-import-delims
You have to keep "where \$CONDITIONS" even though you do not specify one.
You also need to copy "infafas.jar" to shared library.
Please feel free to ask any questions.
Monday, July 27, 2015
Pig: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader - [Solved] - CDH 5.4.X+
Hi All;
With new CDH versions class name for HCatalog loader changed. Unfortunately this is not documented clearly.
Just change
org.apache.hcatalog.pig.HCatLoader to org.apache.hive.hcatalog.pig.HCatLoader
in your pig scripts to solve problem.
I hope, you wont waste too much time like me.
PS: You still need to specify hive-site.xml
With new CDH versions class name for HCatalog loader changed. Unfortunately this is not documented clearly.
Just change
org.apache.hcatalog.pig.HCatLoader to org.apache.hive.hcatalog.pig.HCatLoader
in your pig scripts to solve problem.
I hope, you wont waste too much time like me.
PS: You still need to specify hive-site.xml
Monday, July 13, 2015
Datameer upgrade guide - Step by step instructions
Here is a list of instructions to upgrade Datameer.
It is basically unzip the new
Datameer version into a new folder and copy over several files to the new
version. Then you run a script that upgrades HSQLDB.
Step 0: Save all Datameer setting (Administration -->
Hadoop Cluster
Save all configurations, connection strings and especially
the YARN settings.
Step 1: Stop Datameer
/<datameer installation path>/<Datameer Application
Folder>/bin/conductor.sh stop
Step 2: Backup Datameer directory just in case
Step 3: Unzip new Datameer version
/<datameer installation path>/<New Datameer Application Folder>
Step 4: Give groups and user rights to new datameer
folder similar to old one
Step 5: Adjust Xmx memory settings in das-env.sh to match
old settings
Step 6: Copy the files from the das-data folder of the
old distribution to the new location.
cp -r /<old-location>/das-data /<new-location>/
Step 7: Copy over the native libraries that you have added to Datameer (if this applies)
cp -r /<old-location>/lib/native/* /<new-location>/lib/native/
Step 8: Copy over files from etc/custom-jars (these files could be database drivers or 3rd party libraries).
cp -r <old-location>/etc/custom-jars /<new-location>/
Step 9: Copy Datameer Plugins
cp -r <old-location>/etc/custom-pluginss /<new-location>/
Step 10: Upgrade HSQLDB:
<new location>/bin/upgrade_hsql_db.sh --old-das-data /<old-location>/das-data --new-das-data /<new-location>/das-data
For official instructions you can also visit
Monday, March 2, 2015
Solution to Failed to execute command Create Sqoop Database on service Sqoop 2 error
Depending on java configuration Sqoop2 database creation may fail to update derby drivers and may get "Failed to execute command Create Sqoop Database on service Sqoop 2" error.
To solve this apply following steps:
To solve this apply following steps:
1. Download derby derby client from db-derby-10.11.1.1-bin.zip
2. Extract and copy derby.jar and derbyclient.jar from zip file to /var/lib/sqoop2
3. Copy derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/ as well
4. Delete /opt/cloudera/parcels/CDH-<version>/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby-<version>.jar soft link.
5. Make /opt/cloudera/parcels/CDH-<version>/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/derby.jar
Wednesday, December 17, 2014
UC4 Automic "Automatically deactivate when finished"
You can use following SQL to query "Automatically deactivate when finished" status from UC4 repository.
select OH_IDNR, OH_NAME,
CASE
WHEN JBA_AUTODEACT = 'A' THEN 'Always'
WHEN JBA_AUTODEACT = 'J' THEN 'After error-free execution'
WHEN JBA_AUTODEACT = 'N' THEN 'No'
WHEN JBA_AUTODEACT = 'F' THEN 'After an error-free restart'
end as "Automatically deactivate"
from uc4.oh t1, uc4.JBA t2
where OH_IDNR = JBA_OH_IDNR
select OH_IDNR, OH_NAME,
CASE
WHEN JBA_AUTODEACT = 'A' THEN 'Always'
WHEN JBA_AUTODEACT = 'J' THEN 'After error-free execution'
WHEN JBA_AUTODEACT = 'N' THEN 'No'
WHEN JBA_AUTODEACT = 'F' THEN 'After an error-free restart'
end as "Automatically deactivate"
from uc4.oh t1, uc4.JBA t2
where OH_IDNR = JBA_OH_IDNR
Saturday, December 6, 2014
UC4 status messages - AH_STATUS
Unfortunately UC4 (Automic) software does not have any lookup table for ah_status values.
These are the ah_status messages I stumbled upon among years.
AH_STATUS Status Text
when t1.eh_status = 1300 then 'Preparing '
when t1.eh_status = 1510 then 'Transferred '
when t1.eh_status = 1520 then 'Ready for transfer (before
Connect) '
when t1.eh_status = 1521 then 'Ready to be transferred '
when t1.eh_status = 1529 then 'Ready for generation '
when t1.eh_status = 1530 then 'Ready for start '
when t1.eh_status = 1531 then 'To be called '
when t1.eh_status = 1540 then 'Start initiated '
when t1.eh_status = 1541 then 'Checking '
when t1.eh_status = 1542 then 'Calling '
when t1.eh_status = 1543 then 'Unknown '
when t1.eh_status = 1544 then 'Inconsistent '
when t1.eh_status = 1545 then 'Started '
when t1.eh_status = 1546 then 'Connecting '
when t1.eh_status = 1550 then 'Active '
when t1.eh_status = 1551 then 'Transferring '
when t1.eh_status = 1552 then 'Called '
when t1.eh_status = 1553 then 'Accepted '
when t1.eh_status = 1554 then 'Sampling Files '
when t1.eh_status = 1556 then 'Escalated '
when t1.eh_status = 1560 then 'ProcessFlow is blocked. '
when t1.eh_status = 1561 then 'STOP - Client-wide stop of
automatic processing. '
when t1.eh_status = 1562 then 'HELD - Manual stop has been set.
'
when t1.eh_status = 1563 then 'STOP - Automatic processing has
been stopped. '
when t1.eh_status = 1564 then 'STOP - Queue processing has been
stopped. '
when t1.eh_status = 1570 then 'Skipping '
when t1.eh_status = 1571 then 'Canceling '
when t1.eh_status = 1572 then 'Generating '
when t1.eh_status = 1573 then 'Generated '
when t1.eh_status = 1574 then 'Post processing '
when t1.eh_status = 1575 then 'Ending '
when t1.eh_status = 1600 then 'Internal checking '
when t1.eh_status = 1684 then 'Waiting for Queue slot (max.
parallel tasks exceeded) '
when t1.eh_status = 1688 then 'Waiting for the host of an
AgentGroup. '
when t1.eh_status = 1689 then 'Waiting for resource (max.
FileTransfers exceeded) '
when t1.eh_status = 1690 then 'Waiting for external precondition
'
when t1.eh_status = 1693 then 'Waiting for manual release '
when t1.eh_status = 1694 then 'Waiting for resource (max. Jobs
exceeded) '
when t1.eh_status = 1695 then 'Waiting for restart time '
when t1.eh_status = 1696 then 'Waiting for host '
when t1.eh_status = 1697 then 'Waiting for SYNC '
when t1.eh_status = 1698 then 'Waiting for start time '
when t1.eh_status = 1700 then 'Waiting for predecessor '
when t1.eh_status = 1701 then 'Sleeping '
when t1.eh_status = 1702 then 'Not yet called '
when t1.eh_status = 1703 then 'Waiting for Pre-Conditions '
when t1.eh_status = 1709 then 'Waiting for end of parallel task
'
when t1.eh_status = 1710 then 'Registered '
when t1.eh_status = 1800 then 'ENDED_NOT_OK - Aborted '
when t1.eh_status = 1801 then 'ENDED_NOT_OK - Aborted because of
SYNC condition '
when t1.eh_status = 1802 then 'ENDED_JP_ABEND - Not executed
because of abnormal ProcessFlow end. '
when t1.eh_status = 1810 then 'ENDED_VANISHED - Disappeared '
when t1.eh_status = 1815 then 'ENDED_LOST - Ended undefined
(host terminated prematurely) '
when t1.eh_status = 1820 then 'FAULT_OTHER - Start impossible.
Other error. '
when t1.eh_status = 1821 then 'FAULT_NO_HOST - Start impossible.
Cannot reach host. '
when t1.eh_status = 1822 then 'FAULT_ALREADY_RUNNING - Task is
already running '
when t1.eh_status = 1823 then 'FAULT_POST_PROCESSING - Error in
post processing '
when t1.eh_status = 1824 then 'FAULT_POST_CONDITION - Error in
Post-Condition. '
when t1.eh_status = 1850 then 'ENDED_CANCEL - Manually canceled.
'
when t1.eh_status = 1851 then 'ENDED_JP_CANCEL - ProcessFlow
canceled manually. '
when t1.eh_status = 1852 then 'Rejected '
when t1.eh_status = 1856 then 'ENDED_ESCALATED - Aborted due to
escalation '
when t1.eh_status = 1898 then 'Blocked '
when t1.eh_status = 1899 then 'Status blocked manually removed.
'
when t1.eh_status = 1900 then 'ENDED_OK - Ended normally '
when t1.eh_status = 1901 then 'Confirmed '
when t1.eh_status = 1902 then 'ENDED_QUEUE_END (status for
subordinate tasks of a SAP job that has been ended) '
when t1.eh_status = 1903 then 'ENDED_CONTAINER_END (status for
tasks of an AgentGroup that has been ended) '
when t1.eh_status = 1910 then 'ENDED_EMPTY - Task is empty (STOP
NOMSG) '
when t1.eh_status = 1911 then 'ENDED_TRUNCATE - Transfer
incomplete because of the line limit. '
when t1.eh_status = 1912 then 'ENDED_EMPTY - Nothing found '
when t1.eh_status = 1920 then 'ENDED_INACTIVE - Inactive today
because of the Calendar. '
when t1.eh_status = 1921 then 'ENDED_INACTIVE - Task is not
active because of the definition. '
when t1.eh_status = 1922 then 'ENDED_INACTIVE - Task has
manually been set inactive. '
when t1.eh_status = 1930 then 'ENDED_SKIPPED - Skipped because
of WHEN clause. '
when t1.eh_status = 1931 then 'ENDED_SKIPPED - Skipped because
of SYNC condition. '
when t1.eh_status = 1932 then 'ENDED_SKIPPED - Schedule ended
prematurely. '
when t1.eh_status = 1940 then 'ENDED_TIMEOUT - Not executed
because of timeout (WHEN clause). '
when t1.eh_status = 1941 then 'ENDED_TIMEOUT - Start time
exceeded. '
when t1.eh_status = 1942 then 'ENDED_TIMEOUT - Ended untimely. '
when t1.eh_status = 1999 then 'SYNC check passed. '
