Last year, April -> October, I asked the question about IQ supporting Huge Pages on Linux. It was mentioned that under SA CR 728597 and Red Hat Bug 891857 that there was a bug in the Linux kernel handling of direct I/O while using transparent huge memory pages (a variant of Linux Huge memory pages).

CR 728597:
This problem is related to a possible bug in the transparent huge pages (THP) feature introduced in these operating system versions. Red Hat bug 891857 has been created to track this issue.

The problem can be triggered by calling an external environment, xp_cmdshell, or other procedure that causes a fork while other I/O is occurring. A known limitation with the Linux kernel limits the use of fork while doing O_DIRECT I/O operations. Essentially what can happen is that the data can come from or go to the wrong process’ memory after the fork. SQL Anywhere performs O_DIRECT I/O operations according to the documented safe usage. However, THP appears to cause further problems and the O_DIRECT I/O data comprising database page reads/writes appears to get lost.

http://scn.sap.com/thread/3338917 and http://froebe.net/blog/2013/06/17/does-anyone-have-any-details-on-redhat-linux-bug-891857/

Does anyone know the status of this ongoing FIVE year old issue?

http://scn.sap.com/thread/3505418

If you’re coming from Microsoft SQL Server or Sybase ASE, SAP’s IQ doesn’t have the slew of dbcc commands you might expect. It rolls up all the pertinent dbccs into a single stored procedure. sp_iqcheckdb has four operational modes with a granularity at the database, table or index level:

1. In check mode, sp_iqcheckdb performs an internal consistency check on all IQ indexes and checks that each database block has been allocated correctly. All available database statistics are reported. This mode reads all data pages and can detect all types of allocation problems and most types of index inconsistencies. Check mode should run considerably faster than verify mode for most databases.
2. In verify mode, sp_iqcheckdb performs an intra-index consistency check, in addition to internal index consistency and allocation checking. All available database statistics are reported. The contents of each non-FP index is verified against its corresponding FP index(es). Verify mode reads all data pages and can detect all types of allocation problems and all types of index inconsistencies.
3. In allocation mode, sp_iqcheckdb checks that each database block is allocated correctly according to the internal physical page mapping structures (blockmaps). Database statistics pertaining to allocation are also reported. This mode executes very quickly. Allocation mode, however, does not check index consistency and cannot detect all types of allocation problems.
4. When the Sybase IQ server runs in single-node mode, you can use dropleaks mode with either a database or dbspace target to reset the allocation map for the entire database or specified dbspace targets. If the target is a dbspace, then the dropleaks operation must also prevent read-write operations on the named dbspace. All dbspaces in the database or dbspace list must be online.

No changes are actually performed with the exception of the dropleaks mode. If check or verify detects an issue with allocation, you will need to run sp_iqcheckdb in the dropleaks mode while the database is in single user mode. For any issues with indexes, use sp_rebuildindex on both the index and possibly the affected column of the table.

By default output of sp_iqcheckdb will go to the (instance_name).iqmsg log file but you can redirect it to its own file for easier reading:

set temporary option dbcc_log_progress='on';
sp_iqcheckdb 'verify database' ># filename

Please note that the file name may not contain spaces or special symbols and the path will be local to the IQ box. So, unless you want to hunt for your checkdb output file, please specify the full path. No, you can not put the filename in any type of quotes and you can’t escape interesting characters.

Unless you’re under a serious time constraint, always run sp_iqcheckdb in verify mode to determine structural issues with your IQ databases.

ASE 15.7: Create a Remote Server to SAP IQ 16.0
March 12, 2014

Create the remote server entry in the interfaces file; sql.ini (Windows) or interfaces (UNIX).

[myIQServer]
master=TCP,myIQHost,4091
query=TCP,myIQHost,4091

Read more at ARBIME

Many PDF readers for smart phones (Android/iphone) and tablets manage the pdf files based solely on the Title and Author fields in the PDF file. While for this is fine for your average book, it is not all that helpful with manuals that tend to have abbreviated or no data in the title/author fields. In the case of the manuals for Sybase IQ, I’m unable to load the manuals for say v15.4 and v16.0 as they have the same Title/Author data.

How to fix? Easy. Go get Calibre. Drop the PDF files on to the running Calibre. Edit them by hitting the E key.

In my case, I edited the “Title”, “Author”, “Tags”, “Publisher” and “Languages”:

Calibre doesn’t modify the PDF files themselves so I will need to export the files to a custom directory. In Calibre nomenclature, this is “Saving”. Highlight all the titles you want to export and hit “S” twice. Why twice? No idea. Choose the directory.

You can now copy the exported PDF files to your phone, tablet, whatever without fear of the v16.0 version of the P&T Guide being rejected by Aldiko because the v15.4 version is already added.

Here are the IQ v16.0 SP3 manuals that I’ve ‘fixed’ to work with Aldiko. They are identical to the PDFs on sybooks with the exception of the PDF fields I mentioned previously.

No copyright infringement is intended. SAP/Sybase, please feel free to take these and host them.

IQ 16.0 SP3 – Programming – SAP, Inc_
IQ 16.0 SP3 – Quick Start_ Unix_Linux – SAP, Inc_
IQ 16.0 SP3 – Reference_ Building Blocks, Tables, and Procedures – SAP, Inc_
IQ 16.0 SP3 – Reference_ Statements and Options – SAP, Inc_
IQ 16.0 SP3 – Release Bulletin_ Linux – SAP, Inc_
IQ 16.0 SP3 – SAP Sybase IQ Error Messages – SAP, Inc_
IQ 16.0 SP3 – Unstructured Data Analytics – SAP, Inc_ IQ 16.0 SP3 – User-Defined Functions – SAP, Inc_
IQ 16.0 SP3 – Utility Guide – SAP, Inc_
IQ 16.0 SP3 – Administration_ Backup, Restore, and Data Recovery – SAP, Inc_
IQ 16.0 SP3 – Administration_ Globalization – SAP, Inc_
IQ 16.0 SP3 – Administration_ In-Memory Row-Level Versioning – SAP, Inc_
IQ 16.0 SP3 – Administration_ Load Management – SAP, Inc_
IQ 16.0 SP3 – Administration_ Multiplex – SAP, Inc_
IQ 16.0 SP3 – Administration_ Spatial Data – SAP, Inc_
IQ 16.0 SP3 – Administration_ User Management and Security – SAP, Inc_
IQ 16.0 SP3 – Guide to Licensed Options – SAP, Inc_
IQ 16.0 SP3 – Introduction to SAP Sybase IQ – SAP, Inc_
IQ 16.0 SP3 – Linux Installation and Configuration Guide – SAP, Inc_
IQ 16.0 SP3 – Migration_ Unix_Linux – SAP, Inc_
IQ 16.0 SP3 – New Features Summary – SAP, Inc_
IQ 16.0 SP3 – Performance and Tuning Guide – SAP, Inc_
SAP Control Center 3.3 – Release Bulletin – SAP, Inc_
SAP Control Center for SAP Sybase IQ 3.3 – SAP, Inc_
Unified Agent and Agent Management Console 2.0.2 – User’s Guide – SAP, Inc_

SAP/Sybase’s documentation isn’t very clear for new IQ dbas and developers. Once such item is simply restoring an IQ database on to another system. Unlike ASE, you need to specify the new file locations if they are different than the source server.

Assumptions:

1. IQ software has been installed
2. The new dbfile locations are symbolic links to raw partitions OR the path exists but not the files
3. You have a valid SYSAM license for the new IQ instance.
4. The new IQ instance name is set (via -n instance)
5. The old directory for the .db, .log and .mir exists (use a symbolic link if you wish)

Obtain dbspace file names with sp_iqfile:

select DBFileName, Path, DBFileSize from sp_iqfile();

DBFileName	Path	DBFileSize
'IQ_SYSTEM_MAIN'	'/dba/syb/old_iq/sybdev/IQ_MAIN/old_iqmain001.iq'	'32G'
'IQ_USER_MAIN_FILE_01'	'/dba/syb/old_iq/sybdev/IQ_USER_MAIN/old_iqusermain001.iq'	'1024G'
'IQ_SYSTEM_TEMP'	'/dba/syb/old_iq/sybdev/IQ_TEMP/old_iqtemp001.iqtmp'	'32G'
'IQ_SYSTEM_TEMP_002'	'/dba/syb/old_iq/sybdev/IQ_TEMP/old_iqtemp002.iqtmp'	'32G'

Create a restore.sql file renaming the DBFileName to the new locations:

restore database 'new_iq'
FROM '/dba/backup/sybbackup/old_iq.20140423100111.17760.IQfullbkp'
RENAME IQ_SYSTEM_MAIN TO '/dba/syb/new_iq/sybdev/IQ_MAIN/new_iqmain001.iq'
RENAME IQ_SYSTEM_TEMP TO '/dba/syb/new_iq/sybdev/IQ_TEMP/new_iqtemp001.iq'
RENAME IQ_SYSTEM_TEMP_002 TO '/dba/backup/sybbackup/new_iqtemp002.iq'
RENAME IQ_SYSTEM_MSG TO '/dba/syb/new_iq/instlog/new_iq.iqmsg'
RENAME IQ_USER_MAIN_FILE_01 TO '/dba/syb/new_iq/sybdev/IQ_USER_MAIN/new_iqusermain001.iq';

Stop the destination IQ instance if it is running and start the utility database:

stop_iq

Checking system ...

The following 1 server(s) are owned by 'sybdba'

## Owner          PID   Started  CPU Time  Additional Information
-- ---------  -------  --------  --------  ------------------------------------
1: sybdba       13909     Apr24  00:43:46  SVR:new_iq DB:new_iq PORT:58116
              /dba/syb/new_iq/sybase/IQ-16_0/bin64/iqsrv16 @/dba/syb/new_iq/sybdb/new_iq.cfg /dba/syb/new_iq/sybdb/new_iq.db -gn 65 -o /dba/syb/new_iq/sybase/IQ-16_0/logfiles/

${SYBASE}/IQ-16_0/bin64/start_iq -n utility_db -gu dba -c 48m -gc 20 -iqgovern 30 \
        -gd all -gl all -gm 10 -gp 4096 -ti 4400 -z -zr all -zo $SYBASE/IQ-16_0/logfiles/utility_db.out \
        -o $SYBASE/IQ-16_0/logfiles/utility_db.srvlog -iqmc 100 -iqtc 100 -x "tcpip{port=9000}"

Starting server utility_db on localhost at port 9000 (04/30 09:37:16)

Run Directory       : /dba/syb/new_iq/sybdb
Server Executable   : /dba/syb/new_iq/sybase/IQ-16_0/bin64/iqsrv16
Server Output Log   : /dba/syb/new_iq/instlog/utility_db.srvlog
Server Version      : 16.0.0.653/sp03 16.0.0/Linux 2.6.18-194.el5
Open Client Version : N/A
User Parameters     : '-n' 'utility_db' '-gu' 'dba' '-c' '48m' '-gc' '20' '-iqgovern' '30' '-gd' 'all' '-gl' 'all' '-gm' '10' '-gp' '4096' '-ti' '4400' '-z' '-zr' 'all' '-zo' '/dba/syb/new_iq/instlog/utility_db.out' '-o' '/dba/syb/new_iq/instlog/utility_db.srvlog' '-iqmc' '100' '-iqtc' '100' '-x' 'tcpip{port=9000}'
Default Parameters  : -gn 25
….

Remove the db, log and mir files:

rm instance.db instance.log instance.mir

Restore the full backup:

dbisql -c "uid=dba;pwd=sql;eng=utility_db;dbn=utility_db" -port 9000 -host $( hostname ) -nogui "restore.sql"

Restore the incremental backup(s):

dbisql -c "uid=dba;pwd=sql;eng=utility_db;dbn=utility_db" -port 9000 -host $( hostname ) -nogui "restore_incrementals.sql"

Stop the utility database:

stop_iq

Start the IQ server to ensure it comes up then shut it back down.

If the name of the server has changed (e.g. old_iq -> new_iq), then we need to update the log and mir files. First let’s find out where the log and mir files are currently set to in the db file:

dblog new_iq.db

SQL Anywhere Transaction Log Utility Version 16.0.0.653
"new_iq.db" is using log file "/dba/syb/old_iq/sybdb/old_iq.log"
"new_iq.db" is using log mirror file "/dba/syb/old_iq/sybdb/old_iq.mir"
Transaction log starting offset is 0702994164
Transaction log current relative offset is 0000397583

Set the log file to “new_iq.log”:

dblog -t new_iq.log new_iq.db

SQL Anywhere Transaction Log Utility Version 16.0.0.653
"new_iq.db" was using log file "/dba/syb/old_iq/sybdb/old_iq.log"
"new_iq.db" is using log mirror file "/dba/syb/old_iq/sybdb/old_iq.mir"
"new_iq.db" is now using log file "new_iq.log"
Transaction log starting offset is 0702994164
Transaction log current relative offset is 0000397625

We need to clear the mir file(s) before we can assign a new one:

dblog -r new_iq.db

SQL Anywhere Transaction Log Utility Version 16.0.0.653
"new_iq.db" is using log file "new_iq.log"
"new_iq.db" was using log mirror file "/dba/syb/old_iq/sybdb/im00.mir"
"new_iq.db" is now using no log mirror file
Transaction log starting offset is 0702994164
Transaction log current relative offset is 0000397625

Set the mir file:

dblog -m new_iq.mir new_iq.db

SQL Anywhere Transaction Log Utility Version 16.0.0.653
"new_iq.db" is using log file "new_iq.log"
"new_iq.db" was using no log mirror file
"new_iq.db" is now using log mirror file "new_iq.mir"
Transaction log starting offset is 0702994164
Transaction log current relative offset is 0000397625

Start your IQ instance.

SAP/Sybase’s documentation isn’t very clear for new IQ dbas and developers. Once such item is simply extracting data from the warehouse into files that can be loaded into another IQ system when load table from location simply isn’t available or isn’t practical.

Assumptions:

1. Extraction of table data with the owner of ‘JF’. Replace with the schemas you desire.
2. Exported data will go to the /dba/backup/sybbackup/bkp/export/data_export directory.
3. We will use isql with the parameters to connect stored in the $ISQL environment variable.
4. The max temporary table size is 1TB. Increase as needed.

The simplest method is to simply create a shell script per table to export:

#!/bin/env bash

if [[ ! -d data_export ]]; then
  mkdir data_export
fi

set -A TABLES

TABLES=$( echo "set nocount on
go
select convert(varchar(80), user_name(creator) + '.' + table_name) from systable where user_name(creator) in ('JF') and table_type = 'BASE'
go
exit" | $ISQL -h -b  | sed -e '/affected/d' -e '/---/d' -e '/^$/d' )

for table in ${TABLES}; do
  echo
  echo "Generating /dba/backup/sybbackup/bkp/export/data_export/${table}_export.sql"
  sql_file=/dba/backup/sybbackup/bkp/export/data_export/${table}_export.sql

  echo "set temporary option Temp_Extract_Name1 = '/dba/backup/sybbackup/bkp/export/data_export/${table}.data'
  set temporary option Temp_Extract_Name2 = ''
  set temporary option Temp_Extract_Binary = 'on'
  set temporary option Temp_Extract_Swap = 'off'
    set temporary option TEMP_EXTRACT_SIZE1 = 1073741824
go

  select * from ${table}
go

  set temporary option Temp_Extract_name1 = ''
go" > $sql_file
done

Now that we have the script files, let’s extract the data by running the shell scripts and compress each exported file with the gzip program. You can cancel the export at any time with ctrl-c and restart it after the last exported table:

for TABLE in *.sql; do 
  datafile=$( echo $TABLE | sed -e 's/_export.sql$/.data/' )
  echo $datafile
  gzipfile=${datafile}.gz

  if [ -f $gzipfile ]; then
    echo "$gzipfile already exists"
  else
    $ISQL -i $TABLE 2>&1 > $TABLE.out
    gzip -1 $datafile
  fi
done

Now that we have that data exported, imagine that you copied the files to another system. How do you import that data assuming that the tables have already been completed? Easy. We will create a set of import script files.

LOCAL_PATH=/dba/backup/sybbackup/bkp/import/data_import

for TABLE_gzip in *.gz; do 
  datafile=$( echo $TABLE_gzip | sed -e 's/.gz$//' )
  TABLE_FILE=$( echo $TABLE_gzip | sed -e 's/.data.gz$//' )
  TABLE_OWNER=$( echo $TABLE_FILE | cut -d . -f1 )
  TABLE=$( echo $TABLE_FILE | cut -d . -f2 | sed -e 's/_export$//' )

  if [ -f ${datafile}.done ]; then
    echo "${datafile} already processed"
  else
    # ===================================================================
    # Generate the load commend to load the file
    # ===================================================================
    echo "#!/bin/env bash" > ${TABLE_OWNER}.${TABLE}_import.sh
    echo ". /dba/code/syb/.setenv" >> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "" >> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "echo \"uncompressing $TABLE_gzip\"" >> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "gzip -dc $TABLE_gzip > $datafile" >> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "echo \"importing ${TABLE_OWNER}.${TABLE}\"" >> ${TABLE_OWNER}.${TABLE}_import.sh
    echo '$ISQL -b < <EOF'>> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "select 'Start datetime'=convert(char(25), getdate(), 119), 'TABLENAME=${TABLE_OWNER}.${TABLE}'">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo " ">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "truncate table ${TABLE_OWNER}.${TABLE}">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "commit">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "SET TEMPORARY OPTION IDENTITY_INSERT = '${TABLE_OWNER}.${TABLE}'">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "load table ${TABLE_OWNER}.${TABLE}">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "( ">> ${TABLE_OWNER}.${TABLE}_import.sh
    ../gen_iq_col_list_w_null_byte.sh ip00 $TABLE_OWNER $TABLE | sed -e '/row affected/d;s/ *$//;/^$/d'>> ${TABLE_OWNER}.${TABLE}_import.sh
    echo ")">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "from '${LOCAL_PATH}/${datafile}'">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "escapes off format binary">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "commit">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo " ">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "select 'Start datetime'=convert(char(25), getdate(), 119), 'TABLENAME=${TABLE_OWNER}.${TABLE}'">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "SET TEMPORARY OPTION IDENTITY_INSERT = ''">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "go">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo " ">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "EOF">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "" >> ${TABLE_OWNER}.${TABLE}_import.sh
    echo "rm -f $datafile">> ${TABLE_OWNER}.${TABLE}_import.sh
    echo " ">> ${TABLE_OWNER}.${TABLE}_import.sh
    chmod u+x ${TABLE_OWNER}.${TABLE}_import.sh
  fi
done

If the target system is a different endian (Linux x86-64 -> AIX), replace

echo "escapes off format binary">> ${TABLE_OWNER}.${TABLE}_import.sh

with

echo "escapes off format binary byte order high">> ${TABLE_OWNER}.${TABLE}_import.sh

We simply need to run each import script file:

for import_file in *_import.ksh ; do ./$import_file 2>&1 |tee  ${j}.out ; done

Thanks to Joseph Weaver for supplying this workaround!
In Embarcadero’s Rapid SQL accessing data within SAP IQ can sometimes result in a “SQL Anywhere Error -203: Cannot set a temporary option for user ‘XXXXXX’” error.  This is not an IQ issue.  

The Rapid SQL configuration needs to be updated by disabling the “Enable SET Query Options” by default:

There are times when you need individual users the ability to determine what indexes are suggested by the index advisor but you don’t want or are unable to give them full dba access. All you need to do is grant the execute permission on the sp_iqindexadvice stored procedure.

Grant execute on sp_iqindexadvice to <user>;

create table test (col1 int, col2 varchar(10));
insert into test values (1, 'test1');
insert into test values (2, 'test2');
insert into test values (3, 'test3');
insert into test values (4, 'test4');
commit;

SET OPTION index_advisor = 'ON';
SET OPTION index_advisor_max_rows = 100;
commit;

select * from test where col1 =1 ;
commit;

call sp_iqindexadvice ();</user>

In the MSA and System transactions performedby maint user thread on the sybase-product-futures mailing list, Jeff Tallman mentions that someone needs to create an ISUG enhancement or a Sybase change request (CR).  Mike Harrold let everyone know that even through creating a Sybase CR is put in the Sybase engineering queue, it doesn’t mean that an engineer will actually look at it let alone have product management approve it to be implemented.

Jeff hits the nail on the head.  You have a voice with the ISUG enhancements process. Use it.  And yes, you have to be paid member to vote. Sorry, but we are a non-profit and your membership fees keep us running!  You can vote as a Basic Member – you do not need to join at the higher membership levels (but you should so you can read all the wonderful articles Jeff writes for us).  The more votes, the easier it is for engineering to justify spending staff resources on it when they go to management.  Encourage all of your peers to join and vote as well.  If you work for a business with several Sybase users, convince your boss to pay for a corporate membership and get all your colleagues to vote as well.

Regards,

Mike Harrold
Executive Director

and

Having an “internal enhancement” (aka a CR) doesn’t mean it gets implemented.  It means there’s a CR for it.  It might be a great idea, but without backing (and a user request + an engineer doesn’t mean it has backing; look how long it took for UDFs, mathematical functions, etc.) it doesn’t get allocated any development resources.  Without development resources, it doesn’t get implemented.

Bottom line, a CR doesn’t mean it’s a “planned” feature.

Regards,

Mike Harrold
Executive Director
International Sybase User Group

I would recommend voting for the ISUG enhancements (this week).  I and several other ISUG Board members will be going through the ISUG enhancements this weekend to present a subset to Sybase product management at TechWave next month.

I haven’t seen many ISUG enhancements being requested lately or voted upon for that matter.  The higher the votes for a particular enhancement, the more likely Sybase will pick it up.

So please, everyone go vote on the ISUG enhancements!

The ISUG Technical Journal Sep/Oct 2008 is out.

This issue contains:

• Sybase IQ and Sybase WorkSpace Part II By Mike Crocker
• Perl in the shell By Jason Froebe
• Taming the process nightmare: Consolidating a Horde of Process Diagrams with PowerDesigner By Mike Nicewarner
• Information overload: Weapons-grade data analysis comes of age By Teresa Foster
• Maximizing PowerBuilder and SQLAnywhere Part III SQLAnywhere, a PowerBuilder Developer’s perspective
  By Chris Pollach
• select random (stuff) from SYBASE Command shell by proxy By Bill Grant
• Mobile Enterprise Everywhere: Taking Sybase on the road By Adrian Bridgwater
• Sybase TechWave 2008: Time for the technology tsunami

You need to be an ISUG member to receive the ISUG Technical Journal.  Join ISUG today!

International Sybase User Group

While the CakePHP framework is very good, the dbo “driver” (wrapper) between CakePHP and the PHP Sybase/Microsoft drivers ar not implemented to be actually usable.  Most of the issues are differences of SQL dialect between MySQL and the commercial DBMSs.

The issues tend to be simple ones such as, “select * from my_table limit 10″ to retrieve the first 10 rows not working.  Using “select top 10 col1,col2,….coln from my_table” will.

So how can you help?  Very easily!   The CakePHP website shows us how:

The most intuitive way to write a comparison of two tables and spit out the # of rows that are the same involves a left join:

select 'Investment' as TableName, count(*) as RowCount
from Investment_A a, Investment_B b
where
  a.col1 = b.col1
  AND a.col2 = b.col2
  AND a.col3 = b.col3
  AND a.col4 = b.col4

This returns the correct answer but is very slow. Is there a better way? Of course!

select 'Investment' as TableName, count(*) as RowCount
from
(
 select 1 as num
 FROM (
  select * from Investment_A
  UNION ALL
  select * from Investment_B
 ) tmp
  GROUP BY
    col1, col2, col3, col4
  HAVING COUNT(*) > 1
) tmp2

By pushing the comparison off into the GROUP BY, we leverage the DBMS engine far more efficiently. There are two drawbacks:

1. Readability of the SQL code
2. Far more temporary storage is used for the GROUP BY. There is a real risk of running out of temporary storage if the tables are large.

SAP/Sybase released SAP Sybase IQ 16 Best Practices Guide for their data warehouse product IQ.

This document presents the best practices for IQ 16. It includes recommendations for using and configuring the most important features of IQ, and precautions for avoiding and troubleshooting some of its most common issues. It also highlights IQ 16’s most significant new features and describes the latest enhancements to existing features. This document is intended to serve as a starting point for optimizing your implementation of IQ 16. As a general purpose document, it is not intended to be a comprehensive guide for every environment. Rather, it is a set of guidelines, suggestions and observations on how to better use IQ 16 and its newest capabilities

The manuals for Sybase IQ 16 are also online now.

New features in IQ 16. Most of the changes appear to revolve around multiplex (1 writer and many reader IQ instances) environments with a few minor changes such as the removal of the APPEND LOAD.

If you’re performing updates in a single server environment, you may benefit from RLV (Row Level Versioned) Store:

New in IQ 16 is the RLV (Row Level Versioned) Store – This is an in-memory delta store that is optimized for high performance row-level updates. It is currently supported only in an IQ simplex configuration (single server). The RLV store acts as a staging area for write events. If a table is enabled for the RLV store, then all LOAD TABLE, INSERT, UPDATE, and DELETE events write directly to the RLV store. In-memory data in the RLV store is automatically (and periodically) merged into the IQ main store. You can set parameters for automatic merges, and you can merge ondemand.

YMMV as always.

Many of us are still running SAP/Sybase IQ 15x.

This document discusses Best Practices for IQ 15. It is intended to provide a starting point for optimizing your implementation of IQ 15. It includes an introduction to new features, suggested methods for using and configuring important features of IQ, and some precautions for avoiding common problems. As a general purpose document, it is not intended to be a comprehensive guide for every environment.

It is always a good idea to be aware of any best practices for a piece of software regardless of whether you intend to follow them verbatim or a subset of them.

SAP Sybase IQ 16 Product Overview – Outlines the new features and benefits of SAP Sybase IQ 16, along with what motivated the build of this product version.

Sybase

SAP Sybase IQ 16 In-Database Analytics Option –  This is a technical overview of the SAP Sybse IQ In-Database Analytics Option. This options delivers fast, accurate insights quickly to help make the decisions needed to win in today’s competitive environment.

SAP Sybase IQ 16 Advanced Security Option – SAP Sybase IQ 16 Advanced Security Option Technical Overview. This option allows advanced protection of your valuable data assets without sacrificing flexibility.

SAP Sybase IQ 16 Very Large Database Option – This is a technical overview for the SAP Sybase IQ 16 Very Large Database Option. This option maximizes the value of your growing data assets.

SAP Sybase IQ 16 Unstructured Data Analytics Option – This option allows for the storage and retrieval of unstructured data including full text search and analysis.

SAP Sybase IQ 16 Multiplex Grid Option – The option delivers elastic capacity for high performance Enterprise Data Warehouses.

Lesson 1: Introduction to Sybase IQ Welcome to “Getting started with Sybase IQ Column Store Analytics Server” This book is set up in a way that will put the subject areas most pertinent to you right at your fingertips. This is not a reference book. Instead, it is a how-to book where the information is presented with the goal of getting you started with Sybase IQ as quickly as possible.

Lesson 2: Product Installation and Database Creation – This chapter discusses installing Sybase IQ, verifying the installation, starting up the server and using client tools to connect to it. It will show you how to create a Sybase IQ database. It will also describe the TPCH schema that you will be creating in your Sybase IQ database. This schema will be the basis for the work you will be doing in subsequent chapters of this book.

Lesson 3: Create Schema and Load Data – This chapter discusses creating users in the database you created in the previous chapter, installing the TPCH schema, and loading data into the tables.

Lesson 4: Optimize a Query Using the Sybase IQ Query Plan – In this chapter, you will learn how Sybase IQ executes queries, and how to interpret the Sybase IQ query plan. The focus will be on query optimization on a single IQ server, not distributed query processing in a Multiplex. That is addressed in another lesson, “Scaling Out with Multiplex and Distributed Query Processing”. In this lesson, you will execute a SQL query, and perform some analysis to optimize the query.

Lesson 5: Monitoring Sybase IQ with Sybase Control Center – Once you have your database up and running, there are administrative tasks that you need to know how to perform in order to insure the health – responsiveness and longevity – of your system. This lesson is going to focus on Sybase Control Center – a graphical monitoring and administration tool for all types of Sybase servers, including Sybase IQ.

Lesson 6: Scaling Out with Multiplex and Distributed Query Processing – If your analytics workload becomes too large for a single machine, Sybase IQ can be configured in a clustered configuration called a “Multiplex” that scales to accommodate more users and more complex queries.

Lesson 7: Pattern Matching with a C++ in-Database UDF – When you want to perform analysis on the data in your Sybase IQ database, you normally will write a SQL query or stored procedure. But Sybase IQ has taken analytics to another level with its in-database analytics feature. Sybase IQ has created an API for extending the functionality of the Sybase IQ database server with user defined functions called UDFs.

Sybase

SAP/Sybase’s bold move of an Oracle Enterprise Manager clone, aka, Sybase Control Center (SCC) has been, well, not so good in the sense of being usable. It is currently undergoing a complete rewrite so until then, if you want to use Sybase Central in lieu of SCC, you will need to download Sybase IQ 15.2.

When you install it, choose Custom Install

then deselect Sybase IQ Server and select Sybase Central.

We can download Sybase IQ 15.2 evaluation software edition using the below links
————————————————————————————
Platform: Windows
Version:15.2

http://download.sybase.com/eval/iq152/iq152_windows.zip

————————————————————————————-
Platform: Linux x86-64
Version:15.2

http://download.sybase.com/eval/iq152/iq152_linux64.tgz
————————————————————————————-
Platform: Linux-Power
Version:15.2

http://download.sybase.com/eval/iq152/iq…nuxP64.tgz
————————————————————————————–

On Windows after installation, you will find Sybase Central in your Start -> All Programs -> Sybase -> Sybase IQ 15.2 -> Sybase Central Java Edition

Please note that Sybase Central v6.0 will NOT work with IQ.

If you’re new to IQ (or SQL Anywhere) you may run into an oddity that if you connect with ISQL the following code works but doesn’t work in DBISQL:

SELECT col1_date
FROM mytable
WHERE col1_date >= '06/01/2014'

This is because any connection to IQ using the native connection (SQL Anywhere connection) that is used by dbisql and the like expects string to datetime format of “YYYY-MM-DD HH:NN:SS.SSS”. This is not changeable by setting an option. The string needs to be converted using a style (datetime string style # 103) prior to being used as a SARG.

SELECT col1_date
FROM mytable
WHERE col1_date >= convert(datetime, '06/01/2014', 103)

Even those of us that have been around for eons tend to forget ‘little’ gotchas like this.

From http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc36271.1570/html/blocks/X41864.htm :

The default values (style 0 or 100), and style 9 or 109 return the century (yyyy). When converting to char or varchar from smalldatetime, styles that include seconds or milliseconds show zeros in those positions.

Very simple question. Very simple answer.

select 
    @@max_connections as 'max_connections', 
    count(*) as 'active_connections', 
    (1 - (@@max_connections - count(*)) / convert(numeric, @@max_connections)) * 100 as 'percent_active' 
  from sp_iqconnection();

Output:

 max_connections active_connections percent_active
 --------------- ------------------ ---------------------
             350                 68               19.4286

@@max_ connections: For the network server, the maximum number of active clients (not database connections, as each client can support multiple connections). For Adaptive Server Enterprise, the maximum number of connections to the server. — Sybase IQ > Reference: Building Blocks, Tables, and Procedures > SQL Language Elements > Variables

Many PDF readers for smart phones (Android/iphone) and tablets manage the pdf files based solely on the Title and Author fields in the PDF file. While for this is fine for your average book, it is not all that helpful with manuals that tend to have abbreviated or no data in the title/author fields. In the case of the manuals for Sybase IQ, I’m unable to load the manuals for say v15.4 and v16.0 as they have the same Title/Author data.

How to fix? Easy. Go get Calibre. Drop the PDF files on to the running Calibre. Edit them by hitting the E key.

In my case, I edited the “Title”, “Author”, “Tags”, “Publisher” and “Languages”:

Calibre doesn’t modify the PDF files themselves so I will need to export the files to a custom directory. In Calibre nomenclature, this is “Saving”. Highlight all the titles you want to export and hit “S” twice. Why twice? No idea. Choose the directory.

You can now copy the exported PDF files to your phone, tablet, whatever without fear of the v16.0 version of the P&T Guide being rejected by Aldiko because the v15.4 version is already added.

Here are the IQ v16.0 SP8 manuals that I’ve ‘fixed’ to work with Aldiko. They are identical to the PDFs on sybooks with the exception of the PDF fields I mentioned previously.

No copyright infringement is intended. SAP/Sybase, please feel free to take these and host them.

IQ 16.0 SP8 – SAP IQ Cockpit – SAP, Inc_
IQ 16.0 SP8 – Installation and Configuration Guide [ Windows ] – SAP, Inc_
IQ 16.0 SP8 – Installation and Configuration Guide [ Solaris ] – SAP, Inc_
IQ 16.0 SP8 – Installation and Configuration Guide [ HP-UX ] – SAP, Inc_
IQ 16.0 SP8 – Administration_ User Management and Security – SAP, Inc_
IQ 16.0 SP8 – Reference_ Statements and Options – SAP, Inc_
IQ 16.0 SP8 – User-Defined Functions – SAP, Inc_
IQ 16.0 SP8 – Administration_ In-Memory Row-Level Versioning – SAP, Inc_
IQ 16.0 SP8 – Reference_ Building Blocks, Tables, and Procedures – SAP, Inc_
IQ 16.0 SP8 – Reference_ Programming – SAP, Inc_
IQ 16.0 SP8 – Release Bulletin [ Windows ] – SAP, Inc_
IQ 16.0 SP8 – Release Bulletin [ Linux ] – SAP, Inc_
IQ 16.0 SP8 – Administration_ Multiplex – SAP, Inc_
IQ 16.0 SP8 – What’s New in SAP IQ 16.0 – SAP, Inc_
IQ 16.0 SP8 – Migration [ Windows ] – SAP, Inc_
IQ 16.0 SP8 – Administration_ Load Management – SAP, Inc_
IQ 16.0 SP8 – Guide to Licensed Options – SAP, Inc_
IQ 16.0 SP8 – Migration [ Linux and UNIX ] – SAP, Inc_
IQ 16.0 SP8 – Quick Start [ Windows ] – SAP, Inc_
IQ 16.0 SP8 – Quick Start [ Linux and UNIX ] – SAP, Inc_
IQ 16.0 SP8 – Administration_ Spatial Data – SAP, Inc_
IQ 16.0 SP8 – Performance and Tuning Guide – SAP, Inc_
IQ 16.0 SP8 – Release Bulletin [ IBM AIX ] – SAP, Inc_
IQ 16.0 SP8 – Release Bulletin [ HP-UX ] – SAP, Inc_
IQ 16.0 SP8 – Release Bulletin [ Solaris ] – SAP, Inc_
IQ 16.0 SP8 – Administration_ Database – SAP, Inc_
IQ 16.0 SP8 – Unstructured Data Analytics – SAP, Inc_
IQ 16.0 SP8 – Utility Guide – SAP, Inc_
IQ 16.0 SP8 – Administration_ Backup, Restore, and Data Recovery – SAP, Inc_
IQ 16.0 SP8 – Installation and Configuration Guide [ IBM AIX ] – SAP, Inc_
IQ 16.0 SP8 – Administration_ Globalization – SAP, Inc_
IQ 16.0 SP8 – Installation and Configuration Guide [ Linux ] – SAP, Inc_
IQ 16.0 SP8 – Interactive SQL Guide – SAP, Inc_
IQ 16.0 SP8 – Introduction to SAP IQ – SAP, Inc_

I recently patched an SAP IQ server to 16.0 SP8 PL30 and ran into an interesting error message when trying to start dbisql:

$ dbisql
Interactive SQL could not load the "SQLAnywhere" plug-in.
Its "ngdbc.jar" file has moved or has been deleted. You will not be able to connect to the databases handled by that plug-in.
Interactive SQL could not load the "SybaseIQ" plug-in.
Its "ngdbc.jar" file has moved or has been deleted. You will not be able to connect to the databases handled by that plug-in.
Interactive SQL could not load the "HANA" plug-in.
Its "ngdbc.jar" file has moved or has been deleted. You will not be able to connect to the databases handled by that plug-in.
Interactive SQL could not load the "GenericODBC" plug-in.
Its "ngdbc.jar" file has moved or has been deleted. You will not be able to connect to the databases handled by that plug-in.
Interactive SQL cannot start because it is not installed correctly. No database plug-ins has been registered.
To fix this problem, you should reinstall the program.

If you scan your IQ directory, you will notice there isn’t a “ngdbc.jar” file. You take a look at another IQ box that is working and it doesn’t have the ngdbc.jar file either. The error message is incorrect. The message should report that it isn’t able to access the saip16.jar (or saip11.jar if you’re on IQ 15.x) and/or the jodbc4.jar file in the $SYBASE/IQ-(IQ RELEASE)/java directory (e.g. $SYBASE/IQ-16_0/java).
Verify that the two files exist and the permissions are correct:

316 -rwxr-xr-x.  1 sybase sybase  320071 Mar 20 14:51 jodbc4.jar
112 -rwxr-xr-x.  1 sybase sybase  112325 Mar 20 14:56 saip16.jar

If everything looks okay and it still gives the error, you will need to re-register the plugins. The first thing is to move the dbisql ‘registry':

$ mv $SYBASE/IQ-(IQ RELEASE)/bin64/dbisql_64.rep $SYBASE/IQ-(IQ RELEASE)/bin64/dbisql_64.rep.old

Next, re-register (for IQ 16):

$ cd $SYBASE/IQ-(IQ RELEASE)/java
$ dbisql -Xregister sa16 SybaseIQ com.sybase.saisqlplugin.IQISQLPlugin "$(pwd)/saip16.jar:$(pwd)/jodbc4.jar"

for IQ 15:

$ cd $SYBASE/IQ-(IQ RELEASE)/java
$ dbisql -Xregister sa11 SybaseIQ com.sybase.saisqlplugin.IQISQLPlugin "$(pwd)/saip11.jar:$(pwd)/jodbc4.jar"

dbisql should now work