Showing posts with label SQL code. Show all posts
Showing posts with label SQL code. Show all posts

Monday, February 8, 2021

Embedding Highcharts In SQL Anywhere


Introduction

Highcharts is a bare-metal browser-based JavaScript API for adding interactive charts to HTML pages.

Highcharts is powerful, well documented and (relatively) easy to use, plus it's been around a long time (2009), it's popular and it's commercially successful.

Here's the Highcharts version of Hello, World!:

Demo, Chart!

[ Click the buttons to see data, then move the mouse to see tooltips ]

( if the buttons don't work, try the Static, Chart! demo instead )

Demo, Chart! is one of three demonstrations, all of which look the same but work differently:

  Demo, Chart! Static, Chart! Hello, Chart!
Where does it run? In this blog post In a website (no CORS) In a SQL Anywhere 17 database
HTML location: blogspot.com bcarter.com SQL Anywhere 17 database
Highcharts JavaScript: code.highcharts.com code.highcharts.com SQL Anywhere 17 database
Data location: CSV files in static website bcarter.com CSV files in static website bcarter.com SQL Anywhere 17 database
Data retrieval: Static file via XMLHttpRequest() Static file via XMLHttpRequest() Web service via XMLHttpRequest()
CORS headers: github.com/Rob--W/cors-anywhere Not required Not required

The code for Demo, Chart! is under your fingertips: View page source, then Find <!-- Start Demo, Chart! code in blog post... -->

The code for Static, Chart! is here:

https://www.bcarter.com/blog/Highcharts/static.html
https://www.bcarter.com/blog/Highcharts/demochart1.csv
https://www.bcarter.com/blog/Highcharts/demochart2.csv

The third demo is shown below.


Hello, Chart! Is Embedded In SQL Anywhere

There are advantages to storing code together with the data in a relational database. For example, application deployment, backup, recovery, portability, security all involve a single *.db file, and the Rules For Relational Databases guarantee consistency among various components.

If you have SQL Anywhere 17 installed, embedding Hello, Chart! is a two-step "Download And Go" process:

Step 1: Download four files into a local folder.

This command file uses the wget utility for Windows to download the files into C:\temp:

CD C:\temp
"C:\download\wget\wget.exe" "https://code.highcharts.com/highcharts.js"
"C:\download\wget\wget.exe" "https://code.highcharts.com/modules/data.js"
"C:\download\wget\wget.exe" "https://www.bcarter.com/blog/highcharts/setup_hello.html"
"C:\download\wget\wget.exe" "https://www.bcarter.com/blog/highcharts/setup_hello.sql.txt"
PAUSE

Step 2: Load the code and launch it in a browser.

The following command file
  • creates a new SQL Anywhere 17 database chart.db via dbinit.exe,
  • starts the database via dbsrv17.exe
  • with the builtin HTTP server running on port 54321,
  • loads the code and data by running setup_hello.sql.txt via dbisql.com
  • and then uses the Windows START command to launch the hello.html web service in a browser window:

"%SQLANY17%\bin64\dbinit.exe"^
  -dba dba,sql^
  -mpl 3^
  "C:\temp\chart.db"

"%SQLANY17%\bin64\dbspawn.exe"^
  -f "%SQLANY17%\bin64\dbsrv17.exe"^
  -xs http(port=54321)^
  "C:\temp\chart.db"

"%SQLANY16%\bin64\dbisql.com"^
  -c "ENG=chart; DBN=chart; UID=dba; PWD=sql;"^
  READ ENCODING Cp1252 "C:\temp\setup_hello.sql.txt"

START http://localhost:54321/hello.html

PAUSE

Here's what it looks like in Edge:


How Hello, Chart! Works

Here's how the code works, starting at the point the web browser launches http://localhost:54321/hello.html.

Step 1: The browser passes the request for "hello.html" to the HTTP server embedded inside the SQL Anywhere 17 process dbsrv17.exe, which is listening for HTTP traffic on port 54321 because of the option -xs http(port=54321).

It could use port 80, but this is a demo, and demos are supposed to work, not generate conflicts with existing software :)

Step 2: SQL Anywhere passes the request to the web service named "hello.html" that is defined in this snippet.

CREATE SERVICE "hello.html" TYPE 'RAW' AUTHORIZATION OFF USER DBA
   AS CALL get_static_text_file ( 'hello.html' );

Step 3: The web service passes the request on to the procedure shown in this snippet.

The RESULT clause on line 61 specifies that the procedure will return a result set to the caller (the browser) consisting of a single LONG VARCHAR column.

The CALL on line 75 sets the HTTP Content-Type header to 'text/html' because the input file extension is 'html'.

The SELECT on line 91 gets the text file data from a table rather than an actual file; this is the SELECT that defines the RESULT set returned by this procedure.

CREATE PROCEDURE get_static_text_file ( IN @file_name VARCHAR ( 255 ) )
   RESULT ( text_string LONG VARCHAR )
BEGIN

DECLARE @extension_pos INTEGER;
DECLARE @extension     VARCHAR ( 100 );

SET @extension_pos = LOCATE ( @file_name, '.', -1 );

SET @extension 
   = IF @extension_pos = 0
        THEN ''
        ELSE SUBSTR ( @file_name, @extension_pos + 1 )
     ENDIF;

CALL dbo.sa_set_http_header ( 
   'Content-Type', 
   CASE @extension
      WHEN 'css'  THEN 'text/css' 
      WHEN 'csv'  THEN 'text/csv' 
      WHEN 'htm'  THEN 'text/html'
      WHEN 'html' THEN 'text/html'
      WHEN 'js'   THEN 'application/javascript' 
      WHEN 'json' THEN 'application/json' 
      WHEN 'xml'  THEN 'application/xml' 
      ELSE             'text/plain'
   END CASE ); 

IF EXISTS ( SELECT *
              FROM static_text_file
             WHERE static_text_file.file_name = @file_name ) THEN
   SELECT static_text_file.file_text
     FROM static_text_file
    WHERE static_text_file.file_name = @file_name;
ELSE
   SELECT STRING ( '[', @file_name, ' not found]' );
END IF;

END;

Step 4: At this point in "how the code works", the browser receives the text for hello.html and starts processing by loading two Highcharts Javascript text files from SQL Anywhere...

<script src="get_static_text_file?f=highcharts.js"></script>
<script src="get_static_text_file?f=data.js"></script>

...using another web service that calls the same stored procedure shown earlier:

CREATE SERVICE get_static_text_file TYPE 'RAW' AUTHORIZATION OFF USER DBA
   AS CALL get_static_text_file ( :f );

Here's where all the "static text file" data came from: it was pre-loaded via xp_read_file() calls from actual files into the static_text_file table.

CREATE TABLE static_text_file (
   file_name  VARCHAR ( 255 ) NOT NULL,
   file_text  LONG VARCHAR NOT NULL,
   PRIMARY KEY ( file_name ) );

INSERT static_text_file ON EXISTING UPDATE VALUES 
   ( 'highcharts.js', 
     dbo.xp_read_file ( 'C:\\temp\\highcharts.js' ) );

INSERT static_text_file ON EXISTING UPDATE VALUES 
   ( 'data.js',       
     dbo.xp_read_file ( 'C:\\temp\\data.js' ) );

INSERT static_text_file ON EXISTING UPDATE VALUES 
   ( 'hello.html',    
     dbo.xp_read_file ( 'C:\\temp\\setup_hello.html' ) );
COMMIT;

Step 5: When the browser has finished loading hello.html, it displays this...

...by executing this snippet of HTML:

<b>Hello, Chart!</b>
<p>
<button onClick="showDataSet ( 1 );">Show Data Set 1</button>
<button onClick="showDataSet ( 2 );">Show Data Set 2</button>
<div id="chartDiv" style="margin-top: 0.5em; padding: 0.5em; border-style: solid; border-width: 1px;">
[ <b>Click the buttons</b> to see data, then <b>move the mouse</b> to see tooltips ]
<p style="padding-left: 2em;">
<i>( this is the SQL Anywhere database version of 
<a href="http://sqlanywhere.blogspot.com/2021/02/embedding-highcharts-in-sql-anywhere.html" 
   target="_blank"><b><u>the code in this blog post</u></b></a> )</i>
</div> 

Step 6: When one of the Show Data Set buttons is pressed, the browser executes the following function where

  • the requestUrl string points to a SQL Anywhere web service,

  • the chartXHR.open() sets up a XMLHttpRequest object to perform an asynchronous GET request for the web service and

  • the chartXHR.send() sends the request.

function showDataSet ( dataSet ) { 
   const requestUrl 
      = "get_csv_chart_data?data_set=" + dataSet; // SQL Anywhere service
   chartXHR.open ( "GET", requestUrl, true ); 
   chartXHR.send(); // this will eventually trigger chartXHR.onload
};

Step 7: The web service get_csv_chart_data builds a CSV result set...

Chart Timestamp,Chart Percent
2020-12-01 13:58:54,.1 
2020-12-01 13:59:04,.1 ...

...by executing this code:

CREATE SERVICE get_csv_chart_data TYPE 'RAW' AUTHORIZATION OFF USER DBA
   AS CALL get_csv_chart_data ( :data_set );

CREATE PROCEDURE get_csv_chart_data ( IN @data_set INTEGER )
   RESULT ( csv_string LONG VARCHAR )
BEGIN

CALL dbo.sa_set_http_header ( 'Content-Type', 'text/csv' );

SELECT STRING ( -- convert rows for one data set into CSV data
          'Chart Timestamp,Chart Percent', 
          '\x0d\x0a',
          LIST ( 
             STRING ( DATEFORMAT ( chart_data.chart_timestamp, 'YYYY-MM-DD HH:MM:SS' ),
                      ',',
                      chart_data.chart_percent ),
             '\x0d\x0a'
             ORDER BY chart_data.chart_timestamp ) )
  FROM chart_data
 WHERE chart_data.data_set = COALESCE ( @data_set, 1 );

END;

Here's where the CSV data originally came from; it was pre-loaded into SQL Anywhere using this snippet.

CREATE TABLE chart_data (
   data_set        INTEGER NOT NULL,
   chart_timestamp TIMESTAMP NOT NULL,
   chart_percent   DECIMAL ( 11, 1 ) NOT NULL,
   PRIMARY KEY ( chart_timestamp ) );

INSERT chart_data VALUES ( 1, '2020-12-01 13:58:54', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 13:59:04', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 13:59:14', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 13:59:24', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 13:59:34', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 13:59:44', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 13:59:54', .1 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:00:04', 1.2 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:00:18', 10.4 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:00:28', 23.9 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:00:38', 29.2 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:00:43', 14.3 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:00:53', .2 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:01:03', .7 );
INSERT chart_data VALUES ( 1, '2020-12-01 14:01:13', 1.2 );

INSERT chart_data VALUES ( 2, '2020-12-01 14:01:28', 16.1 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:01:38', 24.1 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:01:47', 26.0 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:01:53', 7.4 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:02:02', .9 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:02:12', .9 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:02:23', 1.0 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:02:32', .1 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:02:46', 28.3 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:02:54', 39.2 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:03:02', .1 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:03:12', 1.0 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:03:22', .4 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:03:32', 1.7 );
INSERT chart_data VALUES ( 2, '2020-12-01 14:03:45', 18.0 );

COMMIT;

Step 8: When the get_csv_chart_data web service returns the CSV result set, it is received by the XMLHttpRequest.onload event handler.

const chartXHR = new XMLHttpRequest();
chartXHR.onload = function() { // asynchronous callback event handler for chartXHR.send()
   if ( chartXHR.status === 200 ) { // HTTP status OK
      Highcharts.chart ( "chartDiv", { // tell Highcharts where to put the chart
         chart: {
            type: "line"
         },
         data: {
            csv: chartXHR.responseText // tell Highcharts where to get the data
         },
         title: {
            text: "Chart"
         },
         xAxis: {
            title: {
               text: "Chart Timestamp"
            }
         },
         yAxis: {
            title: {
               text: "Chart Percent"
            }
         },
      } );
   } else {
      document.getElementById ( "chartDiv" ).innerHTML = "HTTP Status " + chartXHR.status;
      console.log ( "ERROR: chartXHR.status = ", chartXHR.status );
   }
   return false;
};


See Also...

Highcharts Demos highcharts.com/demo
Highcharts Documentation highcharts.com/docs/index
Highcharts JS API Reference api.highcharts.com
Working with Highcharts data.js highcharts.com/docs/working-with-data/data-module
Highcharts CSV sample jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/data/csv
XMLHttpRequest Tutorial javascript.info/xmlhttprequest
Web Services in SQL Anywhere help.sap.com/viewer/98ad9ec940e2465695685d98e308dff5/17.0/en-US/3bd432396c5f10148782bc977a956885.html


Thursday, December 10, 2020

A Simple JSON Example

Question: How do I dump one row to a json file?

Answer: Here's an adhoc query that writes all the Foxhound 5 Monitor Options values for one target database to a json text file:

-- Start - Foxhound5 - 2 Adhoc Query via ISQL
UNLOAD
SELECT REPLACE ( json_raw_table.json_raw_string, ',"', ',\X0D\X0A"' ) AS monitor_options_json
  FROM ( SELECT *
           FROM alerts_criteria
          WHERE alerts_criteria.criteria_set_type = 'Target Specific'
            AND alerts_criteria.sampling_id       = 13 -- Inventory17
            FOR JSON RAW
       ) AS json_raw_table ( json_raw_string ) 
TO 'C:\\temp\\monitor_options.json'
   DELIMITED BY '' 
   ESCAPES OFF 
   HEXADECIMAL OFF 
   QUOTES OFF;

[{"criteria_set_type":"Target Specific",
"sampling_id":13,
"send_alert_emails":"N",
"mail_recipient":"",
"use_html_in_emails":"Y",
"attach_files_to_emails":"Y",
"email_url_host_port":"localhost",
"display_alerts":"Y",
"autodrop_mail_recipients":"",
"autodrop_mail_recipients_same_as_for_alerts":"Y",
"previous_sample_set_number_processed":3935794,
"previous_gathered_sample_finished_at":"2020-12-10 08:28:08.609",
"current_sample_finished_at":"2020-12-10 08:28:08.609",
"alert1_enabled":"Y",
"alert1_is_in_effect":"N",
"alert1_in_effect_at":"2020-12-07 11:13:00.567",
"alert1_all_clear_at":"2020-12-08 07:48:11.438",
"alert1_threshold_lost_sample_accumulated_msec":60000,
"alert1_actual_lost_sample_accumulated_msec":0,
...

The docs are light on simple questions like this, perhaps because they lead to more simple questions...

How do I get SELECT FOR JSON output into a string?

Answer: Use a SELECT INTO clause, or a subquery.

How do I actually code SELECT FOR JSON in a subquery?

Answer: Code a derived table with an alias table name and a single alias column name, because the select list for SELECT * FOR JSON consists of one single column.

How do I get line breaks?

Answer: Use REPLACE, and hope your data doesn't contain any comma-doublequote pairs... or write better code than this example :)

Monday, November 30, 2020

The Future Of SQL Anywhere

Here's a question that was asked today: https://sqlanywhere-forum.sap.com/questions/36291/sql-anywhere-future...

Question: In our company we are starting a new software project and I am in a dilemma about the database. Today we are using SQL Anywhere v17. But as I see it there is not much work going on on it. Is there any development on it? Could we expect a new version in near future?

Answer: I believe SQL Anywhere 17 has a long and successful future as an extremely powerful and stable relational database system. There probably won't be too many (any more?) "RFQ Checkpoint" features any time soon; e.g., no new kitchen-sink-in-the-database features like Java-objects-in-the-database", full-text search, spatial data.

Don't get me wrong, there's a bunch of kitchen-sink features I love; e.g., HTTP, proxy tables, HA, MobiLink, even SQL Remote. My point is, if you have a favorite non-relational, non-SQL feature you'd like stuffed into SQL Anywhere, don't hold your breath. For example, built-in JavaScript charts... but guess what? you can embed Highcharts in SQL Anywhere yourself, and use XMLHttpRequest() to dynamically request database data from the browser.

And if you're looking for further development of the in-memory database feature, or the addition of a columnar data store, well, that'll won't come until we get UltraLite HANA :)

Seriously, the "kitchen-sink" phase is over, SQL Anywhere has entered the "powerful and stable" stage in its life.

Consider this particular Use Case: SQL Anywhere is embedded in the Oracle MICROS RES 3700 Restaurant POS System which is used in hundreds of thousands of bars, restaurants, hotels and other hospitality establishments worldwide.

BTW, Highcharts rocks, so does FreeFind.


Monday, January 21, 2019

Replace SERVER CLASS 'DIRECTORY' with sp_list_directory()

Question: How do I fix these unexplained intermittent errors when using CREATE SERVER CLASS 'DIRECTORY' to get a list of the file names in a folder?

  • From a SELECT statement even though the CREATE EXISTING TABLE works OK:
    SQLCODE = -141, SQLSTATE = 42W33, ERRORMSG() = Table '...' not found
    

  • From a CREATE EXISTING TABLE statement even though the CREATE EXTERNLOGIN is correct:
    SQLCODE = -656, SQLSTATE = HY000, ERRORMSG() = Unable to connect to server '...': DirectoryAccess servers require an externlogin
    
Answer: Switch to using sp_list_directory(); here's how:
  • Set the max_depth parameter to 1 in the sp_list_directory() call if you are using SUBDIRS=0 in the CREATE SERVER statement.

  • Use SELECT TOP 1 and sa_split_list() to get the file name out of the file_path column returned by sp_list_directory().

  • Use LATERAL to join the sp_list_directory() and sa_split_list() calls.

By Example

After several years of intermittent errors, the SERVER CLASS 'DIRECTORY' method was recently judged too fragile for further use in Foxhound. Here is the before-and-after code for the four sections that were changed:

Example 1 BEFORE: SERVER CLASS 'DIRECTORY'
CREATE SERVER local_folder CLASS 'DIRECTORY' USING 'ROOT={@foxhound_folder};SUBDIRS=0';

CREATE EXTERNLOGIN DBA TO local_folder;

CREATE EXISTING TABLE local_file AT 'local_folder;;;.';

SELECT TOP 1 
       local_file.size,
       local_file.file_name
  INTO @size,
       @file_name
  FROM local_file 
 WHERE local_file.file_name LIKE @patch_file_name_like_pattern
   AND NOT EXISTS ( SELECT *
                      FROM rroad_applied_patch
                     WHERE rroad_applied_patch.file_name = local_file.file_name )
 ORDER BY local_file.file_name ASC; 
Example 1 AFTER: sp_list_directory()
SELECT TOP 1 
       sp_list_directory.file_size    AS size,
       local_file.file_name           AS file_name
  INTO @size,
       @file_name
  FROM sp_list_directory ( @foxhound_folder, 1 ),
       LATERAL ( SELECT TOP 1 
                        sa_split_list.row_value AS file_name
                   FROM sa_split_list ( sp_list_directory.file_path, '\\' )
                  ORDER BY sa_split_list.line_num DESC 
               ) AS local_file 
 WHERE sp_list_directory.file_type = 'F'
   AND local_file.file_name        LIKE @patch_file_name_like_pattern
   AND NOT EXISTS ( SELECT *
                      FROM rroad_applied_patch
                     WHERE rroad_applied_patch.file_name = local_file.file_name )
 ORDER BY local_file.file_name ASC;
Example 2 BEFORE: SERVER CLASS 'DIRECTORY'
CREATE SERVER dynamic_source_help_folder
   CLASS 'DIRECTORY' USING 'ROOT=C:/projects/foxhound5/help/dynamic_source;SUBDIRS=0';

CREATE EXTERNLOGIN DBA TO dynamic_source_help_folder;

CREATE EXISTING TABLE dynamic_source_help_folder AT 'dynamic_source_help_folder;;;.';

INSERT temp_help
SELECT dynamic_source_help_folder.file_name ,
       dbo.xp_read_file ( STRING ( 'C:/projects/foxhound5/help/dynamic_source/', dynamic_source_help_folder.file_name ) )
  FROM dynamic_source_help_folder 
 WHERE dynamic_source_help_folder.file_name LIKE '%.html'
 ORDER BY dynamic_source_help_folder.file_name;
Example 2 AFTER: sp_list_directory()
INSERT temp_help
SELECT dynamic_source_help_folder.file_name ,
       dbo.xp_read_file ( STRING ( 'C:/projects/foxhound5/help/dynamic_source/', dynamic_source_help_folder.file_name ) )
  FROM sp_list_directory ( 'C:/projects/foxhound5/help/dynamic_source', 1 ),
       LATERAL ( SELECT TOP 1 
                        sa_split_list.row_value AS file_name
                   FROM sa_split_list ( sp_list_directory.file_path, '\\' )
                  ORDER BY sa_split_list.line_num DESC 
              ) AS dynamic_source_help_folder  
 WHERE sp_list_directory.file_type          = 'F'
   AND dynamic_source_help_folder.file_name LIKE '%.html'
 ORDER BY dynamic_source_help_folder.file_name;
Example 3 BEFORE: SERVER CLASS 'DIRECTORY'
CREATE SERVER backup_folder CLASS 'DIRECTORY' USING 'ROOT={@backup_folder};SUBDIRS=0';
      
CREATE EXTERNLOGIN DBA_activation TO backup_folder;
      
CREATE EXISTING TABLE backup_file AT 'backup_folder;;;.';

INSERT backup_file_drop_down_list_entry
SELECT ROW_NUMBER() OVER ( ORDER BY backup_file.file_name DESC )    AS row_number,
       backup_file.file_name                                         AS file_name
  FROM backup_file 
 WHERE backup_file.file_name LIKE STRING ( 'foxhound5-', @build_number, '-mini-backup-%.txt' )
 ORDER BY backup_file.file_name DESC;
Example 3 AFTER: sp_list_directory()
INSERT backup_file_drop_down_list_entry
SELECT ROW_NUMBER() OVER ( ORDER BY file_name.file_name DESC )    AS row_number,
       file_name.file_name                                         AS file_name
  FROM sp_list_directory ( @backup_folder, 1 ),
       LATERAL ( SELECT TOP 1 
                        sa_split_list.row_value AS file_name
                   FROM sa_split_list ( sp_list_directory.file_path, '\\' )
                  ORDER BY sa_split_list.line_num DESC 
               ) AS file_name  
 WHERE sp_list_directory.file_type = 'F'
   AND file_name.file_name         LIKE STRING ( 'foxhound5-', @build_number, '-mini-backup-%.txt' )
 ORDER BY file_name.file_name DESC;
Example 4 BEFORE: SERVER CLASS 'DIRECTORY'
CREATE SERVER backup_folder CLASS 'DIRECTORY' USING 'ROOT={@backup_folder};SUBDIRS=0';

CREATE EXTERNLOGIN DBA TO backup_folder;

CREATE EXISTING TABLE backup_file AT 'backup_folder;;;.';

FOR f_backup_file AS c_backup_file INSENSITIVE CURSOR FOR
SELECT TOP ALL START AT ( @mini_backup_retention_count + 1 )
       backup_file.file_name AS @file_name
  FROM backup_file
 WHERE backup_file.file_name LIKE STRING ( 'foxhound5-', @build_number, '-mini-backup-%.txt' )
 ORDER BY backup_file.file_name DESC
FOR UPDATE
DO
Example 4 AFTER: sp_list_directory()
FOR f_backup_file AS c_backup_file INSENSITIVE CURSOR FOR
SELECT TOP ALL START AT ( @mini_backup_retention_count + 1 )
       backup_file.file_name AS @file_name
  FROM sp_list_directory ( @backup_folder, 1 ),
       LATERAL ( SELECT TOP 1 
                        sa_split_list.row_value AS file_name
                   FROM sa_split_list ( sp_list_directory.file_path, '\\' )
                  ORDER BY sa_split_list.line_num DESC 
              ) AS backup_file 
 WHERE sp_list_directory.file_type = 'F'
   AND backup_file.file_name       LIKE STRING ( 'foxhound5-', @build_number, '-mini-backup-%.txt' )
 ORDER BY backup_file.file_name DESC
FOR UPDATE
DO


Saturday, October 6, 2018

Characteristic Errors, Revision 8

A characteristic error is a mistake so easy to make it appears you're actively encouraged to make it by the very nature of the software you're using.

Here are the latest entries...

  1. SQL Anywhere: Forgetting to code an INTO @local_variable clause on a SELECT inside a deeply-nested PROCEDURE, thus accidentally returning an additional result set to an higher FROM call, then wondering why you get SQLCODE -866 (returned a result set with a different schema than expected), or SQLCODE -873 (returned multiple result sets), or both.

  2. SQL Anywhere: Expecting to see MESSAGE TO CLIENT output with the SQL Anywhere 17 ISQL GUI. Don't bother looking, it's virtually inaccessible. Use MESSAGE TO CONSOLE instead, or revert to the version 16 ISQL which works just fine with version 17 databases.





Here's the full list...
  1. SQL: Seeing too little data, or no data at all, because a predicate in the WHERE clause effectively turned your OUTER JOIN into an INNER JOIN.

  2. SQL: Seeing too much data because a missing predicate in the WHERE clause effectively turned your INNER JOIN into a CROSS JOIN.

  3. SQL: Getting the wrong COUNT() or SUM() because you forgot to code WHERE ... IS NOT NULL, or you *did* code it when you shouldn't have.

  4. SQL: Getting the wrong answer because you forgot that, in general, NULL values [cough] suck.

  5. SQL Anywhere: Not seeing MESSAGE output because you forgot to run SET TEMPORARY OPTION DEBUG_MESSAGES = 'ON';

  6. SQL Anywhere: Not seeing any data because you forgot ON COMMIT PRESERVE ROWS or NOT TRANSACTIONAL.

  7. SQL Anywhere: Coding ENDIF where END IF was required, or vice versa (before Version 11).

  8. SQL Anywhere: Connecting to the wrong server because you forgot DOBROAD=NONE (before Version 12).

  9. SQL Anywhere: Forgetting the asterisk in SELECT TOP 10 FROM ...

  10. SQL Anywhere: Coding IF NOT VAREXISTS ( 'x' ) THEN ... instead of IF VAREXISTS ( 'x' ) = 0 THEN ...

  11. SQL Anywhere: Coding the wrong magic numbers 1, 2, 3, ... in the get_value() and set_value() calls in an EXTERNAL C DLL function.

  12. SQL Anywhere: Getting proxy table ODBC errors because the engine's running as a service and you've set up a User DSN instead of System DSN.

  13. SQL Anywhere: Getting file-related errors because the file specifications are relative to the server rather than the client.

  14. SQL Anywhere: Getting file-related errors because the engine's running as a service without the necessary permissions.

  15. SQL Anywhere: Coding CREATE TRIGGER IF NOT EXISTS instead of CREATE OR REPLACE TRIGGER, or vice versa for CREATE TABLE (in 11.0.1 or later).

  16. SQL Anywhere: Getting integer arithmetic when you wanted fractional parts because you forgot to CAST.

  17. Stored procedure debugger: Setting it to watch a specific user id other than the one you're using to test your code.

  18. Sybase Central: Setting it to display objects for owner names other than the one you're interested in.

  19. Copy and paste: Forgetting to edit after pasting; e.g., Copy and paste SET @continue = 'Y' into the body of a WHILE loop and then forgetting to change it to 'N'.

  20. MobiLink: Forgetting to call ml_add_column for any of the columns you're trying to synchronize, thus guaranteeing yourself a "Sassen Frassen Fricken Fracken!" moment when you run the first test.

  21. MobiLink: Forgetting to call ml_add_[various] with the NULL parameter to delete old ml_[whatever] rows, thus ending up with thousands of orphan system table rows in the consolidated database.

  22. OLAP Windowing: Coding the wrong combination of ASC and DESC in an inner OVER ORDER BY clause and the outer SELECT ORDER BY: different when they should be the same, the same when they should be different, or some other variation of "wrong combination"...
                SELECT older_sample_set.sample_set_number
                  INTO @20_older_sample_set_number
                  FROM ( SELECT TOP 20
                                ROW_NUMBER() OVER ( ORDER BY rroad_sample_set.sample_set_number ASC ) AS scrolling_row_number,
                                rroad_sample_set.sample_set_number                                    AS sample_set_number
                           FROM rroad_sample_set
                          WHERE rroad_sample_set.sampling_id       = @sampling_id
                            AND rroad_sample_set.sample_set_number < @sample_set_number
                          ORDER BY rroad_sample_set.sample_set_number DESC ) AS older_sample_set
                 WHERE older_sample_set.scrolling_row_number = 20;

  23. MobiLink: Forgetting to call ml_add_column() when trying to use named parameters instead of "?" in versions 10 and 11 MobiLink scripts, resulting in a "What the ... ? Sassen Frassen Fricken Fracken!" moment during the first test (thank you, Jeff Albion).

  24. SQL: Omitting a PRIMARY KEY column from the WHERE clause, thus turning a singleton SELECT (or DELETE!) into something rather more enthusiastic than expected (thank you, Ron Hiner).

  25. HTTP web services: Leaving an & in the code when a ? is required, and vice versa, when editing service URLs; e.g., 'HTTP://localhost:12345/web_service&service_parm2=!parm2'

  26. SQL Anywhere: Forgetting that not all functions look like functions: SELECT CAST ( CURRENT TIMESTAMP, VARCHAR )

  27. Batch file: Trailing spaces on SET commands; e.g., SELECT CAST ( xp_getenv ( 'DEBUG_MESSAGES' ) AS VARCHAR ) returns 'OFF ' instead of 'OFF' after SET DEBUG_MESSAGES=OFF

  28. Forum: Clicking Reply on the main Question or Answer entry instead of the comment you wanted.

  29. SQL Anywhere: Forgetting to run dblog to tell the database file where the log is now, after moving the database and log files to a different folder (thank you, Justin Willey).

  30. SQL Anywhere: Having to look up WAIT in the Help ... every ... single ... time, to be reminded that's it's WAITFOR, not WAIT.

  31. SQL: Forgetting to check the SELECT against the GROUP BY, resulting in "Function or column reference to ... must also appear in a GROUP BY" (thank you, Glenn Paulley).

  32. SQL: Coding too much in the GROUP BY (like, say, the primary key) so every group contains but a single row (thank you, Glenn Paulley).

  33. Design: Forgetting to accomodate or prevent loops in a tree structure, resulting in a tree traversal process that pegs the CPU at 100%... forever (thank you, Ove B).

  34. MobiLink: Unwittingly using a variety of user ids when running sync*.sql, updating MobiLink scripts and running the MobiLink server, resulting in inexplicable inconsistencies.

  35. MobiLink: Accidentally creating multiple script versions and then getting them crossed up when updating MobiLink scripts and running the MobiLink client.

  36. SQL Anywhere: Forgetting to run the 32-bit version of SQL Anywhere when working with Excel proxy tables.

  37. ODBC Administrator: Running the 64-bit version (huh?) of odbcad32.exe (say what?) when you need 32-bit version at C:\WINDOWS\SysWOW64\odbcad32.exe (oh, fer #*@&!!!)

  38. ODBC Administrator: Forgetting to click OK ... twice ... to actually save your new ODBC DSN after celebrating your success with Test Connection.

  39. ODBC Administrator: Setting up an ODBC DSN on the wrong computer: "It goes with the client!" ... but sometimes it's not obvious where the client is actually located.

  40. Security: Forgetting which Windows user id you're using on which system, then spending too much time with Windows menus, firewall software and Google searches before the "Doh!" moment.

  41. SQL: Getting an exception that is not only completely inexplicable, but absolutely impossible for the statement that raised it... until you think to look inside the triggers.

  42. SQL Anywhere: Getting an exception because a FOR loop variable has a scope conflict with a column name, or worse, NOT getting an exception, just a wrong result.

  43. SQL: Forgetting the comma between two columns in SELECT list, thus turning the second column name into a profoundly misleading alias name for the first column.

  44. SQL Anywhere: Coding SET OPTION instead of SET TEMPORARY OPTION, then wondering where the commit came from.

  45. SQL: Not bothering to check that you have the same number of columns and arguments in an INSERT statement and then wondering why you have an error (thank you, Justin Willey).

  46. SQL: Forgetting to omit your autoincrement primary key from your INSERT column name list (thank you, Justin Willey).

  47. Foxhound: Changing the "Default" Monitor option settings, then expecting them to apply to an existing target database.

  48. Foxhound: Creating two separate Foxhound Monitor sessions for the same actual target database, perhaps one using a DSN and the other a DSN-less connection string.

  49. SQL Anywhere: Forgetting to code AUTOSTOP OFF on a START DATABASE statement, then wondering why the database immediately stopped after starting.

  50. SQL Anywhere: Running a new version of dbupgrade.exe on a older version of a database file that is running on an older version of the database server and not noticing it did not have the desired effect.

    For example, running version 16 of dbupgrade.exe that connects to a version 12 database that is running on dbsrv12.exe will appear to run OK but none of the new version 16 stored procedures will exist after the "upgrade":
    SELECT * FROM sa_certificate_info ( 'whatever' );
    
    Could not execute statement.
    Procedure 'sa_certificate_info' not found
    SQLCODE=-265, ODBC 3 State="42S02"
    Line 1, column 1
    (Continuing after error)
    
    The version 16 dbupgrade.exe says it ran "SQL Anywhere Upgrade Utility Version 16.0.0.2512" but the SYSHISTORY table says the version was '12.0.1.4231':
    SQL Anywhere Upgrade Utility Version 16.0.0.2512
    Database upgrade started
    Creating system views
    Creation of system views completed
    Setting option values
    Setting option values completed
    Creating migration procedures
    Creation of migration procedures completed
    Creating jConnect procedures
    Creation of jConnect procedures completed
    Database upgrade finished
    Upgrade completed - shut down database and archive transaction log now
    
    SELECT operation, version, first_time, last_time
      FROM SYS.SYSHISTORY
     WHERE operation = 'UPGRADE';
    
    operation,  version,        first_time,                last_time
    'UPGRADE',  '12.0.1.4231', '2018-09-04 08:36:01.000',  '2018-09-04 08:36:01.000'
    
    To work as expected, a new version of dbupgrade.exe must connect to a database running on the new version of the database server, either by connecting to a database that is already running on the new server or by implicitly starting that database on the new server via the dbupgrade -c'DBF=...' connection parameter.

  51. SQL Anywhere: Not making the SELECT TOP ORDER BY clause sufficiently deterministic, then wondering why it doesn't pick the same top row every time.

    For example, a SELECT TOP 1 with an ORDER BY that doesn't specify enough columns to exactly determine which of 2 or more rows are the TOP 1 will run OK but it may not pick the same top row every time.

  52. SQL Anywhere: Not checking other ORDER BY clauses when changing one of several ORDER BY clauses in the same query; e.g., not checking a ROW_NUMBER() OVER ( ORDER BY ... ) when changing the SELECT ORDER BY, then wondering why there's something funky about the result set.

    Here's an example from down inside Foxhound:
      FROM ( SELECT TOP 500 
                    ROW_NUMBER() OVER ( ORDER BY rroad_v_sample_history_menu_up_count_locator.locator_number ASC ) AS scrolling_row_number,
                    rroad_v_sample_history_menu_up_count_locator.locator_number                                    AS locator_number,
                    rroad_v_sample_history_menu_up_count_locator.secondary_sort_order_number                       AS secondary_sort_order_number
               FROM rroad_v_sample_history_menu_up_count_locator
              ORDER BY rroad_v_sample_history_menu_up_count_locator.locator_number ASC,
                       rroad_v_sample_history_menu_up_count_locator.secondary_sort_order_number ASC 
           ) AS newer_sample_set
     WHERE newer_sample_set.scrolling_row_number = 500;
    

  53. SQL Anywhere: Forgetting to code an INTO @local_variable clause on a SELECT inside a deeply-nested PROCEDURE, thus accidentally returning an additional result set to an higher FROM call, then wondering why you get SQLCODE -866 (returned a result set with a different schema than expected), or SQLCODE -873 (returned multiple result sets), or both.

  54. SQL Anywhere: Expecting to see MESSAGE TO CLIENT output with the SQL Anywhere 17 ISQL GUI. Don't bother looking, it's virtually inaccessible. Use MESSAGE TO CONSOLE instead, or revert to the version 16 ISQL which works just fine with version 17 databases.

Tuesday, September 4, 2018

Characteristic Errors, Revision 7

UPDATE: See the latest version of this article here.


A characteristic error is a mistake so easy to make it appears you're actively encouraged to make it by the very nature of the software you're using.

Here are the latest entries...

  1. SQL Anywhere: Running a new version of dbupgrade.exe on a older version of a database file that is running on an older version of the database server and not noticing it did not have the desired effect.

    For example, running version 16 of dbupgrade.exe that connects to a version 12 database that is running on dbsrv12.exe will appear to run OK but none of the new version 16 stored procedures will exist after the "upgrade":
    SELECT * FROM sa_certificate_info ( 'whatever' );
    
    Could not execute statement.
    Procedure 'sa_certificate_info' not found
    SQLCODE=-265, ODBC 3 State="42S02"
    Line 1, column 1
    (Continuing after error)
    
    The version 16 dbupgrade.exe says it ran "SQL Anywhere Upgrade Utility Version 16.0.0.2512" but the SYSHISTORY table says the version was '12.0.1.4231':
    SQL Anywhere Upgrade Utility Version 16.0.0.2512
    Database upgrade started
    Creating system views
    Creation of system views completed
    Setting option values
    Setting option values completed
    Creating migration procedures
    Creation of migration procedures completed
    Creating jConnect procedures
    Creation of jConnect procedures completed
    Database upgrade finished
    Upgrade completed - shut down database and archive transaction log now
    
    SELECT operation, version, first_time, last_time
      FROM SYS.SYSHISTORY
     WHERE operation = 'UPGRADE';
    
    operation,  version,        first_time,                last_time
    'UPGRADE',  '12.0.1.4231', '2018-09-04 08:36:01.000',  '2018-09-04 08:36:01.000'
    
    To work as expected, a new version of dbupgrade.exe must connect to a database running on the new version of the database server, either by connecting to a database that is already running on the new server or by implicitly starting that database on the new server via the dbupgrade -c'DBF=...' connection parameter.

  2. SQL Anywhere: Not making the SELECT TOP ORDER BY clause sufficiently deterministic, then wondering why it doesn't pick the same top row every time.

    For example, a SELECT TOP 1 with an ORDER BY that doesn't specify enough columns to exactly determine which of 2 or more rows are the TOP 1 will run OK but it may not pick the same top row every time.

  3. SQL Anywhere: Not checking other ORDER BY clauses when changing one of several ORDER BY clauses in the same query; e.g., not checking a ROW_NUMBER() OVER ( ORDER BY ... ) when changing the SELECT ORDER BY, then wondering why there's something funky about the result set.

    Here's an example from down inside Foxhound:
    FROM ( SELECT TOP 500 
                    ROW_NUMBER() OVER ( ORDER BY rroad_v_sample_history_menu_up_count_locator.locator_number ASC ) AS scrolling_row_number,
                    rroad_v_sample_history_menu_up_count_locator.locator_number                                    AS locator_number,
                    rroad_v_sample_history_menu_up_count_locator.secondary_sort_order_number                       AS secondary_sort_order_number
               FROM rroad_v_sample_history_menu_up_count_locator
              ORDER BY rroad_v_sample_history_menu_up_count_locator.locator_number ASC,
                       rroad_v_sample_history_menu_up_count_locator.secondary_sort_order_number ASC 
           ) AS newer_sample_set
     WHERE newer_sample_set.scrolling_row_number = 500;
    



Here's the full list...
  1. SQL: Seeing too little data, or no data at all, because a predicate in the WHERE clause effectively turned your OUTER JOIN into an INNER JOIN.

  2. SQL: Seeing too much data because a missing predicate in the WHERE clause effectively turned your INNER JOIN into a CROSS JOIN.

  3. SQL: Getting the wrong COUNT() or SUM() because you forgot to code WHERE ... IS NOT NULL, or you *did* code it when you shouldn't have.

  4. SQL: Getting the wrong answer because you forgot that, in general, NULL values [cough] suck.

  5. SQL Anywhere: Not seeing MESSAGE output because you forgot to run SET TEMPORARY OPTION DEBUG_MESSAGES = 'ON';

  6. SQL Anywhere: Not seeing any data because you forgot ON COMMIT PRESERVE ROWS or NOT TRANSACTIONAL.

  7. SQL Anywhere: Coding ENDIF where END IF was required, or vice versa (before Version 11).

  8. SQL Anywhere: Connecting to the wrong server because you forgot DOBROAD=NONE (before Version 12).

  9. SQL Anywhere: Forgetting the asterisk in SELECT TOP 10 FROM ...

  10. SQL Anywhere: Coding IF NOT VAREXISTS ( 'x' ) THEN ... instead of IF VAREXISTS ( 'x' ) = 0 THEN ...

  11. SQL Anywhere: Coding the wrong magic numbers 1, 2, 3, ... in the get_value() and set_value() calls in an EXTERNAL C DLL function.

  12. SQL Anywhere: Getting proxy table ODBC errors because the engine's running as a service and you've set up a User DSN instead of System DSN.

  13. SQL Anywhere: Getting file-related errors because the file specifications are relative to the server rather than the client.

  14. SQL Anywhere: Getting file-related errors because the engine's running as a service without the necessary permissions.

  15. SQL Anywhere: Coding CREATE TRIGGER IF NOT EXISTS instead of CREATE OR REPLACE TRIGGER, or vice versa for CREATE TABLE (in 11.0.1 or later).

  16. SQL Anywhere: Getting integer arithmetic when you wanted fractional parts because you forgot to CAST.

  17. Stored procedure debugger: Setting it to watch a specific user id other than the one you're using to test your code.

  18. Sybase Central: Setting it to display objects for owner names other than the one you're interested in.

  19. Copy and paste: Forgetting to edit after pasting; e.g., Copy and paste SET @continue = 'Y' into the body of a WHILE loop and then forgetting to change it to 'N'.

  20. MobiLink: Forgetting to call ml_add_column for any of the columns you're trying to synchronize, thus guaranteeing yourself a "Sassen Frassen Fricken Fracken!" moment when you run the first test.

  21. MobiLink: Forgetting to call ml_add_[various] with the NULL parameter to delete old ml_[whatever] rows, thus ending up with thousands of orphan system table rows in the consolidated database.

  22. OLAP Windowing: Coding the wrong combination of ASC and DESC in an inner OVER ORDER BY clause and the outer SELECT ORDER BY: different when they should be the same, the same when they should be different, or some other variation of "wrong combination"...
    SELECT older_sample_set.sample_set_number
                  INTO @20_older_sample_set_number
                  FROM ( SELECT TOP 20
                                ROW_NUMBER() OVER ( ORDER BY rroad_sample_set.sample_set_number ASC ) AS scrolling_row_number,
                                rroad_sample_set.sample_set_number                                    AS sample_set_number
                           FROM rroad_sample_set
                          WHERE rroad_sample_set.sampling_id       = @sampling_id
                            AND rroad_sample_set.sample_set_number < @sample_set_number
                          ORDER BY rroad_sample_set.sample_set_number DESC ) AS older_sample_set
                 WHERE older_sample_set.scrolling_row_number = 20;

  23. MobiLink: Forgetting to call ml_add_column() when trying to use named parameters instead of "?" in versions 10 and 11 MobiLink scripts, resulting in a "What the ... ? Sassen Frassen Fricken Fracken!" moment during the first test (thank you, Jeff Albion).

  24. SQL: Omitting a PRIMARY KEY column from the WHERE clause, thus turning a singleton SELECT (or DELETE!) into something rather more enthusiastic than expected (thank you, Ron Hiner).

  25. HTTP web services: Leaving an & in the code when a ? is required, and vice versa, when editing service URLs; e.g., 'HTTP://localhost:12345/web_service&service_parm2=!parm2'

  26. SQL Anywhere: Forgetting that not all functions look like functions: SELECT CAST ( CURRENT TIMESTAMP, VARCHAR )

  27. Batch file: Trailing spaces on SET commands; e.g., SELECT CAST ( xp_getenv ( 'DEBUG_MESSAGES' ) AS VARCHAR ) returns 'OFF ' instead of 'OFF' after SET DEBUG_MESSAGES=OFF

  28. Forum: Clicking Reply on the main Question or Answer entry instead of the comment you wanted.

  29. SQL Anywhere: Forgetting to run dblog to tell the database file where the log is now, after moving the database and log files to a different folder (thank you, Justin Willey).

  30. SQL Anywhere: Having to look up WAIT in the Help ... every ... single ... time, to be reminded that's it's WAITFOR, not WAIT.

  31. SQL: Forgetting to check the SELECT against the GROUP BY, resulting in "Function or column reference to ... must also appear in a GROUP BY" (thank you, Glenn Paulley).

  32. SQL: Coding too much in the GROUP BY (like, say, the primary key) so every group contains but a single row (thank you, Glenn Paulley).

  33. Design: Forgetting to accomodate or prevent loops in a tree structure, resulting in a tree traversal process that pegs the CPU at 100%... forever (thank you, Ove B).

  34. MobiLink: Unwittingly using a variety of user ids when running sync*.sql, updating MobiLink scripts and running the MobiLink server, resulting in inexplicable inconsistencies.

  35. MobiLink: Accidentally creating multiple script versions and then getting them crossed up when updating MobiLink scripts and running the MobiLink client.

  36. SQL Anywhere: Forgetting to run the 32-bit version of SQL Anywhere when working with Excel proxy tables.

  37. ODBC Administrator: Running the 64-bit version (huh?) of odbcad32.exe (say what?) when you need 32-bit version at C:\WINDOWS\SysWOW64\odbcad32.exe (oh, fer #*@&!!!)

  38. ODBC Administrator: Forgetting to click OK ... twice ... to actually save your new ODBC DSN after celebrating your success with Test Connection.

  39. ODBC Administrator: Setting up an ODBC DSN on the wrong computer: "It goes with the client!" ... but sometimes it's not obvious where the client is actually located.

  40. Security: Forgetting which Windows user id you're using on which system, then spending too much time with Windows menus, firewall software and Google searches before the "Doh!" moment.

  41. SQL: Getting an exception that is not only completely inexplicable, but absolutely impossible for the statement that raised it... until you think to look inside the triggers.

  42. SQL Anywhere: Getting an exception because a FOR loop variable has a scope conflict with a column name, or worse, NOT getting an exception, just a wrong result.

  43. SQL: Forgetting the comma between two columns in SELECT list, thus turning the second column name into a profoundly misleading alias name for the first column.

  44. SQL Anywhere: Coding SET OPTION instead of SET TEMPORARY OPTION, then wondering where the commit came from.

  45. SQL: Not bothering to check that you have the same number of columns and arguments in an INSERT statement and then wondering why you have an error (thank you, Justin Willey).

  46. SQL: Forgetting to omit your autoincrement primary key from your INSERT column name list (thank you, Justin Willey).

  47. Foxhound: Changing the "Default" Monitor option settings, then expecting them to apply to an existing target database.

  48. Foxhound: Creating two separate Foxhound Monitor sessions for the same actual target database, perhaps one using a DSN and the other a DSN-less connection string.

  49. SQL Anywhere: Forgetting to code AUTOSTOP OFF on a START DATABASE statement, then wondering why the database immediately stopped after starting.

  50. SQL Anywhere: Running a new version of dbupgrade.exe on a older version of a database file that is running on an older version of the database server and not noticing it did not have the desired effect.

    For example, running version 16 of dbupgrade.exe that connects to a version 12 database that is running on dbsrv12.exe will appear to run OK but none of the new version 16 stored procedures will exist after the "upgrade":
    SELECT * FROM sa_certificate_info ( 'whatever' );
    
    Could not execute statement.
    Procedure 'sa_certificate_info' not found
    SQLCODE=-265, ODBC 3 State="42S02"
    Line 1, column 1
    (Continuing after error)
    
    The version 16 dbupgrade.exe says it ran "SQL Anywhere Upgrade Utility Version 16.0.0.2512" but the SYSHISTORY table says the version was '12.0.1.4231':
    SQL Anywhere Upgrade Utility Version 16.0.0.2512
    Database upgrade started
    Creating system views
    Creation of system views completed
    Setting option values
    Setting option values completed
    Creating migration procedures
    Creation of migration procedures completed
    Creating jConnect procedures
    Creation of jConnect procedures completed
    Database upgrade finished
    Upgrade completed - shut down database and archive transaction log now
    
    SELECT operation, version, first_time, last_time
      FROM SYS.SYSHISTORY
     WHERE operation = 'UPGRADE';
    
    operation,  version,        first_time,                last_time
    'UPGRADE',  '12.0.1.4231', '2018-09-04 08:36:01.000',  '2018-09-04 08:36:01.000'
    
    To work as expected, a new version of dbupgrade.exe must connect to a database running on the new version of the database server, either by connecting to a database that is already running on the new server or by implicitly starting that database on the new server via the dbupgrade -c'DBF=...' connection parameter.

  51. SQL Anywhere: Not making the SELECT TOP ORDER BY clause sufficiently deterministic, then wondering why it doesn't pick the same top row every time.

    For example, a SELECT TOP 1 with an ORDER BY that doesn't specify enough columns to exactly determine which of 2 or more rows are the TOP 1 will run OK but it may not pick the same top row every time.

  52. SQL Anywhere: Not checking other ORDER BY clauses when changing one of several ORDER BY clauses in the same query; e.g., not checking a ROW_NUMBER() OVER ( ORDER BY ... ) when changing the SELECT ORDER BY, then wondering why there's something funky about the result set.

    Here's an example from down inside Foxhound:
    FROM ( SELECT TOP 500 
                    ROW_NUMBER() OVER ( ORDER BY rroad_v_sample_history_menu_up_count_locator.locator_number ASC ) AS scrolling_row_number,
                    rroad_v_sample_history_menu_up_count_locator.locator_number                                    AS locator_number,
                    rroad_v_sample_history_menu_up_count_locator.secondary_sort_order_number                       AS secondary_sort_order_number
               FROM rroad_v_sample_history_menu_up_count_locator
              ORDER BY rroad_v_sample_history_menu_up_count_locator.locator_number ASC,
                       rroad_v_sample_history_menu_up_count_locator.secondary_sort_order_number ASC 
           ) AS newer_sample_set
     WHERE newer_sample_set.scrolling_row_number = 500;
    



Wednesday, August 29, 2018

One Of These Statements Is Wrong



Nobody else has this problem?

Right?

Just me?

I thought so :)


Wednesday, December 20, 2017

Sending Emails Via SendGrid SMTP Relay

Once upon a time, it was possible to send emails from inside SQL Anywhere by using xp_sendmail with Google's free smtp.gmail.com service.

It may still be possible, but apparently not for me... all I get is return code 6 (Secure connection error) after clicking on Send Test Alert Email in Foxhound:

Email test result: SMTP start test AutoDrop email failed with return code 6 when sending 
  "Foxhound Test AutoDrop Email (2017-12-01 09:58:38)". Secure connection error
There are a number of alternatives to Google's free SMTP email server; SendGrid is one of them, here's how to sign up:
  • Go to https://sendgrid.com/.

  • Click on Try it for free.

  • Provide a user name and password, a real email address and some other information.

  • Choose SMTP Relay (you're going to use SendGrid's API via SQL Anywhere's xp_sendmail() procedure).

  • Specify a value for API key and receive a big-long-ugly-generated-password in return.

  • Note the following values (you're going to use port 25):
    Server   smtp.sendgrid.net
    Ports 
    25, 587  (for unencrypted/TLS connections)
    465      (for SSL connections)
    Username apikey
    Password big-long-ugly-generated-password
    
Here's how to set up Foxhound's Alert Email feature to use the SendGrid SMTP Relay:
  • Foxhound Options page - 2. Global Email Settings
    Check Use SMTP to send emails 
    SMTP Sender:  your.email@whatever.com
    SMTP Server:  smtp.sendgrid.net
    SMTP Port:    25
    SMTP Timeout: 60
    SMTP Authorization User Name: apikey
    SMTP Authorization Password:  big-long-ugly-generated-password 
    SMTP Certificate Filespec:    [leave empty] 
    

  • Monitor Options page - 7. Email Setup for [Default Settings]
    Use HTML in emails:             [checked] 
    Host[:port] for URLs in emails: localhost
    Send Alert emails:              [unchecked]
    Email address(es) for Alerts:   your.email@whatever.com
    Click on Send Test Alert Email  Test result: OK
    
Here's how to use the SendGrid SMTP Relay in your own SQL:
BEGIN
DECLARE @return_code           INTEGER;
DECLARE @smtp_sender           LONG VARCHAR;
DECLARE @smtp_server           LONG VARCHAR; 
DECLARE @smtp_port             INTEGER; 
DECLARE @timeout               INTEGER;
DECLARE @smtp_sender_name      LONG VARCHAR;
DECLARE @smtp_auth_username    LONG VARCHAR; 
DECLARE @smtp_auth_password    LONG VARCHAR;
DECLARE @trusted_certificates  LONG VARCHAR;
DECLARE @recipient             LONG VARCHAR;
DECLARE @subject               LONG VARCHAR;
DECLARE @message               LONG VARCHAR;

SET @smtp_sender          = 'your.email@whatever.com';
SET @smtp_server          = 'smtp.sendgrid.net'; 
SET @smtp_port            = 25;   -- 25 for standard SMTP, 465 for SSL, or 587 for TLS
SET @timeout              = 60;   -- default is 60 seconds
SET @smtp_sender_name     = 'Foxhound';
SET @smtp_auth_username   = 'apikey';
SET @smtp_auth_password   = 'big-long-ugly-generated-password';
SET @trusted_certificates = NULL;
SET @recipient            = 'your.email@whatever.com';
SET @subject              = STRING ( 'sendgrid test subject at ', CURRENT TIMESTAMP );
SET @message              = STRING ( 'sendgrid test message at ', CURRENT TIMESTAMP );

@return_code = CALL xp_startsmtp ( 
   smtp_sender          = @smtp_sender,  
   smtp_server          = @smtp_server,  
   smtp_port            = @smtp_port,  
   timeout              = @timeout,
   smtp_sender_name     = @smtp_sender_name,
   smtp_auth_username   = @smtp_auth_username, 
   smtp_auth_password   = @smtp_auth_password,
   trusted_certificates = @trusted_certificates );

MESSAGE STRING ( 'xp_startsmtp @return_code = ', @return_code ) TO CLIENT;
MESSAGE STRING ( 'xp_get_mail_error_code() = ',  xp_get_mail_error_code()      ) TO CLIENT;
MESSAGE STRING ( 'xp_get_mail_error_text() = "', xp_get_mail_error_text(), '"' ) TO CLIENT;

@return_code = CALL xp_sendmail ( 
   recipient     = @recipient,  
   subject       = @subject,  
   "message"     = @message );

MESSAGE STRING ( 'xp_sendmail @return_code = ', @return_code ) TO CLIENT;
MESSAGE STRING ( 'xp_get_mail_error_code() = ',  xp_get_mail_error_code()      ) TO CLIENT;
MESSAGE STRING ( 'xp_get_mail_error_text() = "', xp_get_mail_error_text(), '"' ) TO CLIENT;

@return_code = CALL xp_stopsmtp();

MESSAGE STRING ( 'xp_stopsmtp @return_code = ', @return_code ) TO CLIENT;
MESSAGE STRING ( 'xp_get_mail_error_code() = ',  xp_get_mail_error_code()      ) TO CLIENT;
MESSAGE STRING ( 'xp_get_mail_error_text() = "', xp_get_mail_error_text(), '"' ) TO CLIENT;

EXCEPTION WHEN OTHERS THEN
   CALL xp_stopsmtp();

END;
Here's the MESSAGE ... TO CLIENT output:
xp_startsmtp @return_code = 0
xp_get_mail_error_code() = 235
xp_get_mail_error_text() = "Authentication successful "
xp_sendmail @return_code = 0
xp_get_mail_error_code() = 250
xp_get_mail_error_text() = "Mail queued for delivery. "
xp_stopsmtp @return_code = 0
xp_get_mail_error_code() = 250
xp_get_mail_error_text() = "Mail queued for delivery. "
Here's what the email looks like:




Sunday, July 23, 2017

Characteristic Errors, Revision 6

UPDATE: See the latest version of this article here.


A characteristic error is a mistake so easy to make it appears you're actively encouraged to make it by the very nature of the software you're using.

Here's the latest entry...

  1. SQL Anywhere: Forgetting to code AUTOSTOP OFF on a START DATABASE statement, then wondering why the database immediately stopped after starting.
Here's the full list...
  1. SQL: Seeing too little data, or no data at all, because a predicate in the WHERE clause effectively turned your OUTER JOIN into an INNER JOIN.

  2. SQL: Seeing too much data because a missing predicate in the WHERE clause effectively turned your INNER JOIN into a CROSS JOIN.

  3. SQL: Getting the wrong COUNT() or SUM() because you forgot to code WHERE ... IS NOT NULL, or you *did* code it when you shouldn't have.

  4. SQL: Getting the wrong answer because you forgot that, in general, NULL values [cough] suck.

  5. SQL Anywhere: Not seeing MESSAGE output because you forgot to run SET TEMPORARY OPTION DEBUG_MESSAGES = 'ON';

  6. SQL Anywhere: Not seeing any data because you forgot ON COMMIT PRESERVE ROWS or NOT TRANSACTIONAL.

  7. SQL Anywhere: Coding ENDIF where END IF was required, or vice versa (before Version 11).

  8. SQL Anywhere: Connecting to the wrong server because you forgot DOBROAD=NONE (before Version 12).

  9. SQL Anywhere: Forgetting the asterisk in SELECT TOP 10 FROM ...

  10. SQL Anywhere: Coding IF NOT VAREXISTS ( 'x' ) THEN ... instead of IF VAREXISTS ( 'x' ) = 0 THEN ...

  11. SQL Anywhere: Coding the wrong magic numbers 1, 2, 3, ... in the get_value() and set_value() calls in an EXTERNAL C DLL function.

  12. SQL Anywhere: Getting proxy table ODBC errors because the engine's running as a service and you've set up a User DSN instead of System DSN.

  13. SQL Anywhere: Getting file-related errors because the file specifications are relative to the server rather than the client.

  14. SQL Anywhere: Getting file-related errors because the engine's running as a service without the necessary permissions.

  15. SQL Anywhere: Coding CREATE TRIGGER IF NOT EXISTS instead of CREATE OR REPLACE TRIGGER, or vice versa for CREATE TABLE (in 11.0.1 or later).

  16. SQL Anywhere: Getting integer arithmetic when you wanted fractional parts because you forgot to CAST.

  17. Stored procedure debugger: Setting it to watch a specific user id other than the one you're using to test your code.

  18. Sybase Central: Setting it to display objects for owner names other than the one you're interested in.

  19. Copy and paste: Forgetting to edit after pasting; e.g., Copy and paste SET @continue = 'Y' into the body of a WHILE loop and then forgetting to change it to 'N'.

  20. MobiLink: Forgetting to call ml_add_column for any of the columns you're trying to synchronize, thus guaranteeing yourself a "Sassen Frassen Fricken Fracken!" moment when you run the first test.

  21. MobiLink: Forgetting to call ml_add_[various] with the NULL parameter to delete old ml_[whatever] rows, thus ending up with thousands of orphan system table rows in the consolidated database.

  22. OLAP Windowing: Coding the wrong combination of ASC and DESC in an inner OVER ORDER BY clause and the outer SELECT ORDER BY: different when they should be the same, the same when they should be different, or some other variation of "wrong combination"...
    SELECT older_sample_set.sample_set_number
                  INTO @20_older_sample_set_number
                  FROM ( SELECT TOP 20
                                ROW_NUMBER() OVER ( ORDER BY rroad_sample_set.sample_set_number ASC ) AS scrolling_row_number,
                                rroad_sample_set.sample_set_number                                    AS sample_set_number
                           FROM rroad_sample_set
                          WHERE rroad_sample_set.sampling_id       = @sampling_id
                            AND rroad_sample_set.sample_set_number < @sample_set_number
                          ORDER BY rroad_sample_set.sample_set_number DESC ) AS older_sample_set
                 WHERE older_sample_set.scrolling_row_number = 20;

  23. MobiLink: Forgetting to call ml_add_column() when trying to use named parameters instead of "?" in versions 10 and 11 MobiLink scripts, resulting in a "What the ... ? Sassen Frassen Fricken Fracken!" moment during the first test (thank you, Jeff Albion).

  24. SQL: Omitting a PRIMARY KEY column from the WHERE clause, thus turning a singleton SELECT (or DELETE!) into something rather more enthusiastic than expected (thank you, Ron Hiner).

  25. HTTP web services: Leaving an & in the code when a ? is required, and vice versa, when editing service URLs; e.g., 'HTTP://localhost:12345/web_service&service_parm2=!parm2'

  26. SQL Anywhere: Forgetting that not all functions look like functions: SELECT CAST ( CURRENT TIMESTAMP, VARCHAR )

  27. Batch file: Trailing spaces on SET commands; e.g., SELECT CAST ( xp_getenv ( 'DEBUG_MESSAGES' ) AS VARCHAR ) returns 'OFF ' instead of 'OFF' after SET DEBUG_MESSAGES=OFF

  28. Forum: Clicking Reply on the main Question or Answer entry instead of the comment you wanted.

  29. SQL Anywhere: Forgetting to run dblog to tell the database file where the log is now, after moving the database and log files to a different folder (thank you, Justin Willey).

  30. SQL Anywhere: Having to look up WAIT in the Help ... every ... single ... time, to be reminded that's it's WAITFOR, not WAIT.

  31. SQL: Forgetting to check the SELECT against the GROUP BY, resulting in "Function or column reference to ... must also appear in a GROUP BY" (thank you, Glenn Paulley).

  32. SQL: Coding too much in the GROUP BY (like, say, the primary key) so every group contains but a single row (thank you, Glenn Paulley).

  33. Design: Forgetting to accomodate or prevent loops in a tree structure, resulting in a tree traversal process that pegs the CPU at 100%... forever (thank you, Ove B).

  34. MobiLink: Unwittingly using a variety of user ids when running sync*.sql, updating MobiLink scripts and running the MobiLink server, resulting in inexplicable inconsistencies.

  35. MobiLink: Accidentally creating multiple script versions and then getting them crossed up when updating MobiLink scripts and running the MobiLink client.

  36. SQL Anywhere: Forgetting to run the 32-bit version of SQL Anywhere when working with Excel proxy tables.

  37. ODBC Administrator: Running the 64-bit version (huh?) of odbcad32.exe (say what?) when you need 32-bit version at C:\WINDOWS\SysWOW64\odbcad32.exe (oh, fer #*@&!!!)

  38. ODBC Administrator: Forgetting to click OK ... twice ... to actually save your new ODBC DSN after celebrating your success with Test Connection.

  39. ODBC Administrator: Setting up an ODBC DSN on the wrong computer: "It goes with the client!" ... but sometimes it's not obvious where the client is actually located.

  40. Security: Forgetting which Windows user id you're using on which system, then spending too much time with Windows menus, firewall software and Google searches before the "Doh!" moment.

  41. SQL: Getting an exception that is not only completely inexplicable, but absolutely impossible for the statement that raised it... until you think to look inside the triggers.

  42. SQL Anywhere: Getting an exception because a FOR loop variable has a scope conflict with a column name, or worse, NOT getting an exception, just a wrong result.

  43. SQL: Forgetting the comma between two columns in SELECT list, thus turning the second column name into a profoundly misleading alias name for the first column.

  44. SQL Anywhere: Coding SET OPTION instead of SET TEMPORARY OPTION, then wondering where the commit came from.

  45. SQL: Not bothering to check that you have the same number of columns and arguments in an INSERT statement and then wondering why you have an error (thank you, Justin Willey).

  46. SQL: Forgetting to omit your autoincrement primary key from your INSERT column name list (thank you, Justin Willey).

  47. Foxhound: Changing the "Default" Monitor option settings, then expecting them to apply to an existing target database.

  48. Foxhound: Creating two separate Foxhound Monitor sessions for the same actual target database, perhaps one using a DSN and the other a DSN-less connection string.

  49. SQL Anywhere: Forgetting to code AUTOSTOP OFF on a START DATABASE statement, then wondering why the database immediately stopped after starting.



Friday, February 20, 2015

Using CROSS APPLY To Help SELECT TOP 1

Question: How do I find the currently-blocked connections in all the databases being monitored by Foxhound?

Answer: Complete access to the entire Foxhound database for adhoc queries is one of the hallmarks of Foxhound, but sometimes it requires more than a simple SELECT.

Here's a query that answers the question, followed by a sample result set:

WITH block AS (
SELECT sampling_options.sampling_id                  AS sampling_id,
       latest_header.sample_set_number               AS sample_set_number,
       latest_header.sample_finished_at              AS recorded_at, 
       IF sampling_options.selected_tab = 1 
          THEN STRING ( 'DSN: ',    sampling_options.selected_name )  
          ELSE STRING ( 'String: ', sampling_options.selected_name )
       END IF                                        AS target_database,
       blocked_connection.Userid                     AS blocked_Userid,
       blocked_by_connection.Userid                  AS blocked_by_Userid,
       blocked_connection.blocker_reason             AS reason,
       blocked_connection.ReqStatus                  AS ReqStatus,
       blocked_connection.blocker_table_name         AS blocker_table_name  
  FROM sampling_options                                            -- one row per target database
       CROSS APPLY ( SELECT TOP 1 *                                -- most recent successful sample for each target
                       FROM sample_header
                      WHERE sample_header.sampling_id = sampling_options.sampling_id 
                        AND sample_header.sample_lost = 'N'
                      ORDER BY sample_header.sample_set_number DESC ) AS latest_header
       LEFT OUTER JOIN ( SELECT *                                       -- all the blocked connections in the latest sample
                           FROM sample_connection
                          WHERE sample_connection.BlockedOn <> 0 ) AS blocked_connection
          ON  blocked_connection.sampling_id       = sampling_options.sampling_id 
          AND blocked_connection.sample_set_number = latest_header.sample_set_number
       LEFT OUTER JOIN sample_connection AS blocked_by_connection       -- the corresponding blocking connections
          ON  blocked_by_connection.sampling_id       = sampling_options.sampling_id 
          AND blocked_by_connection.sample_set_number = blocked_connection.sample_set_number
          AND blocked_by_connection.connection_number = blocked_connection.BlockedOn )
SELECT *
  FROM block
 ORDER BY block.target_database,
       block.blocked_Userid;
sampling_id, sample_set_number, recorded_at, target_database, blocked_Userid, blocked_by_Userid, reason, ReqStatus, blocker_table_name
16, 2583089, '2015-02-20 09:43:56.544', 'DSN: ddd10', , , , , 
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'c.ryan', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'f.thomson', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'g.mikhailov', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'h.barbosa', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'i.miller', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'n.simpson', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'u.wouters', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'x.wang', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
17, 2583088, '2015-02-20 09:43:56.251', 'DSN: Inventory', 'y.gustavsson', 'e.reid', Row Transaction Intent,  Row Transaction WriteNoPK, BlockedLock, 'inventory'
15, 2583090, '2015-02-20 09:44:01.005', 'DSN: RuralFinds', , , , , 
  • The WITH clause on lines 1 through 28 creates a temporary view that is used in the SELECT * FROM block at the bottom.

    The FROM sampling_options on line 14 selects one row for each target database. The rest of the query is designed to show at least row for each target even if it doesn't have any blocked connection, even if sampling is currently stopped.

  • The CROSS APPLY on lines 15 through 19 selects one sample_header row for each target database. That row is the most recent successful sample for that target. If sampling is running then it will be a recent row. If sampling is stopped then it might be an old row. Either way, it is the "most recent successful sample" for each target database.

    The CROSS APPLY clause is used instead of INNER JOIN because the inner FROM sample_header clause refers to a column in a different table in the outer FROM clause, something you can't do with INNER JOIN.

    Without the CROSS APPLY clause, the SELECT TOP 1 clause wouldn't work properly, and the query would become much more complex... it's the CROSS APPLY that makes the TOP 1 work properly by returning a different TOP 1 row for each row in sampling_options.

  • The two LEFT OUTER JOIN clauses on lines 20 through 28 gather up all the blocked (victim) sample_connection rows plus the corresponding blocking (evil-doer) sample_connection rows.

    LEFT OUTER JOIN clause is used instead of INNER JOIN so the view will return at least one row for each target database even if it doesn't have any blocked connections.
For more queries, see "How do I run adhoc queries on the Foxhound database?"