Chapter 13
Other Data Sources

[p]This chapter documents tags and behaviors which are specific to the Spotlight and custom third-party data sources in Lasso. See the appropriate chapter in this section for information about other data sources including SQL Data Sources, FileMaker Data Sources, and JDBC and ODBC Data Sources. [/p]

• Overview introduces the data sources.
• Feature Matrix includes a table which lists all of the features of each data source and highlights the differences between them.t
• Spotlight Data Source describes how to access the file system indexing in Mac OS X 10.4/10.5 in order to search the contents and attributes of files on the local machine.
• Custom Data Sources describes how to access custom third-party data sources through Lasso.

Overview

[p]The Spotlight data source allows the meta information which is generated by the Spotlight indexer on Mac OS X 10.4/10.5 to be searched by Lasso. This data source provides an effective file system search which is powerful and efficient. [/p]

[p]This chapter also includes information about how to use custom third-party data sources with Lasso. Data sources can be implemented in LassoScript, C/C++ (LCAPI), or Java (LJAPI). Third-party data sources can adhere to the conventions created by the internal data sources or can diverge wildly. [/p]

Table 1: Data Sources

[table][tr][th]Data Source[/th][th]Description[/th][th] [/th][/tr]

[tr][td]Spotlight[/td][td]Supports searching files on Mac OS X 10.4/10.5 systems only. [/td][/tr]

[tr][td]Custom[/td][td]Lasso allows third parties to create data sources using LassoScript, LCAPI, or LJAPI. [/td][/tr]

[/table]

Feature Matrix

[p]The following tables detail the features of each data source in this chapter. Since some features are only available in certain data sources it is important to check these tables when reading the documentation in order to ensure that each data source supports your solutions required features. [/p]

Table 2: Spotlight Data Source

[table][tr][th]Feature[/th][th]Description[/th][th] [/th][/tr]

[tr][td]Platform[/td][td]Mac OS X Only [/td][/tr]

[tr][td]Friendly Name[/td][td]Lasso Connector for Spotlight [/td][/tr]

[tr][td]Internal Name[/td][td]spotlightds. [/td][/tr]

[tr][td]Module Name[/td][td]SpotlightDS.dylib [/td][/tr]

[tr][td]Actions[/td][td]-Search, -Show, -SQL (to specify a raw Spotlight search) [/td][/tr]

[tr][td]Operators[/td][td]-BW, -CN, -EQ, -EW, -GT, -GTE, -LT, -LTE, -NBW, -NCN, -NEW
-OpBegin/-OpEnd with And, Or, Not. [/td][/tr]

[tr][td]Notes[/td][td]This data source differs significantly from other built-in data sources. Read the full details in the Spotlight Data Source section.. [/td][/tr]

[/table]

Table 3: Custom Data Sources

[table][tr][th]Feature[/th][th]Description[/th][th] [/th][/tr]

[tr][td]Module Name[/td][td]Third-party data sources can be implemented in a .lasso or .lassoapp file in LassoStartup, or in a .dll, .dylib, .so, or .jar file in LassoModules. [/td][/tr]

[tr][td]Inline Host Attributes[/td][td]Data source dependent. Third-party data sources may use the -Name and -Port fields in a manner similar to built-in data sources or may use them for custom configuration data. [/td][/tr]

[tr][td]Actions[/td][td]Data source dependent. Third-party data sources may implement standard actions or can implement completely custom actions. [/td][/tr]

[/table]

Spotlight Data Source

[p]The Spotlight data source is used to search the file system on machines with Mac OS X 10.4/10.5. This data source is not supported on earlier versions of Mac OS X or on Windows or Linux. [/p]

[p]The Spotlight data source has one pre-defined host and several databases which represent different metadata sets that can be searched. The Lasso Site Administrator must enable the metadata sets which they want to make available for searching. [/p]

[p]Apple’s developer information page about Spotlight provides a conceptual background to how Spotlight works, what types of metadata are indexed, and how to run Spotlight queries from the command line. [/p]

[pre]http://developer.apple.com/macosx/spotlight.html [/pre]

[note][b]Important: [/b]The Spotlight data source searches the raw contents of files. When the Web site root is searched the raw LassoScript contained within Lasso pages will be searched, not the output of those pages. [/note]

Requirements

[p]The Spotlight data source requires Mac OS X 10.4/10.5. The Spotlight data source can only be used to search for files and folders which have been indexed by the Spotlight indexer. [/p]

[p]Sites will only be able to return Spotlight search results for files contained within their Web server root unless the Server Administrator has granted the site permission to access files outside the root and appropriate file permissions have been granted to the user who is performing the search. [/p]

Spotlight Metadata Sets (Databases)

[p]Lasso defines four Spotlight metadata sets as databases within Lasso Site Administration. A set will also be defined for each volume which is mounted on the machine hosting Lasso Service. Each set must be enabled and appropriate permissions assigned to it before they can be searched by the site user. Some of the sets also require file permissions to be configured. Each metadata set can be enabled by visiting the Site > Data Sources > Databases section of Lasso Site Administration. [/p]

• Spotlight_Site – This set corresponds to the Web site root. Once this “database” is enabled it can be searched by any Lasso use who has permission to access it. No additional file permissions are required.
• Spotlight_User – This set corresponds to the Lasso Professional 8 application folder. The site must be assigned permission to access files outside of the Web server root and the Lasso user must have permission to access those files in order to use this set.
• Spotlight_Computer – This set corresponds to the root of the machine hosting Lasso Service. The site must be assigned permission to access files outside of the Web server root and the Lasso user must have permission to access those files in order to use this set. Note that the search results may contain files which the Lasso user is not capable of manipulating through the file tags.
• Spotlight_Network – This set corresponds to the local area network of the machine hosting Lasso Service. The site must be assigned permission to access files outside of the Web server root and the Lasso user must have permission to access those files in order to use this set.
• /Volumes/MacintoshHD – Each local hard disk, mounted CD or DVD, mounted disk image, and file server will appear as a separate “database” with the name [code]/Volumes/[/code] followed by the volume name.

[note][b]Note: [/b]The [code]-Table[/code], [code]-KeyField[/code], [code]-MaxRecords[/code], and [code]-SkipRecords[/code] tags are not used with the Spotlight data source. [/note]

Spotlight Fields

[p]The fields for Spotlight are defined by the file importers which are active on the system. The following table lists some of the most common field names. See below the table for code which returns a complete list of available field names. [/p]

Table 4: Common Spotlight Field Names

[table][tr][th]Field Name[/th][th]Description[/th][th] [/th][/tr]

[tr][td]Content Created[/td][td]The date/time the contents were created (06/03/20 19:20:58 GMT-08:00) [/td][/tr]

[tr][td]Content Modified[/td][td]The date/time the contents were modified (06/05/04 09:56:25 GMT-07:00) [/td][/tr]

[tr][td]Created[/td][td]The date/time the file was created (06/03/20 19:20:58 GMT-08:00) [/td][/tr]

[tr][td]Display Name[/td][td]The name of the file in the Finder (default.lasso) [/td][/tr]

[tr][td]File Extension Hidden[/td][td]Is the file extension hidden in the Finder (false) [/td][/tr]

[tr][td]File Invisible[/td][td]Is the file invisible in the Finder (false) [/td][/tr]

[tr][td]File Label[/td][td]The ID of the Finder label for the file (0) [/td][/tr]

[tr][td]File Pathname[/td][td]The path to the file (/Library/WebServer/Documents/85/default.lasso). This is the only field which is returned by default if no -ReturnField parameter is specified. [/td][/tr]

[tr][td]Filename[/td][td]The name of the disk file (default.lasso) [/td][/tr]

[tr][td]Group[/td][td]The ID of the group which owns the file (80) [/td][/tr]

[tr][td]Last Opened[/td][td]The date/time the file was last opened (06/05/04 09:48:49 GMT-07:00) [/td][/tr]

[tr][td]Modified[/td][td]The date/time the file was modified (06/05/04 09:56:25 GMT-07:00) [/td][/tr]

[tr][td]Owner[/td][td]The ID of the user who owns the file (501) [/td][/tr]

[tr][td]Size[/td][td]The size of the file in bytes (24666) [/td][/tr]

[tr][td]Text Content[/td][td]The contents of the file. This field can be used for searching, but is not returned in search results. [/td][/tr]

[tr][td]Type[/td][td]The type of the file (text file) [/td][/tr]

[tr][td]*[/td][td]Can be specified as the return field to return all available fields -ReturnField='*'. This is not recommended for general use since generating some of the field values can be performance intensive. [/td][/tr]

[tr][td]kMDItem…[/td][td]It is also possible to use any of the low-level constants which Apple defines for Spotlight queries. These look like kMDItemDisplayName and can be used in place of the friendly field names which Lasso makes available. [/td][/tr]

[/table]

To return the complete list of avaiiable fields:

[p]Use an inline with a [code]-Show[/code] command as in the following code. [/p]

[pre]Inline: -Show, -Database='Spotlight_Site';
Field_Names->Sort & Join('<br>');
/Inline; [/pre]

[pre] Album
Alpha channel
Aperture
Audiences
Audio bit rate
Audio encoding application
… [/pre]

Spotlight Queries

[p]Spotlight queries can make use of any of the fields listed in the table above (or returned by the [code]-Show[/code] inline above). Lasso’s search operators function the same as they do for any database queries. By default all searches of text data are performed as “begins with” searches. For example, the following search would find all disk files whose name begins with [code]default[/code]. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
'Display Name' = 'default';
Records_Array->Join('<br>');
/Inline; [/pre]

[pre] /Library/WebServer/Documents/default.html
/Library/WebServer/Documents/default.lasso [/pre]

[p]The operator can be changed by preceding the search terms with any of Lasso’s built-in operator keywords [code]-BW[/code], [code]-CN[/code], [code]-EQ[/code], [code]-EW[/code], [code]-NBW[/code], [code]-NCN[/code], [code]-NEQ[/code], or [code]-NEW[/code]. (or with [code]-Op='bw'[/code] etc.). For example, the following search would find all disk files which contain the word [code]LassoScript[/code]. [/p]

Inline: -Search,
-Database='Spotlight_Site',
-CN, 'Text Content' = 'LassoScript';
Records_Array->Join('<br>');
/Inline;

[pre] /Library/WebServer/Documents/default.lasso
/Library/WebServer/Documents/include.lasso [/pre]

[p]The [code]-OpLogical[/code] and [code]-OpBegin[/code]/[code]-OpEnd[/code] keywords can be used to create complex nested search. The following code finds all files whose name ends with [code].lasso[/code] or [code].html[/code] and whose contents contains [code]LassoScript[/code]. [/p]

Inline: -Search,
-Database='Spotlight_Site',
-OpLogical='And',
-OpBegin='Or',
-EW, 'Filename' = '.lasso',
-EW, 'Filename' = '.html',
-OpEnd='Or',
-CN, 'Text Content' = 'LassoScript';
Records_Array->Join('<br>');
/Inline;

[pre] /Library/WebServer/Documents/default.lasso [/pre]

[p]When searching fields whose value is an integer, decimal, or boolean value an “Equals” search is used automatically. The operator can be changed by preceding the search terms with any of Lasso’s built-in operator keywords [code]-EQ[/code], [code]-GT[/code], [code]-GTE[/code], [code]-LT[/code], [code]-LTE[/code], or [code]-NEQ[/code] (or with [code]-Op='gt'[/code] etc.). For example, the following search would find all disk files whose size is greater than 32k. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
-GT, 'size' = 32768;
Records_Array->Join('<br>');
/Inline; [/pre]

[pre] /Library/WebServer/Documents/splah.swf [/pre]

Restricting a Query

[p]A query can be restricted to a sub-folder of the Web site root using a [code]File Pathname[/code] term. Most Spotlight searches should be restricted to a folder that only contains the text, image, audio, video, or PDF files which are to be searched. If Spotlight is allowed to search the entire Web server root it will find terms within the raw LassoScript source code of Lasso pages. [/p]

[p]The following code searches only within [code]Images[/code] folders inside the Web root. Note that any path which contains [code]/Images/[/code] is searched including an [code]Images[/code] folder at the root of the Web server and an [code]Images[/code] folder within a sub-folder. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
-GT, 'size' = 32768,
-CN, 'File Pathname' = '/Images/';
Records_Array->Join('<br>');
/Inline; [/pre]

[pre] /Library/WebServer/Documents/Images/logo.gif
/Library/WebServer/Documents/Images/welcome.gif
/Library/WebServer/Documents/Admin/Images/secure.gif [/pre]

[pre]Spotlight Return Fields [/pre]

[p]By default the Spotlight data source only returns the [code]File Pathname[/code] field. Additional fields can be returned by listing the desired field names with -ReturnField. Spotlight queries can return any of the fields listed in the table above (or returned by the [code]-Show[/code] inline above). In addition [code]-ReturnField='*'[/code] can be used as a shortcut to return every available field. [/p]

[p]For example, the following code returns information about an AAC encoded audio file. The [code]-ReturnField[/code] keyword is used to return [code]Album[/code] and [code]Title[/code] information as well as the standard [code]File Pathname[/code]. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
-EW, 'filename' = '.m4a',
-ReturnField='Album',
-ReturnField='Title';
Records;
'Album: ' + (Field: 'Album') + '<br>';
'File Pathname: ' + (Field: 'File Pathname') + '<br>';
'Title: ' + (Field: 'Title') + '<br>';
/Records;
/Inline; [/pre]

[pre] Album: Music From The Good, The Bad And The Ugly & A Fistful Of Dollars & For A Few Dollars More
File pathname: /Library/WebServer/Documents/01 The Good, The Bad And The Ugly.m4a
Title: The Good, The Bad And The Ugly [/pre]

[p]If [code]-ReturnField='*'[/code] is used instead then a wealth of information about the file is returned. Note that the code within the inline uses a conditional to avoid displaying any field which has an empty value. The actual fields which are displayed will depend on what indexers are available. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
-EW, 'filename' = '.m4a',
-ReturnField='*';
Records;
Iterate: Field_Names->Sort &, (Var: 'field');
(Field: $field) != '' ? $field + ': ' + (Field: $field) + '<br>';
/Iterate;
'<hr>';
/Records;
/Inline; [/pre]

[pre] Album: Music From The Good, The Bad And The Ugly & A Fistful Of Dollars & For A Few Dollars More
Audio bit rate: 124.852
Audio encoding application: iTunes v4.0.1, QuickTime 6.4
Channel count: 2
Display name: 01 The Good, The Bad And The Ugly.m4a
Duration: 166.997
File pathname: /Library/WebServer/Documents/01 The Good, The Bad And The Ugly.m4a
Filename: 01 The Good, The Bad And The Ugly.m4a
Modified: 04/02/18 07:56:52 GMT-08:00
Musical genre: Soundtrack
Size: 2722974
Streamable: false
Title: The Good, The Bad And The Ugly
Total bit rate: 124.852
Track number: 1
Type: MPEG-4 Audio File [/pre]

Raw Queries

[p]The Spotlight data source supports the [code]-SQL[/code] action as a method of specifying raw data source queries. The [code][Action_Statement][/code] tag can be used to view the raw query generated by any Spotlight inline. If [code]-StatementOnly[/code] is specified then the raw query will be generated and not actually performed. For example, the following inline generates the raw query which corresponds to the specified search. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
-GT, 'size' = 32768,
-StatementOnly;
Action_Statement;
/Inline; [/pre]

[pre] ((kMDItemFSSize > 32768)) [/pre]

[p]That raw query can then be executed by feeding it into the Spotlight data source through the [code]-SQL[/code] action. [/p]

[pre]Inline: -Search,
-Database='Spotlight_Site',
-SQL='((kMDItemFSSize > 32768))';
Records_Array->Join('<br>');
/Inline; [/pre]

[pre] splah.swf [/pre]

Custom Data Sources

[p]Third-party custom data sources can be implemented in LassoScript, C/C++ (LCAPI), or Java (LJAPI). Third-party data sources can adhere to the conventions created by the internal data sources or can diverge wildly. Consult the documentation of the third-party data source for full details about how it operates. [/p]

[p]See the Custom Data Sources, LCAPI Data Sources, or LJAPI Data Sources chapters in this manual for more information about third-party data sources. The following Tip of the Week contains the source code of a data source which is written in LassoScript. [/p]

[pre]Custom Inline Data Source
http://www.lassosoft.com/TotW.1768.8931.lasso[/pre]