Content Intelligence Workflow Designer Help

Version: 2.0.x
Audience: anonymous
Part Number: MK-HCI007-00

Quick Start: Indexing and searching
Create a data connection to one of your data sources
Create an index collection
Create, edit, and run a workflow
Search for files
Administering Hitachi Content Search
About Hitachi Content Search
Search App
Configuring your user's search experience
Enabling full-content search
Search result highlighting
Making images, audio, and video appear in search results
Query suggestions
Configuring what query suggestions are based on
Adding facets to the Search App
Search result sorting
Configuring per document security
Adding file type icons to search results
Special cases and scenarios
Using aggregations and triggers to monitor a workflow
Create the trigger pipeline
Create the trigger
Run your workflow
Writing HCI custom metadata to HCP objects
Writing file contents as custom metadata to HCP objects
Adding document caching to your workflows
Copying data from Amazon S3 to HCP
Parsing and indexing CSV and log files
Conditionally, separate log files into individual entries
Create a workflow to parse and index individual log entries
Documents
Workflows
Creating a workflow
Viewing workflows
Searching workflows
Workflow inputs
Adding existing inputs to a workflow
Adding new inputs to a workflow
Removing inputs from a workflow
Workflow pipelines
Adding pipelines to a workflow
Reordering pipelines in a workflow
Removing pipelines from a workflow
Workflow outputs
Adding existing outputs to a workflow
Adding new index collections to a workflow
Removing outputs from a workflow
Aggregations
Viewing aggregation values
Aggregation types
Default aggregations
Aggregations, triggers, and workflow notifications
Adding aggregations to workflows
Viewing aggregations for a workflow
Triggers
Trigger expressions
List of trigger comparators
Example trigger expression
Trigger pipelines
Trigger pipeline documents
Example trigger pipeline document
Adding triggers to a workflow
Example trigger pipeline
Workflow metrics information for triggers
Workflow tasks
Running workflow tasks
Run a workflow task
Running multiple workflows at the same time
Configuring where workflows run
Pausing and resuming tasks
Task settings
Document Discovery settings
Performance settings
Metrics settings
Error Handling settings
Memory settings
Reconfiguring tasks
Scheduling tasks
Determining when a running task will stop
Task details, status, and results
Task Status
Status messages
Status icons
Failures
Metrics
Performance
Aggregations task details
Triggers
Viewing workflow task details
Task status icons
Working with fields discovered by a workflow task
Clearing workflow tasks
Starting a task over
Downloading task reports
Testing individual document failures
Retrying task failures
Testing workflows
Deleting a workflow
Importing and exporting workflow components
Exporting workflow components
Importing workflow components
Resuming halted workflows
Actions
Supported actions
Permissions for data connection actions
Performing actions
Execute Action stage advantages
Workflow output action advantages
Data connections
Data connection types and settings
Box data connection (Preview Mode)
Authentication
List-based data connection
Configuration settings
Supported actions
Document fields
Microsoft Sharepoint data connection (Preview Mode)
Authentication
List-based data connection
Configuration settings
Supported actions
Document fields
Microsoft OneDrive for Business data connection
Authentication
List-based data connection
Configuration settings
Supported actions
Document fields
HCP (Hitachi Content Platform) data connection
HCP system requirements
Configuration settings
Supported actions
Authentication and action permissions
HCP object versioning
Checking for updates with an HCP connector
How this data connection determines which file to perform an action on
How this data connection populates HCI_relativePath field
Setting up the HCP connector to perform a Copy File action across HCP tenants
HCP MQE (Hitachi Content Platform Metadata Query Engine) data connection
HCP system requirements
HCP MQE object versioning
Authentication and action permissions
Checking for updates with an MQE connector
Configuration settings
Supported actions
How the data connection determines which file to perform an action on
How this data connection populates the HCI_relativePath field
HCP Anywhere (Single user) data connection
HCP Anywhere system requirements
Checking for updates with an HCP Anywhere single user connector
Configuration settings
Supported actions
How an HCP Anywhere data connection populates HCI_relativePath
HCP Anywhere (System-wide) data connection
HCP Anywhere system and user requirements
Checking for updates with an HCP Anywhere system connector
Configuration settings
Supported actions
How this data connection determines which file to perform an action on
HCP for Cloud Scale Monitoring data connection (Preview Mode)
Configuration settings
Amazon® S3 data connection
Authentication
Checking for updates with an Amazon S3 connector
Configuration settings
Supported actions
How the data connection determines which file to perform an action on
How this data connection populates the HCI_relativePath field
Considerations - Amazon S3 connections
S3 Compatible data connection
Authentication
Connecting to HCP over S3
Checking for updates with an S3 compatible connector
Configuration settings
Supported actions
How the data connection determines which file to perform an action on
How this data connection populates the HCI_relativePath field
Considerations - S3 compatible connections
HCP Monitoring data connection
How this data connection collects information
Supported HCP versions
Fields added to all documents
Fields added by the Node Status API
Fields added by MAPI
Fields added by SNMP
Configuration settings
HCP Syslog Kafka Queue data connection
HCP requirements
Checking for updates with this connector
Configuration settings
Testing this connection
Kafka Queue data connection
Checking for updates with this connector
Configuration settings
Supported actions
PostgreSQL JDBC data connection
Authentication
Avoid restarting workflow tasks that use this data connection
Checking for updates for an SQL data connector
Configuration settings
Supported actions
MySQL and MariaDB JDBC data connection
Authentication
Avoid restarting workflow tasks that use this data connection
Checking for updates for an SQL data connector
Configuration settings for a MySQL and MariaDB JDBC connection
Supported actions
Solr JDBC data connection
Avoid restarting workflow tasks that use this data connection
Checking for updates for an SQL data connector
Configuration settings for a MySQL and MariaDB JDBC connection
Configuration settings for a Solr JDBC data connection
Supported actions
Hadoop File System data connection
Authentication
Checking for updates with a Hadoop data connection
Configuration settings
Supported actions
How this data connection determines which file to perform an action on
How this data connection populates the HCI_relativePath field
Local File System data connection (FOR TEST USE ONLY)
Permissions for use
Connecting to NFS file systems
POSIX metadata
Configuration settings
Supported actions
How this data connection determines where to perform an action
How this data connection populates the HCI_relativePath field
GUID-ADCDEB60-D25B-45B6-B5C0-A4E4D5BD4AE1.html
Permissions for use
Authentication
Configuration settings
Supported actions
How this data connection determines where to perform an action
How this data connection populates the HCI_relativePath field
Solr Query data connections
Solr Query data connection
Internal Index Query data connection
Configuration settings for a Solr Query connector
Configuration settings for an Internal Index Query connector
Crawling behavior for Solr Query connectors
Creating data connections
Related CLI commands - createDataSource
Related REST API methods - POST datasources
Viewing data connections
Related CLI commands - listDataSources
Related REST API methods - GET datasources
Editing data connections
Related CLI commands - editDataSource
Related REST API methods - PUT datasources
Testing data connections
Related CLI commands - testDataSource
Related REST API methods - POST datasources test
Adding data connections to workflows
Deleting data connections
Related CLI commands - deleteDataSource
Related REST API methods - DELETE datasources
Maintaining timestamp metadata on CIFS, HDFS, and LFS connectors
Processing pipelines and stages
Creating and modifying processing pipelines
Creating a new pipeline
Related CLI commands - createPipeline
Related REST API methods - POST pipelines
Adding stages to a pipeline
Related CLI commands - editPipeline
Related REST API methods - PUT pipelines
Configuring stages
Related CLI commands - editPipeline
Related REST API methods - PUT pipelines
Adding conditional statements to a pipeline
Related CLI commands - editPipeline
Related REST API methods - PUT pipelines
Adding actions to a pipeline
Related CLI commands - editPipeline
Related REST API methods - PUT pipelines
Removing stages from a pipeline
Related CLI commands - editPipeline
Related REST API methods - PUT pipelines
Viewing pipelines
Related CLI commands - listPipelines
Related REST API methods - GET pipelines
Built-in stages
Add Metadata stage
Appender stage
Attach Stream stage
Cache Stream stage
Content Class Extraction stage
Date Conversion stage
Decompression stage
DICOM Metadata Extraction stage
Drop Documents stage
Document Security stage
Email Expansion stage
Email Notification stage
Execute Action stage
Filter stage
Field Parser stage
Geocoding stage
HCP Access Log Parser stage
HCP System Events Parser stage
HCP Chargeback Report Parser stage
HCP Resource Metrics Parser stage
JavaScript stage
Single Field Transformation Mode
Document Transformation Mode
Mapping stage
Mbox Expansion stage
MIME Type Detection stage
PST Expansion stage
Read Lines stage
Replace stage
Reject Documents stage
Snippet Extraction stage
SRT Text Extraction stage
Syslog Notification stage
Tagging stage
TAR Expansion stage
Text and Metadata Extraction stage
Tokenizer stage
URL Encoder/Decoder stage
XML Formatter stage
ZIP Expansion stage
Built-in pipelines
Default pipeline
Basic Preprocessing pipeline
Basic Extraction pipeline
Conditional statements
Creating your own stages
Testing pipelines
Testing pipelines
Related CLI commands - runPipelineTest
Related REST API methods - POST GET pipelines
Viewing stream contents
Working with fields discovered by a test
Evaluating pipeline performance
Obtain and analyze pipeline performance data
Reducing time spent indexing
Reducing time spent expanding archive files
Reducing time spent by non-expansion stages
Pipeline execution modes
When to use Workflow-Agent mode
When to use Preprocessing mode
Setting pipeline execution modes
How workflow recursion settings interact with pipeline execution modes
Deleting pipelines
Related CLI commands - deletePipeline
Related REST API methods - DELETE pipelines
Content classes
Creating and editing content classes
Related CLI commands - createContentClass
Related REST API methods - POST contentclasses
Viewing content classes
Related CLI commands - getContentClass
Related REST API methods - GET contentclasses
Testing content classes
Related CLI commands - testContentProperties
Related REST API methods - POST contentclasses test
Using content classes
Deleting content classes
Related CLI commands - deleteContentClass
Related REST API methods - DELETE contentclasses
Index collections
Index collection types and settings
Initial schema options
Adding index collections
Related CLI commands - createIndex
Related REST API methods - POST indexes
Index Collections Overview tab
Viewing index collections and statistics
Related CLI commands - getIndex
Related REST API methods - GET indexes
HCI Index statistics
Apache Solr statistics
Elasticsearch statistics
HDDS statistics
Index shards
Specifying the shard count for an index
Multiple shards on a single-instance system
Balancing shards across instances
Index protection level for HCI Indexes
Editing Index Protection Level for an index
Configuring external indexes
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Index collection schemas
Defined dynamic and copy fields
Field types
Adding and editing fields in an index collection schema
Related CLI commands - editIndex
Related REST API methods - PUT indexes fields
Use cases
Handling multi value fields
Field attributes
Importing fields from a workflow into an index collection
Using Language Select
Directly editing configuration files for HCI Indexes
Related CLI commands - getIndexConfigFile
Related REST API methods - GET configFiles
Querying an index
Related CLI commands - rawQuery
Related REST API methods - POST indexes rawQuery
Query syntax
Deleting an index collection
Related CLI commands - deleteIndex
Related REST API methods - DELETE indexes
Unregistering an index collection
Related CLI commands - unregisterIndex
Related REST API methods - DELETE indexes unregister
Considerations - External Apache Solr index collections
Query settings
Creating and editing query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring advanced search options
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Query setting configuration options
Configuring index display name
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Enabling and disabling query suggestions
Enabling search result download
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring fields in query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring facets in query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Visualizations
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring filtering in query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring sortable fields in query settings
Boosting field relevancy
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring search result layouts
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring access control settings in query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Configuring custom parameters in query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Filtering queries by user
Enabling and disabling query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Deleting query settings
Related CLI commands - editIndex
Related REST API methods - PUT indexes
Index Collection Delete-by-query
Controlling user access to search indexes
Granting limited access to a search index
Granting public access to a search index
Enabling multiple query settings for an index collection
Hitachi Content Intelligence fields
SQL queries
Related CLI commands - rawSqlQuery
Related REST API methods - POST query SQL
Solr index backup and restore
Index backup API
Index restore API
Backup and restore status API
List backups API
Delete backup API
Best practices
For configuring stages and pipelines
Test your pipelines to learn how each stage affects your documents
Order your stages and pipelines correctly
Identify pipeline performance issues
Use conditional statements to limit unnecessary document processing
Process only the documents that your users want
Use the Reject Documents stage to debug your pipelines
Use Workflow Recursion to simplify your pipelines
Use temporary fields
Follow the best practices for each stage type
For running workflow tasks
Schedule tasks to run when user demand on the system is low
Favor processing large numbers of small documents
Optimize your pipeline before changing task performance settings
Experiment with task performance setting changes
Run the Workflow-Agent jobs on as many instances as possible
Optionally, avoid processing old versions of HCP objects
Avoid or minimize metrics collection in production
Avoid unnecessary file rereads
For designing an index
Scale your index appropriately to ensure availability and fast query responses
For HCI Indexes, increase the Index Protection Level setting to maintain index availability and performance
Use the appropriate initial schema value
Favor query settings changes over index collection schema changes
Minimize field attributes in your index collection schema
Minimize the number of copy fields
Do not use dynamic fields in a production index
Use the smallest field types possible for numerical values
Follow this example procedure to create an index for use in production
Use the Index action in your pipeline to test the effects of pipeline stages on your index
For connecting to HCP
HCP data connection
HCP MQE data connection
S3 Compatible data connection
For end-user search experience
Expose as many searchable fields as is useful
Keep searches fast
Enable extra search features
For writing content classes
Have Hitachi Content Intelligence create content properties for you
Keep content property types in separate content classes
Normalize field names within a content class
Troubleshooting
Workflow task issues
Data connection issues
Pipeline and task performance issues
System event issues
Reference
Writing and testing your own plugins
Cloning components
Supported browsers
Accessing other apps
REST API reference
Input and output format
Access and authentication
Viewing and using REST API methods
Error responses
CLI reference
Accessing the CLI tools on a system instance
Accessing the CLI tools from your computer
Syntax
Viewing available commands
Viewing request models
Editing configuration preferences
System error responses
Glossary

Back to home page