From 4b65dfb424395832ce3ddd78061f09453b147e86 Mon Sep 17 00:00:00 2001 From: leto-bbq Date: Thu, 27 Aug 2026 12:25:22 +0800 Subject: [PATCH] docs: add Copy SQL export to TsFile for table model (V2.0.11) --- .../Tools-System/Data-Export-Tool_apache.md | 157 ++++++++++++++++-- .../Tools-System/Data-Export-Tool_apache.md | 157 ++++++++++++++++-- .../Tools-System/Data-Export-Tool_apache.md | 156 +++++++++++++++-- .../Tools-System/Data-Export-Tool_apache.md | 156 +++++++++++++++-- 4 files changed, 582 insertions(+), 44 deletions(-) diff --git a/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md b/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md index 24a4d8716..410ae5f86 100644 --- a/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md +++ b/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md @@ -1,32 +1,39 @@ # Data Export ## 1. Function Overview -The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows: + +IoTDB supports two methods for data export: + +* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory and can export query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) formats. +* Copy SQL Export to TsFile: Write query results back to a TsFile at a specified path via SQL. - + - - - + + + - - + + - + + + + + - +
File FormatIoTDB ToolIoTDB Tool Description
CSVexport-data.sh/batPlain text format for storing structured data. Must follow the CSV format specified below.CSVexport-data.sh/batPlain text format for storing structured data. Must follow the CSV format specified below.
SQLFile containing custom SQL statements.SQLFile containing custom SQL statements.
TsFileTsFileOpen-source time-series file format.
Copy SQL Open-source time-series file format.
- -## 2. Detailed Features +## 2. Data Export Tool ### 2.1 Common Parameters | Short | Full Parameter | Description | Required | Default | |----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |----------------------------------------------| @@ -167,3 +174,131 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. Copy SQL + +> **Note: This feature is supported since V2.0.11.** + +### 3.1 Command + +```SQL +// ---------------------------------------- Copy Statement --------------------------------------------------------- +copyToStatement + : COPY '(' query ')' TO fileName=string ((WITH)? copyToStatementOptions)? + | COPY tableName=qualifiedName ('(' tableColumns=identifierList ')')? TO fileName=string ((WITH)? copyToStatementOptions)? + ; + +copyToStatementOptions + : '(' copyToStatementOption (',' copyToStatementOption)* ')' + ; + +copyToStatementOption + : FORMAT identifier + | TABLE identifier + | TAGS '(' identifierList ')' + | TIME identifier + | MEMORY_THRESHOLD memory=INTEGER_VALUE + ; +``` + +#### Parameters + +| Name | Description | Default | +|---|---|---| +| FORMAT | Export format. Currently only TsFile is supported. | TsFile | +| TABLE | Specifies the table name in the generated TsFile. | If the query SQL involves only one table, that table name is used; otherwise, `default` is used. | +| TIME | Specifies which column in the result set is used as the TIME column.
When manually specified: an error is reported if the column type is not TIMESTAMP or the specified column cannot be found.
When not manually specified: an error is also reported if a column named `time` exists but its type is not TIMESTAMP.
The time column is constructed with the following priority:
1. The column with the same name as the time column of the single table involved in the query
2. The column named "time" with the TIMESTAMP type as the time column
3. Use the current number of written rows for the corresponding device as time to generate the time column, with the column name "time" | - | +| TAGS | Specifies which columns are TAG columns. When there are multiple TAG columns, the order in the final generated table is consistent with the specified order.
When manually specified: an error is reported if a column does not exist, a column type is not STRING, or duplicate column names exist.
If the query involves only one table and all tag columns of the table can be found in the query result set, they are inferred as the tag columns of that table; otherwise, the default value is an empty list, meaning all columns except the TIME column are treated as FIELD columns | Empty list | +| MEMORY_THRESHOLD | Used for memory control when generating the TsFile (unit: byte). An error is reported when the manually specified value is less than or equal to 0. | 32MB | + +#### Result Set + +| Column | Data Type | Description | +|---|---|---| +| path | STRING | Absolute path of the generated target file | +| row_count | INT64 | Total number of written rows | +| device_count | INT64 | Number of generated devices | +| size_in_bytes | INT64 | Size of the generated target file | +| table_name | STRING | Table name in the target file. If it is auto-generated, it will be marked with `(auto_gen)`. | +| time_column | STRING | Name of the time column of the table in the target file. If it is auto-generated, it will be marked with `(auto_gen)`. | +| tag_columns | STRING | Names of the tag columns of the table in the target file, separated by `,`. | + +#### Other Notes + +* File generation location: + * If a file name is specified, the generated TsFile is saved under `${dn_data_dirs}/copy_to` of the DataNode directly connected to the client. When multiple directories are configured, the file is generated according to the strategy of the configuration item `dn_multi_dir_strategy`. + * If a path is specified, the file is saved under the specified path. +* Possible exceptions during execution: + * An error is reported when out-of-order timestamps exist while writing to the TsFile according to the given schema + * An error is reported when the file name is invalid or the target file already exists + * Duplicate column names exist in the query result + * Insufficient disk space + +### 3.2 Examples + +Taking table1 in the [Sample Data](../Reference/Sample-Data.md) as an example + +1. Export all data in table1 to the file copysql1.tsfile via a select statement + +```SQL +IoTDB:database1> copy (select * from table1) to 'copysql1.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql1.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.336s +``` + +2. Export all data in table1 to the file copysql2.tsfile via the table name + +```SQL +IoTDB:database1> copy table1 to 'copysql2.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql2.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.048s +``` + +3. Export part of the data in table1 to the file copysql3.tsfile via the table name (columns) + +```SQL +IoTDB:database1> copy table1 (device_id,temperature) to 'copysql3.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql3.tsfile| 18| 1| 558| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.064s +``` + +4. Export the aggregation results of part of the data in table1 to the file copysql4.tsfile via a select statement + +```SQL +IoTDB:database1> copy (select count(temperature), count(humidity) from table1 group by device_id) to 'copysql4.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql4.tsfile| 2| 1| 543| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.155s +``` + +5. Export part of the data in table1 to the file copysql5.tsfile via a select statement, and specify the target table, time column, and tag columns + +```SQL +IoTDB:database1> copy (select time,region,device_id,temperature from table1 order by time) to 'copysql5.tsfile' (TABLE copytable, TIME time, TAGS (region,device_id)) ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| /iotdb/data/datanode/data/copy_to/copysql5.tsfile| 18| 4| 1199| copytable| time|[region, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +Total line number = 1 +It costs 0.047s +``` diff --git a/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md b/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md index 24a4d8716..410ae5f86 100644 --- a/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md +++ b/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md @@ -1,32 +1,39 @@ # Data Export ## 1. Function Overview -The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows: + +IoTDB supports two methods for data export: + +* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory and can export query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) formats. +* Copy SQL Export to TsFile: Write query results back to a TsFile at a specified path via SQL. - + - - - + + + - - + + - + + + + + - +
File FormatIoTDB ToolIoTDB Tool Description
CSVexport-data.sh/batPlain text format for storing structured data. Must follow the CSV format specified below.CSVexport-data.sh/batPlain text format for storing structured data. Must follow the CSV format specified below.
SQLFile containing custom SQL statements.SQLFile containing custom SQL statements.
TsFileTsFileOpen-source time-series file format.
Copy SQL Open-source time-series file format.
- -## 2. Detailed Features +## 2. Data Export Tool ### 2.1 Common Parameters | Short | Full Parameter | Description | Required | Default | |----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |----------------------------------------------| @@ -167,3 +174,131 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. Copy SQL + +> **Note: This feature is supported since V2.0.11.** + +### 3.1 Command + +```SQL +// ---------------------------------------- Copy Statement --------------------------------------------------------- +copyToStatement + : COPY '(' query ')' TO fileName=string ((WITH)? copyToStatementOptions)? + | COPY tableName=qualifiedName ('(' tableColumns=identifierList ')')? TO fileName=string ((WITH)? copyToStatementOptions)? + ; + +copyToStatementOptions + : '(' copyToStatementOption (',' copyToStatementOption)* ')' + ; + +copyToStatementOption + : FORMAT identifier + | TABLE identifier + | TAGS '(' identifierList ')' + | TIME identifier + | MEMORY_THRESHOLD memory=INTEGER_VALUE + ; +``` + +#### Parameters + +| Name | Description | Default | +|---|---|---| +| FORMAT | Export format. Currently only TsFile is supported. | TsFile | +| TABLE | Specifies the table name in the generated TsFile. | If the query SQL involves only one table, that table name is used; otherwise, `default` is used. | +| TIME | Specifies which column in the result set is used as the TIME column.
When manually specified: an error is reported if the column type is not TIMESTAMP or the specified column cannot be found.
When not manually specified: an error is also reported if a column named `time` exists but its type is not TIMESTAMP.
The time column is constructed with the following priority:
1. The column with the same name as the time column of the single table involved in the query
2. The column named "time" with the TIMESTAMP type as the time column
3. Use the current number of written rows for the corresponding device as time to generate the time column, with the column name "time" | - | +| TAGS | Specifies which columns are TAG columns. When there are multiple TAG columns, the order in the final generated table is consistent with the specified order.
When manually specified: an error is reported if a column does not exist, a column type is not STRING, or duplicate column names exist.
If the query involves only one table and all tag columns of the table can be found in the query result set, they are inferred as the tag columns of that table; otherwise, the default value is an empty list, meaning all columns except the TIME column are treated as FIELD columns | Empty list | +| MEMORY_THRESHOLD | Used for memory control when generating the TsFile (unit: byte). An error is reported when the manually specified value is less than or equal to 0. | 32MB | + +#### Result Set + +| Column | Data Type | Description | +|---|---|---| +| path | STRING | Absolute path of the generated target file | +| row_count | INT64 | Total number of written rows | +| device_count | INT64 | Number of generated devices | +| size_in_bytes | INT64 | Size of the generated target file | +| table_name | STRING | Table name in the target file. If it is auto-generated, it will be marked with `(auto_gen)`. | +| time_column | STRING | Name of the time column of the table in the target file. If it is auto-generated, it will be marked with `(auto_gen)`. | +| tag_columns | STRING | Names of the tag columns of the table in the target file, separated by `,`. | + +#### Other Notes + +* File generation location: + * If a file name is specified, the generated TsFile is saved under `${dn_data_dirs}/copy_to` of the DataNode directly connected to the client. When multiple directories are configured, the file is generated according to the strategy of the configuration item `dn_multi_dir_strategy`. + * If a path is specified, the file is saved under the specified path. +* Possible exceptions during execution: + * An error is reported when out-of-order timestamps exist while writing to the TsFile according to the given schema + * An error is reported when the file name is invalid or the target file already exists + * Duplicate column names exist in the query result + * Insufficient disk space + +### 3.2 Examples + +Taking table1 in the [Sample Data](../Reference/Sample-Data.md) as an example + +1. Export all data in table1 to the file copysql1.tsfile via a select statement + +```SQL +IoTDB:database1> copy (select * from table1) to 'copysql1.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql1.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.336s +``` + +2. Export all data in table1 to the file copysql2.tsfile via the table name + +```SQL +IoTDB:database1> copy table1 to 'copysql2.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql2.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.048s +``` + +3. Export part of the data in table1 to the file copysql3.tsfile via the table name (columns) + +```SQL +IoTDB:database1> copy table1 (device_id,temperature) to 'copysql3.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql3.tsfile| 18| 1| 558| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.064s +``` + +4. Export the aggregation results of part of the data in table1 to the file copysql4.tsfile via a select statement + +```SQL +IoTDB:database1> copy (select count(temperature), count(humidity) from table1 group by device_id) to 'copysql4.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql4.tsfile| 2| 1| 543| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.155s +``` + +5. Export part of the data in table1 to the file copysql5.tsfile via a select statement, and specify the target table, time column, and tag columns + +```SQL +IoTDB:database1> copy (select time,region,device_id,temperature from table1 order by time) to 'copysql5.tsfile' (TABLE copytable, TIME time, TAGS (region,device_id)) ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| /iotdb/data/datanode/data/copy_to/copysql5.tsfile| 18| 4| 1199| copytable| time|[region, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +Total line number = 1 +It costs 0.047s +``` diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md b/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md index dbf3737f1..688ce13f3 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md @@ -2,32 +2,38 @@ ## 1. 功能概述 -数据导出工具 `export-data.sh/bat` 位于 `tools` 目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile（开源时间序列文件格式）格式。具体功能如下： +IoTDB 支持两种方式进行数据导出： + +* 数据导出工具 ：`export-data.sh/bat` 位于 `tools `目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile （开源时间序列文件格式）格式。 +* Copy SQL 导出 TsFile：通过 SQL 将查询结果写回到指定路径的 TsFile 中。 - + - - - + + + - - + + - + - + + + + +
文件格式IoTDB工具IoTDB工具 具体介绍
CSVexport-data.sh/bat纯文本格式，存储格式化数据，需按照下文指定 CSV 格式进行构造CSVexport-data.sh/bat纯文本格式，存储格式化数据，需按照下文指定 CSV 格式进行构造
SQL包含自定义 SQL 语句的文件SQL包含自定义 SQL 语句的文件
TsFileTsFile 开源时序数据文件格式
Copy SQL开源时序数据文件格式
- -## 2. 功能详解 +## 2. 数据导出工具 ### 2.1 公共参数 @@ -177,3 +183,131 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. Copy SQL + +> **注意：该功能自 V2.0.11 版本起支持。** + +### 3.1 运行命令 + +```SQL +// ---------------------------------------- Copy Statement --------------------------------------------------------- +copyToStatement + : COPY '(' query ')' TO fileName=string ((WITH)? copyToStatementOptions)? + | COPY tableName=qualifiedName ('(' tableColumns=identifierList ')')? TO fileName=string ((WITH)? copyToStatementOptions)? + ; + +copyToStatementOptions + : '(' copyToStatementOption (',' copyToStatementOption)* ')' + ; + +copyToStatementOption + : FORMAT identifier + | TABLE identifier + | TAGS '(' identifierList ')' + | TIME identifier + | MEMORY_THRESHOLD memory=INTEGER_VALUE + ; +``` + +#### 参数介绍 + +| 名称 | 说明 | 默认值 | +|---|---|---| +| FORMAT | 导出格式，当前仅有 TsFile | TsFile | +| TABLE | 指定生成的 TsFile 中的 Table 名称 | 如果查询 SQL 仅涉及一个 table，使用这个 table 名，否则使用 `default` | +| TIME | 指定使用结果集中的哪一列作为 TIME 列。
手动指定时：列类型非 TIMESTAMP、找不到指定列均会报错；
未手动指定时：若存在列名 time 但类型非 TIMESTAMP 也会报错。
按以下优先级构造 time 列：
1. 查询仅涉及的一个 table 中的 time 列名称相同的列
2. 寻找列名为 "time" 且类型为 TIMESTAMP 的列作为时间列
3. 使用对应 device 写入的当前行数作为 time 生成时间列，列名为 "time" | - | +| TAGS | 指定哪些列为 TAG 列，有多个 TAG 列时，最终生成的 table 内的顺序和指定的顺序一致。
手动指定时：列不存在、列类型非 STRING、存在重复列名均会报错。
如果查询仅涉及一个 table，且 table 所有 tag 列在查询结果集中可以找到，则推断为这个 table 的 tag 列，否则默认值为空列表，即除了 TIME 列以外其余所有列都被视为 FIELD 列 | 空列表 | +| MEMORY_THRESHOLD | 用于在生成 TsFile 时进行内存控制（单位：byte），手动指定数值小于等于 0 时将报错 | 32MB | + +#### 结果集说明 + +| 列名 | 数据类型 | 说明 | +|---|---|---| +| path | STRING | 生成的目标文件的绝对路径 | +| row_count | INT64 | 总写入行数 | +| device_count | INT64 | 生成的设备数量 | +| size_in_bytes | INT64 | 生成的目标文件大小 | +| table_name | STRING | 目标文件中的表名，如果是自动生成的，会通过 `(auto_gen)` 进行标记 | +| time_column | STRING | 目标文件中的表的 time 列名，如果是自动生成的，会通过 `(auto_gen)` 进行标记 | +| tag_columns | STRING | 目标文件中的表的 tag 列名，以 `,` 分隔 | + +#### 其他注意事项 + +* 对于文件生成位置： + * 如果指定的是文件名，生成的 TsFile 保存在客户端直连的 DataNode 的 `${dn_data_dirs}/copy_to` 下，配置多个目录时，按照配置项 `dn_multi_dir_strategy` 的策略生成； + * 如果指定的是一个路径，则放在指定的路径下。 +* 执行过程中可能出现的异常： + * 按照给定的 schema 写入 TsFile，存在乱序时间戳时报错 + * 非法文件名或目标文件已存在时报错 + * 查询结果中存在重复列名 + * 磁盘空间不足 + +### 3.2 运行示例 + +以[示例数据](../Reference/Sample-Data.md)中 table1 为例 + +1. 通过 select 语句将 table1 中的全部数据导出到文件 copysql1.tsfile + +```SQL +IoTDB:database1> copy (select * from table1) to 'copysql1.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql1.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.336s +``` + +2. 通过表名将 table1 中的全部数据导出到文件 copysql2.tsfile + +```SQL +IoTDB:database1> copy table1 to 'copysql2.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql2.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.048s +``` + +3. 通过表名(列)的方式将 table1 中的部分数据导出到文件 copysql3.tsfile + +```SQL +IoTDB:database1> copy table1 (device_id,temperature) to 'copysql3.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql3.tsfile| 18| 1| 558| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.064s +``` + +4. 通过 select 语句将 table1 中部分数据的聚合结果导出到文件 copysql4.tsfile + +```SQL +IoTDB:database1> copy (select count(temperature), count(humidity) from table1 group by device_id) to 'copysql4.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql4.tsfile| 2| 1| 543| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.155s +``` + +5. 通过 select 语句将 table1 中的部分数据导出到文件 copysql5.tsfile ，并指定目标表、time列及 tag 列 + +```SQL +IoTDB:database1> copy (select time,region,device_id,temperature from table1 order by time) to 'copysql5.tsfile' (TABLE copytable, TIME time, TAGS (region,device_id)) ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| /iotdb/data/datanode/data/copy_to/copysql5.tsfile| 18| 4| 1199| copytable| time|[region, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +Total line number = 1 +It costs 0.047s +``` diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md b/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md index 507e30003..b88ee0f38 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_apache.md @@ -2,32 +2,38 @@ ## 1. 功能概述 -数据导出工具 `export-data.sh/bat` 位于 `tools` 目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile（开源时间序列文件格式）格式。具体功能如下： +IoTDB 支持两种方式进行数据导出： + +* 数据导出工具 ：`export-data.sh/bat` 位于 `tools `目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile （开源时间序列文件格式）格式。 +* Copy SQL 导出 TsFile：通过 SQL 将查询结果写回到指定路径的 TsFile 中。 - + - - - + + + - - + + - + - + + + + +
文件格式IoTDB工具IoTDB工具 具体介绍
CSVexport-data.sh/bat纯文本格式，存储格式化数据，需按照下文指定 CSV 格式进行构造CSVexport-data.sh/bat纯文本格式，存储格式化数据，需按照下文指定 CSV 格式进行构造
SQL包含自定义 SQL 语句的文件SQL包含自定义 SQL 语句的文件
TsFileTsFile 开源时序数据文件格式
Copy SQL开源时序数据文件格式
- -## 2. 功能详解 +## 2. 数据导出工具 ### 2.1 公共参数 @@ -177,3 +183,131 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. Copy SQL + +> **注意：该功能自 V2.0.11 版本起支持。** + +### 3.1 运行命令 + +```SQL +// ---------------------------------------- Copy Statement --------------------------------------------------------- +copyToStatement + : COPY '(' query ')' TO fileName=string ((WITH)? copyToStatementOptions)? + | COPY tableName=qualifiedName ('(' tableColumns=identifierList ')')? TO fileName=string ((WITH)? copyToStatementOptions)? + ; + +copyToStatementOptions + : '(' copyToStatementOption (',' copyToStatementOption)* ')' + ; + +copyToStatementOption + : FORMAT identifier + | TABLE identifier + | TAGS '(' identifierList ')' + | TIME identifier + | MEMORY_THRESHOLD memory=INTEGER_VALUE + ; +``` + +#### 参数介绍 + +| 名称 | 说明 | 默认值 | +|---|---|---| +| FORMAT | 导出格式，当前仅有 TsFile | TsFile | +| TABLE | 指定生成的 TsFile 中的 Table 名称 | 如果查询 SQL 仅涉及一个 table，使用这个 table 名，否则使用 `default` | +| TIME | 指定使用结果集中的哪一列作为 TIME 列。
手动指定时：列类型非 TIMESTAMP、找不到指定列均会报错；
未手动指定时：若存在列名 time 但类型非 TIMESTAMP 也会报错。
按以下优先级构造 time 列：
1. 查询仅涉及的一个 table 中的 time 列名称相同的列
2. 寻找列名为 "time" 且类型为 TIMESTAMP 的列作为时间列
3. 使用对应 device 写入的当前行数作为 time 生成时间列，列名为 "time" | - | +| TAGS | 指定哪些列为 TAG 列，有多个 TAG 列时，最终生成的 table 内的顺序和指定的顺序一致。
手动指定时：列不存在、列类型非 STRING、存在重复列名均会报错。
如果查询仅涉及一个 table，且 table 所有 tag 列在查询结果集中可以找到，则推断为这个 table 的 tag 列，否则默认值为空列表，即除了 TIME 列以外其余所有列都被视为 FIELD 列 | 空列表 | +| MEMORY_THRESHOLD | 用于在生成 TsFile 时进行内存控制（单位：byte），手动指定数值小于等于 0 时将报错 | 32MB | + +#### 结果集说明 + +| 列名 | 数据类型 | 说明 | +|---|---|---| +| path | STRING | 生成的目标文件的绝对路径 | +| row_count | INT64 | 总写入行数 | +| device_count | INT64 | 生成的设备数量 | +| size_in_bytes | INT64 | 生成的目标文件大小 | +| table_name | STRING | 目标文件中的表名，如果是自动生成的，会通过 `(auto_gen)` 进行标记 | +| time_column | STRING | 目标文件中的表的 time 列名，如果是自动生成的，会通过 `(auto_gen)` 进行标记 | +| tag_columns | STRING | 目标文件中的表的 tag 列名，以 `,` 分隔 | + +#### 其他注意事项 + +* 对于文件生成位置： + * 如果指定的是文件名，生成的 TsFile 保存在客户端直连的 DataNode 的 `${dn_data_dirs}/copy_to` 下，配置多个目录时，按照配置项 `dn_multi_dir_strategy` 的策略生成； + * 如果指定的是一个路径，则放在指定的路径下。 +* 执行过程中可能出现的异常： + * 按照给定的 schema 写入 TsFile，存在乱序时间戳时报错 + * 非法文件名或目标文件已存在时报错 + * 查询结果中存在重复列名 + * 磁盘空间不足 + +### 3.2 运行示例 + +以[示例数据](../Reference/Sample-Data.md)中 table1 为例 + +1. 通过 select 语句将 table1 中的全部数据导出到文件 copysql1.tsfile + +```SQL +IoTDB:database1> copy (select * from table1) to 'copysql1.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql1.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.336s +``` + +2. 通过表名将 table1 中的全部数据导出到文件 copysql2.tsfile + +```SQL +IoTDB:database1> copy table1 to 'copysql2.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +| /iotdb/data/datanode/data/copy_to/copysql2.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+ +Total line number = 1 +It costs 0.048s +``` + +3. 通过表名(列)的方式将 table1 中的部分数据导出到文件 copysql3.tsfile + +```SQL +IoTDB:database1> copy table1 (device_id,temperature) to 'copysql3.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql3.tsfile| 18| 1| 558| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.064s +``` + +4. 通过 select 语句将 table1 中部分数据的聚合结果导出到文件 copysql4.tsfile + +```SQL +IoTDB:database1> copy (select count(temperature), count(humidity) from table1 group by device_id) to 'copysql4.tsfile' ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +| /iotdb/data/datanode/data/copy_to/copysql4.tsfile| 2| 1| 543| table1|time(auto_gen)| []| ++-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+ +Total line number = 1 +It costs 0.155s +``` + +5. 通过 select 语句将 table1 中的部分数据导出到文件 copysql5.tsfile ，并指定目标表、time列及 tag 列 + +```SQL +IoTDB:database1> copy (select time,region,device_id,temperature from table1 order by time) to 'copysql5.tsfile' (TABLE copytable, TIME time, TAGS (region,device_id)) ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +| /iotdb/data/datanode/data/copy_to/copysql5.tsfile| 18| 4| 1199| copytable| time|[region, device_id]| ++-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+ +Total line number = 1 +It costs 0.047s +```