skip to main content
Welcome to the Progress DataDirect for JDBC for Apache Spark SQL Driver : Data Types
  

Try DataDirect Drivers Now

Data Types

The following table shows how the driver maps Apache Spark SQL data types to standard JDBC data types.
Table 3. Data Types
Apache Spark SQL
JDBC
Bigint
BIGINT
Binary
VARBINARY or LONGVARBINARY1, 2
Boolean
BOOLEAN
Date
DATE
Decimal
DECIMAL
Double
DOUBLE
Float
REAL
Int
INTEGER
Smallint
SMALLINT
String3, 4
VARCHAR or LONGVARCHAR
Timestamp
TIMESTAMP
Tinyint
TINYINT
Varchar5
VARCHAR

1 If the BinaryDescribeType property is set to varbinary (the default), this data type maps to VARBINARY. If set to longvarbinary, this data type maps to LONGVARBINARY.

2 Supported only for Apache Spark SQL 2.0 and higher.

3 Maximum of 2 GB

4 If the StringDescribeType connection property is set to varchar (the default), the String data type maps to VARCHAR. If StringDescribeType is set to longvarchar, String maps to LONGVARCHAR. StringDescribeType affects all columns reported as String, even columns that were originally cast as Varchar. This is important to note because the Spark Thrift server, when returning result metadata for Varchar columns, reports column type as (12) STRING and precision as 2147483647.

5 When returning result set metadata for Varchar columns, the Spark Thrift server reports the column type as (12) STRING and the precision as 2147483647. For the latest information about this issue, refer to the Apache JIRA SPARK-5918 issue Web page.

* getTypeInfo