Serialized Form (parquet format metadata 2.1.0 API)

Package parquet.format
- Class parquet.format.ColumnChunk extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - file_path
```
String file_path
```
      File where column data is stored. If not set, assumed to be same file as metadata. This path is relative to the current file.
    - file_offset
```
long file_offset
```
      Byte offset in file_path to the ColumnMetaData *
    - meta_data
```
ColumnMetaData meta_data
```
      Column metadata for this chunk. This is the same content as what is at file_path/file_offset. Having it here has it replicated in the file metadata.
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
ColumnChunk._Fields[] optionals
```
- Class parquet.format.ColumnMetaData extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - type
```
Type type
```
      Type of this column *
      
      See Also:
      
      Type
    - encodings
```
List<E> encodings
```
      Set of all encodings used for this column. The purpose is to validate whether we can decode those pages. *
    - path_in_schema
```
List<E> path_in_schema
```
      Path in schema *
    - codec
```
CompressionCodec codec
```
      Compression codec *
      
      See Also:
      
      CompressionCodec
    - num_values
```
long num_values
```
      Number of values in this column *
    - total_uncompressed_size
```
long total_uncompressed_size
```
      total byte size of all uncompressed pages in this column chunk (including the headers) *
    - total_compressed_size
```
long total_compressed_size
```
      total byte size of all compressed pages in this column chunk (including the headers) *
    - key_value_metadata
```
List<E> key_value_metadata
```
      Optional key/value metadata *
    - data_page_offset
```
long data_page_offset
```
      Byte offset from beginning of file to first data page *
    - index_page_offset
```
long index_page_offset
```
      Byte offset from beginning of file to root index page *
    - dictionary_page_offset
```
long dictionary_page_offset
```
      Byte offset from the beginning of file to first (only) dictionary page *
    - statistics
```
Statistics statistics
```
      optional statistics for this column chunk
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
ColumnMetaData._Fields[] optionals
```
- Class parquet.format.DataPageHeader extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - num_values
```
int num_values
```
      Number of values, including NULLs, in this data page. *
    - encoding
```
Encoding encoding
```
      Encoding used for this data page *
      
      See Also:
      
      Encoding
    - definition_level_encoding
```
Encoding definition_level_encoding
```
      Encoding used for definition levels *
      
      See Also:
      
      Encoding
    - repetition_level_encoding
```
Encoding repetition_level_encoding
```
      Encoding used for repetition levels *
      
      See Also:
      
      Encoding
    - statistics
```
Statistics statistics
```
      Optional statistics for the data in this page*
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
DataPageHeader._Fields[] optionals
```
- Class parquet.format.DataPageHeaderV2 extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - num_values
```
int num_values
```
      Number of values, including NULLs, in this data page. *
    - num_nulls
```
int num_nulls
```
      Number of NULL values, in this data page. Number of non-null = num_values - num_nulls which is also the number of values in the data section *
    - num_rows
```
int num_rows
```
      Number of rows in this data page. which means pages change on record boundaries (r = 0) *
    - encoding
```
Encoding encoding
```
      Encoding used for data in this page *
      
      See Also:
      
      Encoding
    - definition_levels_byte_length
```
int definition_levels_byte_length
```
      length of the repetition levels
    - repetition_levels_byte_length
```
int repetition_levels_byte_length
```
      length of the definition levels
    - is_compressed
```
boolean is_compressed
```
      whether the values are compressed. Which means the section of the page between definition_levels_byte_length + repetition_levels_byte_length + 1 and compressed_page_size (included) is compressed with the compression_codec. If missing it is considered compressed
    - statistics
```
Statistics statistics
```
      optional statistics for this column chunk
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
DataPageHeaderV2._Fields[] optionals
```
- Class parquet.format.DictionaryPageHeader extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - num_values
```
int num_values
```
      Number of values in the dictionary *
    - encoding
```
Encoding encoding
```
      Encoding using this dictionary page *
      
      See Also:
      
      Encoding
    - is_sorted
```
boolean is_sorted
```
      If true, the entries in the dictionary are sorted in ascending order *
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
DictionaryPageHeader._Fields[] optionals
```
- Class parquet.format.FileMetaData extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - version
```
int version
```
      Version of this file *
    - schema
```
List<E> schema
```
      Parquet schema for this file. This schema contains metadata for all the columns. The schema is represented as a tree with a single root. The nodes of the tree are flattened to a list by doing a depth-first traversal. The column metadata contains the path in the schema for that column which can be used to map columns to nodes in the schema. The first element is the root *
    - num_rows
```
long num_rows
```
      Number of rows in this file *
    - row_groups
```
List<E> row_groups
```
      Row groups in this file *
    - key_value_metadata
```
List<E> key_value_metadata
```
      Optional key/value metadata *
    - created_by
```
String created_by
```
      String for application that wrote this file. This should be in the format version (build ). e.g. impala version 1.0 (build 6cf94d29b2b7115df4de2c06e2ab4326d721eb55)
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
FileMetaData._Fields[] optionals
```
- Class parquet.format.IndexPageHeader extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
- Class parquet.format.KeyValue extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - key
```
String key
```
    - value
```
String value
```
    - optionals
```
KeyValue._Fields[] optionals
```
- Class parquet.format.PageHeader extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - type
```
PageType type
```
      the type of the page: indicates which of the *_header fields is set *
      
      See Also:
      
      PageType
    - uncompressed_page_size
```
int uncompressed_page_size
```
      Uncompressed page size in bytes (not including this header) *
    - compressed_page_size
```
int compressed_page_size
```
      Compressed page size in bytes (not including this header) *
    - crc
```
int crc
```
      32bit crc for the data below. This allows for disabling checksumming in HDFS if only a few pages needs to be read
    - data_page_header
```
DataPageHeader data_page_header
```
    - index_page_header
```
IndexPageHeader index_page_header
```
    - dictionary_page_header
```
DictionaryPageHeader dictionary_page_header
```
    - data_page_header_v2
```
DataPageHeaderV2 data_page_header_v2
```
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
PageHeader._Fields[] optionals
```
- Class parquet.format.RowGroup extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - columns
```
List<E> columns
```
    - total_byte_size
```
long total_byte_size
```
      Total byte size of all the uncompressed column data in this row group *
    - num_rows
```
long num_rows
```
      Number of rows in this row group *
    - sorting_columns
```
List<E> sorting_columns
```
      If set, specifies a sort ordering of the rows in this RowGroup. The sorting columns can be a subset of all the columns.
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
RowGroup._Fields[] optionals
```
- Class parquet.format.SchemaElement extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - type
```
Type type
```
      Data type for this field. Not set if the current element is a non-leaf node
      
      See Also:
      
      Type
    - type_length
```
int type_length
```
      If type is FIXED_LEN_BYTE_ARRAY, this is the byte length of the vales. Otherwise, if specified, this is the maximum bit length to store any of the values. (e.g. a low cardinality INT col could have this set to 3). Note that this is in the schema, and therefore fixed for the entire file.
    - repetition_type
```
FieldRepetitionType repetition_type
```
      repetition of the field. The root of the schema does not have a repetition_type. All other nodes must have one
      
      See Also:
      
      FieldRepetitionType
    - name
```
String name
```
      Name of the field in the schema
    - num_children
```
int num_children
```
      Nested fields. Since thrift does not support nested fields, the nesting is flattened to a single list by a depth-first traversal. The children count is used to construct the nested relationship. This field is not set when the element is a primitive type
    - converted_type
```
ConvertedType converted_type
```
      When the schema is the result of a conversion from another model Used to record the original type to help with cross conversion.
      
      See Also:
      
      ConvertedType
    - scale
```
int scale
```
      Used when this column contains decimal data. See the DECIMAL converted type for more details.
    - precision
```
int precision
```
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
SchemaElement._Fields[] optionals
```
- Class parquet.format.SortingColumn extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - column_idx
```
int column_idx
```
      The column index (in this row group) *
    - descending
```
boolean descending
```
      If true, indicates this column is sorted in descending order. *
    - nulls_first
```
boolean nulls_first
```
      If true, nulls will come before non-null values, otherwise, nulls go at the end.
    - __isset_bitfield
```
byte __isset_bitfield
```
- Class parquet.format.Statistics extends Object implements Serializable
  - Serialization Methods
    - readObject
```
private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException
```
      Throws:
      
      IOException
      
      ClassNotFoundException
    - writeObject
```
private void writeObject(ObjectOutputStream out)
                  throws IOException
```
      Throws:
      
      IOException
  - Serialized Fields
    - max
```
ByteBuffer max
```
      min and max value of the column, encoded in PLAIN encoding
    - min
```
ByteBuffer min
```
    - null_count
```
long null_count
```
      count of null value in the column
    - distinct_count
```
long distinct_count
```
      count of distinct values occurring
    - __isset_bitfield
```
byte __isset_bitfield
```
    - optionals
```
Statistics._Fields[] optionals
```

Serialized Form

Package parquet.format

Class parquet.format.ColumnChunk extends Object implements Serializable

Serialization Methods

readObject

writeObject

Serialized Fields

file_path

file_offset

meta_data

__isset_bitfield

optionals

Class parquet.format.ColumnMetaData extends Object implements Serializable

Serialization Methods

readObject

writeObject

Serialized Fields

type

encodings

path_in_schema

codec

num_values

total_uncompressed_size

total_compressed_size

key_value_metadata

data_page_offset

index_page_offset

dictionary_page_offset

statistics

__isset_bitfield

optionals

Class parquet.format.DataPageHeader extends Object implements Serializable

Serialization Methods

readObject

writeObject

Serialized Fields

num_values

encoding

definition_level_encoding

repetition_level_encoding

statistics

__isset_bitfield

optionals

Class parquet.format.DataPageHeaderV2 extends Object implements Serializable

Serialization Methods

readObject

writeObject

Serialized Fields

num_values

num_nulls

num_rows

encoding

definition_levels_byte_length

repetition_levels_byte_length

is_compressed

statistics

__isset_bitfield

optionals

Class parquet.format.DictionaryPageHeader extends Object implements Serializable

Serialization Methods

readObject

writeObject

Serialized Fields

num_values

encoding

is_sorted

__isset_bitfield

optionals

Class parquet.format.FileMetaData extends Object implements Serializable

Serialization Methods

readObject

writeObject

Serialized Fields

version

schema

num_rows

row_groups

key_value_metadata

created_by

__isset_bitfield