package io
IO codecs used for compression. See org.apache.spark.io.CompressionCodec.
- Source
- package.scala
Type Members
-
trait
CompressionCodec extends AnyRef
CompressionCodec allows the customization of choosing different compression implementations to be used in block storage.
CompressionCodec allows the customization of choosing different compression implementations to be used in block storage.
- Annotations
- @DeveloperApi()
- Note
The wire protocol for a codec is not guaranteed compatible across versions of Spark. This is intended for use as an internal compression utility within a single Spark application.
-
class
LZ4CompressionCodec extends CompressionCodec
LZ4 implementation of org.apache.spark.io.CompressionCodec.
LZ4 implementation of org.apache.spark.io.CompressionCodec. Block size can be configured by
spark.io.compression.lz4.blockSize
.- Annotations
- @DeveloperApi()
- Note
The wire protocol for this codec is not guaranteed to be compatible across versions of Spark. This is intended for use as an internal compression utility within a single Spark application.
-
class
LZFCompressionCodec extends CompressionCodec
LZF implementation of org.apache.spark.io.CompressionCodec.
LZF implementation of org.apache.spark.io.CompressionCodec.
- Annotations
- @DeveloperApi()
- Note
The wire protocol for this codec is not guaranteed to be compatible across versions of Spark. This is intended for use as an internal compression utility within a single Spark application.
- final class NioBufferedFileInputStream extends InputStream
- class ReadAheadInputStream extends InputStream
-
class
SnappyCompressionCodec extends CompressionCodec
Snappy implementation of org.apache.spark.io.CompressionCodec.
Snappy implementation of org.apache.spark.io.CompressionCodec. Block size can be configured by
spark.io.compression.snappy.blockSize
.- Annotations
- @DeveloperApi()
- Note
The wire protocol for this codec is not guaranteed to be compatible across versions of Spark. This is intended for use as an internal compression utility within a single Spark application.
-
class
ZStdCompressionCodec extends CompressionCodec
ZStandard implementation of org.apache.spark.io.CompressionCodec.
ZStandard implementation of org.apache.spark.io.CompressionCodec. For more details see - http://facebook.github.io/zstd/
- Annotations
- @DeveloperApi()
- Note
The wire protocol for this codec is not guaranteed to be compatible across versions of Spark. This is intended for use as an internal compression utility within a single Spark application.