Pacific-Design.com

    
Home Index

1. Apache Spark

2. CSV To Parquet

Apache Spark / CSV To Parquet /

CSV to Parquet

        SparkConf conf = new SparkConf().setAppName("Generic SQL Component");
        JavaSparkContext sc = new JavaSparkContext(conf);

        SQLContext sqlContext = new SQLContext(sc);
        DataFrame df = sqlContext.read()
                .format("com.databricks.spark.csv")
                .option("inferSchema", "true")
                .option("header", "true")
                .load("/tmp/datafile.csv");

        df.show();