java spark文件读取乱码问题的解决方法

幸鹤梦

目录

• 一、问题
• 二、心酸历程
• 1、textFile+Configuration方式——乱码
• 2、spark.read().option方式——乱码
• 3、newAPIHadoopFile+Configuration——乱码
• 4、newAPIHadoopFile+自定义类——乱码
• 5、newAPIHadoopRDD+自定义类——乱码
• 三、最终解决

一、问题

环境为jdk1.8，spark3.2.1，读取hadoop中GB18030编码格式的文件出现乱码。

二、心酸历程

为了解决该问题，尝试过很多种方法，但都没有成功

1、textFile+Configuration方式——乱码

1. String filePath = "hdfs:///user/test.deflate";
2. //创建SparkSession和SparkContext的实例
3. String encoding = "GB18030";
5. SparkSession spark = SparkSession.builder()
6. .master("local[*]").appName("Spark Example")
7. .getOrCreate();
9. JavaSparkContext sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
10. Configuration entries = sc.hadoopConfiguration();
11. entries.set("textinputformat.record.delimiter", "\n");
12. entries.set("mapreduce.input.fileinputformat.inputdir",filePath);entries.set("mapreduce.input.fileinputformat.encoding", "GB18030");
14. JavaRDD<String> rdd = sc.textFile(filePath);

复制代码

2、spark.read().option方式——乱码

1. Dataset<Row> load = spark.read().format("text").option("encoding", "GB18030").load(filePath);
3. load.foreach(row -> {
4. System.out.println(row.toString());
5. System.out.println(new String(row.toString().getBytes(encoding),"UTF-8"));
6. System.out.println(new String(row.toString().getBytes(encoding),"GBK"));
9. });

复制代码

3、newAPIHadoopFile+Configuration——乱码

1. JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD = sc.newAPIHadoopFile(filePath, TextInputFormat.class, LongWritable.class, Text.class, entries );
3. System.out.println("longWritableTextJavaPairRDD count ="+longWritableTextJavaPairRDD.count());
4. longWritableTextJavaPairRDD.foreach(k->{
6. System.out.println(k._2);
7. });

复制代码

4、newAPIHadoopFile+自定义类——乱码

1. JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD = sc.newAPIHadoopFile(filePath, GBKInputFormat.class, LongWritable.class, Text.class, entries );
3. System.out.println("longWritableTextJavaPairRDD count ="+longWritableTextJavaPairRDD.count());
4. longWritableTextJavaPairRDD.foreach(k->{
6. System.out.println(k._2);
7. });

复制代码

代码中GBKInputFormat.class是TextInputFormat.class复制将内部UTF-8修改为GB18030所得

5、newAPIHadoopRDD+自定义类——乱码

1. JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD1 = sc.newAPIHadoopRDD(entries, GBKInputFormat.class, LongWritable.class, Text.class);
2. System.out.println("longWritableTextJavaPairRDD count ="+longWritableTextJavaPairRDD1.count());
3. longWritableTextJavaPairRDD1.foreach(k->{
4. System.out.println(k._2());
5. });

复制代码

三、最终解决

上述方法感觉指定的字符编码并没有生效不知道为什么，如有了解原因的还请为我解惑，谢谢

最终解决方案如下

1. JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD = sc.newAPIHadoopFile(filePath, TextInputFormat.class, LongWritable.class, Text.class, new Configuration());
3. System.out.println("longWritableTextJavaPairRDD count ="+longWritableTextJavaPairRDD.count());
4. longWritableTextJavaPairRDD.foreach(k->{
5. System.out.println(new String(k._2.copyBytes(), encoding));
6. });
8. JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD1 = sc.newAPIHadoopRDD(entries, TextInputFormat.class, LongWritable.class, Text.class);
10. System.out.println("longWritableTextJavaPairRDD count ="+longWritableTextJavaPairRDD1.count());
11. longWritableTextJavaPairRDD1.foreach(k->{
12. System.out.println(new String(k._2().copyBytes(),encoding));
13. System.out.println(new String(k._2.copyBytes(),encoding));
14. });

复制代码

主要是new String(k._2().copyBytes(),encoding)得以解决

到此这篇关于java spark文件读取乱码问题的解决方法的文章就介绍到这了,更多相关java spark文件读取乱码内容请搜索晓枫资讯以前的文章或继续浏览下面的相关文章希望大家以后多多支持晓枫资讯！

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！

爱生活爱娘活j

感谢楼主，顶。

金蒙雨

顶顶更健康！！！