Ingesting Data from Files with Spark, Part 1: CSV

From Spark in Action, 2nd Ed. by Jean Georges Perrin


Save 37% off Spark in Action, 2nd Ed. Just enter code fccperrin into the discount code box at checkout at

This is the first in a series of 4 articles on the topic of ingesting data from files with Spark. This section deals with ingesting data from CSV.

CSV is probably the most popular data-exchange format around. Due to its age and wide use, this format has many variations on its core structure: separators aren’t always commas, some records may span over multiple lines, there are various ways of escaping the separator, and many more creative considerations. When your customer tells you “I’ll send you a CSV file,” you can certainly nod, smile, and slowly start to freak out on the inside.

Fortunately for you, Spark offers a variety of options for ingesting those CSV files. Ingesting CSV is easy and schema inference is a powerful feature.

Let’s have a look at more advanced examples with more options that illustrate the complexity of CSV files in the outside world. You’ll first look at the file you’ll ingest, and understand its specifications. You’ll then have a look at the result and finally build the mini-application to achieve the result. This pattern repeats for each format.

Figure 1 illustrates the process you’re going to implement.

In listing 1 is an excerpt of a CSV file with two records and a header row. Note that CSV has become a generic term: nowadays, the C means more “character” than comma. You’ll find files where values are separated by semicolons, tabs, pipes (|), and more. For the purist, the acronym may matter, but for Spark, they all fall into the same category.

A few observations:

  • The file isn’t comma-separated but semicolon-separated.
  • I manually added the end of paragraph symbol () to show the end of line. It isn’t in the file.
  • If you look at the record with id 4, there’s a semicolon in the title, which breaks the parsing, therefore this field is surrounded by stars. Keep in mind that this is a stretch example to illustrate some of Spark features.
  • If you look at the record with id 6, you’ll see that the title is split over two lines: there’s a carriage return after Language? And before An.

Listing 1 Complex CSV file (abstract of books.csv)

id;authorId;title;releaseDate;link ¶  
4;1;*Harry Potter and the Chamber of Secrets: The Illustrated Edition (Harr
[CA]y Potter; Book 2)*;10/4/16; ¶
6;2;*Development Tools in 2006: any Room for a 4GL-style Language? ¶
An independent study by Jean Georges Perrin, IIUG Board
Member*;12/28/16;ht [CA]tp:// ¶

Desired output

Listing 2 shows a possible output. I added the paragraph mark to illustrate the new line, as long records aren’t easy to read…

Listing 2 Complex CSV file (abstract of books.csv)

Excerpt of the dataframe content:  
title|releaseDate| link|¶

| 4| 1| Harry Potter and the Chamber of Secrets: The Illustrated Edition (Harry Potter; Book 2)| 10/4/16||¶

| 6| 2|Development Tools in 2006: any Room for a 4GL-style Language? ¶ An independent study by...| 12/28/16||¶ //#B

only showing top 7 rows
Dataframe's schema:
|-- id: integer (nullable = true) ❸
|-- authorId: integer (nullable = true) ❸
|-- title: string (nullable = true)
|-- releaseDate: string (nullable = true) ❶
|-- link: string (nullable = true)

❶ See that our release date is seen as a string, not a date!

❷ The line break that was in your CSV file is still here.

❸ The datatype is an integer: in CSV files, everything is a string, but Spark makes an educated guess!


To achieve the result in listing 2, you’ll have to code something similar to listing 3: first get a session, then configure and run the parsing operation in one call using method chaining. Finally, show some records and display the schema of the dataframe.

Listing 3

package net.jgp.books.sparkWithJava.ch07.lab_100.csv_ingestion;     import org.apache.spark.sql.Dataset;  
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
public class ComplexCsvToDataframeApp { public static void main(String[] args) {
ComplexCsvToDataframeApp app = new ComplexCsvToDataframeApp();
private void start() {
SparkSession spark = SparkSession.builder()
.appName("Complex CSV to Dataframe")
Dataset<Row> df ="csv") ❶
.option("header", "true") ❷
.option("multiline", true) ❸
.option("sep", ";") ❹
.option("quote", "*") ❺
.option("dateFormat", "M/d/y") ❻
.option("inferSchema", true) ❼
System.out.println("Excerpt of the dataframe content:");, 90);
System.out.println("Dataframe's schema:");

❶ The format we want to ingest is CSV

❷ The first line of your CSV file is a header line

❸ Some of our records are splitting over multiple lines, note that you can either use a string or a Boolean, making it easier to load values from a configuration file

❹ The separator between values is a semicolon (;)

❺ The quote character is a star (*)

❻ The date format matches the month/day/year format, as commonly used in the United States (see below)

❼ Spark infers (guesses) the schema (see below)

As you probably guessed, you need to know what your file looks like (separator character, escape character, and so on) before you can configure the parser. Spark won’t infer those, this is part of the contract that comes (or, as in most of the times, you have to guess) with your CSV files.

The schema inference feature is a pretty neat one; but, as you can see here, it didn’t infer that the releaseDate column was a date. One way to tell Spark that it’s a date is to specify a schema.

And that’s where we’re going to stop for this section. Stay tuned for part 2. If you’re interested in some more general information about the book, check it out on liveBook here and see this slide deck.

About the author:
An experienced consultant and entrepreneur passionate about all things data, Jean Georges Perrin was the first IBM Champion in France, an honor he’s now held for ten consecutive years. Jean Georges has managed many teams of software and data engineers.

Originally published at



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Manning Publications

Follow Manning Publications on Medium for free content and exclusive discounts.