To configure a Spring Batch project we will start with the dependencies that we need to use Maven, then we will define the basic configuration and some simple aspects to connect to an embedded database.
What is Spring Batch?
Spring Batch is a framework for process large batches of information in “batch mode”. The execution of batch processes is focused on solving the processing without user intervention and periodically. But to explain it better, we will say that with Spring Batch we can generate jobs (work) and divide them into steps (steps), such as reading data from a database to process them by applying some business rule, and then writing them to a file or back in the database.
We see that a Job can have several Steps and each Step a Reader, a Processor, and a Writer.
What will you do in this example and understand how Spring Batch works?
- Connect to a Database
- Read all the records of a Table
- Process each record and generate new information
- Write the result of the process in another Table of the database.
Let’s see the dependencies you need.
Dependencies needed for Spring Batch
For your project in Spring Batch you need the dependency “org.springframework.boot: spring-boot-starter-batch”. Also, for this example in which we will use a database in memory, we need the dependency of “com.h2database: h2” You will also need “org.springframework.boot: spring-boot-starter-data-jpa” to connect Spring Batch to the database.
These are all the dependencies you will use.
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>
</dependencies>
General structure of your Spring Batch project
Your Spring Batch project will look like this for this example.
What will you do with this tutorial to create a Spring Batch application?
** What we will do in this project example will be to create a Job that reads from a table containing credit cards, then when processing them determine the risk of the card based on an alleged payment date, and at the end save another entity with the result.**
What will you do in this example?
- Read credit cards from the database
- Process the cards and apply a business rule for those cards: calculate the risk of the card according to the payment date.
- The process must generate a new entity with the result of the risk
- In the end, the process result is saved
The Entities
Define two entities, one for the CreditCard and one for CreditCardRisk
package dev.experto.demo.domain;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import java.util.Date;
@Entity
public class CreditCard {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private Long cardNumber;
private Date lastPay;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Long getCardNumber() {
return cardNumber;
}
public void setCardNumber(Long cardNumber) {
this.cardNumber = cardNumber;
}
public Date getLastPay() {
return lastPay;
}
public void setLastPay(Date lastPay) {
this.lastPay = lastPay;
}
@Override
public String toString() {
return "CreditCard{" +
"id=" + id +
", cardNumber=" + cardNumber +
", lastPay=" + lastPay +
'}';
}
}
package dev.experto.demo.domain;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.OneToOne;
import java.util.Date;
@Entity
public class CreditCardRisk {
public static final int HIGH = 3;
public static final int LOW = 2;
public static final int NORMAL = 1;
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private Date date;
private int risk;
@OneToOne(optional = false)
private CreditCard creditCard;
public CreditCardRisk() {
}
public CreditCardRisk(CreditCard creditCard, Date date, int risk) {
this.creditCard = creditCard;
this.date = date;
this.risk = risk;
}
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Date getDate() {
return date;
}
public void setDate(Date date) {
this.date = date;
}
public int getRisk() {
return risk;
}
public void setRisk(int risk) {
this.risk = risk;
}
public CreditCard getCreditCard() {
return creditCard;
}
public void setCreditCard(CreditCard creditCard) {
this.creditCard = creditCard;
}
@Override
public String toString() {
return "CreditCardRisk{" +
"id=" + id +
", date=" + date +
", risk=" + risk +
'}';
}
}
The Repository
You will have two repositories one for each entity that we will use later in the ItemReader and ItemWriter
package dev.experto.demo.repository;
import dev.experto.demo.domain.CreditCard;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface CreditCardRepository extends JpaRepository<CreditCard, Long> {
}
package dev.experto.demo.repository;
import dev.experto.demo.domain.CreditCardRisk;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface CreditCardRiskRespository extends JpaRepository<CreditCardRisk, Long> {
}
The Reader
There are many implementations for Spring Batch Reader available to read different data sources (eg files, database).
Here you will create your own, from the main interface ItemReader. We define the reader that will implement ItemReader and that will use a Spring repository to obtain all CreditCard With the annotation BeforeStep, we perform a read operation on the database before starting the Reader. The _read () _ method will deliver each item in the list to the Processor.
import dev.experto.demo.domain.CreditCard;
import dev.experto.demo.repository.CreditCardRepository;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.annotation.BeforeStep;
import org.springframework.batch.item.ItemReader;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.Iterator;
public class CreditCardItemReader implements ItemReader<CreditCard> {
@Autowired
private CreditCardRepository respository;
private Iterator<CreditCard> usersIterator;
@BeforeStep
public void before(StepExecution stepExecution) {
usersIterator = respository.findAll().iterator();
}
@Override
public CreditCard read() {
if (usersIterator != null && usersIterator.hasNext()) {
return usersIterator.next();
} else {
return null;
}
}
}
The Processor
The Processor will be responsible for receiving a CreditCard and delivering a CreditCardRisk
import dev.experto.demo.domain.CreditCard;
import dev.experto.demo.domain.CreditCardRisk;
import org.springframework.batch.item.ItemProcessor;
import java.time.LocalDate;
import java.time.ZoneId;
import java.util.Date;
import static java.time.temporal.ChronoUnit.DAYS;
public class CreditCardItemProcessor implements ItemProcessor<CreditCard, CreditCardRisk> {
@Override
public CreditCardRisk process(CreditCard item) {
LocalDate today = LocalDate.now();
LocalDate lastDate = item.getLastPay().toInstant().atZone(ZoneId.systemDefault()).toLocalDate();
long daysBetween = DAYS.between(today, lastDate);
int risk;
if (daysBetween >= 20) {
risk = CreditCardRisk.HIGH;
} else if (daysBetween > 10) {
risk = CreditCardRisk.LOW;;
}else {
risk = CreditCardRisk.NORMAL;;
}
CreditCardRisk creditCardRisk = new CreditCardRisk(item, new Date(), risk);
return creditCardRisk;
}
}
The Writer
The Writer also has many implementations in Spring Batch that you can use depending on what or where you are writing.
As we did for the Reader, we will create our own Writer from the interface. The Writer will receive the list of CreditCardRisk that the Processor has processed to save the result in the database.
import dev.experto.demo.domain.CreditCardRisk;
import dev.experto.demo.repository.CreditCardRiskRespository;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.List;
public class CreditCardItemWriter implements ItemWriter<CreditCardRisk> {
@Autowired
private CreditCardRiskRespository respository;
@Override
public void write(List<? extends CreditCardRisk> list) throws Exception {
respository.saveAll(list);
}
}
The Listeners
The Listeners intercept parts of the Job process and receive or listen to what is happening during the execution. Spring Batch has numerous listeners for each part of the job step and you can use them to find out what happens at each stage.
For example, the following JobExecutionListener interface listens to the execution before starting the job and after finishing the Job.
public interface JobExecutionListener {
void beforeJob(JobExecution var1);
void afterJob(JobExecution var1);
}
Then implement the most common listeners to understand them.
package dev.experto.demo.listener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionListener;
import org.springframework.stereotype.Component;
@Component
public class CreditCardJobExecutionListener implements JobExecutionListener {
private static final Logger LOGGER = LoggerFactory.getLogger(CreditCardJobExecutionListener.class);
@Override
public void beforeJob(JobExecution jobExecution) {
LOGGER.info("beforeJob");
}
@Override
public void afterJob(JobExecution jobExecution) {
LOGGER.info("afterJob: " + jobExecution.getStatus());
}
}
package dev.experto.demo.listener;
import dev.experto.demo.domain.CreditCard;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.ItemReadListener;
public class CreditCardIItemReaderListener implements ItemReadListener<CreditCard> {
private static final Logger LOGGER = LoggerFactory.getLogger(CreditCardIItemReaderListener.class);
@Override
public void beforeRead() {
LOGGER.info("beforeRead");
}
@Override
public void afterRead(CreditCard creditCard) {
LOGGER.info("afterRead: " + creditCard.toString());
}
@Override
public void onReadError(Exception e) {
LOGGER.info("onReadError");
}
}
package dev.experto.demo.listener;
import dev.experto.demo.domain.CreditCard;
import dev.experto.demo.domain.CreditCardRisk;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.ItemProcessListener;
public class CreditCardItemProcessListener implements ItemProcessListener<CreditCard, CreditCardRisk> {
private static final Logger LOGGER = LoggerFactory.getLogger(CreditCardItemProcessListener.class);
@Override
public void beforeProcess(CreditCard creditCard) {
LOGGER.info("beforeProcess");
}
@Override
public void afterProcess(CreditCard creditCard, CreditCardRisk creditCardRisk) {
LOGGER.info("afterProcess: " + creditCard + " ---> " + creditCardRisk);
}
@Override
public void onProcessError(CreditCard creditCard, Exception e) {
LOGGER.info("onProcessError");
}
}
package dev.experto.demo.listener;
import dev.experto.demo.domain.CreditCardRisk;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.ItemWriteListener;
import java.util.List;
public class CreditCardIItemWriterListener implements ItemWriteListener<CreditCardRisk> {
private static final Logger LOGGER = LoggerFactory.getLogger(CreditCardIItemWriterListener.class);
@Override
public void beforeWrite(List<? extends CreditCardRisk> list) {
LOGGER.info("beforeWrite");
}
@Override
public void afterWrite(List<? extends CreditCardRisk> list) {
for (CreditCardRisk creditCardRisk : list) {
LOGGER.info("afterWrite :" + creditCardRisk.toString());
}
}
@Override
public void onWriteError(Exception e, List<? extends CreditCardRisk> list) {
LOGGER.info("onWriteError");
}
}
The application.properties file
You are going to configure the database connection here. This project uses an H2 database in memory to simplify the execution of this example.
spring.datasource.url=jdbc:h2:mem:testdb;DB_CLOSE_ON_EXIT=FALSE
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
# create, create-drop, validate, update
spring.jpa.hibernate.ddl-auto = update
The file data.sql
Spring searches by default for files with sql extension inside the resources folder and executes them. This file will then complete the H2 database in memory with information prior to starting the application. In this way, we have data to process in previous memory when starting the Job.
-- risk HIGH
INSERT INTO credit_card (card_number,last_pay) VALUES (9991,CURRENT_DATE()-30);
INSERT INTO credit_card (card_number,last_pay) VALUES (9992,CURRENT_DATE()-21);
-- risk LOW
INSERT INTO credit_card (card_number,last_pay) VALUES (9993,CURRENT_DATE()-20);
INSERT INTO credit_card (card_number,last_pay) VALUES (9994,CURRENT_DATE()-11);
-- risk NORMAL
INSERT INTO credit_card (card_number,last_pay) VALUES (9995,CURRENT_DATE()-10);
INSERT INTO credit_card (card_number,last_pay) VALUES (9996,CURRENT_DATE()-5);
What is a chunk in Spring Batch?
This is a concept that you will use when configuring the Job. A chunk is a processing unit. With this value, you tell Spring Batch to process a certain number of records and when completing the amount send everything to the reader to commit. Note that the class the Item reader interface has the read method that receives a list. This list is the amount of chunk that was previously processed and whose amount you set in the chunk config.
How to set up the Job in Spring Batch
We said before that for Spring Batch a Job contains Steps and each Step generally requires (not always) a Reader, a Processor, and a Writer. Optionally you can add Listener to listen and know what is happening in each part of the batch process.
- The Reader reads data.
- The Processor receives the Reader data and processes it and then delivers it to the Writer.
- The Writer receives the data that was processed and is responsible for saving it.
- Listeners who “listen” to what happens during the process. For this example, we will use some listener, the most common, which we said previously.
Next you will create a configuration class to define all the parts of the Job.
You write this class with @Configuration and @EnableBatchProcessing
Within the class you will call JobBatchConfiguration you must define:
- A Spring bean for the Reader CreditCardItemReader
- A Spring bean for the Processor CreditCardItemProcessor
- A Spring bean for the Writer CreditCardItemWriter
- The beans for the Listeners CreditCardJobExecutionListener , CreditCardItemReaderListener, CreditCardItemProcessListener, CreditCardItemWriterListener
Pay attention to how you define the Job and the Step that will be executed within the Job. To the Job, we indicate the listener that will listen and we establish the step (s). In the definition of the Step, we also indicate the listeners and each of the beans that we mentioned for the reader, processor, and writer. The “chunk” we arbitrarily set it to 100. This value you must think according to the need of your project.
package dev.experto.demo.config;
import dev.experto.demo.domain.CreditCard;
import dev.experto.demo.domain.CreditCardRisk;
import dev.experto.demo.job.CreditCardItemProcessor;
import dev.experto.demo.job.CreditCardItemReader;
import dev.experto.demo.job.CreditCardItemWriter;
import dev.experto.demo.listener.CreditCardItemProcessListener;
import dev.experto.demo.listener.CreditCardIItemReaderListener;
import dev.experto.demo.listener.CreditCardIItemWriterListener;
import dev.experto.demo.listener.CreditCardJobExecutionListener;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.step.tasklet.TaskletStep;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@EnableBatchProcessing
public class JobBatchConfiguration {
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
@Bean
public CreditCardItemReader reader() {
return new CreditCardItemReader();
}
@Bean
public CreditCardItemProcessor processor() {
return new CreditCardItemProcessor();
}
@Bean
public CreditCardItemWriter writer() {
return new CreditCardItemWriter();
}
@Bean
public CreditCardJobExecutionListener jobExecutionListener() {
return new CreditCardJobExecutionListener();
}
@Bean
public CreditCardIItemReaderListener readerListener() {
return new CreditCardIItemReaderListener();
}
@Bean
public CreditCardItemProcessListener creditCardItemProcessListener() {
return new CreditCardItemProcessListener();
}
@Bean
public CreditCardIItemWriterListener writerListener() {
return new CreditCardIItemWriterListener();
}
@Bean
public Job job(Step step, CreditCardJobExecutionListener jobExecutionListener) {
Job job = jobBuilderFactory.get("job1")
.listener(jobExecutionListener)
.flow(step)
.end()
.build();
return job;
}
@Bean
public Step step(CreditCardItemReader reader,
CreditCardItemWriter writer,
CreditCardItemProcessor processor,
CreditCardIItemReaderListener readerListener,
CreditCardItemProcessListener creditCardItemProcessListener,
CreditCardIItemWriterListener writerListener) {
TaskletStep step = stepBuilderFactory.get("step1")
.<CreditCard, CreditCardRisk>chunk(100)
.reader(reader)
.processor(processor)
.writer(writer)
.listener(readerListener)
.listener(creditCardItemProcessListener)
.listener(writerListener)
.build();
return step;
}
}
How to run Spring Batch
This project is create with Maven.
From a Linux console inside the project folder.
$ ./mvnw spring-boot:run
From a windows console inside the project folder.
> mvnw spring-boot:run
Look in more detail at the log output.
- The Job starts with the associated Step and the listener.
- The reader and the listener of the reader are executed.
- The processor and the processor listener are executed.
- The writer and the listener of the writer are executed.
Conclusion:
With the help of this example, you have created a Spring Batch application using some of its most important features. If you need to create batch processes that execute unattended and high-load actions, this is an excellent framework to cover this objective.
Source code of this example
As always I leave you the source code of this example so that you have it at hand.
https://github.com/gustavopeiretti/springbatch-example
https://gitlab.com/gustavopeiretti/springbatch-example