Spring Batch Tutorial 5: Log skipped items in a DataBase.

juin 16, 2011

Log skipped items in a DataBase.

This article is based on the previous article What happends if a process throws an exception ?

The previous article explains how to skip an item in case of error.

A batch is executed without any human interaction.  Due to the nature of a batch program, we must be able to know what are the processed items and also what are items in errors.  In case of errors, we should also know what was the exception.   Like that, we can fix the problem and relaunch a batch process for the skipped items.

The objective of this article is to log all the errors in a table:  Source code

Spring batch provides some  listeners:

  • StepExecutionListener: Listener interface for the lifecycle of a Step.
  • ChunkListener: Listener interface for the lifecycle of a chunk.
  • JobExecutionListener: Provide callbacks at specific points in the lifecycle of a Job.
  • SkipListener: Interface for listener to skipped items.

If we want to keep a track of all skipped items.  The SkipListener is a perfect listener to do that ;)

The SampleSkipListener will be called each time an items is skipped.  But unfortunetly, the listener does not know the  context of the job’s execution.  We have to get this informations by using another Listener: StepExecutionListener.

This Listener is called when the step starts.  At this moment, we can store some informations inside the StepExecutionContext.  This information will be injected into SampleSkipListener via the Spring configuration.

Create a TABLE

Each row contains the skipped item id, the jobExecutionId, the stepExecutionId and a jobName.

CREATE TABLE SKIP_ITEMS (
    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    jobExecutionId INT,
    stepExecutionId INT,
    jobName VARCHAR(100),
    type VARCHAR(100),
    item VARCHAR(100),
    msg VARCHAR(1000),
    runId INT
)

Create the class Model

This class represents a SKIP_ITEMS

package com.alaincieslik.springbatch.article4.model;

public class SkipItems {
    private Long id;
    private String type;
    private String item;
    private String msg;
    private Long runId;
    private Long jobExecutionId;
    private Long stepExecutionId;

	public Long getId() {
		return id;
	}
	public void setId(Long id) {
		this.id = id;
	}
	public String getType() {
		return type;
	}
	public void setType(String type) {
		this.type = type;
	}
	public String getItem() {
		return item;
	}
	public void setItem(String item) {
		this.item = item;
	}
	public String getMsg() {
		return msg;
	}
	public void setMsg(String msg) {
		this.msg = msg;
	}
	public Long getRunId() {
		return runId;
	}
	public void setRunId(Long runId) {
		this.runId = runId;
	}
	public Long getJobExecutionId() {
		return jobExecutionId;
	}
	public void setJobExecutionId(Long jobExecutionId) {
		this.jobExecutionId = jobExecutionId;
	}
	public Long getStepExecutionId() {
		return stepExecutionId;
	}
	public void setStepExecutionId(Long stepExecutionId) {
		this.stepExecutionId = stepExecutionId;
	}

}

Create a DAO for the SkipItems

The interface of our DAO

package com.alaincieslik.springbatch.article4.dao;

import com.alaincieslik.springbatch.article4.model.SkipItems;

public interface SkipItemsDao {

	public void save(SkipItems skipElement);

}

Now the implementation of our DAO.

package com.alaincieslik.springbatch.article4.dao;

import com.alaincieslik.springbatch.article4.model.SkipItems;

public class SkipItemsDaoImpl extends BaseDao implements SkipItemsDao{

	private String insertStatement="INSERT INTO SKIP_ITEMS (type, item, msg,runId,jobExecutionId,stepExecutionId ) VALUES(?,?,?,?,?,?)";

	public void save(SkipItems skipElement){
		simpleJdbcTemplate.update(insertStatement, new Object[]{skipElement.getType(), skipElement.getItem(), skipElement.getMsg(), skipElement.getRunId(), skipElement.getJobExecutionId(), skipElement.getStepExecutionId()});
	}

}

Create a StepExecutionListener

The stepExecutionListener retrieve some informations and save it into the StepExecutionContext.
This stepExecutionContext can be used via the Spring Configuration when the scope is step.

package com.alaincieslik.springbatch.article4;

import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.StepExecutionListener;

public class SampleStepExecutionListener implements StepExecutionListener {

    public void beforeStep(StepExecution stepExecution){
    	Long jobExecutionId =stepExecution.getJobExecutionId();
    	String jobName=stepExecution.getJobExecution().getJobInstance().getJobName();
    	stepExecution.getExecutionContext().put("jobExecutionId", jobExecutionId);
    	stepExecution.getExecutionContext().put("stepExecutionId", stepExecution.getId());
    	stepExecution.getExecutionContext().put("jobName", jobName);
    }

    public ExitStatus afterStep(StepExecution stepExecution){
    	return null;
    }

}

Create a SkipListener

The SkipListener is called each time an item is skipped.  An item can be skip in 3 phases : Read, Process, Write

package com.alaincieslik.springbatch.article4;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.batch.core.SkipListener;

import com.alaincieslik.springbatch.article4.dao.SkipItemsDao;
import com.alaincieslik.springbatch.article4.model.SkipItems;

public class SampleSkipListener extends BaseElement implements SkipListener {

	public final static Log log = LogFactory.getLog(SampleSkipListener.class);
	private SkipItemsDao skipItemsDao;

	public void onSkipInProcess(Object item, Throwable t) {
		log.info("onSkipInProcess:" +item.getClass()+" "+t.getClass());
		if(t instanceof SampleSkipException){
			SampleSkipException sampleSkipException=(SampleSkipException)t;
			skipItemsDao.save(createSkipElement("PROCESS", item.toString(), t.getClass().toString(), sampleSkipException));
		}
	}

	public void onSkipInRead(Throwable t) {
		log.info("onSkipInRead:"+ t.getClass());
		if(t instanceof SampleSkipException){
			SampleSkipException sampleSkipException=(SampleSkipException)t;
			skipItemsDao.save(createSkipElement("READ", "", t.getClass().toString(), sampleSkipException));
		}
	}

	public void onSkipInWrite(Object item, Throwable t) {
		log.info("onSkipInWrite:" +item.getClass()+" "+t.getClass());
		if(t instanceof SampleSkipException){
			SampleSkipException sampleSkipException=(SampleSkipException)t;
			skipItemsDao.save(createSkipElement("WRITE", item.toString(), t.getClass().toString(), sampleSkipException));
		}
	}

	private SkipItems createSkipElement(String type, String item, String msg, SampleSkipException sampleSkipException){
		SkipItems skipElement=new SkipItems();
		skipElement.setType(type);
		skipElement.setItem(item);
		skipElement.setMsg(msg);
		skipElement.setJobExecutionId(getJobExecutionId());
		skipElement.setStepExecutionId(getStepExecutionId());
		return skipElement;
	}
	public SkipItemsDao getSkipItemsDao() {
		return skipItemsDao;
	}

	public void setSkipItemsDao(SkipItemsDao skipItemsDao) {
		this.skipItemsDao = skipItemsDao;
	}
}

BaseElement

The BaseElement is used to store some general informations about the execution.

package com.alaincieslik.springbatch.article4;

public abstract class BaseElement {

	protected Long jobExecutionId;
	protected Long stepExecutionId;
	protected String jobName;

	public String getJobName() {
		return jobName;
	}
	public void setJobName(String jobName) {
		this.jobName = jobName;
	}
	public Long getStepExecutionId() {
		return stepExecutionId;
	}
	public void setStepExecutionId(Long stepExecutionId) {
		this.stepExecutionId = stepExecutionId;
	}
	public Long getJobExecutionId() {
		return jobExecutionId;
	}
	public void setJobExecutionId(Long jobExecutionId) {
		this.jobExecutionId = jobExecutionId;
	}

}

Spring Configuration

In this spring configuration, skipListenerArticle4 bean has a scope=step and we inject jobExecutionId,stepExecutionId, jobName. These informations come from the SampleStepExecutionListener


	




	
 	

	

	

	


Spring Batch configuration

	
		
			
				
				
					
					
				
			
		
	

Issues

  • What happends at the transaction level when an item is skipped?
  • How can we monitor a batch execution?

Links

Package org.springframework.batch.core

8 Responses to “Spring Batch Tutorial 5: Log skipped items in a DataBase.”

  1. excellent!!
    Thanks for the detailed explanations:-)..
    This exactly what I am looking for..

  2. Your atrlcie perfectly shows what I needed to know, thanks!

  3. That addresses several of my concerns atculaly.

  4. 3 comments only? :) weird. thx!

  5. This definitely makes perfect sense to anyone!!!

  6. Your site is really great. Keep going that way.

  7. Hello! Just want to say thank you for this interesting article! =) Peace, Joy.

  8. Amazing article, thanks, I will visit again soon.