org.apache.tephra.hbase.txprune

Class HBaseTransactionPruningPlugin

java.lang.Object

org.apache.tephra.hbase.txprune.HBaseTransactionPruningPlugin

All Implemented Interfaces:

TransactionPruningPlugin

public class HBaseTransactionPruningPlugin
extends Object
implements TransactionPruningPlugin

Default implementation of the TransactionPruningPlugin for HBase. This plugin determines the prune upper bound for transactional HBase tables that use coprocessor TransactionProcessor.

State storage:

This plugin expects the TransactionProcessor to save the prune upper bound for invalid transactions after every major compaction of a region. Let's call this (region, prune upper bound). In addition, the plugin also persists the following information on a run at time t

• (t, set of regions): Set of transactional regions at time t. Transactional regions are regions of the tables that have the coprocessor TransactionProcessor attached to them.
• (t, inactive transaction bound): This is the smallest not in-progress transaction that will not have writes in any HBase regions that are created after time t. This value is determined by the Transaction Service based on the transaction state at time t and passed on to the plugin.

Computing prune upper bound:

In a typical HBase instance, there can be a constant change in the number of regions due to region creations, splits and merges. At any given time there can always be a region on which a major compaction has not been run. Since the prune upper bound will get recorded for a region only after a major compaction, using only the latest set of regions we may not be able to find the prune upper bounds for all the current regions. Hence we persist the set of regions that exist at that time of each run of the plugin, and use historical region set for time t, t - 1, etc. to determine the prune upper bound. From the regions saved at time t, t - 1, etc., the plugin tries to find the latest (t, set of regions) where all regions have been major compacted, i.e, all regions have prune upper bound recorded in (region, prune upper bound).
If such a set is found for time t1, the prune upper bound returned by the plugin is the minimum of

• Prune upper bounds of regions in set (t1, set of regions)
• Inactive transaction bound from (t1, inactive transaction bound)

Above, when we find (t1, set of regions), there may a region that was created after time t1, but has a data write from an invalid transaction that is smaller than the prune upper bounds of all regions in (t1, set of regions). This is possible because (region, prune upper bound) persisted by TransactionProcessor is always the latest prune upper bound for a region.
However a region created after time t1 cannot have writes from an invalid transaction that is smaller than inactive transaction bound at the time the region was created. Since we limit the plugin prune upper bound using (t1, inactive transaction bound), there should be no invalid transactions smaller than the plugin prune upper bound with writes in any transactional region of this HBase instance.

Note: If your tables uses a transactional coprocessor other than TransactionProcessor, then you may need to write a new plugin to compute prune upper bound for those tables.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.hadoop.hbase.client.HConnection	connection
protected DataJanitorState	dataJanitorState
protected org.apache.hadoop.hbase.client.HBaseAdmin	hBaseAdmin
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
HBaseTransactionPruningPlugin()

Method Summary

Modifier and Type	Method and Description
protected void	createPruneTable(org.apache.hadoop.hbase.TableName stateTable) Create the prune state table given the TableName if the table doesn't exist already.
void	destroy() Called once during the shutdown of the Transaction Service.
long	fetchPruneUpperBound(long time, long inactiveTransactionBound) Determines prune upper bound for the data store as mentioned above.
protected SortedSet<byte[]>	getTransactionalRegions()
void	initialize(org.apache.hadoop.conf.Configuration conf) Called once when the Transaction Service starts up.
protected boolean	isTransactionalTable(org.apache.hadoop.hbase.HTableDescriptor tableDescriptor) Returns whether the table is a transactional table.
void	pruneComplete(long time, long maxPrunedInvalid) After invalid list has been pruned, this cleans up state information that is no longer required.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

conf

protected org.apache.hadoop.conf.Configuration conf

hBaseAdmin

protected org.apache.hadoop.hbase.client.HBaseAdmin hBaseAdmin

connection

protected org.apache.hadoop.hbase.client.HConnection connection

dataJanitorState

protected DataJanitorState dataJanitorState

Constructor Detail

HBaseTransactionPruningPlugin

public HBaseTransactionPruningPlugin()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf)
                throws IOException

Description copied from interface: TransactionPruningPlugin

Called once when the Transaction Service starts up.

Specified by:

initialize in interface TransactionPruningPlugin

Parameters:

conf - configuration for the plugin

Throws:

IOException

fetchPruneUpperBound

public long fetchPruneUpperBound(long time,
                                 long inactiveTransactionBound)
                          throws IOException

Determines prune upper bound for the data store as mentioned above.

Specified by:

fetchPruneUpperBound in interface TransactionPruningPlugin

Parameters:

time - start time of this prune iteration in milliseconds

inactiveTransactionBound - the largest invalid transaction that can be possibly removed from the invalid list for the given time. This is an upper bound determined by the Transaction Service, based on its knowledge of in-progress and invalid transactions that may still have active processes and therefore future writes. The plugin will typically return a reduced upper bound based on the state of the invalid transaction data clean up in the data store.

Returns:

prune upper bound for the data store

Throws:

IOException

pruneComplete

public void pruneComplete(long time,
                          long maxPrunedInvalid)
                   throws IOException

After invalid list has been pruned, this cleans up state information that is no longer required. This includes -

• (region, prune upper bound) - prune upper bound for regions that are older than maxPrunedInvalid
• (t, set of regions) - Regions set that were recorded on or before the start time of maxPrunedInvalid
• (t, inactive transaction bound) - Smallest not in-progress transaction without any writes in new regions information recorded on or before the start time of maxPrunedInvalid

Specified by:

pruneComplete in interface TransactionPruningPlugin

Parameters:

time - start time of this prune iteration in milliseconds (same value as passed to TransactionPruningPlugin.fetchPruneUpperBound(long, long) in the same run)

maxPrunedInvalid - the largest invalid transaction that was removed from the invalid list

Throws:

IOException

destroy

public void destroy()

Description copied from interface: TransactionPruningPlugin

Called once during the shutdown of the Transaction Service.

Specified by:

destroy in interface TransactionPruningPlugin

createPruneTable

protected void createPruneTable(org.apache.hadoop.hbase.TableName stateTable)
                         throws IOException

Create the prune state table given the TableName if the table doesn't exist already.

Parameters:

stateTable - prune state table name

Throws:

IOException

isTransactionalTable

protected boolean isTransactionalTable(org.apache.hadoop.hbase.HTableDescriptor tableDescriptor)

Returns whether the table is a transactional table. By default, it is a table is identified as a transactional table if it has a the coprocessor TransactionProcessor attached to it. Should be overriden if the users attach a different coprocessor.

Parameters:

tableDescriptor - HTableDescriptor of the table

Returns:

true if the table is transactional

getTransactionalRegions

protected SortedSet<byte[]> getTransactionalRegions()
                                             throws IOException

Throws:

IOException