General Performance Tuning Guidelines for Oracle RAC.

1) Rule of thumb - It is safe to assume that an application which
scales well in a single instance environment would scale well in
a RAC environment. Opposite of that is also true, If an application
has problems with scalability in a single instance environment,
Then it may not scale well in a RAC environment.

2) Speed up instance recovery. 

i) To speed up the instance recovery time use FAST_START_MTTR_TARGET.

 In a single instance environment instance start-up + crash recovery time
 is controlled by this parameter.
 Setting a value for this parameter would result in a more aggressive
 incremental checkpointing, which in turn would incur higher I/O.

 The estimates for the instance recovery in RAC can found in
 the V$INSTANCE_RECOVERY view through the ESTD_CLUSTER_AVAILABLE_TIME column.

ii) Use RECOVERY_PARALLISM to parallelize instance recovery.
iii) Set PARALLEL_MIN_SERVERS CPU_COUNT-1. This parameter specifies
     the minimum number of parallel execution processes running for the instance.
iv) Speed up transaction recovery after system failure by setting FAST_START_PARALLEL_ROLLBACK.
v) Use asynchronous I/O instead of synchronous I/O.
vi) Instance recovery uses 50 % of the buffer cache for recovery buffers. Make
    sure buffer cache is sufficiently sized.

3) Object level tuning.

i) Avoid full table scans - Full table scans results in higher Global Cache Requests.
   That is because if the requested blocks are not found in the local cache, then
   an attempt is made to find them in another cache.

ii) Use Automatic Segment Space Management - ASSM helps applications with
    high insert intensive nature to avoid issues previously encountered and related
    to free lists. It also provides instance level affinity to table blocks.

iii) Use sequences and increase their cache size. Increase in sequence cache size
     improves instance affinity to index keys deriving their values from sequences.
     If application logic permits use "cache noorder".

iv) Use Range or list partitioning, which results in the workload being directed
    to modify a particular range of values from a particular instance.
v) Use hash partitioning to reduce buffer busy contention.
vi) Reduce excessive parsing in library cache and row cache operations to
    avoid additional interconnect traffic.
vii) Use of tables instead of sequences to generate unique numbers is not
     recommended because it may cause severe contention.
viii) Make sure that index selectivity is good. Unselective indexes degrade
      query performance and also cause inter instance contention.

4) Network Tuning - Make sure that a private network is used for the interconnect. The private network
   link should be in full duplex mode. Also, configure the MTU (Maximum transmission Unit)
   to 9000 or 9k.

5) Reduce Index Block Contention - 
i) Use Global index hash partitioning
ii) If sequences are used, Then increase sequence cache size.
iii) Use Natural keys as opposed to surrogate keys - 
     
Surrogate keys are keys that have no "business" meaning and are used only
for identifying table records.
Example: Keys generated using sequences or via a table in the schema.

Natural keys - Keys which are used by the application or people are natural keys.

Example - Invoice-Numbers, Tax-Ids etc.

iv) Use reverse key indexes.

6) Other Considerations - 

i) You may see High Water Mark contention in applications where data insertion
   is the prevalent feature.
   That is because if data insertion is at a higher rate, new blocks may have to
   be maid available after unfruitful searches for free space. All of this happens
   while holding high water mark enqueue.

So, common wait events for this scenario are as following: -
 a) High percentage of enq: HW - contention
 b) High percentage of gc current  grant events.

Common solution to these problems is to define uniform and large
extent sizes.

ii) Truncating different tables from different instances in RAC does not scale
    well as it requires excessive cross instance calls.