DB Startup Founder

Great MongoDb presentation from Shutterfly

2012-02-12T09:45:00.001-08:00

Talks real numbers and real performance techniques!

http://www.10gen.com/presentations/mongosv-2011/performance-tuning-and-scalability

Tidbit:DB2: identify packages in a bind file

2011-03-04T02:31:00.001-08:00

Use the db2bfd command to determine the names of the packages contained in a bind file:

db2bfd -b db2look.bnd

tidbit:Oracle:identify when row was changed

2011-03-04T02:30:00.000-08:00

Great way to identify when table row was changed (within 3 second precision):

select scn_to_timestamp(ORA_ROWSCN ) from EMP WHERE ROWNUM < 10;

tidbit:Oracle:ITL Deadlock troubleshooting and analysis

2011-03-04T02:27:00.000-08:00

How to identify ITL deadlock:

1. Pick slot number (0x40021) at the top

2. Search for it in a deadlock trace file

3. See if wait event is ITL related.

Resolution: Increase INI_TRANS on table or on index.

---------Blocker(s)-------- ---------Waiter(s)---------

Resource Name process session holds waits process session holds waits

TX-00040021-00001409 66 2515 X 65 1253 S

TX-0005001b-000014d8 65 1253 X 66 2515 S

Dumping one waiter:

inst: 1, sid: 2513, ser: 41

wait event: 'enq: TX - allocate ITL entry'

p1: 'name|mode'=0x54580004

p2: 'usn<<16 | slot'=0x40021

p3: 'sequence'=0x1409

row_wait_obj#: 4294967295, block#: 0, row#: 0, file# 0

min_blocked_time: 0 secs, waiter_cache_ver: 31817

Books:Book about Oracle Founder Larry Ellison

2011-02-20T12:45:00.000-08:00

This book absolutely must read! If you are DBA or start up founder - this book opens really shocking insight how business was and is made in Silicon Valley.

Have you knew that Larry Ellison was not employee #1 of Oracle?

Have you knew that Oracle put on IPO papers that it has mainframe solution while it was not even functional?

Have you knew how exectly Oracle crush the competitors and grew sales?

Read the book!

Tidbit:Oracle:Exadata:Flash Disks Information

2011-02-20T01:27:00.000-08:00

Use below commands to get information on Exadata Cell Flash Disks

CellCLI> list celldisk where diskType = FlashDisk

         FD_00_cell      normal
         FD_01_cell      normal
         FD_02_cell      normal
         FD_03_cell      normal

CellCLI> list celldisk where diskType = FlashDisk DETAIL

         name:                   FD_00_cell
         comment:
         creationTime:           2010-10-21T14:52:28-07:00
         deviceName:             /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01
         devicePartition:        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01
         diskType:               FlashDisk
         errorCount:             0
         freeSpace:              0
         id:                     0000012b-d0c7-42f2-0000-000000000000
         interleaving:           none
         lun:                    /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01
         raidLevel:              "RAID 0"
         size:                   496M
         status:                 normal

         name:                   FD_01_cell
         comment:
         creationTime:           2010-10-21T14:52:30-07:00
         deviceName:             /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02
         devicePartition:        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02
         diskType:               FlashDisk
         errorCount:             0
         freeSpace:              0
         id:                     0000012b-d0c7-61d3-0000-000000000000
         interleaving:           none
         lun:                    /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02
         raidLevel:              "RAID 0"
         size:                   496M
         status:                 normal

         name:                   FD_02_cell
         comment:
         creationTime:           2010-10-21T14:52:31-07:00
         deviceName:             /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03
         devicePartition:        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03
         diskType:               FlashDisk
         errorCount:             0
         freeSpace:              0
         id:                     0000012b-d0c7-695e-0000-000000000000
         interleaving:           none
         lun:                    /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03
         raidLevel:              "RAID 0"
         size:                   496M
         status:                 normal

         name:                   FD_03_cell
         comment:
         creationTime:           2010-10-21T14:52:32-07:00
         deviceName:             /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04
         devicePartition:        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04
         diskType:               FlashDisk
         errorCount:             0
         freeSpace:              0
         id:                     0000012b-d0c7-6cdc-0000-000000000000
         interleaving:           none
         lun:                    /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04
         raidLevel:              "RAID 0"
         size:                   496M
         status:                 normal

blog:design:monetization

2011-02-19T21:36:00.001-08:00

Nice article on blog monetization:

http://www.infobarrel.com/Blogger_Template_Design_For_Monetization

And this one on ad's placement:

http://www.buzzle.com/articles/maximize-your-advertising-revenue-with-your-blog.html

Books: Exadata Books Coming!

2011-02-17T11:26:00.000-08:00

Looks like couple oracle exadata books are in cooking already!

Achieving Extreme Performance with Oracle Exadata (Osborne ORACLE Press Series)

Expert Oracle Exadata

tidbit: blog registration

2011-02-17T11:20:00.001-08:00

Found some very nice site to register blog for search engines:

http://www.wordsinarow.com/blog-registration.html

Exadata Performance Monitoring Part 1

2011-02-17T01:28:00.000-08:00

CELLCLI LIST command can be used to monitor and monitor cell performance metrics

To receive some help on specific performance metric, use

list metricdefinition <metric_name> DETAIL

For Example:

CellCLI> list metricdefinition Io_LOAD DETAIL
         name:                   IO_LOAD
         description:            "Average I/O load for hard disks"
         metricType:             Instantaneous
         objectType:             CELL
         unit:                   Number

To display specific metric value, use:

CellCLI> LIST METRICCURRENT CL_CPUT DETAIL
         name:                   CL_CPUT
         alertState:             normal
         collectionTime:         2011-02-17T00:39:32-08:00
         metricObjectName:       cell
         metricType:             Instantaneous
         metricValue:            30.6 %
         objectType:             CELL
LIST command also accepts where clause (attribute filters):

To Show all immediate non-zero performance metric values:

LIST METRICCURRENT where metricValue > 0

CD_IO_TM_R_LG   CD_disk02_cell                                  36,321 us
CD_IO_TM_R_SM   CD_disk01_cell                                  1,347,527 us
CD_IO_TM_R_SM   CD_disk02_cell                                  1,151,048 us
CD_IO_TM_R_SM   CD_disk03_cell                                  931,721 us
CD_IO_TM_R_SM   CD_disk04_cell                                  2,286,866 us
CD_IO_TM_R_SM   CD_disk05_cell                                  958,574 us

To show historical data, use LIST METRICHISTORY Command:

You can use metricobjectname to show specific disk and collectiontime to show specific timeframe.

Make sure to filter data as default "list metrichistory" would provide too much data to comprehend

CellCLI> list metrichistory CD_IO_RQ_W_SM where metricobjectname = CD_disk01_cell and collectiontime > '2011-02-17T00:53:01-08:00' and metricvalue > 0

CD_IO_RQ_W_SM   CD_disk01_cell 350,071 IO requests 2011-02-17T00:53:19-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 356,091 IO requests 2011-02-17T00:54:19-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 358,111 IO requests 2011-02-17T00:55:19-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 381,131 IO requests 2011-02-17T00:56:19-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 454,151 IO requests 2011-02-17T00:57:19-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 470,171 IO requests 2011-02-17T00:58:19-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 640,191 IO requests 2011-02-17T00:59:20-08:00
CD_IO_RQ_W_SM   CD_disk01_cell 850,211 IO requests 2011-02-17T01:00:20-08:00

Blog monetization - some thoughts

2011-02-15T01:48:00.000-08:00

Since I enabled AdSense on my blog I started to look for what monetization of the blog is. To my surprise, this is a huge industry and it is not based on generation of my money just by creation of a good content.

My "simple minded" understanding of how people earning money on blogs was that one who creates good content, brings more visitors. These visitors suppose to "produce" some clicks on AsSense; at some point this "clickng" would translate into nice check once in a while.

Not that simple - first, there is a "blogsphere war" around - some people prefer blogspot, and some rave about wordpress. Monetization gurus warn about using blogspot, as at some point, it prevents blog owner from converting his blog into "lean, mean money making machine" - which brings in question point - why to make blog to begin with? - to express yourself or to make money? If blog is done just for money - this is garbage content with no spirit and no true continuity, and this garbage will saturate search engines and will only hide good information from people who need it.

I think it should be a matter of balance - it is understandable, that people love rewards and for many monetization is not about money, but rather seeing that their work is been rewarded, may be not in direct way, but a very "real" way.

When you look around - you see tons of advertised blog software and portals that for some "small" money will post your blog information around tons of other blogs, etc. There is other software that will generate dozens (or more) blogs for you with the questionable content (from a copyright perspective) and, again, will supposedly generate money for you.

First I was quite shocked. Does this blogging bacchanalia means that normal blogger, who just writes really good and authentic stuff, has almost zero chances to be heard, unless he joins money making and content faking crowd?

I just touched this topic - may be I wrong, most probably I need to learn much more about the subject - this is quite interesting phenomena.

Overall - we need better "google", better search engines - we need TRUTH SEARCH ENGINES, We need technology that will identify truth and real, good information and will spit all the junk to where it belongs - garbage bin of search engines, not our screens

Surprisingly, I came to this conclusion and that was not my intention at the beginning when I start writing this piece (Honest!)

SQLSPlus/SQLSPlus - SQLPlus for SQL Server

2011-02-07T12:00:00.000-08:00

IMHO this is a very useful application:

SQLS*Plus for SQL Server (http://www.memfix.com/)

When working with SQL Server and trying to use "sqlcmd", it is always coming to mind, that Oracle SQL*Plus is the best database command line application and it would be great to have one for SQL Server - so SQLS*Plus is exactly what is needed.

Beware:Legal Help and Legal Fees

2011-01-03T02:40:00.000-08:00

One of the items that technical founders underestimate and frankly do not understand, is importance of using legal help and also importance of controlling legal fees.

Not having proper agreements and legal setup at the beginning of the company existence will cause many problems going forward.

One of the simple cases: When company is founded, it is very important to set up equity vesting schedule - if you just say that 100% of equity is yours, or split it among the founders, any coming investors will request recapitalization and will re-establish proper vesting from scratch, so after years of work you may end up vesting from zero.

It is best to find legal help through trusted recommendations. Also, you need to make sure that your lawyer has specific experience and references related to work with start up companies - especially your corporate lawyer.

Lawyers like doctors - you need corporate lawyer to deal with company matters, you need IP layers to work with your patents and at some point you many need a litigators well.

Also, as a founder, make sure to retain services of your personal corporate lawyer.

Understand this very clearly - company corporate lawyer is NOT your personal corporate lawyer.

When you starting the company it may seem that way, but you need to recognize clearly - your interests are NOT the company interests. They are aligned in 99% of the cases, but the last 1% is what really matters. Matters especially when investors are being involved.

Silicon Valley is full of stories when founders are totally diluted and left with nothing while companies are sold for 100's of millions of dollars, or when companies are liquidated by investors because they thought that profitable company is progressing too slow for their portfolio.

One sensitive matter here is IP. Founders are brainwashed, that IP should be automatically assigned to the company. The question is WHY? Of course, your business partner would want this, or investors may want this too. But this is clear conflict of interests. Why to do this immediately and to create a huge personal risk exposure?

There are many better options. For example: You may lease IP to your company (for annual fee with interest) and sign agreement that you'll assign it after initial investment of over 1M or even after exit of over 5M, etc (some reasonable condition). This way, if company have to shut down, your IP is not part of the assets and can not be taken from you. Also, since company would own you some money, you'll be establish yourself as one of the creditors. This will also prevent unscrupulous investors from "sending you to cleaners"

Remember, as a technical founder your ideas and your IP is what you have and contribute from an asset perspective and you need to keep it as long as possible and to make sure to be paid for it fairly, not to surrender it just because your partner or corp lawyer says so,

Another thing - always think not just about best possible outcome, but about worst possible outcome as well and make sure to be ready for it. What would happen if your partner leaves tomorrow (for any reason) - can he just walk away with all the equity? - What would happen with the company then? (this is why you need vesting, proper vesting cliff and a right of company to purchase his shares for a very small price, etc). Overall, you need trusted qualified legal advice.

Now in regards to controlling fees - you need to control your talking, control your agenda and control / inspect your legal bills. If you planning to engage into casual conversation with your lawyer - literally - check if this is billable or now. Lawyers derive lots of their income from phone conversation - so be extremely clear, short and precise!

If you plan to engage lawyer into some kind of clearly defined activity - like asking to prepare a document, etc - always ask how long would it take, as well if he can assign this task to associate with the much lower rate.

If your lawyer is dealing with something that can be very taxing for company finance if billed by an hour, request this to be bullied as a project with the cap and in general, try to assign cap (top limit) to the project.

At the end I would like to recommend amazing book that I found after long search - would I read this book earlier, I would have saved thousand of dollars -

Taming the Lawyers: What to Expect in a Lawsuit and How to Make Sure Your Attorney Gets Results (Taking Control)

Beware: Recruiters: How-To notes

2010-12-10T04:32:00.000-08:00

In a start-up you must have best recruiters possible. Unfortunately, quite often, this is not a case. Many recruiters just pick candidates from job search engines, like yahoo jobs, dice or monster and trying to present them to you as if they identified them.

More than that - if you hire person who was found on a search engine, but by any chance (which is quite high) such recruiter also have send you same candidates resume, he can claim that you owe him the fee.

Considering that many of such recruiters keep litigation lawyers on contingency, you can face a lawsuit very fast if you reject to pay such a fee.

I was recently involved into "damage control" situation, when reckless executive behaviors enabled recruiter to sue the company and actually get some money in a process - as for businesses very ofter it is easier to pay than to engage into lengthy lawsuit.

As a person in charge of hiring, it is very important to make sure that agreement clarifies what search of candidates you expect from recruiter.

Another important point - always make sure to have a written agreement in place. Never agree to have couple candidates to be presented to you "just to show how good I am". This is an invitation to a legal troubles.

As a person in charge you already know what percentage / fee you are willing to pay and what other conditions for candidate search you need to have in place. Ask recruiter to send you an agreement, modify it with change tracking, send it back, execute it. After that let him present candidates.

Fees in general:

25% today is a standard proposed fee. I have negotiated 20% with the majority of the search firms. Some companies tried to set 30-35%, but in todays economy this is not reasonable and should not be considered

Other conditions:

Usually recruiter propose 30 days free replacement if candidate left of let go. In complex technical and business positions, this is nearly not enough to understand if person fully fits the position or not.

Always request free candidate replacement within 90-120-180 days - whatever longer you can negotiate. I manage to negotiate 120-180 days with many recruiters.

DB2:SQLPL: Weird look-ahead bug in SQLPL stored procedure SQL0811N

2010-12-06T11:32:00.000-08:00

Just found interesting bug in DB2:

One of the update statements was failing with below SQL Error:

SQL0811N The result of a scalar fullselect, SELECT INTO statement, or VALUES INTO statement is more than one row. SQLSTATE=21000
Nothing unusual, outside of fact that this error was appearing in run time immediately after stored procedure started to execute.

Problem SQL was in the middle of stored procedure body, but it was a FIRST REAL SQL in the body (not call to other stored procedures).

The moral: DB2 executing look-ahead approach in stored procedures and when it finds "naked" SQL it will execute it, ignoring other statements (including debugging ones).

Oracle: Internal structure of "date" data type

2010-12-05T20:22:00.000-08:00

Did some investigation on internal date data format:

A date in binary format contains seven bytes, as shown in table below:

BYTE 1 2 3 4 5 6 7
Meaning Century Year Month Day Hour Minute Second
Example 120 110 11 30 16 18 1

The century and year bytes are in an excess–100 notation.

Dates Before Common Era (BCE) are less than 100.

The era begins on 01–JAN–4712 BCE, which is Julian day 1.

For this date, the century byte is 53, and the year byte is 88.

The hour, minute, and second bytes are in excess–1 notation.

The hour byte ranges from 1 to 24, the minute and second bytes from 1 to 60.

If no time was specified when the date was created, the time defaults to midnight (1, 1, 1).

You can use DUMP function to investigate this:

SQL> create table t ( x date );

Table created.

SQL> insert into t values ( to_date( '03-nov-2010 3:17:01pm', 'dd-mon-yyyy hh:mi:sspm' ) );
1 row created.
SQL> commit;

SQL> select dump(x) from t;
DUMP(X)
--------------------------------------------------------------------------------
Typ=12 Len=7: 120,110,11,3,16,18,2

tidbit:windows:show administrator login

2010-12-05T03:48:00.000-08:00

Needed to be able to see "Administrator" user as a login:

in "regedit" navigate to the following key:

1) HKEY_LOCAL_MACHINE \ SOFTWARE \ Microsoft \ Windows NT \ CurrentVersion \ Winlogon \ SpecialAccounts \ UserList
2) Right-click in the right pane and select New DWORD Value.

Type-in Administrator as the value.

business: Mail Campaign Websites

2010-12-03T17:08:00.000-08:00

Needed to get a good provider for a mail campaigns.

Would very nice URL:

http://email-marketing-service-review.toptenreviews.com/

Looks like Constant Contact is an industry standard, but others like MailChimp and Campaigner are definitely to be considered.

tidbit:windows:reset and re-validate windows registry and system files

2010-12-03T07:27:00.000-08:00

After recent virus attack, I suddenly lost windows search on outlook 2007. I tried many things - reinstalled windows search 4.0, upgraded to Outlook 2010, rebuild indexes - not much help.

When looking after the solution, I have found advice to execute "sfc /scannow" -

This will scan the system files looking for error and will automatically repair or replace any corrupted files.

This actually helped and now I have search functionality restored.

tidbit:Java:Garbage Collection on Netbeans and Eclipse

2010-12-01T01:48:00.000-08:00

Triggering garbage collection (GC) on java-based IDE, such as Netbeans or Eclipse can greatly improve IDE performance, especially if you do not restart IDE for a long time.

Easy way is to start JConsole (standard JDK application), connect to IDE process, select "Memory" tab and click "Perform GC"

BTW - JConsole can be started directly from Netbeans - In "Debug" menu select "Start JConsole"

Check this link for another way to trigger GC from Eclipse:

http://compdottech.blogspot.com/2009/05/trigger-gc-in-eclipse.html

tidbit:DB2:Global temporary table indexes

2010-11-28T23:19:00.000-08:00

Global temporary table is a great way to store session level data in DB2. However, it is important to understand, that these are not 'in-memory' tables and performance of global temporary tables is much lower than arrays (DB2 9.5) or collections (DB2 9.7).

Another very important factor is to make sure to use indexes on global temporary tables. For example:

DECLARE GLOBAL TEMPORARY TABLE session.lookup_data ( row_id integer, value varchar(4000) ) NOT LOGGED on commit PRESERVE rows;

CREATE unique index session.lookup_data_idx ON session.lookup_data (row_id );

Index on global temporary table will provide huge performance improvement.

Another option is "NOT LOGGED" on table, which improves performance, but not drastically.

From a practical perspective, global temporary table are useful when we need to store millions of rows on a session level.

If the need is to store hundreds of rows, arrays are much faster.

As a practical case, recently I switched from session level tables to arrays and data processing performance jumped from 25 records/second to 550 records/second with the use of arrays.

tidbit:Oracle:Exadata Wait Events (cell related)

2010-11-26T13:06:00.000-08:00

Exadata Wait Events:

NAME                                                             WAIT_CLASS
---------------------------------------------------------------- --------------------------------------------------

cell list of blocks physical read                                User I/O
cell manager cancel work request                                 Other
cell manager closing cell                                        System I/O
cell manager discovering disks                                   System I/O
cell manager opening cell                                        System I/O
cell multiblock physical read                                    User I/O
cell single block physical read                                  User I/O
cell smart file creation                                         User I/O
cell smart flash unkeep                                          Other
cell smart incremental backup                                    System I/O
cell smart index scan                                            User I/O
cell smart restore from backup                                   System I/O
cell smart table scan                                            User I/O
cell statistics gather                                           User I/O
cell worker idle                                                 Idle
cell worker online completion                                    Other
cell worker retry                                                Other

tidbit:Oracle:Exadata System Statistics (cell related)

2010-11-26T12:57:00.001-08:00

Exadata cell statistics:

STATISTIC# NAME                                                                  CLASS
---------- ---------------------------------------------------------------- ----------
        51 cell physical IO interconnect bytes                                      64
       224 cell physical IO bytes saved during optimized file creation              64
       225 cell physical IO bytes saved during optimized RMAN file restore          64
       226 cell physical IO bytes eligible for predicate offload                    64
       227 cell physical IO bytes saved by storage index                             8
       228 cell smart IO session cache lookups                                      64
       229 cell smart IO session cache hits                                         64
       230 cell smart IO session cache soft misses                                  64
       231 cell smart IO session cache hard misses                                  64
       232 cell smart IO session cache hwm                                          64
       233 cell num smart IO sessions in rdbms block IO due to user                 64
       234 cell num smart IO sessions in rdbms block IO due to big payload          64
       235 cell num smart IO sessions using passthru mode due to user               64
       236 cell num smart IO sessions using passthru mode due to cellsrv            64
       237 cell num smart IO sessions using passthru mode due to timezone           64
       238 cell num smart file creation sessions using rdbms block IO mode          64
       239 cell physical IO interconnect bytes returned by smart scan               64
       240 cell session smart scan efficiency                                       64
       359 cell scans                                                               64
       360 cell blocks processed by cache layer                                    128
       361 cell blocks processed by txn layer                                      128
       362 cell blocks processed by data layer                                     128
       363 cell blocks processed by index layer                                    128
       364 cell commit cache queries                                                64
       365 cell transactions found in commit cache                                  64
       366 cell blocks helped by commit cache                                       64
       367 cell blocks helped by minscn optimization                                64
       368 cell blocks skipped due to chained rows                                  64
       369 cell simulated physical IO bytes eligible for predicate offload         192
       370 cell simulated physical IO bytes returned by predicate offload          192
       371 cell simulated session smart scan efficiency                            192
       372 cell CUs sent uncompressed                                               64
       373 cell CUs sent compressed                                                 64
       374 cell CUs sent head piece                                                 64
       375 cell CUs processed for uncompressed                                      64
       376 cell CUs processed for compressed                                        64
       377 cell IO uncompressed bytes                                               64
       394 cell index scans                                                         64
       571 cell flash cache read hits                                                8

tidbit:DB2:drop schema

2010-11-26T10:53:00.000-08:00

Simple way to drop DB2 schema:

Db2 drop table ERRORSCHEMA.ERRORTABLE

Db2 "CALL SYSPROC.ADMIN_DROP_SCHEMA('DG', NULL, 'ERRORSCHEMA', 'ERRORTABLE')"

Oracle:Windows:Database Upgrade Fall-back

2010-11-26T10:50:00.000-08:00

Recently I was approached by one of the major customers to assist in creation of database fall-back procedure for a Windows based Oracle 10g RAC system, that is using OCFS to store database data.

The objective was to upgrade clusterware and database from Oracle 10.2.0.2 to 10.2.0.4 and on top of it apply patch 38. Obviously, both ORACLE_HOMES and database itself were to be upgraded.

In a large company world, I would suggest to create separate RAC cluster, setup standby database there and during the brief outage copy control files and redo logs, start-up database and be done with it. This way the fall-back is very simple (start using old system) and most of the work (installing and patching software) can be done without any production impact.

The problem was that customer engaged us only after their fall-back test totally failed and all outage process was based on actually upgrading production system (software and database).

The major issue was that Oracle provided fall-back failed, as downgrade did not worked as expected, as well as nothing was said about proper downgrade of ORACLE_HOMES.

At the end, we arrived to below fall-back procedure:

1) Shutdown all database related Windows services
2) Shutdown clusterware related Windows services
3) Kill ons.exe processes - they sometimes left hanging
4) Take system state backup (IMPORTANT)
5) Backup ocfs and oraclefence drivers
6) Backup database and clusterware Oracle homes
7) Backup Oracle Inventory
8) Backup control files and redo logs
9) Backup OCR using ocrconfig -export <exp_ocr_file.bkp>
10) Backup Voting disk using 'ocopy <voting_file> <exp_voting_file.bkp>'
11) Backup config files (init/tnsnames/listener/sqlnet) (just in case)

During fall-back exercise, we were able to:

1) Restore system state backup in 3 minutes
2) Restore drivers in 3 min (manual copy)
3) Oracle Homes and Inventory Restore was done using rename (1 minute)
4) OCR was restored using "ocrconfig -import" (3 minutes), no need to restore voting disk
5) Database restore from RMAN took 40-50 minutes
6) Redo restore (50GB) took 15 minutes in parallel with database restore
7) Control file restore took 1 minute
8) Database recovery and startup took 5 minutes

Fall-back was 100% successful