Gotcha for Inline View Indexing

Since Domino FP9, IBM has introduced Inline View Indexing to keep critical views up to date.

So I have gone through the instructions in

https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/admn_inline_index_enabling.html

thinking implementing is just easy as ABC but sadly it is not.

I have followed the instructions, restarted Domino, read through it many times but "show dbs * inline views" and "dbcache show inline views" do not show a single view.

Good old IBM Support replied that the DB needs to be on ODS52 and IMAP should not be enabled for the DB.

These requirements are not mentioned in the instructions but my DB has already met the requirements and it is still not working.

In the end, after much research, I found out that transaction logging needs to be enabled in the server for Inline View Indexing to work.

Meanwhile IBM is still working on this problem as I did not update them of my finding.

Beware not to abort compact -REPLICA -RESTART

Due to some of our DBs encountered "Unable to extend an ID table" error, I have scheduled compact -REPLICA -RESTART to run on these DBs every weekend.

One fine weekend, our users decided to access these DBs but compact -REPLICA is still running on 1 large DB.

Thinking that compact -REPLICA replicates the DB to a new .REPL file, if I quit the compact task, Domino should be smart enough to stop the local replication and delete the new .REPL file, leaving everything untouched.

Boy I am so wrong and this caused me several angry users.

What happened?

compact -REPLICA -RESTART was running on a large DB, the documents are sync and it is syncing views to the .REPL file

I stopped compact task in Domino Administrator client, and suddenly the remaining views syncing process completed fast (but actually the remaining views are not sync at all)

[1BA8:0004-2388] Compact -REPLICA, syncing view: Unxxxxx, ID: 003FF26A, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL 
[1BA8:0004-2388] Compact -REPLICA, syncing view: Utxxx, ID: 0000A7A2, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL                   
[1BA8:0004-2388] Compact -REPLICA, syncing view: Wexxxx, ID: 0000A57A, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL 
[1BA8:0004-2388] Compact -REPLICA, syncing view: Wixxx, ID: 0000A552, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL                     
[1BA8:0004-2388] Compact -REPLICA, bring database online E:\Domino\data\xxx.REPL
[1BA8:0004-2388]         Database Name            Refs Mod  FDs  LockWaits/AvgWait #Waiters MaxWaiters Online
[1BA8:0004-2388] Compact -REPLICA, initial population complete for DB: E:\Domino\data\xxx.nsf
[1BA8:0004-2388] Compact -REPLICA, drop all users for E:\Domino\data\xxx.nsf                                         
[1BA8:0004-2388]         Database Name            Refs Mod  FDs  LockWaits/AvgWait #Waiters MaxWaiters Online
[1BA8:0004-2388] E:\Domino\data\xxx.nsf      2   Y   6        0      0        0       4         Y
[0DE4:0009-13D0] drop xxx.nsf
[1BA8:0004-2388] Compact -REPLICA, take database offline for E:\Domino\data\xxx.nsf
[1BA8:0004-2388] Compact -REPLICA, take database offline for E:\Domino\data\xxx.nsf succeeded - offline: 1, references: 2   
[1BA8:0004-2388] Compact -REPLICA, bring database online E:\Domino\data\xxx.nsf
[1BA8:0004-2388]         Database Name            Refs Mod  FDs LockWaits/AvgWait #Waiters MaxWaiters Online
[1BA8:0004-2388] Compact -REPLICA, needs to restart server to complete compaction for DB: E:\Domino\data\xxx.nsf
[1BA8:0004-2388] 20/01/2018 12:29:15 PM  Error compacting E:\Domino\data\xxx.nsf,  compactdb.ind -REPLICA -RESTART -# 4: Program shutdown in progress
[1BA8:0004-2388] 20/01/2018 12:29:15 PM  Database compactor deferring 'restart server' for  E:\Domino\data\xxx.nsf,   compactdb.ind -REPLICA -RESTART -# 4
[1BA8:0002-1A48] 20/01/2018 12:29:16 PM  Database compactor issued a 'restart server', compactdb.ind -REPLICA -RESTART -# 4
[1BA8:0002-1A48] 20/01/2018 12:29:16 PM  Database compactor process shutdown
[0DE4:0009-13D0] restart server

After server was restarted, Domino proceed to replace the DB with the .REPL file! And the problem is Domino does not continue syncing the rest of the views which it does not completed earlier!

Compact -REPLICA, check rename of NSF file with existing restart flag, DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, rename with ORIG file E:\Domino\data\xxx.ORIG, DB: E:\Domino\data\xxx.nsf, deleting ORIG, error: No error
[249C:0092-1C68] Compact -REPLICA, rename of NSF file with existing restart flag, DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, shift: E:\Domino\data\xxx.nsf => E:\Domino\data\xxx.ORIG, E:\Domino\data\xxx.REPL => E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, rename complete for DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, resync of NSF file with existing restart flag, DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, take database offline E:\Domino\data\xxx.ORIG
[249C:0092-1C68] Clearing DBIID D581A1A5 for DB E:\Domino\data\xxx.ORIG
[249C:0092-1C68] Compact -REPLICA, no refresh file E:\Domino\data\xxx.nsf, from DB: E:\Domino\data\xxx.ORIG since:  20/01/2018 08:03:28 AM, data: 20/01/2018 07:01:19 AM, nondata: 20/01/2018 01:00:56 AM
[249C:0092-1C68] Compact -REPLICA, syncing unread from DB: E:\Domino\data\xxx.ORIG to DB: E:\Domino\data\xxx.nsf   
[249C:0092-1C68] Compact -REPLICA, NSFDbCompactSyncFolders: Replicating folders  Since time (20/01/2018 08:03:28 AM), Src time (), Dst time ()
[249C:0092-1C68] [249C:0092-1C68] Compact -REPLICA, NSFDbCompactSyncFolders: NSFStartFolderReplSource->0x3AE=Folders in database are up to date
[249C:0092-1C68] Compact -REPLICA, bring database online E:\Domino\data\xxx.ORIG
[249C:0092-1C68] 20/01/2018 12:30:12 PM  Recovery Manager: Assigning new DBIID for E:\Domino\data\xxx.nsf (need new backup for media recovery).
[249C:0092-1C68] 20/01/2018 12:30:12 PM  Compacting E:\Domino\data\xxx.nsf (), restart completing, -REPLICA -RESTART               [249C:0092-1C68]         Database Name            Refs Mod  FDs LockWaits/AvgWait  #Waiters MaxWaiters Online
[249C:0092-1C68] Compact -REPLICA, delete E:\Domino\data\xxx.ORIG                                       
[249C:0092-1C68] Compact -REPLICA, complete for DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] 20/01/2018 12:30:14 PM  Compacted E:\Domino\data\xxx.nsf (), restart completed, -REPLICA -RESTART
[249C:0092-1C68] 20/01/2018 12:30:14 PM  Compacted E:\Domino\data\xxx.nsf (), increased by 2659328K bytes (<1%), -REPLICA -RESTART

This results in very slow response while our users tried to open the unsync views which are not built yet.

After communicating with our IBM Support, they finally agreed to create SPR # MJCGAWE9MD: Compact -replica Does Not Have A Way To Check What Process It Left Off Before Aborted

There is no way to know when IBM decides to make Domino smarter to safely quit compact -REPLICA so I would strongly advise against stopping compact -REPLICA while it is still running.

Domino 9.0.1 Feature Pack 10 Interim Fix 1 finally released for all platforms

This interim fix was released a few days ago but Windows 64bit version was delayed while Windows 32bit was released and then recalled.

This fix is finally released today for all platforms.

http://www-01.ibm.com/support/docview.wss?uid=swg21657963

Domino XPages Partial Refresh Problem since Domino 9.0.1 FP8

After upgrading from Domino 9.0.1 FP7 to FP8 and later FP9, it seems Domino XPages Partial Refresh fail to work correctly using IE through WebSeal. It was fine all these while on Domino 9.0.1 FP7 and earlier.

Problem does not appears if accessing the web application directly without going through WebSeal. Besides that, it also appears good using Chrome passing through WebSeal.

Checked IE console / debug log and this error is shown:

SCRIPT5007: Unable to get property 'match' of undefined or null reference                     

File: xspClientDojo.js, Line: 5, Column: 45925                          

Workaround:

1. Setup server as Domino 9.0.1 FP7, verified there is no problem.

2. Copy D:\Domino\osgi\shared\eclipse\plugins\com.ibm.xsp.dojo_9.0.1.20160811-1000 folder to another location.

3. Upgrade Domino to 9.0.1 FP9, verified problem appears.

4. Delete D:\Domino\osgi\shared\eclipse\plugins\com.ibm.xsp.dojo_9.0.1.20170803-1411 folder
5. Copy the backed up com.ibm.xsp.dojo_9.0.1.20160811-1000 folder to D:\Domino\osgi\shared\eclipse\plugins\

6. Verified problem no longer appears.

                                                                        

Conclusion:

com.ibm.xsp.dojo_9.0.1.20160811-1000 is working fine but the updated dojo folder is causing this problem.

There are some changes in xspClientDojo.js starting from Domino 9.0.1 FP8 which causes this problem.

Solution:

You may use the workaround above or request a custom hotfix from IBM. This fix is not available in any of the public released FP8 or FP9 Interim Fix or Feature Pack. I am not sure if this is fixed in FP10.

This issue is tracked in SPR #SRKMAS2J84.
https://www-01.ibm.com/support/entdocview.wss?uid=swg1LO93190

IBM Support has built a custom hotfix 901FP9HF212 created on top of 901FP9 IF2 + SRKMAS2J84 plus the following fixes which appears to fix the problem in my test server:

PALT9Z3NZ8:  This.Editor.Getdata Is Not A Function If Doing A Refresh Onclientload                  RGAU9R2HWB:  Customizing The Toolbar In Rich Text Control Of Xpages Does Not Work With Partial Refreshes In Notes 901fp2

RGAUAHBF9U:  Xpages: Djtabcontainer Extlib Control - Toolbar In Rich Text Control Of Xpages Is Not Consistent With Different Tabs

LHEY9MHHFA:  Spellchecker not working for some locales

MKEE9ZBHWW:  XPages, Rich Text control, Spell Check - Danish language using en-US dictionary  

IBM Notes / Domino 9.0.1 Feature Pack 10 Released ! ... but with some problems

I have been tracking the release status of Domino 9.0.1 FP10 and it was stucked at Stage 2: Code Freeze and thus I am thinking it might take some time to be released.

All of a sudden at 1 Feb 2018, it was released to public and it was still showing Stage 2. After a day, it finally jumped to Stage 5: Web Posting

http://www-10.lotus.com/ldd/fixlist.nsf/(Progress)/9.0.1%20FP10

There are tons of new features and bug fixes as mentioned in the release note. I noted there are some changes in this final release note compared to the preliminary release note. It seems the JVM in this feature pack is the newer Java 1.8 SR5 FP6 instead of what was mentioned earlier (SR4 FP10).

Hold on ! Before you rush to download and install it to all your servers. As reported by Daniel, there are some reported problems in this newly released feature pack.
http://blog.nashcom.de/nashcomblog.nsf/dx/notesdomino901fp10-issues-ibm-is-working-on-if1-and-is-listening-for-more-feedback.htm

IBM is aware of the following issues and fixes for several issues are in progress, with delivery of a fix, FP10 IF1, being planned for the week of February 5.

Issues being worked include the following:

SMTP Mails with Umlauts are broken after installing FP10, being tracked under SPR #JBAMAVKUPX
FP10 version incorrect in NSF API NSFDbGetMajMinVersion, being tracked under SPR #KBRNAVLMA3

If you use embedded Sametime and haven't already, please review the information in Sametime Embedded returns error after install of Notes 9.0.1 FP10 on top of Notes 9.0.1 FP9 + Sametime 9.0.x

https://www.ibm.com/developerworks/community/blogs/LotusSupport/entry/Listening_to_your_feedback_on_Notes_Domino_9_0_1_FP10?lang=en

There might be more issues not known yet, I would recommend NOT TO upgrade to FP10 on any of your production servers for now.

[Domino 9.0.1 FP9] Occasional Domino Task Hang on non-transaction logged server

We have upgraded our mail server (non-transaction logged) few month ago from Domino 9.0.1 FP8 to FP9 IF1 and since then we experienced occasional Domino task hang few times in a month.

Domino task such as Indexer and Replicator will hang on a certain mailbox and the mailbox is no longer accessible. Everything looks fine again after we restarted Domino. And when this appears again the next time, it will be a different mailbox.

We managed to run manual NSD when this happened and requested IBM Support to check on this.

It appears that this is a known issue in  Domino 9.0.1 FP9 which happens on non transaction logged server and with warning quota set to DBs (mostly on mail server, as we set quota to mailboxes).

SPR #KBRNASPR6L - Domino server hangs in UBMDelayThreadDebug on non Transaction logging enabled servers.

Solution (Either one the following will prevent this):

1. Submit a PMR to IBM Support to obtain a custom hotfix. (this fix is not available in any of the public released interim fix for Domino 9.0.1 FP9.)

2. Enable transaction logging.

3. Upgrade to Domino 9.0.1 FP10 which includes this fix. (Not recommended for the moment, suggest to wait for Domino 9.0.1 FP10 IF1 which fixes problems in FP10)

Notes client fail to deliver external email when exceeded mailbox quota

We noticed that while mailbox has exceeded a set quota, if user proceed to send an email with both internal and external recipients, only internal recipients will receive the email but Notes client will not deliver the email to external address.

There is no delivery failure message or notification informing the failure of external email delivery. This is a serious problem as there might be many important emails not sent to external without the sender's knowledge, and since it is able to send to internal recipients, the sender might even be convinced it was already sent to external.

I have submitted a case to IBM and this problem is verified and tracked under SPR#NNAI8ZZ4EK http://www-01.ibm.com/support/docview.wss?uid=swg1LO72557, we are currently using IBM Notes 9.0.1 FP9 and we recalled that the problem happens even with Notes 8.5.3 so I assumed that this affects all version of Notes client.

IBM has provided a Notes client hotfix to us but there is no word when will this fix included in any of the future public released Feature Pack or Interim Fix, which is really frustrating.

To anyone out there who encounters this problem, you may submit a case with IBM quoting the SPR number to obtain a fix for this issue.