Sunday, March 4, 2018

Beware not to abort compact -REPLICA -RESTART

Due to some of our DBs encountered "Unable to extend an ID table" error, I have scheduled compact -REPLICA -RESTART to run on these DBs every weekend.

One fine weekend, our users decided to access these DBs but compact -REPLICA is still running on 1 large DB.

Thinking that compact -REPLICA replicates the DB to a new .REPL file, if I quit the compact task, Domino should be smart enough to stop the local replication and delete the new .REPL file, leaving everything untouched.

Boy I am so wrong and this caused me several angry users.

What happened?

compact -REPLICA -RESTART was running on a large DB, the documents are sync and it is syncing views to the .REPL file

I stopped compact task in Domino Administrator client, and suddenly the remaining views syncing process completed fast (but actually the remaining views are not sync at all)

[1BA8:0004-2388] Compact -REPLICA, syncing view: Unxxxxx, ID: 003FF26A, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL 
[1BA8:0004-2388] Compact -REPLICA, syncing view: Utxxx, ID: 0000A7A2, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL                   
[1BA8:0004-2388] Compact -REPLICA, syncing view: Wexxxx, ID: 0000A57A, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL 
[1BA8:0004-2388] Compact -REPLICA, syncing view: Wixxx, ID: 0000A552, from DB: E:\Domino\data\xxx.nsf, to DB: E:\Domino\data\xxx.REPL                     
[1BA8:0004-2388] Compact -REPLICA, bring database online E:\Domino\data\xxx.REPL
[1BA8:0004-2388]         Database Name            Refs Mod  FDs  LockWaits/AvgWait #Waiters MaxWaiters Online
[1BA8:0004-2388] Compact -REPLICA, initial population complete for DB: E:\Domino\data\xxx.nsf
[1BA8:0004-2388] Compact -REPLICA, drop all users for E:\Domino\data\xxx.nsf                                         
[1BA8:0004-2388]         Database Name            Refs Mod  FDs  LockWaits/AvgWait #Waiters MaxWaiters Online
[1BA8:0004-2388] E:\Domino\data\xxx.nsf      2   Y   6        0      0        0       4         Y
[0DE4:0009-13D0] drop xxx.nsf
[1BA8:0004-2388] Compact -REPLICA, take database offline for E:\Domino\data\xxx.nsf
[1BA8:0004-2388] Compact -REPLICA, take database offline for E:\Domino\data\xxx.nsf succeeded - offline: 1, references: 2   
[1BA8:0004-2388] Compact -REPLICA, bring database online E:\Domino\data\xxx.nsf
[1BA8:0004-2388]         Database Name            Refs Mod  FDs LockWaits/AvgWait #Waiters MaxWaiters Online
[1BA8:0004-2388] Compact -REPLICA, needs to restart server to complete compaction for DB: E:\Domino\data\xxx.nsf
[1BA8:0004-2388] 20/01/2018 12:29:15 PM  Error compacting E:\Domino\data\xxx.nsf,  compactdb.ind -REPLICA -RESTART -# 4: Program shutdown in progress
[1BA8:0004-2388] 20/01/2018 12:29:15 PM  Database compactor deferring 'restart server' for  E:\Domino\data\xxx.nsf,   compactdb.ind -REPLICA -RESTART -# 4
[1BA8:0002-1A48] 20/01/2018 12:29:16 PM  Database compactor issued a 'restart server', compactdb.ind -REPLICA -RESTART -# 4
[1BA8:0002-1A48] 20/01/2018 12:29:16 PM  Database compactor process shutdown
[0DE4:0009-13D0] restart server

After server was restarted, Domino proceed to replace the DB with the .REPL file! And the problem is Domino does not continue syncing the rest of the views which it does not completed earlier!

Compact -REPLICA, check rename of NSF file with existing restart flag, DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, rename with ORIG file E:\Domino\data\xxx.ORIG, DB: E:\Domino\data\xxx.nsf, deleting ORIG, error: No error
[249C:0092-1C68] Compact -REPLICA, rename of NSF file with existing restart flag, DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, shift: E:\Domino\data\xxx.nsf => E:\Domino\data\xxx.ORIG, E:\Domino\data\xxx.REPL => E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, rename complete for DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, resync of NSF file with existing restart flag, DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] Compact -REPLICA, take database offline E:\Domino\data\xxx.ORIG
[249C:0092-1C68] Clearing DBIID D581A1A5 for DB E:\Domino\data\xxx.ORIG
[249C:0092-1C68] Compact -REPLICA, no refresh file E:\Domino\data\xxx.nsf, from DB: E:\Domino\data\xxx.ORIG since:  20/01/2018 08:03:28 AM, data: 20/01/2018 07:01:19 AM, nondata: 20/01/2018 01:00:56 AM
[249C:0092-1C68] Compact -REPLICA, syncing unread from DB: E:\Domino\data\xxx.ORIG to DB: E:\Domino\data\xxx.nsf   
[249C:0092-1C68] Compact -REPLICA, NSFDbCompactSyncFolders: Replicating folders  Since time (20/01/2018 08:03:28 AM), Src time (), Dst time ()
[249C:0092-1C68] [249C:0092-1C68] Compact -REPLICA, NSFDbCompactSyncFolders: NSFStartFolderReplSource->0x3AE=Folders in database are up to date
[249C:0092-1C68] Compact -REPLICA, bring database online E:\Domino\data\xxx.ORIG
[249C:0092-1C68] 20/01/2018 12:30:12 PM  Recovery Manager: Assigning new DBIID for E:\Domino\data\xxx.nsf (need new backup for media recovery).
[249C:0092-1C68] 20/01/2018 12:30:12 PM  Compacting E:\Domino\data\xxx.nsf (), restart completing, -REPLICA -RESTART               [249C:0092-1C68]         Database Name            Refs Mod  FDs LockWaits/AvgWait  #Waiters MaxWaiters Online
[249C:0092-1C68] Compact -REPLICA, delete E:\Domino\data\xxx.ORIG                                       
[249C:0092-1C68] Compact -REPLICA, complete for DB: E:\Domino\data\xxx.nsf
[249C:0092-1C68] 20/01/2018 12:30:14 PM  Compacted E:\Domino\data\xxx.nsf (), restart completed, -REPLICA -RESTART
[249C:0092-1C68] 20/01/2018 12:30:14 PM  Compacted E:\Domino\data\xxx.nsf (), increased by 2659328K bytes (<1%), -REPLICA -RESTART

This results in very slow response while our users tried to open the unsync views which are not built yet.

After communicating with our IBM Support, they finally agreed to create SPR # MJCGAWE9MD: Compact -replica Does Not Have A Way To Check What Process It Left Off Before Aborted

There is no way to know when IBM decides to make Domino smarter to safely quit compact -REPLICA so I would strongly advise against stopping compact -REPLICA while it is still running.

Wednesday, February 14, 2018

Domino 9.0.1 Feature Pack 10 Interim Fix 1 finally released for all platforms

This interim fix was released a few days ago but Windows 64bit version was delayed while Windows 32bit was released and then recalled.

This fix is finally released today for all platforms.

Saturday, February 3, 2018

Domino XPages Partial Refresh Problem since Domino 9.0.1 FP8

After upgrading from Domino 9.0.1 FP7 to FP8 and later FP9, it seems Domino XPages Partial Refresh fail to work correctly using IE through WebSeal. It was fine all these while on Domino 9.0.1 FP7 and earlier.

Problem does not appears if accessing the web application directly without going through WebSeal. Besides that, it also appears good using Chrome passing through WebSeal.

Checked IE console / debug log and this error is shown:

SCRIPT5007: Unable to get property 'match' of undefined or null reference                     
File: xspClientDojo.js, Line: 5, Column: 45925                          

1. Setup server as Domino 9.0.1 FP7, verified there is no problem.
2. Copy D:\Domino\osgi\shared\eclipse\plugins\ folder to another location.
3. Upgrade Domino to 9.0.1 FP9, verified problem appears.
4. Delete D:\Domino\osgi\shared\eclipse\plugins\ folder
5. Copy the backed up folder to D:\Domino\osgi\shared\eclipse\plugins\
6. Verified problem no longer appears.
Conclusion: is working fine but the updated dojo folder is causing this problem.

There are some changes in xspClientDojo.js starting from Domino 9.0.1 FP8 which causes this problem.

You may use the workaround above or request a custom hotfix from IBM. This fix is not available in any of the public released FP8 or FP9 Interim Fix or Feature Pack. I am not sure if this is fixed in FP10.

This issue is tracked in SPR #SRKMAS2J84.

IBM Support has built a custom hotfix 901FP9HF212 created on top of 901FP9 IF2 + SRKMAS2J84 plus the following fixes which appears to fix the problem in my test server:

PALT9Z3NZ8:  This.Editor.Getdata Is Not A Function If Doing A Refresh Onclientload                  RGAU9R2HWB:  Customizing The Toolbar In Rich Text Control Of Xpages Does Not Work With Partial Refreshes In Notes 901fp2
RGAUAHBF9U:  Xpages: Djtabcontainer Extlib Control - Toolbar In Rich Text Control Of Xpages Is Not Consistent With Different Tabs
LHEY9MHHFA:  Spellchecker not working for some locales
MKEE9ZBHWW:  XPages, Rich Text control, Spell Check - Danish language using en-US dictionary  

IBM Notes / Domino 9.0.1 Feature Pack 10 Released ! ... but with some problems

I have been tracking the release status of Domino 9.0.1 FP10 and it was stucked at Stage 2: Code Freeze and thus I am thinking it might take some time to be released.

All of a sudden at 1 Feb 2018, it was released to public and it was still showing Stage 2. After a day, it finally jumped to Stage 5: Web Posting

There are tons of new features and bug fixes as mentioned in the release note. I noted there are some changes in this final release note compared to the preliminary release note. It seems the JVM in this feature pack is the newer Java 1.8 SR5 FP6 instead of what was mentioned earlier (SR4 FP10).

Hold on ! Before you rush to download and install it to all your servers. As reported by Daniel, there are some reported problems in this newly released feature pack.

IBM is aware of the following issues and fixes for several issues are in progress, with delivery of a fix, FP10 IF1, being planned for the week of February 5.

Issues being worked include the following:

SMTP Mails with Umlauts are broken after installing FP10, being tracked under SPR #JBAMAVKUPX
FP10 version incorrect in NSF API NSFDbGetMajMinVersion, being tracked under SPR #KBRNAVLMA3

If you use embedded Sametime and haven't already, please review the information in Sametime Embedded returns error after install of Notes 9.0.1 FP10 on top of Notes 9.0.1 FP9 + Sametime 9.0.x

There might be more issues not known yet, I would recommend NOT TO upgrade to FP10 on any of your production servers for now.

[Domino 9.0.1 FP9] Occasional Domino Task Hang on non-transaction logged server

We have upgraded our mail server (non-transaction logged) few month ago from Domino 9.0.1 FP8 to FP9 IF1 and since then we experienced occasional Domino task hang few times in a month.

Domino task such as Indexer and Replicator will hang on a certain mailbox and the mailbox is no longer accessible. Everything looks fine again after we restarted Domino. And when this appears again the next time, it will be a different mailbox.

We managed to run manual NSD when this happened and requested IBM Support to check on this.

It appears that this is a known issue in  Domino 9.0.1 FP9 which happens on non transaction logged server and with warning quota set to DBs (mostly on mail server, as we set quota to mailboxes).

SPR #KBRNASPR6L - Domino server hangs in UBMDelayThreadDebug on non Transaction logging enabled servers.

Solution (Either one the following will prevent this):

1. Submit a PMR to IBM Support to obtain a custom hotfix. (this fix is not available in any of the public released interim fix for Domino 9.0.1 FP9.)

2. Enable transaction logging.

3. Upgrade to Domino 9.0.1 FP10 which includes this fix. (Not recommended for the moment, suggest to wait for Domino 9.0.1 FP10 IF1 which fixes problems in FP10)

Thursday, December 28, 2017

Notes client fail to deliver external email when exceeded mailbox quota

We noticed that while mailbox has exceeded a set quota, if user proceed to send an email with both internal and external recipients, only internal recipients will receive the email but Notes client will not deliver the email to external address.

There is no delivery failure message or notification informing the failure of external email delivery. This is a serious problem as there might be many important emails not sent to external without the sender's knowledge, and since it is able to send to internal recipients, the sender might even be convinced it was already sent to external.

I have submitted a case to IBM and this problem is verified and tracked under SPR#NNAI8ZZ4EK, we are currently using IBM Notes 9.0.1 FP9 and we recalled that the problem happens even with Notes 8.5.3 so I assumed that this affects all version of Notes client.

IBM has provided a Notes client hotfix to us but there is no word when will this fix included in any of the future public released Feature Pack or Interim Fix, which is really frustrating.

To anyone out there who encounters this problem, you may submit a case with IBM quoting the SPR number to obtain a fix for this issue.

Wednesday, December 27, 2017

Sneak peek of Notes / Domino 9.0.1 FP10

IBM Notes / Domino 9.0.1 FP10 is scheduled to be released at Q1 2018.$First?OpenDocument

For those who are interested to know what are included in this Feature Pack 10, here is a sneak peak of the goodies.

This is not the final confirmed features and fixes as it might be subject to any further change by IBM.

New Features in IBM Notes/Domino® 9.0.1 Social Edition Feature Pack 10

The documentation in Knowledgebase will be updated with these new features to coincide with the ship date of 9.0.1 FP10

1) The JVM in Designer is Upgraded to use 1.8 at compile time

With this new upgrade, you can launch Designer client and leverage Java 1.8 compile time features in building applications. For full compatibility, apps compiled in FP10 should be deployed on Notes or Domino FP10 and higher. Java apps may run on Notes/Domino 9.0.1 FP8 and higher as that release upgrade to Java 1.8 runtime. However, Application best practice is to compile with the lowest common denominator to the client/server version deployed.

2) Eclipse Platform Upgraded to 4.6.2 - The 9.0.1 FP10 platform is upgraded from Eclipse 3.4.2 to Eclipse 4.6.2. This includes SWT, OSGi, Jface and other re-lated Eclipse plugins. Notes, Designer, Embedded Sametime, and Connections Plugins now leverage Eclipse 4.6.2. The Domino server uses the OSGi bundles from Eclipse 4.6.2 for servlet and XPages access.

3) Embedded Sametime Upgraded To 901 By Default

The default version of the Embedded Sametime in the Notes Client has been upgraded to the 9.0.1 version by default, which eliminates the need for an add-on installer for Sametime.

4) The GSKit libraries are upgraded to Version for both client & server. GSKit is used for FIPs support, AES , 1024+ RSA, and Diffie-Hellman.

5) Japanese User Interface Update

An optional Japanese User Interface update includes fixes for high-impact localization defects. The 'IBM Notes Client 9.0.1 FP10 Japanese User Interface Update' requires 9.0.1 Japanese Client/Japanese MUI + FP10.

6) Add-on Installer for Notes CCM (Connections Content Manager)

An add-on installer for Notes CCM eliminates the need for special hotfixes but does require 9.0.1 FP10. This optional add-on allows a Notes client to show Libraries from CCM for use with email and calendar.

Note: The Notes CCM now supports the Dutch language in addition to previously supported languages For more details on CCM, see

Fixes contained in this Feature Pack

A plus symbol (+) before the SPR number indicates a fix for a regression bug. A regression bug is an issue that was introduced in a Maintenance Release but did not exist in previous releases of that code stream. For example, a bug that appears in 9.0.1 but did not exist in 9.0.0 is a regression.

If an APAR exists for the corresponding SPR, the APAR number will appear in parentheses next to the SPR number. For example, SPR #nnnnnnnnnn (LO12345).

The Fix List database on developerWorks contains the description of each fix in this Feature Pack, and indicates which platform(s) have been fixed.

9.0.1 Feature Pack 10 Preliminary Fix List descriptions:

Domino® Server

+YNABANLSUB - Fixed an error 4399 "Value is out of range" when running DeleteUser LotusScript. This was a regression introduced in 9.0.1 FP7

LRHG9T78WW (LO83614) - Fixed an HTTP crash in memory allocation when Sametime calls into Domino

MSKAAPEQ7M (LO92705) - Fixed a server crash during ID vault processing

PKIM9XSUTU (LO89753) - Fixed an issue where an SSL certificate validation does not allow partial chains

CRPOA64RNM (LO87648) - Fix an issue where Domino doclinks no longer convert to a usable format for iOS

JAQOAKRTUM (LO91921) - Fixed a Domino Server crash when processing user certificates in the Domino Directory

+JCARAQSJB6 (LO92949) - Fixed an SMTP issue which can cause malformed headers & prevent Internet mail delivery. This was a regression introduced in 9.0.1 FP9

MKIN9Z6UZ2 (LO90767) - Fixed an issue in the web server when processing graphic records

+PMGYAMDCQX (LO92318) - Fixed a Server crash due to a nullhandle of a note. This was a regression introduced in 9.0.1 FP9

RMAA9ENABR (LO78449) - Fixed a Domino Server crash on the Dircat task

TSUAANL4QT (LO92599) - Fixed the error "Value is out of range" in the NotesAdministrationProcess #DeleteUser

+YNABANLSUB (LO92582) - Fixed the error 4399 "Value Is Out Of Range" When Running Deleteuser Lotusscript. This was a regression introduced in 901FP8.

ASHHABSBHK (LO89599) - Fixed an issue where running Dbmt on an NSF file, intermittently leaves the ORIG file.

IISA9QEEAH (LO82584) - Fixed a Domino Server crash due to deadlock in the unread code

+TPON9N3EYE (LO81657) - Fixed an issue that resulted in a message being sent that dropped the 'body' content, resulting in a messaging being sent with only the disclaimer. This would occur with disclaimers enabled and a returnreceipt field set to 0. This was a regression introduced in 9.0.1.

LCAYAC7JU4 (LO89729) - Fixed an issue where delivery failure notifications are going dead in the rather than transferring to O365. Notes.ini parameter POSTMASTER_WITH_BRACKETS=1 needs to be added to enable the fix.

SLAEAGNLBF (LO91113) - Fixed an issue where autoprocessing of update notices took a long time. With this fix, if the buffer used for autoprocessing is maxed out on a server, auto-processing will stop and defer then work to the client. This fix is on by default but can be disabled by setting CSDisableSummaryCacheAutoStop=1

WHAM9KRHHH (LO80695) - Introduced a new dcontroller.ini (ControllerLogLocation) to control the log path for Domino QoS

Notes® Client

+ACVR9HHS55 (LO79672) - Fixed a Notes Client crash when sending mail to groups from recent contacts.This was a regression introduced in 8.5.3 and 9.0

YGAOAFXLAK (LO90918) - Fixed a Notes Client crashes when processing Cross Domain admin requests

ANIAAKYCNY (LO91979) - Fixed the Sametime Awareness icon to display for recipients' alternate name in a Memo

RUYAAM2QGC (LO92259) - Fixed an issue with meetings in summary view showing under wrong date, for federated calendar entries created in a different timezone

CSAOAKF4XX (LO91821) - Fixed a Notes Client startup performance issue caused by Smart Upgrade initialization in a VMWare VDI

JVEKAPMFWR (LO92748 ) - Fixed an issue where mail with Japanese characters are not shown properly on a PC with Chinese regional settings

+JVEKAQ5JAL (LO92827) - Fixed a roaming user issue that results in an empty "At Location" field when a user roams to another computer. This was a regression introduced in 9.0.1 FP7

+JVEKAQSGCC (LO92948) - Fixed an issue with shared, private on first use folder not working as expected in 9.0.1 FP9. Folder can not be viewed in the Designer. This is a regression introduced in 9.0.1 FP9.

+JVEKARBEP2 (LO93058 ) - Fixed an issue where the contents are not displayed after editing if a Richtext field contains an image and "Store contents as HTML and MIME" is enabled. This is a regression introduced in 9.0.1 FP9.

NDDC9J2LG5: (LO79942) - Fixed an issue where the URL in a forwarded email created using Win8.1 Tablet opens in the embedded browser and not the default OS browser

RREN9P9KWR (LO82098) - Fixed an issue where folders will not be expanded when the database is re-opened

+JVEKAR7J8F (LO93043 ) - Fixed an issue where public email groups do not expand. This is a regression introduced in 9.0.1 FP8

XLXZAFZCP2 (LO90940) - Fixed an issue where mails are displayed as unread in Notes after reading the mails on an IMAP client

+YGAOALLM2D (LO92139) - Fixed an issue where users are unable to send encrypted mail. This is a regression introduced in 9.0.1 FP8.


JVEKAPWDR8 (LO92793 ) - Fixed an issue in iNotes when forwarding an email from Google Chrome a lot of white space is added between the new and the forwarded document

+KRAUANKM6K (LO92572) - Fixed an iNotes issue where users are unable to attach two or more files using the Send Function. This was a regression introduced in 9.0.1 FP7

SDOY9LFER6 (LO81030) - Fix an iNotes issue that fails to open attachment on forwarding or replying message on IE with iNoes ActiveX file attachment utility

"IBM Notes Client 9.0.1 FP10 Japanese User Interface Update" Specific SPR's

TITH9GEC9K (LO79111) - Fix an issue where a mail meeting invitation does not have the day in the subject

THIO9KW2FH (LO80689) - To Do From Japanese Notes does not contain the correct Japanese date format characters

SKAI9UW3JX - Fix an issue where the 'IDEXTRAFILENAME' parameter does not work in the Japanese Notes Client

THIO9L22J3 (LO80781) - Fix an issue where the date formats in the subject of Meeting invitations and Group To Do's are SBCS instead of DBCS

TSUA8RA5JW - Fixed an issue where the property box of database is truncated in the Japanese Client

CKUAAHB8PN (LO91232) - Fix an issue where it takes long time to open a DJX address list

Beware not to abort compact -REPLICA -RESTART

Due to some of our DBs encountered "Unable to extend an ID table" error, I have scheduled compact -REPLICA -RESTART to run on thes...