Oplocks will kill us all

[bwolfsohn]

Here's the problem we're facing, and it looks like it will affect all/most of you.. I'm looking at this from both a revenue POV as well as a technical POV.

The new quickbooks is SQL based.
Our xbase++ programs are not sql based, and it's not feasible to convert them.. (feasible being defined in months, not years)

If all the oplocks entries are made on a computer, the new quickbooks crashes (we've been told it crashed on a network).

The only way around this we've found is to set-up a virtual XP machine inside windows 7 and make the entries in the virtual machine. But, this is only available to higher $$ windows 7 versions: professional, ultimate and enterprise.

Questions:
1. Any other approaches to the conflicts i mention here ?
2. Theoretically, doesn't oplocks entries need to be made even on a single user machine ??
scenario: 3 separate xbase++ programs which share some databases. if data it entered in 1 and "saved by xbase++", is it visible in the other 2 programs if no oplocks entries are made ?

[bwolfsohn]

A possible solution would be if oplocks could be user-dependent, i.e. registry entries that execute only when that user is logged on.. i.e. a quickbooks user or a xbase++ user..

But, i don't think they are..

[skiman]

Hi Brian,

I agree that we have to be concerned about this problem. Arctica was nice three years ago. You also saw the demo in Boise.

Now I think we will never see it. Last official announcement was in December 2009. There was promised a podcast in the beginning op 2010. And without Arctica, Xbase++ seems to have a really big problem with data integrity.

Since a few months we are looking to other possibilities. It seems as Xbase++ won't be part of it. I have to decide now, if we want a new version is another language of our software at the end of 2012. This will cost me a lot, but I can't wait forever.

[Cliff Wiernik]

My only exposure to the Virtual XP mode on Windows 7 was less than desirable. Did not work very good with Autocad 2004.

[Tom]

Our software runs at > 1,000 sites and on more than 5,000 workstations, and we don't care for oplocks at all. We did as Steffen first mentioned this possible solution for data problems (in 2004, as I remember), and we found out that almost didn't make any difference. It just caused trouble, nothing else, and it didn't solve a single problem. If a customer faces network problems, we install ADS. That's it. Runs perfect.

The last news I heard about the Arctica project were about problems they have with the documentation. Steffen mentioned some weeks ago, Artica will be delivered this summer.

[rdonnay]

I agree that ADS is the only real solution for now.

I talked to Steffen a few weeks ago, and he confirmed that Arctica is ready except for documentation and installation issues.

Hopefully, we will all have reason to get together again later this year.

[bwolfsohn]

We use ADSserver at a number number of our larger clients with excellent results...

ADSlocal is used at all our other installations, and seems to be subject to the vagaries of oplocks.

But, from a revenue model perspective, not an ideal situation for the small mom and pop who's only using 1 or 2 computers.

Tom, Do i userstand correctly, you make no entries for oplocks and let ads handle everything ??

we're always making oplocks entries whether we use ads or not.

Brian

[Tom]

Hi, Brian.

What I was trying to say: We don't care for damned oplocks with (30 percent) and without (70 percent) ADS. They simply don't make a difference, and all settings we tried just slowed down everything.

In ADS/Server situations, oplocks are not necessary even if they would help without ADS. In ADS situations, files are never opened concurrently - the ADS server just opens them once. Oplocking errors only occur when files are used concurrently.

[rdonnay]

This is from Steffen Pirsig (received today) :

Hi Brian,

just to clarify some of the terms and problems you
are refering in your post.

1.) OPLocking needs to be disabled to have your application
working properly since Windows 2000 SP1 and later. However
with Samba you can configure OPLocking to be enabled/disabled
on a per share base which is perfect.

2.) The issue you have with Windows 7 are definitive not OPLocking
related for a simple reason: W7 doesn't use OPLocking anymore
such as previous versions did. However with Windows 7 or Vista in
a peer network or with a Windows Server 2008 there is another problem
because of how SMB2 works. More details and a solution you can find
here www.alaska-software.com/smb2

3.) The SMB2 problem is a design decision by MS, you can simple
check this by yourself in a network. Open
notepad \\yourserver\yourshare\test.txt, write some text, save it.
Now look at this file from a different workstation in your network.
Use the command shell for that and perform dir. Okay the file is there,
and the size is identical to the size as it was safed with notepad.
Now again, edit the file on your workstation A, safe it. Then go to
workstation B and make dir - look at the file size, do dir for more
then 10 seconds and you will see how this information gets updated
only each 10 seconds. MS did a great trick, with Windows 7: File meta
information such as filesize is updated only each 10 seconds between
clients in a network. This increases speed but of course makes concurrent
file usage impossible. The under 2.) provided MSI package just
configures the timeout to be 0 at the workstation this way file
meta information is with new operating system releases as correct
as it was always with old operating system releases.

4.) SQL is language not a solution to your problem. What you
mean is Client/Server. And of course, if you are doing client
server you are avoiding issues such as OPLocking or SMB2. But
you will have other issues - such as proper configuration of
your network, or proper configuration of your server. Anyway,
Tom is right - ADS is the most cost effective solution right
now.

5.) We are aware that Artica is beyond its schedule and we
are working hard to ensure that the preview/beta can start
as soon as possible. We are right now with packaging and
restructing of the documentation which of course was again
a much more larger workload than initially anticipated. But
we are also sure that Arctica will pay off for your customers
for a simple reason: The migration to SQL client/server, specifically
the PostgreSQL server becomes so easy that even large applications
can be moved to PostgreSQL in just one day.

Just some comments on your posting
With best regards
Steffen F. Pirsig
Alaska Software Inc.

PS: thx Roger for posting this
2PS: thx Tom for bringing that issue to my attention
3PS: thx for your patience with Arctica, we are doing our best

[Auge_Ohr]

hi,

interessing that Statment from Alaska but i do not agree about SMB2.

i did the Sample with "Notepad" and can not agree when use:

Code: Select all

Notepad \\SERVER\TEST\MYTEST.TXT

it will FAIL if you use NET USE X: \\SERVER\TEST with

Code: Select all

Notepad X:MYTEST.TXT

while Vista / Win7 "listen" to Port 139 SMB1 AND Port 445 SMB2
you are not allowed to use "Lanmanager" or "NetBIOS" over Port 139

even Cl*pper can run on SMB2 over Port 445 so SMB2 is not a Problem !!!
here some Testcode, with modification for SMB2 and Xbase++ v1.9.355 SL1

Testcode can run with Cl*pper, harbour and/or Xbase++

Code: Select all

#include "Directry.ch"
#include "common.ch"

#IFDEF __XPP__
   #include "DLL.ch"
   #include "OS.ch"

   #include "Dbfdbe.ch"
   #include "Ntxdbe.ch"
   #include "Foxdbe.ch"
   #include "Cdxdbe.ch"
   #define MSG_DBFDBE_NOT_LOADED   "Database-Engine DBFDBE nicht geladen"
   #define MSG_NTXDBE_NOT_LOADED   "Database-Engine NTXDBE nicht geladen"
   #define MSG_DBFNTX_NOT_CREATED  "DBFNTX Database-Engine;konnte nicht erzeugt werden"

PROCEDURE dbeSys()
   SET COLLATION TO GERMAN
   IF !DbeLoad("DBFDBE",.T.)
      ALERT(MSG_DBFDBE_NOT_LOADED,{"OK"})
   ENDIF
   IF !DbeLoad("NTXDBE",.T.)
      ALERT(MSG_DBFDBE_NOT_LOADED,{"OK"})
   ENDIF
   IF !DbeBuild("DBFNTX","DBFDBE","NTXDBE")
      ALERT(MSG_DBFNTX_NOT_CREATED,{"OK"})
   ENDIF

   IF !DbeLoad("CDXDBE",.T.)
      ALERT("Database-Engine CDXDBE nicht geladen",{"OK"})
   ENDIF
   IF !DbeLoad("FOXDBE",.T.)
      ALERT("Database-Engine FOXDBE nicht geladen",{"OK"})
   ENDIF
   IF !DbeBuild("FOXCDX","FOXDBE","CDXDBE")
      ALERT("FOXCDX Database-Engine;konnte nicht erzeugt werden",{"OK"})
   ENDIF
   //
   // using Cl*pper Comix/SixDrive
   //
   DbeInfo( COMPONENT_DATA, FOXDBE_CREATE_2X, .T. ) 
   DbeInfo( COMPONENT_DATA, FOXDBE_LOCKMODE , FOXDFBE_LOCKMODE_CLIPPER ) 
   //
   DbeInfo(COMPONENT_ORDER, CDXDBE_MODE,CDXDBE_FOXPRO2X )
//
// will give 8999 with Win7 SP1 "RC" 6.1.7601.17105
//
// DbeInfo(COMPONENT_ORDER,DBE_LOCKMODE,LOCKING_EXTENDED)
   DbeSetDefault( "FOXCDX" )

   RETURN

#ENDIF

PROCEDURE Main(cMax)
*LOCAL cHost := "\\A-pc\_e\0"
LOCAL cHost  := "\\A-hp\ALASKA\0"
LOCAL cFile  := "TESTOPS.DBF"
LOCAL cNode  := TRIM(NETNAME())                  // Cl*pper need TRIM()
LOCAL i
LOCAL nStart
LOCAL nStop
LOCAL nRec
LOCAL nper   := 0
LOCAL nMax   := 1000
LOCAL nLast  := 0
LOCAL nError := 0
LOCAL nCount := 0
LOCAL cVersion

LOCAL Use_NTX := .F.           // USE NTX or CDX

DEFAULT cMax TO 1000

   IF PCOUNT() > 0
      IF VAL(cMax) > 0
         nMax  := VAL(cMax)
      ENDIF
   ENDIF

   SET DECIMALS TO 4

   IF .NOT. FILE(cHost+"\"+cFile)
      #IFDEF __XPP__
         MSGBOX("File not found")
      #ELSE
         Alert("File not found")
      #ENDIF
      QUIT
   ENDIF


   #IFDEF __XPP__
      IF FILE(cHost+"\"+STRTRAN(cFile,".DBF",".CDX"))
         Use_NTX := .F.
      ENDIF
   #ELSE
      //
      // Cl*pper v5.2e
      //
      Use_NTX := .T.
   #ENDIF


   CLS
   ? "SMB2 Demo, "+LTRIM(STR(nMax,10,0))+" to add records, using "+IF(Use_NTX,"NTX","CDX")+" please wait ..."
   #IFDEF __XPP__
      IF OnOSVersion() > 5
      // thx goes to
      // Andreas Gehrs-Pahl
      //
         ? "MRXSMB20.sys",cVersion := GetFileVersion('C:\Windows\System32\Drivers\MRXSMB20.sys')
         ? "MRXSMB10.sys",cVersion := GetFileVersion('C:\Windows\System32\Drivers\MRXSMB10.sys')
      ELSE
         ? "MRXSMB.sys",cVersion := GetFileVersion('C:\Windows\System32\Drivers\MRXSMB.sys')
         IF FILE("c:\WINDOWS\system32\drivers\rspndr.sys")
            ? "Responder ",cVersion := GetFileVersion('c:\WINDOWS\system32\drivers\rspndr.sys')
         ELSE
            ? "XP Responder 'c:\WINDOWS\system32\drivers\rspndr.sys' missing"
            ? "see http://support.microsoft.com/kb/922120 ( for XP SP2) "   
            ? "XP SP3 see http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=922120&kbln=de"
         ENDIF
      ENDIF
   #ENDIF
   //
   // Testtime USE
   //
   nStart := SECONDS()
   #IFDEF __XPP__
      IF Use_NTX
         USE (cHost+"\"+cFile) SHARE VIA "DBFNTX"
      ELSE
         USE (cHost+"\"+cFile) SHARE VIA "FOXCDX"
      ENDIF
   #ELSE
      USE (cHost+"\"+cFile) SHARE
   #ENDIF

   IF NetErr()
      #IFDEF __XPP__
         Msgbox("USE SHARE Error")
      #ELSE
         Alert("USE SHARE Error")
      #ENDIF
      QUIT
   ELSE
      nStop  := SECONDS()-nStart
      ? ""
      ? "USE DBF SHARE "+LTRIM(STR(nStop,10,4))+" Sec."
      #IFDEF __XPP__
         IF Use_NTX
            SET INDEX TO (cHost+"\"+"TESTOPS1.NTX"),;
                         (cHost+"\"+"TESTOPS2.NTX"),;
                         (cHost+"\"+"TESTOPS3.NTX"),;
                         (cHost+"\"+"TESTOPS4.NTX")
         ELSE
            SET INDEX TO (cHost+"\"+"TESTOPS.CDX")
         ENDIF
      #ELSE
         SET INDEX TO (cHost+"\"+"TESTOPS1.NTX"),;
                      (cHost+"\"+"TESTOPS2.NTX"),;
                      (cHost+"\"+"TESTOPS3.NTX"),;
                      (cHost+"\"+"TESTOPS4.NTX")
      #ENDIF
   ENDIF
   IF nStop > 1
      ? "if longer than 1sec. use http://support.microsoft.com/kb/150384"
   ENDIF

   //
   // try to use "natural" Order, Index will (must be) updated
   //
   #IFDEF __XPP__
      IF Use_NTX
         SET ORDER TO 0
      ELSE
         ORDSETFOCUS( 0 )
      ENDIF
   #ELSE
      SET ORDER TO 0
   #ENDIF

   nLast := TESTOPS->( LASTREC() )
   //
   // Test APPEND BLANK
   //
   ? ""
   ? "starte DBF APPEND BLANK, please wait ..."
   nStart := SECONDS()
   FOR i := 1 TO nMax
      TESTOPS->( DbAppend() )
      IF NetErr()
         TONE(1234)
         nError++
      ELSE
         nRec := TESTOPS->(RECNO())
         IF TESTOPS->(DbRLock(nRec))
            REPLACE TESTOPS->TESTSTRING   WITH STRZERO(i) // Replicate("A",10)
            REPLACE TESTOPS->TESTNUM      WITH RECNO()
            REPLACE TESTOPS->TESTDATE     WITH DATE()
            REPLACE TESTOPS->TESTTIME     WITH TIME()
            REPLACE TESTOPS->TESTNODE     WITH cNode

            TESTOPS->(DbRUnlock(nRec))
            //
            // need for Win7 "original" if other have SP1 RC
            //
            TESTOPS->(DbSkip(0))
         ELSE
            #IFDEF __XPP__
               Msgbox("DbRLock("+LTRIM(STR(nRec))+") Error")
            #ELSE
               Alert("DbRLock("+LTRIM(STR(nRec))+") Error")
            #ENDIF
         ENDIF
         IF (i % (nMax/100)) == 0
            nStop := SECONDS()-nStart
            nper  := nStop/i
            @ 8,0 Say "Prozent "+LTRIM(STR(i/(nMax/100),10,0 ))+"% per Record "+LTRIM(STR(nper))+" Sec."
         ENDIF
      ENDIF
   NEXT
   nStop  := SECONDS()-nStart

   ? "APPEND BLANK Sec."+LTRIM(STR( nStop ))
   nper  := nStop/nMax
   ? "per Record "+ LTRIM(STR(nper))+" Sec."

   IF nper > 0.04
      IF nper > 0.5
         ? "you have a BIG Performance Problem"
      ELSE
         ? "if Time are to long use http://support.microsoft.com/kb/825433"
      ENDIF
   ENDIF
   ? "LASTREC() and RECCOUNT()",nRec := LASTREC(), RECCOUNT()

   #IFDEF __XPP__
      IF Use_NTX
         SET ORDER TO 2
      ELSE
         ORDSETFOCUS("MYNUM")
      ENDIF
   #ELSE
      SET ORDER TO 2
   #ENDIF
   ? ""
   ? "Index Order  "+OrdSetFocus()
   ? "SEEK for     "+LTRIM(STR(INT(nMax+nLast)))
   SEEK(nMax+nLast)
   IF FOUND()
      ? "FOUND "+LTRIM(STR(nMax+nLast))+" LastRec "+LTRIM(STR(nRec))
   ELSE
      ? "hm ...       "
   ENDIF

   #IFDEF __XPP__
      IF Use_NTX
         SET ORDER TO 1
      ELSE
         ORDSETFOCUS("MYSTRING")
      ENDIF
   #ELSE
      SET ORDER TO 1
   #ENDIF
   ? ""
   ? "Index Order  "+OrdSetFocus()
   ? "SEEK for     "+LTRIM(STR(INT(nMax/2)))
   SEEK(STRZERO(nMax/2))
   IF FOUND()
      ? "FOUND "+LTRIM(STR(nMax/2))+" Record  "+LTRIM(STR(RECNO()))
   ELSE
      ? "hm ...       "
   ENDIF

   //
   // try to use "natural" Order
   //
   #IFDEF __XPP__
      IF Use_NTX
         SET ORDER TO 0
      ELSE
         ORDSETFOCUS( 0 )
      ENDIF
   #ELSE
      SET ORDER TO 0
   #ENDIF

   TESTOPS->(DbGoTop())
   ? ""
   ? "COUNT TO nCount FOR "+cNode
   nStart := SECONDS()

   COUNT TO nCount FOR TESTOPS->TESTNODE = cNode NEXT nMax+nLast

   nStop  := SECONDS()-nStart
   ? LTRIM(STR(nCount))+" Records "+LTRIM(STR(nStop,10,4))+" Sec. "+LTRIM(STR(nStop/nCount,10,4))+" per/Rec"

   CLOSE DATABASE

*   IF (nRec % (nMax+nLast)) == 0
*      WAIT "is it "+LTRIM(STR(nRec))+"? press any key ..."
*   ELSE
*      WAIT "seems NOT all Records in DBF, only "+LTRIM(STR(nRec))+" ? press any key ..."
*   ENDIF

   ? ""
   IF nError > 0
      WAIT "seems NOT all Records in DBF, only "+LTRIM(STR(nRec))+" ? press any key ..."
   ELSE
      WAIT "is it "+LTRIM(STR(nRec))+" ? press any key ..."
   ENDIF

   CLS

RETURN

#IFDEF __XPP__

FUNCTION NETNAME()
LOCAL nDll, cName := SPACE( 255 ), nSize := 255, cReturn := ''
   nDll := DllLoad( "kernel32.dll" )
   IF nDll <> 0
      IF !EMPTY( DllCall( nDll, DLL_STDCALL, "GetComputerNameA", @cName, @nSize ) )
         cReturn := LEFT( cName, nSize )
      ENDIF
      DllUnload( nDll )
   ENDIF
RETURN ( cReturn )

#ELSE
   // Clipper v5.2e does have NetName()
#ENDIF

FUNCTION MyCount( cNode )
LOCAL nCount := 0

  DBEVAL( {|| nCount++ },{|x| TRIM(TESTOPS->TESTNODE) = cNode}, , , , .T. )
*  TESTOPS->(DbGotop())
*  DBEval( {|| nCount := nCount + 1}, {|| TESTOPS->TESTNODE = cNode},,,, .F. )

   ? cNode, nCount

RETURN nCount

Function GetFileVersion(cFile)
LOCAL cFileName := GetShortFileName(cFile)
LOCAL nStartAt  := 0
LOCAL nInfoLen  := DllCall("Version.dll", DLL_STDCALL, ;
                           "GetFileVersionInfoSizeA", cFileName, @nStartAt)
LOCAL cVersion  := Space(nInfoLen)
   if nInfoLen > 0 .and. ;
      DllCall("Version.dll", DLL_STDCALL, "GetFileVersionInfoA", ;
              cFileName, 0, nInfoLen, @cVersion) # 0
      cVersion := upper(cVersion)
      if At("FILEVERSION", cVersion) == 0
         cVersion := StrTran(cVersion, chr(0) + chr(0), '>[@]<')
         cVersion := StrTran(cVersion, chr(0), '')
         cVersion := upper(StrTran(cVersion, '>[@]<', chr(0)))
         if (nStartAt := At("FILEVERSION", cVersion)) > 0
            nStartAt += 13
         endif
      elseif (nStartAt := At("FILEVERSION", cVersion)) > 0
         nStartAt += 12
      endif
      if nStartAt > 0
         nInfoLen := At(chr(0), cVersion, nStartAt)
         cVersion := substr(cVersion, nStartAt, nInfoLen - nStartAt)
      else
         cVersion := ''
      endif
   endif
return (alltrim(cVersion))

Function GetShortFileName(cLongName)
LOCAL cShortName := Space(255)
LOCAL nNameLen   := 1
   if empty(cLongName)
      return ('')
   else
      if Set(_SET_CHARSET) == CHARSET_OEM
         cLongName := ConvToAnsiCp(cLongName)
      endif
      nNameLen := DllCall("Kernel32.DLL", DLL_STDCALL, "GetShortPathNameA",;
                          cLongName, @cShortName, len(cShortName))
   endif
return (left(cShortName, nNameLen))

STATIC FUNCTION OnOSVersion()
LOCAL cVersion := OS(OS_VERSION)
LOCAL nVersion := 0
LOCAL nPosi

   nPosi := AT(".",cVersion)
   IF nPosi > 0
      nVersion := VAL(SUBSTR(cVersion,1,nPosi-1))
   ELSE
   ENDIF

RETURN nVersion

ad. Test Source was not "lates" Version ... here with GetFileVersion()