New library PowerPar for parallel programming

[SlavkoDam]

Hi,

I developed a new PowerPar library for parallel (concurrent) application development, with the following features:

- Multi-processor-threading - Execute multiple processes (UDFs) on multiple logical processors in parallel. Each process is executed in its own thread.
- Process synchronization - Create events, mutexes, semaphores and timers.
- Process communication - Create slots and pipes.

You can download documentation of PowerPar library on this link:

https://www.sd-softdesign.com/download.htm.

[SlavkoDam]

Hello,

I added a new functionality in the library. A function to run in parallel can be not only from the current XPP application, but also from an external XPP application. This is achieved with an additional parameter in the ProcExe() function, which defines the name of the application in which is the function to run. The default value is the current application.

In this way the main application will be a controller and the external application will be its service. And a service can run in parallel thread on a separate processor. This is much simpler and faster then to create and maintain an XPP service application, for the same purpose.

[Victorio]

Hi Slavko,

I am interesting about Powerpar, how much CPU support this ? I am using multicpu system in Xbase but at this moment I am stop at 32 CPU, no more with 32 application.
Have you support for more than 32 cpu ? (I need 48)

Viktor

[SlavkoDam]

Hi Victorio,

I suggest you to download documentation, test version, and demo program for PowerPar, and try to get info you are asking for your system. Test NUMA and CPU function, and look the CPU numbers in demo program. I never used system with 48 CPUs.

That article on ILX portal gives only theoretical answer and not practical. I suppose you don't want random allocations of CPUs, but want to schedule them to processes as you wish. If you use the proposed Alaska method, it may occur that multiple of your processes run on the same CPU, and you didn't want that initially.

[Victorio]

Yes, you are true, I do not want randomly allocate CPU, I must have control on which CPU running proces because after processing I join multiple data from all CPUs to one with main control applicaton.

Alaska or some other source ?, that _sysSetCPU cannot process larger number (binary), as 32 bit in 32 bit application,this is Xbase++.
I write small testing program and in fact when set larger than 32CPU, crashed with error.
If you use also _syssetcpu, that also have limit 32CPU.

I will try your demo,

My application which use my client processing many and very large databases, one searching time is several hours. Therefore, we have requested the server operator to create a virtual server with 48CPUs, which will significantly shorten processing. Also this need more RAM, every CPU need about 2 or 3GB, summary on server is 48GB RAM. I hope this will enough to secure processing without memory crashes.

[k-insis]

There is point in processing time and load when xbase is just too slow and you should really consider offloading that processing to something faster than xbase compiled program.

Even with slicing task into pararell executed xbase exes those might hardly reach 20-25% of individual vCPU load and will just saturate IOPS of data system.

My application which use my client processing many and very large databases, one searching time is several hours. Therefore, we have requested the server operator to create a virtual server with 48CPUs, which will significantly shorten processing. Also this need more RAM, every CPU need about 2 or 3GB, summary on server is 48GB RAM. I hope this will enough to secure processing without memory crashes.

[Tom]

If you use the proposed Alaska method, it may occur that multiple of your processes run on the same CPU, and you didn't want that initially.

But this is a task without a solution anyway. Load-balancing and predictions of utilization on an OS like Windows never work efficiently, since they are based on a snapshot an things change within a nanosecond. All efforts to develop a smart system for this ended with the recommendation to randomly select a processor. Alaska is simply right with this. The probability that multiple people will have to work on the same processor must be accepted. The software then should offer the option to change the processor (for the entire application, if your library is not in use). You will end up with the same problem if you have ten threads running on difference CPUs and another user starts the same shit.

The only way out for this is to let the OS decide which ressources should be used how, but this won't work with Xbase++-applications.

[SlavkoDam]

Well, do as you like and as you know. No one force you to do the other way.

Anyway, PowerPar allows programmatically and explicitly multi-CPU usage in Xbase++. No other software can do that. Alaska should be proud that this can be achieved in Xbase++, despite the fact that PowerPar broke Alaska one-CPU limit per app.

The main purpose of PowerPar is not multi-user app. But, there is no problem for PowerPar working with multi-users. Each process for each user is started in its own thread on the specified CPU, all in parallel, and this solution works stable and safe. Any OS recommendation for multi-CPU usage is utopia an unpredictable, no one rely on that.

[skiman]

Hi,

If you can check what the load is for each CPU it makes sense. Otherwise if for a thread there is a fixed cpu number assigned, you can't know if this CPU already has a huge load or not.

I agree with Tom, randomly use the different CPU's is a good solution. I change the CPU if another thread is started. I know this is for the complete application. This way all cpu's are used randomly. When looking at a server with 25 users and 10 cpu's the load is quite split according them all.

I don't use cpu 0 and use the highest cpu only for reporting. Since reporting can have a heavy load, it won't slowdown other users.

This system is implemented since years and is working very well.

[SlavkoDam]

Hi,

Yes, I can check the load of each CPU, but that is not the point. Your system has nothing to do with PowerPar library. Your system is multi-users oriented, PowerPar is multi-process-CPUs oriented. It is not the same, but you don't understand that.

As I wrote on the German Forum, you guys don't have the wright applications for Powerpar to use for. You don't know anything about true parallel programming and multi-computer (supercomputer) systems which are being used for decades for special expert systems for massive data processing. You don't have that kind of applications, you are developing simple commercial applications. So, drop any discussion about PowerPar since whatever you say is wrong and not adequately. You just don't understand it and its purpose. Keep track with your current systems and what you know well. You don't know for better and it is better for you.