bb.donnay-software.com

Victorio,
If you want to speed up your massive data file processing, you have to create a multi-threaded application. In that case, the size of RAM memory is of no matter. You have to allocate to each thread a small file part or small array, and synchronize and join the results of all threads. By my experience, Xbase++ works very slow with large arrays. I transferred all that processing to C and that works very very fast. In general, multi small parts processing is crucial in massive data processing.
You have a 24 CPU server, why don't you use it in the right way?

hi,

You have a 24 CPU server, why don't you use it in the right way?

i don´t think that 24 CPU will help when using Xbase++ as Xbase++ can only use 1 x CPU
Multi-Threading, under Xbase++, does not use multi-CPU . it use same Time-Slice of 1 x CPU

Jimmy,

Xbase++ can only use 1 x CPU. Multi-Threading, under Xbase++, does not use multi-CPU.

This is not true at all. You should read the SetLogicalProcessor() and GetLogicalProcessorCount() functions. This is what Alaska documentation says:

SetLogicalProcessor( [<nLogicalProcessor>] ) --> nOldLogicalProcessor

Parameters
<nLogicalProcessor>
The optional numeric parameter <nLogicalProcessor> specifies the logical processor the application is assigned to.

Return
The numeric value returned identifies the logical processor where the application was previously assigned to.

Description
A processor is generally a unit able to execute operations. A Logical Processor is the term used to represent the fact that today:

- A processor can be virtual (vmware, MS hyper-v).
- A physical CPU/Processor can have multiple cores/processors.
- A hyper-threading capable core is divided into two logical processing units.

// Assign the Process randomly to a logical processor unit

PROCEDURE Main
SetLogicalProcessor( RandomInt(GetLogicalProcessorCount()) )
RETURN

In a multi-threaded application you can use SetLogicalProcessor() in each thread. I tested that and it works well. Besides, you can do the same task with respective WinAPI functions. With WinAPI functions you can also monitor the execution times of all processors and make your own plan of their load/assignment.

hi,

This is not true at all.
You should read the SetLogicalProcessor() and GetLogicalProcessorCount() functions.
This is what Alaska documentation says:

WHERE DO ALASKA SAY IT
do you have any Sample to show the Effect

https://doc.alaska-software.com/content ... cessor.cxp

// Assign the Process randomly to a logical processor unit

Hi,

SlavkoDam wrote: ↑Thu Jun 27, 2024 1:24 am Jimmy,

Xbase++ can only use 1 x CPU. Multi-Threading, under Xbase++, does not use multi-CPU.

This is not true at all. You should read the SetLogicalProcessor() and GetLogicalProcessorCount() functions. This is what Alaska documentation says:

Xbase++ does not use multi-CPU.

When you change the processor every thread is using the same processor. You can't use multiple processors at the same time.

I did a test as this:
Start thread1 on cpu 1: you see the activity on cpu 1.
Start thread2 on cpu 2: you see the activity on cpu 2. Also the activity of thread one is now on cpu 2.
Start thread3 on cpu 3: you see the activity of thread 1 and thread2 and thread3 on cpu 3.

Also they don't say in the Alaska docs that you can use multiple cpu's.

I'm changing the cpu at the start of each thread in my application. On a server with 30 users connected via terminal services this give a rather well balanced use of the available cpu's on the server. I reserve 2 cpu's for two specific tasks.
On a 16 processor system.
cpu 14 is for my soap and rest-api server.
cpu 15 is for the creation of reports.

This way the users don't notice some loss of speed when a big report is generated or there are a lot of requests on the soap or rest-api server.

I am using multiprocessor , all 24 CPUs, but not as thread and as external exe modules, where modules run on 1,2,3,4,--- CPUs.
I have main program, and this controlling start modules , divide process to 24 parts and every part send to module on own CPU.
Main program controlling how many modules still running (every module EXE create temporary file runcpu01.txt,runcpu02.txt... etc, and after processing this temporary file delete) When no temporary file exist, main program know, process ended.
This system works fine, it doesn't cut the processing down to exactly 1/24 of the time, but it's close to it
but I thought that if I run a thread on another CPU, in one main program (EXE) that each thread cannot run in parallel, each on its own CPU, I have to try it, it would be interesting, and much easier.
here is how use every CPU my program , I wrote about it several years ago.
Important is, that this system need more memory, minimal 2GB on every CPU, for 24 CPU enough 48GB, when RAM is for example 8GB, procesing is slowly because writing to HDD,(or SSD) as virtual memory.

Hi,

These WinAPI functions allow scheduling a thread(s) to run on a single or multiple processors.

GetProccesAffinityMask() obtains the processors that a process and its threads are allowed to run on.
SetThreadIdealProcessor() runs the thread on the specified processor.
SetThreadAffinityMask() forces a thread to run on a specific subset of processors.
SetProcessAffinityMask() forces all threads of a process to run on a specific subset of processors.

hi,

we know GetProccesAffinityMask() and SetThreadAffinityMask() and had the Discussion long before when Alaska had disable it after v1.6
we have Sample Source\samples\solution\smp\smprun.prg to switch CPU but ALWAYS hole App will run on same CPU

we can use Concept from Chris or Victorio and use multible App each on different CPU but there is NO WAY to use a Thread on other CPU than Main under Xbase++.

You are dreaming about Xbase++ MT, like using 64 Bit

Slavko, we all say the same.

Learned that long ago the hard way on multi socket Citrix system -
xbase only runs on single core of multi core processors and does not provide
anything to expand processing across multiple cores within single process.

They had some multi core experiment in 1.6 but it was plagued by suboptimal reliability and performance and was dumped in 1.7

And if you do not randomize CPU selection at startup it will run on core 0 of multicore CPU all the time.
Not to say there is no information on selection of "P" vs "E" cores for xpp program too.

Only way to harness is to split application into multiple exe that do processing on their own
but communicate back to main process like OP Claudio did.

At that time it is just simpler to use more modern and faster language for processing and put it onto application server to do heavy lifting.

SlavkoDam wrote: ↑Thu Jun 27, 2024 12:27 pm Hi,

These WinAPI functions allow scheduling a thread(s) to run on a single or multiple processors.

GetProccesAffinityMask() obtains the processors that a process and its threads are allowed to run on.
SetThreadIdealProcessor() runs the thread on the specified processor.
SetThreadAffinityMask() forces a thread to run on a specific subset of processors.
SetProcessAffinityMask() forces all threads of a process to run on a specific subset of processors.