I have a powerful workstation for running copernic server and rescaning the folders for changes takes more than 3 weeks for 4 million files.
i'm sure - increasing CrawlingThreadPoolSize to 4 or 8 would now harm my machine
regards
Andreas Korntheuer
0 Votes
12 Comments
Sorted by
L
LUST & Partners Jacky d'Hoestposted
almost 3 years ago
I found out that you can change the crawling value to whatever value you want when you first quit CDS, second delete the queue and third change the value in the xml. You can then restart CDS and start an update. Even "12" was possible. Scanning is then superfast. Downside is that sometimes CDS is then unstable, meaning it freezes at 99 % of indexing. You can then only restart CDS. The value of extraction can never be higher then "4".
2 Votes
A
Alexander Borouhinposted
about 3 years ago
Currently both thread pools seem to be limited by 4 threads max. At least, 4/4 configuration works for me, anything higher gets reset to 4/4. But still it is very low limit. While initial indexing of ca. 1 million documents takes days, and each index update - hours, I'd prefer to utilize all the 16 cores to the maximum to speed up this process, scheduling the indexing time so that it doesn't affect other tasks.
1 Votes
R
Roeland Mertensposted
over 3 years ago
What I report
now will sound a bit dumb – but here it is.
In total I
repeated the test to modify both parameters 4 times. The first 3 times the values
“2/4” were immediately overwritten to “1/2” upon restart of the app. I’ve done
another test now once again … and the”2/4” values stayed on!
Go
figure …
The only difference
with the 4th test may be that at long last the initial indexing had
been finalized; so the app was busy indexing the first 3 times, and idle at the
4th time. I have no other explanation.
Thx for all the replies.
0 Votes
M
Mirco Persechinoposted
over 3 years ago
Hi Mertens,
Make sure you completely exit Copernic Desktop Search BEFORE making any changes in the config file. I think that could be the issue here. So completely exit CDS, then make changes to your config file extractionthread and then start Copernic. It should be okay.
0 Votes
L
LUST & Partners Jacky d'Hoestposted
over 3 years ago
My configuration runs like this :
Only If you go higher the values are adjusted.
0 Votes
M
Mirco Persechinoposted
over 3 years ago
Did you try 2 and 2, start and see if it returns back to 2 and 1, then try with 2 and 3.
It shouldn't behave like that. I'll make a few checks with our developpers.
0 Votes
R
Roeland Mertensposted
over 3 years ago
Not so. As I have just said, with the settings as suggested, at the next start it overwrites both settings to those "2" and "1" values again.
0 Votes
M
Mirco Persechinoposted
over 3 years ago
Hi Mertens,
"ExtractionThreadPoolSize" can be set to a maximum of 4.
"CrawlingThreadPoolSize", maximum 2.
If you exceed those numbers, the config file will simply reset to its default value.
1 Votes
R
Roeland Mertensposted
over 3 years ago
Same problem here.
I use a high-end PC (with i.a. an AMD Threadripper Ryzen CPU with 24 cores). With the indexing performance set at unrestricted, the indexing process stays well below 3% CPU load all the time. That is of course ridiculous for this kind of PC and the time waste is enormous.
Remarkably, after manually adjusting the parameters ("CrawlingThreadPoolSize" and "ExtractionThreadPoolSize"), at each startup my settings are immediately overwritten and reset to those meager "2" and "1" values again.
Ask me why ... !
0 Votes
A
Andreas Korntheuerposted
over 3 years ago
i think, this is not the main problem in my case.
the process Copernic.Plugins.PluginsService.exe is still at 0% cpu usage
if copernic do a ocr rescaning this process should be more than 0% cpu usage
the process Copernic.DesktopSearch.exe use about 15% cpu usage.
if there are more than 2 Crawling Threads the indexing should be faster.
regards
0 Votes
L
LUST & Partners Jacky d'Hoestposted
over 3 years ago
If you are using the OCR-plugin, that is the reason, why it takes so long. I have an index with 1.300.000 items and with the OCR-Pugin a full update takes more than three weeks. Without it, it still takes 10 days and the reason for that is the fact that CDS every time reindexes all emails. It does not restrict itself to the new or changed emails ...
I complained several times about that without any succes.
2 Votes
A
Andreas Korntheuerposted
over 3 years ago
please reply - rescaning the folders for changes should not take 3 weeks
Hi Team,
why there is a limitation to the CrawlingThreadPoolSize to max 2?
mentioned here --> https://copernicsearch.freshdesk.com/en/support/discussions/topics/47000642738/page/last#post-47001465793
I have a powerful workstation for running copernic server and rescaning the folders for changes takes more than 3 weeks for 4 million files.
i'm sure - increasing CrawlingThreadPoolSize to 4 or 8 would now harm my machine
regards
Andreas Korntheuer
0 Votes
12 Comments
LUST & Partners Jacky d'Hoest posted almost 3 years ago
I found out that you can change the crawling value to whatever value you want when you first quit CDS, second delete the queue and third change the value in the xml. You can then restart CDS and start an update. Even "12" was possible. Scanning is then superfast. Downside is that sometimes CDS is then unstable, meaning it freezes at 99 % of indexing. You can then only restart CDS. The value of extraction can never be higher then "4".
2 Votes
Alexander Borouhin posted about 3 years ago
Currently both thread pools seem to be limited by 4 threads max. At least, 4/4 configuration works for me, anything higher gets reset to 4/4. But still it is very low limit. While initial indexing of ca. 1 million documents takes days, and each index update - hours, I'd prefer to utilize all the 16 cores to the maximum to speed up this process, scheduling the indexing time so that it doesn't affect other tasks.
1 Votes
Roeland Mertens posted over 3 years ago
What I report now will sound a bit dumb – but here it is.
In total I repeated the test to modify both parameters 4 times. The first 3 times the values “2/4” were immediately overwritten to “1/2” upon restart of the app. I’ve done another test now once again … and the”2/4” values stayed on!
Go figure …
The only difference with the 4th test may be that at long last the initial indexing had been finalized; so the app was busy indexing the first 3 times, and idle at the 4th time. I have no other explanation.
Thx for all the replies.
0 Votes
Mirco Persechino posted over 3 years ago
Hi Mertens,
Make sure you completely exit Copernic Desktop Search BEFORE making any changes in the config file. I think that could be the issue here. So completely exit CDS, then make changes to your config file extractionthread and then start Copernic. It should be okay.
0 Votes
LUST & Partners Jacky d'Hoest posted over 3 years ago
My configuration runs like this :
Only If you go higher the values are adjusted.
0 Votes
Mirco Persechino posted over 3 years ago
Did you try 2 and 2, start and see if it returns back to 2 and 1, then try with 2 and 3.
It shouldn't behave like that. I'll make a few checks with our developpers.
0 Votes
Roeland Mertens posted over 3 years ago
Not so. As I have just said, with the settings as suggested, at the next start it overwrites both settings to those "2" and "1" values again.
0 Votes
Mirco Persechino posted over 3 years ago
Hi Mertens,
"ExtractionThreadPoolSize" can be set to a maximum of 4.
"CrawlingThreadPoolSize", maximum 2.
If you exceed those numbers, the config file will simply reset to its default value.
1 Votes
Roeland Mertens posted over 3 years ago
Same problem here.
I use a high-end PC (with i.a. an AMD Threadripper Ryzen CPU with 24 cores). With the indexing performance set at unrestricted, the indexing process stays well below 3% CPU load all the time. That is of course ridiculous for this kind of PC and the time waste is enormous.
Remarkably, after manually adjusting the parameters ("CrawlingThreadPoolSize" and "ExtractionThreadPoolSize"), at each startup my settings are immediately overwritten and reset to those meager "2" and "1" values again.
Ask me why ... !
0 Votes
Andreas Korntheuer posted over 3 years ago
i think, this is not the main problem in my case.
the process Copernic.Plugins.PluginsService.exe is still at 0% cpu usage
if copernic do a ocr rescaning this process should be more than 0% cpu usage
the process Copernic.DesktopSearch.exe use about 15% cpu usage.
if there are more than 2 Crawling Threads the indexing should be faster.
regards
0 Votes
LUST & Partners Jacky d'Hoest posted over 3 years ago
If you are using the OCR-plugin, that is the reason, why it takes so long. I have an index with 1.300.000 items and with the OCR-Pugin a full update takes more than three weeks. Without it, it still takes 10 days and the reason for that is the fact that CDS every time reindexes all emails. It does not restrict itself to the new or changed emails ...
I complained several times about that without any succes.
2 Votes
Andreas Korntheuer posted over 3 years ago
please reply - rescaning the folders for changes should not take 3 weeks
0 Votes
Login or Sign up to post a comment