[obspy-users] FDSN downloader spamming new processes

Lion Krischer lion.krischer at erdw.ethz.ch
Fri Mar 1 11:56:54 CET 2019


Hi Florian,

Tobias opened the issue here: https://github.com/obspy/obspy/issues/2342

I'm having some troubles reproducing it. Can you join the discussion on
github and help us narrow it down?

All the best,

Lion


On 01.03.19 11:32, Tobias Megies wrote:
> Hi Florian,
>
> you could try to use the get_waveforms_bulk() method instead of the for
> loop, see if that helps. In general the bulk requests are more
> efficient, as they avoid http comms overhead, but I'm not sure how this
> plays out with the routed clients exactly..
>
> But, this is definitly something we want to look into, so I'm opening an
> issue on github for this.
>
> cheers,
> Tobias
>
>
> On 26/02/2019 15:11, Florian Fuchs wrote:
>> Dear ObsPy team,
>>
>> I am facing a potential issue with the ObsPy FDSN client. I am using the
>> EIDA routing client to download waveform data and metadata for many
>> stations. The way I do it is a simple loop over all stations, using the
>> get_waveforms() function in the loop. In principle it looks like this:
>>
>> from obspy.clients.fdsn import RoutingClient as Client
>> client = Client('eida-routing')
>> for station in long_list :
>> data = client.get_waveforms(...)
>> data.write(filename)
>>
>> The problem is, that each time data is actually downloaded, ObsPy seems
>> to start several new processes in the background, which are not being
>> closed until the end of the python script. Thus, for each station in the
>> for-loop new additional processes are started but not closed again.
>> Since I have a very long stationlist and many days that I loop over,
>> basically I end up spawning more processes than the systen can handle,
>> ultimately resulting in the message: "Can't start new thread"
>>
>> This problem only occurs with the fdsn routing client, but e.g. not with
>> the ArcLink client. I did not try the non-routed fdsn client. Same
>> behaviour with ObsPy versions 1.1.0 and 1.1.1 ...
>>
>> Any suggestions how to avoid this? Or did I just code it in a stupid way?
>>
>> Thanks in advance and best regards,
>> Florian
>>
>> _____________________________________ Dr. Florian Fuchs Department of
>> Meteorology and Geophysics University of Vienna UZA II, 2D508
>> Althanstraße 14 1090 Vienna Austria Phone: +43 1 4277 53726 Mobile: +43
>> 664 60277 53726 www: http://imgw.univie.ac.at
>> _____________________________________
>>
>> _______________________________________________
>> obspy-users mailing list
>> obspy-users at lists.swapbytes.de
>> https://lists.swapbytes.de/mailman/listinfo/obspy-users
>>


More information about the obspy-users mailing list