[obspy-users] FDSN downloader spamming new processes

Tobias Megies megies at geophysik.uni-muenchen.de
Fri Mar 1 11:32:16 CET 2019


Hi Florian,

you could try to use the get_waveforms_bulk() method instead of the for
loop, see if that helps. In general the bulk requests are more
efficient, as they avoid http comms overhead, but I'm not sure how this
plays out with the routed clients exactly..

But, this is definitly something we want to look into, so I'm opening an
issue on github for this.

cheers,
Tobias


On 26/02/2019 15:11, Florian Fuchs wrote:
> Dear ObsPy team,
> 
> I am facing a potential issue with the ObsPy FDSN client. I am using the
> EIDA routing client to download waveform data and metadata for many
> stations. The way I do it is a simple loop over all stations, using the
> get_waveforms() function in the loop. In principle it looks like this:
> 
> from obspy.clients.fdsn import RoutingClient as Client
> client = Client('eida-routing')
> for station in long_list :
> data = client.get_waveforms(...)
> data.write(filename)
> 
> The problem is, that each time data is actually downloaded, ObsPy seems
> to start several new processes in the background, which are not being
> closed until the end of the python script. Thus, for each station in the
> for-loop new additional processes are started but not closed again.
> Since I have a very long stationlist and many days that I loop over,
> basically I end up spawning more processes than the systen can handle,
> ultimately resulting in the message: "Can't start new thread"
> 
> This problem only occurs with the fdsn routing client, but e.g. not with
> the ArcLink client. I did not try the non-routed fdsn client. Same
> behaviour with ObsPy versions 1.1.0 and 1.1.1 ...
> 
> Any suggestions how to avoid this? Or did I just code it in a stupid way?
> 
> Thanks in advance and best regards,
> Florian
> 
> _____________________________________ Dr. Florian Fuchs Department of
> Meteorology and Geophysics University of Vienna UZA II, 2D508
> Althanstraße 14 1090 Vienna Austria Phone: +43 1 4277 53726 Mobile: +43
> 664 60277 53726 www: http://imgw.univie.ac.at
> _____________________________________
> 
> _______________________________________________
> obspy-users mailing list
> obspy-users at lists.swapbytes.de
> https://lists.swapbytes.de/mailman/listinfo/obspy-users
> 

-- 
EMail: tobias.megies at lmu.de
WWW: https://www.geophysik.uni-muenchen.de

Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck

Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970

Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München

Tel: +49 (0) 89 2180-4326
Fax: +49 (0) 89 2180-4205


More information about the obspy-users mailing list