Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#1184 closed defect/bug (fixed)

maptool crash when converting an OSM pbf file.

Reported by: tuxmaster Owned by: KaZeR
Priority: major Milestone: version 0.5.1
Component: mapdrivers/OSM Version: git master
Severity: normal Keywords: OSM, maptool, performance
Cc:

Description

Steps to reproduce the crash.

  1. load the osm pbf from http://download.geofabrik.de/europe/germany-latest.osm.pbf
  2. load http://download.geofabrik.de/europe/germany.poly
  3. run osmupdate to update the map. "osmupdate -B=germany.poly germany-latest.osm.pbf germany-neu.osm.pbf"
  4. run "maptool --protobuf -i germany-neu.osm.pbf germany-neu.bin"

maptool crash with an segfault. I have talk the the developer of osmupdate/osmconvert, and we have test the map. The map seen to be ok. Used maptool from SVN 5718.

Change History (6)

comment:1 Changed 6 years ago by tryagain

Hi!

Thank you for your report.

maptool requires lots of RAM and disk space during run, and it does not report low memory and low disk space related problems at all. To give you a hint, you'll need around 2 gigabytes of RAM and 100G of disk space to process the whole planet file with default slice size (1G).

A few more questions about your situation:

  • What OS do you run it on?
  • What is the size of your germany-neu.osm.pbf?
  • What filesystem do you have on the disk which holds the file?
  • How much RAM does your system have?
  • How much free space is left on the filesystem after the crash?
  • What maptool reports right before crashing?
  • What do say the most recent output lines starting with PHASE, PROGRESS.

We would highly appreciate if you could to provide a backtrace of your crash. To get it, do the following steps:

  1. Make sure you have not a stripped down version of maptool. For example, shell command
    file maptool
    
    should contain words 'not stripped' in output. If you have a stripped down version, following step will not provide too much useful information.
  1. Run maptool under gdb like this:
    $ gdb --args maptool --protobuf -i germany-neu.osm.pbf germany-neu.bin
    (gdb) run
    ....
    Segmentation fault
    (gdb) bt
    

then copy and paste the bt output here.

comment:2 Changed 6 years ago by tuxmaster

Hello, here the facts, I hope it will help. I think, the system is not the limit.

OS: Fedora 19 x86_64 map size: 1805093922 1. Dez 19:38 Deutschland.osm.pbf RAM:32GB free space on file system: 1,3TB Filesystem :ext4 free space on /tmp after crash: 16GB the last output before crash: PROGRESS: Phase 1: collecting data 0:00 0 MB

As an additional hint, I have run it with --slice-size 5338709120 but not other results. file maptool: maptool: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x9469efb4606f78494e05277c79a03c3091979ebc, not stripped

the bt output #0 0x00000000004213b5 in read_blob (f=f@entry=0xcc3760, header=0xcd1180) at osm_protobuf.c:73 #1 0x000000000042170f in map_collect_data_osm_protobuf (in=0xcc3760, osm=0x7fffffffdf60) at osm_protobuf.c:365 #2 0x000000000041569c in osm_collect_data (suffix=0x47766f "", p=0x7fffffffdef0) at maptool.c:466 #3 main (argc=<optimized out>, argv=<optimized out>) at maptool.c:870

comment:3 Changed 6 years ago by tryagain

hm, looks like the alloca() function has failed to allocate memory needed to hold the blob data.

I have two ideas for now...

First one would be that your map data contains too big blob data that can't fit into the stack. You should be able to solve this problem by increasing ulimit -s value or setting it to unlimited. Maximum pbf blob size is 32Mb, and trying to allocate it on the stack is not a good idea. We'll have to switch to malloc there.

Another idea is that something has forced the read_blob function to be optimized as inline. Aren't there some related warnings in the build logs? Then stripping the static declaration from that function definition should workaround your problem. To fix this case, we review our build system.

comment:4 Changed 6 years ago by tuxmaster

current: limit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 254807 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

ulimit -s hard: ulimit -s: unlimited

this will work: Werkzeuge/maptool --protobuf --slice-size 533870912 -i Deutschland.osm.pbf Deutschland.bin PROGRESS: Phase 1: collecting data 0:00 0 MB flush_nodes 0 flush_nodes 0 flush_nodes 0 flush_nodes 0 flush_nodes 1 PROGRESS: Phase 2: counting references and resolving ways 3:03 47 MB 5 slices slice 0 of 4 slice 1 of 4 slice 2 of 4 slice 3 of 4 slice 4 of 4 PROGRESS: Phase 3: converting ways to pois 3:51 47 MB PROGRESS: Phase 4: finding intersections 3:52 47 MB PROGRESS4: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 3:52 47 MB PROGRESS4: Processed 0 nodes (0 out) 28546343 ways 0 relations 0 tiles 4:04 47 MB PROGRESS4: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 4:04 47 MB PROGRESS4: Processed 0 nodes (0 out) 31145679 ways 0 relations 0 tiles 4:14 47 MB PROGRESS4: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 4:44 47 MB PROGRESS4: Processed 0 nodes (0 out) 33028574 ways 0 relations 0 tiles 4:54 47 MB PROGRESS4: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 5:14 47 MB PROGRESS4: Processed 0 nodes (0 out) 34485837 ways 0 relations 0 tiles 5:24 47 MB PROGRESS4: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 5:44 47 MB PROGRESS4: Processed 0 nodes (0 out) 35185626 ways 0 relations 0 tiles 5:54 47 MB PROGRESS: Phase 5: generating coastlines 5:55 47 MB tile_collector_finish tile_collector_finish foreach done tile_collector_finish destroy done Level=14 Level=14 * * * * Level=13 * * * * Level=12 * * * * Level=11 * * * * Level=10 * * * * Level=9 * * * * Level=8 * * * * Level=7 * * * * Level=6 * * * * Level=5 * * * * Level=4 * * * * Level=3 * * * * Level=2 * * * * Level=1 * * * * tile_collector_finish done PROGRESS: Phase 6: assinging towns to countries 5:55 47 MB PROGRESS: Phase 7: sorting countries 8:11 287 MB PROGRESS: Phase 8: generating turn restrictions 8:12 287 MB PROGRESS: Phase 9: processing associated street relations 8:46 287 MB PROGRESS: Phase 10: generating tiles 9:02 287 MB PROGRESS10: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 9:02 287 MB PROGRESS: sorting 154078 tiles PROGRESS: sorting 154078 tiles done PROGRESS: merged 129349 tiles PROGRESS: sorting 50058 tiles PROGRESS: sorting 50058 tiles done PROGRESS: merged 2411 tiles PROGRESS: sorting 49594 tiles PROGRESS: sorting 49594 tiles done PROGRESS: merged 564 tiles PROGRESS: sorting 49318 tiles PROGRESS: sorting 49318 tiles done PROGRESS: merged 195 tiles PROGRESS: sorting 49211 tiles PROGRESS: sorting 49211 tiles done PROGRESS: merged 65 tiles PROGRESS: sorting 49185 tiles PROGRESS: sorting 49185 tiles done PROGRESS: merged 23 tiles PROGRESS: sorting 49177 tiles PROGRESS: sorting 49177 tiles done PROGRESS: merged 13 tiles PROGRESS: sorting 49175 tiles PROGRESS: sorting 49175 tiles done PROGRESS: merged 7 tiles PROGRESS: sorting 49173 tiles PROGRESS: sorting 49173 tiles done PROGRESS: merged 5 tiles PROGRESS: sorting 49173 tiles PROGRESS: sorting 49173 tiles done PROGRESS: merged 4 tiles PROGRESS: sorting 49173 tiles PROGRESS: sorting 49173 tiles done PROGRESS: merged 4 tiles PROGRESS: sorting 49172 tiles PROGRESS: sorting 49172 tiles done PROGRESS: merged 2 tiles PROGRESS: sorting 49172 tiles PROGRESS: sorting 49172 tiles done PROGRESS: merged 0 tiles PROGRESS10: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 899737 tiles 9:21 287 MB PROGRESS: Phase 11: assembling map 9:22 287 MB Maximum slice size 533870912 Slice 0 is of size 533815488 Slice 1 is of size 533769628 Slice 2 is of size 533607496 Slice 3 is of size 533743944 Slice 4 is of size 533842172 Slice 5 is of size 533830344 Slice 6 is of size 502530576 PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 9:22 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 9:36 287 MB PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 11:04 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 11:18 287 MB PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 12:47 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 13:02 287 MB PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 14:31 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 14:46 287 MB PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 16:16 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 16:31 287 MB PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 17:45 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 18:00 287 MB PROGRESS11: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 19:10 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 0 tiles 19:24 287 MB PROGRESS11: Processed 9086001 nodes (0 out) 35266900 ways 0 relations 49173 tiles 20:20 287 MB PROGRESS: Phase 12: done 20:20 287 MB

comment:5 Changed 6 years ago by tryagain

  • Resolution set to fixed
  • Status changed from new to closed

tuxmaster,

Thank you for reporting this and your help in further investigation.

Since r5725 maptool should not require ulimit magic to process your file.

So I'm closing the ticket as fixed now.

comment:6 Changed 6 years ago by usul

  • Component changed from core to mapdrivers/OSM
  • Keywords OSM maptool performance added
  • Milestone set to version 0.5.1
Note: See TracTickets for help on using tickets.