Opened 4 years ago

Closed 2 months ago

#1155 closed defect/bug (fixed)

Maptool memory usage optimization

Reported by: tryagain Owned by: tryagain
Priority: critical Milestone: version 0.5.1
Component: tools Version: git master
Severity: normal Keywords: map, maptool, osm, extractor
Cc: cp15, https://wiki.navit-project.org/index.php/user:tryagain

Description

It looks like we have hit our hardware limits with maptool. It has now peak memory consumption above 10G, but server cant give it so much.

That might be related to osm growth, our recent code changes.

I'm aware of memory leaks in the relation processing code, and will attempt to address them when i get to my dev environment next week.

For now, i see two possible solutions, one is to split maptool run into two parts with -s and -e options to drop any leaks between runs. If that won't help, i will try to reduce -S value (slice size), which though might increase processing time.

I do not think that increasing swap size is a way to go, but depending on how efficiently we'll be able to address memory apettites of maptool, server might require a memory upgrade.

tryagain

Change History (6)

comment:1 Changed 4 years ago by usul

  • Cc cp15 added
  • Keywords map maptool osm extractor added

user:cp15 mailed me, that there is some memory leak within maptool.

As it's a heavy problem, if we couldn't create our planet files, IMHO we need to put it in focus for the next minor release.

comment:2 Changed 4 years ago by tryagain

Surely, we shuldnt wait for any releases but fix it asap.

Current state of the workaraunds described above: Attempt to split maptool runs failed, maptool segfaulted on the second run with -s 10 key.

So now its running with lowered by 1 gb slice size.

I have investigated the crash a bit, and its reproduceable with the sample map of Munchen. So theres a bug in phase 10 maptool code, which doesnt allow to restart maptool from phase 10. We have to address this issue too.

comment:3 Changed 4 years ago by tryagain

  • Owner changed from KaZeR to tryagain
  • Priority changed from blocker to critical
  • Status changed from new to assigned

Now all mirrors have been updated.

Peak memory consumption was around 8gb, slice size was decreased by 1G.

As we managed to have a workaround, I think priority can now be lowered a bit.

comment:4 Changed 4 years ago by tryagain

  • Cc https://wiki.navit-project.org/index.php/user:tryagain added

comment:5 Changed 2 years ago by kazer

  • Severity set to normal

As maptool has been fixed and this ticket isn't really actionnable should we close this ticket? We know that some work is needed regarding maptool anyway.

comment:6 Changed 2 months ago by kazer

  • Resolution set to fixed
  • Status changed from assigned to closed

No activity for some time, closing for now. Feel free to reopen.

Note: See TracTickets for help on using tickets.