Index

Show enters and exits. Hide enters and exits.

01:37:49evanwho's got a working heap dump?
01:37:55evan<== that guy
01:46:28BrianRicecool
02:00:25slavaevan: heap dump?
02:00:59evanslava: memory debugging tool
02:01:07evanwrites the whole heap out to a file to be analyzed
02:01:27slavaoh, makes sense
02:01:37slavadoes it dump the ivars of each object too?
02:02:18evanyep
02:02:24evanwith references
02:03:46slavaone cool thing you can do with that is make your inspector tool and other tools that look at objects access all ivars with one extra level of indirection
02:04:00slavaso you can make a proxy type that wraps an object stored in such a dump
02:04:17slavaand then do offline debugging with the same tools you use in the running image
02:04:22evanthats true.
04:44:18BrianRicehm not a bad idea
05:39:31mahargevan: does it have enough information to rebuild an entire interpreter state (less, obviously, any open external resources or C extension objects)?
05:39:48mahargbecause you could do a lot more than debug with that
05:46:47evanmaharg: you mean... an image? :D
05:47:11evanI could dump all CallFrame state
05:47:23evanthen it would be a matter of reconstructing it to be used.
05:53:19slavaand then you'd have serializable continuations
05:53:36slavastore a fiber in the database and resume execution in another VM
05:53:36evanbada-boom.
05:53:45evanslava: how do you serialize a call stack?
05:54:22evanor, rather, how do you unserialize a call stack
05:59:41slavawe don't :)
05:59:48evanthat makes it easy!
05:59:55slavain fact I'm going to remove general continuations at some point altogether, so we won't have a 'callstack' as a first-class type anymore
06:00:11evanthe image restarts at a fixed position then
06:00:14slavawell, how would you serialize a call frame that's executing compiled code?
06:00:14evanthat you can't return past
06:00:24evanslava: you can't!
06:00:27slavaoh, images can get loaded at an arbitrary base address
06:00:27evanthats why i was wonder!
06:00:30evanyou're magic
06:00:31evanso who knows.
06:00:46slavaso its a matter of fixing up the return address in each frame
06:01:07slavatransporting a callstack between images is much harder though; your code may have compiled slightly differently, etc
06:01:16evantotally.
06:01:23slavaif you compile the same word twice in factor, it might end up with a different register allocation the second time because of weird hash effects and such
06:01:35slavaso you'd have to juggle spill slots around somehow. too magic
06:01:36evanfun!
06:01:46evanyes, bigger fish to fry.
06:01:52evanlet Dan figure that out.
06:01:53slavabut yeah, you can reify a continuation, save the image, start it again, and reflect that continuation
06:01:53evan:D
06:02:19slavabut that's just a matter of going through and adding (new_address - old_address) to each return address and spilled pointer :)
06:02:46slavasbcl doesn't do image relocation on startup, which means they have to mmap the image at a fixed address, and update their VM for every minor point release of every OS that changes the address space slightly
06:02:55evanyep
06:02:55slavaand on openbsd which has address space randomization, they're out of luck
06:03:02evandoesn't emacs do that too?
06:03:05slavayep
06:03:13slavathe bookkeeping required for image relocation is similar to what you need for precise GC
06:03:18evanyeah
06:03:25slavasbcl has a conservative GC; every stack location that looks like a pointer is treated like one
06:03:31evanouch.
06:03:33slavaso they have a copying collector, but if something is referenced from the call stack it gets pinned down
06:04:10evanewww.
06:04:26slavaon platforms with lots of registers, like PPC, its precise, because they divide the register file into two -- tagged pointers and untagged integers
06:04:32slavaso the GC traces every tagged register as a root
06:04:43slavabut on x86, you don't have enough registers to be able to do that, and they don't do GC maps either
06:05:10evanaah.
06:06:05slavamoral of the story: design decisions that made sense on early 90's SPARC workstations don't necessarily make sense today, hence whiny RISC trolls who pine for the good old days
06:06:23evanhah
06:06:52slavathe main thing I hate about x86 is not the register shortage but the fact that some instructions have fixed register assignments
06:07:26slavasome recent research shows how to do graph coloring register allocation in polynomial time, but when you add fixed registers to the mix its still NP-complete. if only intel knew at the time :)
06:08:14slavaits totally random too. multiplication can take operands in any register, but shifts must have the shift count in ECX. why are shifts special? why ECX?
06:08:15evanyou mean like the ones to push and pop from the stack?
06:08:26evanah yeah
06:08:27evanthat stuff.
06:08:41evanthats "optimization"!
06:08:41slavano, like IDIV storing the result in EAX, shifts with ECX, and some of the atomic compare and swap stuff is hard-wired in weird ways too
06:09:11slavayeah, you save one whole byte every time you divide
06:09:21slavatoo bad you generate a ton of unnecessary MOVs :)
06:10:13slavaI always thought it was odd that Forth had a /MOD word until I learned that on x86, IDIV computes both the div and the mod
06:10:50slavaPowerPC doesn't even have mod, instead you compute x-(x/y)*y
06:47:27evanbrixen: http://gist.github.com/444643
06:48:09brixensweet
06:49:19evanall from the heap dump
06:49:32evanit's got all the references between objects too
06:50:20brixenvery cool
06:50:59evanthats a dump from a script that jsut calls dump_heap
06:51:03evanso it's pretty much just the startup.
06:51:29brixen20k tuples to start up
06:51:30brixenwow
06:51:39evanyeah, seems high doesn't it.
06:51:44evani need to verify these numbers.
06:51:45brixena bit
07:51:09jaribbrixen: no, on 1.9 configure crashes since `` will raise Errno::ENOENT if the command isn't found
09:08:02khaasejust added rbx to the rack ci: http://ci.rkh.im/job/Rack-Rubinius/2/console
09:08:13khaaseany idea where this comes from?
09:09:03khaasehuh, that spec passes for me locally
10:05:23jaribbrixen: created #378
10:08:25jaribbrixen: the build still fails after fixing configure though http://gist.github.com/444768
17:07:51iamstefapparently im missing something... 1.0.1 has some sort of limited debugger support, but i can seem to figure out how to enable the debugger
17:08:02iamstefanyone got a sec to explain this, little detail im missing?
17:12:47cyndisrequire 'debugger'; Debugger.start
18:35:20boyscoutAdd a number of GC stats to the query agent - 5e5679d - Evan Phoenix
18:45:54boyscoutCI: rubinius: 5e5679d successful: 3456 files, 13603 examples, 41167 expectations, 0 failures, 0 errors
19:15:36dbussinkevan: cool though that it's exactly 20k tuples ;)
19:19:26dbussinkjarib: what ruby version are you using?
19:21:02evandbussink: yeah, thats funny isn't it
19:21:17evandbussink: i thought it might be wrong, but adding some more objects dumps it up, so it's right
19:21:48dbussinkevan: how funny that numbers like this trip people up ;)
19:21:57dbussinkany idea where that number is coming from?
19:22:19dbussinkevan: btw, would it be easy to do ivar packing for included modules?
19:22:40dbussinkevan: i wonder if it would be possible to detect it with datamapper, or perhaps otherwise expose an api for it
19:22:53dbussinkwould be nice that like loading 100 db records is nicer on memory pressure :)
19:26:21evandbussink: actually, i can add it for included modules
19:26:39evanthe current code includes ivars seen in superclasses
19:26:49evani can easily have it include ivars found in include modules
19:28:00dbussinkevan: dunno if dm is simple enough in it's usage for proper auto detection though, dunno if adding an api for it would be useful?
19:28:22dbussinkevan: because dm tries to be pretty lazy in actually setting stuff etc.
19:28:46dbussinkbut it could likely be very useful to register ivars that are very likely to be filled
19:28:51evanthere is an API
19:29:03evanModule#packed!([:@name, ...])
19:29:35dbussinkevan: ah, cool, maybe i'll try playing with it to see how much it helps :)
19:29:53dbussinkalso saw it didn't only use a lot less memory on that benchmark, but it was also 10x faster for me here
19:30:11evanwhat was 10x faster?
19:31:27dbussinkevan: this one: https://gist.github.com/40f0ec7be1b89ebee06a
19:32:11evanthats 10x faster on rbx with autopacking? or 10x faster than MRI?
19:32:30dbussinkevan: 10x faster with autopacking compared to not packing
19:32:42dbussinkevan: that was the revision before you made autopacking the default
19:32:43evanah yep
19:32:45evanyep yep
19:33:10dbussinkpretty impressive improvement :)
19:33:33evanyeah! i'm happy it's worked out so well.
19:33:52evanthere are still directions to take it
19:33:55evanlike included modules
19:34:01dbussinkevan: how smart is it? it considers each @var during class loading?
19:34:14evanand the JIT making use of packing
19:34:18dbussinkdoes it work with stuff like class_eval etc.?
19:34:27dbussinkah, would be nice if the jit could trigger it :)
19:34:38dbussinkbased on history usage of ivars for a class of objects :)
19:34:52evanit considers ivars used in methods when a method is added to a class
19:34:54evanso stuff like
19:34:57dbussinklike, ok, the last 1000 objects of this type used all these ivars, let's pack that
19:34:58evaninstance_eval { @name = 1 }
19:35:04evanwill add @name to the wrong place
19:35:05evanbut thats ok
19:35:26evanit does all the work before an instance of the class is created
19:35:26dbussinkwell, datamapper defines some readers / writers with class_eval
19:35:44evanwhen a instance is created, it crystalizes the format
19:35:55evanafter than, the format is fixed.
19:36:14dbussinkevan: http://github.com/datamapper/dm-core/blob/master/lib/dm-core/model/property.rb#L221-237
19:36:18dbussinkdoes stuff like that
19:36:40dbussinkthis is called by property :name, String for example that is in a class body
19:36:57evanthat should actualy work ok
19:37:05evanbecause the class_eval will create the method and add it to the class
19:37:16evanat which time that method will be looked at to see what ivars are used
19:37:30evanso long as this is called before there is an instance of that class
19:37:31evanshould be fine
19:37:50dbussinkcan i easily see what exactly is packed for an object?
19:37:55evanyeah
19:38:03evanthe @seen_ivars ivar in class
19:38:05evanjust add
19:38:09evanattr_reader :seen_ivars
19:38:45dbussinki'll try with a script to see if it works or not :)
19:39:06evank
19:39:12evanit happens at method addition time
19:39:18evannot parse time
19:39:19evanwhich helps
19:39:29evanbecause meta programming shouldn't defeat it.
19:41:11dbussinkevan: hmm, should seen_ivars be on the instance or class?
19:41:19evanclass
19:41:22evansorry
19:41:24evani meant
19:41:26evanclass Class
19:41:28dbussinkbecause it's returning nil on both now for me
19:41:30evanattr_reader :seen_ivars
19:41:30evanend
19:42:42evandbussink: http://gist.github.com/445211
19:42:44evanline 17
19:42:46evanoutputs
19:42:51evan[:@name, :@thing, :@more, :@stuff, :@lots, :@crap, :@woo, :@mmmm, :@foo]
19:43:39dbussinkevan: hmm, this is returning nil for me: https://gist.github.com/b09acf1dec9a186bd383
19:44:06dbussinkthat should go through that whole layer of meta programming though, maybe it gets confused somewhere
19:44:17evancould be
19:44:17dbussinkevan: or doesn't it work because that property method comes from a module?
19:44:26evandoes it just create accessors and call them?
19:44:33evandoes it actually use @<name> syntax?
19:44:56dbussinkevan: well, the accessors created do use that, if you look at the code i linked to
19:45:04dbussinkevan: http://github.com/datamapper/dm-core/blob/master/lib/dm-core/model/property.rb#L221-237
19:45:28dbussinkevan: instance_variable_name there is something like @id, @name and @age in this example
19:45:34evanwhat is instance_variable_name?
19:45:37evank
19:46:52evanwhat does chainable do?
19:46:56dbussinkevan: if i output instance_variable_name and self in that code snippet, i get @id and Person
19:47:04evana class_eval on the block or something?
19:47:16dbussinkevan: ah, that might be it, it creates an anonymous module from the block and then includes it
19:47:21evanthats it.
19:47:33dbussinkso people can override it later on with their modules and call super()
19:47:38evanyep
19:47:47evanso the class really doesn't have any ivars
19:47:54evanyou're putting them in that anonymous module
19:48:08dbussinkbut also getting ivars from modules would fix this then right?
19:48:51dbussinkevan: yep, removed the chainable to test it and then it works :)
19:49:01evanthere are 2 things
19:49:08evan1) i should teach the code about included modules
19:49:18evan2) seen_ivars is only the ivars in that exact module
19:49:29evanbut at crystalize time
19:49:33evanall the seen_ivars are consulted.
19:50:07dbussinkevan: ah ok, so would it be useful to add some __ methods perhaps for this so people can look at it?
19:50:22dbussinkevan: or other methods that aren't likely to clash name wise :P
19:50:28evani could add a method to return the definitive list
19:51:26dbussinkthat could be useful i guess for people who would want to verify that it works for their objects
19:51:29evandbussink: after an instance is created
19:51:36evanyou can look at @packed_ivar_info on the class
19:51:41evanwhich shows the mapping of ivar => slot
19:51:52evanthat will include ivars found in superclasses
19:52:40evandbussink: i'll go ahead and add Class.memory_info
19:52:45evanie
19:52:49evanClass.memory_info(Blah)
19:53:03evanwhich will report the byte size, packed ivars, etc.
19:53:16dbussinkevan: cool :)
19:53:47dbussinkevan: funny thing was that sam smoot (who started datamapper originally) was musing about exactly this yesterday in #datamapper :)
19:54:02dbussinkthat he thought something like this would be really awesome
19:54:03evanhehe
19:54:23dbussinkbut i'm pretty happy that it's smart enough that besides the included module stuff, it already works :)
19:54:34evanyeah
19:54:43evanby delaying the calculations you can't take back
19:54:46evanit can be pretty smart
19:54:46dbussinkguess that means that datamapper internals are also pretty reasonable and do too weird things :P
19:54:55dbussinkdon't do
19:55:18evan:)
19:55:48dbussinkanother benefit of being declarative :P
19:55:53evandbussink: there is already Rubinius.memory_size actually
19:56:04dbussinkevan: which i can give a class?
19:56:05evanwhich is just the byte size
19:56:12evani think just an object
19:56:23evani need another one that knows about classes
19:56:29evanand will give info about the classes instances
19:59:59dbussinkevan: this is also really the stuff that makes rbx awesome, being able to get information like this from the system etc.
20:03:56skaarevan: is the Rubinius Fiber available by default? I see it in the source tree, but rbx (at least in irb mode) doesn't seem to find it
20:04:32cyndisRubinius::Fiber
20:12:18evanskaar: the API is experimental
20:12:21evanso it's under Rubinius
20:18:32skaarevan: ah, thanks, got it
20:28:53dbussinkfinally a pretty exiting world cup match :)
20:30:00evanyeah
20:30:03evanthat one was entertaining.
20:30:40dbussinkand a good result for the dutch too :p
20:30:58dbussinkevan: i was actually surprised by how much attention it gets in the usa
20:30:59evansince you guys killed denmark?
20:31:33dbussinkwe've qualified now, netherlands will be either first or second in the group now
20:32:14evanthe US needs to be beat algeria
20:32:15dbussinkdenmark and japan have to play for the other place, so they can't both get to 6 points
20:32:28evanat least we control our own fate now.
20:32:44dbussinkhehe, and no one else to blame if it fails :P
20:32:49evanyep.
20:33:05evanwell, we can blame that ref.
20:33:18dbussinkthat's true yeah, was a perfectly legal goal
20:33:25evanyeah
20:33:28dbussinkbut on the other hand, you can thank the english :p
20:33:29evantotal bullshit.
20:33:34evanthere were 3 fouls in the box too
20:33:36evanby slovenia
20:34:55dbussinkhaven't seen that match actually, only small parts of the summary
20:35:04dbussinkbut did see that denied third goal
20:38:32evanI only watched the 2nd half.
20:38:40evansince it was on early here.
20:39:34dbussinkactually watched the first game from the netherlands in our bed in the hotel in new york early in the morning :p
20:39:43evanha!
20:39:55evani'll bet you were happy :)
20:40:06tarcieri=======<() ~ ♪ ~♫ BZZZZZZZZZZZZZZZZZZZZZZZ
20:40:16dbussinkgood start of the day :)
20:40:22evantarcieri: :D
20:41:00dbussinktarcieri: http://i.imgur.com/WXNmF.jpg
20:42:29tarcieriheh
21:05:14dbussinkevan: did that manual inlining you did here make a big difference? http://github.com/evanphx/rubinius/commit/0f068f5b84ea873392584116ceb34727902aab68
23:56:31jaribdbussink: using 1.9.1p378