Index

Show enters and exits. Hide enters and exits.

00:22:29ice799brixen: INT2FIX(RHASH(obj)->tbl->num_entries) is painful, but it can be dealt with. depending on how much suffering you can handle, i suppose.
00:28:36brixenice799: it can be dealt with by removing the totally unnecessary static from rb_hash_size
00:28:46ice799well, sure.
00:28:54brixenunless someone can point out a good reason for it being there
00:28:59brixenwhich no one has
00:29:03evannope!
00:29:11ice799there is probably no good reason for it, but you can be backward compatible with that usage
00:29:16ice799if you want to be.
00:29:27brixenI don't follow
00:29:38brixenwhat if your hash in not based on st?
00:29:51brixenthere is no tbl->num_entries to return
00:30:02ice799Heh. How much pain do you want?
00:30:05brixenwithout going through gymnastics
00:30:09brixenexactly ;)
00:30:15ice799ok, well withough gymnastics, you are fucked.
00:36:14ice799Ah, so you guys do the GIL, thing, too, huh?
00:36:28brixenfor now, yes
00:36:29ice799Where was I when that became the new hotness?
00:36:35brixenit will go away eventually
00:36:45brixennot the new hotness, just the GIL :)
00:37:08brixenit was a couple months back
00:37:23ice799Eh, I don't like the whole GIL thing very much. Why even use native threads?
00:37:45ice799I don't like it mostly because you don't have fine-grained control on the OS scheduler, do when you decide to start the race for the GIL
00:38:02ice799you can't really do very much to ensure that a specific person doesn't get starved to shit.
00:38:32ice799Presumably one could solve this with a condition var, or some other locking primitive. But it seems that not building your own locking primitives is -also- the new hotness, sadly.
00:39:27brixencrosses fingers hoping for an abort
00:40:05evanice799: i don't really like the GIL either
00:40:10evanbut it's a stepping stone
00:40:32ice799IMHO, I personally like green threads over the pthread+GIL thing that everyone is doing.
00:41:01evani like green threads too
00:41:22evanbut their correct implementation when using the C stack is non-trivial.
00:41:36ice799It's not all that hard, imho.
00:41:48slavanative threads are a better bet
00:41:59ice799i pushed some patches to MRI that fixed a lot of the problems MRI had with their green thread implementation
00:42:03evanwe want to eliminate the GIL eventually anyway, like I said.
00:42:12ice799Word.
00:42:32evanand i'm open to MxN threads as well.
00:42:42ice799Word.
00:43:03evanI need to implement stack saving soon, for continuations
00:43:15evanusing that same mechanism, we could play with green threads
00:43:27ice799Well, I'd like to say that I'd implement it, because it seems that mri will not accept my patches unless they are ported to 1.8 head first.
00:43:33evanit's trickier than on MRI
00:43:33ice799er
00:43:37evanbecause our GC is accurate
00:43:38ice799but, rather.
00:43:54ice799tricky = fun, though.
00:43:59evanare your patches for 1.8.6?
00:44:02evanyes, quite fun!
00:44:11ice799evan: 1.8.6 and 1.8.7
00:44:19ice799Essentially, I moved the thread stacks to the heap
00:44:25ice799to allow constant time thread switching in MRI
00:44:28evanah yes!
00:44:30evani saw those patches.
00:44:31ice799instead of the brain dead memcpy'ing.
00:44:33evanvery nice.
00:44:36ice799thanks.
00:44:46evanyes, rather than doing the stack_extend() + memcpy over the stack memory
00:44:52evanwhich always seemed ultra evil to me
00:44:58evanbut i guess necessary
00:45:06evanslava tells the story about having to do that on windows
00:45:18evanthat true slava?
00:45:20slavaits only a problem if you have C frames mixed in
00:45:29ice799there was no reason for mri to do that, honestly.
00:45:29evanassume you do
00:45:31slavabecause of how windows does seh
00:45:39slavabut why would you?
00:45:51slavadoes ruby code ever call C which calls ruby?
00:45:55evanHAH
00:45:56yakischlobayes.
00:46:00evanthats the whole way MRI is implemented.
00:46:21slavawhat about rubinius?
00:46:30evanmuch much much less
00:46:35evanbut we've got the interpreter
00:46:38evanwhich would be a C frame.
00:46:55evanso unless we compile everything upfront, there would be C frames due to mixed mode
00:47:03ice799word.
00:47:24slavathat's a problem since you can't control the C frame's stack layout, to know where it ends, so it might make thread switching harder
00:47:39ice799what do you mean you can't control it?
00:47:43slavayou can use llvm to generate an interpeter on startup though, instead of writing it in C
00:47:49ice799you can write whatever you want to the stack, so you can forge stack frames if you wish.
00:47:57slavaice799: not with portable C++
00:48:19ice799Meh.
00:49:14ice799when/if I merge my junk to 1.8, i'll probably write it so that the x86 path is fast
00:49:20ice799and everyone else uses the shitty slow memcpy path.
00:49:32ice799and then if non-x86 people come forward to fix their paths, good for them.
00:49:48slavaits compiler specific really
00:49:55ice799Huh?
00:50:01ice799As long as you conform to the x86 ABI, who cares?
00:50:01slavaif you have an accurate gc, you need to know what offsets in the stack frame hold pointers into the heap
00:50:16slavaand you can't assume the compiler puts locals in the same place
00:50:34evanwe force the compiler to use certain sections of memory for managed pointers
00:50:38evanusing a shadow stack
00:51:00ice799Ah, I see.
00:51:06evanso it's a matter of walking the shadow stack and reconstructing it when it's been copied to another region of memory
00:51:06ice799well, with gymnastics it could be solved, I bet.
00:51:13ice799yeah what i'm saying is
00:51:14slavaevan: that works
00:51:15ice799don't do any copying.
00:51:26ice799just point the initial thread stack at a chunk of memory on the heap.
00:51:27slavahow big do you make your per-thread stacks though?
00:51:28ice799and then never copy.
00:51:41ice799Use the same default that pthread uses, 1meg.
00:51:49ice799pthreads don't grow, and no one seems to care.
00:52:17ice799Of course, you could include a Thread.stack_size= to let people tune if they like.
00:52:27ice799slava: you can also grow if you want to do gymnastics.
00:52:49ice799you could just create a guard page under the thread stack, when it faults
00:52:53ice799you know the guy has overflown
00:52:57ice799so then just try a mremap
00:53:02ice799tell it not to move the mapping.
00:53:05slavai'd like to see you get that working reliably with MRI :)
00:53:12ice799slava: it is working reliably.
00:53:20slavaactually, because MRI is so buggy, something like this is made easier. a few more crashes now and then won't be noticable
00:53:26ice799Hah.
00:53:30slavasince a typical setup runs multiple web servers which restart evey few hours
00:53:32ice799It is reliable, my friend.
00:53:34slavaor so I've heard anyway
00:54:19ice799slava: all of that is already written. I haven't done the mremap because no one cares about their thread stacks growing, it seems.
00:54:32ice799slava: And it all seems to work pretty well. ::shrug::
00:54:39slavayou can't guarantee that mremap doesn't move your data
00:54:44ice799yes you can.
00:54:46ice799there is a flag.
00:54:49ice799man mremap
00:55:00slavain that case it won't always be able to grow the stack
00:55:03ice799If it can't do the mapping, it'll just fail and you'll just raise a stackoverflow
00:55:09slavayour trick with the guard page will only work if you don't move the stack from the fault handler
00:55:21ice799Huh ?
00:55:34ice799Faults get handled on a separate signal stack.
00:55:43slavathat's not what I was talkinga bout
00:55:59slavaif you compute an address relative to the stack pointer, then do something which grows the stack, your address might not be valid anymore, if the stack moved
00:56:09ice799Correct, which is why you don't move them.
00:56:26ice799you call mremap and tell it not to move the mapping. If that fails, and you STILL want to grow it
00:56:32ice799you can do more gymnastics to make it happen.
00:56:33ice799But again.
00:56:37ice799Pthreads don't grow.
00:56:39ice799No one cares.
00:57:14ice799Anyway, if you'd like to still grow after all that failure, you can allocate more stack space somewhere else in memory, and then just forge a frame.
00:57:37ice799but, really, why even bother?
00:57:45ice799just set it a meg, if it faults, too bad.
01:06:19brixenevan: sanity check, should a Handle ever get deleted when references_ > 0 ?
01:06:39evanhm
01:06:43evani'd think no
01:06:53brixenwell then houston, we have a problem
01:07:15brixenlet's see if I can figure it out
01:07:37evank
01:12:03evanarg
01:12:09evani'm flummuxed on a debugging problem
01:12:10evanbbiab.
02:27:08evanbrixen: any luck?
02:40:25brixenevan: well, it's not collecting any handles with positive references, that I can see
02:41:03brixenbut http://gist.github.com/134308
02:41:21brixenshould it ever call get_object() on a handle that has <= 0 references?
02:41:39brixenI would think not
02:41:54brixenbecause if the GC got there first, that object would no longer be valid
02:41:55brixenright?
02:42:22brixencooking dinner, brb..
02:43:43evanyeah
02:43:45evanthats true
02:43:57evanthe checksum basically validates that
02:44:06evanand it's a bit stronger by testing a full 32bits
02:44:43brixenthe problem I see is this (if I've got this right)...
02:45:00brixena C ext allocs some memory that has a VALUE element
02:45:10brixenit does mem->obj = rb_something
02:45:28brixenNMF is destroyed and it does handle->deref()
02:45:41brixenand now mem->obj is a handle with 0 references
02:45:55brixenright?
02:46:11brixenI don't see how to keep mem->obj alive
02:46:23evanit can be kept alive via ANY reference
02:46:30evanso if it's not seen via rb_gc_mark
02:46:36evanit just have to be reference by anything else
02:46:41evanto keep the ref count above 0, and alive
02:46:47brixenright
02:46:48evansome way MRIs GC works
02:46:53brixenwell, wait
02:47:00evans/some/same/
02:47:12brixenreferenced by what exactly, to keep it alive
02:47:28brixenit can't just be a value in an alloc'd bit of memory, can it?
02:47:43evanthats not strong enough to keep it alive, no.
02:47:48evanbecause we can't see that.
02:47:50brixenI think that is the issue here
02:47:54brixenright
02:48:34brixenbecause the bonus memory is hanging off of the parser or emitter struct, which are themeselves put in a Data_Wrap_Struct
02:48:44brixenbut no the bonus pointer
02:48:49brixens/no/not/
02:48:54brixenunless I'm missing something
02:48:58evanok
02:49:03evani had to check the handle code
02:49:22evanif what you said happens happens
02:49:31evanand references_ drops to 0
02:49:34evanit's put into weak mode
02:49:45evanwhich means the handle stays alive if the object the handle points to stays alive
02:50:10evanso if there is any way the object the handle points to is alive, the handle should be alive
02:50:15brixenright
02:50:18evanif the object is GC'd, the handle is GC'd
02:50:22brixenok, I was tracking that right then
02:51:04brixenso, a call to get_object() when the handle references == 0 is valid
02:51:26evanyes
02:51:44evanreferences_ is strong_handle_references
02:51:57evanmeaning, that something in capi has said "this handle needs to stay alive for me"
02:52:02brixenbut, when the gc runs, if references == 0 and the CApiHandle is not seen, the GC collects it
02:52:05brixenright?
02:52:14evanwell, no.
02:52:19evanit shouldn't.
02:52:37evanif references is 0 AND the object the handle points to is being GCd
02:52:39evanthe handle is GCd
02:52:50brixenright
02:52:52evanso, for example
02:52:54brixenthat's what I meant
02:52:55evanrb_cObject
02:52:57evanreturns a handle
02:53:00evanthat never dies
02:53:06evanbecause Object is never out of scope
02:53:21brixengotcha
02:53:22evanok, just wanted to be clear
02:53:26brixenyeah
02:53:32evanthis is tricky
02:53:44evanbecause i've built a garbage collection system on a garbage collection system
02:54:13brixenso, if nothing references the CApiHandle object AND references == 0, GC collects (just to be sure I can state this in words)
02:54:16brixenright?
02:54:17evanthis is both awesome and crazy at the same time
02:54:25evans/this/which/
02:54:54seydarhypothetically speaking if I wanted to eavesdrop on the conversation, what is the background I'd need to know?
02:54:56evanwait.
02:55:02evanwhy do you say CApiHandle there
02:55:24brixenvm/builtin/capi_handle.hpp
02:55:31evanno.
02:55:35evanthen
02:55:38evanok.
02:55:48brixenNativeMethodFrame::get_handle
02:55:50evanhere's how it goes.
02:56:01evancapi::Handle.object_
02:56:09evanpoints to a managed object
02:56:29evanif references_ > 0, the GC keeps Handle.object_ alive
02:56:43evanif references_ == 0, the GC checks if Handle.object_ is still alive
02:56:47evanif it is, nothing happens
02:57:08evanif it's becoming garbage, the capi::Handle is deallocated
02:57:24evanCApiHandle is simply a wrapper for capi::Handle.object_
02:57:59evancapy::Handle.object_->get_ivar("capi_handle") == CApiHandle that wraps capi::Handle
02:58:52brixenand "the GC checks if Handle.object_ is still alive" means finding the object when the GC does the transitive closure walk, yes?
02:59:02evancorrect.
02:59:05brixenk
02:59:34brixenI still think these assumptions are breaking down, but I need to trace this through the syck code
02:59:46evancould very well be
03:00:07evananswer the question: how is ->bonus seen by the GC
03:00:14brixenyes
03:00:19evanvia rb_gc_mark? rb_global_variable?
03:00:38brixenI need to figure it out, but yes, that's the question I have too
03:00:42evanis it simply shoved into an array or somewhere else the GC sees
03:00:47brixenso I'm on the path :)
03:00:51evanthat 3rd one leverages the fact that MRI is non-moving
03:00:55evanand ANY reference keeps an object alive
03:00:56brixenright
03:01:07evanso you toss a ref somewhere the GC can see
03:01:23evanthen you can have a zillion places that VALUE is stored in unmanaged memory
03:02:04evan(this, for those playing along at home, is why no one has ever fit a generational GC to MRI successfully)
03:02:09brixengot it
03:02:14evanok, need to run to the store.
03:02:15evani'll be back later.
03:02:17brixenok
03:04:08seydarbrixen: if it's safe to interrupt, _why_ are there a zillion places that VALUE is stored?
03:05:40brixenseydar: in MRI a VALUE is a pointer to an object
03:06:56seydarand the pointers are located in unmanaged memory?
03:07:56brixenwell, generally a pointer to unmanaged memory is associated with an MRI object using Data_Wrap_Struct
03:08:16brixenwhich creates a managed object that contains the pointer
03:08:47brixenok, eating dinner, bbiab..
03:08:53seydarso i learned recently, but what exactly causes the pointer to be duplicated?
03:09:02seydarlame. dinner should be eaten according to EST
03:09:46brixensorry :P
03:10:02brixenthe issue here isn't pointer duplication
03:10:11brixennot sure what the problem with that would be exactly
03:10:26brixenanyway, bb20m EST ;)
03:10:27seydarI'm trying to decipher "<@evan> then you can have a zillion places that VALUE is stored in unmanaged memory"
03:10:36seydari'll be waiting
03:10:46seydarrue_XIV: ping!
04:23:08seydarpax, children
07:44:47brixenevan: see vm/builtin/data.cpp
07:45:18brixenevan: but, trying to implement ::mark, I'm running into vm.cpp : 58 current_mark(NULL)
07:45:50brixenwhich, when rb_gc_mark executes, segfaults of course
07:46:00brixenbecause gc is NULL
07:49:24brixenhmm
07:52:39evanhm.
07:53:04evanbrixen: ah ah!
07:53:05evanof course!
07:53:08evanall my fault!
07:54:50brixenevan: well, I'm a little further
07:54:59evandid you get the current_mark passed down?
07:55:01evanthats all that needed to do
07:55:08brixenyeah, one sec
07:55:10brixenI'll diff ya
07:55:13evank
07:55:21brixenI was totally chasing my tail on the damn handle
07:55:26brixenfacepalms
07:55:29evani was going to say, just throw it into a thread local
07:55:32evanyeah
07:55:34brixenhandles*
07:55:39evanwhen it was all my fault!
07:55:41evansorry :/
07:55:44brixenwhich seem to be working fine
07:55:48brixennah, my fault
07:55:56brixenI have a totally simple repro too!
07:55:58brixensec..
07:58:12brixenhttp://gist.github.com/134406
07:58:25brixenI just followed the existing code for ::mark
07:58:36brixenI wasn't reading the commented code at first
07:58:47brixenbut why does it only call for one Data object?
07:58:56brixenit misses all the rest
07:59:08brixenand they get collected
07:59:45brixenoh, wait, 0x1b228c0 gets deleted too
08:00:17evan::mark should be called on all live Data objects
08:01:16brixenwell, every one of those dates is in an array, and the date obj should have a data_wrap_struct from Date#to_yaml
08:01:35brixenand that struct has a function to mark the bonus fields
08:02:09evanso is this not working?
08:02:34brixenit appears that ::mark is not called on those data objects, yes
08:03:03evanDate (not Data) uses Data_Wrap_Struct?
08:03:06evanis that what i'm seeing?
08:03:19brixenwell
08:04:00brixenhm, do you have the mri code handy?
08:04:05evanyeah
08:04:12brixenI can take you through the yaml code
08:04:23evanfffun!
08:04:26evan0xfffun!
08:04:28brixensee lib/yaml/rubytypes.rb
08:04:29brixenheh
08:04:33brixenclass Date
08:04:54evanok, i see it.
08:04:54brixenthen lib/yaml.rb : def YAML.quick_emit
08:05:10evanok
08:05:23brixenthen syck/rubyext.c syck_emitter_s_alloc
08:05:34brixenand syck_emitter_reset
08:05:46brixenquick_emit will create the emitter obj
08:05:53evanok
08:05:57brixenwhich will call syck_emitter_s_alloc
08:06:02brixenand syck_emitter_reset
08:06:10brixenwhich is Emitter#initialize
08:06:31brixensyck_emitter_s_alloc call Data_Wrap_Struct
08:06:31evanok
08:07:02evanyep
08:07:10evanand i see bonus-> fields set in _reset()
08:07:15brixenright
08:07:22evanand I see syck_mark_parser
08:07:27evanmarking bonus fields
08:07:48brixenin the bottom part of that gist, line 288
08:08:07brixenthere is only one call to Data::Info::mark before all those deletes
08:08:13brixenthat's what I don't understand
08:08:33evanoh
08:08:35evanthats easy.
08:08:43evanthere is only one Data_Wrap_Struct in use
08:08:55evanthe syck parser/emitter object itself
08:09:21evanparser_s_alloc is the only thing here that creates Data objects
08:09:34evanok, there are a few more
08:09:38evanbut thats the main one for this
08:09:54evanyou call it, and it calls rb_gc_mark
08:09:57evanon all the bonus fields, etc.
08:10:22evanerr., i meant syck_emitter_s_alloc
08:10:24evanbut same diff really.
08:10:27evanit's the same pattern.
08:10:46brixenhmm, I was thinking quick_emit was creating an emitter for each obj
08:11:06evani don't think so
08:11:11evanI think it calls reset on the same one
08:11:17brixenyeah, checking now
08:11:21evanyeah, that emitter.reset
08:11:47brixenI didn't think of that, since opts = {} in both quick_emit and in Date#to_yaml call to quick_emit
08:11:48evanhm, it does call Emitter.new.set_resolver()
08:11:49evanthough...
08:12:20brixendef YAML.emitter; Emitter.new.set_resolver( YAML.resolver ); end
08:12:27evanhm.
08:12:33brixenemitter in emitter.reset has to be that method
08:12:34evanoh!
08:12:35evander.
08:12:42evanall the other ones are dead already!
08:12:48brixen:C
08:12:48evanthey're already garbage
08:12:59evanso of course ::mark is called on them
08:13:27brixenyou mean delete?
08:13:30evanno
08:13:36evani mean that the emitters live a very short time
08:13:49evanso they are created, do their business, and become garbage
08:13:56evanbefore the GC runs even once
08:14:08evanthe case you're seeing
08:14:20evanyou managed to catch one live, during a GC
08:14:30brixenok
08:14:40evanif you want to validate this
08:14:54evanyou could add a counter to find_lost_souls in baker.cpp
08:15:00evanto count the number of Data objects seen as dead.
08:15:06brixenok
08:15:09evanor register that Data needs cleanup
08:15:17evanand you could print out all the dead Data objects
08:15:37brixenso, why are we trying to access a deleted object then
08:15:42brixenwhat am I missing
08:16:16evanok, lets go back to your problem.
08:16:29brixenvery bottom of the gist
08:16:47evanyes, i see us marking bonus->port
08:16:51evanbut then, right below
08:16:53evanwe delete it
08:17:19evanit would seem THATS the issue.
08:17:37evanso, what is rb_gc_mark doing
08:17:47evanbecause it's not saying "HEY ASSHOLE, I NEED THIS TO STAY ALIVE" properly
08:18:01brixenwell, that and we never mark 0x1b22910
08:18:07brixenand that's the one we try to access
08:18:10brixennot the one we marked
08:18:23evanwell
08:18:23evanok
08:18:33evanprint out all handles passed to rb_gc_mark
08:18:36evanand validate that
08:18:40brixenk
08:18:50evanmy hunch is you'll see it marked.
08:19:08evanbecause, here's what i'm guessing is happening.
08:19:12evanreferences_ is at 0
08:19:25evanwell, hm.
08:19:29evanlet me think about this...
08:20:13evanbrixen: where did you put that "deleteing <addr>" line?
08:20:19evanwhat file/line
08:21:04brixenin ~Handle()
08:21:23brixenvm/capi/handle.cpp
08:21:29evanok
08:21:37evanso do 2 things
08:21:41evanin rb_gc_mark
08:21:59evanon line 22, print out handle and handle->object()
08:22:18evanand in ~Handle(), print out object() as well
08:22:29evanwe need to match things up
08:22:33brixenk
08:22:41brixenthere are only 3 calls to rb_gc_mark
08:22:56evanthats fine
08:23:00evanthats EXACTLY what I expect
08:23:03brixenk
08:23:08evanthe 3 that emitter_mark calls
08:23:12evanfor the 3 fields in bonus
08:23:17brixenyeah
08:23:46brixenin rb_gc_mark do you want the original object() or res?
08:24:12evanwell, i want object() after the conditional code has run
08:24:20evanso we need to know what is handle now pointing to
08:24:24brixenk
08:24:27evanres might be NULL, we don't need to know that
08:24:37evanwe need to know handle->object()
08:24:48brixengot it
08:27:47brixenhttp://gist.github.com/134413
08:28:34evanah ha!
08:28:43evanwe did mark the handle we tried to access
08:28:55evanand we then delete it!
08:28:56evanOOPS
08:28:57evan:D
08:29:23brixenI see
08:29:47evanso why did we delete a handle we just marked?
08:30:02brixensomeone erased the chalk mark?
08:30:26evanwell, lets...
08:30:41evanwe're in a young gen sweep, yes?
08:30:55brixenyes
08:31:13brixenI was looking at ObjectMemory::prune_handles earlier
08:31:39brixenand baker.cpp : 130
08:31:45evank.
08:31:52brixennothing was standing out such as to poke me in the eye or anything
08:31:53evani was trying to find prune_handles
08:32:00evanah ah!
08:32:28evanok!
08:32:33evanso, baker.cpp:130
08:32:43evanthat code doesn't run for our handle in question
08:32:47evanbecause it's weak!
08:32:55brixenahh indeed
08:33:01evanwe instead allow the GC to find it
08:33:27evanok
08:33:44evanso the code at 157 must be wrong
08:34:38brixen157?
08:34:52evanobjectmemory.cpp:157
08:34:57brixenah
08:35:16evanif we marked 0x132dd20
08:35:29evanthen the code on 157 should come up false
08:35:38evanbecause it is young and it should be forwarded
08:36:20evanor it's not young
08:36:34evanso this code should blow by
08:37:37evanagardiner: hey there!
08:37:40brixenagardiner!
08:38:03agardinerhowdy!
08:38:11agardinerhow are you both?
08:38:20evanbrixen: on line 158, print out if obj is young and if obj is forwarded
08:38:40evanoh good
08:38:42evanwe're debugging
08:38:43evan:D
08:38:56agardinerwhatcha debugging?
08:39:00evanbrixen: and print out obj's address
08:39:02evanbrixen: the GC
08:39:06evaner.
08:39:09evanagardiner: the GC!
08:39:13agardinerfun!
08:39:51agardinerstill using immix?
08:39:59evanyep
08:40:06evanwe're debugging the funnest part
08:40:12evanthe capi Handle system
08:40:30agardinerah...
08:40:46agardinerdoes that use pinning?
08:41:03evannope
08:41:15evanwould be neat to use it though
08:41:20evanbut we'd have to be able to pin young objects
08:41:26evansomething i haven't implemented
08:42:35agardinerhave you come across anyone else using immix for real?
08:42:45evannot yet
08:42:48evanit works great
08:42:53evani'm sure someone will.
08:42:57agardinercutting edge!
08:42:59agardiner:-)
08:43:37evandaring even!
08:44:11agardinerso how goes progress?
08:44:18evangoing well
08:44:21agardinerseems like things are stabilising somewhat?
08:44:23evanbeen working on performance for a while
08:44:26evanyeah
08:44:26brixenevan: prune: 0x1b22930, 0x132dd20, weak: 1, young: 1, forwarded: 0
08:44:38evanwtf.
08:44:43evanwhy is it not forwarded?
08:44:43brixenI don't get it
08:44:55evansend me your diff again
08:45:03evansomething smells.
08:46:12brixenwell, this has the Data::Info::mark diff http://gist.github.com/134406
08:46:36evanno
08:46:38evani need the whole diff
08:46:42evangive me the whole thing
08:46:43evanall your changes.
08:47:08brixenincluding the syck/yaml import?
08:47:17evanok, minus that.
08:47:22evanat least everything in vm
08:47:27brixensec..
08:48:10evanFUCk.
08:48:26evandamnit.
08:48:32evani really wonder if this is going to be it.
08:48:35evango into vm.hpp
08:48:39evanchange current_mark to be
08:48:45evanObjectMark* current_mark;
08:49:17brixenhttp://gist.github.com/134418
08:49:28brixenk
08:50:13evanthen change your ::mark to do
08:50:30evanObjectMark* cur = ->current_mark;
08:50:35evan->current_mark = &mark;
08:50:42evan(*data...)();
08:50:46evan->current_mark = cur;
08:51:48evanand change rb_gc_mark to do
08:51:53evan->current_mark->call(..)
08:52:24evanare you testing this running the full specs?
08:52:34brixenyeah
08:52:48evanok
08:52:52evantry that
08:52:54evantell me what happens
08:53:02brixenrunning the code at the top of this is my repro http://gist.github.com/134406
08:53:06brixenbut I can run the full specs
08:53:21evanno no
08:53:22evanthats fine
08:53:29evanstick with what you've been using
08:53:36brixenk
09:00:30brixensame result
09:00:44evank
09:00:53evanset a breakpoint in rb_gc_mark
09:00:59evanon the call to ->call()
09:01:05evanyou're going to step into that
09:01:10evanand figure out wtf is going on
09:01:16brixenk
09:06:55evanif it's a young object
09:06:57evanand it is
09:07:02evanthen it should be forwarded
09:07:07evanand the bit set to indicate as such
09:07:15evanif it's not, we need to figure out why
09:08:00brixenk
09:27:37brixenwth http://gist.github.com/134432
09:30:57brixenhm, well the copy should not be forwarded
09:31:14evanoh.
09:31:15brixenthe original object->forwarded_p() is true
09:31:15evangeez.
09:31:20evani'm fucking stupid.
09:31:28evanyep.
09:31:35evani've got the logic all backwards.
09:32:44evanok, so do
09:34:00evancheck_forwards ? (obj->young_object_p() && young.validate(obj) != cValid) : !obj->marked_p()
09:34:07evanshould do the trick
09:34:21brixenahh ok
09:34:36brixenI couldn't figure out the logic with forwarded_p() :P
09:34:59evani got confused because most code is looking at refs and asking "was the object seen?"
09:35:26evanrarely is code expecting to find the forwarded copy somewher
09:35:29evanthats why
09:35:34brixenmakes sense
09:36:21evanooh yeah
09:36:23evancheck this out: http://gist.github.com/134435
09:36:41evanthe IC stats
09:37:28brixenwhoa
09:37:39evan97.16% of ICs had one class ever seen
09:37:47agardinerhow do you get 0 class? is that an IC that hasn't been called at all?
09:37:51evanyeah
09:37:58evanthose are ICs that haven't been used
09:38:01evanfor whatever reason
09:38:10brixeninteresting jump from 3 to 3+
09:38:11evanmethods not executed, branches not taken, etc.
09:38:27evanyeah, thats actually consistent with the research
09:38:49brixenso 3+ is basically megamorphic
09:38:50evanthere is a rise in the graph toward the mega-morphic end
09:39:06evanbecause there is a bunch of utility code that ends up being megamorphic
09:39:16brixencan you print those out? ie code location?
09:39:26evangreat example: Module#attr_accessor
09:39:39evanbecause it calls attr_reader and attr_writer internally
09:40:05evanand the receiver is pretty much always seen as a metaclass
09:40:11evanthus, different.
09:40:23agardinerdoes this mean have you implemented PICs now?
09:40:25evanbrixen: which do you want printed out?
09:40:34evanagardiner: i could implement them in a jiff
09:40:36evanwas going to tomorrow
09:41:28brixenevan: the code location of the megamorphic calls
09:41:33agardinercool... wonder if it will help much. If you recall, I implemented something simple in shotgun, but didn't commit it because it made no difference to perf
09:41:42evanyeah
09:41:47evani doubt it will make much perf difference
09:42:20evanbrixen: i could probably figure out a way to, yes.
09:42:56agardinergeez, i can't believe that was like a year ago...
09:43:01evani know
09:43:03evantime flies
09:43:13agardinerdoesn't it...
09:48:38brixenevan: I think we should re-convene further into the AM
09:48:39brixenhttp://gist.github.com/134438
09:48:56evanyeah
09:48:57brixensee line 16
09:48:59evani think so too
09:49:31evansee ya in the morning.
09:49:35brixenk
09:49:38brixennite!
09:49:57agardinernight
09:50:07brixenmorning agardiner! heh
09:50:16agardiner:-D
09:50:21brixenI can't believe how much earlier it is in the UK
09:50:28evanseriously.
09:50:29evanoh btw
09:50:38evanbin/mspec ci -T -Xic.stats
09:50:47evan1 class: 97.12%
09:51:02evan1 class: 110288 97.1255%
09:51:02evan2 classes: 1728 1.52177%
09:51:03agardinerah... its later. coming up for 10am Tues
09:51:12brixeninteresting
09:51:21brixenevan: you should try that with rails -h right quick
09:51:29evanumm
09:51:41evanlets see how quickly the gem installs
09:51:46brixenheh
09:51:50brixenI thought you had it already
09:52:00agardinerthis is a striaght count of ICs by classes, right - not based on usage?
09:52:00brixenrun ic.stats on gem install!
09:52:03evanI did at one point
09:52:12evanagardiner: no, thats ALL usage
09:52:43evanthats saying, 97.12% of ICs only ever seen one receiver class
09:52:50evanonly ever see
09:53:05evanbrixen: ok, running...
09:53:18evanwow, this is awesome
09:53:20agardinerright, but the 1% that see more than 1 class might be called many more (or less times), no?
09:53:25evan1 class: 26018 97.1836%
09:53:25evan2 classes: 333 1.24384%
09:53:30evanthats for rails -h
09:54:00brixenagardiner: ah interesting point
09:54:09evanagardiner: i've got hit counters on all the entries
09:54:10brixenevan: do you have a call count in the ICs?
09:54:13evanyes.
09:54:17evancall counters per class
09:54:24brixennice
09:54:26agardinerthat would be interesting
09:54:34evantell ya what
09:54:36evani'll find out
09:54:38evanIN THE MORNING!
09:54:41brixenheh
09:54:42agardiner:-D
09:54:43brixenOK
09:54:52evannite / morning boys
09:54:56agardinerget some sleep then, you slacker! :-P
09:55:06evani'm off to sleeps!
09:55:08agardinernite
09:55:17brixenwill have nightmares about handles and forwarded objects
09:55:34evanhehe
09:55:35agardinerhah, my nightmare last night was workflow
09:55:39evansleeps
17:24:50evanmorning
17:25:00evanbrixen: i realized last night as I was brushing my teeth whats going on
17:26:33sbryant_workmorning
17:26:42evanmornig
17:26:44evanmorning
17:29:13agardinermorning
17:29:13sbryant_workCrazy weekend?
17:33:26evansbryant_work: no, not too crazy
17:33:27evanyou?
17:34:08sbryant_workDecently crazy.
17:34:40sbryant_workWent out drinking with a friend. We end up in a "fancy" bar and he goes to pay.
17:34:50sbryant_workWe get up to leave and he takes off.
17:34:57sbryant_workLike bolts down the street.
17:35:13sbryant_workhe was already drunk and thought everyone was bailing on the check.
17:35:22sbryant_workSome people do not hold their booze!
17:36:38brixenevan: morning
17:36:45brixenwhat's going on? :)
17:36:45evanbrixen: morning
17:36:48evanok
17:37:03evanthere are actually 2 cases for objects seen on line 157
17:37:25evan1) objects marked which have been marked via rb_gc_mark
17:37:37evan2) objects which have been marked, but not via rb_gc_mark
17:37:45brixenok
17:38:23evanby changing it like we did, we made one case work, and one case break
17:38:45evaneasiest fix is to rearrange the code a little
17:38:49brixenk
17:38:49evanso that if check_forwards is set
17:38:52evanand the object is young
17:39:10evanif it's cValid, don't do anything with it, keep going
17:39:33evanif it's forwarded, do the code on 172
17:39:44evanif it's neither, do the code on 160
17:40:44brixenk
17:41:13agardinerright, i'm off...
17:41:18agardinercatch you later
17:41:20brixensee ya agardiner!
18:11:29brixenwoot
18:11:47brixenoh dr phoenix, your Rx appears to work, confirm I read your handwriting
18:11:48brixenhttp://gist.github.com/134686
18:12:06brixentracking down an issue with Qnil passed to rb_gc_mark now
18:12:16evanah
18:12:17evanthats easy
18:12:24brixenbut it ran for a long time, through many collections
18:12:28evanyou probably just need to check for references
18:12:31evanyay!
18:12:31brixenyeah
18:12:36brixenyay indeed!
18:12:39evanoh btw, interesting numbers: http://gist.github.com/134687
18:12:42evanjust ran those
18:13:01brixenwow, that is a long tail
18:13:17evanyep
18:13:28sbryant_workWhat do those numbers mean?
18:13:46brixenand it would be interesting to see the proportion of calls in that 2.88%
18:13:53evansbryant_work: thats enumerating all the InlineCaches in the system
18:13:57evanie, every sendsite
18:13:58brixenie the tail
18:14:27evanand this says that 97.12% of sends always had the same receiver class
18:14:29brixenevan: you might just have gotten us to the point where PIC shows results!
18:14:40sbryant_workThanks!
18:14:46evanperhaps!
18:15:17evani'm going to generate some stats that take the number of hits into account
18:15:25brixencoool
18:24:38brixenwooot!
18:24:41brixenit finished
18:24:59evanWOO
18:25:00brixenhad to check that handle itself was a reference in rb_gc_mark
18:25:07evanyep.
18:25:09evanmake sense
18:25:14brixenexcelente
18:25:19evanyay!
18:25:25brixenfucking crazy how cool these handles are
18:25:30evanyeah
18:25:31brixentake that syck!
18:25:38evani'm really happy with how they turned out
18:25:51brixenme too
18:28:12sbryant_workneat-o
18:28:23sbryant_workI wish this damn project would end so I could work on more interesting things :-\
18:28:39evansbryant_work: what you work on?
18:28:55sbryant_workA rails app this legal company bought.
18:29:08sbryant_workThey want to do things that aren't very rails "friendly"
18:29:24sbryant_workSo it's a lot of extra code, doing ugly things and waiting on red tape
18:29:31evanfun. :/
18:29:44sbryant_workBut after it's done they told me I could work on whatever I'd like
18:30:22sbryant_workImagine having 7 different database (with all the same structure) and you have to pull data from each of them on a single request.
18:31:06evanhehe
18:31:09sbryant_workOh and you have another database for the "core" of the application
18:31:12evanI just called a variable
18:31:14evanthese_hits
18:31:22evanbecause i've got 3 other variables named varations of hits
18:31:24sbryant_workSo the 7 different databases aren't directly related to your site.
18:31:57sbryant_workI'm wrapping things in modules based on the database they talk to
18:31:57eneboevan: Are your stats generated via a rubyspec run?
18:32:07evanthe last gist was
18:32:08evanyes
18:32:11enebook
18:32:16sbryant_workso they do things like set_table_name and establish_connect :db
18:32:26evani'm generating another one now
18:32:32evanthat takes into account the usage of the IC
18:32:46eneboI have always been interested in gather these stats too
18:32:47sbryant_workis this for bench.rubini.us?
18:32:48evansbryant_work: ug. yes, i've done that too.
18:32:54evansbryant_work: partly
18:32:56enebofor various site caches
18:33:07sbryant_workevan: I tried to dynamically generate all the code
18:33:28sbryant_workbut that didn't work out as planned because I couldn't get that call to establish_connect be lazily evaluated
18:33:50evanok: http://gist.github.com/134687
18:33:52evancheck out the bottom
18:34:27evanthat bottom one is how many hits an IC containing that number of tracked classes
18:34:53evanso, interestingly, there are very few megamorphic Ics
18:34:57evanbut they're hit A LOT.
18:35:10evanactually
18:35:16evan I might be skewing this...
18:35:23evanlet me check.
18:37:46evanhm, so for the megamorphic counters
18:38:16evanthere is a counter thats incremented everytime there is no more room for any more tracked classes
18:38:30evanwhich would be run everytime the IC is wrong
18:38:39brixenwell, looks like ~20% in 2-10
18:38:49brixenso a PIC should make a big diff
18:39:07evanperhaps up 4
18:39:13evanafter that, it's not as useful
18:39:44brixenyeah, 2-4 is > 15%
18:39:57evanlet me try something...
18:40:19brixenman, I'd love to know where these are
18:40:23brixenlike the 6 one
18:40:29eneboevan: I am wonder which sites are in 10+....
18:40:35evanyeah
18:40:40brixenenebo: heh, me too
18:40:41evaneveryone wants to know where they are!
18:40:47enebofind them!!!! :)
18:40:49brixenpublish a zine!
18:40:54evangets out his cartographers hat
18:41:00brixeninquiring minds want to know
18:41:09brixenuse your 3gs compass!
18:41:16evanhah
18:41:57sbryant_workAre you going IC hit hunting?
18:42:32evaninteresting
18:42:40evanmy change didn't change the stats, thats good
18:43:41evanok, lets see
18:43:57evanif i can print out the file:line of all megamorphic sends
18:46:26enebosome hash call in map :)
18:46:39evanprobably something like that
18:46:48evanwe'll see shortly!
18:49:04sbryant_workThis channel is pretty exciting today.
18:49:45boyscoutBetter explanation of the benchmark graphs. - 34443d5 - Brian Ford
18:49:45boyscoutAdded rb_io_write and specs. - a3c74cb - Brian Ford
18:49:45boyscoutAdded rb_gc_mark_maybe. - 26c1844 - Brian Ford
18:49:45boyscoutRake task to compile syck extension. - 7cf24f4 - Brian Ford
18:49:45boyscoutAdded Syck extension. - 8776651 - Brian Ford
18:49:46boyscoutAlways compile extensions with debug symbols. - 3ffd142 - Brian Ford
18:49:48boyscoutDon't bus error if rb_global_variable gets invalid address. - 13bb5be - Brian Ford
18:49:50boyscoutAdded rb_hash_size. - 869ec1d - Brian Ford
18:49:50evanWOO
18:49:52boyscoutAdded OBJ_TAINT, OBJ_TAINTED. - e823303 - Brian Ford
18:49:54boyscoutFixes for Syck extension integrated in rbx. - e4ee79b - Brian Ford
18:49:55evanKEEP COMIN'
18:49:56boyscoutRemoved failing tags for YAML. - 7b3e2ad - Brian Ford
18:49:57evanGO GO GO
18:49:58boyscoutRemoved RbYAML. - 7f63316 - Brian Ford
18:50:00boyscoutFixed sprintf format warning on gcc 4.3. - fbbcf02 - Brian Ford
18:50:01evanCHUG
18:50:02boyscoutFixed rb_gc_mark to check handles are references. - 835263c - Brian Ford
18:50:03evanCHUG
18:50:04boyscoutDr Phoenix's Rx for the treatment of weak and strong handles. - ee676d8 - Brian Ford
18:50:04evanCHUG
18:50:08brixenhaha
18:50:11evanHAHAH
18:50:13evani love that last one
18:50:24brixenI think I get point for best commit msg in a while :)
18:50:29brixenpoints*
18:50:30evanyou do
18:50:34evanbrixen += 5
18:50:40brixenrock
18:51:25brixenscouts breakfast while boyscout chugs
18:52:04sbryant_workThat was awesome.
18:54:17evanwoo
18:54:22evani think i've got IC locations
18:54:33brixenschweet
18:57:50boyscoutCI: ee676d8 success. 2709 files, 10776 examples, 33796 expectations, 0 failures, 0 errors
18:57:57evanwoo
18:59:46brixenyay
19:01:52evanok
19:01:54evancheck out http://gist.github.com/134687
19:02:03evani've printed out the file:line of the megamorphic sites
19:02:15evanand run them through a script that shows the code at that location
19:02:24evani'll run this on the specs now.
19:03:40evana lot a lot of these are class/module related
19:03:43evanwhich makes sense
19:06:42eneboand may or may not be hot on a more conhesive set of code (e.g. something which runs the same code a lot versus something which spans the entire runtime+libraries)
19:06:45brixeninteresting!
19:06:47evanspecs!
19:06:48evanhttp://gist.github.com/134720
19:07:35brixenline #s might be a bit off
19:07:49brixeneg kernel/common/marshal.rb 273
19:08:47evanif they're off
19:08:54evanthey're usually one line too far down
19:09:01brixenyeah
19:09:08evanyeah, like that one
19:09:15evanit's object_id there probably
19:09:28evanhey
19:09:29evanyou know
19:09:33evani should be printing out the name of the IC
19:09:41brixenso, am I reading right, #object_id, #send, #kind_of?
19:09:43evanand all the classes recorded there
19:09:47brixenyeah, I was thinking that
19:45:10evanok, print out the names and classes helps
19:45:16evanthe line numbers are definitely off
19:45:19evanneed to fix that.
19:46:07evanlet me gist this...
19:46:49evanhttp://gist.github.com/134687
19:47:09evanok, you can see how the mega's are basically not even using the cache
19:47:20evanwhen it's called on classe
19:47:30evanit ends up just cycling through
19:47:43evanmaking little to no use of the cache
19:47:47brixeninteresting
19:49:35evani could adjust the caching
19:49:46evanso that when it adds a new class
19:49:52evanit kicks out any class that had 0 hits
19:50:02evanrather than letting it fill up with classes with 0 hits
19:50:21brixensounds reasonable
19:50:28evanlets see what that does...
19:50:42evanthis is some fun research
19:50:49brixenindeed
19:51:18sbryant_worksounds like a plan
19:53:12evanwow
19:53:22evandoing that means there are 3 times fewer mega-morphic sites
19:54:04evanbecause megamorphic now means at 10+ receiver class that had at least one cache hit
19:55:05evanhttp://gist.github.com/134687
19:55:17evanbottom one is the new logic, reusing any tracking spot with 0 hits
19:55:38sbryant_workwow.
19:55:53sbryant_workOrders of magnitude difference.
19:56:10evanyeah
19:56:15evani'll run the specs now.
20:04:29evanhehe
20:04:30evanrock
20:04:39sbryant_workwhat's the results?
20:04:43evan__metaclass_init__ is megamorphic
20:04:48evanand all the results are
20:04:53evanMetaClas:<metaclass>
20:04:57evanie, a metaclass of a metaclass
20:05:06evansbryant_work: it's crashing on the specs
20:05:08evani need to figure out why
20:06:53sbryant_workgood luck!
20:07:37evanI find it interesting how badly ICs deal with Metaclasses
20:08:29brixenMetaclass, the IC repellant
20:09:56evani wonder if i can find any info on how smalltalk dealt with this
20:10:03evanthey've got the same problem here
20:10:11evansince their classes have metaclasses in the same way
20:11:59brixenevan: this is interesting http://www.jot.fm/issues/issue_2009_01/article4/
20:12:08brixensearch for metaclass
20:13:05brixenObj-C 2.0 is sounding kinda hip
20:16:32brixenevan: do you have this paper http://portal.acm.org/citation.cfm?id=141936.141947
20:17:48evanmy acm membership expired, maybe I should renew it
20:17:56evanyeah, i've read that one
20:18:05evanit's all about these crazy techniques for organizing method tables
20:18:08evanas I recall
20:18:56evanok, time to push this.
20:25:16evanthat paper uses lazy, versioned IC clearing
20:25:19evaninteresting
20:25:45brixencan you email it to me?
20:25:56evanthe first one
20:26:00evanthe jot.fm one
20:26:13brixenahh
20:26:28brixenhaven't finished reading it
20:27:36evani'm skimming
20:40:58boyscoutFix mis-cache of methods called via #send - 3b291e0 - Evan Phoenix
20:40:58boyscoutAdd klass tracking in ICs, add ability to disable the IC - 362c395 - Evan Phoenix
20:40:58boyscoutFix up IC profiling, add -Xic.stats - 66c60d8 - Evan Phoenix
20:45:37boyscoutCI: 66c60d8 success. .................................................................................................... .................................................................................................... .................................................................................................... ...............................................................................
20:47:04brixenhm
20:47:40evan:/
20:48:17evani'm having it manually build again
20:50:28boyscoutCI: 66c60d8 success. 2709 files, 10776 examples, 33796 expectations, 0 failures, 0 errors
20:51:08evanhm, ok
20:51:11evanwell
20:51:15evanthat means it's lunch time.
20:51:57brixenyah, moi aussi