Da
Show enters and exits. Hide enters and exits.
| 00:40:18 | headius | hey guys, I thought of another concern for capi |
| 00:40:27 | headius | that pushed me back into the "we won't implement it" column |
| 00:40:42 | headius | it seems like it will make it impossible to get rid of the gil |
| 00:41:18 | headius | since you could have threads manipulating data structs in both C and Ruby at the same time, which for you guys with copying will be a showstopper |
| 00:42:08 | headius | the only way it could work is if all threads have to freeze when extension code is running, which means you'd have to wait for them to reach safe points every time |
| 00:42:53 | headius | it's also totally incompatible with MVM unless you use the new MVM extension stuff ruby 1.9 guys have been working on |
| 00:48:21 | rue | Well, no, it just means that synchronisation may be needed when an extension accesses Ruby data |
| 00:48:38 | slava | VM extensions |
| 00:48:38 | slava | lol |
| 00:48:48 | slava | just port everything to FFI mang |
| 00:49:14 | rue | slava: They put too much logic in the extensions |
| 00:49:39 | sbryant | slava: in a perfect world, but support Capi will make rubinius that much more attractive from an adoption standpoint |
| 00:49:45 | headius | rue: synchronization on every access would be even worse |
| 00:50:03 | headius | since you'd have to have synchronization on the ruby side too, and you wouldn't know when there might be an extension modifying it |
| 00:50:09 | slava | sbryant: depends on how much capi affects stability, performance, concurrency, etc |
| 00:50:15 | headius | you'd essentially have to make all core collections be synchronized |
| 00:50:18 | slava | sbryant: people who want perfect compatibility with extensins will use mri anyway |
| 00:50:31 | headius | slava: yeah, that's about what I figure too |
| 00:50:34 | rue | headius: *Maybe* synchronise |
| 00:50:44 | sbryant | slava: I agree-ish |
| 00:50:52 | sbryant | It's a pretty gray area. |
| 00:51:02 | headius | rue: there is no such thing as maybe synchronize, unless you're going to add another conditional check on every access to a collection to see if it's been leaked to the outside world |
| 00:51:10 | headius | and then hope that across threads that flag isn't stale in a cache somewhere |
| 00:51:14 | rue | 'Course there is |
| 00:51:31 | rue | I do not mean a runtime check |
| 00:52:28 | rue | The current handle implementation uses actual handles, which should take care of objects the GC moved |
| 00:53:21 | evan | headius: some of those are probably true |
| 00:53:33 | evan | headius: but thats still not a good enough reason to not do it |
| 00:53:44 | headius | well, I disagree on that point |
| 00:53:44 | evan | we just fall on different sides of this line |
| 00:53:56 | evan | sure, you're allowed to disagree. |
| 00:53:58 | rue | Which data structures are being manipulated? |
| 00:54:17 | rue | That is, without a user explicitly doing so without synchronising themselves |
| 00:54:24 | headius | you should probably make sure people know that's going to be a problem with native exts |
| 00:54:49 | headius | if we could get enough people showing that native exts prevent parallel threading in the C impls we could start weaning more people off them |
| 00:55:07 | headius | and with things like the swig ffi generator, they shouldn't need them in most cases anyway |
| 00:55:40 | headius | this might be a good point to bring up during our talk |
| 00:55:47 | sbryant | Probably a stupid question but is there an FFI for MRI (1.8.x and 1.9.x) ? |
| 00:55:50 | headius | you like C exts, but do you know what you're sacrificing? |
| 00:55:56 | evan | headius: i think we can wean them off without saying "fuck you, rewrite them now." |
| 00:55:57 | rue | sbryant: ruby-ffi |
| 00:56:38 | rue | I am not above flat-out lying about it to get them to use FFI, but I still do not understand which data it is that we are worried about |
| 00:56:42 | headius | I dunno, I'm kind of a "fuck you" kind of gut |
| 00:56:50 | evan | i know you are |
| 00:56:53 | evan | i'm just not :) |
| 00:57:06 | headius | rue: like array data...you guys have to copy it in and out if people directly access it |
| 00:57:18 | headius | that only works if you can guarantee only one side or the other is accessing it at a time |
| 00:57:27 | headius | with parallel threads, you can't |
| 00:57:47 | evan | headius: i also thing there is a middle ground |
| 00:57:49 | rue | How is that different from manipulating an Array in two threads just in Ruby? |
| 00:57:58 | evan | just like people have migrated their extensions for 1.9 |
| 00:58:09 | evan | providing a saner C API can help mitigate |
| 00:58:20 | evan | and having our own capi impl means we can create those sane apis |
| 00:58:22 | evan | and provide them |
| 00:58:28 | evan | and provide shims to use them on 1.8 |
| 00:59:01 | rue | Nah. |
| 00:59:10 | headius | evan: certainly, and if people migrate to the sane API (which would probably look more like JNI) I'm all for implementing it |
| 00:59:16 | evan | right |
| 00:59:21 | headius | but implementing what we have right now is incompatible with parallel execution |
| 00:59:22 | evan | well, money talks |
| 00:59:28 | headius | so we can't really help on that end |
| 00:59:30 | evan | so having such an API means people can actually migrate |
| 00:59:36 | rue | Just provide an easy way to compile a library to which FFI can be attached |
| 00:59:45 | evan | headius: then hold off |
| 00:59:48 | headius | I think they might migrate this far and no further unless MRI starts to push people too |
| 00:59:51 | evan | let us weather this storm |
| 00:59:57 | headius | you need to get MRI on board |
| 00:59:58 | evan | and help write new, saner APIs |
| 01:00:07 | headius | or it's never going to happen |
| 01:00:10 | evan | not entirely |
| 01:00:13 | evan | because we can provide a shim.c and shim.h |
| 01:00:22 | evan | that implement the new APIs for 1.8 |
| 01:00:27 | headius | we need to get MRI to ship a ruby-safe.h though |
| 01:00:34 | headius | so people have the option of moving to these APIs |
| 01:00:34 | evan | doing so also means MRI is more likely to get involved |
| 01:00:39 | evan | because it lowers the barrier for them. |
| 01:02:08 | headius | yeah, don't think I don't appreciate what you're doing in this area |
| 01:02:24 | headius | I'm just bummed we can't implement what you have now without totally crippling threads |
| 01:02:29 | evan | sure |
| 01:02:34 | evan | it's a bummer |
| 01:02:36 | evan | but don't forget |
| 01:02:48 | evan | for us, this is an integration API |
| 01:02:56 | evan | in the same way you have all the code to integrate with java |
| 01:03:06 | evan | so it just pays off less for you than us |
| 01:03:08 | headius | oh yeah, I realize that |
| 01:03:15 | headius | you don't have a java to fall back on |
| 01:03:20 | evan | right |
| 01:03:25 | headius | I know you guys have to do something |
| 01:03:36 | headius | I just keep checking back on subtend to see if we'd be able to do it |
| 01:03:54 | headius | wmeissner convinced me again it's just not feasible |
| 01:03:57 | rue | headius: Am I missing something, or are you saying that the GIL is needed to protect against people who do not synchronise their threaded code? |
| 01:04:24 | headius | rue: I'm saying you could get native extensions and ruby code seeing different *copies* of the data at the same time |
| 01:04:25 | evan | headius: :D |
| 01:04:30 | headius | and having native code wipe out ruby code's changes |
| 01:04:35 | evan | headius: we can play a lot more tricks to make it work than you can |
| 01:04:43 | headius | if you didn't segfault, you'd surely corrupt |
| 01:04:45 | evan | headius: like me integrating support for it directly into the GC |
| 01:04:54 | evan | than you can't play |
| 01:04:55 | evan | rather. |
| 01:05:00 | headius | evan: copying is going to be the nonstarter though |
| 01:05:05 | evan | why |
| 01:05:13 | headius | you can't be copying data around and still allow concurrent access |
| 01:06:14 | evan | oh that |
| 01:06:19 | evan | well, JNI copies |
| 01:06:21 | evan | what does it do? |
| 01:06:36 | headius | in JNI you use an API to manipulate data struct internals that are shared with Java |
| 01:06:41 | headius | you never get a pointer to anything |
| 01:06:53 | evan | we never give pointers either |
| 01:07:14 | evan | so if thats the only requirement, we're good |
| 01:07:34 | headius | what about rarry ptr, or whatever it's called |
| 01:07:47 | rue | Those changes are synched, too? |
| 01:07:48 | evan | thats a pointer to C struct we allocate on the C heap |
| 01:08:00 | evan | rue: yeah, they're sync'd at critical points |
| 01:08:07 | headius | which contains what? |
| 01:08:15 | evan | an array of handles |
| 01:09:21 | brixen | also, modifying an ext to not use RSTRING and RARRAY is easier than rewriting it |
| 01:10:03 | headius | fair enough, I'm not arguing that...and if they don't use those they'll be closer to something we can support |
| 01:10:07 | headius | I'm arguing that they won't |
| 01:10:19 | brixen | they have the option to |
| 01:10:27 | rue | It *is* certainly possible to get bad data, but only if the user fails to synchronise |
| 01:10:29 | headius | I'll have to get wmeissner to talk to you guys |
| 01:10:32 | brixen | I'm not sure they'd rewrite in ffi either |
| 01:10:35 | headius | see if things have improved |
| 01:10:41 | brixen | but we give them that option too |
| 01:10:51 | evan | headius: on another topic |
| 01:10:53 | headius | he took a recent look and what he described that we'd have to do was simply not doable |
| 01:10:58 | evan | headius: how do ya want to handle the talk? |
| 01:10:58 | sbryant | headius: if you make the barrier of entry low, desire for portability will drive those simple rewrites |
| 01:11:02 | evan | just sync up on details in vegas? |
| 01:11:16 | sbryant | rather simple changes, not rewrites |
| 01:11:17 | headius | evan: yeah, we don't talk until day 2 |
| 01:11:22 | evan | headius: i'd love to hear his asessment |
| 01:11:24 | rue | headius: I think GC is where you would have a problem |
| 01:11:25 | headius | I'll get some time this coming week to work on slides |
| 01:11:27 | evan | headius: because maybe we've overlooked something |
| 01:11:42 | headius | evan: do you have a current plan for eliminating the gil? |
| 01:11:53 | headius | because I think that's where a lot of complexity comes in |
| 01:11:55 | evan | nothing solid no |
| 01:12:11 | evan | but the plan is how john rose pointed out to me |
| 01:12:26 | evan | which is take the one lock and first make it 2 locks |
| 01:12:27 | evan | then 3 |
| 01:12:28 | evan | then 4 |
| 01:12:42 | evan | which is take the one lock and first make it 2 locks |
| 01:12:43 | evan | then 3 |
| 01:12:45 | evan | then 4 |
| 01:12:52 | evan | (^^ for headius) |
| 01:13:12 | sbryant | evan: that's going to be painful :( |
| 01:13:19 | headius | bleh hotel wireless |
| 01:13:21 | headius | I sent this last: if you haven't started planning for that I'm worried you're painting yourselves into a corner |
| 01:13:24 | headius | I missed anything after |
| 01:13:43 | evan | sbryant: i'm open to other suggestions |
| 01:13:48 | evan | headius: nothing after that |
| 01:15:19 | headius | bleh |
| 01:15:21 | headius | I'm on another host now |
| 01:15:24 | headius | I'll read log for a sec |
| 01:15:27 | evan | heh |
| 01:15:29 | evan | nothing was said |
| 01:15:39 | evan | we waited for ya :) |
| 01:16:34 | headius | sigh...internet on the road |
| 01:16:37 | headius | logs are behind |
| 01:16:48 | evan | you hear me now? |
| 01:16:51 | headius | yes |
| 01:16:53 | evan | k |
| 01:16:56 | sbryant | evan: bah, new MBP isn't as fast as I'd like it to be, sorry for the delay. It's going to be painful no matter what you do. At least in my experience. |
| 01:17:06 | evan | sbryant: right |
| 01:17:24 | headius | logs must update every ten mins or so, eh? |
| 01:17:29 | evan | dunno |
| 01:17:51 | evan | sbryant: planning to remove the gil at this point would introduce too much additional work |
| 01:18:15 | evan | it's certainly on our minds, but we're not many/any decisions based on it atm |
| 01:18:18 | sbryant | evan: I have no perspective, there for cannot offer a valuable opinion |
| 01:18:27 | evan | sbryant: :D no prob. |
| 01:18:32 | sbryant | and I just Englished all over the place :D |
| 01:19:33 | sbryant | evan: what's currently on rubinius roadmap? |
| 01:19:48 | evan | brixen and I were just discussing that |
| 01:20:03 | evan | GIL removal is not an the roadmap for 1.0 |
| 01:20:09 | evan | it's in the cloud after 1.0 |
| 01:20:11 | sbryant | Understandable |
| 01:20:18 | rue | There really are not that many things that need to be locked |
| 01:21:09 | headius | it's not locking as much as preventing segfaults when people read across each others boundaries |
| 01:21:32 | headius | you can do all sorts of concurrent modification of structures in java, but you only get java exceptions |
| 01:21:37 | rue | Segfaults will teach them |
| 01:21:39 | headius | not crashes |
| 01:21:58 | headius | rue: heh, sounds like a 'fuck you' approach :) |
| 01:22:05 | sbryant | haha. |
| 01:22:11 | evan | headius: the handle API abstracts access pretty clearly |
| 01:22:46 | headius | did you and john get to talk much about how to eliminate the gil? |
| 01:22:47 | evan | so there is very limited chance for the user to corrupt a GC object |
| 01:22:49 | rue | headius: Yeah. Personally I have no problem with crashing if the user fails to synch their code |
| 01:22:51 | evan | headius: yeah |
| 01:22:55 | evan | for about an hour |
| 01:23:18 | evan | rue: esp if we convert the segfault into an exception! |
| 01:23:19 | evan | bingo! |
| 01:23:30 | sbryant | I'm all or that. |
| 01:23:32 | sbryant | for* |
| 01:23:36 | evan | i've done that before |
| 01:23:41 | rue | I was thinking more like converting it into some type of ASCII middle finger |
| 01:23:43 | evan | rubinius has had that feature on and off |
| 01:24:02 | evan | SEGFAULT DETECTED: Please wait while we call your mom. |
| 01:24:18 | sbryant | Just rickroll them. |
| 01:25:25 | evan | hehe |
| 01:25:59 | sbryant | He'll never let you down and hurt you. |
| 01:26:13 | evan | haha |
| 01:26:19 | sbryant | :D |
| 01:26:24 | evan | PONY TIME DETECTED: Preparing cake party... |
| 01:26:30 | sbryant | haha |
| 01:27:42 | sbryant | Does anyone here use emacs for development and if so do you have any tips for rubinius development/debuging |
| 01:28:01 | evan | <= macvim + gdb |
| 01:28:19 | sbryant | Yeah, I'm an emacs whore |
| 01:28:46 | sbryant | took me a while to learn to love it, but now I have Stockholm Syndrome and couldn't be happier. |
| 01:32:43 | brixen | evan: I'm tempted to leave MetaClass at global namespace Ruby should have that class anyway ;) |
| 01:32:43 | rue | cgdb is fun |
| 01:32:54 | brixen | evan: should I namespace it? |
| 01:33:10 | brixen | er because Ruby* |
| 01:33:18 | rue | MetaClass is probably the likeliest name conflict after Tuple |
| 01:33:32 | evan | nah |
| 01:33:36 | evan | leave it toplevel |
| 01:33:38 | brixen | k |
| 01:33:44 | brixen | and IncludedModule? |
| 01:33:51 | evan | mm |
| 01:33:53 | evan | namespace it. |
| 01:33:55 | brixen | k |
| 01:37:35 | sbryant | brixen: did you ever resolve that problem with pup.rb? |
| 01:37:45 | evan | yeah |
| 01:37:47 | evan | i solved it. |
| 01:37:50 | evan | like usual |
| 01:37:50 | evan | :D |
| 01:38:21 | sbryant | you're the man now dog :) |
| 01:38:36 | evan | do people even remember what movie that was from? |
| 01:38:45 | yakischloba | finding forrester? |
| 01:38:47 | sbryant | Finding Forrester |
| 01:38:51 | evan | double ding! |
| 01:38:53 | evan | ok, good. |
| 01:38:54 | sbryant | evan: seems so :) |
| 01:38:57 | evan | you can use it then. |
| 01:39:02 | evan | :D |
| 01:39:07 | yakischloba | (took me a second. star wars?) |
| 01:39:13 | sbryant | evan: it was on the teevee the other day |
| 01:39:28 | sbryant | In glorious HD. |
| 01:40:09 | evan | hah |
| 01:55:54 | evan | rad |
| 01:56:09 | evan | i've got code exposed to LLVM enough that it's making some rad optimizations |
| 01:56:25 | sbryant | evan: neat! |
| 01:56:27 | evan | http://gist.github.com/100866 |
| 01:56:34 | sbryant | evan: tailcall recursion optiomizations?> |
| 01:56:36 | evan | thats for |
| 01:56:42 | evan | def blah(obj); 1 == obj; end |
| 01:56:56 | evan | check out the bottom define |
| 01:57:00 | evan | thats the optimized version |
| 01:57:14 | sbryant | reading now |
| 01:57:41 | evan | line 81 is particularly awesome |
| 01:58:02 | sbryant | I wish this made more sense to me |
| 01:58:06 | evan | i'll explain |
| 01:58:14 | evan | see %arg_val on line 81? |
| 01:58:27 | sbryant | yeah |
| 01:58:57 | evan | thats obj, but accessed directly from the argument array |
| 01:59:04 | evan | which is passed in by the caller |
| 01:59:18 | evan | thats awesome because llvm has seen through 2 levels of storage to do that |
| 01:59:37 | evan | it 1) saw the move from the arguments array to the locals array |
| 01:59:45 | evan | and 2) saw the move from the locals array to the stack |
| 01:59:50 | evan | so it eliminates using both of those |
| 01:59:54 | sbryant | that is pretty jawsome |
| 01:59:55 | evan | and goes directly to the source |
| 02:00:00 | sbryant | <3 LLVM |
| 02:00:47 | evan | oh |
| 02:01:03 | sbryant | Is LLVM going to be required for rubinius? |
| 02:01:03 | evan | it also converted stack push/poping into direct stack references |
| 02:01:11 | sbryant | Oh that is awesome. |
| 02:01:28 | evan | ie, the stack pointer was competely eliminated |
| 02:01:32 | evan | it %stack_top in the top version |
| 02:01:33 | sbryant | Yeah I got that. |
| 02:01:35 | evan | it's |
| 02:01:42 | evan | you can see it being used |
| 02:01:50 | evan | and it completely eliminates it in the bottom one |
| 02:01:50 | sbryant | So instead of a dereference it can access the memory directly |
| 02:01:57 | evan | right |
| 02:02:09 | evan | instead of manipulating the stack pointer to point to the current top of the stack |
| 02:02:17 | evan | it did all the stack movement calculation at compile time |
| 02:02:29 | evan | and access the stack at fixed offsets instead |
| 02:03:24 | evan | sbryant: LLVM will be required for the JIT, yeah. |
| 02:03:25 | sbryant | is LLVM going to be required going forward? |
| 02:03:27 | sbryant | Or is it already |
| 02:03:29 | sbryant | ahh |
| 02:03:30 | sbryant | gotcha |
| 02:03:30 | evan | we currently ship with a version of LLVM |
| 02:03:44 | evan | so we'll probably do 2 things |
| 02:03:49 | evan | 1) upgrade our builtin version |
| 02:03:59 | evan | 2) only use it if there isn't an installed version |
| 02:04:12 | sbryant | I like that appraoch |
| 02:04:32 | evan | probably provide a way for the results of doing an internal build be saved |
| 02:04:37 | evan | ie, install that version |
| 02:05:05 | yakischloba | http://pastie.org/private/hvecgtowd5rxchurpunqa this a readline issue? need to update it or something? |
| 02:05:24 | evan | hrm |
| 02:05:26 | evan | yakischloba: what OS ? |
| 02:05:29 | yakischloba | tiger |
| 02:05:37 | evan | i thought we fixed this... |
| 02:05:59 | yakischloba | have i done something funky to my system then? what is the issue |
| 02:06:28 | evan | hm, no. |
| 02:07:17 | evan | hrm |
| 02:07:28 | yakischloba | fresh pull, btw |
| 02:07:32 | evan | yeah |
| 02:07:33 | evan | it's not that |
| 02:07:44 | evan | yakischloba: whats the output of |
| 02:07:47 | evan | sw_vers |
| 02:07:48 | evan | for you? |
| 02:08:05 | yakischloba | http://pastie.org/private/hvecgtowd5rxchurpunqa |
| 02:08:20 | evan | hm |
| 02:09:02 | evan | yakischloba: ok, do this |
| 02:09:06 | evan | gdb --args bin/rbx -v |
| 02:09:09 | evan | then, in gdb |
| 02:09:11 | evan | b mmap |
| 02:09:12 | evan | run |
| 02:09:17 | evan | when it stops on mmap |
| 02:09:19 | evan | do |
| 02:09:20 | evan | bt |
| 02:09:23 | evan | and paste the results |
| 02:10:15 | yakischloba | k sec |
| 02:11:00 | yakischloba | http://pastie.org/private/hvecgtowd5rxchurpunqa |
| 02:11:44 | yakischloba | http://pastie.org/private/sozmvti2abt6rxayaowmg |
| 02:12:05 | evan | ah yes. |
| 02:12:14 | evan | ok, that makes sense. |
| 02:13:17 | evan | yakischloba: i'll try and get to that a bit later |
| 02:13:22 | evan | it's something we need to adapt in the code |
| 02:13:33 | evan | or if you want |
| 02:13:33 | yakischloba | sure, np. holler at me when you push a fix, if you dont mind |
| 02:13:41 | evan | you could try replacing mmap with malloc |
| 02:13:47 | evan | see if you can get it going that way |
| 02:14:04 | yakischloba | bleh. i don't have time to get into it right now, i was just trying to build so i could play with it later this evening |
| 02:14:13 | evan | ah ok. |
| 02:22:01 | sbryant | evan: so back to our LLVM love fest, what else is LLVM doing for rubinius? |
| 02:32:45 | evan | sbryant: providing a JIT is a lot |
| 02:32:48 | evan | thats it's main duty |
| 02:35:08 | sbryant | evan: makes sense |
| 17:11:49 | brixen | evan: you mentioned renaming StaticScope and VariableScope, want me to do that while I'm namespacing all these? |
| 17:29:49 | rue | `sed s/StaticScope/Ponies/g vm/**/*` |
| 17:35:00 | brixen | you forgot the kernel files |
| 17:41:55 | rue | It is more fun this way |
| 17:42:59 | brixen | we've got a lot of impl classes, I've got 11 of them namespaced |
| 19:00:04 | evan | allo |
| 19:03:08 | scoopr | halo |
| 19:03:22 | scoopr | howsit going |
| 19:03:39 | evan | oh, good |
| 19:03:44 | evan | it's the day before moving day |
| 19:03:47 | evan | when means it's moving day |
| 19:08:05 | scoopr | ah right |
| 19:08:23 | scoopr | everything packed? ;) |
| 19:08:29 | evan | not really, no. |
| 19:08:35 | evan | i'm moving just over 1 mile |
| 19:08:51 | evan | which means we've procrastinated |
| 19:08:59 | evan | thankfully, we've got awesome boxes this time |
| 19:22:40 | boyscout | Search parent VariableScopes for eval locals - 51ad981 - Evan Phoenix |
| 19:22:54 | evan | brixen: that should fix locals in irb |
| 19:24:07 | boyscout | CI: 51ad981 success. 1503 files, 7246 examples, 23660 expectations, 0 failures, 0 errors |
| 19:25:09 | brixen | evan: sweet |
| 20:57:46 | rue | Knew it |