Index

Show enters and exits. Hide enters and exits.

01:20:35boyscoutAdd ability to set thread stack size - aa660db - Evan Phoenix
01:20:35boyscoutRework capi Handles entirely - 4c6a1ef - Evan Phoenix
01:20:48boyscoutCI: Build 4c6a1ef failed. http://ci.rubini.us/rubinius/builds/4c6a1efb16afc006fbcaff727b2e6315f5d4d264
01:21:19brixenuh oh, might have missed a file
01:22:37boyscoutAdd missing files - 07dfb90 - Evan Phoenix
01:23:36rueNeat, real handles.
01:25:09evanrue: yep
01:25:15evanthey basically mimic the MRI memory model now
01:31:27boyscoutCI: 07dfb90 success. 1503 files, 7245 examples, 23659 expectations, 0 failures, 0 errors
03:42:39mdalessioHey all. I'm trying to install a gem, and getting an odd error.
03:42:42mdalessioThe command I'm running: ./bin/rbx -S gem install rake
03:42:46mdalessioThe error message: ERROR: While executing gem ... (Zlib::GzipFile::Error) invalid compressed data -- crc error
03:43:35mdalessioI'm on latest master
04:55:34brixenmdalessio: that is a weird error
04:55:51brixenI've seen it/heard of it but really don't know the cause
04:56:00brixenI think it could be your zlib
04:56:08mdalessiobrixen: agreed. just wanted to do a sanity check, that I'm not doing something obviously wrong
04:56:13brixenrbx uses FFI bindings to zlib
04:56:24brixenit's probably not anything in rbx
04:56:49brixenwhat platform are you on btw?
04:57:01mdalessioubuntu hardy
04:57:42brixenhad you seen? http://www.ruby-forum.com/topic/148609
04:59:30mdalessioi did, but nothing i could find there was really relevant. i'm looking at strace output now ...
06:34:58boyscoutFixes to run the sqlite3-ruby gem - 01bd2d1 - Evan Phoenix
06:39:56boyscoutCI: 01bd2d1 success. 1503 files, 7245 examples, 23659 expectations, 0 failures, 0 errors
07:37:46headiusanyone around that knows about the current status of the capi work?
07:39:18evanyep
07:39:22evanwhat you wanna know?
07:39:34evanbrixen and I have been working on it a bunch
09:13:46headiusevan: oh sorry, I didn't notice
09:14:07headiusI was just wondering if you had any feeling for the road forward
09:14:28headiuswe've always hoped there might be a way to load sane extensions directly, but everything I've seen tells me sane extensions are extremely rare
09:14:52headiusor that the performance impact of trying to make them work would be so bad as to invalidate the utility of loading them in the first place
09:15:33headiusI think this is going to impact all new impls that want to load C extensions....like macruby
09:15:48headiusin order for macruby to support this they'll have to do exactly what you do, which is in turn exactly what we'd have to do in JRuby
09:16:05headiusso the exploration you're doing could be applicable to all three of us, and I'm obviously interested in that
09:17:42boyscoutAdded -gc option to show GC stats. - 9d7db4b - Brian Ford
09:17:43boyscoutFixed rake build tasks using profiler_flags. - bd46f4c - Brian Ford
09:17:43boyscoutFixed profiler. - 51cb55a - Brian Ford
09:18:21boyscoutCI: Build 51cb55a failed. http://ci.rubini.us/rubinius/builds/51cb55a979779ef47707dfcf1943461103e9a5b4
09:18:49brixenfscking gcc
09:19:53kronos_vano:)
09:20:59headiusso I guess I'm mostly interested because if you're successful, it will be possible for us to implement the same set of APIs and possible reuse a lot of your capi wrapper code with a JNI equivalent
09:22:41brixenheadius: evan just implemented gc'd constant values for handles
09:22:59brixenwe're running bigdecimal, digest, and sqlite3
09:23:04brixenand readline
09:23:23brixenand mongrel
09:23:43headiushave you done any measurement of performance? I remember you guys talking recently about having to copy stuff in and out for several capi functions
09:23:51headiusthat seems like it would be a nonstarter
09:24:36brixenno, haven't really looked at perf
09:25:44headiusit's at least promising that you can load those libraries and get them to function
09:25:53brixenfucking gcc 4.3
09:26:08headiusmy other concern is that the ability to load those libraries will make people just give up on doing ports of them that probably would perform a lot better
09:26:36headiusso they'll try running stuff in rubinius or macruby or jruby with the native extension, decide it's just slower, and go back to MRI's comparatively faster extension support
09:27:06headiusI'm sure you guys see now a lot of the reasons JNI is structured the way it is
09:27:28headiuseverything has to be pushed and pulled into the managed handles because things can relocate
09:27:32slavaJNI is a bit too low-level. Would've been nice if Java came with an FFI
09:27:44headiusslava: FFI works, but you still need a way of getting data out to it
09:28:01headiusMRI's extension API, and the parts of it rbx supports, does not support any abstraction of where the data is located
09:28:37headiusso while on MRI it's fine, because there's no concurrent threading, no GC running during native code, rbx is penalized because it has to ensure the data presented to C code is fixed in memory
09:29:02headiusI've never doubted that it's possible to implement enough of the C api to get things to *work* but to work with acceptable performance is another matter entirely
09:29:33headiuswe actually had a start on our own capi layer, and came to the point where it would be necessary to copy state in and out
09:29:43headiusat that point we didn't figure it was worth continuing, since the performance would always blow
09:31:10headiusI think rbx and macruby will have the same problem, though I'd love to be proven wrong
09:32:09headiusso yeah, I would love to talk more about it
09:32:14headiuswhen you guys have time
09:35:22brixenheadius: well, we'd love to see all the gems and libraries rewritten in FFI
09:35:30brixenbut that's not happening any time soon
09:35:39headiusright, it certainly isn't
09:35:41brixenand we've had enough issues with bugs in rewritten stuff
09:35:53brixenthat it pays to run, even if running slower
09:36:09brixenI'm not sure perf will be that big an issue on basic stuff
09:36:27brixenyou should check out evan's work from the past 2 days
09:36:29headiusand fwiw I am trying to be constructive with this discussion...we feel the pain of not being able to run native extensions and would whole-heartedly support work on a common set of capi shims if it seemed like it might be feasible, both functionality and performance-wise
09:36:54brixenI'm not sure how much there is in common
09:37:05brixenit's pretty impl specific
09:37:14headiusat some level you have a representation of a handle to a ruby object and logic for getting data out of it
09:37:16headiusthat would be impl specific
09:37:31headiusbut the basic operations would be the same
09:37:42headiusgiven a ruby string, get a reference to a c-str array
09:37:45brixenwell, a lot of capi is rb_funcall
09:37:55brixenand what isn't is calls to our C++ classes
09:38:04brixenso, it's pretty impl specific
09:38:25headiusmm, so you're not building any API that abstracts your C++ logic
09:38:28brixenthe handles are gc'd now
09:38:39brixenso that's also impl specific
09:38:41headiusyes, I saw something about reference-conted
09:39:03brixennope, inside the rb_xxx functions, it calls C++ stuff directly
09:39:13brixencan't see any reason to abstract that
09:39:14headiusmmm ok...that's too bad
09:39:27headiuswell, except that if your C++ classes change in the future you'll have to fix all that
09:39:32brixensure
09:39:41headiusthe reason for abstracting anything, really
09:39:45brixena lot of stuff is rb_funcalls though
09:41:36headiusyeah, that much we had working on wmeissner's work
09:41:57headiusbut every extension we encountered needed to get access to C string data, for example, which meant copying stuff around
09:42:27headiusand some went as far as accessing hash internals using st.c functions
09:43:26boyscoutFixes for gcc 4.3. - 6becfe1 - Brian Ford
09:43:54brixenyeah, the one's that access st just plain suck ass
09:44:06headiusthere's simply no way to support them
09:44:24headiuspushing and pulling an st.c hash would be absurd
09:44:27brixennarray does RCLASS(a)->m_tbl == RCLASS(b)->m_tbl
09:44:31rueheadius: The trivial Mongrel+Rack script I wrote ran about 5% slower than MRI, meaning the difference in the C part was negligible.
09:44:33brixenas a type_of
09:44:39headiusyeah, exactly, shit like that makes me lose hope
09:45:00headiusrue: how did you isolate the time spent in C stuff?
09:45:12headiusbecause other parts could be faster and compensating
09:45:47rueWe do not run Ruby faster than MRI
09:45:48brixenheh, we don't have a lot of "faster parts" atm
09:45:51headiusbrixen: we actually used to have checks like that in JRuby internals, so I think it's a CRubyistm
09:46:01ruebrixen: Should just use 4.3
09:46:16brixenrue: 4.3 should just go commit suicide
09:46:21headiuswell, that's somewhat promising, but mongrel is probably not a good example
09:46:33headiusthe transfer across the native/ruby boundary is pretty much a single shot
09:46:41headiusand entirely from native to ruby, rather than the other way around
09:46:43brixenmongrel has a lot of global data though
09:47:01brixenit was broken before evan's handle rework
09:47:36headiusrue: can I see that script?
09:48:20ruebrixen: 4.3 works just fine, and catches quite a few problems that need catching
09:49:03boyscoutCI: 6becfe1 success. 1503 files, 7245 examples, 23659 expectations, 0 failures, 0 errors
09:49:44ruehttp://gist.github.com/87260
09:50:41headiusjust hitting it with ab or something?
09:50:55rueYeah
09:51:46headiusso yeah, I guess I've made my case...5% worse than MRI is promising
09:52:10headiusI suppose it's going to be a growing corpus of knowledge about whether this is feasible, since we don't know what other hassles are out there
09:52:23headiusbrixen: how does the capi handlg DataWrapStruct right now? I know nokogiri uses that heavily
09:52:37brixenhandles it ok
09:52:45brixencreates a Data instance
09:54:25headiusrue: what kind of numbers were you getting for MRI on this?
09:54:29headiusit seems slower than it should be
09:54:43headius92rps?
09:55:58headiushmm
09:55:58headiusrack.errors: #<Rack::Lint::ErrorWrapper:0x12c7ef4 @error=#<IO:0x2f7c4>>
09:56:04headiusI wonder if I should be seeing that
10:02:44headiusrue: without lint and showexceptions I get closer to 609 rps
10:02:53headiuscan you re-run in rbx without those?
10:19:25headiusmaybe I can re-run...does this work against master?
10:33:16headiusbrixen: you around still?
11:02:30scooprheh http://gist.github.com/99691
12:29:43headiusrue: FYI, I did get that script to run on rbx and wasn't able to reproduce your performance observation
12:29:45headiusit also crashed: http://gist.github.com/99739
12:30:08headiusmaybe I'm doing something wrong with it...I just did bin/rbx gem install mongrel and gem install rack and then ran my modified version of your script that doesn't use lint or showexceptions
12:31:42headiusit seems to die some time after 800 requests
12:32:05headiuswith 500 requests I can get about 96rps
12:32:15headiusMRI does in the neighborhood of 600
12:32:26headiuslet me know if there's something else I should be doing to run this
14:56:37rueI went to the zo
14:56:44rueZoo, too. Both were fun
15:18:31scooprnice
20:30:11boyscoutDeclare Time#initialize private instead of public. - c214841 - Tilman Sauerbeck
20:30:12boyscoutUnMarshal's methods are private, except unmarshal(). - 65adf3f - Tilman Sauerbeck
20:34:33boyscoutCI: 65adf3f success. 1503 files, 7245 examples, 23659 expectations, 0 failures, 0 errors
21:56:52boyscoutFix latent parsing problem - 2242394 - Evan Phoenix
22:03:55boyscoutCI: 2242394 success. 1503 files, 7245 examples, 23659 expectations, 0 failures, 0 errors
23:16:14boyscoutChanged profiler cumulative seconds method. - dd65a98 - Brian Ford
23:23:36boyscoutCI: dd65a98 success. 1503 files, 7245 examples, 23659 expectations, 0 failures, 0 errors